Mirroring support method, mirroring support device, and program

Abstract

A mirroring support method according to an embodiment of the present invention supports the mirroring of a speaker by a listener in a bilateral communication via video. In the mirroring support method, a computer performs a feeling estimation step of estimating a feeling of the speaker and a feeling of the listener, an intermediate feeling calculation step of calculating an intermediate feeling between the feeling of the speaker and the feeling of the listener, a change amount calculation step of calculating a change amount necessary to convert a facial expression corresponding to the feeling of the listener into a facial expression corresponding to the intermediate feeling for each of one or more feature points set on a face image of the listener, and a facial expression conversion step of converting the facial expression corresponding to the feeling of the listener by using the change amount.

Description

[Technical field]

[0001] The present invention relates to a mirroring support method, a mirroring support device, and a program. [Background technology]

[0002] There is known a video chat system that can express emotions in sync with the interlocutor during remote communication such as at a customer center or in a remote class (see, for example, Non-Patent Document 1). In the video chat system described in Non-Patent Document 1, for example, when the speaker smiles, the listener's facial expression is also transformed into a smile. This is a pseudo-generation of a psychological technique known as "mirroring," which imitates the facial expression of the interlocutor, and has been shown to have the effect of facilitating smooth communication. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Suzuki, K., Yokoyama, M., Yoshida, S., Mochizuki, T., Nunobiki, J., Narumi, T., Tanigawa, T., and Hirose, M. (2018). Extending remote communication using synchronized facial deformation technology. Transactions of the Information Processing Society of Japan, 59(1), 52-60. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, when the listener's facial expression is converted to the speaker's facial expression, if the converted facial expression is too different from the listener's emotion, the facial expression conversion may feel unnatural. This is thought to be because the non-verbal elements other than the converted facial expression are based on the listener's emotion.

[0005] For example, if a speaker is smiling while a listener is feeling sad, the listener may respond with a nonchalant tone of voice, such as "That's right..." In this case, the listener's facial expression is transformed into a smile, but the listener responds with a nonchalant tone of voice, which is likely to make the speaker feel uncomfortable.

[0006] An embodiment of the present invention has been made in consideration of the above points, and has an object to realize mirroring that causes less awkwardness during communication. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, a mirroring assistance method of one embodiment is a mirroring assistance method that supports a listener in mirroring a speaker in two-way communication via video, and includes a computer-executed emotion estimation step of estimating the emotion of the speaker and the emotion of the listener, an intermediate emotion calculation step of calculating an intermediate emotion between the emotion of the speaker and the emotion of the listener, a change amount calculation step of calculating, for each of one or more feature points set on a facial image of the listener, an amount of change for converting a facial expression corresponding to the listener's emotion into an facial expression corresponding to the intermediate emotion, and an expression conversion step of converting the facial expression corresponding to the listener's emotion using the amount of change. Effect of the Invention

[0008] It is possible to achieve mirroring that creates a natural feeling when communicating. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a mirroring support apparatus according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of a mirroring support device according to the present embodiment. [Diagram 3] FIG. 13 is a diagram illustrating an example of an emotion vector. [Figure 4]10 is a flowchart illustrating an example of a mirroring assistance process according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described. In this embodiment, a mirroring support device 10 capable of realizing mirroring that is less awkward during communication (particularly remote communication) such as video chat or video call will be described.

[0011] Here, as described above, when the listener's facial expression is converted to the speaker's facial expression, if the converted facial expression and the listener's emotion are far apart, an awkward feeling will be generated. Therefore, in the mirroring support device 10 according to the present embodiment, the listener's facial expression is not converted to the speaker's facial expression itself, but the listener's facial expression is converted to an expression corresponding to an emotion between the listener's and the speaker's emotions after estimating them. This reduces the listener's awkwardness in response to the facial expression conversion, and increases the mirroring effect on the speaker, enabling smooth communication. In addition, by using the mirroring support device 10 according to the present embodiment, the listener's emotional labor burden, such as suppressing his / her own emotions and expressing different emotions, can be reduced.

[0012] The mirroring support device 10 according to the present embodiment is used in combination with a system for remote communication such as video chat or video call between two people. In this case, the terminal used by the speaker may function as the mirroring support device 10, the terminal used by the listener may function as the mirroring support device 10, or both of these terminals may function as the mirroring support device 10. In addition, a server or the like that realizes a video chat or video call between the terminal used by the speaker and the terminal used by the listener may function as the mirroring support device 10.

[0013] It should be noted that the distinction between speaker and listener is a matter of convenience, and during a dialogue or conversation between two people, the same person may sometimes be the speaker and sometimes be the listener.

[0014] <Hardware Configuration of the Mirroring Support Device 10> The hardware configuration of the mirroring support device 10 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, the mirroring support device 10 according to this embodiment is realized by the hardware configuration of a general computer or computer system, and has an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a processor 105, and a memory device 106. Each of these hardware components is connected to each other via a bus 107 so as to be able to communicate with each other.

[0015] The input device 101 is, for example, a keyboard, a mouse, a touch panel, various physical buttons, etc. The display device 102 is, for example, a display, etc. Note that the mirroring support device 10 does not necessarily have to have at least one of the input device 101 and the display device 102, for example.

[0016] The external I / F 103 is an interface with an external device such as a recording medium 103a. The mirroring support device 10 can read and write data from and to the recording medium 103a via the external I / F 103. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), a SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0017] The communication I / F 104 is an interface for connecting the mirroring support device 10 to a communication network. The processor 105 is, for example, various types of arithmetic devices such as a central processing unit (CPU) or a graphics processing unit (GPU). The memory device 106 is, for example, various types of storage devices such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a random access memory (RAM), a read only memory (ROM), etc.

[0018] The mirroring support device 10 according to this embodiment has the hardware configuration shown in Fig. 1, and is therefore capable of implementing the mirroring support process described below. Note that the hardware configuration shown in Fig. 1 is merely an example, and the mirroring support device 10 may have, for example, multiple processors 105 or multiple memory devices 106. Furthermore, the mirroring support device 10 may have various other hardware in addition to the hardware shown in the figure.

[0019] <Functional configuration of the mirroring support device 10> The functional configuration of the mirroring support device 10 according to this embodiment is shown in Fig. 2. As shown in Fig. 2, the mirroring support device 10 according to this embodiment has an emotion estimation unit 201, an emotion vectorization unit 202, an intermediate emotion calculation unit 203, a change amount calculation unit 204, and a facial expression conversion unit 205. Each of these units is realized, for example, by processing executed by the processor 105 of one or more programs installed in the mirroring support device 10.

[0020] Moreover, the mirroring support apparatus 10 according to this embodiment has a change amount database 206. The change amount database 206 is realized by, for example, an auxiliary storage device such as an HDD or SSD. Note that the change amount database 206 may also be realized by, for example, a storage device (such as a database server) connected to the mirroring support apparatus 10 via a communication network.

[0021] The emotion estimation unit 201 estimates the emotions of the speaker and the listener, respectively. Here, the emotion estimation unit 201 may estimate the emotions of the speaker and the listener, respectively, using known technology. For example, the emotion estimation unit 201 may estimate the emotion of a person (speaker, listener) using image data of the face of the person by emotion recognition technology (https: / / emotion-ai.userlocal.jp / face). With this emotion recognition technology, a value that quantifies each of five emotions, "happiness," "anger," "sadness," "surprise," and "neutral," is obtained as an estimation result of the person's emotion. Therefore, in this case, for example, the emotion with the highest value may be estimated as the emotion of the person.

[0022] Note that the use of the above emotion recognition technology is just one example. In addition to technology that estimates emotions from data of an image of a person's face (hereinafter also referred to as a facial image), for example, technology that estimates a person's (speaker, listener) emotions from the results of converting the person's (speaker's or listener's) voice into text using voice recognition may be used, or emotions may be estimated by combining multiple known technologies.

[0023] The emotion vectorization unit 202 vectorizes the emotions of the speaker and the listener estimated by the emotion estimation unit 201. Hereinafter, the vectorized emotions are referred to as "emotion vectors". In addition, the emotion vector is a vector whose size is the radius of the emotion circle model of Russell's emotion circle model and whose direction is the direction from the origin to the emotion when the emotion estimated by the emotion estimation unit 201 is mapped onto the emotion circle model of Russell. In addition, the Russell emotion circle model is a model in which emotions (such as "surprise", "excitement", "happiness", "joy", "happy", "satisfied", "easy", "relaxed", "boredom", "slack", "boredom", "depression", "sadness", "unpleasant", "irritated", "dissatisfied", "anger", "worry", "fear", etc.) are mapped in a two-dimensional space with the axes of emotional valence and arousal level. For details of the Russell emotion circle model, please refer to, for example, Reference 1.

[0024] FIG. 3 shows an example of emotion vectors when the speaker's emotion is estimated to be "joy" and the listener's emotion is estimated to be "sadness." As shown in FIG. 3, the speaker's emotion vector is a vector whose size is the radius of the emotion circle model and whose orientation is the direction from the origin to the emotion "joy." Similarly, the listener's emotion vector is a vector whose size is the radius of the emotion circle model and whose orientation is the direction from the origin to the emotion "sadness." The size of the radius of the emotion circle model can be set arbitrarily; for example, radius = 1.

[0025] The intermediate emotion calculation unit 203 calculates an intermediate vector (hereinafter referred to as intermediate emotion vector) between the speaker's emotion vector and the listener's emotion vector obtained by the emotion vectorization unit 202. Here, the intermediate emotion calculation unit 203 calculates the sum of the emotion vectors normalized by its magnitude as the intermediate emotion vector, for example, using the idea of ​​resultant force described in Reference 2. Specifically, for example, when the radius of the emotion circle model is 1, the intermediate emotion calculation unit 203 calculates the sum of the speaker's emotion vector and the listener's emotion vector normalized by its magnitude (that is, the sum of the speaker's emotion vector and the listener's emotion vector is calculated and the magnitude is set to 1) as the intermediate emotion vector. As a result, a vector that represents an emotion intermediate between the speaker's emotion and the listener's emotion is obtained as the intermediate emotion vector.

[0026] An example of an intermediate emotion vector between a speaker's emotion vector representing the emotion "joy" and a listener's emotion vector representing the emotion "sadness" is shown in Fig. 3. In the example shown in Fig. 3, the intermediate emotion vector represents the emotion "relaxed."

[0027] The change amount calculation unit 204 refers to the change amount database 206 and calculates the amount of change in feature points for converting the listener's facial expression into an expression corresponding to the emotion represented by the intermediate emotion vector calculated by the intermediate emotion calculation unit 203. Here, feature points are points that characterize a person's facial expression on a face image, such as points on the face outline, points on the eyes and their outline, points on the nose and their outline, points on the mouth and their outline, points on the eyebrow outline, etc. These feature points are set in advance on the listener's face image. Please refer to, for example, Non-Patent Document 1 for information on what points on a face image can be set as feature points.

[0028] The facial expression conversion unit 205 changes each feature point on the listener's face image using the amount of change in each feature point calculated by the change amount calculation unit 204. In this way, the listener's facial expression is converted into an expression that corresponds to the emotion represented by the intermediate emotion vector.

[0029] The change amount database 206 stores the change amount of each feature point for converting a neutral expression into an expression corresponding to each emotion. Here, the change amount of each feature point is calculated in advance by a known method and stored in the change amount database 206. One such method is to collect an image of a listener with a neutral expression and an image of the listener with an expression corresponding to each emotion, and then calculate the change amount of each feature point from these images. Alternatively, the change amount of each feature point may be calculated by performing an experiment equivalent to the "design experiment of facial expression deformation amount" described in Non-Patent Document 1.

[0030] The change amount database 206 may store the change amount of each feature point for converting a facial expression corresponding to each emotion into a facial expression corresponding to another emotion.

[0031] <Mirroring support process> The mirroring support process according to this embodiment will be described with reference to Fig. 4. In the following, it is assumed that a face image of a speaker and a face image of a listener (including a moving image in which these face images are frame images) are provided to the mirroring support device 10. The following mirroring support process is repeatedly executed at predetermined time intervals while remote communication such as a video chat or a video call is being carried out between the speaker and the listener.

[0032] First, the emotion estimation unit 201 estimates the emotion of the speaker and the emotion of the listener, respectively, using a face image of the speaker and a face image of the listener (step S101).

[0033] Next, the emotion vectorization unit 202 vectorizes the speaker's emotion and the listener's emotion estimated in the above step S101 (step S102), thereby obtaining the speaker's emotion vector and the listener's emotion vector.

[0034] Next, the intermediate emotion calculation unit 203 calculates an intermediate emotion vector using the speaker's emotion vector and the listener's emotion vector obtained in step S102 above (step S103).

[0035] Next, the change amount calculation unit 204 refers to the change amount database 206 and calculates the change amount of the feature points for converting the listener's facial expression into an expression corresponding to the emotion represented by the intermediate emotion vector calculated in step S103 above (step S104).

[0036] For example, when the emotion represented by the listener's emotion vector is "emotion A" and the emotion represented by the intermediate emotion vector is "emotion B", the change amount calculation unit 204 refers to the change amount database 206 and calculates the change amount of each feature point for converting the facial expression corresponding to "emotion A" to the facial expression corresponding to "emotion B". For example, when the change amount of each feature point for converting a neutral expression to a facial expression corresponding to each emotion is stored in the change amount database 206, if the change amount for converting a certain feature point from a neutral expression to a facial expression corresponding to "emotion A" is Δ1=(Δx1, Δy1) and the change amount for converting a neutral expression to a facial expression corresponding to "emotion B" is Δ2=(Δx2, Δy2), it is possible to calculate ||Δ2-Δ1||2 using the L2 norm. In this way, the change amount of the feature point representing the content such as "lower the feature point corresponding to the corner of the mouth by 1 mm" is calculated.

[0037] Then, the facial expression conversion unit 205 changes each feature point on the listener's face image by using the amount of change of each feature point calculated in the above step S104 (step S105). That is, the facial expression conversion unit 205 changes the position of each feature point on the listener's face image by the amount of change of the feature point by using the amount of change of each feature point calculated in the above step S104. This converts the listener's facial expression into an expression corresponding to the emotion represented by the intermediate emotion vector. The facial image of this converted expression is displayed on the speaker's terminal. Note that the function of displaying the facial image of the converted expression on the speaker's terminal may be included in the mirroring support device 10, or may be included in a device, equipment, system, program, etc. different from the mirroring support device 10.

[0038] Generally, it is considered that the emotions of the speaker and the listener do not change for several to several tens of seconds. Therefore, the change of each feature point in the above step S105 may be performed for, for example, a facial image of the listener for several to several tens of seconds. In this case, for example, the above steps S101 to S104 are performed every time span of several to several tens of seconds, and the above step S105 is performed for each facial image of the listener (i.e., a facial image included as a frame image in the video).

[0039] <Summary> As described above, the mirroring support device 10 according to the present embodiment considers the emotions of the speaker and the listener, and converts the listener's expression into an expression that represents the intermediate emotion therebetween. As a result, the discomfort of the speaker with respect to the expression conversion of the listener is reduced, the mirroring effect on the speaker can be increased, and smooth communication can be achieved. In addition to this, it is also possible to reduce the burden of the listener's emotional labor, such as the listener suppressing their own emotions and expressing different emotions.

[0040] <Modification Example> Hereinafter, a modification example of the present embodiment will be described.

[0041] <<Modification Example 1>> In the present embodiment, Russell's emotion circular model is used as the emotion model, and emotion vectors are defined on this emotion circular model. However, the present invention is not limited to this, and emotion vectors may be defined using other emotion models.

[0042] <<Modification Example 2>> In the present embodiment, the emotion that is intermediate between the emotions of the two parties is calculated using emotion vectors. However, the emotion that is intermediate between the emotions of the two parties may be calculated by other methods (for example, an index for calculating the intermediate emotion from the emotions of the two parties).

[0043] <<Modification Example 3>> In the present embodiment, the emotion that is intermediate between the emotions of the two parties is calculated. However, the present invention is not limited to this, and any emotion between the emotions of the two parties may be used, not necessarily the intermediate emotion. For example, assuming that the emotion vector of the speaker is v1, the emotion vector of the listener is v2, and the parameter is a (where 0 < a < 1), av1 + (1 - a)v2 may be used as the intermediate emotion vector.

[0044] Further, for example, as time passes, an emotion approaching the emotion of the speaker or the listener may be calculated as an intermediate emotion between the two. For example, taking a parameter whose value increases according to the elapsed time as t (where 0 < t ≤ 1), tv1 + (1 - t)v2 may be used as the intermediate emotion vector, or (1 - t)v1 + tv2 may be used as the intermediate emotion vector.

[0045] The present invention is not limited to the above-described embodiments specifically disclosed, and various modifications, changes, combinations with known technologies, etc. are possible without departing from the scope of the claims.

[0046] [References] Reference 1: J. A. Russell, "A circumplex model of affect." Journal of personality and social psychology, vol. 39, no. 6, p. 1161, 1980. Reference 2: Reiko Aruga, Junji Watanabe, Sumihisa Nukumi: Impression evaluation regarding emotional expression of agents by scaling of figures, Proceedings of the Human Interface Symposium 2017 (2017).

Explanation of Signs

[0047] 10 Mirroring Support Device 101 Input Device 102 Display Device 103 External I / F 103a Recording Medium 104 Communication I / F 105 Processor 106 Memory Device 107 Bus 201 Emotion Estimation Unit 202 Emotion Vectorization Unit 203 Intermediate Emotion Calculation Unit 203 204 Change Amount Calculation Unit 205 Facial Expression Conversion Unit 206 Change Amount Database

Claims

1. A method for supporting mirroring of a listener to a speaker in two-person video communication, comprising: an emotion estimation step of estimating the emotion of the speaker and the emotion of the listener; an intermediate emotion calculation step for calculating an intermediate emotion between the emotion of the speaker and the emotion of the listener; a change amount calculation step of calculating a change amount for converting a facial expression corresponding to the emotion of the listener into a facial expression corresponding to the intermediate emotion, for each of one or more feature points set on the face image of the listener; a facial expression conversion step of converting a facial expression corresponding to the emotion of the listener using the amount of change; The computer can perform mirroring assistance methods.

2. further executing an emotion vectorization step of generating a first emotion vector by vectorizing the emotion of the speaker and a second emotion vector by vectorizing the emotion of the listener; The intermediate emotion calculation step includes: The mirroring assistance method according to claim 1 , further comprising the step of calculating an emotion represented by a third emotion vector calculated by adding the first emotion vector and the second emotion vector as the intermediate emotion.

3. The intermediate emotion calculation step includes: The first emotion vector is v １ , the second emotion vector v ２ , parameter a (where 0<a<1), and a １ +(1-a)v ２ The method for assisting mirroring according to claim 2 , further comprising the step of calculating the third emotion vector as the third emotion vector.

4. The intermediate emotion calculation step includes: The first emotion vector is v １ , the second emotion vector v ２ , t is a parameter whose value increases with the passage of time of the video (where 0<t≦1), and tv １ +(1-t)v ２ , or (1-t)v １ +tv ２ The method for assisting mirroring according to claim 2 , further comprising the step of calculating the third emotion vector as the third emotion vector.

5. The change amount calculation step includes:

5. A mirroring assistance method according to claim 1, further comprising the steps of: referring to a database in which is stored, for each of the one or more feature points, the amount of change required to convert a neutral expression into an expression corresponding to each emotion, and calculating the amount of change required to convert an expression corresponding to the listener's emotion into an expression corresponding to the intermediate emotion.

6. A mirroring support device that supports a listener in mirroring a speaker in two-person video communication, comprising: an emotion estimation unit that estimates an emotion of the speaker and an emotion of the listener; an intermediate emotion calculation unit that calculates an intermediate emotion between the emotion of the speaker and the emotion of the listener; a change amount calculation unit that calculates a change amount for converting a facial expression corresponding to the emotion of the listener into a facial expression corresponding to the intermediate emotion, for each of one or more feature points set on the face image of the listener; a facial expression conversion unit that converts a facial expression corresponding to the emotion of the listener using the amount of change; A mirroring support device having the above configuration.

7. A program for causing a computer to execute the mirroring support method according to any one of claims 1 to 5.

Images