Emotion expression adding method, emotion expression adding device, and program

Abstract

The present invention facilitates understanding of the emotion of a person in a video by causing a computer to execute: an identification procedure in which a form of expression that is easy for a user to understand in relation to a given emotion of a person is identified on the basis of information inputted by the user; and an addition procedure in which, if it is estimated that a person included in a video displayed for the user has expressed the given emotion, information indicating the given emotion in the form of expression identified through the identification procedure is added to the video.

Description

[Technical field]

[0001] The present invention relates to an emotional expression adding method, an emotional expression adding device, and a program. [Background technology]

[0002] Emotions are what people feel in response to certain events, and people tend to interpret certain emotions differently. In communication, the receiver of emotional information infers the sender's emotions from linguistic information and non-verbal information such as tone of voice and gestures, but it is not always easy to understand the emotions as intended by the sender of emotional information. In the case of web conferences, it becomes more difficult because there is less information that can be used to read the sender's emotions compared to face-to-face conferences.

[0003] A technology has been proposed that estimates emotions, creates CG effects for those emotions, and then adds those CG effects to the sender's video to emphasize the emotion, thereby communicating the emotion in an easy-to-understand way to the receiver. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Daiki Yokoyama, Sachiko Kodama, "Emotion FX: An application for automatically generating CG effects that emphasize facial emotions in videos in real time", [online], Internet<URL:http: / / www.interaction-ipsj.org / proceedings / 2020 / data / pdf / 1A-10.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] In conventional technology, emotions can be conveyed to the receiver in an easy-to-understand manner by adding CG effects corresponding to the estimated emotion. However, because not all receivers will necessarily have the same understanding of the added information, conventional technology that adds the effects uniformly does not make it easier to convey the same emotion to everyone. In other words, while one receiver may understand the sender's intention, another receiver may receive it differently from the sender's intention, or it may make it more difficult for the receiver to understand.

[0006] The present invention has been made in consideration of the above-mentioned points, and has an object to make it easier to understand the emotions of people in a video. [Means for solving the problem]

[0007] In order to solve the above problem, based on information input by a user, Identifying a characteristic of the user from among predetermined characteristics, and based on the identified characteristic, The computer executes a specification step of specifying an expression manner in which a certain human emotion is easily understandable to the user, and an addition step of adding information indicating the certain emotion in the expression manner specified by the specification step to the video when it is estimated that a person included in the video displayed to the user has expressed the certain emotion. Effect of the Invention

[0008] This makes it easier to understand the emotions of people in the video. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a hardware configuration of an emotional expression adding device 10 according to a first embodiment. [Diagram 2] 1 is a diagram illustrating an example of a functional configuration of an emotional expression adding device 10 according to a first embodiment. [Diagram 3] FIG. 4 is a diagram showing an example of a correspondence table between user characteristics and additional contents in the first embodiment. [Figure 4]FIG. 11 is a diagram illustrating an example of a functional configuration of an emotional expression adding device 10 according to a second embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of a questionnaire screen. [Figure 6] FIG. 11 is a diagram illustrating an example of evaluation information. [Figure 7] FIG. 13 is a diagram illustrating an example of a functional configuration of an emotional expression adding device 10 according to a third embodiment. [Figure 8] FIG. 13 is a diagram showing an example of a set of additional content candidates corresponding to a user characteristic “extroversion: high” in the third embodiment. [Figure 9] FIG. 13 is a diagram illustrating an example of a functional configuration of an emotional expression adding device 10 according to a fourth embodiment. [Figure 10] FIG. 23 is a diagram showing an example of a set of additional content candidates corresponding to a user characteristic “extroversion: high” in the fourth embodiment. [Figure 11] FIG. 13 is a diagram illustrating an example of a sum result. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of a hardware configuration of an emotional expression adding device 10 in a first embodiment. The emotional expression adding device 10 in Fig. 1 has a drive device 100, an auxiliary storage device 102, a memory device 103, a CPU 104, an interface device 105, a display device 106, an input device 107, etc., which are all connected to each other via a bus B.

[0011] A program for implementing the processing in the emotional expression adding device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 via the drive device 100 into the auxiliary storage device 102. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0012] The memory device 103 reads out and stores the program from the auxiliary storage device 102 when an instruction to start the program is received. The CPU 104 realizes the functions related to the emotional expression adding device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network. The display device 106 displays a GUI (Graphical User Interface) or the like according to a program. The input device 107 is composed of a keyboard, a mouse, etc., and is used to input various operational instructions.

[0013] In this embodiment, the emotional expression adding device 10 is a terminal used by each user who communicates through video transferred over a network, such as in a Web conference. However, the emotional expression adding device 10 may also be a server (for example, a group of computers on the cloud side) that relays a Web conference. In this embodiment, a user who transmits video (the user who conveys emotion through the video) is particularly called a "sender." Also, when simply referring to a "user," the user in question means a receiver.

[0014] Fig. 2 is a diagram showing an example of a functional configuration of the emotional expression adding device 10 in the first embodiment. In Fig. 2, the emotional expression adding device 10 has a user characteristic identification unit 11, a characteristic addition content selection unit 12, a sender emotion estimation unit 13, a video adding unit 14, and a video display unit 15. Each of these units is realized by a process in which one or more programs installed in the emotional expression adding device 10 are executed by the CPU 104.

[0015] The user characteristic identification unit 11 identifies the characteristics of a user (hereinafter, referred to as "user characteristics") based on information input by the user who is the recipient of the emotion. The user characteristics refer to the characteristics of the user's way of feeling and thinking about an event.

[0016] Specifically, the user characteristic identification unit 11 accepts responses from the user to a questionnaire such as Big Five or a cognitive characteristic test when, for example, a program that causes a computer to function as the user characteristic identification unit 11 is initially started, and identifies the user characteristic based on the responses. For example, the user characteristic identification unit 11 selects one type (type) as the user characteristic from a plurality of types (types) that are predetermined (classified) for human characteristics, such as "extroversion: high", "openness: high", "conscientiousness: high", etc. Note that the type of user characteristic in this embodiment is based on the Big Five. Identification of user characteristics from responses to questionnaires such as the Big Five or cognitive characteristic tests may be performed based on the information disclosed in, for example, "Namikawa Tsutomu, Tani Iori, Wakita Takafumi, Kumagai Ryuichi, Nakane Ai, Noguchi Hiroyuki, Development of a shortened version of the Big Five scale and examination of its reliability and validity, Psychological Research, Vol. 83, No. 2, pp. 91-99, https: / / www.jstage.jst.go.jp / article / jjpsy / 83 / 2 / 83_91 / _pdf / -char / ja".

[0017] The characteristic addition content selection unit 12 selects a set of contents (hereinafter referred to as "additional contents") to be added to the video in accordance with the sender's emotions in order to express various emotions of the sender based on the user characteristics (hereinafter referred to as "target user characteristics") specified by the user characteristic specification unit 11. Such selection is performed based on a correspondence table between user characteristics and addition content sets, for example, as shown in FIG. 3. The characteristic addition content selection unit 12 selects an addition content set (additional contents for each emotion of the sender) included in a row corresponding to the target user characteristics in the correspondence table. For example, when the target user characteristic is "openness: high", the addition content set in the second row (an addition content set in which "quarter note" is happiness, "face with crying left eye" is sadness, and "blue veins" is anger) is selected. That is, since the most easily understandable addition content is considered to differ depending on the user characteristics, different addition contents are selected even if the contents express the same emotion. For example, FIG. 3 is based on an example in which a heart mark is easier to understand as happiness than a musical note mark for a person with "openness: high". Hereinafter, the set of additional contents selected by the characteristic additional content selection unit 12 will be referred to as a "set of target additional contents."

[0018] The sender emotion estimation unit 13 estimates the emotion expressed by the sender from the video of the sender (video including the sender as a subject) periodically (for example, every 0.1 second video section) during a Web conference, for example. The emotion estimation based on the video can be performed using a known technology such as "Cognitive Services Face API (https: / / azure.microsoft.com / ja-jp / services / cognitive-services / face / #demo)". The sender emotion information, which is the estimation result of the sender emotion estimation unit 13, may include not only the type of emotion (joy, sadness, anger, etc.) but also the intensity of the emotion (low, medium, high, etc.).

[0019] The video adding unit 14 identifies, from the set of target added contents, added contents corresponding to the (sender's) emotion indicated by the sender emotion information output from the sender emotion estimating unit 13, as an expression form that makes it easy for the user to understand the emotion. The video adding unit 14 adds (superimposes or synthesizes) the identified added contents to the sender's video by image processing, thereby generating a video to which the added contents have been added (hereinafter referred to as a "video with added contents").

[0020] The video display unit 15 displays the video with the additional content on the display device 106 .

[0021] As described above, according to the first embodiment, the additional content changes according to the characteristics of the receiver with respect to a certain emotion of the sender. Therefore, the emotion can be transmitted according to the characteristics of the receiver, and the emotion of the sender can be easily conveyed to each receiver. In other words, the emotion of the person in the video can be easily understood.

[0022] Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be described. Points not specifically mentioned in the second embodiment may be the same as those in the first embodiment.

[0023] In the second embodiment, a method is disclosed in which user evaluation information on a video with additional content is obtained in advance, optimal additional content is selected based on the evaluation information, and the selected additional content is used to express the sender's feelings.

[0024] Fig. 4 is a diagram showing an example of a functional configuration of the emotional expression adding device 10 in the second embodiment. In Fig. 4, the same parts as or corresponding parts to Fig. 2 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0025] 4, the emotional expression adding device 10 further includes an evaluation information acquiring unit 16 and an evaluation addition content selecting unit 17. These units are realized by processing executed by the CPU 104 of one or more programs installed in the emotional expression adding device 10. It should be noted that the emotional expression adding device 10 in the second embodiment does not include the user characteristic identifying unit 11.

[0026] The evaluation information acquisition unit 16 identifies the emotion that the user understands for each additional content based on the information input by the user who is the recipient of the emotion. For example, the evaluation information acquisition unit 16 displays a questionnaire screen when a program that causes a computer to function as the evaluation information acquisition unit 16 is initially started, and acquires information (hereinafter referred to as "evaluation information") indicating how the user understands each additional content through the questionnaire screen.

[0027] FIG. 5 is a diagram showing an example of a questionnaire screen. As shown in FIG. 5, the questionnaire screen 510 includes a video display area 511 and a confidence level input area 512. The evaluation information acquisition unit 16 displays (plays) in the video display area 511 a video randomly selected from a plurality of videos (videos with added content) in which a person is shown and in which added content has been previously synthesized. For example, for each type of emotion and for each of a plurality of types of added content, a plurality of videos in which the added content has been synthesized for a video of a person having the facial expression of the emotion are created in advance, and the plurality of videos are displayed in a random order. The user (receiver) refers to the person (facial expression) in the video displayed in the video display area 511 and the added content synthesized with the video, and inputs the degree of ease of understanding of various emotions from the video (the facial expression of the person and the added content) (hereinafter referred to as "confidence level") in the confidence level input area 512. The evaluation information acquisition unit 16 repeats displaying random images in the image display area 511 and accepting input of confidence levels for various emotions for the images. As a result, evaluation information as shown in Fig. 6 is obtained. As shown in Fig. 6, the evaluation information is information indicating the confidence levels for various emotions for each added content. Added content with a higher confidence level for a certain emotion indicates that the emotion is easier for the user to understand.

[0028] The evaluation addition content selection unit 17 selects the optimum addition content for the user for each type of emotion (i.e., for each column in the table in FIG. 6) based on the evaluation information (FIG. 6) acquired by the evaluation information acquisition unit 16. As an example of a method for selecting addition content for a certain emotion, two specific examples (1) and (2) are shown below.

[0029] (1) Select the additional content with the highest confidence level. This is because the highest confidence level is considered to be the emotion that is easiest for the user to understand. In the case of FIG. 6, for example, for joy, the additional content in the first row is selected.

[0030] (2) For each confidence level, the difference between the highest confidence level among different emotions with the same added content is taken (if the added content has the highest confidence level, the next highest value is taken; otherwise, the highest confidence level is taken). The added content with the highest difference among the same emotion type is selected. This is because if the added content has a higher confidence level than other emotion types, it is thought that it will be less likely to be understood as a different emotion. Therefore, in the case of Figure 6, the added content in the second row is selected for joy.

[0031] The result of selection by any one of the above methods for each emotion is set as a target additional content set in the second embodiment.

[0032] Therefore, in the second embodiment, the video adding unit 14 specifies the additional content corresponding to the (sender's) emotion indicated by the sender emotion information output from the sender emotion estimating unit 13, from among the target additional content set selected by the evaluation additional content selecting unit 17, as an expression form that makes the emotion easy for the user to understand. The video adding unit 14 adds (superimposes or synthesizes) the specified additional content to the sender's video by image processing, thereby generating an additional content-added video to which the additional content has been added.

[0033] The rest is similar to the first embodiment.

[0034] As described above, according to the second embodiment, it is possible to obtain the same effects as those of the first embodiment.

[0035] Next, the third embodiment will be described. In the third embodiment, differences from the first or second embodiment will be described. Points not specifically mentioned in the third embodiment may be the same as those in the first embodiment. In the third embodiment, a first method in which the first and second embodiments are combined is disclosed.

[0036] Fig. 7 is a diagram showing an example of a functional configuration of an emotional expression adding device 10 according to the third embodiment. In Fig. 7, the same parts as or corresponding parts to Fig. 2 or Fig. 4 are given the same reference numerals, and description thereof will be omitted as appropriate.

[0037] The characteristic addition content selection unit 12 selects (specifies) a set of candidates for the addition content for each emotion (hereinafter, referred to as an “addition content candidate set”) based on the target user characteristic specified by the user characteristic specification unit 11.

[0038] A set of additional content candidates is created in advance for each user characteristic. Fig. 8 is a diagram showing an example of a set of additional content candidates corresponding to a user characteristic of "extroversion: high" in the third embodiment. As shown in Fig. 8, the set of additional content candidates includes a set of additional content candidates for each type of emotion.

[0039] The characteristic additional content selection unit 12 selects one additional content candidate set corresponding to the target user characteristic from the additional content candidate sets prepared for each characteristic. Hereinafter, the additional content candidate set selected by the characteristic additional content selection unit 12 is referred to as the "target additional content candidate set."

[0040] For example, when a program that causes a computer to function as the evaluation information acquisition unit 16 is initially started, the evaluation information acquisition unit 16 acquires, for each type of emotion, evaluation information indicating how the user understands each candidate of additional content included in the target additional content candidates for that type of emotion. Specifically, the evaluation information acquisition unit 16 accepts input of evaluation information (FIG. 6) for each type of emotion and for each candidate by displaying a questionnaire screen 510 shown in FIG. 5, in which a video of a person having an expression of that emotion is synthesized with each candidate of additional content corresponding to that emotion from the set of target additional content candidates.

[0041] The rest is the same as in the second embodiment.

[0042] As described above, according to the third embodiment, it is possible to obtain the same effects as those of the above embodiments.

[0043] Next, a fourth embodiment will be described. In the fourth embodiment, differences from the first to third embodiments will be described. Points not specifically mentioned in the fourth embodiment may be the same as the first to third embodiments. In the fourth embodiment, a second method in which the first and second embodiments are combined is disclosed.

[0044] Fig. 9 is a diagram showing an example of a functional configuration of an emotional expression adding device 10 according to the fourth embodiment. In Fig. 9, the same parts as or corresponding parts to Fig. 2, Fig. 4 or Fig. 7 are given the same reference numerals, and description thereof will be omitted as appropriate.

[0045] 9, the emotional expression adding device 10 further includes a synthesis unit 18. The synthesis unit 18 is realized by a process in which one or more programs installed in the emotional expression adding device 10 are caused to be executed by the CPU 104.

[0046] The characteristic additional content selection unit 12 selects a set of additional content candidates based on the target user characteristics identified by the user characteristic identification unit 11 .

[0047] A set of additional content candidates is created in advance for each user characteristic. FIG. 10 is a diagram showing an example of a set of additional content candidates corresponding to the user characteristic "extroversion: high" in the fourth embodiment. As shown in FIG. 10, in the fourth embodiment, a set of additional content candidates corresponding to a certain user characteristic includes, for each type of emotion, a set of additional content candidates for the emotion and a score for each candidate for a person having the user characteristic. The numerical value in each column of the first row of FIG. 10 is the score for the additional content candidate corresponding to the column. The score is a numerical value indicating the degree of ease of understanding of each additional content for a person having the characteristic depending on the magnitude of the value, and the larger the value, the easier it is to understand.

[0048] A table (additional content candidate set) as shown in FIG. 10 can be obtained by, for example, having subjects having the user characteristics answer a questionnaire conducted by the evaluation information acquisition unit 16 for each user characteristic. Specifically, about 10 to 15 subjects are collected for each user characteristic. The answers are analyzed for each user characteristic, and scores are assigned to each additional content in order of the best results. For example, scores are assigned to each additional content in order of the number of subjects who answered that their confidence level was 80 or higher. The score is an index independent of the confidence level. The score is an index common to people having a certain user characteristic, and the confidence level is an index based on each individual's sense. For example, if there are people A and B who have the same characteristic "extroversion: high", the scores of A and B will be the same, but the confidence levels of A and B may differ.

[0049] The characteristic additional content selection unit 12 selects one additional content candidate set corresponding to the target user characteristic from the additional content candidate sets prepared for each user characteristic. Hereinafter, the additional content candidate set selected by the characteristic additional content selection unit 12 is referred to as the "target additional content candidate set."

[0050] The evaluation information acquiring unit 16 acquires the evaluation information (FIG. 6) in the same manner as in the second embodiment.

[0051] As in the second embodiment, the evaluation additional content selection unit 17 selects an optimal set of additional content for the user based on the evaluation information (FIG. 6) acquired by the evaluation information acquisition unit 16. However, in the fourth embodiment, the set of additional content selected by the evaluation additional content selection unit 17 is not definitive, but is treated as a candidate set of additional content (hereinafter, referred to as a "candidate set of additional content").

[0052] For each combination of emotion type and added content (each added content included in the target added content candidate set), the synthesis unit 18 adds up the score for that combination in the target added content candidate set (Figure 10) selected by the characteristic added content selection unit 12 and the confidence level for that combination in the added content set candidate selected by the evaluation added content selection unit 17 (Figure 6).

[0053] Fig. 11 is a diagram showing an example of the summation result. For example, in Fig. 11, 70 in the first column of joy is the sum (total) of 50 in the first column of joy in Fig. 10 and 20 in the first row of joy in Fig. 6.

[0054] The synthesis unit 18 selects (characterizes) additional content to be synthesized into a video for each emotion based on the sum calculated for each additional content for each emotion type. Specifically, the synthesis unit 18 selects, for each emotion type, additional content corresponding to the maximum sum among the sums calculated for the emotion type, as additional content for the emotion. For example, in the example of FIG. 11, additional content with a sum of 70 is selected for joy. A set of additional contents selected by the synthesis unit 18 for each emotion (additional content set) is regarded as a target additional content set in the fourth embodiment.

[0055] Therefore, in the fourth embodiment, the video addition unit 14 adds (superimposes or synthesizes) additional content corresponding to the sender's emotion estimated by the sender emotion estimation unit 13 from the set of target additional content selected by the synthesis unit 18 onto the sender's video by image processing, thereby generating a video with additional content to which the additional content has been added.

[0056] As described above, according to the fourth embodiment, it is possible to obtain the same effects as those of the above embodiments.

[0057] In the above-mentioned embodiments, the user characteristic identifying unit 11, the characteristic addition content selecting unit 12, and the video adding unit 14, or the evaluation information acquiring unit 16, the evaluation addition content selecting unit 17, and the video adding unit 14, or the user characteristic identifying unit 11, the characteristic addition content selecting unit 12, the video adding unit 14, the evaluation information acquiring unit 16, and the evaluation addition content selecting unit 17, or the user characteristic identifying unit 11, the characteristic addition content selecting unit 12, the video adding unit 14, the evaluation information acquiring unit 16, the evaluation addition content selecting unit 17, and the synthesis unit 18 are examples of an identifying unit. The video adding unit 14 is an example of an adding unit.

[0058] Although the embodiment of the present invention has been described in detail above, the present invention is not limited to such specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0059] 10 Emotional Expression Addition Device 11 User characteristic identification unit 12. Characteristic addition content selection section 13 Sender Emotion Estimation Unit 14 Video Addition Section 15 Video display section 16 Evaluation information acquisition unit 17 Evaluation addition content selection section 18 Synthesis section 100 Drive device 101 Recording media 102 Auxiliary storage device 103 Memory device 104 CPU 105 Interface device 106 Display device 107 Input Devices B Bus

Claims

1. a specification step of specifying a characteristic of the user from among predetermined characteristics based on information input by the user, and specifying an expression manner for a certain emotion of a person that is easy for the user to understand based on the specified characteristic; an adding step of adding information indicating the certain emotion in the expression manner specified by the specifying step to the video when it is estimated that a person included in the video displayed to the user has expressed the certain emotion; The method for adding an emotional expression is characterized in that the method is executed by a computer.

2. The identification step identifies an expression mode corresponding to the identified characteristic.

2. The method for adding an emotional expression according to claim 1.

3. The identification step includes: a first step of identifying a set of expressions corresponding to the identified characteristic; a second step of having the user input a degree of ease of understanding of the certain emotion for a plurality of expression modes included in the set of expression modes identified in the first step, and identifying an expression mode that is easy for the user to understand from among the plurality of expression modes based on the degree; 2. The method for adding an emotional expression according to claim 1, further comprising:

4. The identification step includes: a first step of identifying a set of expressions corresponding to the identified characteristic; a second step of having the user input a degree of ease of understanding of the certain emotion for a plurality of expression modes; a third step of identifying an expression mode that is easy for the user to understand the certain emotion, for each expression mode included in the set of expression modes identified in the first step, based on a degree of ease of understanding of the certain emotion by the expression mode for a person having the identified characteristic and a degree of ease of understanding identified in the second step for the expression mode; 2. The method for adding an emotional expression according to claim 1, further comprising:

5. a specification unit that specifies a characteristic of the user from among predetermined characteristics based on information input by the user, and specifies an expression manner for a certain emotion of a person that is easy for the user to understand based on the specified characteristic; an adding unit that, when it is estimated that a person included in a video displayed to the user has expressed the certain emotion, adds information indicating the certain emotion in the expression manner specified by the specifying unit to the video; An emotional expression adding device comprising:

6. 5. A program for causing a computer to execute the emotional expression adding method according to claim 1.

Images