Information processing device, information processing method, and program

Abstract

The present technology pertains to an information processing device, an information processing method, and a program that enable appropriate evaluation of the degree of session dominance of participants participating in an online session, such as a video chat, on the basis of not only the amount of speech. A degree of dominance of a session by a participant is estimated based on facial images of the participant participating in the session. This technology can be applied to a video chat or the like using a terminal such as a smartphone.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program capable of appropriately evaluating the dominance of a participant participating in a conversation based on not only the amount of speech. Background Art

[0002] Patent document 1 discloses a technology for appropriately transmitting information in a situation where a user's hands cannot be used. Specifically, Patent document 1 discloses a technology involving a display device that can be mounted on the head, such as a head-mounted display, and an imaging unit capable of imaging lips and eyes, in which word recognition and expression recognition in a captured image are performed based on the movement of the lips, and associated features (stamps) are transmitted according to the results. Patent document 2 discloses a technology for pre-storing voice data of sounds made by a user and recognizing speech based on a video image captured by imaging the movement of the user's lips, and a technology for creating sound using the text of speech recognized by speech recognition and the stored voice data. Patent document 3 discloses a technology for estimating the satisfaction of a conversation between multiple people.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2021-157681

[0006] Patent Document 2: Japanese Patent Application Publication No. 2019-208138

[0007] Patent Document 3: Japanese Patent Application Publication No. 2018-169506 Summary of the invention

[0008] Problems to be solved by the present invention

[0009] In a conversation such as a video chat (online chat), there may be a case where a user is considered to be participating in the conversation as if the user is talking even when the user is not talking. Therefore, the dominance of a participant participating in the conversation cannot be properly evaluated only from the amount of talking (the proportion of time spent talking).

[0010] The present technology has been made in view of such circumstances, and aims to enable appropriate evaluation of the dominance of a participant participating in a conversation based on not only the amount of speech.

[0011] Solution to the problem

[0012] An information processing device or program according to the present technology is an information processing device including a processing unit that estimates the dominance of a participant over a conversation based on a facial image of the participant participating in the conversation, or a program that causes a computer to function as such an information processing device.

[0013] An information processing method according to the present technology is an information processing method including a processing unit, the processing unit estimating a dominance of a participant over a conversation based on a facial image of the participant participating in the conversation.

[0014] In the information processing device, the information processing method, and the program according to the present technology, the dominance of a participant over a conversation is estimated based on the facial image of the participant participating in the conversation. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram illustrating an example configuration of an information processing system according to an embodiment to which the present technology is applied.

[0016] Figure 2 is a view for explaining detection of facial landmarks.

[0017] Figure 3 is a view for explaining detection of facial landmarks.

[0018] Figure 4 is a diagram showing an example of an action correspondence table.

[0019] Figure 5 It is indicated by Figure 1 A flowchart of an example of a processing procedure performed by an information processing device in FIG.

[0020] Figure 6 is a block diagram illustrating an example configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION

[0021] In the following description, embodiments of the present technology will be explained with reference to the drawings.

[0022] <<Data Processing System According to the Present Embodiment>>

[0023] Figure 1 is a block diagram illustrating an example configuration of an information processing system according to an embodiment to which the present technology is applied.

[0024] exist Figure 1In the data processing system of the information processing system according to the present embodiment, for example, a terminal 1 and a terminal 2 such as a smart phone, a tablet or a personal computer (PC) are included. In the following description, it is assumed that the terminal 1 and the terminal 2 are used for a one-to-one video chat, for example, via a smart phone. One of the terminal 1 and the terminal 2 is also referred to as the user's terminal 1, and the other is also referred to as the other party's terminal 2. Since the user's terminal 1 and the other party's terminal 2 or the terminal 1 and the terminal 2 have similar configurations and are capable of performing similar processing, the configuration of the user's terminal 1 (also referred to as terminal 1 for short) and the processing to be performed by the user's terminal 1 are described herein. Note that the other party's terminal 2 only needs to have a configuration for performing a video chat with the user's terminal 1, and is not limited to any specific configuration.

[0025] The terminal 1 includes an imaging unit 11, a sound acquisition unit 12, an image processing unit 13, a display unit 14, a communication unit 15, an image acquisition unit 16, a conversation state determination unit 17, a sound processing unit 18, a sound output unit 19, and a data learning unit 20. The imaging unit 11 continuously captures a video (image) of a subject, and acquires a moving image including frames formed at predetermined time intervals. The imaging unit 11 is intended to capture an image of the face of a call party as a subject, and may be, for example, a built-in camera generally provided in a smartphone, etc., and captures an image of the face of a user (first call party) of the user's terminal 1. However, the imaging unit 11 may be an external camera, because in the case where an external camera generally included in a smartphone, etc. is pointed to the user of the smartphone during a conversation and in the case where the photographer is different from the first call party, the first call party is captured using the external camera. That is, the imaging unit 11 may be one or more cameras included in the terminal 1, and the user may specify a camera to be used as the imaging unit 11, or the camera capturing the face may be automatically switched to the imaging unit 11. The image acquired by the imaging unit 11 is supplied to the image processing unit 13.

[0026] The sound acquisition unit 12 collects voices around the terminal 1 and acquires sound (sound signal) as an electrical signal. The sound acquisition unit 12 may be, for example, a microphone generally included in a smart phone or the like. However, the sound acquisition unit 12 may be an external device connected to the terminal 1, such as a headset or a Bluetooth (registered trademark) headset. The sound acquired by the sound acquisition unit 12 is supplied to the image processing unit 13.

[0027] The image processing unit 13 performs image processing on the image supplied from the imaging unit 11 (also referred to as the self-image) and the image from the terminal 2 of the other party supplied from the image acquisition unit 16 (also referred to as the image of the other party), and supplies information (evaluation information) for determining (evaluating) the conversation state between the first call party and the other party (also referred to as the second call party) to the conversation state determination unit 17. In addition, the image processing unit 13 generates an image for display based on the self-image and the image of the other party, and supplies the generated image to the display unit 14. The image for display may be, for example, in a form in which the self-image is superimposed on the image of the other party or in a form in which the image of the other party and the self-image are switched. In addition, the image processing unit 13 supplies the sound from the terminal 1 of the user supplied from the sound acquisition unit 12 (also referred to as the self-sound) to the sound processing unit 18 and the data learning unit 20, and supplies the self-image to the communication unit 15 and the data learning unit 20. Note that the image processing unit 13 will be described in detail later.

[0028] The display unit 14 displays the image for display from the image processing unit 13. The display unit 14 may be, for example, a display generally included in a smartphone or the like.

[0029] The communication unit 15 controls communication with external devices and performs communication with the terminal 2 of the other party. The communication may include, for example, a wired communication network such as a local area network (LAN) or a wide area network (WAN), a wireless communication network such as a mobile communication network or a wireless local area network (WLAN), or a combined communication network. The network may include the Internet using a communication protocol such as a transmission control protocol / Internet protocol (TCP / IP).

[0030] The image acquisition unit 16 acquires an image of the other party transmitted from the terminal 2 of the other party via the communication unit 15, and supplies the acquired image to the image processing unit 13. The conversation state determination unit 17 determines a conversation state such as the dominance of a conversation in the current call based on the evaluation information from the image processing unit 13. The conversation state as a determination result is supplied to the image processing unit 13 and the sound processing unit 18. The conversation state determination unit 17 will be described in detail later.

[0031] The sound processing unit 18 acquires the sound (also referred to as the other party's sound) sent from the other party's terminal 2 via the communication unit 15. Based on the conversation state from the conversation state determination unit 17, the sound processing unit 18 performs sound processing on the other party's sound, such as voice conversion with pitch shift (change in sound pitch) or equalizer (voice effect). The other party's sound after sound processing is supplied to the sound output unit 19. In addition, the sound processing unit 18 can acquire the own sound from the image processing unit 13, and perform sound processing on the own sound based on the conversation state in a manner similar to that for the other party's sound. The own sound that has undergone sound processing is supplied to the communication unit 15, and is sent from the communication unit 15 to the other party's terminal 2.

[0032] The sound output unit 19 outputs the sound from the other party of the sound processing unit 18 as a sound wave. The sound output unit 19 may be, for example, a speaker generally included in a smart phone, etc. However, the sound output unit 19 may be an external device connected to the terminal 1, such as a headset or a Bluetooth (registered trademark) headset.

[0033] The data learning unit 20 learns the expression (expression change) of the first party to add points to or subtract points from the dominance of the conversation based on the self-image and self-voice from the image processing unit 13. The learning result is supplied to the image processing unit 13. The data learning unit 20 will be described in detail later.

[0034] <Details of the Image Processing Unit 13, the Dialogue State Determination Unit 17, and the Sound Processing Unit 18>

[0035] The image processing unit 13 includes a face recognition unit 31, an expression recognition unit 32, and an expression conversion unit 33. The face recognition unit 31 recognizes the face (face image) of the first party included in the self-image from the imaging unit 11. The expression recognition unit 32 performs expression recognition on the face recognized by the face recognition unit 31, and estimates the dominance of the first party over the conversation based on the recognized expression. Note that the term "expression" includes facial movements.

[0036] The dominance of the conversation indicates the degree to which each of the first call party and the second call party can be regarded as dominating the dialogue (conversation) between the first call party and the second call party. For example, it is assumed that the dominance of the first call party over the conversation becomes higher as the first call party's speaking time becomes longer. In addition, in the case where the first call party does not speak, when the first call party "nods (moves the neck up and down)", "listens with a smile (corners of the mouth raised)", etc., it can be considered that the first call party is actively participating in the conversation. Therefore, it is assumed that the dominance of the first call party over the conversation becomes higher as the time or number of times it is determined that the first call party shows a positive expression (reaction) such as nodding, agreeing, or doing some other habitual behavior related to the conversation is greater. On the contrary, in the case where the first call party is not listening as in "eyes looking in some other direction (line of sight pointing outside the screen)", or in the case of "he / she wants to speak but cannot do so (pressing lips)", it can be considered that the first call party is not actively participating in the conversation. Note that the case where the line of sight is directed outside the screen means the direction in which the line of sight is away from the display unit 14 or the imaging unit 11. It is assumed that the dominance of the first party to the conversation becomes lower as the time or number of times it is determined that the first party to the conversation shows such a negative expression (reaction) to the conversation becomes greater.

[0037] The expression recognition unit 32 of the image processing unit 13 can recognize the expression of the second call party based on the image of the other party, and estimate the dominance of the second call party over the conversation in a manner similar to that used for the dominance of the first call party over the conversation. Note that the dominance of one of the first call party and the second call party over the conversation can be estimated by the user's terminal 1, while the dominance of the other party can be estimated by the other party's terminal 2. In this case, the image processing unit 13 of the user's terminal 1 can obtain both the dominance of the first call party over the conversation and the dominance of the second call party over the conversation by acquiring the dominance of the conversation estimated by the other party's terminal 2 via communication.

[0038] It is assumed that a value is added to or subtracted from the dominance of the conversation for the first and second parties under the same conditions (by the same evaluation method). For example, the dominance of the conversation by the first and second parties is represented by x1 and x2, respectively. It is assumed that each time the first or second party speaks for one second, one is added to the dominance x1 or x2 of the corresponding party over the conversation; and each time the first or second party agrees once, one is added to the dominance x1 or x2 of the corresponding party over the conversation. In this case, the value indicated by the dominance x1 or x2 of one of the first and second parties over the conversation is a value indicating the time or number of times he / she actively participates (or is regarded as participating) in the conversation during the period from the start of the conversation to the current time point. Therefore, to be precise, the dominance x1 or x2 does not directly indicate the degree to which each of the first and second parties can be regarded as dominating the conversation. For convenience, the parameters x1 and x2 referred to as dominance are now referred to as conversation participation evaluation values ​​x1 and x2, and the dominance of each of the first call party and the second call party is represented by the parameters X1 and X2. In this case, the dominance X1 may be a value obtained according to x1 / (x1+x2), and the dominance X2 may be a value obtained according to x2 / (x1+x2). That is, the dominance X1 and X2 are values ​​indicating their ratios, and may be the composition ratios of the corresponding conversation participation evaluation values ​​x1 and x2 relative to the total number (sum) of the conversation participation evaluation values ​​x1 and x2.

[0039] The expression recognition unit 32 estimates the dominance X1 (or conversation participation evaluation value x1 ) of the first party and the dominance X2 (or conversation participation evaluation value x2 ) of the second party over the conversation, and supplies the results to the dialogue state determination unit 17 .

[0040] The dialogue state determination unit 17 compares the dominance X1 of the first call party to the conversation supplied from the expression recognition unit 32 with the dominance X2 of the second call party to the conversation, and determines whether there is a difference between them. Whether there is a difference between the dominance X1 and the dominance X2 can be determined by, for example, whether the difference between the dominance X1 and the dominance X2 is equal to or greater than a predetermined critical value. The critical value may be a value set or changed by the user (the first call party), or may be a fixed value. For example, in the case where the dominance X1 and the dominance X2 are expressed in percentages, when C% (for example, C is 60) is set as a critical value, the dialogue state determination unit 17 determines whether the difference between the dominance X1 and the dominance X2 is C% or more. Alternatively, the dialogue state determination unit 17 may determine whether the dominance X1 is (50-C / 2)% or less, or (50+C / 2)% or more, or may determine whether only one of the conditions is satisfied. For example, in the case where the first party is a person who is not good at speaking but likes to listen, the critical value C can be set to a relatively large value such as 60%, and in this case, a check can be made to determine whether the dominance X1 of the first party over the conversation is (50-60 / 2)%=20% or less. The result of the determination made by the dialogue state determination unit 17 (determination result) is supplied to the image processing unit 13 and the sound processing unit 18.

[0041] In the case where a determination result indicating that there is a difference between the dominance X1 of the first caller and the dominance X2 of the second caller over the conversation is supplied from the conversation state determination unit 17, the expression conversion unit 33 of the image processing unit 13 changes the expression of the second caller in the image of the other party by image processing, and guides them to reduce the difference. The image of the other party whose expression is changed by image processing is a display image displayed on the display unit 14 and visually recognized by the first caller. For example, in the case where a determination result indicating that the dominance X1 is excessively lower than the dominance X2 and there is a difference between them is supplied, the expression conversion unit 33 changes the expression of the second caller so that the dominance X1 of the first caller over the conversation becomes higher. As a specific example, the expression conversion unit 33 performs conversion so that the corners of the mouth in the facial image of the second caller in the image of the other party are raised. As a result, the face of the second caller looks more smiling, and the sense of affirmation increases. Therefore, the first caller is guided to increase the amount of speaking (dominance X1). In the case where a determination result indicating that dominance X1 is excessively higher than dominance X2 and there is a difference between them is supplied, the expression conversion unit 33 changes the expression of the second party so that dominance X1 of the first party over the conversation is reduced. As a specific example, the expression conversion unit 33 performs conversion so that the corners of the mouth in the facial image of the second party in the image of the other party are lowered. As a result, the sense of negation increases with the impression received from the face of the second party, and thus the first party is guided to reduce the amount of speaking (dominance X1).

[0042] Furthermore, in the case where a determination result indicating that there is a difference between the dominance X1 of the first caller and the dominance X2 of the second caller over the conversation is supplied from the conversation state determination unit 17, the expression conversion unit 33 can also change the expression of the first caller in the self-image from the imaging unit 11 by image processing, and guide them to reduce the difference. In this case, the self-image of the first caller whose expression is changed by image processing is a display image displayed on the display unit of the other party's terminal 2 through communication and visually recognized by the second caller. For example, in the case where a determination result indicating that the dominance X1 is excessively lower than the dominance X2 and there is a difference between them is supplied, the expression conversion unit 33 changes the expression of the first caller so that the dominance X2 of the second caller over the conversation is reduced. In the case where a determination result indicating that the dominance X1 is excessively higher than the dominance X2 and there is a difference between them is supplied, the expression conversion unit 33 changes the expression of the first caller so that the dominance X2 of the second caller over the conversation is increased.

[0043] Note that one of the change of the expression of the first caller in the own image and the change of the expression of the second caller in the other party's image may be performed by the user's terminal 1, and the other may be performed by the other party's terminal 2. In some cases, the other party's terminal 2 may not have such a function of changing the expression, or in some cases, the user's terminal 1 may only change the expression of one party. In this embodiment, for ease of explanation, it is assumed that the user's terminal 1 has a function of changing only the expression of the second caller in the other party's image using the expression conversion unit 33.

[0044] In the case where a determination result indicating that there is a difference between the dominance X1 of the first calling party and the dominance X2 of the second calling party over the conversation is supplied from the conversation state determination unit 17, the sound processing unit 18 changes the sound quality of the other party's voice from the other party's terminal 2 by sound processing such as voice conversion to which pitch shift or equalizer (voice effect) is applied, and guides them to reduce the difference. The voice of the other party whose sound quality is changed by sound processing is the sound output by the sound output unit 19 and listened to by the first calling party. For example, in the case where a determination result indicating that the dominance X1 is excessively lower than the dominance X2 and there is a difference between them is supplied, the sound processing unit 18 changes the voice of the second calling party (the sound quality of the other party's voice) so that the dominance X1 of the first calling party over the conversation increases. As a specific example, the sound processing unit 18 performs voice conversion for raising the pitch (voice pitch) of the other party's voice. As a result, the voice of the second calling party sounds more positive than his / her usual voice, and thus guides the first calling party to increase the amount of speaking (dominance X1). In the case where a determination result indicating that the dominance X1 is excessively higher than the dominance X2 and there is a difference between them is supplied, the sound processing unit 18 changes the voice of the second party (the sound quality of the other party's voice) so that the dominance X1 of the first party over the conversation is reduced. As a specific example, the sound processing unit 18 performs voice conversion to lower the pitch of the other party's voice. As a result, the voice of the second party sounds more negative than his / her usual voice, leading the first party to reduce the amount of speaking (dominance X1).

[0045] Furthermore, in the case where a determination result indicating that there is a difference between the dominance X1 of the first call party and the dominance X2 of the second call party over the conversation is supplied from the conversation state determination unit 17, the sound processing unit 18 can also change the sound quality of the own voice from the sound acquisition unit 12 by sound processing, and guide them to reduce the difference. In this case, the own voice whose sound quality is changed by sound processing is the voice output from the sound output unit of the other party's terminal 2 through communication and listened to by the second call party. For example, in the case where a determination result indicating that the dominance X1 is excessively lower than the dominance X2 and there is a difference between them is supplied, the sound processing unit 18 changes the sound quality of the own voice so that the dominance X2 of the second call party over the conversation is reduced. In the case where a determination result indicating that the dominance X1 is excessively higher than the dominance X2 and there is a difference between them is supplied, the sound processing unit 18 changes the sound quality of the own voice so that the dominance X2 of the second call party over the conversation is increased.

[0046] Note that one of the change in the sound quality of the other party's voice and the change in the sound quality of one's own voice may be performed by the user's terminal 1, and the other may be performed by the other party's terminal 2. In some cases, the other party's terminal 2 may not have such a function of changing the voice, and in some cases, the user's terminal 1 may only change the sound quality of one party's voice. In this embodiment, for ease of explanation, it is assumed that the user's terminal 1 has a function of changing only the sound quality of the other party's voice using the sound processing unit 18.

[0047] <Details of Expression Recognition Unit 32>

[0048] The expression recognition unit 32 recognizes the facial expression of the first call party based on the self-image from the imaging unit 11, and estimates the dominance of the first call party over the conversation (conversation participation evaluation value x1) based on the recognized expression. Note that the expression recognition unit 32 can estimate the dominance of the second call party over the conversation (conversation participation evaluation value x2) in a manner similar to that used for the dominance of the first call party over the conversation (conversation participation evaluation value x1) based on the image of the other party from the other party's terminal 2. However, in the present embodiment, the dominance of the second call party over the conversation is supplied from the other party's terminal 2, and its description is not given herein.

[0049] The expression recognition unit 32 includes a facial landmark recognition unit 41 and an action correspondence table 42. The facial landmark recognition unit 41 detects (recognizes) facial landmarks to recognize the facial expression of the first caller in the self-image. Figure 2 As shown, the facial landmarks LM indicate feature points detected from the facial image FA, and as Figure 3As shown, for example, 68 feature points are indicated. The detection of the facial landmark LM can be performed using "OpenFace" (Tadas Baltrusaitis, Peter Robinson, Louis-Philippe Morency, "OpenFace: an open source facial behavior analysis toolkit", 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1-10, 2016) which is a facial recognition application. In addition, the detection of the facial landmark LM can also be performed using the function of an application such as "ARFaceAnchor" (https: / / developer.apple.com / documentation / arkit / arfaceanchor) which is an application used in a mobile terminal such as a smartphone or a tablet, or can be performed using an inference model generated by a machine learning technology. Taking the case of using "ARFaceAnchor" as an example, the facial landmark recognition unit 41 can obtain the degree of opening of the mouth as a coefficient called jawOpen while detecting the facial landmark, and obtain various states of the facial landmark as coefficients, such as 1.0 when the mouth is fully open, and 0 when the mouth is not open at all. Since the first party must move the mouth to speak, the facial landmark recognition unit 41 determines that the first party is in a speaking state based on the state of the facial landmark when the state of the mouth changes. For example, the facial landmark recognition unit 41 increments the value of the first party's dominance over the conversation (conversation participation evaluation value x1) by 1 each time the first party is determined to be in a speaking state for, for example, one second.

[0050] In addition, the facial landmark recognition unit 41 detects the facial expressions of the first party through changes in the facial landmarks LM. The facial expressions are indicated by a combination of facial movement elements (e.g., 44 kinds) called "action units (AUs)", which are the smallest units of facial expressions. The action correspondence table 42 specifies conditions for determining positive facial expressions that indicate that the first party is considered to be actively participating in the conversation even when the first party is not speaking, and conditions for determining negative facial expressions that indicate that the first party can be considered to be not actively participating in the conversation. In addition, the action correspondence table 42 specifies values ​​to be added or subtracted from the first party's dominance over the conversation (conversation participation evaluation value x1) when facial expressions corresponding to these specified conditions are detected. Figure 4 An example of the action correspondence table 42 is shown. For example, in the case of detecting the expression action using, for example, "OpenFace", the facial landmark recognition unit 41 can obtain the coefficient of each action unit. The coefficient of each action unit corresponds to the ratio of the expression action of each action unit included in the expression action of the first caller. The facial landmark recognition unit 41 obtains the coefficient of each action unit when detecting the expression action of the first caller, and detects the expression action of the first caller based on the coefficient of each action unit obtained. Figure 4 The expression action that satisfies the condition is detected among the expression actions in the action correspondence table 42 shown in the figure. For example, when the coefficient of the expression action of the action unit "pressing lips" is 0.3 or higher and the action lasts for two seconds or longer, the expression action that satisfies the condition is detected. Figure 4 The expression action condition indicated in the first line of . Figure 4 As specified in , the facial landmark recognition unit 41 subtracts "1" from the dominance of the first call party over the conversation (conversation participation evaluation value x1). That is, it is determined that the action is a negative expression action that the first call party can be regarded as not actively participating in the conversation, and "1" is subtracted from the dominance of the first call party over the conversation (conversation participation evaluation value x1).

[0051] On the other hand, when the coefficient of the expression action of the action unit called "tight neck" changes by 0.2 or more within two seconds, it is detected that the expression action satisfies Figure 4 The expression action conditions shown in the second row of . Figure 4 As specified in , the facial landmark recognition unit 41 adds "1" to the dominance of the first call party over the conversation (conversation participation evaluation value x1). That is, it is determined that the action is a positive expression action that the first call party can be regarded as actively participating in the conversation, and "1" is added to the dominance of the first call party over the conversation (conversation participation evaluation value x1). In some cases, such data in the action correspondence table 42 can be created in advance, or in some cases can be learned during the conversation and added according to the characteristics of the first call party's expression action.

[0052] <Creation of Data in Action Correspondence Table 42>

[0053] Now, the case where the data in the action correspondence table 42 is learned during the conversation and added according to the characteristics of the facial expressions of the first party is described. Figure 1 In the example, the data learning unit 20 operates when the sound components other than human voice (the speech sound of the first party) included in the own sound acquired by the sound acquisition unit 12 are at a predetermined level or lower. The data learning unit 20 includes a sound recognition unit 51, a sound-to-text conversion unit 52, an emotion analysis unit 53, and an expression learning unit 54.

[0054] The voice recognition unit 51 acquires the own voice acquired by the voice acquisition unit 12 via the image processing unit 13, recognizes (extracts) human voice (speech sound) from the acquired own voice, and supplies the human voice to the voice-to-text conversion unit 52. The voice-to-text conversion unit 52 converts the speech sound from the voice acquisition unit 12 into text, and supplies the text data to the emotion analysis unit 53. The emotion analysis unit 53 detects emotions as emotion information based on the meaning of the text based on the text data from the voice-to-text conversion unit 52, and supplies the emotion information to the expression learning unit 54.

[0055] The expression learning unit 54 learns the emotion information from the emotion analysis unit 53 and the movement of the facial landmarks in the self-image when the emotion information is detected. Information about the facial landmarks in the self-image is supplied to the expression learning unit 54 from the expression recognition unit 32 (facial landmark recognition unit 41) of the image processing unit 13. As a result, the expression learning unit 54 can learn the expression action performed by the first call party for the emotion of the first call party indicated by the emotion information, and associate the expression action of the first call party with the emotion of the first call party at that time. For example, in the case where the first call party says a word of negative emotion such as "It's too difficult" after moving his / her mouth, the expression action of moving the mouth can be associated with the negative emotion. For the expression action in the case of positive emotion, the dominance of the first call party over the conversation (conversation participation evaluation value x1) is added (for example, +1). For the expression action in the case of negative emotion, the dominance of the first call party over the conversation (conversation participation evaluation value x1) is subtracted (for example, -1). In this way, the data in the action correspondence table 42 can be generated. The generated data is accumulated in the data accumulation unit 61 , and is placed in a usable state as data in the action correspondence table 42 of the expression recognition unit 32 at an appropriate timing.

[0056] <Processing procedure for adjusting the degree of dominance of user's terminal 1 over a session>

[0057] Figure 5 4 is a flowchart showing an example of a processing procedure to be performed by the user's terminal 1. Note that the processing to be performed by the data learning unit 20 to create the data in the action correspondence table 42 is not explained herein.

[0058] In step S1, the imaging unit 11 starts acquiring a self-image. Thereafter, the acquisition of the self-image is continuously performed. In step S2, the image processing unit 13 (face recognition unit 31) determines whether a face is included in the self-image acquired in step S1. If the result in step S2 is negative, the processing in step S2 is repeated. If the result in step S2 is positive, the processing moves to step S3 and step S6. Note that the processing in steps S3 to S5 and the processing in steps S6 and S7 are performed in parallel.

[0059] In step S3, the image processing unit 13 (the facial landmark recognition unit 41 of the expression recognition unit 32) detects the facial landmarks of the first caller in the self-image, and detects the states of the facial landmarks of the lips and the mouth. In step S4, the image processing unit 13 (the facial landmark recognition unit 41 of the expression recognition unit 32) determines whether the values ​​(coordinates) of the facial landmarks of the lips and the mouth have changed by a certain value or more. If the result in step S4 is negative, the processing in step S4 is repeated. If the result in step S4 is positive, the processing moves to step S5. In step S5, the image processing unit 13 (the facial landmark recognition unit 41 of the expression recognition unit 32) determines that the first caller is in a speaking state. At this time, the image processing unit 13 (the facial landmark recognition unit 41 of the expression recognition unit 32) increases the dominance of the first caller over the conversation (the conversation participation evaluation value x1). For example, the image processing unit 13 adds "1" to the dominance of the first caller over the conversation (the conversation participation evaluation value x1), or increases the duration (seconds) of the detection of the speaking state.

[0060] In step S6, the image processing unit 13 (the facial landmark recognition unit 41 of the expression recognition unit 32) acquires the state of the facial landmark (the expression action of the first caller corresponding to the condition specified in the action correspondence table 42) based on the action correspondence table 42. In step S7, the image processing unit 13 (the facial landmark recognition unit 41 of the expression recognition unit 32) performs addition or subtraction corresponding to the expression action acquired in step S6 on the dominance of the first caller over the conversation (the conversation participation evaluation value x1) based on the action correspondence table 42. In step S8, the conversation state determination unit 17 compares the dominance X1 of the first caller over the conversation with the dominance X2 of the second caller over the conversation.

[0061] In step S9, the conversation state determination unit 17 determines whether there is a difference between the dominance X1 of the first call party and the dominance X2 of the second call party. If the result in step S9 is negative, the processing from step S2 is repeated. If the result in step S9 is positive, the processing moves to step S10. In step S10, the image processing unit 13 (expression conversion unit 33) converts the expression of the second call party in the image of the other party so as to reduce the difference between the dominance X1 of the first call party and the dominance X2 of the second call party. In addition, the sound processing unit 18 converts the sound quality of the other party's voice so as to reduce the difference between the dominance X1 of the first call party and the dominance X2 of the second call party. Note that in some cases, only one of the expression conversion of the image processing unit 13 and the sound quality conversion of the sound processing unit 18 is performed. After step S10, the processing returns to step S2, and the processing from step S2 is repeated.

[0062] According to the present technology described above, the dominance of a conversation can be estimated by considering facial expression reactions such as nodding, agreeing, and doing some other habitual behavior, rather than determining which of the first party (conversation participant) and the second party (the other party) is talking more actively based on the conversation volume (speaking volume) alone. In addition, in the case of determining the dominance only by an image, the accuracy becomes lower when different cameras are used at different distances or the object is facing sideways. However, the state of facial landmarks is used, and therefore the position and direction of the face have little influence on the accuracy.

[0063] In addition, in video chats such as online consultation or online courses where there is a teaching party and a party being taught through video chat, there may be few words from the party being taught, or only the teaching party may keep talking. In this case, the user wants to speak but always listens, or cannot ask what he / she wants to ask, resulting in a decrease in user satisfaction. In addition, there are services such as online campus tours for introducing places, and there is a situation where one party (or both parties) use video chat to introduce places while being outdoors. In this case, it is also conceivable to conduct video chats in an external environment with a lot of noise.

[0064] According to the present technology, these problems are solved, and conversation bias in one-to-one video calls is solved. When the user's state and situation are estimated by facial images and it is determined that there is a bias in the conversation, the conversation bias is eliminated through user feedback. In the case where outdoor use is assumed, the user situation is determined by image rather than by voice. The conversation bias is determined by the value of dominance. The dominance of the conversation is calculated by adjusting the dominance using facial reactions in addition to the speaking state. Therefore, even if the voice of one party occupies most of the conversation, it is difficult to determine that the conversation is biased if the non-speaking party frequently nods or responds.

[0065] <Implementation Method (Use Case)>

[0066] Figure 1 The information processing system shown and others may adopt the following embodiments.

[0067] (First Embodiment)

[0068] In services such as online consultation or online courses, it is possible that the information processing system automatically selects a mode of the other party. Alternatively, in a service for selecting the best conversation partner such as a matching service, it is conceivable to perform matching based on the dominance of the conversation. For high and low tendencies of dominance of the conversation, people with opposite tendencies can be automatically matched or people with high dominance of the conversation can be matched with people with low dominance of the conversation, so that people who want to talk can talk more and be satisfied, and people who do not want to talk more do not need to talk more and can be satisfied.

[0069] (Second Embodiment)

[0070] In the first embodiment, only the conversation amount (speaking amount) is used. However, as an application method, matching can also be performed based not only on the dominance of the conversation but also on the actions in the action correspondence table in the following manner: a person who frequently looks at the other party (a person who looks at the call screen) is matched with a person who makes a lot of reactions without any problem, and a person who hardly looks at the other party is matched with a person who hardly reacts without any problem.

[0071] (Third Embodiment)

[0072] In addition to the above, by providing a parameter indicating that speech is easy to hear because a person who speaks with a wide mouth (a person with a high average value of a specific coefficient of mouth opening) opens his mouth clearly, an application method for training or educating high-quality hosts using matching services or conversation proficiency in such services as an indicator can be considered.

[0073] <Sample configuration of computer>

[0074] The above series of processing can be performed by hardware or software. In the case of performing a series of processing by software, the program forming the software is installed in the computer. Here, examples of the computer include a computer integrated into dedicated hardware, a general-purpose personal computer that can perform various functions by installing various programs in the computer, etc.

[0075] Figure 6 : is a block diagram showing an example configuration of hardware of a computer that executes the above-described series of processes according to a program.

[0076] In the computer, a central processing unit (CPU) 201 , a read only memory (ROM) 202 , and a random access memory (RAM) 203 are connected to each other via a bus 204 .

[0077] The input / output interface 205 is further connected to the bus 204. The input / output interface 205 is connected to an input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210.

[0078] The input unit 206 includes a keyboard, a mouse, a microphone, etc. The output unit 207 includes a display, a speaker, etc. The storage unit 208 includes a hard disk, a nonvolatile memory, etc. The communication unit 209 includes a network interface, etc. The drive 210 drives a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0079] In the computer configured as described above, the CPU 201 loads a program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 , for example, and executes the program to perform the above-described series of processing.

[0080] The program executed by the computer (CPU 201) can be provided, for example, by being recorded as a package medium or the like on the removable medium 211. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0081] In the computer, when the removable medium 211 is set in the drive 210, the program can be installed in the storage unit 208 via the input / output interface 205. In addition, the program can be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. Alternatively, the program can be pre-installed in the ROM 202 or the storage unit 208.

[0082] Note that the program executed by the computer may be a program for executing the processing in time series according to the sequence described herein, or may be a program for executing the processing in parallel or at necessary timing such as when a call is made.

[0083] Here, the processing to be executed by the computer according to the program is not necessarily performed in time order according to the order shown in the flowchart. In other words, the processing to be executed by the computer according to the program includes processing executed in parallel or independently (for example, parallel processing or object-oriented processing).

[0084] In addition, the program may be executed by a single computer (processor), or may be executed by a plurality of computers in a distributed manner. In addition, the program may be transmitted to a remote computer and executed there.

[0085] In addition, in this specification, a system means a collection of multiple components (devices, modules (parts), etc.), and it is not important whether all the components are in the same housing. Therefore, multiple devices housed in different housings and interconnected through a network and a single device including multiple modules housed in a single housing are both systems.

[0086] In addition, a component described as one device (or processing unit) may, for example, be divided and formed into a plurality of devices (or processing units). Conversely, the components described above as a plurality of devices (or processing units) may be collectively formed into one device (or processing unit). In addition, components other than the above components may be added to the configuration of each device (or each processing unit). In addition, as long as the configuration and operation of the entire system are substantially the same, a portion of the configuration of a certain device (or processing unit) may be included in the configuration of a certain other device (or a certain other processing unit).

[0087] Furthermore, the present technology can be implemented, for example, as cloud computing in which one function is shared and processed by a plurality of devices in a distributed manner through a network.

[0088] In addition, the above-mentioned program can be executed by any desired device, for example. In this case, the device is only required to have necessary functions (functional blocks, etc.) and obtain necessary information.

[0089] In addition, for example, each step in the above flowchart may be performed by a single device, or may be performed by multiple devices in a shared manner. In addition, in the case where a single step includes multiple processes, the multiple processes included in the single step may be performed by a single device, or may be performed by multiple devices in a shared manner. In other words, the multiple processes included in the single step may also be performed as processes in multiple steps. Conversely, the processes described as multiple steps may be collectively performed as a single step.

[0090] Note that in the program to be executed by the computer, the processing in the steps of writing the program may be performed in chronological order according to the order described in this article, or may be performed in parallel or independently at necessary timings such as when a call is made. That is, as long as there is no contradiction, the processing in the various steps may be performed in an order different from the above order. In addition, the processing in the steps of writing the program may be performed in parallel with the processing according to some other program, or may be performed in combination with the processing according to some other program.

[0091] Note that, as long as there is no contradiction, the various techniques of the present technology described herein can be implemented independently of each other. It goes without saying that any of the various techniques of the present technology can be implemented in combination. For example, some or all of the present technology described in any embodiment can be implemented in combination with some or all of the present technology described in other embodiments. In addition, some or all of the above-mentioned present technology can be implemented in combination with some other technologies not described above.

[0092] <Sample combination of configuration>

[0093] Note that the present technology may also have the following configurations.

[0094] (1) An information processing device comprising:

[0095] A processing unit estimates, based on facial images of participants participating in a conversation, the dominance of the participants over the conversation.

[0096] (2) The information processing device according to (1), wherein:

[0097] The processing unit detects the facial expressions of the participant based on facial landmarks detected from the facial image.

[0098] (3) The information processing device according to (2), wherein:

[0099] The processing unit detects the facial expressions of the participant based on facial landmarks detected from the facial image.

[0100] (4) The information processing device according to (3), wherein:

[0101] The processing unit detects the facial expression of the participant through a combination of facial expression movement elements.

[0102] (5) The information processing device according to any one of (1) to (4), wherein:

[0103] The processing unit estimates the dominance based on the facial expressions of the participant when the participant speaks and the facial expressions of the participant when the participant does not speak, which are recognized based on the facial image.

[0104] (6) The information processing device according to any one of (1) to (5), wherein:

[0105] The processing unit obtains the dominance of the other party of the participant over the conversation, the other party participating in the conversation, and

[0106] The dominance of the participant is estimated as a value indicating a ratio to the dominance of the other party.

[0107] (7) The information processing device according to (5), wherein:

[0108] When the facial expression of the participant when the participant is silent is regarded as an facial expression of active participation in the conversation, the processing unit increases the dominance of the participant.

[0109] (8) The information processing device according to (7), wherein:

[0110] When the facial expression of the participant is nodding, agreeing or smiling, the processing unit determines that the facial expression of the participant is regarded as an facial expression of actively participating in the conversation.

[0111] (9) The information processing device according to any one of (5) to (8), wherein:

[0112] When the facial expression of the participant when the participant is silent is regarded as a facial expression of not actively participating in the conversation, the processing unit reduces the dominance of the participant.

[0113] (10) The information processing device according to (9), wherein:

[0114] When the participant's expression action is an expression action of pressing lips or when the sight is away from the direction of the imaging unit that captures the facial image, the processing unit determines that the participant's expression action is an expression action regarded as not actively participating in the conversation.

[0115] (11) The information processing device according to any one of (1) to (10), further comprising:

[0116] a display unit that displays a facial image of the other party of the participants participating in the conversation; and

[0117] A conversion unit changes a part of the facial image of the other party displayed on the display unit to convert the expression of the other party according to the dominance of the participant over the conversation.

[0118] (12) The information processing device according to (11), wherein:

[0119] The conversion unit changes the corners of a mouth in the facial image of the other party.

[0120] (13) The information processing device according to any one of (1) to (12), wherein:

[0121] The conversion unit changes a part of a facial image of the participant or the other of the participants so that the dominance of the participant satisfies a preset condition.

[0122] (14) The information processing device according to any one of (1) to (13), further comprising:

[0123] a voice output unit that outputs a voice of the other party of the participants participating in the conversation; and

[0124] A voice processing unit changes the sound quality of the other party's voice output from the voice output unit according to the degree of dominance of the participant over the conversation.

[0125] (15) The information processing device according to (14), wherein:

[0126] The sound output unit changes the sound quality of the other party's voice by adopting pitch shifting or an equalizer.

[0127] (16) The information processing device according to any one of (1) to (15), wherein:

[0128] The sound processing unit changes the sound quality of the voice of the participant or the other of the participants so that the dominance of the participant satisfies a preset condition.

[0129] (17) The information processing device according to any one of (1) to (16), wherein:

[0130] The processing unit matches the participant participating in the conversation with another party of the participant according to the dominance of the participant.

[0131] (18) The information processing device according to (17), wherein:

[0132] The processing unit performs matching between the other party whose tendency is opposite to that of the participant and the participant, with respect to the high and low tendencies of the dominance, the other party participating in the conversation.

[0133] (19) An information processing method comprising:

[0134] The dominance of a participant participating in a conversation over the conversation is estimated by a processing unit of the information processing device based on a facial image of the participant.

[0135] (20) A program for causing a computer to:

[0136] A processing unit estimates, based on facial images of participants participating in a conversation, the dominance of the participants over the conversation.

[0137] Note that the present embodiment is not limited to the above-described embodiment, and various modifications may be made to them without departing from the scope of the present disclosure. In addition, the effects described herein are merely examples and are not restrictive, and some other effects may be achieved.

[0138] Reference numerals list

[0139] 1 User's terminal

[0140] 2 The other party's terminal

[0141] 11 Imaging Unit

[0142] 12Sound Acquisition Unit

[0143] 13 Image Processing Unit

[0144] 14 display units

[0145] 15 Communication Unit

[0146] 16 Image acquisition unit

[0147] 17Dialogue state determination unit

[0148] 18 sound processing units

[0149] 19 Sound output unit

[0150] 20 Data Learning Units

[0151] 31Facial recognition unit

[0152] 32 Expression recognition units

[0153] 33 Expression conversion unit

[0154] 41Facial Landmark Recognition Unit

[0155] 42 Action Correspondence Table

[0156] 51 Voice Recognition Unit

[0157] 52 Voice to Text Conversion Units

[0158] 53 Sentiment Analysis Unit

[0159] 54 Expression Learning Unit

[0160] 61 Data accumulation unit

Claims

1. An information processing device, comprising: A processing unit estimates, based on facial images of participants participating in a conversation, the dominance of the participants over the conversation.

2. The information processing device according to claim 1, wherein: The processing unit estimates the dominance of the participant based on the facial expressions of the participant.

3. The information processing device according to claim 2, wherein: The processing unit detects the facial expressions of the participant based on facial landmarks detected from the facial image.

4. The information processing device according to claim 3, wherein: The processing unit detects the facial expression of the participant through a combination of facial expression movement elements.

5. The information processing device according to claim 1, wherein: The processing unit estimates the dominance based on the facial expressions of the participant when the participant speaks and the facial expressions of the participant when the participant does not speak, which are recognized based on the facial image.

6. The information processing device according to claim 1, wherein: The processing unit obtains the dominance of the other party of the participant over the conversation, the other party participating in the conversation, and The dominance of the participant is estimated as a value indicating a ratio to the dominance of the other party.

7. The information processing device according to claim 5, wherein: When the facial expression of the participant when the participant is silent is regarded as an facial expression of active participation in the conversation, the processing unit increases the dominance of the participant.

8. The information processing device according to claim 7, wherein: When the facial expression of the participant is nodding, agreeing or smiling, the processing unit determines that the facial expression of the participant is regarded as an facial expression of actively participating in the conversation.

9. The information processing device according to claim 5, wherein: When the facial expression of the participant when the participant is silent is regarded as a facial expression of not actively participating in the conversation, the processing unit reduces the dominance of the participant.

10. The information processing device according to claim 9, wherein: When the participant's expression action is an expression action of pressing lips or when the sight is away from the direction of the imaging unit that captures the facial image, the processing unit determines that the participant's expression action is an expression action regarded as not actively participating in the conversation.

11. The information processing device according to claim 1, further comprising: a display unit that displays a facial image of another party of the participants participating in the conversation; as well as A conversion unit changes a part of the facial image of the other party displayed on the display unit to convert the expression of the other party according to the dominance of the participant over the conversation.

12. The information processing device according to claim 11, wherein: The conversion unit changes the corners of a mouth in the facial image of the other party.

13. The information processing device according to claim 1, wherein: The conversion unit changes a part of a facial image of the participant or the other of the participants so that the dominance of the participant satisfies a preset condition.

14. The information processing device according to claim 1, further comprising: a voice output unit configured to output a voice of another party of the participants participating in the conversation; as well as A voice processing unit changes the sound quality of the other party's voice output from the voice output unit according to the degree of dominance of the participant over the conversation.

15. The information processing device according to claim 14, wherein: The sound output unit changes the sound quality of the other party's voice by adopting pitch shifting or an equalizer.

16. The information processing device according to claim 1, wherein: The sound processing unit changes the sound quality of the voice of the participant or the other of the participants so that the dominance of the participant satisfies a preset condition.

17. The information processing device according to claim 1, wherein: The processing unit matches the participant participating in the conversation with another party of the participant according to the dominance of the participant.

18. The information processing device according to claim 17, wherein: The processing unit performs matching between the other party whose tendency is opposite to that of the participant and the participant, with respect to the high and low tendencies of the dominance, the other party participating in the conversation.

19. An information processing method, comprising: The dominance of a participant participating in a conversation over the conversation is estimated by a processing unit of the information processing device based on a facial image of the participant.

20. A program for causing a computer to: A processing unit estimates, based on facial images of participants participating in a conversation, the dominance of the participants over the conversation.

Images