Information processing device, method, and program
The information processing device addresses the challenge of conveying emotions in virtual spaces by recognizing and controlling avatar expressions using emotional recognition units and models, resulting in improved emotional communication in virtual environments.
Patent Information
- Application Number
- PCT/JP2024/016190
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-04-25
- Publication Date
- 2024-11-21
AI Technical Summary
Communication in virtual spaces, such as the Metaverse, often conveys less information than face-to-face interactions, as subtle facial expressions and movements may not be accurately reflected in avatars, leading to potential misinterpretation of emotional cues.
An information processing device and method that includes an emotional expression recognition unit, an emotional expression control section, and an avatar generation section, which recognize and control the emotional expression of a user's avatar based on sensing data, using models like RUSSELL's circular model and Facial Action Units (AU) to generate avatars that reflect controlled emotional expressions, thereby enhancing emotional conveyance in virtual communication.
This solution enables smoother and more accurate emotional communication in virtual spaces by generating avatars that reflect controlled emotional expressions, allowing users to convey emotions effectively without overwhelming or misrepresenting their emotional states.
Smart Images

Figure JP2024016190_21112024_PF_FP_ABST
Abstract
Description
Information processing device, method, and program
[0001] The present technology relates to an information processing device, method, and program, and more particularly to an information processing device, method, and program that enable smoother communication in a virtual space.
[0002] Communication using avatars in virtual spaces has become widespread.
[0003] For example, Patent Document 1 describes a technology that acquires user data from voice, movement, and images and generates facial expressions for an avatar in real time.
[0004] Japanese Patent Application Laid-Open No. 2020-068404
[0005] However, communication in a virtual space conveys less information than face-to-face communication. Even if the same subtle facial expressions and movements as in face-to-face communication are recognized and reflected in the avatar, the information may not be conveyed as expected by the user.
[0006] The present technology has been developed in light of these circumstances, and is intended to enable smoother communication.
[0007] An information processing device according to one aspect of the present technology includes an emotional expression recognition unit that recognizes an emotional expression of a user in accordance with a result of sensing, an emotional expression control unit that controls the emotional expression of an avatar of the user at the time of the sensing based on the recognized emotional expression, and an avatar generation unit that generates an avatar that reflects the controlled emotional expression.
[0008] In one aspect of the present technology, a user's emotional expression is recognized based on the results of sensing, and the emotional expression of the user's avatar at the time of the sensing is controlled based on the recognized emotional expression, and an avatar reflecting the controlled emotional expression is generated.
[0009] 15 is a block diagram showing an example configuration of an ambient computing system according to an embodiment of the present technology. FIG. 16 is a block diagram showing an example configuration of a communication system according to a first embodiment of the present technology. FIG. 17 is a diagram showing an example of first emotional expression control. FIG. 18 is a flowchart explaining the processing of the communication system of FIG. 2. FIG. 19 is a diagram showing a modified example of the first emotional expression control of FIG. 3. FIG. 19 is a block diagram showing an example configuration of a communication system according to a second embodiment of the present technology. FIG. 20 is a diagram showing an example of an AU score used in the second emotional expression control. FIG. 21 is a diagram showing an example of a threshold value set for AU. FIG. 19 is a diagram showing control processing of the communication system of FIG. 6. FIG. 21 is a block diagram showing a first modified example of the communication system of FIG. 6. FIG. 22 is a diagram showing an example of a selection screen. FIG. 23 is a diagram showing control processing of the communication system of FIG. 10. FIG. 24 is a diagram showing another example of the control processing of the communication system of FIG. 10. FIG. 25 is a flowchart explaining the processing of the communication system of FIG. 10. FIG. 26 is a block diagram showing a second modified example of the communication system of FIG. 6. FIG. 27 is a diagram showing control processing of the communication system of FIG. 15.
[0010] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Overall system 2. First embodiment of the present technology (relaxation of emotional expression) 3. Second embodiment of the present technology (emphasis on ambiguous emotional expression) 4. Others
[0011] <1. Overall System> <System Configuration> FIG. 1 is a diagram showing an example of the configuration of an ambient computing system according to an embodiment of the present technology.
[0012] The ambient computing system 1 in FIG. 1 is a system that detects information about the user's situation and surrounding environment, controls each device based on the detected information and prior information, and provides the services desired by the user.
[0013] In providing such services, the ambient computing system 1 provides, as a function related to the present technology, a service in which, for example, the emotional expression of a user is recognized based on the user's sensing results, the emotional expression of an avatar at the time of sensing is controlled based on the user's emotional expression recognized at the time of sensing, and an avatar with the controlled emotional expression is presented.
[0014] The ambient computing system 1 is configured to include a user interface 21, an environmental prior information acquisition unit 22, a user prior information acquisition unit 23, a collaborative sensing unit 24, a surrounding environment detection unit 25, a user state detection unit 26, a user response detection unit 27, a context management unit 28, an equipment control unit 29, an environmental context model memory unit 30, a user context model memory unit 31, and a service list memory unit 32.
[0015] In this specification, the term "context" refers to information including user information, environmental information, and the relationship between these pieces of information. The term "context model" refers to a model used by the context management unit 28 when learning, and is also a model (i.e., an algorithm or machine learning model) used by the context management unit 28 to build personalized services for users.
[0016] The user interface 21 performs interface functions for the user, such as sending notifications to the user and receiving instructions from the user.
[0017] The ambient computing system 1 has two paths for acquiring information.
[0018] The first path is a path for acquiring prior information in advance based on user input, and is realized by the prior environment information acquisition unit 22 and the prior user information acquisition unit 23.
[0019] The environmental prior information acquisition unit 22 acquires prior information about the environment based on a user input, and outputs the acquired environmental prior information to the context management unit 28 as environmental information.
[0020] As a function related to the present technology, in particular, the environmental prior information acquisition unit 22 outputs, for example, information about the environment in the virtual space (whether it is a world modeled after a city, whether it is attending a class, whether it is a conversation between two people, etc.) to the context management unit 28 as environmental information.
[0021] The user prior information acquisition unit 23 acquires prior information about the user based on the user input, and outputs the acquired user prior information to the context management unit 28 as user information.
[0022] As a function related to the present technology, in particular, the user advance information acquisition unit 23 outputs, for example, information indicating the type of viewing content being viewed by the user to the context management unit 28 as user information.
[0023] The second path is a path that detects the environment and the user's current state information (in this technology, the state that the user is watching VR) through a recognizer that performs sensing by coordinating multiple sensing devices such as IoT (Internet of Things) and handles the sensing information, and is also capable of multimodal processing. Multimodal processing refers to processing multiple types of information at once. The second path is realized by a cooperative sensing unit 24 as a recognizer, a surrounding environment detection unit 25, a user state detection unit 26, and a user response detection unit 27.
[0024] The collaborative sensing unit 24 acquires sensing information from multiple sensing devices and performs predetermined signal processing on the acquired information to make it easier to handle in subsequent processing, such as removing noise components and performing data cleansing to synchronize data. The collaborative sensing unit 24 outputs the signal-processed information to the detection unit corresponding to the information, among the surrounding environment detection unit 25, the user state detection unit 26, and the user response detection unit 27.
[0025] The surrounding environment detection unit 25 detects surrounding information such as images, temperature, and sound from the information supplied by the collaborative sensing unit 24, and outputs information regarding the environment obtained as a result of the detection (hereinafter referred to as environmental information) to the context management unit 28.
[0026] The user state detection unit 26 detects information about the user from the information supplied from the collaborative sensing unit 24. For example, the user state detection unit 26 performs individual user identification by facial recognition using RGB images, detects the user's behavior from skeletal structure detection and environmental map information, and detects the user's emotions, such as comfort or discomfort, by multimodally processing facial expressions acquired from an RGB camera and vital information, such as body temperature, acquired from a wearable device.
[0027] The user state detection unit 26 outputs information about the user obtained as a result of the detection (hereinafter referred to as user information) to the context management unit 28 .
[0028] As a function related to the present technology, in particular, the user state detection unit 26 detects, for example, an emotional expression of the user and outputs the user information obtained as a result to the context management unit 28. Emotional expression means conveying an emotion (feeling) to another person or conveying an emotion to another person, and in this specification, emotional expression includes the emotion itself.
[0029] The user response detection unit 27 detects the user's response to the service provided by the context management unit 28 from the information supplied from the collaborative sensing unit 24, and outputs the user's response information and feedback information obtained as a result of the detection to the context management unit 28.
[0030] As a function related to the present technology, in particular, the user response detection unit 27 detects the user's emotional expression in response to services such as the presentation of an avatar provided by the context management unit 28, and outputs the resulting user response information and feedback information to the context management unit 28.
[0031] The context management unit 28 learns an environmental context model, such as an environment that is likely to be created in a certain time period and in a certain area, by accumulating information supplied from the surrounding environment detection unit 25, the user state detection unit 26, and the user response detection unit 27. The context management unit 28 stores the learned environmental context model in the environmental context model storage unit 30.
[0032] The context management unit 28 learns a model relating to the user's habitual behavioral patterns, such as what activity the user is performing in what situation, as a user context model by accumulating information supplied from the surrounding environment detection unit 25, the user state detection unit 26, and the user response detection unit 27. The context management unit 28 stores the learned user context model in the user context model storage unit 31.
[0033] The context management unit 28 learns each context model, and when information indicating a similar situation is obtained, it controls the device control unit 29 to provide personalized services that use the user context model to anticipate actions for the user from devices that are linked to the ambient computing system 1.
[0034] The context management unit 28 receives user responses when a personalized service is provided via the user response detection unit 27, thereby obtaining feedback information on whether the service was actually appropriate, and re-learning each context model to improve the accuracy of each context model.
[0035] Furthermore, by repeatedly learning context models of habitual behavioral patterns, the context management unit 28 can also model behavioral norms that indicate which options a user is likely to choose in certain situations, making it possible to provide a personalized service to the user in advance, even if the user does not habitually do something.
[0036] As a function related to this technology, in particular, the context management unit 28, for example, recognizes the user's emotional expression, controls the emotional expression of the avatar based on the recognized emotional expression of the user, and controls the device control unit 29 to provide the controlled avatar information from an HMD (head-mounted display), which is one of the devices that the ambient computing system 1 is linked to.
[0037] Under the control of the context management unit 28, the device control unit 29 causes a device (for example, an HMD in the case of the present technology) linked to the ambient computing system 1 to provide personalized services.
[0038] The environmental context model storage unit 30 stores the environmental context model. The environmental context model storage unit 30 also stores, as environmental context, information used when learning the environmental context model and information including the relationships between the information.
[0039] The user context model storage unit 31 stores a user context model, and also stores information used when learning the user context model and information including the relationship between the information and the user context as user context.
[0040] The service list storage unit 32 stores a list of devices that perform services that can be provided to users.
[0041] Hereinafter, the functions of the present technology in the ambient computing system 1 of FIG. 1 will be described in detail.
[0042] 2. First Embodiment of the Present Technology (Relaxation of Emotional Expression) Configuration of Communication System FIG. 2 is a block diagram showing an example configuration of a communication system according to a first embodiment of the present technology.
[0043] FIG. 2 shows an example of the configuration of a communication system 51 having the functions of the present technology, which is part of the ambient computing system 1 of FIG.
[0044] The communication system 51 is configured to include a sensing unit 61 , an emotional expression recognition unit 62 , an emotional expression control unit 63 , an avatar generation unit 64 , and a presentation control unit 65 .
[0045] The sensing unit 61 corresponds to the collaborative sensing unit 24, the surrounding environment detection unit 25, the user state detection unit 26, and the user response detection unit 27 in Fig. 1. The emotional expression recognition unit 62, the emotional expression control unit 63, and the avatar generation unit 64 correspond to the context management unit 28 in Fig. 1. The presentation control unit 65 corresponds to the device control unit 29 in Fig. 1.
[0046] When a user participates in a virtual space such as the Metaverse using a head-mounted display (HMD) and communicates using an avatar, the sensing unit 61 may be, for example, a sensor such as a camera in the HMD worn by the user, which captures an image of the user's face and detects facial expressions. In this case, the sensing unit 61 may also be a sensor provided in a controller held by the user, which detects the user's body movements.
[0047] Alternatively, if a user is participating in a virtual space using a PC (personal computer) or smartphone, the sensing unit 61 consists of a web camera or a smartphone camera, and by performing sensing, it acquires images of the user, detects facial expressions, and detects the user's body movements.
[0048] The sensing unit 61 outputs information indicating the detected facial expressions and body movements of the user to the emotional expression recognition unit 62. If the sensing unit 61 is composed of a microphone, it collects sound and outputs information about the collected sound to the emotional expression recognition unit 62.
[0049] The emotional expression recognition unit 62 recognizes emotions and emotional expressions from the user's facial expressions and body movement information detected by the sensing unit 61. For example, emotions and emotional expressions such as anger and joy are detected. In this specification, the emotional expressions include angry emotions, facial expressions that appear to be angry, expressions that show anger, gestures that show disbelief, and arrogant attitudes.
[0050] 2 recognizes emotional expressions from the user's facial expressions and body movement information detected by the sensing unit 61. The emotional expression recognition unit 62 outputs information indicating the recognized emotional expression (x, y in the case of FIG. 2 ) to the emotional expression control unit 63.
[0051] The emotional expression control unit 63 outputs control information (x', y' in the case of FIG. 2) for controlling the emotional expression of the user's avatar to the avatar generation unit 64 based on the information indicating the emotional expression supplied from the emotional expression recognition unit 62. Note that x, y and x', y' will be described later with reference to FIG. 3.
[0052] The avatar generation unit 64 generates an avatar of the user that reflects the control information supplied from the emotion expression control unit 63 , and outputs information indicating the generated avatar to the presentation control unit 65 .
[0053] The presentation control unit 65 causes an avatar corresponding to the information supplied from the avatar generation unit 64 to be presented.
[0054] As described above, the communication system 51 recognizes the user's emotional expression based on the user's sensing results, controls the emotional expression of the avatar at the time of sensing based on the user's emotional expression recognized at that time, and presents an avatar with the controlled emotional expression.
[0055] <First Emotion Expression Control> FIG. 3 is a diagram showing an example of the first emotion expression control.
[0056] Hereinafter, the first emotional expression control by the emotional expression control unit 63 will be described with reference to FIG.
[0057] The emotional expression recognition unit 62 estimates emotional expressions from facial expressions, voice, and body movements using, for example, the algorithm described in “Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition”, D. Kollias, S. Zafeiriou, arxiv.org / abs / 1811.07770v2, 13 Dec 2019, and maps them onto Russell's circular ring model.
[0058] FIG. 3A is a simplified diagram showing Russell's circular ring model.
[0059] In Russell's circumplex model, the X-axis ranges from -1 to 1 and indicates unpleasant or pleasant feelings, and the Y-axis ranges from -1 to 1 and indicates non-arousal or non-arousal feelings, with larger absolute values indicating stronger feelings.
[0060] The emotional expression recognition unit 62 outputs the two axes x and y in the Russell's circular model to the emotional expression control unit 63 as information indicating the emotional expression.
[0061] For example, if a user feels and expresses "anger" in communication in a virtual space, the emotional expression recognition unit 62 detects the expression of "anger" felt by the user (for example, x = -1.0, y = 1.0). Since the information that can be transmitted in a virtual space is limited, if that emotional expression is directly reflected in an avatar, it may be perceived by the other person as an overly strong emotional expression.
[0062] The emotional expression control unit 63 performs control to soften the emotional expression recognized by the emotional expression recognition unit 62. That is, the emotional expression control unit 63 sets conversion coefficients in a conversion coefficient table according to the X-axis value and Y-axis value recognized by the emotional expression recognition unit 62, and reflects the conversion coefficients in the X-axis value and Y-axis value.
[0063] FIG. 3B is a diagram showing an example of a conversion coefficient table.
[0064] In the conversion coefficient table, values on the X and Y axes corresponding to the conversion coefficients are shown.
[0065] A conversion factor of 0.6 corresponds to an X-axis value less than -0.8 and a Y-axis value greater than 0.8. A conversion factor of 0.7 corresponds to an X-axis value greater than or equal to -0.8 but less than -0.6, and a Y-axis value greater than 0.6 but less than 0.8. A conversion factor of 0.8 corresponds to an X-axis value greater than or equal to -0.6 but less than -0.4, and a Y-axis value greater than 0.4 but less than 0.6.
[0066] For example, as shown in A of FIG. 3, when the value (-1.0) of the recognition point X of anger on Russell's circular ring model is smaller than -0.8 and the value (1.0) of the recognition point Y is larger than 0.8 (-1 or close to 1 = strong emotion), the emotion expression control unit 63 outputs control information (x', y') obtained by multiplying x and y by a conversion coefficient of 0.6 to the avatar generation unit 64.
[0067] At this time, the avatar generation unit 64 generates an avatar by suppressing (reflecting) the avatar's expression (skeletal movement) that represents (the strength of) emotion to 60%.
[0068] This allows you to communicate with the other person that you feel angry, without using overly strong language that may shock or offend them, and allows you to continue communicating.
[0069] Furthermore, body movements that express anger, such as clenching hands or crossing arms, are described in "A Bimodal Face and Body Gesture Database for Automatic Analysis of Human Nonverbal Affective Behavior," Proceedings of the 18th International Conference on Pattern Recognition (ICPR'06).
[0070] <Processing of the Communication System> FIG. 4 is a flowchart illustrating the processing of the communication system 51 of FIG.
[0071] In step S11, the sensing unit 61 is made up of a web camera or a smartphone camera, and performs sensing to acquire an image of the user, detect facial expressions, and detect the user's body movements.
[0072] In step S12 , the emotional expression recognition unit 62 recognizes an emotional expression from the user's facial expression and body movement information detected by the sensing unit 61 .
[0073] In step S13, the emotional expression control unit 63 controls the emotional expression of the user's avatar based on the information indicating the emotional expression supplied from the emotional expression recognition unit 62. For example, the emotional expression control unit 63 outputs control information for controlling the emotional expression of the user's avatar to the avatar generation unit 64.
[0074] In step S14 , the avatar generation unit 64 generates an avatar of the user that reflects the control information supplied from the emotion expression control unit 63 , and outputs information indicating the generated avatar to the presentation control unit 65 .
[0075] In step S15, the presentation control unit 65 presents an avatar corresponding to the information supplied from the avatar generation unit 64. Thereafter, the processing of the communication system 51 ends.
[0076] <Modification of First Emotion Expression Control> FIG. 5 is a diagram showing a modification of the first emotion expression control of FIG.
[0077] In FIG. 5, the user feels "uncomfortable" in communication in the virtual space, and the emotional expression recognition unit 62 detects the expression of "uncomfortable" that the user feels (for example, x=-1.0, y=0).
[0078] The emotional expression control unit 63 sets conversion coefficients and controlled values of a plurality of proposed proposals in a conversion coefficient table according to the X-axis value and Y-axis value recognized by the emotional expression recognition unit 62, and reflects the conversion coefficients and controlled values in the X-axis value and Y-axis value.
[0079] FIG. 5B is a diagram showing an example of a conversion coefficient table.
[0080] The conversion coefficient table shows a plurality of proposals, and pre-control values of the X-axis and Y-axis corresponding to the conversion coefficients of each proposal, as well as post-control values of the Y-axis.
[0081] In Proposal 1, a conversion coefficient of 0.6 corresponds to an X-axis value less than −0.8 and a Y-axis value of 1, and the Y-axis value after control becomes +0.2.
[0082] In proposal 2, a conversion coefficient of 0.6 corresponds to an X-axis value smaller than −0.8 and a Y-axis value of 1, and the Y-axis value after control is 0.
[0083] In proposal 3, a conversion coefficient of 0.6 corresponds to an X-axis value smaller than −0.8 and a Y-axis value of 1, and the Y-axis value after control is −0.2.
[0084] For example, as shown in A of FIG. 5, when discomfort (x=-1.0, y=0) is recognized on the Russell ring model, the emotional expression control unit 63 multiplies the value of X by a coefficient of 0.6, and controls the value of X to -0.6.
[0085] On the other hand, the emotional expression control unit 63 controls the value of Y to +0.2, 0, and −0.2 for proposals 1 to 3, respectively.
[0086] The avatar generation unit 64 generates an avatar so that if the value of Y is +0.2, it includes a worried expression; if it is 0, it includes only a displeased expression; and if it is −0.2, it includes a sad expression.
[0087] The presentation control unit 65, under the control of the emotion expression control unit 63, presents, for example, proposals 1 to 3 so that the user can select one, as shown in Fig. 11 described later. The presentation control unit 65 controls the presentation of an avatar corresponding to the proposal selected by the user from proposals 1 to 3.
[0088] As described above, suggestions may be presented that include not only "discomfort" but also different expressions surrounding "discomfort," allowing the user to select an avatar to be output depending on how they want to communicate.
[0089] 2. Second Embodiment of the Present Technology (Emphasis on Ambiguous Expressions) Configuration of HMD Communication System FIG. 6 is a block diagram showing an example configuration of a communication system according to a second embodiment of the present technology.
[0090] FIG. 6 shows an example of the configuration of a communication system 101 having the functions of the present technology, which is part of the ambient computing system 1 of FIG.
[0091] The communication system 101 differs from the communication system 51 of FIG. 2 in that the emotional expression recognition unit 62 is replaced with an emotional expression recognition unit 111 and the emotional expression control unit 63 is replaced with an emotional expression control unit 112.
[0092] The emotional expression recognition unit 111 recognizes emotional expressions from the user's facial expressions and body movement information detected by the sensing unit 61. The emotional expression recognition unit 111 outputs information indicating the recognized emotional expression (in the case of FIG. 6 , an AU (Facial Action Unit) score = 0 to 1) to the emotional expression control unit 112. The AU score will be described later with reference to FIG. 7 .
[0093] The emotional expression control unit 112 outputs control information for controlling the emotional expression of the user's own avatar (in the case of Figure 6, control information indicating whether or not to perform a specified action) to the avatar generation unit 64 based on the information indicating the emotional expression supplied from the emotional expression recognition unit 111.
[0094] As described above, the communication system 101 recognizes the user's emotional expression based on the user's sensing results, controls the emotional expression of the avatar at the time of sensing based on the user's emotional expression recognized at that time, and presents an avatar with the controlled emotional expression.
[0095] <Second Emotion Expression Control> FIG. 7 is a diagram showing an example of an AU score used in the second emotion expression control.
[0096] The second emotional expression control by emotional expression control unit 112 will be described below with reference to FIG.
[0097] The emotional expression recognition unit 111 can acquire information equivalent to the Facial Action Unit described in Observer-Based_Measurement_of_Facial_Expression_wi0-7695, for example.
[0098] Facial Action Units are assigned numbers.
[0099] 7 shows, as an example of an AU, an AU related to the movement (expression) of asking a question because one does not understand. An AU related to the movement of asking a question is an AU that represents the facial expression, gesture, and expression that a person who asks a question tends to make. Asking a question is something that is done when one does not understand something, and in this case, possible movements include furrowing one's eyebrows, pursing one's mouth, or slightly opening one's mouth to ask a question, and among the AUs, these correspond to AU4, AU17, AU25, etc.
[0100] A of Fig. 7 is a diagram showing a movement to which AU4 is assigned. A of Fig. 7 shows the face of a person furrowing their eyebrows, that is, AU4 represents the movement of furrowing their eyebrows.
[0101] B of Fig. 7 is a diagram showing a movement to which AU 17 is assigned. B of Fig. 7 shows the face of a person making a movement of sagging their mouth, that is, AU 17 represents the movement of sagging their mouth.
[0102] Fig. 7C is a diagram showing a movement to which AU 25 is assigned. Fig. 7C shows the face of a person performing a movement of slightly opening their mouth, that is, AU 25 represents the movement of slightly opening their mouth.
[0103] Each AU takes a value between 0 and 1, and the greater the magnitude of the movement, the closer it approaches 1.
[0104] The emotional expression recognition unit 111 outputs the AU score (AU value) to the emotional expression control unit 63 .
[0105] The emotional expression control unit 112 performs control to emphasize, as necessary, the emotional expression recognized by the emotional expression recognition unit 111. The emotional expression control unit 112 determines whether to emphasize an emotional expression based on the AU score supplied from the emotional expression recognition unit 111 and a threshold value set for the AU.
[0106] FIG. 8 is a diagram showing an example of threshold values set for AU.
[0107] FIG. 8 shows the score range, threshold value, and emphasizing emotional expression set for the AU related to the question.
[0108] When the only AU related to the question is AU4, the score range is set to 0 to 1, the threshold is set to 0.8, and the emphasized emotional expression is set to "raise hand."
[0109] When the AUs related to the question are AU4 and AU17, the score range is set to 0 to 1, the threshold is set to 0.6, and the emotional expression to be emphasized is set to "raise hand."
[0110] When the AUs related to the question are AU4 and AU25, the score range is set to 0 to 1, the threshold is set to 0.6, and the emotional expression to be emphasized is set to "raise hand."
[0111] For example, when a user is listening to a lecture in a virtual space, the emotional expression control unit 112 determines whether the user does not understand and wants to ask a question based on the size of AU4, AU17, and AU25.
[0112] When the value of AU4 is 0.8 or more, the emotional expression control unit 112 determines that the user does not understand and wants to ask a question.
[0113] When the values of AU4 and AU17 are 0.6 or more, the emotional expression control unit 112 determines that the user does not understand and wants to ask a question.
[0114] When the values of AU4 and AU25 are equal to or greater than 0.6, the emotional expression control unit 112 determines that the user does not understand and wants to ask a question.
[0115] When the emotional expression control unit 112 determines that the user is in a state where he or she wants to ask a question, it outputs control information to the avatar generation unit 64 indicating that the user should raise his or her hand as an expression that conveys that the user wants to ask a question.
[0116] The avatar generation unit 64 generates an avatar with its hand raised, and outputs information indicating the generated avatar to the presentation control unit 65 .
[0117] If the emotional expression control unit 112 determines that the user is not in a state where he or she wants to ask a question, control information indicating that the user should not raise his or her hand is output to the avatar generation unit 64 .
[0118] The above-mentioned function can be turned on when the user wants to actively listen to a lecture. For example, if it is detected that the user is passive while listening to a lecture, it is possible to control the avatar to appear as if the user is actively listening.
[0119] <Control Processing> FIG. 9 is a diagram showing the control processing of the communication system 101 of FIG.
[0120] For example, as shown in FIG. 9A, while listening to a lecture in virtual space 131, the user opens his mouth to ask a question.
[0121] The sensing unit 61 outputs information indicating the detected facial expressions and body movements of the user to the emotional expression recognition unit 111. The emotional expression recognition unit 111 recognizes the emotional expression of the user that has been detected by the sensing unit 61, that is, that the user has opened their mouth, and outputs an AU25 score of 0.8 to the emotional expression control unit 112.
[0122] When the values of AU4 and AU25 are 0.6 or more, the emotional expression control unit 112 determines that the user does not understand and wants to ask a question, and outputs control information indicating that the user should raise their hand to the avatar generation unit 64 as an expression conveying that they want to ask a question.
[0123] The avatar generation unit 64 generates an avatar of the user that reflects the control information supplied from the emotion expression control unit 112 , and outputs information indicating the generated avatar to the presentation control unit 65 .
[0124] The presentation control unit 65 causes the avatar 132B supplied from the avatar generation unit 14 to be presented in the virtual space 131, as shown in B of FIG. 9 .
[0125] In other words, under the control of the emotional expression control unit 112, the user's emotional expression at the time of detection is emphasized, and as a result, avatar 132B is presented in the virtual space 131, raising its hand to ask a question to avatar 132A, who is the avatar of the lecturer in the class.
[0126] <Configuration of HMD Communication System> FIG. 10 is a block diagram showing a first modified example of the communication system of FIG.
[0127] The communication system 151 in Fig. 10 differs from the communication system 101 in Fig. 6 in that a context update unit 161 and a user context model storage unit 162 are added. The communication system 151 also differs from the communication system 101 in Fig. 6 in that the avatar generation unit 64 is replaced with an avatar generation unit 163 and the presentation control unit 65 is replaced with a presentation control unit 164.
[0128] The context update unit 161 corresponds to a part of the context management unit 28 in FIG. 1, and the user context model storage unit 162 corresponds to the user context model storage unit 31 in FIG.
[0129] The context update unit 161 updates the user context model stored in the user context model storage unit 162 based on the user's response of selecting one of multiple avatars, which is supplied via the sensing unit 61, the emotional expression recognition unit 111, and the emotional expression control unit 112.
[0130] In this way, the user's unique emotional expressions (for example, tilting their head, shaking their head, or widening their eyes when they don't understand something) and the emotional expression that the user likes (raising their hand) among the emotional expressions controlled when that expression occurs are learned as a user context model and can be used the next time it is expressed.
[0131] The user context model storage unit 162 stores the user context model.
[0132] The avatar generation unit 163 generates multiple avatars for the user that reflect the control information supplied from the emotion expression control unit 112, and outputs selection screen information to the presentation control unit 164 for allowing the user to select from the multiple generated avatars.
[0133] The presentation control unit 164 controls the presentation of a selection screen that allows the user to select from a plurality of avatars, based on the selection screen information supplied from the avatar generation unit 163 .
[0134] As described above, the communication system 151 recognizes the user's emotional expression based on the user's sensing results, controls the emotional expression of the avatar at the time of sensing based on the user's emotional expression recognized at the time of sensing, and presents an avatar with the controlled emotional expression.
[0135] <Selection Screen> FIG. 11 is a diagram showing an example of a selection screen.
[0136] FIG. 11 shows a selection screen 171 whose presentation is controlled by the presentation control unit 164 .
[0137] An avatar 172A, an avatar 172B, and an avatar 172C are presented on the selection screen 171. The user can move a cursor 173 to select one of the avatars 172A to 172C.
[0138] The selection screen 171 may be presented superimposed on the virtual space of the HMD, similar to the presentation of an avatar, or may be presented on a presentation unit separate from the virtual space.
[0139] <Control Processing> FIG. 12 is a diagram showing the control processing of the communication system 151 of FIG.
[0140] For example, as shown in FIG. 12A, during communication in the virtual space 131, the user tilts his / her head because he / she does not understand something.
[0141] The sensing unit 61 outputs information indicating the detected facial expressions and body movements of the user to the emotional expression recognition unit 111. The emotional expression recognition unit 111 outputs an AU having a number corresponding to the emotional expression of the user tilting their head detected by the sensing unit 61 to the emotional expression control unit 112.
[0142] The emotional expression control unit 112 determines that the user is in an unsure state based on the AU number, value, and threshold value, and outputs control information indicating that the user is unsure to the avatar generation unit 163.
[0143] The avatar generation unit 163 generates a plurality of avatars representing an unknown state, and outputs selection screen information for selecting from the plurality of generated avatars to the presentation control unit 65 .
[0144] Based on the selection screen information supplied from the avatar generation unit 163, the presentation control unit 65 presents a selection screen 171 on which avatars 181A to 181C can be selected, for example, by superimposing it on the virtual space 131, as shown in B of Figure 12.
[0145] The user can move cursor 182 to select one of avatars 181A to 183C.
[0146] A response from the user selecting one of the multiple avatars is supplied to the context update unit 161 and the avatar generation unit 163 via the sensing unit 61 , the emotional expression recognition unit 111 and the emotional expression control unit 112 .
[0147] The context update unit 161 updates the user context model stored in the user context model storage unit 162 based on the supplied user response.
[0148] The avatar generation unit 163 outputs information indicating an avatar based on the user's response to the presentation control unit 164 .
[0149] The presentation control unit 164 causes the avatar 183B corresponding to the information supplied from the avatar generation unit 163 to be presented in the virtual space 131 .
[0150] That is, under the control of the emotional expression control unit 112, the emotional expression of the user at the time of detection is emphasized, and as a result, avatar 183B is presented in the virtual space 131 with arms outstretched, shoulders raised, and expressing that it does not understand, in contrast to avatar 183A, which is the avatar of the communication partner.
[0151] At this time, a small screen 184 for monitoring an actual image of the user as a result of sensing may be displayed in a corner of the virtual space 131.
[0152] As described above, avatars are presented that emphasize small movements that are difficult to convey, i.e., emotional expressions, so communication in virtual space can be carried out smoothly.
[0153] FIG. 13 is a diagram showing another example of the control process of the communication system 151 of FIG.
[0154] For example, as shown in FIG. 13A, during communication in the virtual space 131, the user mutters "Huh?" because he or she has difficulty hearing.
[0155] The sensing unit 61 senses information indicating the user's facial expressions and body movements as well as information about the user's voice, and outputs this information to the emotional expression recognition unit 111. The emotional expression recognition unit 111 adds the user's voice saying "Huh?" to the AU number and value detected by the sensing unit 61, and outputs this to the emotional expression control unit 112.
[0156] The emotion expression control unit 112 outputs the AU number and value, the threshold value, and the voice to the context update unit 161. For example, the user's voice saying "Huh?" is learned in the user context model storage unit 162 as a context expressing that it is difficult to hear. The context update unit 161 searches the user context model storage unit 162 and extracts the user's voice saying "Huh?" as a context expressing that it is difficult to hear.
[0157] The emotion expression control unit 112 outputs control information based on the extracted context to the avatar generation unit 163. The avatar generation unit 163 generates an avatar that expresses difficulty in hearing, and outputs information indicating the generated avatar to the presentation control unit 164.
[0158] The presentation control unit 164 causes the avatar 191B supplied from the avatar generation unit 163 to be presented in the virtual space 131, as shown in B of FIG.
[0159] That is, under the control of the emotional expression control unit 112, the user's expression at the time of detection is emphasized, and as a result, avatar 191B that emphasizes that the user has difficulty hearing is presented to avatar 191A, the avatar of the communication partner.
[0160] As described above, by storing and learning user responses and feedback as a user context model, even if the user expresses it in a unique way, for example, when the user feels that they have difficulty hearing, an avatar that emphasizes that they have difficulty hearing can be presented, which allows for smooth communication in the virtual space.
[0161] <Processing of the Communication System> FIG. 14 is a flowchart illustrating the processing of the communication system 151 of FIG.
[0162] Steps S111 to S113 in FIG. 14 are the same as steps S11 to S13 in FIG. 4, and therefore a description thereof will be omitted.
[0163] In step S114, the avatar generation unit 163 generates multiple avatars for the user that reflect the control information supplied from the emotional expression control unit 112, and outputs selection screen information consisting of the multiple generated avatars to the presentation control unit 164.
[0164] In step S115, the presentation control unit 164 controls the presentation of a selection screen that allows the user to select from a plurality of avatars.
[0165] The user selects one avatar.
[0166] In step S116, the sensing unit 61 outputs the detected user response to the avatar generation unit 163 and the context update unit 161 via the emotional expression recognition unit 111 and the emotional expression control unit 112.
[0167] In step S117, the avatar generation unit 163 outputs information indicating the selected avatar to the presentation control unit 164 based on the user's response, and the presentation control unit 164 presents the selected avatar based on the information supplied from the avatar generation unit 163.
[0168] In step S117, the context update unit 161 uses control information of the user's preference (e.g., information indicating that the currently selected expression should be used if the user does not understand) based on the user's response to update the context model in the user context model storage unit 162. At that time, the conversion coefficient table and the AU score threshold value may also be updated to suit the user's preference.
[0169] This makes it possible to generate and present an avatar using the updated context model (user-preferred control information) from the next time onwards.
[0170] In the process of Figure 14, an example has been described in which the context model is updated based on the user's response, but it is also possible to sense the user who looks at the presented avatar, recognize their emotional expression, and receive feedback information as to whether the user likes the presented avatar.
[0171] In this case, the emotional expression recognition unit 111 may use the X-axis (pleasant or unpleasant) of Russell's circular model, as was done by the emotional expression recognition unit 62, and send information indicating that the user likes the item if it is pleasant, and that the user does not like the item if it is unpleasant, to the context update unit 161, thereby updating the context model in the user context model storage unit 162.
[0172] <Configuration of HMD Communication System> FIG. 15 is a block diagram showing a second modified example of the communication system of FIG.
[0173] The communication system 201 differs from the communication system 101 of FIG. 6 in that a fixed control unit 211 is added.
[0174] The fixed control unit 211 corresponds to a part of the context management unit 28 in FIG.
[0175] The fixation control unit 211 fixes the avatar information generated by the avatar generation unit 64 based on the control information, or releases the fixation, in response to a user instruction.
[0176] As described above, the communication system 201 recognizes the user's emotional expression based on the user's sensing results, controls the emotional expression of the avatar at the time of sensing based on the user's emotional expression recognized at that time, and presents an avatar with the controlled emotional expression.
[0177] <Control Processing> FIG. 16 is a diagram showing the control processing of the communication system 201 of FIG.
[0178] For example, as shown in A of FIG. 16, in a virtual space 131, a user is attending a class.
[0179] The sensing unit 61 outputs information indicating the detected facial expressions and body movements of the user to the emotional expression recognition unit 111. The emotional expression recognition unit 111 outputs an AU score to the emotional expression control unit 112 based on the expression of the user staring at something detected by the sensing unit 61.
[0180] The emotional expression control unit 112 determines that the subject is in a state of understanding based on the AU score, and outputs control information indicating that the subject is nodding in understanding to the avatar generation unit 64.
[0181] The avatar generation unit 64 generates an avatar of the user that reflects the control information supplied from the emotion expression control unit 112 , and outputs information indicating the generated avatar to the presentation control unit 65 .
[0182] The presentation control unit 65 causes an avatar 231B corresponding to the information supplied from the avatar generation unit 64 to be presented in the virtual space 131, as shown in B of FIG.
[0183] That is, under the control of emotion expression control unit 112, the emotion expression of the user at the time of detection is emphasized, and as a result, avatar 231B is presented nodding in understanding to avatar 231A, which is the avatar of the lecturer of the class.
[0184] At this time, the user checks himself / herself on a small screen 232 for monitoring his / her sensed image (appearance), and is presented with avatar 231B nodding in understanding. The user turns off the control and instructs that the presented avatar be fixed to avatar 231B. The fixing control unit 211 fixes the avatar in the presented state by controlling the avatar generation unit 64 in accordance with the user's instruction.
[0185] This allows, for example, avatar 231B to be presented nodding in understanding, allowing the user to relax and listen to the lecture while eating snacks, for example.
[0186] <4. Other> <Effects of the Present Technology> With the present technology, the emotional expression of a user is recognized according to the results of sensing, the emotional expression of the user's avatar at the time of sensing is controlled based on the recognized emotional expression of the user, and an avatar that reflects the controlled emotional expression is generated.
[0187] This allows for smooth communication between users in the virtual space.
[0188] In the above explanation, for the sake of simplicity, the first emotional expression control and the second emotional expression control are described separately, but both emotional expression controls may be performed by a single emotional expression recognition unit and emotional expression control unit.
[0189] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.
[0190] FIG. 17 is a block diagram showing an example of the hardware configuration of a computer 900 that executes the above-described series of processes according to a program.
[0191] A CPU (Central Processing Unit) 901 , a ROM (Read Only Memory) 902 , and a RAM (Random Access Memory) 903 are interconnected by a bus 904 .
[0192] An input / output interface 910 is also connected to the bus 904. An input unit 911 including a keyboard, a mouse, etc., and an output unit 912 including a display, a speaker, etc. are connected to the input / output interface 910. Also connected to the input / output interface 910 are a storage unit 913 including a hard disk, a nonvolatile memory, etc., a communication unit 914 including a network interface, etc., and a drive 915 that drives a removable recording medium 921.
[0193] In the computer 900 configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing it.
[0194] The program executed by the CPU 901 is recorded on a removable recording medium 921, or is provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and installed in the storage unit 913.
[0195] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.
[0196] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.
[0197] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.
[0198] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.
[0199] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.
[0200] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.
[0201] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.
[0202] <Examples of Combinations of Configurations> The present technology may also be configured as follows. (1) An information processing device comprising: an emotional expression recognition unit that recognizes an emotional expression of a user according to a result of sensing; an emotional expression control unit that controls an emotional expression of an avatar of the user at the time of the sensing based on the recognized emotional expression of the user; and an avatar generation unit that generates an avatar that reflects the controlled emotional expression. (2) The information processing device described in (1), wherein the emotional expression control unit controls an emotional expression of the user's avatar in a direction that softens the recognized emotional expression of the user. (3) The information processing device described in (1), wherein the emotional expression control unit controls an emotional expression of the user's avatar in a direction that emphasizes the recognized emotional expression of the user. (4) The information processing device described in any of (1) to (3), wherein the emotional expression recognition unit recognizes the emotional expression of the user using Russell's circumplex model. (5) The information processing device described in any of (1) to (3), wherein the emotional expression recognition unit recognizes the emotional expression of the user using a Facial Action Unit. (6) The information processing device according to any one of (1) to (5), further comprising a presentation control unit that controls presentation of a generated avatar. (7) The information processing device according to (6), wherein the presentation control unit controls presentation of a selection screen for selecting from a plurality of avatars whose emotional expressions are controlled. (8) The information processing device according to (7), further comprising an update unit that updates a user context model based on emotional expression control applied to an avatar selected by the user from the plurality of avatars whose emotional expressions are controlled, wherein the emotional expression control unit controls emotional expression of the user's avatar by referring to the updated user context model. (9) The information processing device according to (8), wherein the update unit updates the user context model based on the emotional expression of the user recognized in accordance with the result of the sensing. (10) The information processing device according to any one of (6) to (9), wherein the presentation control unit controls presentation to a head-mounted display.(11) The information processing device according to any of (1) to (10), further comprising a fixation control unit that fixes the presented avatar to an avatar that reflects a controlled emotional expression in accordance with an operation of the user. (12) An information processing method in which an information processing device recognizes a user's emotional expression in accordance with a result of sensing, controls the emotional expression of the user's avatar at the time of the sensing based on the recognized emotional expression of the user, and generates an avatar that reflects the controlled emotional expression. (13) A program that causes a computer to function as an emotional expression recognition unit that recognizes a user's emotional expression in accordance with a result of sensing, an emotional expression control unit that controls the emotional expression of the user's avatar at the time of the sensing based on the recognized emotional expression of the user, and an avatar generation unit that generates an avatar that reflects the controlled emotional expression.
[0203] 1 Ambient computing system, 21 User interface, 22 Environmental prior information acquisition unit, 23 User prior information acquisition unit, 24 Collaborative sensing unit, 25 Surrounding environment detection unit, 26 User state detection unit, 27 User response detection unit, 28 Context management unit, 29 Device control unit, 30 Environmental context model storage unit, 31 User context model storage unit, 32 Service list storage unit, 51 Communication system, 61 Sensing unit, 62 Emotion expression recognition unit, 63 Emotion expression control unit, 64 Avatar generation unit, 65 Presentation control unit, 101 Communication system, 111 Emotion expression recognition unit, 112 Emotion expression control unit, 151 Communication system, 161 Context update unit, 162 User context model storage unit, 163 Avatar generation unit, 164 Presentation control unit, 201 Communication system, 211 Fixed control unit
Claims
1. An information processing device comprising: an emotional expression recognition unit that recognizes an emotional expression of a user based on a result of sensing; an emotional expression control unit that controls the emotional expression of the user's avatar at the time of the sensing based on the recognized emotional expression of the user; and an avatar generation unit that generates an avatar that reflects the controlled emotional expression.
2. The information processing device according to claim 1, wherein the emotional expression control unit controls the emotional expression of the user's avatar in a direction to alleviate the recognized emotional expression of the user.
3. The information processing device according to claim 1, wherein the emotional expression control unit controls the emotional expression of the user's avatar in a direction that emphasizes the recognized emotional expression of the user.
4. The information processing device according to claim 1, wherein the emotional expression recognition unit recognizes the emotional expression of the user using Russell's circular model.
5. The information processing device according to claim 1, wherein the emotional expression recognition unit recognizes the emotional expression of the user using a Facial Action Unit.
6. The information processing device according to claim 1, further comprising a presentation control unit that controls the presentation of the generated avatar.
7. The information processing device according to claim 6, wherein the presentation control unit controls the presentation of a selection screen for selecting from a plurality of avatars whose emotional expression is controlled.
8. An information processing device as described in claim 7, further comprising an update unit that updates a user context model based on emotional expression control applied to an avatar selected by the user from the plurality of avatars whose emotional expression has been controlled, wherein the emotional expression control unit controls the emotional expression of the user's avatar by referring to the updated user context model.
9. The information processing device according to claim 8, wherein the update unit updates the user context model based on an emotional expression of the user recognized according to a result of the sensing.
10. The information processing device according to claim 6, wherein the presentation control unit controls presentation to a head mounted display.
11. The information processing device according to claim 1, further comprising a fixing control unit that fixes the presented avatar to an avatar that reflects a controlled emotional expression in response to an operation by the user.
12. An information processing method in which an information processing device recognizes an emotional expression of a user based on a sensing result, controls the emotional expression of the user's avatar at the time of the sensing based on the recognized emotional expression of the user, and generates an avatar that reflects the controlled emotional expression.
13. A program that causes a computer to function as an emotional expression recognition unit that recognizes an emotional expression of a user based on the results of sensing, an emotional expression control unit that controls the emotional expression of the user's avatar at the time of said sensing based on the recognized emotional expression of the user, and an avatar generation unit that generates an avatar that reflects the controlled emotional expression.
Citation Information
Patent Citations
Computer program, and method
JP2021189544A
