Display control device, display control method, and display control program

Abstract

A display control apparatus according to an aspect of the present disclosure includes: an acquisition unit that acquires an action of a user in a real space as input information; a determination unit that determines whether an input corresponding to a pre-registered specific action is included in the input information; and a display control unit that reflects an action of the user corresponding to the input information in the avatar in the virtual space, and reflects a specific action in the avatar in the virtual space in a case where it is determined that an input corresponding to the specific action is included in the input information, and displays the avatar in the virtual space.

Description

Technical Field

[0001] The present disclosure relates to a display control device, a display control method, and a display control program for expressing an avatar in virtual reality. Background Art

[0002] Various types of technology for presenting a three-dimensional virtual space called a metaverse or the like configured in a computer or a computer network to a user have been put into practical use.

[0003] For example, a technology has been proposed for providing users with an avatar with good operability or realizing a background display that conforms to reality by applying technology related to virtual reality (VR) or augmented reality (AR) to expressions in a virtual space (for example, Patent Document 1).

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: WO 2014 / 119098 A Summary of the invention

[0007] Technical issues

[0008] However, there is still room for improvement in the methods of expressing avatars in virtual spaces.

[0009] For example, in a virtual space, users can use avatars to communicate closely with each other. Therefore, it is expected that the service side providing the virtual space not only reproduces realistic actions in avatars that conform to reality, but also provides avatars that express various emotional expressions and various actions that users like.

[0010] Therefore, the present disclosure proposes a display control device, a display control method, and a display control program that can variously express avatars in a virtual space.

[0011] Solution to the problem

[0012] In order to solve the above-mentioned problems, a display control device according to one aspect of the present disclosure includes: an acquisition unit, which acquires the behavior of a user in a real space as input information; a determination unit, which determines whether an input corresponding to a pre-registered specific action is included in the input information; and a display control unit, which reflects the behavior of the user corresponding to the input information in an avatar in a virtual space, and further reflects the specific action in the avatar in the virtual space when it is determined that the input information includes an input corresponding to a specific action, so as to display the avatar in the virtual space. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1is a diagram showing an overview of a display control process according to an embodiment.

[0014] Figure 2 is a diagram illustrating an example of a determination process according to an embodiment.

[0015] Figure 3 : is a diagram showing a configuration example of a display control device according to an embodiment.

[0016] Figure 4 : is a table showing an example of a posture storage unit according to the embodiment.

[0017] Figure 5 This is a flowchart showing the flow of display control processing according to the embodiment.

[0018] Figure 6 FIG. 1 is a diagram for explaining a first variation of the display control process of the embodiment.

[0019] Figure 7 FIG. 2 is a diagram for explaining a first variation of the display control process of the embodiment.

[0020] Figure 8 FIG. 1 is a diagram for explaining a second variation of the display control process of the embodiment.

[0021] Fig. 9 FIG. 2 is a diagram for explaining a second variation of the display control process of the embodiment.

[0022] Fig.10 FIG. 3 is a diagram for explaining a second variation of the display control process of the embodiment.

[0023] Fig.11 FIG. 1 is a diagram for explaining a third variation of the display control process of the embodiment.

[0024] Fig.12 FIG. 2 is a diagram for explaining a third variation of the display control process of the embodiment.

[0025] Fig.13 FIG. 3 is a diagram for explaining a third variation of the display control process of the embodiment.

[0026] Fig.14 FIG. 4 is a diagram for explaining a third variation of the display control process of the embodiment.

[0027] Fig.15 FIG. 5 is a diagram for explaining a third variation of the display control process of the embodiment.

[0028] Fig.16 FIG. 6 is a diagram for explaining a third variation of the display control process of the embodiment.

[0029] Fig.17 FIG. 1 is a diagram for explaining a fourth modification of the display control process of the embodiment.

[0030] Fig.18 FIG. 2 is a diagram for explaining a fourth modification of the display control process of the embodiment.

[0031] Fig.19 FIG. 3 is a diagram for explaining a fourth modification of the display control process of the embodiment.

[0032] Fig. 20 FIG. 1 is a diagram for explaining a fifth modification of the display control process of the embodiment.

[0033] Fig.21 FIG. 2 is a diagram for explaining a fifth modification of the display control process of the embodiment.

[0034] Fig. 22 FIG. 3 is a diagram for explaining a fifth variation of the display control process of the embodiment.

[0035] Fig.23 FIG. 4 is a diagram for explaining a fifth variation of the display control process of the embodiment.

[0036] Fig.24 FIG. 5 is a diagram for explaining a fifth variation of the display control process of the embodiment.

[0037] Fig.25 is a hardware configuration diagram showing an example of a computer that realizes the function of the display control device. DETAILED DESCRIPTION

[0038] Hereinafter, the embodiments will be described in detail based on the drawings. Note that in each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant description will be omitted.

[0039] The present disclosure will be described in the order of the following items.

[0040] 1. Implementation Method

[0041] 1-1. Overview of Display Control Processing According to Embodiment

[0042] 1-2. Configuration of display control device according to embodiment

[0043] 1-3. Processing Procedure According to Embodiment

[0044] 1-4. Variations of Processing According to Embodiments

[0045] 1-4-1. Expression of activity between avatars

[0046] 1-4-2. Emotional expression of avatars

[0047] 1-4-3. Visualization of dialogue

[0048] 1-4-4. Proximity effect between avatars

[0049] 1-4-5. Expression of avatar display and interaction

[0050] 1-5. Modification

[0051] 2. Other Implementation Methods

[0052] 3. Effects of the display control device according to the present disclosure

[0053] 4. Hardware Configuration

[0054] (1. Implementation Method)

[0055] (1-1. Overview of Display Control Processing According to Embodiment)

[0056] Will refer to Figure 1 An example of a display control process according to an embodiment will be described. Figure 1 is a diagram showing an overview of a display control process according to an embodiment.

[0057] The display control process of the embodiment is performed by the display control device 100 ( Figure 1 (not shown). The display control device 100 is an information processing terminal for implementing VR or AR technology. That is, in an embodiment, the display control device 100 displays a three-dimensional virtual space called a metaverse (hereinafter referred to as "virtual space") configured in a computer or a computer network on a display unit (such as a display). In addition, the display control device 100 displays an avatar as a character that imitates the user in the virtual space. For example, the display control device 100 is implemented by a desired information processing device such as a personal computer (PC), a tablet terminal, or a smart phone. Note that the display control device 100 may be, for example, a head-mounted display (HMD), AR glasses, or the like.

[0058] For example, the display control device 100 uses an image sensor (e.g., a camera) to shoot a user in the real space, and reflects the user's body posture or facial expression in the avatar in the virtual space. Therefore, the user can experience the feeling as if the virtual self exists in the virtual space. In addition, since multiple avatars (in other words, multiple users) can exist in the virtual space at the same time, the user can communicate similarly to the real space via the avatar. For example, the user can greet another user or have a conversation (voice chat or text chat) with another user via the avatar to deepen the interaction with other users.

[0059] Since the virtual space can be expressed in various modes, the virtual space can be configured not only as a realistic space that conforms to the real space, but also as a space involving cartoon expressions that the user likes. In this case, it is expected that the provider of the virtual space can not only reproduce realistic actions that conform to reality in the avatar as the expression of the avatar, but also provide various emotional expressions and various actions that the user likes.

[0060] Therefore, the display control device 100 according to the present disclosure implements the above processing using the following configuration. For example, the display control device 100 obtains the behavior of the user in the real space as input information, and determines whether the input corresponding to the pre-registered specific action is included in the input information. Then, the display control device 100 reflects the behavior of the user corresponding to the input information in the avatar in the virtual space, and in the case of determining that the input information includes the input corresponding to the specific action, further reflects the specific action in the avatar in the virtual space.

[0061] Specifically, the display control device 100 pre-registers comical actions as specific actions (hereinafter referred to as "postures") performed by the avatar, and the comical actions include cartoon expressions such as thumbs up, posing each avatar's favorite posture, or high-fives. In addition, in the case where the action for stimulating the posture is received as input from the user, the display control device 100 reflects the posture in the avatar instead of the user's own action. Note that in the case where the display control device 100 does not receive the posture from the user, the user's body posture itself can be reflected in the avatar. As described above, the display control device 100 reflects postures associated with both the user's intentional actions and the user's unintentional actions in the avatar. Therefore, the display control device 100 can display an avatar with expressive and comical actions in the virtual space, rather than just imitating the user's actions. Through such expressions, the display control device 100 can construct a virtual space that promotes communication between users or promotes mutual love between users.

[0062] In the following, reference will be made to Figure 1 The above display control process is described. Figure 1As shown in the left part of , the display control device 100 displays the avatar 10A existing in the virtual space in the display screen example 20. At this time, the display control device 100 uses the camera to continuously capture the user corresponding to the avatar 10A (that is, the user whose actions or facial expressions are reflected in the avatar 10A).

[0063] Then, the display control device 100 acquires input information input from the user based on the image data obtained from the camera. Note that the input information refers to information acquired from various sensors including the camera, and is collectively referred to as information for displaying an avatar or the like.

[0064] As an example, the display control device 100 acquires the user's facial expression 21 using a technique of extracting facial expression based on an image as input information. The display control device 100 also acquires the user's skeletal structure 22 by using a technique of estimating the skeletal structure based on an image. The display control device 100 also acquires body posture information 23 indicating the movement of the user's arm or hand using a hand tracking technique. Note that the display control device 100 can acquire not only image data but also information of voice or text input from the user as input information. Then, the display control device 100 integrates this information and receives input information to be reflected in the avatar 10A from the user.

[0065] For example, based on the user's input information, the display control device 100 obtains the gesture of the avatar 10A nodding, shaking its head to express a willingness to refuse, or tilting its head to express curiosity as the user's unintentional action, which is not a specific action. Note that the display control device 100 can obtain various emotional expressions of the user, such as the user's smile or expression or surprised face or expression, as unintentional actions. In the present disclosure, these are collectively referred to as "unintentional actions". Figure 1 The example on the left shows an example in which an unintentional action of the user raising a hand is reflected in the avatar 10A and displayed in the virtual space.

[0066] On the other hand, the display control device 100 may also determine whether the user is performing a specific action for stimulating a gesture based on the acquired image data. That is, the display control device 100 determines whether an input corresponding to a pre-registered specific action is included in the input information. Figure 1 As shown in the right part of , the display control device 100 displays the avatar 10B existing in the virtual space in the screen display example 25. At this time, the display control device 100 uses the camera to continuously capture the user corresponding to the avatar 10B (that is, the user whose actions or facial expressions are reflected in the avatar 10B).

[0067] Note that Figure 1In the example of , for easy comparison and understanding, avatar 10A and avatar 10B are drawn separately; however, in practice, unintentional actions and intentional actions are expressed in the same avatar. In the following, when there is no need to distinguish avatar 10A, avatar 10B and other avatars, they are simply collectively referred to as "avatar 10".

[0068] When the user performs a specific action, the display control device 100 accepts a request to switch to an input mode for determining the specific action. For example, when the avatar 10B is expected to make a characteristic gesture based on the user's specific action, the user requests the display control device 100 to switch to an input mode for intentionally making a gesture.

[0069] The user then performs a characteristic action, such as "making a pre-registered gesture with the index finger and thumb in front of the camera." Figure 1 Similar to the left part of , the display control device 100 acquires the user's behavior, for example, the user's facial expression 26, the user's bone structure 27, and body posture information 28 based on the captured image. Then, the display control device 100 integrates the acquired information and determines whether the characteristics of the user's behavior match the characteristics pre-registered for stimulating gestures. Note that various known types of technology can be used for the acquisition of the user's behavior and the determination process of the gesture.

[0070] When determining that the behavior input from the user has a movement having characteristics sufficient to trigger a gesture, the display control device 100 reflects the gesture in the avatar 10B. Figure 1 As shown in the right part of , the avatar 10B makes a gesture with the index finger and thumb. At this time, the display control device 100 can perform a funny expression including a rendering effect showing radial lines around the avatar 10B together with the gesture. With such an expression, the user can distinguish whether the action of the avatar 10B is a pre-registered specific gesture or the action of the user himself. In other words, the user intentionally makes the avatar 10B perform this specific gesture. Therefore, in the present disclosure, these are collectively referred to as "intentional actions".

[0071] The display control device 100 can reflect both unintentional actions and intentional actions in the avatar 10 to express them as reactions of the avatar 10. Therefore, the display control device 100 can display the avatar 10 in a virtual space with various expressions including not only the user's original actions but also cartoon expressions.

[0072] At the same time, as described above, the user can switch between the input modes of unintentional action and intentional action to reflect the input information through unintentional action or intentional action in the avatar 10. For example, such switching can be performed based on the user's explicit designation (manual operation such as pressing a button or voice input). In addition, the display control device 100 can also pre-set an area in the space where the user performs intentional action, and switch the input mode based on this setting. This will be described by referring to Figure 2 Give a description. Figure 2 is a diagram illustrating an example of a determination process according to an embodiment.

[0073] Figure 2 An example is shown in which a user inputs his / her behavior by being photographed by the camera 141 of the display control device 100. At this time, the user performs a gesture 30 in a preset space 31 (for example, a space within a predetermined distance from the camera 141). The display control device 100 uses, for example, image analysis, a depth sensor, etc. to determine whether the user's behavior is performed in the space 31. Then, in the case where the user's behavior is performed in the space 31, the display control device 100 considers that the behavior will stimulate an intentional action, and determines the stimulation of the gesture. At the same time, the display control device 100 reflects the user's behavior observed in the space 32, which is an area outside the space 31, in the avatar 10 as an unintentional action.

[0074] That is, in the case where the position is at a distance greater than the predetermined distance from the camera 141, the display control device 100 does not determine the input from the user as an action for stimulating a gesture, and determines whether the action is for stimulating a gesture only with respect to the action performed by the user in the space 31. Therefore, the user can desirably reflect unintentional actions and intentional actions in the avatar 10 without explicitly specifying the switching of the input mode.

[0075] Note that the switching of input modes is not limited to Figure 2 Example shown. For example, in the case where the user is performing voice chatting (i.e., voice input), the display control device 100 can automatically switch to an input mode for extracting unintentional actions (skeleton detection). Alternatively, in the case where the user is performing text chatting (i.e., manual character input), the display control device 100 can automatically switch to an input mode for extracting intentional actions (gesture detection).

[0076] In addition, the display control device 100 can perform a mix of actions that reflect two types of movements in the middle of the execution mode switching. In addition, when the user is performing a user interface (UI) operation, the display control device 100 can turn off the input mode itself so that actions with different intentions are not reflected in the avatar 10. Alternatively, the display control device 100 can activate the gesture input mode only within a predetermined time period after performing the desired key operation. In addition, the display control device 100 can require the user to enter a predetermined word when performing a gesture to prevent incorrect gesture input. In addition, when the mute button for turning off the voice input from the user is pressed, the display control device 100 can activate the gesture input mode, and in other cases the mode can be set to the input mode of unintentional actions. In addition, when the input mode of unintentional actions is turned off, the display control device 100 can simultaneously turn off the voice input and switch to the privacy mode (not accepting input from the user). As described above, the display control device 100 can use different input modes according to various modes.

[0077] As mentioned above Figure 1 and Figure 2 As described above, according to the display control process of the embodiment, since both unintentional actions and intentional actions can be desirably reflected in the avatar 10, the avatar 10 can be displayed in various expressions.

[0078] (1-2. Configuration of Display Control Device According to Embodiment)

[0079] Next, the configuration of the display control device 100 will be described. Figure 3 1 is a diagram showing a configuration example of a display control device 100 according to an embodiment. Figure 3 One of the components of the display control system 1 of the embodiment shown. The display control system 1 includes a display control device 100 and an analysis server 50 .

[0080] The analysis server 50 performs various types of analysis processing described later, and provides the analysis results to the display control device 100. The analysis server 50 performs, for example, information processing with a relatively high processing load that is not performed by the display control device 100, such as text-based sentiment analysis and machine learning processing for image recognition. For example, the analysis server 50 is a cloud server or the like managed by a business operator or the like that operates a service related to a virtual space.

[0081] like Figure 3 As shown, the display control device 100 includes a communication unit 110, a storage unit 120, a control unit 130, a sensor unit 140, and a display unit 150. Note that the display control device 100 may include an input unit (keyboard, touch panel, etc.) that receives various operations from a user operating the display control device 100.

[0082] The communication unit 110 is implemented by, for example, a network interface card (NIC), a network interface controller, etc. The communication unit 110 is connected to the network N in a wired or wireless manner, and transmits and receives information to and from the analysis server 50 and other devices via the network N. The network N is implemented, for example, by a wireless communication standard or scheme such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), ultra-wideband (UWB), or low power wide area (LPWA).

[0083] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0084] The storage unit 120 stores various types of information related to the display control process of the embodiment. In the embodiment, the storage unit 120 includes a gesture storage unit 121.

[0085] Will refer to Figure 4 The posture storage unit 121 is described. Figure 4 is a table showing an example of the posture storage unit 121 of the embodiment. Figure 4 As shown, the gesture storage unit 121 has items such as “gesture ID”, “gesture”, and “determination item”.

[0086] "Posture ID" is identification information for identifying a posture. "Posture" indicates the content of a specific posture. "Determination item" indicates an item for which a feature amount is extracted, the feature amount being used to determine whether the user's behavior causes the posture to be triggered. Note that although Figure 4 Although not shown in the figure, the information of each gesture may include action data (animation of bone structure or facial expression, rendering effect, etc.) for moving the 3D model of the avatar 10 according to the gesture.

[0087] For example, Figure 4 The example shown indicates that the gesture with gesture ID "A01" corresponds to "thumbs up", and whether the gesture is caused to be triggered is determined by the user's "hand position" and "arm movement".

[0088] Back to Figure 3 , the description will be continued. The sensor unit 140 includes sensors that detect various types of environmental information. For example, the sensor unit 140 includes a camera 141 as an imaging device, which includes an outward-facing camera that captures the outside of the display control device 100, an inward-facing camera that captures the user side, and the like.

[0089] For example, the camera 141 identifies a subject (e.g., a user located in the real space) located in front of the display control device 100. In this case, the camera 141 may acquire an image of the user as the subject, and calculate the distance from the display control device 100 (in other words, the camera 141) to the subject based on the parallax between the images captured by the stereo camera or the analysis of the captured image data. Alternatively, the sensor unit 140 may detect the distance in the real space using a depth sensor capable of detecting the distance of any subject such as a user.

[0090] In addition, the sensor unit 140 includes a microphone 142 capable of measuring the user's voice and surrounding sounds. For example, when performing voice chatting, the user can perform voice input by using the microphone 142 included in the display control device 100.

[0091] Furthermore, for example, in the case where the display control device 100 is a wearable device such as an HMD, in addition to the camera 141 and the microphone 142, the sensor unit 140 may also have a function of detecting various types of information about the user's actions, such as the orientation, inclination, movement, and travel speed of the user's body. Specifically, the sensor unit 140 detects information about the user's head and posture, the movement of the user's head and body (acceleration and angular velocity), the field of view direction, the viewpoint movement speed, and the like as information about the user's actions. For example, the sensor unit 140 functions as various motion sensors such as a three-axis acceleration sensor, a gyro sensor, and a speed sensor, and detects information about the user's operation. More specifically, the sensor unit 140 detects components in the yaw direction, the pitch direction, and the roll direction as the movement of the user's head, thereby detecting a change in at least one of the position of the head or the user's posture.

[0092] Note that the sensor unit 140 is not necessarily included in the display control device 100, and may be, for example, an external sensor connected to the display control device 100 in a wired or wireless manner. For example, the camera 141 is not necessarily included in the display control device 100, and may be an imaging device installed outside the display control device 100.

[0093] The display unit 150 displays various types of information output from the control unit 130. For example, the display unit 150 is a display that outputs a video to a user. Note that the display unit 150 may include a sound output unit (eg, a speaker) that outputs sound.

[0094] The control unit 130 is implemented by, for example, a central processing unit (CPU), a microprocessing unit (MPU), a GPU, etc. (using a RAM, etc. as a work area to execute a program stored inside the display control device 100 (for example, a display control program according to the present disclosure)). The control unit 130 is also a controller and can be implemented by, for example, an integrated circuit such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0095] like Figure 3 As shown, the control unit 130 includes an acquisition unit 131 , a determination unit 132 , and a display control unit 133 .

[0096] The acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires input information to be reflected in the avatar 10 by sensing the user using the sensor unit 140. Specifically, the acquisition unit 131 acquires the behavior of the user in the real space as input information.

[0097] In addition, the acquisition unit 131 may acquire various operation and setting requirements from the user. For example, the acquisition unit 131 receives a designation from the user as to whether a specific action is reflected in the avatar 10. That is, the acquisition unit 131 receives an instruction from the user as to whether to set an input mode for inputting an unintentional action or an input mode for inputting an intentional action (gesture).

[0098] In addition, when acquiring input information, the acquisition unit 131 may use the sensor unit 140 or the like to acquire surrounding information or the like. Specifically, Figure 2 As shown, the acquisition unit 131 acquires the spatial position when the user's behavior is photographed from the camera 141. Therefore, the determination unit 132 described later can determine at what distance from the camera 141 the gesture performed by the user is photographed, and thus the mode can be automatically switched to the gesture mode.

[0099] Note that the acquisition unit 131 can acquire not only information about the user actually photographed by the camera 141, but also information about multiple users connected to the virtual space. That is, the acquisition unit 131 can acquire the behavior of multiple users as input information. By using such input information of multiple users, the determination unit 132 can determine the activity of the conversation of multiple users as described later, etc.

[0100] In addition, the acquisition unit 131 may receive various requests sent from the user via the UI. For example, the acquisition unit 131 receives various operations for services related to the virtual space, such as a request to change the appearance of an avatar or to purchase, select, or exchange an object associated with an avatar (e.g., an item in the virtual space).

[0101] The determination unit 132 performs various determinations for display control of the avatar 10. For example, the determination unit 132 determines whether an input corresponding to a pre-registered specific action is included in the input information from the user. The input corresponding to the pre-registered specific action refers to a registered behavior of a user activating a gesture.

[0102] For example, in the case where there is a designation of an input mode from the user (designation of a gesture mode), the determination unit 132 determines whether an input corresponding to a specific action is included in the input information. That is, the determination unit 132 determines whether the behavior performed by the user in front of the camera 141 is an intentional action for stimulating a gesture.

[0103] Furthermore, in the case where the mode is changed according to the distance, the determination unit 132 does not determine whether the input corresponding to the specific action is included in the input information when the position is more than a predetermined distance away from the camera 141, and determines whether the input corresponding to the specific action is included in the input information when the position is within a predetermined distance from the camera 141. That is, the determination unit 132 performs posture determination on an action such as a gesture performed by the user at a position relatively close to the camera 141, and does not perform posture determination on an action such as a gesture performed by the user in an area outside the set spatial area, and determines that the action is an unintentional action. Therefore, the determination unit 132 can freely switch the input mode to express the avatar 10.

[0104] The display control unit 133 controls the display unit 150 to display the information output from the control unit 130. That is, the display control unit 133 outputs the virtual space video rendered as the video content to the display. Note that the output destination device is not limited to the built-in display, but may be a separate information device such as a smartphone or a video output device such as a television.

[0105] For example, the display control unit 133 reflects the behavior of the user corresponding to the input information in the avatar 10 in the virtual space based on the input information acquired from the user, such as an image capturing the user, a voice uttered by the user, or a text input by the user. Furthermore, in the case where it is determined that the input information includes an input corresponding to a specific action, the display control unit 133 reflects the gesture as the specific action in the avatar 10 in the virtual space, and displays the avatar 10 in the virtual space.

[0106] For example, in the case of a mode for inputting an unintentional action, the display control unit 133 reflects the bone structure data of the user acquired based on the image capturing the user, etc., in the avatar 10. Meanwhile, in the case of a gesture mode, in the case where the determination unit 132 determines that the user's behavior causes a gesture to be triggered, the display control unit 133 reflects a gesture including a specific animation or presentation effect that has been pre-registered in the avatar 10. In other words, the gesture of the avatar 10 is expressed by animation, accompanied by a special action or presentation such as a thumbs-up, a high five, or applause. Note that, along with the gesture, the display control unit 133 can also control the voice output (pre-registered laughter), etc., of the avatar.

[0107] (1-3. Processing Procedure According to Embodiment)

[0108] Next, we will refer to Figure 5 The procedure of the processing according to the embodiment is described. Figure 5 This is a flowchart showing the flow of display control processing according to the embodiment.

[0109] like Figure 5 As shown, the display control apparatus 100 acquires an image capturing the user using the sensor unit 140 (step S11). Note that, as described above, the display control apparatus 100 may acquire the user's voice, already input text, etc. together with the image.

[0110] Then, the display control device 100 converts the image of the user into bone structure data based on a known posture estimation technique or the like (step S12 ).

[0111] Subsequently, the display control apparatus 100 determines whether the current mode is the gesture mode (step S13). If the current mode is the gesture mode (step S13; Yes), the display control apparatus 100 determines the position and movement of feature points in the skeleton structure data (step S14).

[0112] Then, the display control device 100 determines whether the feature point position and movement correspond to the registered gesture (step S15). If the feature point position and movement correspond to the registered gesture (step S15; yes), the display control device 100 reads the motion data of the gesture from the storage unit 120 (step S16). Then, the display control device 100 reflects the read motion data in the avatar 10 (step S18).

[0113] If the current mode is not in the gesture mode (step S13; No), the display control device 100 converts the bone structure data obtained from the user into the bone structure data of the avatar 10 without performing gesture determination (step S17). Then, the display control device 100 reflects the bone structure data in the avatar 10 (step S18).

[0114] In addition, also in step S15, when the feature point position and movement do not correspond to the posture (step S15; No), the display control device 100 converts the bone structure data obtained from the user into the bone structure data of the avatar 10 (step S17). Then, the display control device 100 reflects the bone structure data in the avatar 10 (step S18).

[0115] Thereafter, the display control device 100 displays the avatar 10 in the virtual space based on the information reflected in the avatar 10 (step S19 ). The display control device 100 repeats such display control processing for each predetermined frame acquired by the sensor unit 140 .

[0116] (1-4. Variation of Processing According to Embodiment)

[0117] (1-4-1. Expression of activity between avatars)

[0118] The expression of the avatar 10 with the unintentional motion and the intentional motion as described above can be combined with various other expressions. Hereinafter, a modification of the display control process of the embodiment will be described.

[0119] First, the first variation will be described. In the first variation, the display control device 100 displays various visual effects according to the activity of multiple avatars. Note that, in other words, the activity of the avatar means the activity of the user. Therefore, the display control device 100 can express that active communication is taking place between multiple users in the virtual space in a form that can be seen by other users.

[0120] That is, in the first variant, the acquisition unit 131 acquires the behavior of multiple users as input information. Then, the determination unit 132 determines the activity of the interaction between multiple users in the virtual space based on the behavior of the multiple users. In addition, the display control unit 133 displays the presentation effect according to the activity of the interaction near the multiple users.

[0121] Specifically, the determination unit 132 determines the activeness of the interaction based on the number of users participating in the interaction, the frequency of conversations between users, and the degree of overlap of voices of multiple users.

[0122] Furthermore, the display control unit 133 may determine the expression or occurrence frequency of the presentation effect based on the activeness of the interaction, the orientation of the body of the avatar 10 , and the content of the conversation of the plurality of users.

[0123] In the following, reference will be made to Figure 6 The first variation is described in detail. Figure 6 FIG. 1 is a diagram for explaining a first variation of the display control process of the embodiment.

[0124] Figure 6The example shown in FIG. 1 shows that avatars 10A, avatar 10B, and avatar 10C among a plurality of avatars actively interact with each other through greetings or conversations. At this time, the display control device 100 determines the activity of the interaction between the plurality of users, and displays an arc presentation effect 200 or a radial presentation effect 201 around the avatars based on the activity that has been determined. In this way, the display control device 100 can perform a display so that even if not participating in an actual conversation, a plurality of avatars that are actively interacting with each other can be distinguished only by observing the situation from a distance.

[0125] Note that the liveliness between avatars can be derived from various parameters. For example, in the case where the liveliness between avatars is through voice communication such as voice chat, parameters such as the volume, speed and pitch of the voice, emotion, frequency of overlap, amount of filler, or the presence or absence of specific language expressions are used. That is, the liveliness of communication in the virtual space is not necessarily measured only by the volume of the voice or the amount of dialogue.

[0126] For example, in a stereo environment such as a virtual three-dimensional space, simultaneous speech occurs as in a real space. Therefore, the display control device 100 can use, for example, "overlap (multiple people speaking simultaneously)" as a high importance (high weight) parameter for activity determination.

[0127] In addition, the display control device 100 may cause the analysis server 50 or the like to analyze the speech to analyze the emotion of each speech, and may calculate the activity as high in the case of including more emotional speech. Specifically, the display control device 100 may calculate the activity as high for an interaction including a call expressing an exclamation or strong emotion, such as "イェーイ (Japanese)" or "Wow! Yeah! (English)". Alternatively, the display control device 100 may calculate the activity as high based on the frequency of occurrence of greetings or words (positive words based on sentiment analysis) containing strong congratulations or happy intentions such as "Cheers!"

[0128] Furthermore, the display control apparatus 100 may change the presentation (visual) effect to be displayed according to the activity level.

[0129] For example, the display control device 100 may change the size or frequency of the presentation effect indicated by the arc according to the activity. Specifically, when the activity is relatively high, the display control device 100 may display a large arc in a manner that covers all multiple users participating in the conversation. Therefore, the display control device 100 can accurately convey the activity of multiple users to a user who is observing the multiple users from a distance.

[0130] In addition, even if the conversation is not necessarily active, the display control device 100 can also display the presentation effect to a large extent. For example, in a case where multiple users are paying attention to an event in a virtual space, the interaction of multiple users is determined to be active despite the small amount of conversation. In this case, for example, the display control device 100 can determine the activity based on the eye contact of the avatars or interactions other than conversations (the degree of closeness of the distance between users, etc.). The display control device 100 then displays an arc or radial presentation effect that is larger than usual or brighter than usual based on the activity. Therefore, the display control device 100 can indicate to other users that multiple users are actively interacting with each other without relying on conversations.

[0131] Next, we will refer to Figure 7 An example of a process in which the display control apparatus 100 determines the activity level is described. Figure 7 FIG. 2 is a diagram for explaining a first variation of the display control process of the embodiment.

[0132] Figure 7 2 is a diagram showing a composite waveform 205 obtained from a conversation between multiple users. Figure 7 In the figure, range 206 indicates that the voice volume of multiple users is measured to be a high waveform, which indicates that a conversation with relatively large voices has been carried out. In addition, range 207 indicates a waveform in which a large amount of voices of multiple users are combined, which indicates that an active conversation is carried out.

[0133] The display control device 100 determines the activity of multiple users based on this waveform analysis. For example, the display control device 100 determines the activity based on the characteristics of the volume change within a predetermined time period. Alternatively, the display control device 100 can estimate the number of participants in the conversation based on the waveform analysis, and when the estimated number of participants is large, determine that the activity is high.

[0134] In addition, the display control device 100 can change the expression of the presentation effect based on the waveform. For example, the display control device 100 can change the color or density (frequency of occurrence) of the arc or radial shape to be displayed, the speed at which the arc or radial shape expands, etc. according to the amplitude of the waveform.

[0135] In addition, the display control device 100 does not need to perform arc expression as a presentation effect, but can display on balloons with continuous sharp angles according to voice, conversation content or emotion. Therefore, the display control device 100 can express that, for example, multiple users are arguing with each other. In this case, the display control device 100 can display a presentation effect such as a balloon in a three-dimensional display (applying an edge shader), which can be formed regardless of the viewing angle.

[0136] In addition, the display control device 100 can change the speed at which the presentation effect occurs according to the distance from the observer to the multiple users. Specifically, the display control device 100 can display the presentation effect more slowly as the distance becomes longer, and display the presentation effect more quickly as the distance becomes shorter. Therefore, the observing user can grasp at a glance how far away the actively interacting users are.

[0137] (1-4-2. Emotional expression of the avatar)

[0138] Next, a second modification will be described. In the second modification, the display control apparatus 100 acquires an input in which an emotion is strongly indicated, such as a smile of a user, and reflects the gesture of the smile in the avatar 10 if it is determined that the user is smiling.

[0139] That is, in the second modification, the determination unit 132 determines whether an input corresponding to a specific action associated with a specific emotion such as laughter is included in the input information from the user. In the case where it is determined that the input information includes a specific action associated with a specific emotion, the display control unit 133 reflects the specific action associated with the specific emotion in the avatar 10.

[0140] At this time, the determination unit 132 may determine whether the input information includes an input corresponding to a specific action associated with a specific emotion by comparing voiceprint information associated with a specific emotion, which is pre-registered by the user, with the input information.

[0141] Furthermore, the display control unit 133 may display the avatar 10 reflecting a specific action associated with a specific emotion, and display a presentation effect associated with the specific emotion near the avatar 10 .

[0142] Will refer to Figure 8 and subsequent figures describe the second variation in detail. Figure 8 FIG. 1 is a diagram for explaining a second variation of the display control process of the embodiment.

[0143] Figure 8 The example shown in 2 indicates that the avatar 10 is making a gesture associated with "laughing" as an example of a specific emotion. At this time, the display control device 100 can display a presentation effect 210 of a funny expression associated with laughing around the avatar 10. That is, the display control device 100 can display the avatar 10 rich in emotional expression by causing the avatar 10 to perform a laughing gesture in response to a laughing user.

[0144] The display control device 100 can determine that the user is smiling by some methods. This will be referred to as Fig. 9 Give a description. Fig. 9 FIG. 2 is a diagram for explaining a second variation of the display control process of the embodiment.

[0145] Fig. 9 215 is a diagram showing an example of a speech waveform 215 acquired from a user. In the speech waveform 215, when the ambient sound (background noise) when the user is performing a voice chat or the like is measured, the measured value is set as a measurement result 216. In addition, in the speech waveform 215, the speech waveform when the user is performing a voice chat or the like is represented as a measurement result 217. At this time, the peak of the measurement result 217 is indicated by a line of the result 218. The difference between the result 218 and the measurement result 216 is indicated by the difference 219.

[0146] That is, when the user uses the microphone 142 for voice chatting, etc., the display control device 100 measures the ambient sound (background noise). Then, the display control device 100 measures the user's voice, estimates the peak of the volume as laughter, and measures the difference 219 from the ambient sound. In this way, the display control device 100 registers the difference between the peak of each user's voice and the ambient sound.

[0147] Then, when the user utters a voice and a voice exceeding the difference 219 (or equivalent voice) is observed next time, the display control apparatus 100 determines that the user smiles. According to this method, the display control apparatus 100 can determine whether the user smiles based on the volume.

[0148] Note that, as another method, the display control device 100 may learn voiceprint information when the user laughs, and perform determination based on such learning results. This will be described by referring to Fig.10 Give a description. Fig.10 FIG. 3 is a diagram for explaining a second variation of the display control process of the embodiment.

[0149] Fig.10 2 is a diagram showing an example of a speech waveform 220 acquired from a user. A range 221 of the speech waveform 220 indicates a waveform when the user's laughter is measured. The display control device 100 (or the analysis server 50) accumulates the waveform when laughter is measured for each user, and learns the features of the accumulated waveform by a known speech learning method (e.g., deep learning for extracting characteristic patterns). Therefore, the display control device 100 can generate a learned voiceprint pattern indicating a state of "laughing" for each user.

[0150] Then, when the user utters a voice next time and the voice waveform is acquired, the display control apparatus 100 can determine whether the voice includes laughter by comparing the voice waveform with the learned voiceprint pattern. Therefore, the display control apparatus 100 can determine whether the user laughs with high accuracy.

[0151] Note that human laughter is different, and the emotions contained in laughter include not only happiness but also various emotions. Therefore, in the machine learning model that determines laughter through learning, the display control device 100 can generate a model in a manner that classifies emotions, etc., rather than binary determining whether the user is laughing or not laughing.

[0152] In addition, several methods can be envisioned as methods of acquiring laughter as learning data. For example, the display control device 100 acquires a waveform indicating laughter based on the user's explicit designation. Specifically, when the user presses a button for causing the display control device 100 to acquire (record) laughter, the display control device 100 acquires the user's voice. At this time, the user laughs in the desired manner. Therefore, the display control device 100 can acquire a voiceprint pattern indicating the user's laughter (i.e., teacher data with a label of "laugh" attached). Alternatively, the display control device 100 can acquire the user's voiceprint pattern in a manner that accumulates only the user's laughter, using a general machine learning model for speech recognition that determines the user's laughter. Generally, since the characteristics of laughter vary from individual to individual, it is desirable for the display control device 100 to acquire and learn the voiceprint pattern of each individual user.

[0153] Note that the display control apparatus 100 can acquire voices based on various user emotions, not limited to laughter. By reflecting the expression based on the acquired voice in the avatar 10 , the display control apparatus 100 can also express the user's own emotional expression in the avatar 10 .

[0154] As an example, the display control device 100 may acquire the characteristics of each user's filler and reflect the characteristics of the user who uttered the filler in the avatar 10. It can be said that the filler such as "Hmm..." uttered by the user or the habit of saying "Sounds good" indicates the characteristics of the user. Therefore, the display control device 100 can reproduce the characteristics of the user in the virtual space by learning the filler and speaking habits of each user and reflecting the filler or speaking habits in the avatar 10 when the filler or speaking habits are observed. In this case, the display control device 100 can also reflect the characteristics of the user in the avatar 10 by adding emphasized and cartoon expressions, such as expressing the voice uttered as the filler or speaking habits in a balloon in 3D.

[0155] (1-4-3. Visualization of dialogue)

[0156] Next, a third variation will be described. In the third variation, a method will be described in which, when the display control device 100 displays a conversation between avatars in a virtual space, not only text indicating the contents of the conversation is displayed in a general balloon, but also the contents of the conversation are expressed in various expressions.

[0157] For example, in the third modification, the display control unit 133 may display the words spoken by the avatar 10 based on the input information, and further display the history of the words in a manner shifting upward from the avatar.

[0158] Furthermore, the display control unit 133 may display the words spoken by the avatar 10 as icons representing the semantic contents with pictures based on the semantic contents included in the input information.

[0159] In addition, the determination unit 132 may determine the distance between the position of the display of the word observed in the virtual space (i.e., the position where the avatar as the observer is located) and the position of the avatar 10 that spoke the word. In this case, the display control unit 133 displays the history of the icon or word as a presentation effect that does not include the semantic content of the word or icon based on the determined distance. Note that as an example of a presentation effect that does not include any semantic content, as will be described later, there is a case where text indicating the content of the conversation is not displayed, and only the history of the active conversation is expressed in the form of a signal fire or a balloon.

[0160] In addition, the determination unit 132 may determine the positional relationship between the position in the virtual space where the display of the word is observed and the position of the avatar 10 that uttered the word. In this case, the display control unit 133 may determine the display angle and three-dimensional display of the word observed in the virtual space based on the determined positional relationship.

[0161] As described above, in the third modification, the display control apparatus 100 displays icons or displays a conversation including a visual effect, instead of displaying a general conversation displayed as text, etc. Therefore, the display control apparatus 100 can express at a glance that the avatars are actively talking to each other.

[0162] Will refer to Fig.11 and subsequent figures describe the third variation in detail. Fig.11 FIG. 1 is a diagram for explaining a third variation of the display control process of the embodiment.

[0163] Fig.11 The example shown in 230 indicates that the avatar 10A is having a conversation with the avatar that is the observer by saying "There is a movie I want to see!" Since the avatar 10A is talking near the observer, this conversation is displayed as text in the balloon 230.

[0164] Meanwhile, the content of the conversation conducted by the avatar 10B on the far side of the avatar 10A is indicated by the icon 231. Specifically, the display control device 100 performs content and sentiment analysis on the conversation of the avatar 10B through the analysis server 50, and displays the analysis result as the icon 231. Fig.11In the example of FIG. 1 , icon 231 indicates that avatar 10B has had a conversation having content such as going to a movie, then riding in a car, and dining out.

[0165] As described above, for a conversation with an avatar 10B that is separated from the observer by more than a predetermined distance, the display control device 100 does not display the text as it is, but displays an icon (image, etc.) indicating the content of the conversation. Therefore, the observer can grasp the rough content of the conversation and situations such as how actively the conversation is going on without reading the character information of the distant conversation in detail.

[0166] Furthermore, the display control device 100 can display the conversation between avatars in an animation in which the past history flows upward, like icon 231. Therefore, the display control device 100 can send the flow of conversation to other users in a visually easy-to-understand manner, and can also allow a user at a distance to grasp how actively the conversation is going.

[0167] This will be seen by referring to Fig.12 Give a description. Fig.12 FIG. 2 is a diagram for explaining a third variation of the display control process of the embodiment.

[0168] Fig.12 A situation where multiple avatars 235 observed at a very far distance from the observer are talking is shown. At this time, in the conversation of the multiple avatars 235, the display including semantic contents such as text or icons is omitted, and only the history 236 of the conversation in a balloon or a circular shape is displayed (this display may be referred to as a "text balloon").

[0169] As described above, the history 236 of the conversation is animated upward in the order of the past history. Fig.12 In the example of FIG. 2 , the conversations of the multiple avatars 235 are expressed as signal fires in the virtual space. Therefore, even a distant observer can know at which locations in the virtual space the conversations are actively taking place.

[0170] In addition, the display control device 100 can change the expression method differently according to the distance between the observer and the avatar 10. This will be described by referring to Fig.13 Give a description. Fig.13 FIG. 3 is a diagram for explaining a third variation of the display control process of the embodiment.

[0171] Fig.13 2 is a diagram showing different expressions of a conversation 240 near the observer, a conversation 241 conducted at a medium distance from the observer, and a conversation 242 conducted at a very large distance from the observer.

[0172] For example, in the conversation 240, the conversation content of the avatar is displayed as text, and the details of the conversation are displayed. Note that in the conversation 240, not only text but also image information shared between users can be displayed, such as images sent by users during text chat. Therefore, the observer can roughly grasp the type of topic of the ongoing communication without reading the text.

[0173] For example, in the conversation 241, the text display is omitted for the conversation content of the avatar, and an icon or the like is inserted in the middle, thereby displaying so that the observer can grasp the rough content of the conversation. For example, in the conversation 241, in the case where two avatars are talking, the observer can grasp the content of the conversation at the granularity of the degree of the avatars talking. By the way, the old history of the conversation 241 can be displayed in a manner that text strings merge with each other when moving upward.

[0174] For example, in conversation 242, the display of semantic content such as text or icons is omitted for the conversation content of the avatar, and only the history of the conversation is displayed like a signal fire. At this time, the display control device 100 can perform a display so that the past history (i.e., the upper conversation history) becomes more transparent, just like in a signal fire in a real space. In addition, the display control device 100 can display a display indicating each line that is smaller than those lines of conversation 240 and conversation 241. That is, the display control device 100 can change the display bar, such as a balloon for displaying a line, according to the distance.

[0175] As described above, by changing the expression of the conversation according to the distance, the display control apparatus 100 can provide the user with a UI that does not complicate the display even when active communication is performed in a virtual space.

[0176] In addition, the display control device 100 can change the expression method differently according to the positional relationship between the observer and the avatar 10. This will be explained by referring to Fig.14 Give a description. Fig.14 FIG. 4 is a diagram for explaining a third variation of the display control process of the embodiment.

[0177] Fig.14 The left figure in shows a situation where a conversation 250 is being conducted near the observer. Specifically, in the conversation 250, the avatar 10A and the avatar 10B are conducting a conversation. In addition, the conversation 250 includes a conversation history 251 of the avatar 10A and a conversation history 252 of the avatar 10B.

[0178] Fig.14The right figure in FIG. 2 shows a bird's-eye view display 255 of the conversation 250 observed from above. As shown in the bird's-eye view display 255, it is based on the premise that the avatar 10A and the avatar 10B do not face the observer 256, and stand at a slightly inclined angle when viewed from the observer 256.

[0179] At this time, if Fig.14 As shown in the left figure in , the display control device 100 displays the conversation history 251 at a certain angle based on the positional relationship between the observer 256 and the avatar 10A (in this example, the angle formed by the line of sight of the observer 256 and the orientation of the avatar 10A). Similarly, the display control device 100 displays the conversation history 252 at a certain angle based on the positional relationship between the observer 256 and the avatar 10B.

[0180] With such an expression, the observer 256 can recognize the positional relationship with the avatar 10A or the avatar 10B simply by viewing the conversation history.

[0181] In addition, the display control device 100 can perform a display in which, in addition to the angle, the presentation effect is also included in the conversation content itself such as text. This will be described by referring to Fig.15 Give a description. Fig.15 FIG. 5 is a diagram for explaining a third variation of the display control process of the embodiment.

[0182] Fig.15 A situation is shown in which the avatar 10A and the avatar 10B are having a conversation, and a text 260 , a presentation effect 261 , and a text 262 indicating the contents of the conversation are displayed.

[0183] For example, the text 260 may include highlighting, such as being displayed in bold compared to other texts. For example, when the user says a word that strongly expresses emotion and is pre-registered (eg, "crazy"), the display control apparatus 100 automatically performs highlighting.

[0184] Incidentally, the presentation effect 261 is a character icon including a three-dimensional display. For example, when the user says a word (such as "crazy") that strongly expresses emotion and is pre-registered, the presentation effect 261 is automatically displayed.

[0185] In addition, the text 262 includes a three-dimensional display. For example, when the user says a word (such as "surprise") that is pre-registered and strongly expresses emotion, or when the user's voice volume is greater than the normal volume, the text 262 is displayed based on a predetermined determination standard.

[0186] Note that these special text display and presentation effects can be displayed based on sentiment analysis in the user's conversation, etc. For example, the display control device 100 collaborates with the analysis server 50 to perform a positive / negative determination on the content input in the text chat using sentiment analysis. The display control device 100 then automatically generates appropriate visual effects for the balloons in which the text is displayed. Specifically, in the case where the conversation is positive, the display control device 100 may display balloons or text with bright colors, animations of characters flicking, and the like. Alternatively, in the case where the conversation is negative, the display control device 100 may display balloons with cold colors, or may display balloons or text with animations that make it look like the text is crying, and the like.

[0187] Additionally, these balloons and text can be presented through a historical display (known as a word cloud or the like) where the size of the text varies depending on how often the words appear in the conversation.

[0188] Additionally, these types of text displays, etc. can be accompanied by three-dimensional physical properties. This will be discussed in detail in the following sections. Fig.16 Give a description. Fig.16 FIG. 6 is a diagram for explaining a third variation of the display control process of the embodiment.

[0189] Fig.16 2 is a diagram showing an example of a text 270 and a text 275 having a three-dimensional display. The text 270 has a shape as if the characters are floating from a balloon. Therefore, in a display 271 in which the text 270 is viewed from the side, the character portion is expressed as floating from the balloon.

[0190] In addition, the text 275 has a shape as if the characters are captured in a balloon. Therefore, in the display 276 when the text 275 is viewed from the side, the character part is expressed in a manner included in the cloud-like balloon. Therefore, in the viewer's view, the characters are expressed as a two-dimensional plate, and the balloon is expressed in a shape like a three-dimensional cloud or a rubber balloon.

[0191] Notice, Fig.16 The three-dimensional display shown in the other figures can also be displayed from the back. In this case, the image or text in the three-dimensional display is horizontally inverted and displayed.

[0192] As described above, the display control apparatus 100 can also perform various expressions on text in a conversation or the like, thereby visually entertaining the user.

[0193] (1-4-4. Proximity effect between avatars)

[0194] Next, a fourth modification will be described. In the fourth modification, an example is shown in which the display control device 100 performs such expression that encourages interaction when avatars are close to each other.

[0195] For example, in the fourth modification, the determination unit 132 may determine whether the first avatar is included in the area recognized by the second avatar. Then, in the case where it is determined that the first avatar is included in the area recognized by the second avatar, the display control unit 133 may adjust the travel path of the first avatar so that the first avatar approaches the second avatar.

[0196] In addition, the determination unit 132 may determine whether the first avatar has approached the second avatar in the virtual space. In the case of determining that the first avatar has approached the second avatar, the display control unit 133 reflects a specific action corresponding to the approach in at least one of the first avatar or the second avatar.

[0197] That is, in the fourth modification, the display control device 100 causes a special display or action to operate so that avatars can recognize each other's presence and communicate in the vicinity of each other, thereby promoting interaction between users.

[0198] Will refer to Fig.17 and subsequent figures describe the fourth variation in detail. Fig.17 FIG. 1 is a diagram for explaining a fourth modification of the display control process of the embodiment.

[0199] Fig.17 The first diagram shown in FIG. 1 shows a case where the avatar 10D approaches the place where the avatar 10A, the avatar 10B, and the avatar 10C are talking. At this time, the avatar 10A recognizes or recognizes through the initially set viewpoint recognition that the avatar 10D has entered the area determined to be close.

[0200] like Fig.17 As shown in the second figure in FIG. 1 , when the avatar 10A recognizes the approach to the avatar 10D, the avatar 10A causes a specific action to be triggered. For example, the avatar 10A performs a gesture of raising a hand to greet. At this time, the display control device 100 can automatically display a friendly message, etc., to encourage the avatar 10A to participate in the conversation with the avatar 10D.

[0201] Then, when avatar 10D responds to the greeting and indicates an interest in participating in the conversation between avatar 10A and the other avatars, avatar 10D participates in the conversation, such as Fig.17 As shown in the third figure.

[0202] In these actions, the display control device 100 performs correction so that the avatar 10A and the avatar 10D make eye contact, or so that the avatar 10A and the avatar 10D perform a greeting gesture, etc. Therefore, the display control device 100 can make the avatars actively interact with each other.

[0203] For this automatic conversation participation, the display control device 100 performs control processing based on some conditions. For example, the display control device 100 determines that one or more avatars are already in a conversation (chatting) state. In addition, the display control device 100 determines that another avatar has approached the area where the conversation is taking place. When these conditions are met, the display control device 100 corrects the display so that the avatars can easily make eye contact or perform gestures.

[0204] This will be seen by referring to Fig.18 Give a description. Fig.18 FIG. 2 is a diagram for explaining a fourth modification of the display control process of the embodiment.

[0205] like Fig.18 As shown, the display control device 100 recognizes a proximity area 280 of the avatar 10A and a proximity area 281 of the avatar 10D. In addition, the display control device 100 recognizes a distance 282 between the avatar 10A and the avatar 10D. Then, when the distance 282 falls below or equal to a predetermined distance, the display control device 100 performs correction so that, for example, the avatar 10A and the avatar 10D face each other, thereby performing correction so that eye contact is made.

[0206] Then, the display control device 100 recognizes that the proximity region 280 of the avatar 10A and the proximity region 281 of the avatar 10D have come into contact with each other. In this case, the display control device 100 stimulates the avatar 10A and the avatar 10D to perform a gesture of greeting each other.

[0207] Furthermore, when the avatar 10A and the avatar 10D enter each other's proximity area, the display control device 100 starts a conversation between the two, such as a voice chat. At this time, the display control device 100 newly recognizes a conversation area 283 in which the conversation between the avatar 10A and the avatar 10D is conducted.

[0208] Note that, in a case where the user expresses an intention to refuse a conversation while making eye contact or making a greeting gesture, the display control device 100 may perform correction in such a manner as not to bring the avatars close to each other.

[0209] In addition, when avatars are physically close to each other, the display control device 100 can perform control to prioritize voice chat. Therefore, the display control device 100 can create an environment in which a newly participating avatar can easily talk with an unspecified number of avatars that are already participating in the conversation. Note that in the case where the user has muted voice input, the display control device 100 can prioritize text chat.

[0210] In addition, regarding the control of bringing avatars closer to each other to encourage interaction, the display control device 100 may pre-set areas where conversations are encouraged and areas where conversations are prohibited. This will be described by referring to Fig.19Give a description. Fig.19 FIG. 3 is a diagram for explaining a fourth modification of the display control process of the embodiment.

[0211] Fig.19 290 . This is a diagram showing a situation where a certain event is performed in a virtual space. For example, this is based on the premise that in the event, a specific avatar appears on a stage 290. In this case, the display control device 100 sets a dialogue prohibition area 292 and a dialogue prohibition area 293 as a passage, etc., so that the avatar does not stay in the passage leading to the stage 290 or at a position away from the stage 290. At the same time, the display control device 100 sets a dialogue area 294 around the stage 290, etc., to actively encourage dialogue around the stage 290 or around a monument 291.

[0212] In this case, the display control device 100 does not perform correction to cause the avatar to start a conversation in the conversation prohibited area 292 or the conversation prohibited area 293. On the other hand, the display control device 100 performs correction to cause the avatar to start a conversation in the conversation area 294. Therefore, the display control device 100 can actively guide the conversation between the avatars in the area where the conversation is recommended in the virtual space.

[0213] Note that the display control device 100 can correct the travel path of the avatar 10D so that the avatar 10D can easily participate in the conversation. For example, the display control device 100 can correct the walking path or walking speed so that the path that the avatar 10D intends to walk automatically approaches the approach area of ​​the avatar 10A or the area where the avatar 10A and other avatars are having a conversation. In addition, the display control device 100 can correct the orientation of the body so that the avatar 10D can easily make eye contact with the avatar 10A while walking.

[0214] (1-4-5. Expression of avatar display and interaction)

[0215] Next, a fifth modification will be described. In the fifth modification, an example is shown in which the display control apparatus 100 displays a UI for changing the appearance of the avatar 10 or exchanging items for changing the appearance between avatars.

[0216] For example, in the fifth modification, the acquisition unit 131 acquires from the user a request to change the appearance of the avatar 10. Then, the display control unit 133 reflects the change based on the request in the avatar 10, and changes the appearance and presentation effect of the avatar 10 in the virtual space.

[0217] In addition, the acquisition unit 131 may acquire from the user a request to exchange any of the plurality of objects associated with the first avatar with the second avatar. In this case, when the request is acquired, the display control unit 133 displays a list of the plurality of objects associated with the first avatar near the first avatar.

[0218] In addition, the acquisition unit 131 may acquire an instruction from the user to select an object to be exchanged with the second avatar from among the multiple objects displayed in the list. In this case, when the instruction is acquired, the display control unit 133 displays the object in a floating manner from the list display, and moves the floating object in the virtual space to overlap with the second avatar according to the operation received from the user.

[0219] As described above, in the fifth modification, the display control apparatus 100 performs display control to change the appearance of the avatar 10 or actively encourage the exchange of items between the avatars 10. Therefore, the display control apparatus 100 can visually entertain the user and actively promote interaction between users.

[0220] Will refer to Fig. 20 and subsequent figures describe the fifth variation in detail. Fig. 20 FIG. 1 is a diagram for explaining a fifth modification of the display control process of the embodiment.

[0221] Fig. 20 The first figure shows an example in which a change option 300 is displayed as a UI so that the user changes the appearance of the avatar 10A. In this example, it is based on the premise that the user has selected option 301.

[0222] Fig. 20 The second figure shows a case where the display control apparatus 100 is changing the appearance of the avatar 10A. When changing the appearance of the avatar 10A, the display control apparatus 100 may display a presentation effect 302 in which the avatar 10A is covered with smoke.

[0223] Fig. 20 The third figure shows a case where the display control device 100 has changed the appearance of the avatar 10A. The display control device 100 displays the changed appearance of the avatar 10A in the virtual space. In this way, the user can change the appearance of the avatar 10A as desired.

[0224] In the virtual space, the user can exchange items for changing the appearance of the avatar 10 with other users. This will be explained by referring to Fig.21 Give a description. Fig.21 FIG. 2 is a diagram for explaining a fifth modification of the display control process of the embodiment.

[0225] exist Fig.21, a UI display for exchanging items between the avatar 10A and the observer is shown. The avatar 10A presents the observer with an exchange option item 310. The observer is also presented with an exchange option item 311. Then, when the observer selects the exchange button 312, the items are exchanged with each other. Note that in the virtual space, there may be a concept indicating which user (avatar) has ownership of each item. The display control device 100 may perform internal processing so that ownership is also exchanged together with the exchange of items.

[0226] This exchange of items between avatars can be displayed in a different UI. This will be shown by referring to Fig. 22 Give a description. Fig. 22 FIG. 3 is a diagram for explaining a fifth variation of the display control process of the embodiment.

[0227] Fig. 22 3 is a diagram showing that the avatar 10A and the avatar 10B attempt to exchange the item 320. In this case, the user corresponding to the avatar 10A selects the item 320 on the screen display using a pointing device such as a mouse.

[0228] Then, the display control apparatus 100 enlarges the display around the item 320. In the enlarged view, the user can move the item 320 closer to the avatar 10B, for example, by moving the arm in the real space. In this case, the display control apparatus 100 displays the skeleton display 321 of the avatar 10A corresponding to the user's arm, and moves the item 320 displayed near the arm of the avatar 10A toward the avatar 10B.

[0229] When the item 320 is moved to the skeleton display 322 closer to the avatar 10B than a predetermined distance, the display control apparatus 100 determines that the item 320 has been transferred. Then, the display control apparatus 100 displays the item 320 near the skeleton display 322, and transfers the ownership of the item 320 to the avatar 10B.

[0230] As described above, the display control apparatus 100 can establish item exchange in expression as if the avatar is actually transferring the item, instead of displaying a UI indicating the item exchange.

[0231] Furthermore, as described above, in the case where the item is used to change the appearance of the avatar 10, etc., the display control device 100 may perform a UI display corresponding to the portion to be changed. This will be described by referring to Fig.23 Give a description. Fig.23 FIG. 4 is a diagram for explaining a fifth variation of the display control process of the embodiment.

[0232] like Fig.23As shown, in the avatar 10A, the head 330, the face 332, the body 334, and the legs 338 are set as parts for changing the appearance. In this case, the user can display a list of items that can be changed in each part. For example, the display control device 100 displays an item option 331 corresponding to the head 330. Similarly, the display control device 100 displays an item option 333 corresponding to the face 332, an item option 335 corresponding to the body 334, and an item option 337 corresponding to the legs 338. In addition, the display control device 100 can also display an item option 339 for changing the appearance of the entire avatar 10A.

[0233] That is, the display control apparatus 100 displays target item options and the like in a popup manner for each portion requesting change in the appearance of the avatar 10A. Then, the display control apparatus 100 selects an item for changing the appearance of the avatar 10A according to the user's operation.

[0234] Note that the display control device 100 may, for example, overlay items existing in the hierarchical structure under a single selected item in the UI display of the item. For example, in the case where glasses are selected as an item for changing the face 332 of the avatar 10A, the display control device 100 may also display a plurality of item options included in the glasses category. In this case, after selecting one of the item options 333, the user may select an item in a lower hierarchical structure, for example, in a one-stroke writing manner.

[0235] In addition, when selecting an item, the display control device 100 can set an appropriate shortcut according to the type of device used by the user. For example, in item selection, the display control device 100 can receive input through a typing operation of a keyboard, etc., as well as input through a pointing device such as a mouse or a user's body gesture.

[0236] In addition, the display control device 100 can perform item transfer control using UI display. This will be described by referring to Fig.24 Give a description. Fig.24 FIG. 5 is a diagram for explaining a fifth variation of the display control process of the embodiment.

[0237] Fig.24 The example shown in shows a case where the user selects an item 341 from the item options 340 owned by the avatar 10A. For example, the user selects the item 341 by long pressing on the screen with a finger or a pointing device. In this case, the display control device 100 switches the display of the item 341 to such a display (floating display) as if it floats from the item options 340.

[0238] The user drags the item 341 in the floating display to the avatar 10B. Through such an operation, the display control device 100 determines that the item 341 is delivered to the avatar 10B. In this way, the display control device 100 can use various UIs or various types of screen displays to perform processing such as item exchange between users. Note that in an embodiment, the content exchanged by avatars in a virtual space is described as an item; however, the mode type of the item can be set as desired by the service side that provides the virtual space. For example, an item can be data such as a non-fungible token (NFT).

[0239] (1-5. Modification)

[0240] The processing according to the above embodiment may be accompanied by various modifications. For example, Figure 3 Each of the devices in illustrative examples conceptually shows a function in the display control system 1 and may have various modes according to the embodiment.

[0241] For example, in the above embodiment, an example has been described in which the display control device 100 performs processing in cooperation with the analysis server 50. However, the display control device 100 may perform the processing performed by the analysis server 50 on the display control device 100 itself.

[0242] In addition, the display control device 100 may include a plurality of devices. For example, the display control device 100 may be divided into a first information processing device, the first information processing device acquires the user's behavior as an image and performs processing related to the acquired image, and a second information processing device performs processing related to the input of the user's voice, text, etc. In this case, the first information processing device acquires the user's behavior as an image, performs posture estimation based on the image, and performs hand-raising determination, head orientation determination, gesture determination, etc. In addition, the second information processing device may perform laughter determination based on voice, or may perform sentiment analysis based on text, etc.

[0243] In addition, the display control device 100 may be a device in which a display unit and an information processing unit are separately configured. In this case, the information processing unit of the display control device 100 may be a desired information processing device such as a server or a PC.

[0244] (2. Other Implementation Methods)

[0245] The processing according to the above embodiment may be performed in various different embodiments other than the above embodiment.

[0246] In the processes described in the above embodiments, all or part of the processes described as being automatically performed may be performed manually, or all or part of the processes described as being manually performed may be automatically performed by known methods. In addition, unless otherwise specified, the processes, specific names, and information including various types of data or parameters shown above or in the accompanying drawings may be modified as needed. For example, the various types of information shown in the accompanying drawings are not limited to the information shown.

[0247] In addition, each component of each device shown in the drawings is conceptual in function and is not necessarily physically configured as shown in the drawings. That is, the specific form of distribution or integration of the devices is not limited to that shown in the drawings, and all or part of them may be functionally or physically distributed or integrated in any unit according to various loads, usage status, etc.

[0248] In addition, the above embodiments and modifications may be appropriately combined within the range where processing contents do not conflict.

[0249] Furthermore, the effects described herein are merely examples and are not restrictive, and other effects may be achieved.

[0250] (3. Effects of the display control device according to the present disclosure)

[0251] As described above, the display control device according to the present disclosure (display control device 100 in the embodiment) includes an acquisition unit (acquisition unit 131 in the embodiment), a determination unit (determination unit 132 in the embodiment), and a display control unit (display control unit 133 in the embodiment). The acquisition unit acquires the behavior of the user in the real space as input information. The determination unit determines whether the input corresponding to a pre-registered specific action is included in the input information. The display control unit reflects the behavior of the user corresponding to the input information in the avatar in the virtual space, and further reflects the specific action in the avatar in the virtual space when it is determined that the input information includes an input corresponding to a specific action, so as to display the avatar in the virtual space.

[0252] As described above, the display control device according to the present disclosure displays an avatar by combining an unintentional motion accompanying a user's natural motion with an intentional motion for activating a gesture. Therefore, the display control device can display an avatar in a virtual space with various expressions including not only the user's original motion but also cartoon expressions.

[0253] Furthermore, the acquisition unit receives a designation from the user as to whether to reflect a specific action in the avatar. In the case of the user designation, the determination unit determines whether the input information includes an input corresponding to the specific action.

[0254] In addition, the acquisition unit acquires the position of the user when the behavior is photographed from the imaging device. The determination unit does not determine whether the input corresponding to the specific action is included in the input information when the position is more than a predetermined distance away from the imaging device, and determines whether the input corresponding to the specific action is included in the input information when the position is within a predetermined distance from the imaging device.

[0255] As described above, when the user specifies to perform an intentional action, or when the user performs a gesture in a specific area, the display control device according to the present disclosure reflects the intentional action in the avatar. Therefore, the display control device can perform avatar expression according to the user's intention.

[0256] In addition, the acquisition unit acquires the behaviors of the plurality of users as input information. The determination unit determines the activity of the interaction between the plurality of users in the virtual space based on the behaviors of the plurality of users. The display control unit displays the presentation effect according to the activity of the interaction near the plurality of users.

[0257] Furthermore, the determination unit determines the activeness of the interaction based on the number of users participating in the interaction, the frequency of conversations between the users, and the degree of overlap of the voices of the plurality of users.

[0258] Furthermore, the display control unit determines the expression or appearance frequency of the presentation effect based on the activeness of the interaction, the orientation of the body of the avatar, and the content of the conversation of the plurality of users.

[0259] As described above, by visually displaying the activeness of a conversation between avatars, the display control device according to the present disclosure can effectively show a situation where avatars actively interact with each other even for a user who is not participating in the conversation.

[0260] In addition, the determination unit determines whether the input information includes an input corresponding to a specific action associated with a specific emotion. When it is determined that the input information includes a specific action associated with a specific emotion, the display control unit reflects the specific action associated with the specific emotion in the avatar to display the avatar in the virtual space.

[0261] The determination unit also determines whether the input information includes an input corresponding to a specific action associated with the specific emotion by comparing voiceprint information associated with the specific emotion, which is pre-registered by the user, with the input information.

[0262] Furthermore, the display control unit displays the avatar reflecting a specific action associated with the specific emotion, and displays a presentation effect associated with the specific emotion near the avatar.

[0263] As described above, the display control device according to the present disclosure can reflect gestures associated with specific emotions such as laughter in the avatar. Therefore, the display control device can also express various emotional expressions in the virtual space, just like in the real space.

[0264] The display control unit displays the words spoken by the avatar based on the input information, and further displays the history of the words in a manner shifting upward from the avatar.

[0265] Furthermore, the display control unit displays the word spoken by the avatar as an icon representing the semantic content with a picture based on the semantic content included in the input information.

[0266] Furthermore, the determination unit determines a distance between a position observing the display of the word and a position of the avatar who uttered the word in the virtual space. The display control unit displays a history of the icon or word as a presentation effect excluding the semantic content of the word or icon based on the determined distance.

[0267] Furthermore, the determination unit determines a positional relationship between a position in the virtual space where the display of the word is observed and a position of the avatar who uttered the word. The display control unit determines a display angle and a three-dimensional display of the word observed in the virtual space based on the determined positional relationship.

[0268] As described above, the display control device according to the present disclosure can display in such a manner that the user can grasp at a glance where active communication is taking place by expressing the history of conversation in various modes without complicating the screen display.

[0269] Furthermore, the determination unit determines whether the first avatar is included in an area recognized by the second avatar in the virtual space. If it is determined that the first avatar is included in the area recognized by the second avatar, the display control unit adjusts a travel path of the first avatar so that the first avatar approaches the second avatar.

[0270] In addition, the determination unit determines whether the first avatar has approached the second avatar in the virtual space. When it is determined that the first avatar has approached the second avatar, the display control unit reflects a specific action corresponding to the approach in at least one of the first avatar or the second avatar to display the first avatar and the second avatar.

[0271] As described above, the display control device according to the present disclosure can promote avatars to actively interact with each other by correcting the behavior between avatars. Therefore, the display control device can increase the chance of dialogue between users in the virtual space, thereby activating the interaction in the virtual space.

[0272] Furthermore, the acquisition unit acquires a request for changing the appearance of the avatar from the user. The display control unit reflects the change based on the request in the avatar, and changes the appearance and presentation effect of the avatar in the virtual space.

[0273] The acquisition unit acquires a request from the user for an arbitrary object among a plurality of objects associated with the first avatar to be exchanged with the second avatar. When the request is acquired, the display control unit displays a list of the plurality of objects associated with the first avatar near the first avatar.

[0274] In addition, the acquisition unit acquires an instruction from the user to select an object to be exchanged with the second avatar from among the multiple objects displayed in the list. When the instruction is acquired, the display control unit displays the object in a floating manner from the list display, and moves the floating object in the virtual space to overlap with the second avatar according to the operation received from the user.

[0275] As described above, the display control device according to the present disclosure reduces the stress of the user in the interaction such as exchanging items between users by providing a UI with good operability. Therefore, the display control device can support active interaction between users.

[0276] (4. Hardware Configuration)

[0277] An information device such as the display control device 100 according to the above-described embodiment is composed of, for example, a display control device having Fig.25 The configuration shown is implemented by a computer 1000. Hereinafter, the display control device 100 will be described as an example. Fig.25 1 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the display control device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input and output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0278] The CPU 1100 operates according to a program stored in the ROM 1300 or the HDD 1400 and controls each of the components. For example, the CPU 1100 loads a program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0279] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, a program depending on the hardware of the computer 1000 , and the like.

[0280] The HDD 1400 is a computer-readable recording medium that non-transitorily records a program to be executed by the CPU 1100 , data used by such a program, etc. Specifically, the HDD 1400 is a recording medium that records a display control program according to the present disclosure as an example of the program data 1450 .

[0281] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.

[0282] The input and output interface 1600 is an interface for connecting the input and output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input and output interface 1600. The CPU 1100 also sends data to an output device such as a display, a speaker, or a printer via the input and output interface 1600. In addition, the input and output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium. The medium refers to, for example, an optical recording medium such as a digital versatile disk (DVD) or a phase change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a magnetic tape medium, a magnetic recording medium, or a semiconductor memory.

[0283] For example, in the case where the computer 1000 functions as the display control device 100 according to the embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 130 or other units by executing the display control program loaded on the RAM 1200. The HDD 1400 also stores the display control program or data according to the present disclosure in the storage unit 120. Note that although the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program, as another example, these programs may be acquired from another device via the external network 1550.

[0284] Note that the present technology may also have the following configurations.

[0285] (1) A display control device comprising:

[0286] an acquisition unit, wherein the acquisition unit acquires the behavior of the user in the real space as input information;

[0287] a determining unit that determines whether an input corresponding to a pre-registered specific action is included in the input information; and

[0288] A display control unit reflects the behavior of the user corresponding to the input information in an avatar in the virtual space, and further reflects the specific action in the avatar in the virtual space when it is determined that the input information includes an input corresponding to the specific action, so as to display the avatar in the virtual space.

[0289] (2) The display control device according to (1), wherein:

[0290] The acquisition unit

[0291] receiving a designation from the user as to whether to reflect the particular action in the avatar, and

[0292] The determining unit

[0293] In the case where there is the designation of the user, it is determined whether an input corresponding to the specific action is included in the input information.

[0294] (3) The display control device according to (1) or (2), wherein:

[0295] The acquisition unit

[0296] acquiring from an imaging device the position of the user when the behavior is captured, and

[0297] The determining unit

[0298] When the distance between the position and the imaging device is greater than a predetermined distance, it is not determined whether the input corresponding to the specific action is included in the input information, and when the position is within the predetermined distance from the imaging device, it is determined whether the input corresponding to the specific action is included in the input information.

[0299] (4) The display control device according to any one of (1) to (3), wherein:

[0300] The acquisition unit

[0301] Get the behavior of multiple users as input information,

[0302] The determining unit

[0303] determining the activity of the interaction between the plurality of users in the virtual space based on the behaviors of the plurality of users, and

[0304] The display control unit

[0305] The presentation effect is displayed according to the activity of the interaction near the multiple users.

[0306] (5) The display control device according to (4), wherein:

[0307] The determining unit

[0308] The activity of the interaction is determined based on the number of users participating in the interaction, the frequency of conversations between the users, and the degree of overlap of the voices of the multiple users.

[0309] (6) The display control device according to (5), wherein:

[0310] The display control unit

[0311] The expression or occurrence frequency of the presentation effect is determined based on the liveliness of the interaction, the orientation of the body of the avatar, and the content of the conversation between the plurality of users.

[0312] (7) The display control device according to any one of (1) to (6), wherein:

[0313] The determining unit

[0314] determining whether an input corresponding to a specific action associated with a specific emotion is included in the input information, and

[0315] The display control unit

[0316] In a case where it is determined that the input information includes the specific action associated with the specific emotion, the specific action associated with the specific emotion is reflected in the avatar, and the avatar is displayed in the virtual space.

[0317] (8) The display control device according to (7), wherein:

[0318] The determining unit

[0319] Whether an input corresponding to the specific action associated with the specific emotion is included in the input information is determined by comparing voiceprint information associated with the specific emotion with the input information, the voiceprint information being pre-registered by the user.

[0320] (9) The display control device according to (7) or (8), wherein:

[0321] The display control unit

[0322] The avatar reflecting the specific action associated with the specific emotion is displayed, and a rendering effect associated with the specific emotion is displayed near the avatar.

[0323] (10) The display control device according to any one of (1) to (9), wherein:

[0324] The display control unit

[0325] The words spoken by the avatar are displayed based on the input information, and the history of the words is further displayed in a manner of shifting upward from the avatar.

[0326] (11) The display control device according to (10), wherein:

[0327] The display control unit

[0328] The words spoken by the avatar are displayed as icons based on semantic content included in the input information, the icons representing the semantic content with pictures.

[0329] (12) The display control device according to (11), wherein:

[0330] The determining unit

[0331] determining a distance in the virtual space between a location observing a display of the word and a location of an avatar speaking the word, and

[0332] The display control unit

[0333] The history of the icon or the word is displayed based on the determined distance as a presentation effect of the semantic content that does not include the word nor the icon.

[0334] (13) The display control device according to (11) or (12), wherein:

[0335] The determining unit

[0336] determining a positional relationship in the virtual space between a location observing a display of the word and a location of an avatar speaking the word, and

[0337] The display control unit

[0338] A display angle and a three-dimensional display of the words observed in the virtual space are determined based on the determined positional relationship.

[0339] (14) The display control device according to any one of (1) to (13), wherein:

[0340] The determining unit

[0341] determining whether the first avatar is included in a range recognized by the second avatar in the virtual space, and

[0342] The display control unit

[0343] In a case where it is determined that the first avatar is included in the area recognized by the second avatar, a travel path of the first avatar is adjusted so that the first avatar approaches the second avatar.

[0344] (15) The display control device according to (14), wherein:

[0345] The determining unit

[0346] determining whether the first avatar has approached the second avatar in the virtual space, and

[0347] The display control unit

[0348] In a case where it is determined that the first avatar has approached the second avatar, a specific action corresponding to the approach is reflected in at least one of the first avatar or the second avatar to display the first avatar and the second avatar.

[0349] (16) The display control device according to any one of (1) to (15), wherein:

[0350] The acquisition unit

[0351] obtaining a request from the user to change the appearance of the avatar, and

[0352] The display control unit

[0353] The changes based on the request are reflected in the avatar to change the appearance and presentation effect of the avatar in the virtual space.

[0354] (17) The display control device according to any one of (1) to (16), wherein:

[0355] The acquisition unit

[0356] obtaining a request from the user to exchange any of a plurality of objects associated with the first avatar with a second avatar, and

[0357] The display control unit

[0358] When the request is obtained, a list of the plurality of objects associated with the first avatar is displayed near the first avatar.

[0359] (18) The display control device according to (17), wherein:

[0360] The acquisition unit

[0361] acquiring, from the user, an instruction to select any object to be exchanged with the second avatar from among the plurality of objects displayed in the list, and

[0362] The display control unit

[0363] When the instruction is acquired, the object is displayed in a manner of floating from the displayed list, and the floating object is moved in the virtual space to overlap with the second avatar according to an operation received from the user.

[0364] (19) A display control method, comprising:

[0365] By computer

[0366] Obtaining the user's behavior in the real space as input information;

[0367] determining whether an input corresponding to a pre-registered specific action is included in the input information; and

[0368] The user's behavior corresponding to the input information is reflected in an avatar in the virtual space, and when it is determined that the input information includes an input corresponding to the specific action, the specific action is further reflected in the avatar in the virtual space to display the avatar in the virtual space.

[0369] (20) A display control program for causing a computer to function as a display control device, wherein the display control device comprises:

[0370] an acquisition unit, wherein the acquisition unit acquires the behavior of the user in the real space as input information;

[0371] a determining unit that determines whether an input corresponding to a pre-registered specific action is included in the input information; and

[0372] A display control unit reflects the behavior of the user corresponding to the input information in an avatar in the virtual space, and further reflects the specific action in the avatar in the virtual space when it is determined that the input information includes an input corresponding to the specific action, so as to display the avatar in the virtual space.

[0373] Reference numerals list

[0374] 10 Incarnation

[0375] 50 Analysis Server

[0376] 100 display control device

[0377] 110 Communication Unit

[0378] 120 storage units

[0379] 130 control unit

[0380] 131 Acquisition Unit

[0381] 132 Determine the unit

[0382] 133 Display control unit

[0383] 140 sensor units

[0384] 141 Camera Device

[0385] 142 Microphones

[0386] 150 display units

Claims

1. A display control device, comprising: an acquisition unit, wherein the acquisition unit acquires the behavior of the user in the real space as input information; a determining unit that determines whether an input corresponding to a pre-registered specific action is included in the input information; as well as A display control unit reflects the behavior of the user corresponding to the input information in an avatar in the virtual space, and further reflects the specific action in the avatar in the virtual space when it is determined that the input information includes an input corresponding to the specific action, so as to display the avatar in the virtual space.

2. The display control device according to claim 1, wherein: The acquisition unit receiving a designation from the user as to whether to reflect the particular action in the avatar, and The determining unit In the case where there is the designation of the user, it is determined whether an input corresponding to the specific action is included in the input information.

3. The display control device according to claim 1, wherein: The acquisition unit acquiring from an imaging device the position of the user when the behavior is captured, and The determining unit When the distance between the position and the imaging device is greater than a predetermined distance, it is not determined whether the input corresponding to the specific action is included in the input information, and when the position is within the predetermined distance from the imaging device, it is determined whether the input corresponding to the specific action is included in the input information.

4. The display control device according to claim 1, wherein: The acquisition unit Get the behavior of multiple users as input information, The determining unit determining the activity of the interaction between the plurality of users in the virtual space based on the behaviors of the plurality of users, and The display control unit The presentation effect is displayed according to the activity of the interaction near the multiple users.

5. The display control device according to claim 4, wherein: The determining unit The activity of the interaction is determined based on the number of users participating in the interaction, the frequency of conversations between the users, and the degree of overlap of the voices of the multiple users.

6. The display control device according to claim 5, wherein: The display control unit The expression or occurrence frequency of the presentation effect is determined based on the liveliness of the interaction, the orientation of the body of the avatar, and the content of the conversation between the plurality of users.

7. The display control device according to claim 1, wherein: The determining unit determining whether an input corresponding to a specific action associated with a specific emotion is included in the input information, and The display control unit In a case where it is determined that the input information includes the specific action associated with the specific emotion, the specific action associated with the specific emotion is reflected in the avatar, and the avatar is displayed in the virtual space.

8. The display control device according to claim 7, wherein: The determining unit Whether an input corresponding to the specific action associated with the specific emotion is included in the input information is determined by comparing voiceprint information associated with the specific emotion with the input information, the voiceprint information being pre-registered by the user.

9. The display control device according to claim 7, wherein: The display control unit The avatar reflecting the specific action associated with the specific emotion is displayed, and a rendering effect associated with the specific emotion is displayed near the avatar.

10. The display control device according to claim 1, wherein: The display control unit The words spoken by the avatar are displayed based on the input information, and the history of the words is further displayed in a manner of shifting upward from the avatar.

11. The display control device according to claim 10, wherein: The display control unit The words spoken by the avatar are displayed as icons based on semantic content included in the input information, the icons representing the semantic content with pictures.

12. The display control device according to claim 11, wherein: The determining unit determining a distance in the virtual space between a location observing a display of the word and a location of an avatar speaking the word, and The display control unit The history of the icon or the word is displayed based on the determined distance as a presentation effect of the semantic content that does not include the word nor the icon.

13. The display control device according to claim 11, wherein: The determining unit determining a positional relationship in the virtual space between a location observing a display of the word and a location of an avatar speaking the word, and The display control unit A display angle and a three-dimensional display of the words observed in the virtual space are determined based on the determined positional relationship.

14. The display control device according to claim 1, wherein: The determining unit determining whether the first avatar is included in a range recognized by the second avatar in the virtual space, and The display control unit In a case where it is determined that the first avatar is included in the area recognized by the second avatar, a travel path of the first avatar is adjusted so that the first avatar approaches the second avatar.

15. The display control device according to claim 14, wherein: The determining unit determining whether the first avatar has approached the second avatar in the virtual space, and The display control unit In a case where it is determined that the first avatar has approached the second avatar, a specific action corresponding to the approach is reflected in at least one of the first avatar or the second avatar to display the first avatar and the second avatar.

16. The display control device according to claim 1, wherein: The acquisition unit obtaining a request from the user to change the appearance of the avatar, and The display control unit The changes based on the request are reflected in the avatar to change the appearance and presentation effect of the avatar in the virtual space.

17. The display control device according to claim 1, wherein: The acquisition unit obtaining a request from the user to exchange any of a plurality of objects associated with the first avatar with a second avatar, and The display control unit When the request is obtained, a list of the plurality of objects associated with the first avatar is displayed near the first avatar.

18. The display control device according to claim 17, wherein: The acquisition unit acquiring, from the user, an instruction to select any object to be exchanged with the second avatar from among the plurality of objects displayed in the list, and The display control unit When the instruction is acquired, the object is displayed in a manner of floating from the displayed list, and the floating object is moved in the virtual space to overlap with the second avatar according to an operation received from the user.

19. A display control method, comprising: By computer Obtaining the user's behavior in the real space as input information; determining whether an input corresponding to a pre-registered specific action is included in the input information; as well as The user's behavior corresponding to the input information is reflected in an avatar in the virtual space, and when it is determined that the input information includes an input corresponding to the specific action, the specific action is further reflected in the avatar in the virtual space to display the avatar in the virtual space.

20. A display control program for causing a computer to function as a display control device, wherein: The display control device comprises: an acquisition unit, wherein the acquisition unit acquires the behavior of the user in the real space as input information; a determining unit that determines whether an input corresponding to a pre-registered specific action is included in the input information; and A display control unit reflects the behavior of the user corresponding to the input information in an avatar in the virtual space, and further reflects the specific action in the avatar in the virtual space when it is determined that the input information includes an input corresponding to the specific action, so as to display the avatar in the virtual space.

Images