Information processing device and information processing method

Abstract

This information processing device for displaying an avatar that is a user's clone in a virtual space comprises a processor. The processor controls, as an action mode, an action reflection mode in which an avatar being operated by a user of the information processing device reflects an action of the user, and an action non-reflection mode in which the avatar does not reflect the action of the user. In the case of an action in the action reflection mode, the processor performs control so that the action of the user is reflected on the avatar, and in the case of an action in the action non-reflection mode, performs control so that the action of the user is not reflected on the avatar or is only partially reflected.

Description

Information processing device and information processing method

[0001] The present invention relates to an information processing device and an information processing method, and more particularly to an information processing device and an information processing method for displaying and operating an avatar, an alter-ego character that resembles the user, in a virtual space.

[0002] In recent years, virtual reality (VR) technology, which creates a virtual space (virtual world) and virtual objects that resemble reality and allows users to experience the sensation of being in the virtual space, has been widely used, and information processing devices that allow users to easily handle virtual spaces and virtual objects have become widespread. One example is a head-mounted display (HMD) that is worn on the head. A user wearing an HMD can display and operate an avatar that represents their own self in the virtual space, allowing them to have an experience that is different from the real world and communicate with other users via other avatars operated by other users.

[0003] One way to experience a virtual world is to use a personal computer or smartphone and operate an avatar displayed on a display using a keyboard, mouse, etc., but if you want to immerse yourself more deeply in the virtual world, a more suitable method is to wear a goggle-shaped HMD on your head that covers the area around the wearer's (user's) eyes, blocking out the outside world (real world), and displays a virtual world image on a display in front of your eyes, and then operate the avatar using gestures or input on a virtual keyboard.

[0004] Furthermore, when a user wears a closed-type HMD and communicates with other users through a separate avatar operated by the other user, it is expected that communication with other users will become smoother by appropriately reflecting the actions (gestures, etc.) performed by the user in the real world in the avatar in the virtual space.

[0005] As an example of a method for reflecting a user's actions in real space on an avatar in virtual space, Patent Document 1 discloses a method and system for generating a three-dimensional representation of a subject, which comprises: "comparing a depth image of the subject with a plurality of representative images, each representative image being associated with an individual parametric model of the subject; identifying one of the plurality of representative images as the representative image most similar to the depth image of the subject; selecting a parametric model associated with the representative image most similar to the depth image; and fitting the selected parametric model to the depth image of the subject, thereby generating a three-dimensional representation of the subject (excerpt from abstract)."

[0006] JP 2015-15021 A

[0007] Patent Literature 1 describes a technology that more accurately reflects a user's body shape in an avatar. However, there is a problem in that no consideration or suggestion is given to how to deal with security risks that may arise when other users observe and infer highly confidential information such as user IDs, passwords, emails, and documents from the avatar's movements that reflect and reproduce the user's actions. Another problem is that the user's state may be revealed by the avatar's movements even when the user is in a location where they do not want to be tracked or during times when they do not want to be seen. This security risk increases the more accurately the avatar reflects the user's actions.

[0008] On the other hand, accurately reflecting a user's movements in their avatar increases the sense of unity between the avatar and the user operating it, allowing the avatar's movements to more closely resemble natural human movements. This has the advantage of making it easier to feel immersed in the virtual space when interacting with other avatars in the virtual space or viewing one's own avatar from a third-person perspective. Therefore, there is a need to meet the demand for avatar movements to reflect the user's movements.

[0009] The present invention has been made to solve the above-mentioned problems, and aims to reconcile the conflicting demands of having the user's movements reflected more accurately in the avatar, while at the same time preventing information leakage due to the avatar's movements and disclosure of the user's situation to others.

[0010] To briefly outline a representative invention among those disclosed in this application, the present invention is an information processing device that displays an avatar, which is a user's alter ego, in a virtual space, and is equipped with a processor. The processor controls two operation modes: an action reflection mode in which the user's actions are reflected in the avatar operated by the user of the information processing device, and a non-action reflection mode in which the user's actions are not reflected in the avatar. When operating in the action reflection mode, the processor controls the user's actions to be reflected in the avatar, and when operating in the non-action reflection mode, the processor controls the user's actions to not be reflected in the avatar, or to be only partially reflected.

[0011] According to the present invention, it is possible to achieve both of the conflicting demands of more accurately reflecting the user's movements in the avatar, while preventing information leakage due to the avatar's movements and disclosure of the user's status to others. Note that other issues, configurations, and effects will become clear from the description of the following embodiments.

[0012] FIG. 1 is an example of a diagram for schematically explaining the appearance of an example of an information processing device according to the present embodiment; FIG. 2 is another example of a diagram for schematically explaining the appearance of an example of an information processing device according to the present embodiment; FIG. 3 is another example of a diagram for schematically explaining the appearance of an example of an information processing device according to the present embodiment; FIG. 4 is another example of a diagram for schematically explaining the appearance of an example of an information processing device according to the present embodiment; FIG. 5 is a diagram for explaining screen transitions in an example of an information processing device according to the present embodiment; FIG. 6 is another example of a diagram for schematically explaining the appearance of an example of an information processing device according to the present embodiment; FIG. 7 is a block diagram showing an example of the configuration of an information processing device according to the present embodiment; FIG. 8 is an example of a flowchart for explaining the basic operation of the information processing device according to the present embodiment; FIG. 9 is an example of a flowchart for explaining detailed operation of the information processing device according to the present embodiment.

[0013] The information processing device and information processing method according to the present embodiment can reduce the risk of highly confidential information leaking based on the avatar's movements by switching the degree to which an avatar placed and displayed in a virtual space follows the user's movements depending on the user's movements. Therefore, the present invention can increase the commercial value of information processing devices and methods to which the present invention is applied, and is therefore expected to contribute to Goal 8.2 of the United Nations' Sustainable Development Goals (SDGs) (increasing economic productivity through diversification, technological improvement, and innovation, particularly in industries that increase the value of goods and services and labor-intensive industries).

[0014] An example of an embodiment of the present invention will be described below with reference to the drawings. A specific example of an information processing device according to this embodiment will be described using a see-through HMD (Head-Mounted Display) that is worn on a user's head, blocks out the outside world, and displays an avatar representing the user as a virtual object on a display in front of the user's eyes. FIGS. 1A, 1B, 1C, 1D, and 1E are schematic diagrams illustrating the exterior of an example of an information processing device according to a first embodiment. Virtual space images can be rendered from a so-called first-person perspective, in which the virtual space is viewed from the same perspective as the user's avatar, thereby rendering the user's avatar within the virtual space image. Alternatively, virtual space images can be rendered from a so-called third-person perspective, in which the user views the virtual space from a perspective different from the user's avatar, thereby rendering the user's avatar within the virtual space image. While FIGS. 1A to 1E illustrate virtual space images from a third-person perspective, this embodiment can also be applied to rendering virtual space images from a first-person perspective.

[0015] 1A , an HMD 100 is worn on the head of a user 10. The HMD 100 includes a camera 101 that captures the external world around the user, a distance measurement sensor 102 that measures the distance and angle to an object and captures the shape of the object in three dimensions, a left eye gaze sensor 103 and a right eye gaze sensor 104 that detect the gaze of the left and right eyes of the user 10, a microphone 106 that collects sounds such as those uttered by the user 10, and a speaker 107 that produces sounds based on audio data.

[0016] The HMD 100 can capture gesture operations 130 of the user 10's hands or the like using a camera 101 and a distance sensor 102. The HMD 100 also performs wireless communication with an external information server 112 via an external network 111. The information server 112 stores information about the virtual space and virtual space object information that represents user avatars appearing in the virtual space and common virtual objects within the virtual world. The HMD 100 transmits and receives the virtual space information, virtual space object information, and the like to and from the information server 112.

[0017] A display screen 120a displayed on the display of the HMD 100 displays a virtual space image based on virtual space information, and further displays, based on virtual space object information, virtual space objects such as an avatar 121 of user 10, other avatars 123 of other users 10X (see Figures 1D and 1E), and an entrance door 124 to a facility that only pre-registered users (avatars) can enter, and an intercom 125.

[0018] Although not shown, another user 10X (see FIGS. 1D and 1E) operating another avatar 123 also views the virtual space displayed on the HMD 100 of the user 10 from a viewpoint set by the other user 10X (see FIGS. 1D and 1E). The HMD 100X worn by the other user 10X shown in FIGS. 1D and 1E is the same as the HMD 100 worn by the user 10, and therefore the components of the HMD 100X are given the same reference numerals as those used for the components of the HMD 100, and redundant explanations will be omitted. From the perspective of the HMD 100, the HMD 100X corresponds to an external device to which avatar information in the motion reflection mode and avatar information in the non-motion reflection mode is sent.

[0019] If, while the virtual space image of Figure 1A is being displayed, user 10 thinks of having avatar 121 enter a pre-registered facility, in Figure 1A, in an action reflection mode in which the action state of user 10 is reproduced and reflected in avatar 121, user 10 performs gesture operation 130 so that the hand of avatar 121 pushes open entrance door 124 (dashed frame 131), or user 10 performs gesture operation 130 so that the hand of avatar 121 presses intercom 125 (dashed frame 132).

[0020] 1B , by the avatar 121 performing such actions to enter the pre-registered facility, a user ID and password input screen 141, which are virtual objects, and a virtual keyboard 142 for keystroke input are displayed on the display screen 120b. The user ID and password input screen 141 and the virtual keyboard 142 are virtual objects that are displayed on the display of the HMD 100 and are therefore visible only to the user 10. The user ID and password input screen 141 displays a user ID input field 143, a password input field 144, and a login button 145. In addition, the virtual keyboard 142 displays an input key 146 and a return key 147.

[0021] First, the user 10 performs a hand gesture operation 130 to select a user ID input field 143 on the displayed input screen 141. The selection operation performed by the user 10's hand gesture operation 130 is displayed on the display screen 120b as a virtual object 148 of the user 10's hand. The user 10 can perform the selection operation while visually checking the movement of the hand during selection. The virtual object 148 of the user 10's hand is a virtual object that can only be seen by the user 10. At this time, the avatar 121 may be in an operation reflection mode that reflects the operating state of the user 10. Therefore, the avatar 121 reproduces the hand gesture operation 130 of the user 10, as indicated by a dashed frame 149.

[0022] On the other hand, when the display screen 120b is displayed on the HMD 100 of the user 10, as shown in Fig. 1D , a virtual space image similar to the display screen 120b in Fig. 1B is displayed on the display screen 120d of the HMD 100X worn by the other user 10X, as seen from a viewpoint set by the other user 10X. On the display screen 120d, the avatar 121 reproduces the hand gesture operation 130 of the user 10, as indicated by a dashed frame 149. Note that the input screen 141 for entering a user ID or password and the virtual keyboard 142 are not displayed on the display screen 120d.

[0023] 1B , when the user ID input field 143 is selected on the input screen 141, the virtual object 148 of the hand of the user 10 that was displayed on the input screen 141 is erased, and a virtual object 151 of the hand of the user 10 is displayed on the virtual keyboard 142 as shown in FIG. 1C . This indicates that an operation on the virtual keyboard 142, rather than an operation on the input screen 141, has become active. Next, the user 10 performs a hand gesture operation 130 to type input keys on the input keys 146 of the virtual keyboard 142. The results of the key input on the input keys 146 are sequentially displayed in the user ID input field 143 while the user ID information is being input. At this time, the virtual object 151 of the hand of the user 10 who is inputting the user ID information is displayed on the display screen 120c as a virtual object visible only to the user 10, and the user 10 can perform a selection operation while visually checking the movement of the hand during input.

[0024] Meanwhile, while the user ID information is being typed, the HMD 100 switches to a non-motion reflection mode in which the motion state of the user 10 is not reflected in the avatar 121, and the avatar 121 performs an alternative motion that differs from the motion state of the user, such as temporarily stopping the hand movements of the avatar 121, as illustrated by the dashed frame 152. At this time, a virtual space image in which the hand movements of the avatar 121 are temporarily stopped is displayed on the HMD 100X worn by the other user 10X, as shown in FIG. 1E.

[0025] In this way, when the avatar 121 is controlled in the non-motion reflection mode, the hand movements of the avatar 121 are not reflected in the movements of the avatar 121 that are visually recognized by other users 10X, which is one of the features of the information processing device according to this embodiment. In this way, the hand and finger movements of the avatar 121 do not reproduce the key typing actions of the user 10 on the virtual keyboard 142, and therefore it is possible to prevent other users 10X from guessing the contents of the key inputs made by the user 10 on the virtual keyboard 142 from the hand and finger movements of the avatar 121.

[0026] As shown in FIG. 1C , when typing on the input keys 146 of the virtual keyboard 142 is completed, the return key 147 is pressed, and the user ID input process is terminated. Then, the virtual object 151 of the hand of the user 10 displayed on the virtual keyboard 142 is erased, and a virtual object 148 of the hand of the user 10 is displayed on the input screen 141, as shown in FIG. 1B . This indicates that operations on the input screen 141, rather than operations on the virtual keyboard 142, are now active. Furthermore, once the input operations on the virtual keyboard 142 are completed, the system returns to an operation reflection mode in which the motion state of the user 10 is reflected in the avatar 121. Accordingly, the avatar 121 displayed on the HMD 100X of the other user 10X is also displayed in a state that reflects the motion state of the user 10.

[0027] In this way, the avatar 121 does not reproduce the gesture operation of the user 10 indicating the user ID information input operation, but performs an alternative operation, so that other users operating other avatars 123 located near the avatar 121 cannot observe and guess the user ID information input operation of the user 10 by observing the avatar 121 through the other avatars 123. This makes it possible to avoid and eliminate the security risk of other users guessing the user ID information of the user 10.

[0028] 1B , similar to the above-described selection operation of the user ID input field 143 and the input operation of the user ID information, the password input field 144 is selected in the operation reflection mode by the hand gesture operation 130 of the user 10, and then the password information is typed on the virtual keyboard 142 in the non-operation reflection mode. That is, first, the operation of selecting the password input field 144 on the input screen 141 is performed by the hand gesture operation 130 of the user 10. At this time, too, a virtual object 148 of the hand of the user 10 is displayed on the display screen 120b as a virtual object visible only to the user 10, and the user 10 can perform the selection operation while visually checking the movement of the hand during selection. The avatar 121 is in the operation reflection mode that reflects the operation state of the user 10, and the avatar 121 reproduces the hand gesture operation 130 of the user 10, as indicated by the dashed frame 149.

[0029] When the password input field 144 is selected on the input screen 141, the virtual object 148 of the hand of the user 10 that was displayed on the input screen 141 is erased, and a virtual object 151 of the hand of the user 10 is displayed on the virtual keyboard 142. This indicates that an operation on the virtual keyboard 142, rather than an operation on the input screen 141, has become active. Next, the user 10 performs a hand gesture operation 130 to type input keys on the input keys 146 of the virtual keyboard 142. The results of the key input on the input keys 146 are sequentially displayed in the password input field 144 while the password information is being input. At this time, too, the virtual object of the hand of the user 10 entering the password information is displayed on the display screen 120 as a virtual object 151 that can only be seen by the user 10, allowing the user 10 to perform an input operation while visually checking the movement of the hand during input.

[0030] Meanwhile, while the password information is being typed, the HMD 100 switches to a non-motion reflection mode in which the motion state of the user 10 is not reflected in the avatar 121, and the avatar 121 performs an alternative motion that differs from the motion state of the user 10. When the typing on the input keys 146 of the virtual keyboard 142 is finished, the return key 147 is pressed, and the password input process ends. Then, the virtual object 151 of the hand of the user 10 displayed on the virtual keyboard 142 is erased, and a virtual object 148 of the hand of the user 10 is displayed on the input screen 141. This indicates that the operation on the input screen 141, rather than the operation on the virtual keyboard 142, has become active. Furthermore, with the input operation on the virtual keyboard 142 finished, the HMD 100 returns to a motion reflection mode in which the motion state of the user 10 is reflected in the avatar 121. In this way, the avatar 121 does not reproduce the gesture operation of the user 10 indicating the password information input operation, but performs an alternative operation, so that other users operating other avatars 123 located near the avatar 121 cannot observe and guess the password information input operation of the user 10 by observing the avatar 121 through the other avatars 123. This makes it possible to avoid and eliminate the security risk of other users guessing the password information of the user 10.

[0031] After typing in the user ID information and password information, when the user 10 presses the login button 145 by gesture operation 130, the entrance door 124 to the facility opens, allowing the avatar 121 to enter the pre-registered facility.

[0032] The above-described screen transitions will be described with reference to FIG. 1F . FIG. 1F is a diagram illustrating screen transitions in an example of the information processing device according to this embodiment. The display screen 120a in FIG. 1F shows a virtual space image in an action reflection mode in which the movements of the user 10 are reflected in the avatar 121. On the display screen 120a, the HMD 100 worn by the user 10 and the HMD 100X worn by another user 10X each display a virtual space image in which the avatar 121 reflecting the movements of the user 10 is positioned, as seen from the viewpoint of each user.

[0033] Next, when the user 10 activates the input screen 141, the HMD 100 displays a display screen 120b in which the input screen 141 and the virtual keyboard 142 are superimposed on a virtual space image in an action reflection mode state in which the movements of the user 10 are reflected in the avatar 121. At this time, the HMD 100X of the other user 10X displays a display screen 120d in which a virtual space image in an action reflection mode state in which the movements of the user 10 are reflected in the avatar 121 is displayed.

[0034] Next, when the user 10 activates the virtual keyboard 142, the avatar 121 switches to a non-motion reflection mode that does not reflect the movements of the user 10. The HMD 100 displays a display screen 120c in which an input screen 141 and a virtual keyboard 142 are superimposed on a virtual space image in which the avatar 121 that does not reflect the movements of the user 10 is placed. At this time, the HMD 100X of the other user 10X also displays a display screen 120e in which a virtual space image in which the avatar 121 that does not reflect the movements of the user 10 is placed is displayed. When the user 10 deactivates the virtual keyboard 142 on the HMD 100, the avatar 121 switches to the non-motion reflection mode, and the avatar 121 that reflects the movements of the user 10 is also displayed on the HMD 100X.

[0035] When the user 10 successfully logs in to the entrance door 124, the HMD 100 and the HMD 100X both display a display screen 120f showing the avatar 121 having passed through the entrance door 124.

[0036] Thus, when the user 10 types in the user ID information and password information, the HMD 100 operates in a non-action reflection mode in which the user's action state is not reflected in the avatar, preventing other users operating nearby avatars from observing and guessing highly confidential information such as the user ID information and password information. This avoids and eliminates the security risk of other users 10X guessing the user 10's user ID information and password information. In other words, when the HMD 100 inputs highly confidential user ID information and password information, it switches from an action reflection mode in which the user's action is reflected in the avatar to a non-action reflection mode in which the action is not reflected, thereby avoiding and eliminating the risk of the highly confidential user ID information and password information being stolen.

[0037] In addition, as an alternative action of the avatar 121 in the non-motion reflection mode, in addition to temporarily stopping the hand movements of the avatar 121, the avatar 121 may perform an alternative action different from the motion state of the user 10, such as randomly moving the hands or repeating only a predetermined movement. Furthermore, in the non-motion reflection mode, the reflection of the user 10's actions to the avatar 121 may be limited, or the accuracy of the reflection of the user 10's actions to the avatar 121 may be reduced. For example, when typing, the accuracy of the reflection to the avatar 121 may be reduced by reflecting the actions once every few keystrokes or once every few seconds. In this case, there is an advantage that having the avatar 121 move somewhat is preferable to having the avatar 121 not move at all, as it allows the user 10 to know that the avatar 121 is doing something.

[0038] 1B and 1C show a case where the input screen 141 for the user ID and password and the virtual keyboard 142 are displayed simultaneously on the display screen 120 of the HMD 100, but the virtual keyboard 142 may not be displayed during the selection operation of the user ID input field 143 or the password input field 144 on the input screen 141, and may be displayed on the display screen 120 only when the virtual keyboard 142 is to be used after the user ID input field 143 or the password input field 144 has been selected.

[0039] As described above, the processor 320 (see FIG. 3 ) of the HMD 100 may determine whether the user's motion status is a highly confidential motion based on the display content of the HMD 100 and switch between the motion reflection mode and the non-motion reflection mode, or the user 10 may manually switch between the motion reflection mode and the non-motion reflection mode. For example, the virtual keyboard 142 may be provided with an motion mode switching button in addition to the input key 146 and the return key 147, and when the virtual keyboard 142 is displayed on the display screen 120, the motion reflection mode and the non-motion reflection mode may be manually switched depending on whether the motion mode switching button is pressed. Alternatively, the non-motion reflection mode may be maintained throughout the period from when the user ID input field 143 or the password input field 144 is selected until the user ID information or password information is entered. Furthermore, the non-motion reflection mode may be maintained during the selection of the user ID input field 143 or the password input field 144 on the input screen 141, in which the motion status of the user 10 is not reflected in the avatar 121. Furthermore, the operation mode may be switched by the HMD 100 at any predetermined time desired by the user, such as when the user temporarily leaves his / her seat while working or studying, during a break, or after leaving work, and then switched to the operation reflection mode when the user starts working or studying.

[0040] (Another Example of Alternative Action) Next, another example of an alternative action that an avatar is made to perform in a non-action reflection mode in which the user's actions are not reflected in the avatar will be described with reference to the drawings. Figure 2 is a diagram showing a schematic external view of an embodiment of an information processing device in which the keystroke input action of user 10 is not reflected in avatar 201, which is an avatar of user 10, when user 10 is performing various highly confidential execution processes. In Figure 2, parts with the same reference numerals as those in Figures 1A, 1B, 1C, 1D, and 1E perform the same actions as those already described, and therefore detailed description thereof will be omitted.

[0041] 2 , when the user 10 performs a hand gesture operation 230 to type on a virtual keyboard 203 as a highly confidential execution process, the avatar 201 does not mirror the user 10's typing action and performs a keystroke input action on a virtual personal computer 202, but instead performs an alternative action different from the user's motion, such as raising the hand high as shown in a dashed frame 205. Furthermore, the virtual personal computer 202 may be made invisible to other avatars 206. Examples of highly confidential execution processes include a user registration process for a new service, a chat input process (chat limited to a specific user; open chat that can be read by anyone does not need to be confidential), an email creation process, and a document creation / editing process set to a predetermined importance level or higher (creation / editing of a document with a low importance level does not need to be confidential).

[0042] At this time, a virtual object 204 of the user's 10 hand and a virtual keyboard 203 during the execution of the process are displayed on the display screen 120 as virtual objects visible only to the user 10, allowing the user 10 to perform typing operations while visually checking the movements of the user's hand during typing. As a result, the avatar 201 performs an alternative operation without reflecting the typing operations of the user 10, so that other users operating other avatars 206 located near the avatar 201 cannot observe and infer the typing operations of the user 10 by observing the operations of the avatar 201. This avoids and eliminates the security risk of other users inferring the processing operations being executed by the user 10.

[0043] 1A, 1B, 1C, 1D, and 1E show a case where the non-action reflection mode is set when user ID information and password information are typed on the virtual keyboard 142. However, when the user ID and password input screen 141 and the virtual keyboard 142 for typed input are displayed on the display screen 120 of the HMD 100, the mode may be switched to a non-action reflection mode in which the user's actions are not reflected on the avatar.

[0044] In this case, when inputting user ID information or password information, reflection of the information in avatar 201 can be avoided at an early stage of the input process, thereby reliably eliminating security risks. Furthermore, when a predetermined application (document creation / editing application, email application, chat application, etc.) is running, the system may be switched to a non-action reflection mode in which the actions of user 10 are not reflected in avatar 201, regardless of whether virtual keyboard 203 is displayed. In this case, security risks can be eliminated at all times while the application is running, regardless of whether avatar 201 is running the application due to a user operation.

[0045] Furthermore, when the user 10 operating the avatar 201 is in a predetermined location designated as confidential, such as a bathroom or a bath, or when the user 10 is in a highly confidential location or time period, such as during work hours, where the user does not want to be tracked even if the user's actions do not pose a security concern, the device may be switched to a non-motion reflection mode, in which the user's actions are not reflected on the avatar 201. The user's location may be confirmed, for example, by capturing an image of the user's surroundings using the camera 101 (see FIG. 3 ) of the HMD 100 to generate an external image, and then performing subject recognition processing on the external image by the processor 320 (see FIG. 3 ) to identify the user's location. If the user 10 is in a bathroom or a bath, the device may be forcibly switched to the non-motion reflection mode. By presetting locations and time periods where the user 10 does not want to be seen, the user's actions can be hidden by switching to the non-motion reflection mode, in which the user's actions are not reflected on the avatar 201, regardless of whether the user's actions are highly confidential or the type of application the user 10 is using. Examples of highly confidential locations and time periods include safe rooms and locker rooms, and for security reasons, the system may be switched to a non-action reflection mode in which the actions of the user 10 are not reflected on the avatar 201.

[0046] Next, a configuration example of the information processing apparatus according to this embodiment will be described with reference to a block diagram. Fig. 3 is a block diagram showing a configuration example of the information processing apparatus according to this embodiment.

[0047] 3 , an HMD 100 that is worn on the head of a user 10 and displays a display screen 120 including an avatar 121 representing the user 10 is configured appropriately using a camera 101, a distance measurement sensor 102, a left-eye gaze sensor 103, a right-eye gaze sensor 104, a positioning sensor 301, an attitude sensor 302, a microphone 106, a speaker 107, an operation input interface 306, a display 307, a vibrator 308, a processor 320, a memory 330 that stores a program 331 and information data 332, a network communication interface 346, and a short-range wireless communication interface 347, and each component is connected to each other via a bus 360. The HMD 100 also wirelessly communicates virtual space information, virtual space object information, and the like with an external information server 112 via an external network 111 via the network communication interface 346.

[0048] On the other hand, the HMD 100X, which is worn on the head of another user and displays a display screen including another avatar 123 that is the other user's alter ego, has a configuration similar to the HMD 100 and wirelessly communicates virtual space information, virtual space object information, and the like with an external information server 112 via an external network 111. The external information server 112 holds, as virtual space information, configuration information related to background information and facilities within the virtual space. The external information server 112 also holds, as virtual space object information, setting information related to each user's avatar 121 and the other avatars 123, and authentication information (user ID information, password information, etc.) required when each user's avatar 121 and the other avatars 123 enters facilities within the virtual space.

[0049] Next, each component of the HMD 100 will be described in detail.

[0050] The camera 101 captures the user's surroundings and converts light incident from a lens into an electrical signal using an image sensor to obtain a camera-captured image. The HMD 100 extracts gesture operations of the user 10, such as hand movements, from the camera-captured image captured by the camera 101, and obtains gesture operation information that is reflected in the movement of the avatar 121.

[0051] The ranging sensor 102 is a sensor that can measure the distance and angle to an object and capture the shape of the object in three dimensions. Examples of such sensors include a LiDAR (Light Detection and Ranging) sensor that irradiates an object with laser light such as infrared light, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object; a TOF (Time of Flight) sensor that measures the distance by irradiating a subject with pulsed light and measuring the reflection time for each pixel; and a millimeter-wave radar that emits millimeter-wave radio waves, captures the reflected waves, and detects the distance to the object and the state of the object. The HMD 100 uses the ranging sensor 102 to measure the distance and direction to a gesture-operated object, such as the hand of the user 10, and can acquire gesture operation information to be reflected in the movement of the avatar 121 from the shape of the object calculated based on the measurement information.

[0052] The left eye gaze sensor 103 and the right eye gaze sensor 104 can detect the movement and direction of the left and right eyes, respectively, and capture the gaze of the user 10. The process of detecting eye movement can utilize well-known techniques commonly used for eye tracking processing. For example, a method using corneal reflex is known in which an infrared light-emitting diode (LED) is irradiated onto the face, an image is captured with an infrared camera, and the position on the cornea of ​​the light reflected by the infrared LED (corneal reflex) is used as a reference point to detect the movement of the eye and gaze based on the position of the pupil relative to the position of the corneal reflex. Character input is possible by aligning the gaze with each key on the virtual keyboard 142. The left eye gaze sensor 103 and the right eye gaze sensor 104 may be a left internal camera and a right internal camera capable of capturing images of the areas around the left and right eyes of the user 10, respectively.

[0053] The positioning sensor 301 receives radio signals from GPS (Global Positioning System) satellites in the sky to detect the current location, and can identify the position of the HMD 100 and the current location of the user 10 wearing the HMD 100.

[0054] The posture sensor 302 is composed of an acceleration sensor 303, a gyro sensor 304, a geomagnetic sensor 305, and the like, and detects posture states such as direction and tilt. The acceleration sensor 303 is a sensor that detects acceleration, which is a change in speed per unit time, and can capture movement, vibration, shock, and the like. The gyro sensor 304 is a sensor that detects angular velocity in the rotation direction, and can capture vertical, horizontal, and diagonal posture states. The geomagnetic sensor 305 is a sensor that detects the Earth's magnetic force, and detects the direction in which the HMD 100 is facing. Based on the posture of the user, such as direction and tilt, detected by these sensors, the user's motion state can be grasped and acquired. Furthermore, the grasped motion state of the user can be reflected in the avatar.

[0055] The operation input interface 306 is an interface with input means such as a hand, line of sight, or pointer, and is used by the user 10 to set and input desired information. An input operation screen such as a user ID and password input screen 141, a virtual keyboard 142, and an operation menu is displayed on the display screen 120 of the display 307, and input information is captured based on the position pointed to by the hand or the position of the gaze on the input operation screen. Examples of operation input include selection setting of a user ID input field 143 and a password input field 144, and input setting of user ID information and password information using the virtual keyboard 142.

[0056] In a masked HMD worn on the user's head, the display 307 is configured with a display device using a liquid crystal panel, a hologram, etc., and displays virtual objects in a virtual space. In reality, a display for the left eye and a display for the right eye are installed in front of the left eye and right eye, respectively, and an image for the left eye and an image for the right eye are displayed on each display, with a predetermined sense of distance due to the left-right parallax.

[0057] The processor 320 is composed of a CPU or the like, and controls each component of the HMD 100 by executing an operating system (OS) 333 and an application program 334 for operation control, etc., stored in the memory 330, thereby realizing functions of the OS, middleware, applications, etc. In the processor 320, an avatar generation processing unit 321, a user operation state confidentiality identification processing unit 322, an operation mode switching processing unit 323, and an avatar substitute operation processing unit 324 control the operation of each function.

[0058] The memory 330 is configured with a nonvolatile storage device or the like, and stores various programs 331 and information data 332 handled by the processor 320, etc. The programs 331 include an operating system 333 and various application programs 334 for controlling operations used by the processor 320. The information data 332 stores confidential user ID information 335, password information 336, virtual space object information 337 consisting of information about the virtual space and virtual objects in the virtual space including the avatar, confidential user operation state information 338 indicating the confidential operation state of the user, and avatar alternative operation information 339 indicating alternative operations of the avatar that differ from the user operation state.

[0059] The microphone 106 collects the user's own speech and external sounds and converts them into audio data. The user 10 speaks a voice indicating an instruction such as an input operation, and the voice is collected by the microphone 106 to capture the instruction information such as the input operation, thereby enabling the user to conveniently execute an operation in response to the instruction information.

[0060] The speaker 107 outputs sound based on the audio data, and can notify the user 10 of notification instruction information by sound. For example, a message such as "The non-operation reflection mode has been entered" can be issued to improve usability.

[0061] The vibrator 308 generates vibrations under the control of the processor 320, and converts notification information sent by the HMD 100 to the user 10 into vibrations. The vibrator 308 transmits vibrations to the head of the user wearing the HMD 100, and can notify the user 10 by vibrations corresponding to a message such as "The non-motion reflection mode has been entered," thereby improving usability.

[0062] The network communication interface 346 is a communication interface that performs wireless communication with an external information server 112 via the external network 111 using base station communication or the like. The network communication interface 346 transmits and receives virtual space information and virtual space object information stored in the information server 112 to and from the information server 112, enabling the user 10 to experience a virtual space that is different from the real world. In addition, communication with other users can be enabled through the avatars of other users, which are virtual space object information. As base station communication, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access), GSM (registered trademark) (Global System for Mobile communications), LTE (Long Term Evolution, registered trademark), LTE-Advanced, Mobile WiMAX (Worldwide Interoperability for Microwave Access, registered trademark), or WiMAX2 may be used.

[0063] The short-range wireless communication interface 347 is a communication interface through which the HMD 100 and the information server 112 perform short-range wireless communication when the information server 112 is within a range where short-range wireless communication is possible. Short-range wireless communication is performed using, for example, an electronic tag, but is not limited to this, and may also be performed using Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or a wireless LAN (IEEE802.11a, IEEE802.11b, IEEE802.11g, etc.).

[0064] Next, the operation of each processing unit performed by the processor 320 will be described.

[0065] The avatar generation processing unit 321 generates images that constitute a virtual space based on the virtual space information. The avatar generation processing unit 321 also generates an avatar 121, which is an avatar of the user 10, based on the virtual space object information and places the avatar 121 in the virtual space.

[0066] The user operation state confidentiality identification processing unit 322 identifies whether a user's operation is confidential. For example, the user's operation is identified as confidential when the user 10 types user ID information 335, password information 336, etc., on the virtual keyboard 142, 203, or when the user registers for a new service (obtaining a new user ID and setting a password associated with the obtained user ID), inputs chat information, creates an email, or creates or edits a document with a predetermined importance level or higher. The user operation state confidentiality identification processing unit 322 may also identify the user 10 as confidential regardless of the user's operation when the user ID or password input screen 141 or the virtual keyboard 142 is displayed, when a predetermined application software is running, or when the user is in a predetermined location or time period. The user operation state confidentiality identification processing unit 322 may also identify the user 10 as confidential when the user registers, inputs or creates email or chat information, or creates or edits a document or scheduler.

[0067] The operation mode switching processing unit 323 switches between an operation reflection mode in which the user's operation state is reflected in the avatar and a non-operation reflection mode in which the user's operation state is not reflected in the avatar, depending on the identification result of the user operation state confidentiality identification processing unit 322 as to whether the user's operation is confidential or not.

[0068] As an example of processing by the operation mode switching processor 323, in the operation reflection mode, the avatar generation processor 321 may be configured to reflect the sensor information output by the orientation sensor 302 in the avatar by not preventing the avatar generation processor 321 from acquiring the sensor information output by the orientation sensor 302. Since the sensor information acquired from the orientation sensor 302 indicates the user's movement, by reflecting the sensor information output by the orientation sensor 302 in the avatar, the user's movement can be reflected in the avatar's movement.

[0069] On the other hand, in the non-motion-reflection mode, the motion mode switching processing unit 323 may control the avatar generation processing unit 321 to prevent the avatar generation processing unit 321 from acquiring the sensor information, thereby preventing the user's motion from being reflected in the avatar. For example, in the non-motion-reflection mode, the motion mode switching processing unit 323 may completely or partially block the output of the sensor information to the avatar generation processing unit 321 (equivalent to thinning out the sensor information along the time axis, etc.), or may disable the sensor information even if the avatar generation processing unit 321 acquires it (the avatar generation processing unit 321 does not use the sensor information in the avatar movement control process). Also, in the non-motion-reflection mode, the motion mode switching processing unit 323 may control the output of alternative motion information generated by the avatar alternative motion processing unit 324 to the avatar generation processing unit 321, and the avatar generation processing unit 321 may control the motion of the avatar based on the alternative motion information rather than the sensor information output by the posture sensor 302.

[0070] When the operation mode switching processing unit 323 switches the operation mode of the HMD 100 to a non-operation reflection mode in which the user's operation state is not reflected in the avatar, the avatar alternative operation processing unit 324 controls the avatar's operation so that it does not reflect the user's operation state but instead performs an alternative operation that is different from the user's operation state.

[0071] With the above configuration, the HMD 100 displays the avatar 121, which is an avatar of the user 10, generated by the avatar generation processing unit 321 in the virtual space. In the HMD 100, the user motion state confidentiality identification processing unit 322 identifies whether the user motion state is confidential, and if the user motion state is not confidential, the motion mode switching processing unit 323 switches the motion mode of the HMD 100 to a motion reflection mode in which the motion state of the user 10 is reflected in the avatar 121 displayed in the virtual space. As a result, the avatar 121 operates by reflecting the user's gestures and the like. On the other hand, if the user motion state is confidential, the motion mode switching processing unit 323 switches the motion mode of the HMD 100 to a non-motion reflection mode in which the motion state of the user 10 is not reflected in the avatar 121 displayed in the virtual space. As a result, the avatar 121 does not reflect the user's gestures and the like. Furthermore, the avatar substitute action processing unit 324 performs substitute actions that differ from the user's motion state as the avatar motion, thereby avoiding or eliminating the security risk of other users observing and guessing highly confidential information such as user IDs, passwords, emails, and documents, and also enabling users to operate an avatar in a virtual space, have an experience different from the real world, and communicate with other users via their avatars, without their motion state in confidential places or at confidential times being discovered.

[0072] Next, the operation of the information processing device according to this embodiment will be described using a flowchart. Fig. 4 is an example of a flowchart illustrating the basic operation of the information processing device according to this embodiment. Fig. 5 is an example of a flowchart illustrating, as a detailed example of the flowchart of the information processing device according to this embodiment, the process of authenticating an avatar to enter a facility or equipment for which a user has registered using user ID information and password information.

[0073] In FIG. 4, the HMD 100 displays the avatar 121 or 201, which is an avatar of the user 10, in the virtual space under the control of the avatar generation processing unit 321 (S401).

[0074] Further, under the control of the user action state confidentiality identification processing unit 322, it is identified whether or not the action state of the user operating the avatar 121, 201 displayed in the virtual space is confidential (S402).

[0075] If the motion state of the user 10 is not confidential, the process branches to "No" in S402, and the motion mode switching processing unit 323 switches the motion mode of the HMD 100 to a motion reflection mode in which the motion state of the user 10 is reproduced and reflected in the avatar 121 or 201 (S403). In response to this, the avatar generation processing unit 321 reflects the motion of the user 10 in the avatar 121 or 201. Then, the process proceeds to step S405.

[0076] On the other hand, if the user's operating state is confidential, the processing of S402 branches to the ``Yes'' side, and the operating mode switching processing unit 323 switches the operating mode of the HMD 100 to a non-operation reflection mode in which the operating state of the user 10 is not reproduced or reflected in the avatar 121 or avatar 201.

[0077] Furthermore, the avatar substitute action processing unit 324 causes the avatar 121 or avatar 201 to perform a substitute action different from the action state of the user 10 (S404). Thereafter, it is confirmed whether or not the display of the avatar 121 or avatar 201 is to be ended (S405), and if the display of the avatar 121 or avatar 201 is not to be ended (S405: No), the process returns to step S402, where it repeats the process of identifying whether or not the action state of the user 10 is confidential and switching to the action reflection mode or the non-action reflection mode.

[0078] On the other hand, when the display of the avatar 121 or avatar 201 is finished (S405: Yes), the entire sequence ends. This avoids or eliminates the security risk of highly confidential information such as a user ID, password, email, or document being observed and guessed by other users 10X, and also allows the user to operate the avatar 121 or avatar 201 in the virtual space without the user's behavior in a confidential place or time being discovered.

[0079] 5, the avatar generation processing unit 321 first displays an avatar 121 representing the user 10 in a virtual space. At this point, the HMD 100 may be in an action reflection mode that reproduces and reflects the user 10's actions. The user 10 uses a gesture operation to cause the avatar 121 to perform an action indicating the start of entry into a facility or equipment for which the user is registered, such as pushing open the entrance door 124 with the avatar 121's hand (dashed frame 131) or pressing the intercom 125 with the avatar 121's hand (dashed frame 132) (S501, see FIG. 1A). In response to the avatar action, the HMD 100 displays a user ID and password input screen 141 and a virtual keyboard 142 on the display screen 120b displayed on the display 307 (S502, see FIG. 1B). The avatar 121 controlled in the action reflection mode is displayed on the HMD 100X worn by another user 10X (see FIG. 1D).

[0080] When the user 10 selects the user ID input field 143 on the input screen 141 by a gesture operation (S503), the HMD 100, under the control of the user motion state confidentiality identification processing unit 322 and the operation mode switching processing unit 323, switches the operation mode of the HMD 100 to a non-motion reflection mode in which the user's motion is not reflected in the avatar (S504, see FIG. 1C ). That is, based on the user 10's selection of the user ID input field 143, the user motion state confidentiality identification processing unit 322 identifies that a highly confidential user ID input process will be performed after this. Then, based on the identification result of the user motion state confidentiality identification processing unit 322, the operation mode switching processing unit 323 switches the operation mode of the HMD 100 to a non-motion reflection mode in which the user's motion is not reflected in the avatar. As a result, the avatar 121 controlled in the non-motion reflection mode is also displayed on the HMD 100X worn by the other user 10X (see FIG. 1E ).

[0081] Then, in the HMD 100, the user ID information, which is the input information for the selected user ID input field 143, is input via the virtual keyboard 142 by a gesture operation by the user 10 (S505). Note that, since the gesture operation by the user 10 is not reflected in the movement of the avatar 121 during the input operation of the user ID information, other users 10X cannot guess the highly confidential user ID information via other avatars 123 that can observe the movement of the avatar 121, and this can avoid and eliminate the risk of the user ID information being stolen by other users 10X.

[0082] In the HMD 100, when the input of the user ID information, which is the input information, into the selected input field is completed (S506), the operation mode switching processing unit 323 controls the HMD 100 to switch the operation mode of the HMD 100 to an operation reflection mode in which the user's operation is reflected in the avatar (S507, see FIGS. 1B and 1D). Furthermore, since the input process of the user ID information has been completed but the input process of the password information has not been performed (S508), the process returns to sequence S503, and next, the user 10 selects the password input field 144 on the input screen 141 by a gesture operation (S503). In the HMD 100, when the password input field 144 is selected, the user operation state confidentiality identification processing unit 322 and the operation mode switching processing unit 323 control the HMD 100 to switch the operation mode of the HMD 100 to a non-operation reflection mode in which the user's operation is not reflected in the avatar (S504, see FIGS. 1C and 1E). Then, in the HMD 100, the password information, which is the input information to the selected password input field 144, is input via the virtual keyboard 142 by a gesture operation by the user 10 (S505).

[0083] In the HMD 100, when the input of password information into the selected password input field is completed (S506), the operation mode switching processing unit 323 controls the operation mode of the HMD 100 to switch to an operation reflection mode in which the user's actions are reflected in the avatar (S507).

[0084] After completing the input of both the user ID information and password information through the above sequence (S508), the user 10 then performs a gesture operation to press the login button 145 on the input screen 141. The user ID information entered in the user ID input field 143 and the password information entered in the password input field 144 are then sent to the information server 112, where they are compared with the user ID information and password information pre-registered in the information server 112 for authentication. If the authentication process confirms that the user 10 is a registered user, the avatar 121 of the user 10 can enter the facility or equipment for which the highly confidential user ID information and password information are registered (S509, see FIG. 1F ).

[0085] In the above process, during the input operation of the user ID information and password information, the actions of the user 10 are not reflected in the actions of the avatar 121, so even if other users observe the actions of the avatar 121 through the other avatars 123, they cannot observe and infer the highly confidential user ID information or password information, thereby avoiding and eliminating the risk of the user ID information or password information being stolen by other users. That is, in the HMD 100, during the input process of the highly confidential user ID information and password information, by switching from an action reflection mode in which the user's actions are reflected in the avatar's actions to a non-action reflection mode in which the actions are not reflected, the risk of the highly confidential user ID information and password information being stolen can be avoided and eliminated.

[0086] Also known is an HMD 100 that includes a mouth camera at the bottom of the housing of the HMD 100 that can capture images of the area around the user's mouth. The mouth camera captures and captures the user's mouth movements (the shape of the user's mouth when speaking), and the captured mouth movements of the user 10 are reflected in the mouth movements (the shape of the user's mouth when speaking) of the avatar 121, thereby making the avatar 121's movements during conversation more realistic. Meanwhile, known technologies have already made it possible to perform confidential execution processes, such as inputting user ID information and password information, and creating and editing emails, chats, documents, etc., using voice input (voice control). In this case, if the user 10 wears the HMD 100 that can reflect the user's mouth movements in the avatar 121's mouth movements, and the avatar accurately reflects the user's mouth movements as they are spoken, there is a risk that the user's ID, password, created document, etc. may be observed and inferred from the avatar 121's mouth movements. In contrast, when a confidential execution process is performed by voice input, in a non-action reflection mode in which the movements of the user 10 are not reflected in the movements of the avatar 121, the mouth movements of the user 10 are not reflected in the mouth movements of the avatar 121, and an alternative process is performed for the mouth movements of the avatar 121. Or, at least, the mouth movements are not reflected accurately. Thus, even in voice input processing, it is possible to avoid and eliminate the risk that the contents of the confidential execution process will be guessed and leaked.

[0087] Also known is an HMD 100 that includes an eye-capturing camera inside the housing of the HMD 100 that can capture images of the area around the user's eyes. The eye-capturing camera captures and captures the eye movements (gaze direction) of the user 10, and the captured eye movements (gaze direction) of the user 10 are reflected in the eye movements of the avatar 121, thereby making the facial expressions of the avatar 121 more realistic. Meanwhile, publicly known technology has already made it possible for the user 10 to input characters by focusing their gaze on each key of the virtual keyboard 142 for a predetermined period of time, instead of using typing gestures to select each key. Given this, if a user wearing an HMD that can reflect the eye movements of the user 10 in the eye movements of the avatar 121 attempts to enter a user ID, password, created document, etc. by focusing their gaze on the virtual keyboard, there is a risk that the character input of the user ID, password, created document, etc. may be observed and inferred from the avatar's gaze movements. In contrast, when confidential character input processing is performed using gaze detection, in a non-action reflection mode in which the user's movements are not reflected in the avatar's movements, the user's eye movements are not reflected in the avatar's eye movements, and alternative processing is performed on the avatar's eye movements. Or, at least, they are not reflected accurately. Therefore, even when characters are input using a virtual keyboard by focusing the eyes on the virtual keyboard, the risk of the contents of the confidential character input processing being guessed and leaked can be avoided.

[0088] The above explanation has focused on the case of a closed-type HMD that blocks the outside world and displays a virtual space and virtual space objects on a display in front of the eyes, but this technology can also be applied to a so-called optically transparent HMD that is not a closed-type but has an optically transparent display in front of the eyes.

[0089] Furthermore, even when a personal computer or the like is used as a method for experiencing a virtual space, it is conceivable that, in order to facilitate smoother communication with other users, the user's own movements can be captured using a camera mounted on the top of the personal computer monitor and reflected on an avatar. In this case, if the avatar accurately reflects the operation state of the keyboard or mouse, there is a risk that highly confidential input content can be observed and inferred by observing the avatar's movements, just as in the case of a non-viewable HMD. That is, even when a personal computer or the like is used as a method for experiencing a virtual space, it is preferable to use a non-action reflection mode in which the user's movement state is not reflected on the avatar during input processing of user ID information, password information, etc. The same applies when a smartphone or the like is used as a method for experiencing a virtual space.

[0090] Furthermore, since the user's 10 movements can be inferred not only from the movements of the avatar 121 but also from, for example, the movements of a pet near the avatar 121, the user's 10 movements may not be reflected in virtual objects near the avatar 121. Furthermore, since the movements of the avatar 121 affect the surrounding environment, the user's 10 movements may be inferred by observing the surrounding environment. For example, the user's 10 movements may be reflected in a reflecting object installed in the virtual space image. Therefore, when the avatar 121's movements are set to a non-motion reflection mode that does not reflect the user's 10 movements, the surrounding environment (for example, an area within a predetermined distance from the avatar 121 operated by the user 10) may also be controlled so that it does not change in conjunction with the avatar 121 movements.

[0091] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0092] Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0093] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the program, table, and file that implements each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communication network.

[0094] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines are necessarily shown on the product. In reality, it can be assumed that almost all components are interconnected.

[0095] The above embodiments include the following inventions: (Supplementary Note 1) An information processing device that displays an avatar representing a user in a virtual space, comprising a processor, the processor controls operation modes including an action reflection mode in which actions of the user are reflected in an avatar operated by the user of the information processing device, and a non-action reflection mode in which actions of the user are not reflected in the avatar, when operating in the action reflection mode, the processor controls the information processing device to reflect the actions of the user in the avatar, and when operating in the non-action reflection mode, the processor controls the information processing device to not reflect, or only partially reflect, the actions of the user in the avatar. (Supplementary Note 2) An information processing method in an information processing device that displays an avatar representing a user in a virtual space, wherein the information processing device: executes a confidentiality identification step of identifying whether the motion state of a user of the information processing device is a highly confidential motion; if the motion state of the user is identified in the confidentiality identification step as not a highly confidential motion, executes an action reflection step of reflecting the user's motion in an avatar operated by the user; and if the motion state of the user is identified in the confidentiality identification step as a highly confidential motion, executes a non-action reflection step of not reflecting the user's motion in the avatar, or only partially reflecting the user's motion.

[0096] 10: User, 10X: Other users, 100: HMD, 100X: HMD, 101: Camera, 102: Distance measurement sensor, 103: Left eye gaze sensor, 104: Right eye gaze sensor, 106: Microphone, 107: Speaker, 111: External network, 112: Information server, 120: Display screen, 120a: Display screen, 120b: Display screen, 120c: Display screen, 120d: Display screen, 120e: Display screen, 120f: Display screen, 121: Avatar, 123: Other avatars , 124: entrance door, 125: intercom, 130: gesture operation, 131: dashed frame, 132: dashed frame, 141: input screen, 142: virtual keyboard, 143: user ID input field, 144: password input field, 145: login button, 146: input key, 147: return key, 148: virtual object, 149: dashed frame, 151: virtual object, 152: dashed frame, 201: avatar, 202: personal computer, 203: virtual Virtual keyboard, 204: virtual object, 205: dashed frame, 206: other avatar, 230: gesture operation, 301: positioning sensor, 302: attitude sensor, 303: acceleration sensor, 304: gyro sensor, 305: geomagnetic sensor, 306: operation input interface, 307: display, 308: vibrator, 320: processor, 321: avatar generation processing unit, 322: user action state confidentiality identification processing unit, 323: operation mode switching processing unit, 324: avatar alternative action processing unit, 330: memory, 331: program, 332: information data, 333: operating system, 334: application program, 335: user ID information, 336: password information, 337: virtual space object information, 338: confidential user action state information, 339: avatar alternative action information, 346: network communication interface, 347: short-range wireless communication interface, 360: bus

Claims

1. An information processing device that displays an avatar representing a user in a virtual space, comprising a processor that controls operation modes between an action reflection mode in which the user's actions are reflected in the avatar operated by the user of the information processing device, and a non-action reflection mode in which the user's actions are not reflected in the avatar, and when operating in the action reflection mode, controls the information processing device to reflect the user's actions in the avatar, and when operating in the non-action reflection mode, controls the information processing device to not reflect the user's actions in the avatar, or to only partially reflect the user's actions in the avatar.

2. An information processing device according to claim 1, wherein the processor determines whether to execute the operation reflection mode or the non-operation reflection mode based on at least one of the user's operation states, and switches between the operation reflection mode and the non-operation reflection mode.

3. An information processing device according to claim 2, wherein the processor switches the operation mode to the non-operation reflection mode when the user's operation state is a highly confidential operation.

4. An information processing device according to claim 3, wherein the processor determines that the operation is highly confidential when at least one of the following operations is being performed: inputting at least one of a user ID and password, user registration, inputting and creating email or chat, and creating and editing a document or scheduler.

5. An information processing device according to claim 3, wherein the processor determines that the user's behavior is highly confidential when the user is in a predetermined location or during a predetermined time period that has been determined to be confidential.

6. An information processing device according to claim 2, further comprising a display, wherein the processor determines that the user's operating state is a highly confidential operation when an input screen for inputting at least one of a user ID and a password is displayed on the display, or when a virtual keyboard is displayed on the display.

7. An information processing device according to claim 1, wherein the processor controls the avatar to temporarily stop its movement, to make the avatar perform a random movement, or to make the avatar perform an alternative movement different from the user's actual movement while the non-motion reflection mode is being executed.

8. An information processing device according to claim 1, further comprising a communication interface for transmitting and receiving data to and from an external device, wherein the processor, when operating in the action reflection mode, controls the communication interface to transmit avatar information in which the user's actions are reflected in the avatar to the external device, and when operating in the non-action reflection mode, controls the communication interface to transmit avatar information in which the user's actions are not reflected in the avatar or only partially reflected to the external device.

9. An information processing method in an information processing device that displays an avatar representing a user in a virtual space, wherein the information processing device executes a confidentiality identification step of identifying whether the motion state of a user of the information processing device is a highly confidential motion, and if the motion state of the user is identified as not being a highly confidential motion in the confidentiality identification step, executes an action reflection step of reflecting the user's motion in an avatar operated by the user, and if the motion state of the user is identified as being a highly confidential motion in the confidentiality identification step, executes a non-action reflection step of not reflecting the user's motion in the avatar, or only partially reflecting the user's motion.

10. An information processing method according to claim 9, wherein the information processing device, when executing the action reflection step, transmits avatar information in which the user's action is reflected in the avatar to an external device, and, when executing the non-action reflection step, transmits avatar information in which the user's action is not reflected in the avatar, or avatar information in which the user's action is only partially reflected, to the external device.

Images