PROGRAM, INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING SYSTEM, AND INFORMATION PROCESSING METHOD

Abstract

To provide a program for improving virtual experience of a user.SOLUTION: A program to be executed by a computer for providing a user B with virtual experience includes: a step of executing first action in a virtual space 200B; a step of executing second action with respect to another user A different from the user B, on the basis of the execution of the first action; and a step of executing third action in the virtual space 200B on the basis of a reaction to the second action.SELECTED DRAWING: Figure 20

Description

[Technical field]

[0001] The present disclosure relates to a program, an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] As described in Non-Patent Document 1, a service is known that allows a plurality of users sharing a virtual space to enjoy communication involving body language using virtual hands or the like. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Social VR Demo - Selfie Stick and 360 Photo Spheres - Oculus”, [online], April 13, 2016, Oculus, [Retrieved August 10, 2017], Internet<https: / / youtu.be / -pumFtAjgLY?t=5> Summary of the Invention [Problem to be solved by the invention]

[0004] However, in services such as the one described in Non-Patent Document 1 above, there is room for improvement in order to improve the virtual experience of users.

[0005] The present disclosure aims to enhance a user's virtual experience. [Means for solving the problem]

[0006] According to one aspect of the program disclosed herein, A computer for providing a virtual experience to a user, performing a first action in a virtual space; executing a second action on a user different from the user based on the execution of the first action; and executing a third action in the virtual space based on a reaction to the second action. Effect of the Invention

[0007] According to the present disclosure, the virtual experience of a user can be improved. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing a virtual space distribution system according to an embodiment of the present invention (hereinafter simply referred to as this embodiment). [Diagram 2] FIG. 2 is a schematic diagram showing a user terminal; [Diagram 3] FIG. 2 is a diagram showing the head of a user wearing an HMD. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of a control device. [Diagram 5] 11 is a flowchart showing a process for displaying a visual field image on an HMD. [Figure 6] FIG. 2 is an xyz space diagram showing an example of a virtual space. [Figure 7] State (a) is a yx plan view of the virtual space shown in Fig. 6. State (b) is a zx plan view of the virtual space shown in Fig. 6. [Figure 8] FIG. 2 is a diagram showing an example of a visual field image displayed on an HMD. [Figure 9] FIG. 2 is a diagram illustrating a hardware configuration of a server illustrated in FIG. [Figure 10] State (a) is a diagram showing a virtual space provided to user A. State (b) is a diagram showing a virtual space provided to user B. [Figure 11] FIG. 11 is a sequence diagram illustrating an example of a process for synchronizing the movements of avatars between user terminals. [Figure 12]11 is an example of a flowchart for explaining the information processing method according to the present embodiment in a case where a user A has removed the HMD. [Figure 13] State (a) is an example of a flowchart for explaining a process for determining whether or not user A has removed the HMD using an HMD sensor. State (b) is an example of a flowchart for explaining a process for determining whether or not user A has removed the HMD using a wearing sensor. [Figure 14] 11 is a diagram showing an example of a virtual space provided to a user B for explaining an information processing method according to the present embodiment. FIG. [Figure 15] 13 is a diagram showing another example of the virtual space provided to user B for explaining the information processing method according to the embodiment. FIG. [Figure 16] 11 is an example of a flowchart for explaining the information processing method according to the present embodiment in a case where user A puts on the HMD again. [Figure 17] 13 is a diagram showing another example of the virtual space provided to user B for explaining the information processing method according to the embodiment. FIG. [Figure 18] 13 is an example of a flowchart for explaining an information processing method according to Modification 1. [Figure 19] FIG. 13 is a diagram showing an example of a virtual space 200B provided to a user B for illustrating an information processing method according to a first modified example. [Figure 20] FIG. 13 is a diagram showing an example of a virtual space 200B provided to a user B for illustrating an information processing method according to a first modified example. [Figure 21] FIG. 13 is a diagram showing an example display of a user terminal 5 associated with a user A, for illustrating an information processing method according to a first modified example. [Figure 22] FIG. 13 is a diagram showing an example of a virtual space 200B provided to a user B for illustrating an information processing method according to a first modified example. [Figure 23] FIG. 13 is a diagram showing an example of a virtual space 200B provided to a user B for illustrating an information processing method according to a first modified example. [Figure 24]13 is an example of a flowchart for explaining an information processing method according to Modification 1 in a case where a user A does not wear an HMD within a certain time period after receiving read information. [Diagram 25] 13 is an example of a flowchart for explaining an information processing method according to Modification 2. [Figure 26] FIG. 13 is a diagram showing an example of a virtual space 200B provided to a user B for illustrating an information processing method according to a second modification. [Figure 27] FIG. 13 is a diagram showing an example of a virtual space 200B provided to a user B for illustrating an information processing method according to a second modification. [Figure 28] FIG. 13 is a diagram showing an example of a virtual space 200B provided to a user B for illustrating an information processing method according to a second modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Description of the embodiment shown in the present disclosure] An overview of the embodiments disclosed herein will be described. (1) An information processing method executed by a computer in a virtual space distribution system including a first user terminal having a first head-mounted device worn on the head of a first user, and a second user terminal having a second head-mounted device worn on the head of a second user, The information processing method includes: (a) generating virtual space data defining a virtual space including a first avatar associated with the first user and a second avatar associated with the second user; (b) updating a field of view image displayed on the second head mounted device based on the movement of the second head mounted device and the virtual space data; (d) when non-wearing information indicating that the first user is not wearing the first head mounted device is received, indicating to the second user by the second user terminal that the first user is not wearing the first head mounted device.

[0010] According to the above method, when non-wearing information indicating that the first user is not wearing a first head mounted device (hereinafter, first HMD) is received, the second user terminal presents to the second user that the first user is not wearing the first HMD. In this way, the second user can easily know that the first user is not wearing the first HMD while communicating with the first user in the virtual space. Therefore, a rich virtual experience can be provided to the user.

[0011] In particular, if the second user does not know that the first user is not wearing the first HMD, the second user may feel uncomfortable with the first user not showing any reaction. In this way, by making the second user aware that the first user is not wearing the first HMD, it is possible to avoid a situation in which the second user feels uncomfortable with the first user not showing any reaction.

[0012] (2) The step (d) The information processing method described in item (1), comprising a step of visualizing, when the non-wearing information is received, on a field of view image displayed on the second head mounted device, that the first user is not wearing the first head mounted device.

[0013] According to the above method, when non-wearing information indicating that the first user is not wearing the first head mounted device (first HMD) is received, information indicating that the first user is not wearing the first HMD is visualized on a field of view image displayed on a second head mounted device (second HMD). In this way, while communicating with the first user in the virtual space, the second user can easily understand that the first user is not wearing the first HMD by visually recognizing the information indicating that the first user is not wearing the first HMD through the second HMD.

[0014] (3)(e) when face information indicating a facial expression of the first avatar is received, updating a facial expression of the first avatar based on the face information; The step (d) The information processing method according to item (1) or (2), further comprising a step of visualizing a first avatar, the facial expression of which is set to a first manner, on a field of view image displayed on the second head-mounted device when the non-wearing information is received.

[0015] According to the above method, when non-wearing information indicating that the first user is not wearing the first head-mounted device (first HMD) is received, the first avatar with a facial expression set to a first manner is visualized on the field of view image displayed on the second head-mounted device (second HMD). In this way, while communicating with the first user in the virtual space, the second user can easily understand that the first user is not wearing the first HMD by visually confirming that the facial expression of the first avatar is a predetermined expression.

[0016] In particular, when the first user is not wearing the first HMD, the facial expression of the first avatar is not updated at all. Therefore, if the second user does not know that the first user is not wearing the first HMD, the second user may feel uncomfortable with the first user. In this way, by making the second user aware that the first user is not wearing the first HMD, it is possible to avoid a situation in which the second user feels uncomfortable with the first user who shows no reaction.

[0017] (4) The information processing method according to item (3), wherein the first aspect is a facial expression of the first avatar that is set as a default.

[0018] According to the above method, when non-wearing information indicating that the first user is not wearing the first head mounted device (first HMD) is received, the facial expression of the first avatar becomes the default facial expression of the first avatar. In this way, when the second user is communicating with the first user in the virtual space, the second user can easily understand that the first user is not wearing the first HMD by visually confirming that the facial expression of the first avatar has become the default facial expression.

[0019] (5) The information processing method described in item (3), wherein the first manner is a facial expression preselected as an manner indicating that the first user is not wearing the first head mounted device.

[0020] According to the above method, when non-wearing information indicating that the first user is not wearing the first head-mounted device (first HMD) is received, the facial expression of the first avatar is set to an expression (hereinafter, referred to as a selected expression) previously selected by the first user as a mode indicating that the first user is not wearing the first head-mounted device. In this way, when the second user is communicating with the first user in the virtual space, the second user can easily understand that the first user is not wearing the first HMD by visually confirming that the facial expression of the first avatar has become the selected expression.

[0021] (6) the first user terminal further includes a wearing sensor configured to detect whether the first user is wearing the first head-mounted device; The information processing method according to any one of items (1) to (5), wherein, when the wearing sensor detects that the first user is not wearing the first head-mounted device, the first user terminal outputs the non-wearing information, and the second user terminal receives the output non-wearing information from the first user terminal.

[0022] According to the above method, it is determined that the first user is not wearing the first head mounted device (first HMD) according to information transmitted from the wearing sensor of the first user terminal. Then, non-wearing information indicating that the first user is not wearing the first HMD is transmitted. In this way, it is possible to automatically determine that the first user is not wearing the first HMD according to the information output from the wearing sensor of the first user terminal.

[0023] (7)(f) An information processing method described in any one of items (1) to (6), further comprising a step of, when receiving wearing information indicating that the first head-mounted device has been removed from the first user and then put back on, indicating to the second user by the second user terminal that the first user is wearing the first head-mounted device.

[0024] According to the above method, when wearing information indicating that the first head mounted device (first HMD) is removed from the first user and then worn again is received, the second user terminal presents to the second user that the first user is wearing the first HMD. In this way, the second user can easily understand that the first user has worn the first HMD again. Therefore, a rich virtual experience can be provided to the user.

[0025] (8) A program for causing a computer to execute the information processing method according to any one of items (1) to (7).

[0026] The above program makes it possible to provide a user with a rich virtual experience.

[0027] (9) A virtual space distribution system comprising a first user terminal having a first head-mounted device worn on the head of a first user, and a second user terminal having a second head-mounted device worn on the head of a second user, and configured to execute the information processing method described in any one of items (1) to (7).

[0028] According to the above virtual space distribution system, a rich virtual experience can be provided to the user.

[0029] (10) a processor; and a memory storing computer readable instructions, When the computer-readable instructions are executed by the processor, the device performs the information processing method described in any one of items (1) to (7).

[0030] The above device can provide a rich virtual experience to the user.

[0031] (11) A computer for providing a user with a virtual experience, moving an operation object in a virtual space in response to a movement of a part of the user's body; executing a first action by the operation object on a virtual object whose movement in the virtual space is controllable based on a movement of another user different from the user; and executing a second action on the other user based on the execution of the first action.

[0032] According to the program, it is possible to improve the virtual experience of the user. In particular, by performing actions between multiple users who share a virtual space through intuitive operations using operation objects, communication in the virtual space can be smoothly performed without losing the sense of immersion.

[0033] (12) The program according to item (11), further comprising: executing the second action when the first action is executed in a state in which the virtual object is not being controlled by the other user.

[0034] According to the above program, by executing the second action on the virtual object in an uncontrolled state, it is possible to attract other users into the virtual space.

[0035] (13) The program according to item (11) or (12), wherein the first action includes an action of contacting the virtual object during a repetitive motion of the operation object.

[0036] (14) The program according to item (13), wherein the first action includes a contact action with the virtual object during the repetitive motion at an acceleration equal to or greater than a certain value.

[0037] (15) The program according to item (13) or (14), wherein the first action includes an action of passing through relative coordinates associated with the virtual object a predetermined number of times or more within a certain period of time.

[0038] According to these programs, erroneous detection of the motion of the control object can be prevented by not taking into consideration the case where the control object accidentally comes into contact with the virtual object.

[0039] (16) The program according to any one of items (11) to (15), wherein the second action includes a notification encouraging the other user to join the virtual space.

[0040] According to the above program, it is possible to easily let other users sharing the virtual space know that, for example, a multiplayer game has been proposed, thereby promoting the use of multiplayer games.

[0041] (17) transitioning the virtual object from a first state to a second state when a reaction to the second action is performed; Item (16) is a program for causing the computer to execute a step of transitioning the virtual object to a third state if the virtual object is controlled by the other user within a certain period of time after the transition to the second state.

[0042] According to the above program, the user who performed the first action can easily know that there has been a reaction from another user, or that the virtual object has been controlled by another user.

[0043] (18) The program according to item (17), further causing the computer to execute a step of returning the virtual object to the first state if the virtual object is not controlled by the other user within a certain period of time after the transition to the second state.

[0044] According to the above program, the user who performed the first action can easily understand that the virtual object is not in a state where it is controlled by another user despite the execution of the second action.

[0045] (19) The program according to any one of items (16) to (18), wherein the content of the notification varies depending on at least one of the strength of the motion of the first action on the virtual object and the type of the first action.

[0046] According to the above method, other users can grasp the level of desire for multiplayer from the user who performed the first action, and can decide whether or not to join the virtual space depending on the level of desire.

[0047] (20) executing the first action with the operation object on another virtual object whose action in the virtual space is computer-controllable; A program described in any one of items (11) to (19), for causing the computer to further execute the step of transitioning a game provided in the virtual space from a first scene to a second scene based on the execution of the first action.

[0048] According to the above program, the user can progress through the game without losing the sense of immersion in the virtual space.

[0049] (21) An information processing device for providing a user with a virtual experience, comprising: Under the control of a processor included in the information processing device, moving an operation object in a virtual space in response to a movement of a part of the user's body; executing a first action by the operation object on a virtual object whose action in the virtual space is controllable based on a movement of another user different from the user; and executing a second action on the other user based on the execution of the first action.

[0050] According to the information processing device, the virtual experience of the user can be improved.

[0051] (22) An information processing system for providing a user with a virtual experience, comprising: Under the control of a plurality of processors included in the information processing system, moving an operation object in a virtual space in response to a movement of a part of the user's body; executing a first action by the operation object on a virtual object whose action in the virtual space is controllable based on a movement of another user different from the user; and executing a second action on the other user based on the execution of the first action.

[0052] According to the above information processing system, the virtual experience of the user can be improved.

[0053] (23) An information processing method implemented by a computer for providing a user with a virtual experience, comprising: moving an operation object in a virtual space in response to a movement of a part of the user's body; executing a first action by the operation object on a virtual object whose movement in the virtual space is controllable based on a movement of another user different from the user; and executing a second action on the other user based on the execution of the first action.

[0054] According to the above information processing method, the user's virtual experience can be improved.

[0055] (24) A computer for providing a user with a virtual experience, performing a first action in a virtual space; executing a second action on a user different from the user based on the execution of the first action; and executing a third action in the virtual space based on a reaction to the second action.

[0056] According to the program, the virtual experience of the user can be improved. In particular, by performing a third action in a virtual space based on a reaction to a second action performed on another user, a new virtual experience can be provided in which the boundary between the real space and the virtual space is seamless.

[0057] (25) The program described in item (24), wherein the first action is an action for identifying the other user, and the third action is an action executed by a virtual object associated with the other user in the virtual space.

[0058] According to the above program, actions are executed between a plurality of users who share a virtual space, thereby enabling smooth communication in the virtual space.

[0059] (26) causing the computer to further execute a step of moving an operation object in the virtual space in response to a movement of a part of the body of the user; The program according to item (24) or (25), wherein the first action is executed using the operation object.

[0060] According to the above program, a plurality of users sharing a virtual space can execute actions through intuitive operations using operation objects, without losing the sense of immersion of the users.

[0061] (27) The program according to any one of items (24) to (26), wherein the second action includes a notification encouraging the other user to join the virtual space.

[0062] According to the above program, it is possible to easily let other users sharing the virtual space know that, for example, a multiplayer game has been proposed, thereby promoting the use of multiplayer games.

[0063] (28) The program described in any one of items (24) to (27), wherein the third action includes transitioning an action performed by a virtual object associated with the other user in the virtual space from a first state to a second state.

[0064] According to the above method, the user who performed the first action can easily know whether or not there has been a reaction from other users.

[0065] (29) An information processing device for providing a user with a virtual experience, comprising: Under the control of a processor included in the information processing device, performing a first action in a virtual space; executing a second action on a user different from the user based on the execution of the first action; and executing a third action in the virtual space based on a reaction to the second action.

[0066] According to the information processing device, the virtual experience of the user can be improved.

[0067] (30) An information processing system for providing a user with a virtual experience, comprising: Under the control of a plurality of processors included in the information processing system, performing a first action in a virtual space; executing a second action on a user different from the user based on the execution of the first action; and executing a third action in the virtual space based on a reaction to the second action.

[0068] According to the above information processing system, the virtual experience of the user can be improved.

[0069] (31) An information processing method implemented by a computer for providing a user with a virtual experience, comprising: performing a first action in a virtual space; executing a second action on a user different from the user based on the execution of the first action; and executing a third action in the virtual space based on a reaction to the second action.

[0070] According to the above information processing method, the user's virtual experience can be improved.

[0071] [Details of the embodiment shown in this disclosure] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. For the sake of convenience, the description of elements having the same reference numbers as elements already described in the description of the present embodiment will not be repeated.

[0072] First, an outline of the configuration of the virtual space delivery system 100 will be described with reference to FIG. 1. FIG. 1 is a schematic diagram of the virtual space delivery system 100 (hereinafter, simply referred to as the delivery system 100). As shown in FIG. 1, the delivery system 100 includes a user terminal 1A (first user terminal) operated by a user A (first user) wearing a head mounted device (HMD) 110 (first HMD), a user terminal 1B (second user terminal) operated by a user B (second user) wearing an HMD 110 (second HMD), and a server 2. The user terminals 1A and 1B are communicatively connected to the server 2 via a communication network 3 such as the Internet. In this embodiment, the virtual space includes a VR (Virtual Reality) space. In addition, hereinafter, for convenience of explanation, the user terminals 1A and 1B may be collectively referred to simply as a user terminal 1. Furthermore, the users A and B may be collectively referred to simply as a user U. In this embodiment, the user terminals 1A and 1B have the same configuration.

[0073] Next, the configuration of the user terminal 1 will be described with reference to Fig. 2. Fig. 2 is a schematic diagram showing the user terminal 1. As shown in Fig. 2, the user terminal 1 includes a head mounted device (HMD) 110 worn on the head of a user U, headphones 116, a face camera 117, a microphone 118, a position sensor 130, an external controller 320, and a control device 120.

[0074] The HMD 110 includes a display unit 112, an HMD sensor 114, a gaze sensor 140, a face camera 113, and a wearing sensor 115. The display unit 112 includes a non-transmissive display device configured to completely cover the field of view (visual field) of a user U wearing the HMD 110. This allows the user U to be immersed in a virtual space by only viewing a visual field image displayed on the display unit 112. The display unit 112 may include a left-eye display unit configured to provide an image to the left eye of the user U, and a right-eye display unit configured to provide an image to the right eye of the user U. The HMD 110 may also include a transmissive display device. In this case, the transmissive display device may be temporarily configured as a non-transmissive display device by adjusting its transmittance.

[0075] The HMD sensor 114 is mounted near the display unit 112 of the HMD 110. The HMD sensor 114 includes at least one of a geomagnetic sensor, an acceleration sensor, and a tilt sensor (angular velocity sensor, gyro sensor, etc.), and can detect various movements (tilt, etc.) of the HMD 110 worn on the head of the user U.

[0076] The gaze sensor 140 has an eye tracking function that detects the line of sight of the user U. The gaze sensor 140 may include, for example, a right-eye gaze sensor and a left-eye gaze sensor. The right-eye gaze sensor may obtain information regarding the rotation angle of the right eye's eyeball by irradiating, for example, infrared light onto the right eye of the user U and detecting the reflected light reflected from the right eye (particularly, the cornea and the iris). On the other hand, the left-eye gaze sensor may obtain information regarding the rotation angle of the left eye's eyeball by irradiating, for example, infrared light onto the left eye of the user U and detecting the reflected light reflected from the left eye (particularly, the cornea and the iris).

[0077] The face camera 113 is configured to acquire images (particularly, moving images) showing the eyes (left eye and right eye) and eyebrows (left eyebrow and right eyebrow) of the user U when the HMD 110 is worn by the user U. The face camera 113 is disposed at a predetermined position inside the HMD 110 so as to be able to capture images of the eyes and eyebrows of the user U. The frame rate of the moving images acquired by the face camera 113 may be higher than the frame rate of a visual field image (described later) displayed on the HMD 110.

[0078] The wearing sensor 115 is configured to generate event information for indicating whether the user U is wearing the HMD 110. Specifically, when the user U removes the HMD 110 (when the state of the HMD 110 transitions from a wearing state to a non-wearing state), the wearing sensor 115 generates event information and transmits the event information to the control device 120. Also, when the user U wears the HMD 110 (when the state of the HMD 110 transitions from a non-wearing state to a wearing state), the wearing sensor 115 generates event information and transmits the event information to the control device 120. Also, the configuration of the wearing sensor 115 is not particularly limited. For example, it is assumed that the elastic force of a spring provided in the HMD 110 changes when the user U wears the HMD 110. In this case, the wearing sensor 115 may generate event information based on the change in the elastic force of the spring. Furthermore, it is assumed that when the user U wears the HMD 110, a current flows between two pads that are provided on the HMD 110 and contact a part of the body (the nose) and that change. In this case, the wearing sensor 115 may generate event information based on the change in the current that flows between the two pads.

[0079] The headphones 116 (audio output unit) are respectively worn on the left and right ears of the user U. The headphones 116 are configured to receive audio data (electrical signals) from the control device 120 and output audio based on the received audio data. The microphone 118 (audio input unit) is configured to collect audio uttered by the user U and generate audio data (electrical signals) based on the collected audio. Furthermore, the microphone 118 is configured to transmit the audio data to the control device 120.

[0080] The face camera 117 is configured to acquire an image (particularly, a moving image) showing the mouth of the user U and its surroundings. The face camera 117 may be disposed at a position facing the mouth of the user U so as to be able to capture an image of the mouth of the user U and its surroundings. The face camera 117 may also be connected to the HMD 110. The frame rate of the moving image acquired by the face camera 117 may be higher than the frame rate of the field of view image displayed on the HMD 110.

[0081] The position sensor 130 is, for example, configured by a position tracking camera, and configured to detect the positions of the HMD 110 and the external controller 320. The position sensor 130 is communicably connected to the control device 120 wirelessly or by wire, and configured to detect information on the positions, inclination, or emission intensity of a plurality of detection points (not shown) provided on the HMD 110. Furthermore, the position sensor 130 is configured to detect information on the positions, inclination, and / or emission intensity of a plurality of detection points (not shown) provided on the external controller 320. The detection points are, for example, light-emitting units that emit infrared rays or visible light. The position sensor 130 may also include an infrared sensor or a plurality of optical cameras.

[0082] The external controller 320 is used to control the movement of the hands of an avatar displayed in a virtual space by detecting the movement of a part of the user U's body (a part other than the head, which is the hand of the user U in this embodiment). The external controller 320 has a right-hand external controller 320R (hereinafter simply referred to as the controller 320R) operated by the right hand of the user U, and a left-hand external controller 320L (hereinafter simply referred to as the controller 320L) operated by the left hand of the user U. The controller 320R is a device that indicates the position of the user U's right hand and the movement of the fingers of the right hand. The right hand of the avatar present in the virtual space moves in response to the movement of the controller 320R. The controller 320L is a device that indicates the position of the user U's left hand and the movement of the fingers of the left hand. The left hand of the avatar present in the virtual space moves in response to the movement of the controller 320L.

[0083] The control device 120 is a computer configured to control the HMD 110. The control device 120 identifies position information of the HMD 110 based on information acquired from the position sensor 130, and can accurately associate the position of the virtual camera in the virtual space (the position of the avatar) with the position of the user U wearing the HMD 110 in the real space based on the identified position information. Furthermore, the control device 120 identifies the motion of the external controller 320 based on information acquired from the position sensor 130 and / or a sensor built in the external controller 320, and can accurately associate the motion of the hand of the avatar displayed in the virtual space with the motion of the external controller 320 in the real space based on the identified motion of the external controller 320. In particular, the control device 120 identifies the motion of the controller 320L based on information acquired from the position sensor 130 and / or a sensor built in the controller 320L, and can accurately associate the motion of the left hand of the avatar displayed in the virtual space with the motion of the controller 320L in the real space (the motion of the left hand of the user U) based on the identified motion of the controller 320L. Similarly, the control device 120 can identify the movement of the controller 320R based on information obtained from the position sensor and / or the sensor built into the controller 320R, and based on the identified movement of the controller 320R, accurately match the movement of the right hand of the avatar displayed in the virtual space with the movement of the controller 320R in the real space (the movement of the right hand of the user U).

[0084] Furthermore, the control unit 121 can associate the facial expression (state) of the user U with the facial expression (state) of the avatar of the user U based on the moving images acquired by the face cameras 113 and 117. The process of associating the facial expression of the user U with the facial expression of the avatar will be described in detail later.

[0085] Furthermore, the control device 120 can identify the gaze of the right eye and the gaze of the left eye of the user U based on information transmitted from the gaze sensor 140 (the left eye gaze sensor and the right eye gaze sensor) and identify a gaze point that is an intersection of the gaze of the right eye and the gaze of the left eye. Furthermore, the control device 120 can identify the gaze of both eyes of the user U (the gaze of the user U) based on the identified gaze point. Here, the gaze of the user U is the gaze of both eyes of the user U, and coincides with the direction of a straight line passing through the gaze point and the midpoint of the line segment connecting the right eye and the left eye of the user U.

[0086] Next, a method for acquiring information on the position and tilt of the HMD 110 will be described with reference to FIG. 3. FIG. 3 is a diagram showing the head of a user U wearing the HMD 110. Information on the position and tilt of the HMD 110 linked to the movement of the head of the user U wearing the HMD 110 can be detected by the position sensor 130 and / or the HMD sensor 114 mounted on the HMD 110. As shown in FIG. 2, three-dimensional coordinates (uvw coordinates) are defined with the head of the user U wearing the HMD 110 as the center. The vertical direction in which the user U stands upright is defined as the v axis, the direction perpendicular to the v axis and passing through the center of the HMD 110 is defined as the w axis, and the direction perpendicular to the v axis and the w axis is defined as the u axis. The position sensor 130 and / or the HMD sensor 114 detects angles around each uvw axis (i.e., tilt determined by the yaw angle indicating rotation around the v axis, the pitch angle indicating rotation around the u axis, and the roll angle indicating rotation around the w axis). The control device 120 determines angle information for controlling the visual axis of the virtual camera based on the detected angle changes around each of the uvw axes.

[0087] Next, the hardware configuration of the control device 120 will be described with reference to Fig. 4. Fig. 4 is a diagram showing the hardware configuration of the control device 120. As shown in Fig. 4, the control device 120 includes a control unit 121, a storage unit 123, an I / O (input / output) interface 124, a communication interface 125, and a bus 126. The control unit 121, the storage unit 123, the I / O interface 124, and the communication interface 125 are connected to each other via the bus 126 so as to be able to communicate with each other.

[0088] The control device 120 may be configured as a personal computer, a smartphone, a phablet, a tablet, or a wearable device separate from the HMD 110, or may be built into the HMD 110. In addition, some of the functions of the control device 120 may be mounted on the HMD 110, and the remaining functions of the control device 120 may be mounted on another device separate from the HMD 110.

[0089] The control unit 121 includes a memory and a processor. The memory is, for example, a ROM (Read Only Memory) in which various programs and the like are stored, a RAM (Random Access Memory) having multiple work areas in which various programs and the like executed by the processor are stored, and the like. The processor is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and / or a GPU (Graphics Processing Unit), and is configured to load a program designated from various programs incorporated in the ROM onto the RAM and execute various processes in cooperation with the RAM.

[0090] In particular, the processor may load a control program on the RAM and execute the control program in cooperation with the RAM, so that the control unit 121 controls various operations of the control device 120. The control unit 121 displays a visual field image on the display unit 112 of the HMD 110 based on the visual field image data. This allows the user U to be immersed in the virtual space.

[0091] The memory unit (storage) 123 is, for example, a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a USB flash memory, and is configured to store programs and various data. The memory unit 123 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. The memory unit 123 may also store an authentication program for a user U and data related to various images and objects (e.g., avatars, etc.). Furthermore, a database including a table for managing various data may be constructed in the memory unit 123.

[0092] The I / O interface 124 is configured to communicatively connect the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118 to the control device 120, and is configured, for example, by a Universal Serial Bus (USB) terminal, a Digital Visual Interface (DVI) terminal, a High-Definition Multimedia Interface (HDMI) (registered trademark) terminal, etc. The control device 120 may be wirelessly connected to the HMD 110, the position sensor 130, the external controller 320, the headphones 116, the face camera 117, and the microphone 118.

[0093] The communication interface 125 is configured to connect the control device 120 to a communication network 3 such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The communication interface 125 includes various wired connection terminals for communicating with external devices such as the server 2 via the communication network 3 and various processing circuits for wireless connection, and is configured to comply with a communication standard for communicating via the communication network 3.

[0094] Next, a process for displaying a visual field image on the HMD 110 will be described with reference to Figs. 5 to 8. Fig. 5 is a flowchart showing a process for displaying a visual field image on the HMD 110. Fig. 6 is an xyz space diagram showing an example of a virtual space 200. State (a) of Fig. 7 is a yx plan view of the virtual space 200 shown in Fig. 6. State (b) of Fig. 7 is a zx plan view of the virtual space 200 shown in Fig. 6. Fig. 8 is a diagram showing an example of a visual field image V displayed on the HMD 110.

[0095] As shown in FIG. 5, in step S1, the control unit 121 (see FIG. 4) generates virtual space data indicating the virtual space 200 including the virtual camera 300 and various objects. As shown in FIG. 6, the virtual space 200 is defined as a celestial sphere centered on the central position 210 (only the upper half of the celestial sphere is illustrated in FIG. 6). In addition, an xyz coordinate system with the central position 210 as the origin is set in the virtual space 200. The virtual camera 300 defines a visual axis L for identifying a visual field image V (see FIG. 8) displayed on the HMD 110. The uvw coordinate system defining the visual field of the virtual camera 300 is determined so as to be linked to the uvw coordinate system defined with the head of the user U in the real space as the center. In addition, the control unit 121 may move the virtual camera 300 in the virtual space 200 in conjunction with the movement in the real space of the user U wearing the HMD 110.

[0096] Next, in step S2, the control unit 121 specifies the field of view CV (see FIG. 7) of the virtual camera 300. Specifically, the control unit 121 acquires information about the position and inclination of the HMD 110 based on data indicating the state of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. Next, the control unit 121 specifies the position and orientation of the virtual camera 300 in the virtual space 200 based on the information about the position and inclination of the HMD 110. Next, the control unit 121 determines the visual axis L of the virtual camera 300 from the position and orientation of the virtual camera 300, and specifies the field of view CV of the virtual camera 300 from the determined visual axis L. Here, the field of view CV of the virtual camera 300 corresponds to a partial area of ​​the virtual space 200 that is visible to the user U wearing the HMD 110 (in other words, corresponds to a partial area of ​​the virtual space 200 displayed on the HMD 110). The field of view CV has a first area CVa set as an angular range of a polar angle α centered on the visual axis L in the xy plane shown in state (a) of Fig. 7, and a second area CVb set as an angular range of an azimuth angle β centered on the visual axis L in the xz plane shown in state (b) of Fig. 7. The control unit 121 may specify the line of sight of the user U based on data indicating the line of sight of the user U transmitted from the gaze sensor 140, and may determine the direction of the virtual camera 300 (the visual axis L of the virtual camera) based on the specified line of sight of the user U and information regarding the position and inclination of the HMD 110. As will be described later, the control unit 121 may determine the face direction of the avatar of the user U based on information regarding the position and inclination of the HMD 110.

[0097] In this way, the control unit 121 can specify the field of view CV of the virtual camera 300 based on data from the position sensor 130 and / or the HMD sensor 114. Here, when the user U wearing the HMD 110 moves, the control unit 121 can update the field of view CV of the virtual camera 300 based on data indicating the movement of the HMD 110 transmitted from the position sensor 130 and / or the HMD sensor 114. That is, the control unit 121 can update the field of view CV according to the movement of the HMD 110. Similarly, when the line of sight of the user U changes, the control unit 121 may update the field of view CV of the virtual camera 300 based on data indicating the line of sight of the user U transmitted from the gaze sensor 140. That is, the control unit 121 may change the field of view CV according to the change in the line of sight of the user U.

[0098] Next, in step S3, the control unit 121 generates visual field image data indicating the visual field image V to be displayed on the display unit 112 of the HMD 110. Specifically, the control unit 121 generates the visual field image data based on the virtual space data defining the virtual space 200 and the visual field CV of the virtual camera 300.

[0099] Next, in step S4, the control unit 121 displays the visual field image V on the display unit 112 of the HMD 110 based on the visual field image data (see FIG. 7). In this way, the visual field CV of the virtual camera 300 changes in response to the movement of the user U wearing the HMD 110, and the visual field image V displayed on the display unit 112 of the HMD 110 changes, so that the user U can be immersed in the virtual space 200.

[0100] The virtual camera 300 may include a left-eye virtual camera and a right-eye virtual camera. In this case, the control unit 121 generates left-eye field of view image data indicating a field of view image for the left eye based on the virtual space data and the field of view of the left-eye virtual camera. Furthermore, the control unit 121 generates right-eye field of view image data indicating a field of view image for the right eye based on the virtual space data and the field of view of the right-eye virtual camera. After that, the control unit 121 displays a left-eye field of view image on the left-eye display unit based on the left-eye field of view image data, and displays a right-eye field of view image on the right-eye display unit based on the right-eye field of view image data. In this way, the user U can visually recognize the field of view image three-dimensionally due to the parallax between the left-eye field of view image and the right-eye field of view image. The virtual cameras may be arranged at the eye position of an avatar operated by a user, as described later. For example, the left-eye virtual camera may be arranged at the left eye of the avatar, while the right-eye virtual camera may be arranged at the right eye of the avatar.

[0101] 5 may be executed for each frame (a still image constituting a moving image). For example, when the frame rate of a moving image is 90 fps, the processes of steps S1 to S4 may be executed repeatedly at intervals of ΔT=1 / 90 (seconds). In this manner, the processes of steps S1 to S4 are executed repeatedly at predetermined intervals, so that the field of view of virtual camera 300 is updated in response to the operation of HMD 110, and the field of view image V displayed on display unit 112 of HMD 110 is updated.

[0102] Next, the hardware configuration of the server 2 shown in Fig. 1 will be described with reference to Fig. 9. Fig. 9 is a diagram showing the hardware configuration of the server 2. As shown in Fig. 9, the server 2 includes a control unit 23, a storage unit 22, a communication interface 21, and a bus 24. The control unit 23, the storage unit 22, and the communication interface 21 are connected to each other via the bus 24 so as to be able to communicate with each other. The control unit 23 includes a memory and a processor, and the memory is composed of, for example, a ROM, a RAM, etc., and the processor is composed of, for example, a CPU, an MPU, and / or a GPU.

[0103] The memory unit (storage) 22 is, for example, a large-capacity HDD. The memory unit 22 may store a control program for causing a computer to execute at least a part of the information processing method according to the present embodiment, or a control program for realizing sharing of a virtual space by a plurality of users. The memory unit 22 may also store user management information for managing each user, and data related to various images and objects (e.g., avatars, etc.). The communication interface 21 is configured to connect the server 2 to the communication network 3.

[0104] Next, an example of a process for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B will be described with reference to Figs. 1, 10, and 11. State (a) of Fig. 10 is a diagram showing a virtual space 200A provided to the user A. State (b) of Fig. 10 is a diagram showing a virtual space 200B provided to the user B. Fig. 11 is a sequence diagram for describing an example of a process for synchronizing the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B. In this description, as a prerequisite, as shown in Fig. 10, the avatar 4A (first avatar) associated with the user terminal 1A (user A) and the avatar 4B (second avatar) associated with the user terminal 1B (user B) share the same virtual space. In other words, it is assumed that the user A and the user B share one virtual space via the communication network 3.

[0105] As shown in state (a) of FIG. 10, the virtual space 200A of the user A includes an avatar 4A and an avatar 4B. The avatar 4A is operated by the user A and is linked to the movement of the user A. The avatar 4A has a left hand (a part of the virtual body of the avatar 4A) that is linked to the movement of the controller 320L of the user terminal 1A (movement of the left hand of the user A), a right hand that is linked to the movement of the controller 320R of the user terminal 1A (movement of the right hand of the user A), and a face whose expression is linked to the facial expression of the user A. The face of the avatar 4A has a plurality of facial parts (for example, eyes, eyebrows, mouth, etc.). The avatar 4B is operated by the user B and is linked to the movement of the user B. Avatar 4B has a left hand that is linked to the movement of controller 320L of user terminal 1B that indicates the movement of the left hand of user B, a right hand that is linked to the movement of controller 320R of user terminal 1B that indicates the movement of the right hand of user B, and a face whose expression is linked to the facial expression of user B. The face of avatar 4B has multiple facial parts (for example, eyes, eyebrows, mouth, etc.).

[0106] It is also assumed that the avatar 4A is not visualized in the virtual space 200A provided to the user A. In this case, the avatar 4A placed in the virtual space 200A includes at least a virtual camera 300 that moves in conjunction with the movement of the HMD 110 of the user terminal 1A.

[0107] The positions of the avatars 4A and 4B may be specified according to the position of the HMD 110 of the user terminals 1A and 1B. Similarly, the face directions of the avatars 4A and 4B may be specified according to the inclination of the HMD 110 of the user terminals 1A and 1B. Furthermore, the hand movements of the avatars 4A and 4B may be specified according to the operation of the external controller 320 of the user terminals 1A and 1B. In particular, the left hand movements of the avatars 4A and 4B may be specified according to the operation of the controller 320L of the user terminals 1A and 1B, and the right hand movements of the avatars 4A and 4B may be specified according to the operation of the controller 320R of the user terminals 1A and 1B. Furthermore, the facial expressions of the avatars 4A and 4B may be specified according to the facial expressions (states) of the users A and B.

[0108] Also, a virtual camera 300 (FIG. 6) may be disposed at each eye of the avatars 4A and 4B. In particular, a left-eye virtual camera may be disposed at the left eye of the avatars 4A and 4B, and a right-eye virtual camera may be disposed at the right eye of the avatars 4A and 4B. In the following description, it is assumed that the virtual cameras 300 are disposed at the eyes of the avatars 4A and 4B. For this reason, it is assumed that the field of view CV of the avatar 4A coincides with the field of view CV (see FIG. 7) of the virtual camera 300 disposed at the avatar 4A. Similarly, it is assumed that the field of view CV of the avatar 4B coincides with the field of view CV (see FIG. 7) of the virtual camera 300 disposed at the avatar 4B.

[0109] 10(b), the virtual space 200B of the user B includes an avatar 4A and an avatar 4B. The positions of the avatars 4A and 4B in the virtual space 200A may correspond to the positions of the avatars 4A and 4B in the virtual space 200B.

[0110] It is also assumed that the avatar 4B is not visualized in the virtual space 200B provided to the user B. In this case, the avatar 4B placed in the virtual space 200B includes at least a virtual camera 300 that moves in conjunction with the movement of the HMD 110 of the user terminal 1B.

[0111] 11, in step S10, the control unit 121 of the user terminal 1A generates voice data of the user A. For example, when the user A inputs voice into the microphone 118 (voice input unit) of the user terminal 1A, the microphone 118 generates voice data indicating the input voice. After that, the microphone 118 transmits the generated voice data to the control unit 121 via the I / O interface 124.

[0112] Next, in step S11, the control unit 121 of the user terminal 1A generates control information for the avatar 4A, and then transmits the generated control information for the avatar 4A and voice data indicating the voice of user A (voice data of user A) to the server 2. Thereafter, the control unit 23 of the server 2 receives the control information for the avatar 4A and the voice data of user A from the user terminal 1A (step S12).

[0113] Here, the control information of the avatar 4A is information necessary for controlling the movement of the avatar 4A. The control information of the avatar 4A may include information on the position of the avatar 4A (position information), information on the direction of the face of the avatar 4A (face direction information), information on the state of the hands (left hand and right hand) of the avatar 4A (hand information), and information on the facial expression of the avatar 4A (face information).

[0114] The facial information of the avatar 4A includes information indicating the states of a plurality of facial parts. The information indicating the states of the plurality of facial parts includes information (eye information) indicating the state of the eyes of the avatar 4A (such as the shape of the whites of the eyes and the position of the pupil relative to the whites of the eyes), information (eyebrow information) indicating the eyebrows of the avatar 4A (such as the position and shape of the eyebrows), and information (mouth information) indicating the state of the mouth of the avatar 4A (such as the position and shape of the mouth).

[0115] In detail, the eye information of the avatar 4A includes information indicating the state of the left eye of the avatar 4A and information indicating the state of the right eye of the avatar 4A. The eyebrow information of the avatar 4A includes information indicating the state of the left eyebrow of the avatar 4A and information indicating the state of the right eyebrow of the avatar 4A.

[0116] The control unit 121 of the user terminal 1A acquires an image showing the eyes (left and right eyes) and eyebrows (left and right eyebrows) of the user A from the face camera 113 mounted on the HMD 110, and then identifies the state of the eyes (left and right eyes) and eyebrows (left and right eyebrows) of the user A based on the acquired image and a predetermined image processing algorithm. Next, the control unit 121 generates information indicating the state of the eyes (left and right eyes) of the avatar 4A and information indicating the state of the eyebrows (left and right eyebrows) of the avatar 4A based on the identified state of the eyes and eyebrows of the user A.

[0117] Similarly, the control unit 121 of the user terminal 1A acquires an image showing the mouth and its surroundings of the user A from the face camera 117, and identifies the state of the mouth of the user A based on the acquired image and a predetermined image processing algorithm. Next, the control unit 121 generates information indicating the state of the mouth of the avatar 4A based on the identified state of the mouth of the user A.

[0118] In this way, the control unit 121 of the user terminal 1A can generate eye information of the avatar 4A corresponding to the eyes of the user A, eyebrow information of the avatar 4A corresponding to the eyebrows of the user A, and mouth information of the avatar 4A corresponding to the mouth of the user A.

[0119] Furthermore, the control unit 121 may specify the facial expression (e.g., a smile) of the user A from among a plurality of types of facial expressions (e.g., a smile, a sad expression, a neutral expression, an angry expression, a surprised expression, a troubled expression, etc.) stored in the storage unit 123 or memory based on the image captured by the face camera 113, the image captured by the face camera 117, and a predetermined image processing algorithm. Thereafter, the control unit 121 can generate face information indicating the facial expression (e.g., a smile) of the avatar 4A based on the specified facial expression of the user A. In this case, the facial expression data including the plurality of types of facial expressions of the avatar 4A and the facial information of the plurality of avatars 4A, each of which is associated with one of the plurality of types of facial expressions, may be stored in the storage unit 123.

[0120] For example, when the control unit 121 identifies the facial expression of the avatar 4A as a smile, it acquires face information indicating the smile of the avatar 4A based on the facial expression (smile) and expression data of the avatar 4A. Here, the face information indicating the smile of the avatar 4A includes eye information, eyebrow information, and mouth information of the avatar 4A when the facial expression of the avatar 4A is a smile.

[0121] Next, in step S13, the control unit 121 of the user terminal 1B generates control information for the avatar 4B and transmits the generated control information for the avatar 4B to the server 2. Thereafter, the control unit 23 of the server 2 receives the control information for the avatar 4B from the user terminal 1B (step S14).

[0122] Here, the control information of the avatar 4B is information necessary for controlling the movement of the avatar 4B. The control information of the avatar 4B may include information on the position of the avatar 4B (position information), information on the direction of the face of the avatar 4B (face direction information), information on the state of the hands (left hand and right hand) of the avatar 4B (hand information), and information on the facial expression of the avatar 4B (face information).

[0123] The facial information of avatar 4B includes information indicating the states of a plurality of facial features. The information indicating the states of a plurality of facial features includes information (eye information) indicating the state of avatar 4B's eyes (such as the shape of the whites of the eyes and the position of the pupil relative to the whites of the eyes), information (eyebrow information) indicating the position and shape of the eyebrows, and information (mouth information) indicating the state of avatar 4B's mouth (such as the position and shape of the mouth). The method of acquiring the facial information of avatar 4B is the same as the method of acquiring the facial information of avatar 4A.

[0124] Next, the server 2 transmits the control information of the avatar 4B to the user terminal 1A (step S15), while transmitting the control information of the avatar 4A and the voice data of the user A to the user terminal 1B (step S19). After that, the control unit 121 of the user terminal 1A receives the control information of the avatar 4B in step S16, and then updates the states of the avatars 4A and 4B based on the control information of the avatars 4A and 4B, and updates the virtual space data indicating the virtual space 200A (see state (a) in FIG. 10) (step S17).

[0125] In detail, the control unit 121 of the user terminal 1A updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the facial orientation of the avatars 4A and 4B based on the facial orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the facial information of the avatars 4A and 4B. In this manner, the virtual space data indicating the virtual space 200A including the updated avatars 4A and 4B is updated.

[0126] Then, the control unit 121 of the user terminal 1A determines the field of view CV of the avatar 4A (virtual camera 300) based on the position and inclination of the HMD 110, and updates the field of view image displayed on the HMD 110 based on the updated virtual space data and the field of view CV of the avatar 4A (step S18).

[0127] Meanwhile, after receiving the control information of avatar 4A and the voice data of user A in step S20, the control unit 121 of user terminal 1B updates the states of avatars 4A and 4B based on the control information of avatars 4A and 4B, and then updates the virtual space data indicating virtual space 200B (see state (b) in Figure 10) (step S21).

[0128] In detail, the control unit 121 of the user terminal 1B updates the positions of the avatars 4A and 4B based on the position information of the avatars 4A and 4B. The control unit 121 updates the facial orientation of the avatars 4A and 4B based on the facial orientation information of the avatars 4A and 4B. The control unit 121 updates the hands of the avatars 4A and 4B based on the hand information of the avatars 4A and 4B. The control unit 121 updates the facial expressions of the avatars 4A and 4B based on the facial information of the avatars 4A and 4B. In this way, the virtual space data indicating the virtual space including the updated avatars 4A and 4B is updated.

[0129] Then, the control unit 121 of the user terminal 1B determines the field of view CV of the avatar 4B (virtual camera 300) based on the position and inclination of the HMD 110, and updates the field of view image displayed on the HMD 110 based on the updated virtual space data and the field of view CV of the avatar 4B (step S22).

[0130] Thereafter, the control unit 121 of the user terminal 1B processes the voice data of the user A based on the received voice data of the user A, information on the position of the avatar 4A included in the control information of the avatar 4A, information on the position of the avatar 4B, and a predetermined voice processing algorithm. Thereafter, the control unit 121 transmits the processed voice data to the headphones 116 (voice output unit), and the headphones 116 outputs the voice of the user A based on the processed voice data (step S23). In this way, a voice chat (VR chat) between users (avatars) can be realized in the virtual space.

[0131] In this embodiment, after the user terminals 1A and 1B transmit the control information of the avatar 4A and the control information of the avatar 4B to the server 2, the server 2 transmits the control information of the avatar 4A to the user terminal 1B, and transmits the control information of the avatar 4B to the user terminal 1A. In this way, it is possible to synchronize the movements of the avatars 4A and 4B between the user terminal 1A and the user terminal 1B.

[0132] Next, the information processing method according to the present embodiment when the HMD 110 of the user A is removed will be described mainly with reference to Fig. 12 and Fig. 14. Fig. 12 is an example of a flowchart for explaining the information processing method according to the present embodiment when the HMD 110 is removed by the user A. Fig. 14 is a diagram showing an example of a virtual space 200B provided to the user B for explaining the information processing method according to the present embodiment. Note that in the flowchart shown in Fig. 12, for convenience of explanation, the process in which the control unit 121 of the user terminal 1A transmits the control information of the avatar 4A to the server 2 and the process in which the control unit 121 of the user terminal 1B transmits the control information of the avatar 4B to the server 2 are omitted.

[0133] In this embodiment, it is assumed as a prerequisite that an avatar 4A and an avatar 4B share the same virtual space as shown in Fig. 14. In other words, it is assumed that users A and B share one virtual space via a communication network 3.

[0134] The virtual space 200B of the user B includes an avatar 4A and an avatar 4B. The control unit 121 of the user terminal 1B updates the virtual space data indicating the virtual space 200B. For convenience of explanation, in this description, the virtual space 200A provided to the avatar 4A is omitted from illustration.

[0135] 12, in step S30, the control unit 121 of the user terminal 1A determines whether the user A has removed the HMD 110. In this regard, the control unit 121 may determine whether the user A has removed the HMD 110 based on tilt information of the HMD 110 transmitted from the HMD sensor 114 or event information transmitted from the wearing sensor 115. The HMD 114 and the wearing sensor 115 function as sensors configured to detect whether the HMD 110 is being worn.

[0136] With reference to state (a) of FIG. 13, a process of determining whether or not the user A has removed the HMD 110 using the HMD sensor 114 will be described. State (a) of FIG. 13 is an example of a flowchart for describing a process for determining whether or not the user A has removed the HMD 110 using the HMD sensor 114. As shown in state (a) of FIG. 13, the control unit 121 receives tilt information indicating the tilt (roll angle, yaw angle, pitch angle) of the HMD 110 from the HMD sensor 114 (step S301). Next, the control unit 121 determines whether or not the tilt of the HMD 110 is larger than a predetermined tilt based on the received tilt information (S302). Here, the predetermined tilt may be appropriately set by the user. It is assumed that the tilt of the HMD 110 does not become larger than the predetermined tilt while the user A is wearing the HMD 110. That is, while the user A is wearing the HMD 110, the inclination of the HMD 110 is equal to or less than the predetermined inclination, whereas when the user A removes the HMD 110, the HMD 110 is significantly inclined, and the inclination of the HMD 110 may exceed the predetermined inclination. From this viewpoint, the predetermined inclination is set.

[0137] For example, when the control unit 121 determines that the pitch angle θ of the HMD 110 is greater than a predetermined angle θth (YES in step S302), as an example of the tilt of the HMD 110, it identifies that the user A has removed the HMD 110 (in other words, the user A is not wearing the HMD 110) and then executes the process of step S31. On the other hand, when the control unit 121 determines that the pitch angle θ of the HMD 110 is equal to or smaller than the predetermined angle θth (NO in step S302), it identifies that the user A is wearing the HMD 110 and waits until it receives tilt information of the HMD 110 from the HMD sensor 114.

[0138] Next, with reference to state (b) of FIG. 13, a process of determining whether or not the user A has removed the HMD 110 using the wearing sensor 115 will be described. State (b) of FIG. 13 is an example of a flowchart for describing a process for determining whether or not the user A has removed the HMD 110 using the wearing sensor 115. As shown in state (b) of FIG. 13, the control unit 121 determines whether or not event information has been received from the wearing sensor 115 (step S303). Here, the event information transmitted from the wearing sensor 115 is information indicating that the state of the HMD 110 of the user terminal 1A has changed between a wearing state and a non-wearing state. For example, when the user A removes the HMD 110 (when the state of the HMD 110 transitions from the wearing state to the non-wearing state), the wearing sensor 115 outputs the event information. Also, when the user A wears the HMD 110 (when the state of the HMD 110 transitions from the non-wearing state to the wearing state), the wearing sensor 115 outputs the event information.

[0139] When the control unit 121 determines that the event information has been received from the wearing sensor 115 (YES in step S303), it specifies that the user A has removed the HMD 110 (in other words, the user A is not wearing the HMD 110) and then executes the process of step S31. On the other hand, when the control unit 121 determines that the event information has not been received from the wearing sensor 115 (NO in step S303), it specifies that the user A is wearing the HMD 110 and then waits until the event information is received from the wearing sensor 115.

[0140] 12, when the control unit 121 determines that the user A has removed the HMD 110 (in other words, the user A is not wearing the HMD 110) (YES in step S30), the control unit 121 generates information indicating that the user A is not wearing the HMD 110 (non-wearing information) (step S31). After that, the control unit 121 transmits the non-wearing information to the server 2 via the communication network 3 (step S32).

[0141] Next, after receiving the non-wearing information from the user terminal 1A, the server 2 transmits the non-wearing information to the user terminal 1B via the communication network 3 (step S33). After that, the control unit 121 of the user terminal 1B sets the facial expression of the avatar 4A arranged in the virtual space 200B to a default facial expression (in other words, a facial expression set as a default) after receiving the non-wearing information from the server 2 (step S34). Here, the default facial expression of the avatar 4A refers to the facial expression of the avatar 4A in the initial state before starting to update the facial expression of the avatar 4A. When the default facial expression is a smile, the facial expression of the avatar 4A is set to a smile, as shown in FIG. 14. In addition, the control unit 121 updates the movement of the avatar 4B (including the facial expression of the avatar 4B) based on the control information of the avatar 4B.

[0142] In step S34, the control unit 121 may set the facial expression of the avatar 4A to an expression selected in advance by the user A as a mode indicating that the user A is not wearing the HMD 110. In this case, information regarding the facial expression of the avatar 4A selected by the user A may be transmitted from the user terminal 1A to the user terminal 1B and then stored in advance in the storage unit 123 of the user terminal 1B.

[0143] Next, the control unit 121 generates a speech bubble object 42A associated with the avatar 4A in the virtual space 200B, as shown in Fig. 14. The speech bubble object 42A displays information indicating that the user A is not wearing the HMD 110 (for example, "User A is away").

[0144] The control unit 121 may generate a telop object 43A in the virtual space 200B instead of the speech bubble object 42A (see FIG. 15). As shown in FIG. 15, the telop object 43A displays information indicating that the user A is not wearing the HMD 110 (for example, "User A is away"). The telop object 43A may be placed near the avatar 4A, or may be linked to the movement of the field of view CV of the avatar 4B so that the telop object 43A is always placed within the field of view CV of the avatar 4B. Furthermore, a telop indicating information indicating that the user A is not wearing the HMD 110 may be superimposed on the field of view image displayed on the HMD 110.

[0145] Thereafter, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the field of view CV of the avatar 4B in accordance with the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and displays the field of view image on the HMD 110 based on the updated field of view image data (step S36).

[0146] On the other hand, the control unit 121 of the user terminal 1A, after transmitting the non-wearing information to the server 2 via the communication network 3, transitions from the active mode to the sleep mode (step S37). In this case, it is preferable that the mode of the control unit 121 is the sleep mode, while the mode of the sensor (the wearing sensor 115 or the HMD sensor 114) is the active mode.

[0147] According to this embodiment, when user A is not wearing the HMD 110, the user terminal 1B presents to user B that user A is not wearing the HMD 110. In particular, according to this embodiment, when user A is not wearing the HMD 110, information indicating that user A is not wearing the HMD 110 is visualized on the field of view image displayed on the HMD 110 of the user terminal 1B. In this regard, the avatar 4A, whose facial expression is set to a default facial expression, and the speech bubble object 42A are visualized on the field of view image displayed on the HMD 110 of the user terminal 1B.

[0148] In this way, while communicating with user A in virtual space 200B, user B can easily understand that user A is not wearing HMD 110 by visually checking the default facial expression (e.g., a smile) of avatar 4A and the information displayed in balloon object 42A. Therefore, a rich virtual experience can be provided to the user.

[0149] In particular, user B may feel uncomfortable with user A not showing any reaction if he / she is not aware that user A is not wearing the HMD 110. In this way, by making user B aware that user A is not wearing the HMD 100, it is possible to avoid a situation in which user B feels uncomfortable with user A not showing any reaction (for example, the uncanny valley phenomenon).

[0150] Furthermore, according to this embodiment, it is determined that user A is not wearing the HMD 110 according to information (tilt information or event information) transmitted from a sensor (HMD sensor 114 or wearing sensor 115) of user terminal 1A. Then, non-wearing information indicating that user A is not wearing the HMD 110 is transmitted to server 2. In this way, it is possible to automatically determine that user A is not wearing the HMD 110 according to information output from the sensor of user terminal 1A.

[0151] In this embodiment, the user terminal 1A transmits the non-wearing information to the user terminal 1B via the server 2, but instead of the non-wearing information, the tilt information transmitted from the HMD sensor 114 may be transmitted to the user terminal 1B. In this case, the control unit 121 of the user terminal 1B may determine that the user A is not wearing the HMD 110 based on the received tilt information, and then execute the processes defined in steps S34 to S36.

[0152] Next, an information processing method according to this embodiment in a case where user A who has removed the HMD 110 puts the HMD 110 back on will be described with reference mainly to Fig. 16 and Fig. 17. Fig. 16 is an example of a flowchart for explaining the information processing method according to this embodiment in a case where user A puts the HMD 110 back on. Fig. 17 is a diagram showing an example of a virtual space 200B provided to user B for explaining the information processing method according to this embodiment.

[0153] 16, in step S40, the control unit 121 of the user terminal 1A determines whether the user A has worn the HMD 110 again. In this regard, the control unit 121 may determine whether the user A has worn the HMD 110 again based on the tilt information of the HMD 110 transmitted from the HMD sensor 114 or the event information transmitted from the wearing sensor 115. For example, when the control unit 121 determines that the event information has been received from the wearing sensor 115, it specifies that the user A has worn the HMD 110, and then executes the process of step S41. On the other hand, when the control unit 121 determines that the event information has not been received from the wearing sensor 115, it specifies that the user A has not worn the HMD 110, and then waits until the event information is received from the wearing sensor 115.

[0154] If the determination result in step S40 is YES, the control unit 121 transitions from the sleep mode to the active mode (step S41). Next, the control unit 121 that has transitioned to the active mode generates control information for the avatar 4A and generates wearing information indicating that the user A has worn the HMD 110 (step S42). After that, the control unit 121 transmits the control information and wearing information for the avatar 4A to the server 2 via the communication network 3 (step S43).

[0155] Next, after receiving the control information and wearing information of the avatar 4A from the user terminal 1A, the server 2 transmits the control information and wearing information of the avatar 4A to the user terminal 1B via the communication network 3 (step S44). Thereafter, the control unit 121 of the user terminal 1B updates the movement of the avatar 4A based on the control information of the avatar 4A, and updates the movement of the avatar 4B based on the control information of the avatar 4B (step S45). In particular, the control unit 121 updates the facial expression of the avatar 4A based on the facial information of the avatar 4A included in the control information of the avatar 4A, and updates the facial expression of the avatar 4B based on the facial information of the avatar 4B included in the control information of the avatar 4B.

[0156] Next, the control unit 121 generates a speech bubble object 44A associated with the avatar 4A in the virtual space 200B (step S46), as shown in Fig. 17. The speech bubble object 44A displays information indicating that the user A has put on the HMD 110 (for example, "User A has returned to his / her seat").

[0157] Thereafter, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the field of view CV of the avatar 4B in accordance with the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and displays the field of view image on the HMD 110 based on the updated field of view image data (step S47).

[0158] According to this embodiment, when user A removes the HMD 110 and then puts it back on, the user terminal 1B presents to user B that user A is wearing the HMD 110. In this way, user B can easily understand that user A has put the HMD 110 back on by visually checking the updated facial expression of the avatar 4A and the speech bubble object 44A. Therefore, a rich virtual experience can be provided to the users.

[0159] In the description of this embodiment, information indicating whether or not user A is wearing the HMD 110 is visualized on the visual field image, but this embodiment is not limited to this. For example, a voice guidance indicating that user A is not wearing the HMD 110 (e.g., "User A has left the seat") or a voice guidance indicating that user A is wearing the HMD 110 (e.g., "User A has returned to his / her seat") may be output from the headphones 116 (voice output unit) of the user terminal 1B. In this case, the control unit 121 of the user terminal 1A transmits voice guidance data together with non-wearing information (or wearing information) to the user terminal 1B via the server 2.

[0160] In addition, in the description of this embodiment, it is assumed that the virtual space data indicating the virtual space 200B is updated on the user terminal 1B side, but the virtual space data may be updated on the server 2 side. Furthermore, it is assumed that the visual field image data corresponding to the visual field image is updated on the user terminal 1B side, but the visual field image data may be updated on the server 2 side. In this case, the user terminal 1B displays the visual field image on the HMD 110 based on the visual field image data transmitted from the server 2.

[0161] Furthermore, the order of the processes defined by the steps shown in FIGS. 12 and 16 is merely an example, and the order of these steps can be changed as appropriate.

[0162] (Variation 1) Next, with reference to Figs. 18 to 25, a first modified example of the information processing method according to the present embodiment in a case where the HMD 110 of the user A is removed will be described. Fig. 18 is an example of a flowchart for explaining the information processing method according to the first modified example. Figs. 19, 20, 22, and 23 are diagrams showing an example of a virtual space 200B provided to the user B in order to explain the information processing method according to the first modified example. Fig. 21 is a diagram showing an example of a display of the user terminal 5 associated with the user A in order to explain the information processing method according to the first modified example. Fig. 24 is an example of a flowchart for explaining the information processing method according to the first modified example in a case where the user A does not wear the HMD within a certain time from receiving the read information.

[0163] 18, the control unit 121 of the user terminal 1A determines whether the user A has removed the HMD 110 (step S50). The process of determining whether the user A has removed the HMD 110 using the HMD sensor 114 is as described in the above embodiment, and therefore a detailed description thereof will be omitted.

[0164] In step S50, when it is determined that the user A has removed the HMD 110 (in other words, the user A is not wearing the HMD 110) (YES in step S50), the control unit 121 generates information (non-wearing information) indicating that the user A is not wearing the HMD 110 (step S51). After that, the control unit 121 transmits the non-wearing information to the server 2 via the communication network 3 (step S52). Note that in step S51, the control unit 121 may generate positional relationship information indicating the positional relationship between the HMD 110 and the external controller 320, and transmit the positional relationship information together with the non-wearing information to the server 2.

[0165] Next, after receiving the non-wearing information from the user terminal 1A, the server 2 transmits the non-wearing information to the user terminal 1B via the communication network 3 (step S53).

[0166] On the other hand, the control unit 121 of the user terminal 1A, after transmitting the non-wearing information to the server 2 via the communication network 3, transitions from the active mode to the sleep mode (step S54). In this case, it is preferable that the mode of the control unit 121 is the sleep mode, while the mode of the sensor (the wearing sensor 115 or the HMD sensor 114) is the active mode.

[0167] Next, the control unit 121 of the user terminal 1B sets the posture of the avatar 4A placed in the virtual space 200B to a default posture (in other words, a posture set as a default) based on the non-wearing information received from the server 2 (step S55). In the first modified example, the virtual space is assumed to be a room where a partner for multiplayer is searched for, but is not limited to this. Here, the default posture of the avatar 4A refers to the posture of the avatar 4A in its initial state. If the default posture is a lying position, the posture of the avatar 4A is set to a lying position on the floor, as shown in FIG. 19.

[0168] In step S55, the control unit 121 may set the posture of the avatar 4A to a posture selected in advance by the user A, as a mode indicating that the user A is not wearing the HMD 110. In this case, information on the posture of the avatar 4A selected by the user A may be transmitted from the user terminal 1A to the user terminal 1B, and then stored in advance in the storage unit 123 of the user terminal 1B.

[0169] Furthermore, when the user 1B has received positional relationship information indicating the positional relationship between the HMD 110 and the external controller 320, in step S55, the control unit 121 may set the posture of the avatar 4A based on the positional relationship information. That is, when the user A is not wearing the HMD 110, the HMD 110 and the external controller 320 are often placed on the floor or a desk in the real space. In this case, the avatar 4A in the virtual space 200B may be set to a posture as if it is collapsed on the floor based on the positional relationship information between the HMD 110 and the external controller 320.

[0170] Next, the control unit 121 of the user terminal 1B sets the facial expression of the avatar 4A placed in the virtual space 200B to a default facial expression (in other words, a facial expression set as a default) (step S56). If the default facial expression is eyes closed, the facial expression of the avatar 4A is set to eyes closed as shown in FIG.

[0171] In step S56, the control unit 121 may set the facial expression of the avatar 4A to an expression selected in advance by the user A as a mode indicating that the user A is not wearing the HMD 110. In this case, information regarding the facial expression of the avatar 4A selected by the user A may be transmitted from the user terminal 1A to the user terminal 1B and then stored in advance in the storage unit 123 of the user terminal 1B.

[0172] Next, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the field of view CV of the avatar 4B in accordance with the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and displays the field of view image on the HMD 110 based on the updated field of view image data (step S57).

[0173] In this way, since the posture and face of avatar 4A are displayed in the default setting (lying down with eyes closed) in virtual space 200B, user B can easily recognize that user A is not wearing HMD 110 and is not participating in the content provided in virtual space 200B.

[0174] Next, after updating the field of view image in step S56, the control unit 121 of the user terminal 1B determines whether or not, in the virtual space 200B, the avatar 4B associated with user B has performed a specific action (an example of a first action) on the avatar 4A associated with user A (step S58).

[0175] The specific action that the avatar 4B executes with respect to the avatar 4A includes, for example, an action involving physical interference between the avatar 4A and the avatar 4B. The physical interference between the avatar 4A and the avatar 4B is detected using a collider (collision object) for detecting a collision with another object. Although not shown, the collider is associated with the body of the avatar 4A, 4B, and is used for collision judgment (hit judgment) between the avatar 4A and the avatar 4B. For example, when the virtual hand 400 of the avatar 4B, to which a collider is set, comes into contact with the avatar 4A, to which a collider is set, it is determined that the avatar 4B and the avatar 4A have come into contact. The collider may be set to the entire range of the body of the avatar 4A, 4B, or may be set to a specific part of the body of the avatar 4A, 4B (for example, the hand, foot, head, shoulder, waist, etc.). In particular, it is preferable that a collider is set to each fingertip of the virtual hand 400, and the hit judgment is also detected independently for each finger. This allows accurate collision detection between the finger and other objects. Note that the collider is configured as a transparent object, and is not displayed within the field of view image V (FIG. 8).

[0176] In the above example, the control unit 121 specifies the hit determination for the avatar 4B against the avatar 4A based on the movement of the avatar 4B (for example, the movement of the virtual hand 400), but is not limited to this example. For example, the control unit 121 may specify the hit determination for the avatar 4A based on the movement of the user B (for example, the movement of the hand of the user B).

[0177] The specific action including a movement involving physical interference may include, for example, a movement of the avatar 4B touching the avatar 4A using the virtual hand 400, a movement of grabbing and shaking the body of the avatar 4A, a movement of hitting the avatar 4A (for example, a slap on the face or shoulder of the avatar 4A), etc. The specific action may be a movement including a repetitive movement, that is, a movement of contacting the avatar 4A in the process of the repetitive movement of the virtual hand 400. In this way, even if the avatar 4B accidentally touches the avatar 4A due to an erroneous operation by the user B, it can be detected that this is not a specific action, and erroneous detection of the movement of the virtual hand 400 can be prevented.

[0178] For example, as shown in Fig. 20, when avatar 4B grasps the body (e.g., the torso) of avatar 4A with virtual hand 400 and the virtual hand 400 makes a repetitive movement, control unit 121 determines that a shaking action has been performed as the specific action. The grasping (selection) of the body of avatar 4A by virtual hand 400 can be realized, for example, by user B operating (pressing) a key (not shown) of external controller 320 with a collider of virtual hand 400 in contact with a collider of the body of avatar 4A. This shaking action may include a contact action with avatar 4A during the repetitive movement of virtual hand 400 with an acceleration equal to or greater than a certain value. For example, in the case of the state shown in FIG. 20, when the virtual hand 400 is moved back and forth in the z-axis direction while grasping the body of the avatar 4A, if the acceleration in the +z-axis direction is equal to or greater than a certain value and the acceleration in the -z-axis direction is equal to or greater than a certain value for a predetermined number of times in succession, this may be regarded as a shaking motion. However, the direction in which the virtual hand 400 is moved back and forth is not limited to the z-axis direction, and may be any direction. The shaking motion may also include a motion in which the virtual hand 400 moves back and forth through one or more relative coordinates associated with the avatar 4A a predetermined number of times or more within a certain period of time. For example, in the case of the state shown in FIG. 20, the one or more relative coordinates may include a first relative coordinate set at an arbitrary position in the +z-axis direction and a second relative coordinate set at an arbitrary position in the -z-axis direction based on the part of the body of the avatar 4A grasped by the virtual hand 400. In this example, when the virtual hand 400 grasps the body of the avatar 4A and moves back and forth in the z-axis direction, the first relative coordinates move back and forth a predetermined number of times within a certain time, and the second relative coordinates move back and forth a predetermined number of times within a certain time, this may be regarded as a shaking motion. However, the direction in which the virtual hand 400 moves back and forth is not limited to the z-axis direction, and may be any direction. In this case, the relative coordinates for the virtual hand 400 may also be set in any direction.

[0179] It is preferable to determine that a specific action has been performed when the user B himself / herself recognizes the action as shaking. In other words, since it is expected that the shaking direction is not uniform and varies depending on the user, the moving direction of the virtual hand 400 may be in any direction and does not need to be limited to a specific direction. However, when it is necessary to strictly determine the movement of the specific action or to give reality to the movement of the avatar 4B (for example, to prevent the virtual hand 400 from sinking into the avatar 4A), the control unit 121 may determine that a shaking action has been performed when a repetitive movement is performed in a specific direction, taking into account the moving direction of the virtual hand 400.

[0180] On the other hand, when the virtual hand 400 contacts the avatar 4A at a certain acceleration or more (collider collision) in a state where the avatar 4B is not gripping the body of the avatar 4A with the virtual hand 400, the control unit 121 specifies that a slapping action has been executed as the specific action. In order to prevent erroneous detection of the action of the virtual hand 400, it is preferable that the slapping action includes a repetitive movement of the virtual hand 400 at a certain acceleration or more and a plurality of contact actions (collider collisions) with the avatar 4A in the process of the repetitive movement (specifically, a reciprocating slapping action). For example, in the case of the state shown in FIG. 19, the virtual hand 400 is moved reciprocatingly in the y-axis direction, and when the acceleration in the +y-axis direction is a certain value or more and the acceleration in the -y-axis direction is a certain value or more, the slapping action may be determined as a predetermined number of contacts between the collider of the virtual hand 400 and the collider of the face of the avatar 4A. However, the direction in which the virtual hand 400 is moved reciprocatingly is not limited to the y-axis direction, and may be any direction. The reciprocating slapping action may be specified, for example, by the virtual hand 400 of the avatar 4B reciprocating a relative coordinate associated with the avatar 4A a predetermined number of times within a certain time period and by a contact action (hit determination) with the avatar 4A multiple times within the certain time period. For example, in the case of the state shown in FIG. 19, the one or more relative coordinates may be a first relative coordinate set at an arbitrary position in the +y-axis direction and a second relative coordinate set at an arbitrary position in the -y-axis direction with respect to the face of the avatar 4A as a reference. In this example, the virtual hand 400 may be moved reciprocatingly in the y-axis direction, and in a state where the first relative coordinate is moved reciprocatingly a predetermined number of times within a certain time period and the second relative coordinate is moved reciprocatingly a predetermined number of times within a certain time period, the slapping action may be specified when the collider of the virtual hand 400 contacts the collider of the face of the avatar 4A a predetermined number of times or more. However, the direction in which the virtual hand 400 is moved reciprocatingly is not limited to the y-axis direction and may be any direction. In this case, the relative coordinate with respect to the virtual hand 400 may also be set in any direction. However, the control unit 121 may take into consideration the moving direction of the virtual hand 400 and determine that a hitting motion has been performed when the virtual hand 400 has made a repetitive motion in a specific direction.

[0181] The avatar 4A may be set to move in accordance with the movement of the avatar 4B. For example, as shown in Fig. 20, when the avatar 4B executes a shaking action on the avatar 4A, the control unit 121 may set the movement of the avatar 4A so that the avatar 4A is shaken in accordance with the shaking action. However, when the avatar 4B executes a shaking action or a hitting action on the avatar 4A, the avatar 4A may be set not to move in accordance with the movement of the avatar 4B, that is, not to move.

[0182] Next, when the control unit 121 determines that the avatar 4B has performed a specific action on the avatar 4A (YES in step S58), the control unit 121 generates information indicating that the specific action has been performed by the user B (action execution information) (step S60). After that, the control unit 121 transmits the action execution information to the server 2 via the communication network 3 (step S61).

[0183] Next, the server 2 generates information (participation invitation notification information) regarding a participation invitation notification for inviting user A to participate in the content provided in the virtual space 200, based on the action execution information received from the user terminal 1B (step S62). The participation invitation notification is a notification for inviting user A, who has removed the HMD 110, to participate in the content provided in the virtual space 200. The participation invitation notification information may include, for example, the type of the specific action, the strength of the motion of the specific action, etc., in addition to the identification information of the avatar 4B that performed the specific action and the identification information of the avatar 4A that received the specific action.

[0184] Next, the server 2 identifies the personal information of the user A based on the generated participation inquiry notification information (step S63). For example, the personal information of the user A (for example, an email address or an SNS (Social Networking Service) address, etc.) is registered in advance in an information table stored in the storage unit 22 of the server 2 and managed in association with the identification information of the avatar 4A. In step S63, the server 2 refers to the information table stored in the storage unit 22 and acquires the personal information of the user A based on the identification information of the avatar 4A included in the participation inquiry notification information. However, the personal information of the user A may be managed by a server different from the server 2. An example of a server different from the server 2 may be a server capable of distributing a game program to a user whose personal information is registered, that is, a server that plays the role of a game distribution platform, but is not limited thereto.

[0185] Next, the server 2 transmits participation inquiry notification information to the destination (e.g., the email address of user A) acquired in step S63 (step S64). The participation inquiry notification information of user A can be transmitted using a communication tool such as email, chat, or SNS.

[0186] Next, the user terminal 5 receives the participation inquiry notification information by the communication tool, and displays a reception notification of the participation inquiry notification information on the display unit 6 (step S65). The user terminal 5 is a terminal device owned by the user A, and may be, for example, a portable terminal such as a smartphone, a PDA (Personal Digital Assistant), a tablet or phablet computer, or a wearable device, or may be a personal computer, a video game device, or the like.

[0187] Next, the user terminal 5 starts a communication tool based on an input operation by the user A who has seen the reception notification, and displays a message M1 on the display unit 6 as shown in FIG. 21 (step S66). The message M1 includes information indicating that the user B is inquiring the user A to participate in the virtual space 200 (for example, "You are currently receiving communication from [other player name (e.g., user B)] in [content name (e.g., the name of the content provided in the virtual space 200)]. Start [content] and come meet him!", etc.). Note that if the user terminal 5 is a personal computer or a video game device and the image of the content provided in the virtual space 200 is also displayed on the display of these devices, a message regarding the participation inquiry notification may be displayed on the display. The message M1 may also be a message urging the user A to wear the HMD 110 of the user terminal 1A.

[0188] Next, the user terminal 5 determines that user A has read message M1 based on the message M1 being displayed on the display unit 6, and generates read information indicating that message M1 has been read (step S67). The read information is generated based on user A's reaction to message M1, and may be generated based on, for example, an email reply or the allocation of "Like!" points. Thereafter, the user terminal 5 transmits the read information to the server 2 via the communication network (step S68).

[0189] Next, after receiving the read information from the user terminal 5, the server 2 transmits the read information to the user terminal 1B via the communication network 3 (step S69).

[0190] Next, the control unit 121 of the user terminal 1B sets the facial expression of the avatar 4A placed in the virtual space 200B to the updated facial expression based on the read information received from the server 2 (step S70). For example, as shown in Fig. 22, the control unit 121 updates the facial expression of the avatar 4A from a state in which the eyes are closed (an example of a first state) to a state in which the eyes are open (an example of a second state).

[0191] In step S70, the control unit 121 may update the posture of the avatar 4A based on the read information received from the server 2. For example, the posture of the avatar 4A that is lying down may be updated to a sitting position.

[0192] Next, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the field of view CV of the avatar 4B according to the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and displays the field of view image on the HMD 110 based on the updated field of view image data (step S71). This allows the user B to easily understand that the user A has reacted to the participation invitation notification by visually confirming that the avatar 4A has opened its eyes.

[0193] On the other hand, in step S72, the server 2 transmits the read information to the user terminal 1A in addition to the user terminal 1B.

[0194] Next, the control unit 121 of the user terminal 1A determines whether the user A has worn the HMD 110 within a certain time period from the reception of the read information (step S73). The control unit 121 may determine whether the user A has worn the HMD 110 based on the tilt information of the HMD 110 transmitted from the HMD sensor 114 or the event information transmitted from the wearing sensor 115. The determination of whether the user A has worn the HMD 110 may be based on tilt information indicating the tilt (roll angle, yaw angle, pitch angle) of the HMD 110 acquired by the HMD sensor 114, and may be determined based on whether the tilt of the HMD 110 is smaller than a predetermined tilt. As described above, while the user A is wearing the HMD 110, the tilt of the HMD 110 is equal to or smaller than the predetermined tilt, and therefore, if the tilt of the HMD 110 is smaller than the predetermined tilt, it is determined that the user A is wearing the HMD 110.

[0195] When it is determined in step S73 that the user A has worn the HMD 110 (YES in step S73), the control unit 121 of the user terminal 1A transitions from the sleep mode to the active mode (step S74). Next, the control unit 121 that has transitioned to the active mode generates control information for the avatar 4A and generates wearing information indicating that the user A has worn the HMD 110 (step S75). After that, the control unit 121 transmits the control information and wearing information for the avatar 4A to the server 2 via the communication network 3 (step S76).

[0196] Next, after receiving the control information and wearing information of the avatar 4A from the user terminal 1A, the server 2 transmits the control information and wearing information of the avatar 4A to the user terminal 1B via the communication network 3 (step S77).

[0197] Next, the control unit 121 of the user terminal 1B updates the movement of the avatar 4A based on the control information of the avatar 4A, and updates the movement of the avatar 4B based on the control information of the avatar 4B (step S78). In particular, the control unit 121 updates the posture of the avatar 4A based on the wearing information of the avatar 4A. The control unit 121 may also update the facial expression of the avatar 4B based on the facial information of the avatar 4B included in the control information of the avatar 4B. Specifically, the control unit 121 sets the posture of the avatar 4A to a sitting up state (an example of the third state) as shown in FIG. 23, for example.

[0198] 17, the control unit 121 may generate a speech bubble object associated with the avatar 4A in the virtual space 200B. The speech bubble object displays information indicating that the user A has put on the HMD 110 (for example, "User A has returned to his / her seat").

[0199] Thereafter, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the field of view CV of the avatar 4B in accordance with the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and displays the field of view image on the HMD 110 based on the updated field of view image data (step S79).

[0200] On the other hand, in step S73, if it is determined that the user A is not wearing the HMD 110 within a certain time from the reception of the read information (NO in step S73), the process proceeds to the process in Fig. 24. Next, as shown in Fig. 24, the control unit 121 generates non-wearing information indicating that the user A is not wearing the HMD 110 (step S80). After that, the control unit 121 transmits the non-wearing information to the server 2 via the communication network 3 (step S81).

[0201] Next, after receiving the non-wearing information from the user terminal 1A, the server 2 transmits the non-wearing information to the user terminal 1B via the communication network 3 (step S82).

[0202] Next, the control unit 121 of the user terminal 1B sets the facial expression of the avatar 4A arranged in the virtual space 200B to a default expression based on the non-wearing information received from the server 2 (step S83). For example, if the default facial expression is eyes closed, the facial expression of the avatar 4A is updated to eyes closed as shown in FIG. 19. Note that, in the process of step S83, the control unit 121 of the user terminal 1B may measure a certain time from the reception of the read information, and if the control information and the wearing information of the avatar 4A are not received within the certain time, the facial expression of the avatar 4A may be set to the default expression. Note that, if the eyes of the avatar 4A are open based on the read information in step S70 of FIG. 18, the facial expression of the avatar 4A is returned to the eyes closed in this step S83. On the other hand, if the participation invitation notice has not been read (NO in step S66), the avatar 4A maintains the state in which the eyes are closed, and therefore the facial expression of the avatar 4A is not changed in step S83.

[0203] Thereafter, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the field of view CV of the avatar 4B in accordance with the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and displays the field of view image on the HMD 110 based on the updated field of view image data (step S84).

[0204] As described above, the information processing method according to the first modification includes the steps of moving the virtual hand 400 in the virtual space 200 in response to the movement of the user B, performing an action (first action) such as a shaking action by the virtual hand 400 on the avatar 4A whose movement in the virtual space 200 can be controlled based on the movement of the user A, and performing a participation invitation notification (second action) to invite the user A (the user terminal 5 associated with the user A) to participate in the virtual space 200 based on the execution of the action. According to this method, a predetermined action is executed between a plurality of users A and B who share the virtual space 200 by an intuitive operation using the virtual hand 400, so that communication in the virtual space 200 can be smoothly performed without impairing the immersive feeling of the user. Thus, according to the first modification, a user interface (so-called diegetic UI) that does not require menu operation or key operation for communication between users is provided, thereby improving the virtual experience of the user.

[0205] In particular, by transmitting a participation invitation notification to the user terminal 5 associated with the user A based on a specific action performed by the user B on the avatar 4A, the user A can easily recognize that the user B has invited him / her to participate in the content provided in the virtual space 200. This can promote the use of content that requires multiple users to play simultaneously in the virtual space 200 (for example, a multiplayer game).

[0206] Furthermore, in the information processing method according to the first modification, when a specific action is performed by user B on avatar 4A while avatar 4A is not being controlled by user A (i.e., user A is not wearing HMD 110), a participation invitation notification is sent to user A. Therefore, user A who is not wearing HMD 110 can be attracted to content provided in virtual space 200, and participation in the content can be further promoted. In particular, in a VR game, when there are few users (avatars) waiting in a room, that is, when there are few candidates for multiplay, users (avatars) enter the room for the time being, but often remove the HMD and wait until a candidate for multiplay appears, so that such waiting users can be effectively attracted to multiplay in a VR game.

[0207] The information processing method according to the first modification may further include a step of transitioning the avatar 4A in the virtual space 200B from an eyes-closed state (first state) to an eyes-open state (second state) when the user A reacts to the participation inquiry notification (for example, when the message M1 has been read), and a step of transitioning the avatar 4A in the virtual space 200B to a standing state (third state) when the avatar 4A is controlled by the user A within a certain period of time after the avatar 4A has transitioned to the eyes-open state. This allows the user B to easily grasp that there has been a reaction from the user A and that the user A is wearing the HMD 110 and controlling the avatar 4A.

[0208] On the other hand, if the avatar 4A is not controlled by the user A within a certain period of time after the avatar 4A transitions to the state with its eyes open, the avatar 4A in the virtual space 200B may be returned to the state with its eyes closed. This allows the user B to easily understand that the avatar 4A is not controlled by the user A despite the participation inquiry notification being sent (i.e., the user A is not participating in the content). Therefore, the user B can shake the avatar 4A again to inquire about participation in the virtual space again, or can inquire about participation in the virtual space of an avatar associated with another user other than the user A who shares the virtual space 200.

[0209] The state in which the avatar 4A is not controlled by the user A may include not only the case in which the user A has removed the HMD 110, but also the case in which the user A has not logged in to the content provided in the virtual space 200. When the user A has not logged in to the content, the avatar 4A does not usually exist in the virtual space 200B provided by the user terminal 1B of the user B. However, if the user B has registered the user A as a friend, the avatar 4A of the user A can be made to exist in the virtual space 200B even if the user A has not logged in to the content. The presence or absence of this friend registration may be registered in the server 2, for example, together with the personal information of each user. In this case, the control unit 121 of the user terminal 1B can set the posture and facial expression of the avatar 4A arranged in the virtual space 200B to defaults based on the fact that the user A has not logged in to the content (step S54 in FIG. 18). In this way, by having avatar 4A associated with user A who is not logged in to the content exist in virtual space 200B, and sending a participation invitation notification to user A based on the execution of a specific action from avatar 4B to avatar 4A, it is possible to provide motivation for user A to log in to the content.

[0210] Also, the content of the participation inquiry notification (message M1) may be different depending on the strength of the motion of the specific action by the avatar 4B against the avatar 4A. For example, a plurality of thresholds may be set in stages for the acceleration of the motion by the virtual hand 400, and the content of the message M1 may be changed depending on the threshold of the acceleration. For example, the message M1 may include information on the participation desire level, and the information on the participation desire level may be changed depending on the strength of the shaking motion. Specifically, when the shaking motion is intense, the message M1 may be displayed with content indicating that the participation desire level from the user B is high (for example, "Participation desire level: ☆☆☆ (three stars)"). Note that a predetermined threshold may be set for the number of times the virtual hand 400 passes through the relative coordinates associated with the avatar 4A, and the strength of the motion of the specific action may be specified depending on the threshold of the number of passes.

[0211] Furthermore, the content of the message M1 may be changed when multiple avatars simultaneously (or in cooperation) perform a specific action on avatar 4A. In this case, it is preferable to provide dedicated colliders for each specific part of avatar A's body so that avatar A can be grabbed by multiple avatars at the same time. This allows multiple avatars to perform a shaking motion on avatar 4A. Note that in this case, it is preferable to increase the participation desire level as the number of users performing a specific action on avatar 4A increases.

[0212] Furthermore, the content of the message M1 may differ depending on the type of specific action performed on the avatar 4A. For example, if the avatar 4B slaps the avatar 4A, the participation desire level displayed in the message M1 may be higher than if the avatar 4B shakes the avatar 4A.

[0213] In this way, by varying the content of the participation inquiry notification depending on the strength of the motion of the specific action by avatar 4B against avatar 4A and the type of the specific action, user A can understand the degree of desire to participate in the content from user B. Therefore, user A can decide whether or not to participate in the content depending on the degree of desire.

[0214] In addition, in the present embodiment, an action involving physical interference is exemplified as the specific action by the avatar 4B to the avatar 4A, but the specific action is not limited to this example. The specific action may be any action for identifying the user A, and for example, the specific action executed by the avatar 4B to the avatar 4A may be identified by gaze, voice, laser pointer, facial expression (smile, glare, etc.), clapping, etc. Also, the user B may select an avatar (user) to which he / she wishes to invite participation in the content from a list of avatars (list of users) displayed in the virtual space 200B. That is, the specific action executed by the avatar 4B may be a direct action or an indirect action to the avatar 4A. Also, a direct action and an indirect action may be combined.

[0215] In addition, the timing of sending the participation inquiry notification may be changed depending on the contents provided in the virtual space 200. For example, when the number of users required to start a multiplay is three or more, when the control unit 121 of the user terminal 1B determines that the user B has performed a specific action on the last avatar among the multiple avatars required to start a multiplay, the control unit 121 of the user terminal 1B may send a participation inquiry notification to each user associated with the multiple avatars who have performed the specific action collectively. Note that even when the number of users required to start a multiplay is three or more, the control unit 121 of the user terminal 1B may send a participation inquiry notification to each avatar individually at the timing when the avatar 4B has performed a specific action on each avatar.

[0216] In the above embodiment, the facial expression of the avatar 4A, which had closed eyes, is updated to open eyes based on the reaction of the user A to the participation inquiry notification (e.g., message M1 regarding the participation inquiry notification has been read), but this is not limited to the example. For example, the eyes of the avatar 4A may be updated to open eyes based on the execution of a specific action on the avatar 4A by the avatar 4B. This allows the user B to easily understand that the specific action has been appropriately executed on the avatar 4A. The eyes of the avatar 4A may be updated to open eyes based on the user A putting on the HMD 110.

[0217] Moreover, the information processing method according to the first modification includes a step of executing a predetermined action (an example of a first action) involving physical interference from user B to user A in the virtual space 200, a step of executing a participation inquiry notice (an example of a second action) to user A based on the execution of the action, and a step of updating the posture and / or facial expression of the avatar 4A in the virtual space 200 (an example of a third action) based on the reaction of user A to the participation inquiry notice (for example, the message M1 related to the participation inquiry notice has been read). According to this method, the posture and facial expression of the avatar 4A associated with user A in the virtual space 200B are updated based on the reaction of user A to the participation inquiry notice transmitted to user A, thereby enabling smooth communication between users A and B in the virtual space 200. In this way, a seamless virtual experience that blurs the boundary between the real space and the virtual space can be provided by reflecting the reaction of user A to the participation inquiry notice in the real space in the virtual space 200.

[0218] (Variation 2) Next, with reference to Fig. 25 to Fig. 28, a modified example 2 of the information processing method according to this embodiment will be described. In the modified example 2, an information processing method at the start of a game in which a game played only by user B (a so-called single-play game) is provided in the virtual space 200 (200B) is illustrated. Fig. 25 is an example of a flowchart for explaining the information processing method according to the modified example 2. Figs. 26 to 28 are diagrams showing an example of the virtual space 200B provided to user B for explaining the information processing method according to the modified example 2. As shown in Fig. 26, in the modified example 2, the virtual space 200B includes an avatar 4B and an avatar 4C. The avatar 4C is an avatar 4C (a so-called Non Player Character (NPC)) that can be controlled by the server 2.

[0219] 25, first, the control unit 121 of the user terminal 1B accepts a game start input based on an operation by the user B (step S100). Next, the control unit 121 transmits game start input information to the server 2 via the communication network 3 (step S101).

[0220] Next, the server 2 reads out data of an opening movie from the storage unit 22 based on the game start input information received from the control unit 121 (step S102). The opening movie is a video that is played in the virtual space 200B as a scenario scene (an example of a first scene) that is introduced at the start of the game. Next, the server 2 transmits data of the opening movie to the control unit 121 of the user terminal 1B via the communication network 3 (step S103).

[0221] Next, the control unit 121 of the user terminal 1B plays back the opening movie in the virtual space 200B based on the data of the opening movie received from the server 2 (step S104).

[0222] During or after the opening movie is played, the control unit 121 of the user terminal 1B sets the posture and facial expression of the avatar 4C to the default posture and facial expression (step S105). If the default posture is lying down and the default facial expression is eyes closed, then the posture of the avatar 4C is set to lying down and the facial expression of the avatar 4C is set to eyes closed, as shown in FIG.

[0223] Next, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the field of view CV of the avatar 4B in accordance with the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and displays the field of view image on the HMD 110 based on the updated field of view image data (step S106).

[0224] Next, after updating the field of view image in step S106, the control unit 121 determines whether or not the avatar 4B associated with the user B has performed a specific action (an example of a first action) on the avatar 4C associated with the server 2 (step S107). The process of determining the specific action is similar to that of the first modification, and therefore a detailed description thereof will be omitted. In order to encourage the avatar 4B to perform a specific action on the avatar 4C, the control unit 121 may display a message such as "Please wake up avatar 4C" in the virtual space 200B after playing the opening movie in step S104.

[0225] Next, when the control unit 121 determines that the avatar 4B has performed a specific action on the avatar 4C (YES in step S107), the control unit 121 generates action execution information indicating that the specific action has been performed by the user B (step S108). After that, the control unit 121 transmits the action execution information to the server 2 via the communication network 3 (step S109).

[0226] Next, the server 2 generates scene transition information based on the action execution information received from the user terminal 1B (step S110). The scene transition information is information for transitioning the game provided in the virtual space 200B from a scenario scene to a game scene (an example of a second scene). Next, the server 2 transmits the scene transition information to the control unit 121 of the user terminal 1B via the communication network 3 (step S111).

[0227] Next, the control unit 121 of the user terminal 1B updates the settings of the posture and facial expression of the avatar 4C based on the scene transition information received from the server 2 (step S112). Specifically, for example, as shown in Fig. 28, the posture of the avatar 4C is set to a sitting up state, and the facial expression of the avatar 4C is set to an open-eyed state.

[0228] Next, the control unit 121 generates a speech bubble object 45A associated with the avatar 4C in the virtual space 200B (step S113). The speech bubble object 45A displays information indicating that the avatar 4C has joined the group of the avatar 4B (for example, "Avatar 4C has joined the group.").

[0229] Thereafter, the control unit 121 updates the virtual space data indicating the virtual space 200B, and updates the field of view CV of the avatar 4B in accordance with the movement of the HMD 110 of the user terminal 1B. Next, the control unit 121 updates the field of view image data based on the updated virtual space data and the updated field of view CV of the avatar 4B, and displays the field of view image on the HMD 110 based on the updated field of view image data (step S114).

[0230] After that, based on the scene transition information, the control unit 121 starts displaying the game scene (step S115).

[0231] As described above, the information processing method according to the second modification includes a step of executing a specific action such as a shaking action by the virtual hand 400 on the avatar 4C whose action in the virtual space 200B is computer-controllable, and a step of transitioning a game provided in the virtual space 200B from a scenario scene (first scene) to a game scene (second scene) based on the execution of the specific action. Conventionally, in order to start a game scene, a user interface (e.g., "Press Any Button") for progressing the game is displayed after the scenario scene is played back, and the game scene is started by the user pressing a predetermined button on the external controller 320. In contrast, according to the method according to the second modification, a game scene is started based on the execution of a specific action by the user B on the avatar 4C, so that the game can be progressed without impairing the user B's immersion in the virtual space 200B. Note that, in the second modification, a case in which the processing of the second modification is executed in a system having the server 2 and the user terminal 1B has been described as an example, but the present invention is not limited thereto, and the processing may be executed by the user terminal 1B alone.

[0232] Moreover, the information processing method for progressing the game according to the second modification is not limited to a scene when the game is started, but can also be applied to a scene when the game is resumed in the middle of the game. For example, in a case where a game played by a party organized by a plurality of avatars (for example, avatar 4B and avatar 4C) is provided in the virtual space 200B, the game may be resumed when it is determined that a specific action is performed from avatar 4B to avatar 4C after the game progress is interrupted. Specifically, in a scene where the party stays at an inn, the avatar 4C is still sleeping the next morning. In this state, the avatar 4B may perform a specific action such as a shaking action or a hitting action on the avatar 4C, so that the avatar 4C wakes up and the progress of the game is resumed. According to such a method, the progress of the game can be resumed without impairing the user B's immersion in the virtual space 200B.

[0233] Furthermore, the order of the processes defined by the steps shown in FIGS. 18, 24, and 25 is merely an example, and the order of these steps can be changed as appropriate.

[0234] In addition, in the above description of the embodiment and the modified example, it is assumed that the wearing or non-wearing of the HMD 110 is detected by the tilt information of the HMD 110 transmitted from the HMD sensor 114 or the event information transmitted from the wearing sensor 115, but this is not limited to the example. For example, the control unit 121 of the user terminal may detect the movement of the HMD 110 by the position sensor 130 and / or the HMD sensor 114, and when the movement draws a predetermined trajectory, may determine that the user has worn the HMD 110 or removed the HMD 110. In addition, the control unit 121 may detect the movement of the HMD 110 by the position sensor 130 and / or the HMD sensor 114, and may determine that the user is not wearing the HMD 110, for example, when the position of the HMD 110 does not change at all for a certain period of time. Furthermore, the control unit 121 may detect the wearing or non-wearing of the HMD 110 by detecting the relative positional relationship between the HMD 110 and the external controller 320. For example, the control unit 121 may determine that the user is not wearing the HMD 110 when the HMD 110 and the external controller 320 are separated by a certain distance or more.

[0235] In addition, in the above description of the embodiment and the modified example, it is assumed that the virtual space data showing the virtual space 200B is updated on the user terminal 1B side, but the virtual space data may be updated on the server 2 side. Furthermore, it is assumed that the visual field image data corresponding to the visual field image is updated on the user terminal 1B side, but the visual field image data may be updated on the server 2 side. In this case, the user terminal 1B displays the visual field image on the HMD 110 based on the visual field image data transmitted from the server 2.

[0236] Furthermore, the order of the processes defined by the steps shown in FIGS. 12 and 16 is merely an example, and the order of these steps can be changed as appropriate.

[0237] In addition, in the present embodiment, a virtual space (VR space) in which the user is immersed by the HMD device has been described as an example, but a see-through HMD device may be adopted as the HMD device. In this case, a virtual experience in an augmented reality (AR) space or a mixed reality (MR) space may be provided to the user by outputting a field of view image in which a part of an image constituting the virtual space is synthesized in a real space visually recognized by the user through the see-through HMD device. In this case, an action on a target object in the virtual space may be generated based on the movement of the user's hand instead of the virtual hand. Specifically, the processor may specify coordinate information of the position of the user's hand in the real space and define the position of the target object in the virtual space in relation to the coordinate information in the real space. This allows the processor to grasp the positional relationship between the user's hand in the real space and the target object in the virtual space 2, and to execute processing corresponding to the above-mentioned collision control between the user's hand and the target object. As a result, it becomes possible to give an action to the target object based on the movement of the user's hand.

[0238] In addition, in order to realize various processes executed by the control unit 121 of the user terminal 1 by software, a control program for causing a computer (processor) to execute various processes may be pre-installed in the storage unit 123 or memory. Alternatively, the control program may be stored in a computer-readable storage medium such as a magnetic disk (HDD, floppy disk), an optical disk (CD-ROM, DVD-ROM, Blu-ray (registered trademark) disk, etc.), a magneto-optical disk (MO, etc.), or a flash memory (SD card, USB memory, SSD, etc.). In this case, the storage medium is connected to the control device 120, and the control program stored in the storage medium is installed in the storage unit 123. Then, the control program installed in the storage unit 123 is loaded onto the RAM, and the processor executes the loaded program, whereby the control unit 121 executes various processes.

[0239] The control program may also be downloaded from a computer on the communication network 3 via the communication interface 125. In this case as well, the downloaded control program is installed in the storage unit 123.

[0240] Although the embodiment of the present disclosure has been described above, the technical scope of the present invention should not be interpreted as being limited by the description of the present embodiment. The present embodiment is an example, and it is understood by those skilled in the art that various modifications of the embodiment are possible within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and its equivalents. [Explanation of symbols]

[0241] 1, 1A, 1B: User terminal 2: Server 3: Communication network 4A, 4B: Avatar 21: Communication interface 22: Storage part 23: Control unit 24: Bus 42A, 44A, 45A: Speech bubble object 43A: Caption object 100: Virtual space distribution system (distribution system) 110: HMD (head mounted device) 112: Display section 113: Face camera 114: HMD sensor 115: Wearable sensor 116: Headphones 117: Face Camera 118: Microphone 120: Control device 123: Storage section 124: I / O interface 125: Communication interface 126: Bus 130: Position sensor 140: Gaze sensor 200, 200A, 200B: Virtual space 210: Center position 300: Virtual camera 320: External controller 320L: Left-handed external controller (controller) 320R: Right-hand external controller (controller) 400: Virtual hand U, A, B: User M1: Message

Claims

1. Computer, providing means for providing a virtual experience to a user via a device operated by the user; a first execution means for executing an action for identifying another user different from the user in a virtual space; a second execution means for executing a process of notifying the other user of a request to participate in the virtual space based on the execution of the action; A program that functions as The action includes a specific action by a first virtual object associated with the user on a second virtual object associated with the other user; The notification includes an indication of a degree of desire indicating a degree to which the user desires the other user to participate in the virtual space, The second execution means changes an indication of the desirability included in the notification depending on at least one of a strength of the motion of the action on the second virtual object and a type of the action.

2. The program according to claim 1 , wherein the specific action includes at least one of a physical interference between the first virtual object and the second virtual object, a gaze from the first virtual object to the second virtual object, a voice, a laser pointer, a facial expression, and clapping.

3. Computer, providing means for providing a virtual experience to a user via a device operated by the user; a first execution means for executing a first action in a virtual space, the first action being an action for identifying another user different from the user; a second execution means for executing a process of notifying the other user of a request to participate in the virtual space based on the execution of the action; A program that functions as The action includes a specific action by a first virtual object associated with the user on a second virtual object associated with the other user; The notification includes an indication of a degree of desire indicating a degree to which the user desires the other user to participate in the virtual space, The second execution means changes an indication of the desirability included in the notification depending on the strength of the action performed on the second virtual object.

4. causing the computer to further function as a moving means for moving an operation object within the virtual space in response to a movement of a part of the body of the user; The program according to claim 1 , wherein the action is executed using the operation object.

5. The program according to claim 4 , wherein the specific action includes an action of the first virtual object shaking or hitting the second virtual object using the control object.

6. The program according to claim 1 , wherein the notification is sent to a terminal other than the device and the computer, the terminal being associated with the other user.

7. Computer, providing means for providing a virtual experience to a user via a device operated by the user; a first execution means for executing an action for identifying another user different from the user in a virtual space; a second execution means for executing a process of notifying the other user of a request to participate in the virtual space based on the execution of the action; An information processing system that functions as The action includes a specific action by a first virtual object associated with the user on a second virtual object associated with the other user; The notification includes an indication of a degree of desire indicating a degree to which the user desires the other user to participate in the virtual space, An information processing system in which the display of the desirability included in the notification is made different depending on at least one of the strength of the movement of the action on the second virtual object and the type of the action.

8. Computer, providing means for providing a virtual experience to a user via a device operated by the user; a first execution means for executing an action for identifying another user different from the user in a virtual space; a second execution means for executing a process of notifying the other user of a request to participate in the virtual space based on the execution of the action; An information processing system that functions as The action includes a specific action by a first virtual object associated with the user on a second virtual object associated with the other user; The notification includes an indication of a degree of desire indicating a degree to which the user desires the other user to participate in the virtual space, An information processing system in which the display of the desirability included in the notification is changed depending on the strength of the action performed on the second virtual object.

Images