Information processing apparatus, head-mounted type display device and information processing method

Abstract

To enable a user who is wearing a head-mounted type display device to easily and visually recognize a back direction of a virtual object.SOLUTION: An information processing apparatus comprises: selection means that selects a virtual object existing in a visual line position of a user who is wearing a head-mounted type display device as a control object; determination means that determines whether or not the user has performed a predetermined action for visually recognizing a back direction of the control object; and control means that controls display of the control object to be able to visually recognize the back direction of the control object in response to the predetermined action of the user.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, a head-mounted information display device, and an information processing method. [Background technology]

[0002] In recent years, the real world and the virtual world have been combined to create new experiences through XR (Extended Reality). XR technology is a general term for virtual reality, mixed reality, augmented reality, and alternative reality technologies. Virtual reality (VR) technology is a technology that allows users to experience a virtual space as if it were real. Mixed reality (MR) technology is a technology that allows users to experience mixed reality by seamlessly fusing reality and virtual reality. Augmented reality (AR) is a technology that overlays a virtual world onto real space and projects it. Substitutional reality (SR) is a technology that overlays and projects past images onto the real world.

[0003] One type of XR technology is the XR system that uses a head mounted display (HMD). The XR system allows a user wearing an HMD to experience an XR space by superimposing virtual objects on the actual outside world.

[0004] The HMD is equipped with a function for detecting the user's gaze position, and is capable of display control utilizing information on the gaze position. For example, Patent Document 1 discloses that a virtual object related to a real object for which an attention action has been performed is displayed in response to the continuation of the attention action over a predetermined reference time. Patent Document 2 discloses that an image visually recognized by a user changes depending on a combination of the user's head direction and gaze direction. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2016-81339 A [Patent Document 2] JP 2016-82466 A Summary of the Invention [Problem to be solved by the invention]

[0006] When a user wearing an HMD wants to temporarily see what is behind a virtual object, the user can move the virtual object using a user interface such as hand gestures, or move to a position where the user can see what is behind the virtual object. However, it can be difficult to see what is behind a virtual object when the range of movement is limited, when the user wants to control the virtual object without changing their standing position, or when the user cannot move their arms and legs freely.

[0007] An object of the present invention is to provide an information processing device that enables a user wearing a head-mounted display device to easily view what is behind a virtual object. [Means for solving the problem]

[0008] The information processing device according to the present invention comprises a selection means for selecting a virtual object present at a gaze position of a user wearing a head-mounted display device as a control object, a determination means for determining whether or not the user has performed a predetermined action for visually recognizing an area behind the control object, and a control means for controlling display of the control object in response to the user performing the predetermined action so that an area behind the control object can be visually recognized. It is a sign. Effect of the Invention

[0009] According to the present invention, it is possible to enable a user wearing a head-mounted display device to easily view behind a virtual object. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating an example of the appearance of an HMD. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of an HMD. [Diagram 3] 11 is a flowchart illustrating a process of the HMD. [Figure 4] FIG. 2 is a diagram illustrating a three-axis acceleration sensor and a three-axis angular velocity sensor. [Diagram 5] 13 is a flowchart illustrating a process of determining a peering motion. [Figure 6] 11A and 11B are diagrams illustrating a determination of a peering operation. [Figure 7] FIG. 2 is a diagram illustrating metadata assigned to an object. [Figure 8] 11 is a flowchart illustrating a display control process of a control object. [Figure 9] FIG. 13 is a diagram illustrating a moving direction of a control object. [Figure 10] 11A and 11B are diagrams illustrating display control of a control object. [Figure 11] FIG. 13 is a diagram illustrating highlighting of a control object. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Embodiment> Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] (Device Configuration) An example of the configuration of an HMD100, which is a head-mounted display device according to this embodiment, will be described with reference to Figs. 1 and 2. Fig. 1 is a diagram illustrating an example of the appearance of the HMD100. The HMD100 shown in Fig. 1 is a display device that is worn on the head of a user. The HMD100 includes operation members such as a power switch and a button for controlling device settings. A user can operate the HMD100 via the operation members or a controller that can communicate with the HMD100. The HMD100 includes an image processing unit that generates virtual objects and synthesizes images.

[0013] The head mounting member 101 is a member for stably fixing the HMD 100 to the user's head. The head mounting member 101 fixes the HMD 100 to the user's head so that the HMD 100 can move without shifting in accordance with the movement of the user's head. Note that the head mounting member 101 is not limited to a member for fixing the HMD 100 to the user's head, and may be a member for fixing the HMD 100 by hanging it over the user's ear.

[0014] The display device 102 displays a virtual object or a composite image of an image of real space and a virtual object in front of the eyes of a user who is wearing the HMD 100. The display device 102 is, for example, a liquid crystal display or an organic EL (ElectroLuminescence) display. The display device 102 includes, for example, a liquid crystal panel, a driver circuit that controls the liquid crystal panel, and a memory that holds the image to be displayed. The display device 102 may be a non-transmissive display unit, or may be an optically transmissive display unit that allows the real space to be viewed through a lens.

[0015] The image capturing device 103 is a device (camera unit) that captures the environment around the user. The image capturing device 103 may capture the environment in front of the user as the environment around the user. The environment in front of the user may be, for example, the front of the user's head.

[0016] Fig. 2 is a block diagram showing an example of the configuration of the HMD 100. The HMD 100 shown in Fig. 2 includes an imaging unit 200, a gaze detection unit 201, a movement detection unit 202, a rotation detection unit 203, an information processing unit 204, an image processing unit 205, and a display unit 206. Note that the information processing unit 204 and the image processing unit 205 will be described as processing units included in the HMD 100, but the processing or a part of the processing of the information processing unit 204 and the image processing unit 205 may be executed by an external information processing device capable of communicating with the HMD 100. In this case, the information processing device executes the processing of the information processing unit 204 and the image processing unit 205, and controls the display of a virtual object displayed by the display unit 206 of the HMD 100.

[0017] The information processing unit 204 includes a captured image acquisition unit 207, an object holding unit 208, an object generation unit 209, a line of sight information acquisition unit 210, a movement amount calculation unit 211, a rotation amount calculation unit 212, and a motion determination unit 213. The image processing unit 205 includes an object selection unit 214, an object control unit 215, and a display image generation unit 216.

[0018] The imaging unit 200 has an optical system, an image sensor, a driver circuit for controlling the image sensor, an A / D conversion circuit for converting a signal acquired by the image sensor into a digital signal, and a development circuit for developing the obtained digital signal into an image. The imaging unit 200 transmits captured image data (captured image) to a captured image acquisition unit 207 of the information processing unit 204. The captured image acquisition unit 207 transmits the image data received from the imaging unit 200 to a display image generation unit 216 of the image processing unit 205. The captured image acquisition unit 207 also transfers the captured image to an object generation unit 209.

[0019] The gaze detection unit 201 detects the gaze direction of a user who is a wearer of the HMD 100. The gaze detection unit 201 may be, for example, a device used in a single-lens reflex camera or the like, which detects the gaze direction from the relationship between the reflected image of the light source due to corneal reflection and the pupil by illuminating the user's eyeball with an infrared light emitting diode.

[0020] The gaze information acquisition unit 210 of the information processing unit 204 acquires information on the gaze (hereinafter also referred to as gaze information) detected by the gaze detection unit 201. The gaze information acquisition unit 210 transmits the acquired gaze information to the object control unit 215.

[0021] The movement detection unit 202 detects the movement of the HMD 100. The movement detection unit 202 can detect the movement of the HMD 100 using, for example, position information from a global positioning system (GPS) and the output of an acceleration sensor.

[0022] The movement amount calculation unit 211 of the information processing unit 204 acquires information on the movement of the HMD 100 detected by the movement detection unit 202, and calculates the movement direction and movement amount of the HMD 100. The movement amount calculation unit 211 transmits the calculated movement direction and movement amount of the HMD 100 (hereinafter also referred to as movement information) to the motion determination unit 213.

[0023] The rotation detection unit 203 detects the rotation of the HMD 100. The rotation detection unit 203 can detect the rotation of the HMD 100 using, for example, the output of a rotation displacement sensor. The rotation amount calculation unit 212 of the information processing unit 204 acquires information on the rotation detected by the rotation detection unit 203 and calculates the rotation direction and rotation amount (hereinafter also referred to as rotation information) of the HMD 100. The rotation amount calculation unit 212 transmits the calculated rotation direction and rotation amount of the HMD 100 to the operation determination unit 213.

[0024] The motion determination unit 213 determines the motion of the HMD 100 based on the movement direction and movement amount of the HMD 100 received from the movement amount calculation unit 211 and the rotation direction and rotation amount of the HMD 100 received from the rotation amount calculation unit 212. The motion determination unit 213 determines the motion of the HMD 100 based on the information of the determined motion (hereinafter The image processing unit 205 transmits the object information (hereinafter, also referred to as operation information) to the object control unit 215 of the image processing unit 205.

[0025] The object holding unit 208 holds data related to the virtual space, such as data (shape information, position and orientation information, etc.) of virtual objects that configure the virtual space and data of a light source that illuminates the virtual space, in a storage unit such as a memory. The object selection unit 214 of the image processing unit 205 selects a virtual object to be displayed from the object holding unit 208, and transfers data of the selected virtual object to the object control unit 215. The object holding unit 208 may also transmit data of the virtual object to the object generation unit 209, causing it to generate another virtual object.

[0026] The object generation unit 209 generates a virtual object based on the image data transferred from the captured image acquisition unit 207 and the virtual object data transferred from the object holding unit 208. The object generation unit 209 transmits the generated virtual object to the object selection unit 214 of the image processing unit 205.

[0027] The object selection unit 214 selects a virtual object to be displayed to the user from the object holding unit 208 and the object generation unit 209. The object selection unit 214 transmits data of the selected virtual object to be displayed to the object control unit 215. The data of the virtual object to be displayed transmitted to the object control unit 215 includes display image data, arrangement position data in the display angle of view, overlap order data with other virtual objects, and the like.

[0028] The object control unit 215 controls the display of a virtual object selected by the object selection unit 214, based on the line-of-sight information received from the line-of-sight information acquisition unit 210 and the operation information of the HMD 100 received from the operation determination unit 213. The control of the display of a virtual object includes moving, reducing, making transparent, and hiding the virtual object. The object control unit 215 controls the display of the virtual object by changing the data of the virtual object based on the line-of-sight information and the operation information. The object control unit 215 transmits the data of the virtual object to which the display control has been applied to the display image generation unit 216.

[0029] The display image generation unit 216 generates a composite image of the image data of the captured real space transferred from the captured image acquisition unit 207 and the virtual object received from the object control unit 215. The display image generation unit 216 transmits the generated composite image to the display unit 206.

[0030] (HMD processing) The processing of the HMD 100 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart illustrating the processing of the HMD 100. The processing shown in Fig. 3 is started when a user wearing the HMD 100 turns on the power switch of the HMD 100 and a virtual object is displayed on the display device 102.

[0031] In step S300, the gaze information acquisition unit 210 acquires the user's gaze position on the display screen of the display device 102 based on information about the user's gaze detected by the gaze detection unit 201. The gaze information acquisition unit 210 transmits information about the user's gaze position to the object control unit 215.

[0032] In step S301, the object control unit 215 determines whether or not a virtual object is present at the user's gaze position based on the data of the virtual object selected by the object selection unit 214 and the gaze information received from the gaze information acquisition unit 210. If a virtual object is present at the gaze position, the process proceeds to step S302. If a virtual object is not present at the gaze position of the user, the process returns to step S300.

[0033] The object control unit 215 may determine whether or not the virtual object at the user's gaze position is the foreground virtual object. If the virtual object at the user's gaze position is the foreground virtual object, the process proceeds to step S302. If the virtual object at the user's gaze position is not the foreground virtual object, the process returns to step S300.

[0034] In step S302, the object control unit 215 selects a virtual object present at the gaze position as a control object. In step S303, the gaze information acquisition unit 210 determines whether or not the viewing time for the control object selected in step S302 is longer than a predetermined time t (time). That is, the gaze information acquisition unit 210 determines whether or not the gaze position directed at the control object has not changed during the predetermined time t. The predetermined time t can be, for example, a time that can be determined as the user gazing at the control object.

[0035] If the viewing time for the control object is longer than the predetermined time t, the process proceeds to step S304. If the viewing time for the control object is equal to or shorter than the predetermined time t, the process returns to step S300.

[0036] In step S304, the motion determination unit 213 determines the motion of the HMD 100. The HMD 100 is fixed to the user's head, and the motion determination unit 213 can determine the motion of the user's head by determining the motion of the HMD 100. Specifically, the motion determination unit 213 acquires the movement direction and movement amount of the HMD 100 (user's head) from the movement detection unit 202. In addition, the motion determination unit 213 acquires the rotation direction and rotation amount of the HMD 100 (user's head) from the rotation detection unit 203. The motion determination unit 213 determines the motion of the HMD 100 (user) based on at least one of the movement direction, movement amount, rotation direction, and rotation amount of the HMD 100 (user's head). Note that the determination process of the motion determination unit 213 in step S304 may be executed before the process of step S303 is started.

[0037] In step S305, the motion determination unit 213 determines whether or not a predetermined motion has been performed by the user. The predetermined motion is a motion for the user wearing the HMD 100 to visually check behind the control object selected in step S302, such as a motion of actually peering into the rear of the control object. Note that the predetermined motion is not limited to a motion of peering into the rear, and may be a motion that is predefined as a motion for visually checking the rear. For example, the predetermined motion may be a motion of turning the face in the direction in which the control object is to be moved while looking at the control object.

[0038] The motion determination unit 213 can determine whether or not the user has performed a predetermined motion based on at least one of the movement direction, movement amount, rotation direction, and rotation amount of the HMD 100. For example, the motion determination unit 213 can determine that the user has performed a predetermined motion when the movement amount of the HMD 100 (user's head) is a distance within a predetermined range. Furthermore, the motion determination unit 213 can determine that the user has performed a predetermined motion when the rotation amount of the HMD 100 (user's head) is an angle within a predetermined range. The motion determination unit 213 may determine that the user has performed a predetermined motion when both the movement amount of the user's head is a distance within a predetermined range and the rotation amount is an angle within a predetermined range are satisfied.

[0039] Whether the amount of movement of the user's head is within a predetermined range is determined by a first threshold value that defines the predetermined range. and a second threshold (>first threshold), and it can be determined whether the amount of movement of the user's head is equal to or greater than the first threshold and equal to or less than the second threshold. Whether the amount of rotation of the user's head is within a predetermined angle range can be determined by setting a third threshold and a fourth threshold (>third threshold) that define the predetermined range, and it can be determined whether the amount of rotation of the user's head is equal to or greater than the third threshold and equal to or less than the fourth threshold.

[0040] If the user has performed the predetermined action, the process proceeds to step S306. If the user has not performed the predetermined action, the process returns to step S300.

[0041] In step S306, the object control unit 215 controls the display of the control object selected in step S302 in response to the user performing a predetermined action. The object control unit 215 controls the display of the control object so that the user can see behind the control object. The object control unit 215 controls the display of the control object by performing at least one of the display controls of movement, reduction, transparency, and non-display on the control object, for example.

[0042] The object control unit 215 transmits data of the control object to be controlled for display and data of the virtual object selected by the object selection unit 214 to the display image generation unit 216. The display image generation unit 216 generates a composite image of the virtual object including the control object and the captured image, and transmits the generated composite image to the display unit 206. The display unit 206 displays the composite image received from the display image generation unit 216 on the display device 102.

[0043] In step S307, the gaze information acquisition unit 210 determines whether the user's gaze position has changed. For example, the gaze information acquisition unit 210 can determine that the user's gaze position has changed when the amount of change from the gaze position acquired in step S300 is greater than a preset threshold. If the user's gaze position has changed, the process proceeds to step S309. If the user's gaze position has not changed, the process proceeds to step S308.

[0044] In step S308, the motion determination unit 213 determines whether or not a predetermined motion by the user has been completed. For example, the motion determination unit 213 determines whether or not a predetermined motion has been completed based on the amount of movement and the amount of rotation from the position and orientation of the HMD 100 (the position and orientation of the user's head) when it is determined in step S305 that the predetermined motion has been performed. The motion determination unit 213 can determine that a predetermined motion has been completed when the amount of movement is greater than a preset threshold value of the amount of movement, when the amount of rotation is greater than a preset threshold value of the amount of rotation, or when both the amount of movement and the amount of rotation are greater than their respective threshold values. When the predetermined motion has been completed, the process proceeds to step S309. When the predetermined motion has not been completed, the display control of the control object is not restored, and the process returns to step S307.

[0045] In step S309, a predetermined action such as the user wearing the HMD 100 actually looking into the control object has ended, and the object control unit 215 returns the display control performed in step S306 to the original state.

[0046] (Example of peeking behavior) An example of determining the peering motion will be described with reference to Figs. 4(A), 4(B), 5, and 6(A) to 6(C). The peering motion is an example of a predetermined motion. In the following example, the movement detection unit 202 includes an accelerometer (three-axis acceleration sensor) that detects acceleration in three axial directions, and the rotation detection unit 203 includes an angular velocity meter (three-axis angular velocity sensor, gyro sensor) that detects angular velocity in three axial directions.

[0047] 4(A) and 4(B) illustrate a three-axis acceleration sensor and a three-axis angular velocity sensor. 4A. The three-axis acceleration sensor of the movement detection unit 202 is mounted on the HMD 100 and detects the acceleration of the HMD 100 in three axial directions. As shown in FIG. 4A, the three-axis acceleration sensor can detect the movement directions of the X-axis (left and right), Y-axis (up and down), and Z-axis (front and back) and the amount of movement in each direction. The output of the three-axis acceleration sensor is transmitted to a movement amount calculation unit 211. The movement amount calculation unit 211 can obtain the movement direction and amount of movement of the HMD 100 based on the output data of the three-axis acceleration sensor.

[0048] The three-axis angular velocity sensor of the rotation detection unit 203 is mounted on the HMD 100 and detects the angular velocity of the HMD 100 in three axial directions. As shown in Fig. 4(B), the three-axis angular velocity sensor can detect the rotation directions of pitch (rotation about an axis in the left-right direction), yaw (rotation about an axis in the up-down direction), and roll (rotation about an axis in the front-rear direction) and the amount of rotation in each direction. The output of the three-axis angular velocity sensor is transmitted to the rotation amount calculation unit 212. The rotation amount calculation unit 212 can obtain the rotation direction and amount of rotation of the HMD 100 based on the output data of the three-axis angular velocity sensor.

[0049] Fig. 5 is a flow chart illustrating a process of determining whether a peek action has been performed. The process shown in Fig. 5 is a specific example of a process of determining whether a predetermined action has been performed in steps S304 and S305 in Fig. 3. Fig. 6 is a diagram for explaining the determination of a peek action. In the following description, the three-axis acceleration sensor and the three-axis angular velocity sensor are referred to as an acceleration sensor and an angular velocity sensor, respectively.

[0050] 5, the movement detection unit 202 acquires the output of the acceleration sensor. The movement detection unit 202 transmits the acquired output of the acceleration sensor to the movement amount calculation unit 211. In step S501, the movement amount calculation unit 211 determines whether or not an initial value of acceleration is stored. The initial value of acceleration is used to calculate the movement direction and movement amount of the HMD 100.

[0051] If the initial value of acceleration has been saved, the process proceeds to step S503. If the initial value of acceleration has not been saved, the process proceeds to step S502. In step S502, the movement amount calculation unit 211 saves the output value of the acceleration sensor acquired in step S500 as the initial value of acceleration. The process returns to step S500.

[0052] In step S503, the rotation detection unit 203 acquires the output of the angular velocity sensor. The rotation detection unit 203 transmits the acquired output of the angular velocity sensor to the rotation amount calculation unit 212. In step S504, the rotation amount calculation unit 212 determines whether or not an initial value of the angular velocity is stored. The initial value of the angular velocity is used to calculate the rotation direction and the rotation amount of the HMD 100.

[0053] If the initial value of the angular velocity has been saved, the process proceeds to step S506. If the initial value of the angular velocity has not been saved, the process proceeds to step S505. In step S505, the rotation amount calculation unit 212 saves the output value of the angular velocity sensor acquired in step S503 as the initial value of the angular velocity. The process returns to step S504.

[0054] In step S506, the movement amount calculation unit 211 acquires a movement direction and a movement amount as movement information based on the amount of change in acceleration from the initial value, and the rotation amount calculation unit 212 acquires a rotation direction and a rotation amount as rotation information based on the amount of change in angular velocity from the initial value.

[0055] In step S507, the motion determination unit 213 determines whether the movement amount of the HMD 100 is within a predetermined range. The distance may be specified by a lower limit and a upper limit. Further, the distance thresholds that define the predetermined range may be set separately for the up-down direction, the left-right direction, and the front-rear direction.

[0056] In the example of FIG. 6(A), the threshold of the amount of movement of the user's head in the forward upward direction is set to the amount of movement A. In addition, in the example of FIG. 6(B), the threshold of the amount of movement of the user's head in the upper left direction or the upper right direction is set to the amount of movement B. Regarding the amount of movement acquired in step S506, the motion determination unit 213 can determine that the amount of movement is within a predetermined range when the amount of movement in the forward upward direction is within the range of 0 to the amount of movement A and the amount of movement in the upper left direction or the upper right direction is within the range of 0 to the amount of movement B. Note that the threshold of the distance for determining the amount of movement is not limited to the examples of FIG. 6(A) and 6(B), and may be set according to the type of motion of the predetermined motion. Also, the threshold of the distance for determining the amount of movement may be set for each user according to the characteristics of the user's motion.

[0057] In step S507, if the amount of movement of the HMD 100 is within the predetermined range, the process proceeds to step S510. If the amount of movement of the HMD 100 is not within the predetermined range, the process proceeds to step S508.

[0058] In step S510, the movement determination unit 213 determines whether or not the rotation amount of the HMD 100 is within a predetermined range of angles. The threshold angle that defines the predetermined range may be specified by an upper limit value and a lower limit value of the rotation angle.

[0059] In the example of FIG. 6(C), the threshold value of the rotation amount of the user's head in the forward left direction or forward right direction is set to the rotation amount C. Regarding the rotation amount acquired in step S506, when the rotation amount in the forward left direction or forward right direction is within the range of 0 to the rotation amount C, the motion determination unit 213 can determine that the rotation amount is an angle within a predetermined range. Note that the angle threshold value for determining the rotation amount is not limited to the example of FIG. 6(C), and may be set according to a predetermined motion. Also, the angle threshold value for determining the rotation amount may be set for each user according to the characteristics of the user's movement.

[0060] In step S510, if the amount of rotation of the HMD 100 is within the predetermined range of angles, the process proceeds to step S511. If the amount of rotation of the HMD 100 is not within the predetermined range of angles, the process proceeds to step S508.

[0061] In step S508, the rotation amount calculation unit 212 updates the initial value of the angular velocity to the output value of the angular velocity sensor acquired in step S503. In step S509, the movement amount calculation unit 211 updates the initial value of the acceleration to the output value of the acceleration sensor acquired in step S500. After the initial values ​​of the angular velocity and acceleration are updated, the process returns to step S500.

[0062] In step S511, the motion determination unit 213 determines that the motion of the user's head is a peek motion. Note that in the process shown in Fig. 5, the motion determination unit 213 determines that a peek motion has been performed when both the movement amount and the rotation amount of the HMD 100 are within their respective predetermined ranges, but is not limited to this. The motion determination unit 213 may determine that a peek motion has been performed if the movement amount is a distance within a predetermined range, or may determine that a peek motion has been performed if the rotation amount is an angle within a predetermined range.

[0063] 5, the motion determination unit 213 determines whether the motion of the user's head is a peek motion based on whether the movement amount and rotation amount of the HMD 100 satisfy a predetermined condition, but may further use conditions for the movement direction and rotation direction to make the determination. For example, the motion determination unit 213 determines that the motion is a peek motion when the movement amount is a distance within a predetermined range and the movement direction is in front of the user, and determines that the motion is a peek motion when the movement amount is a distance within a predetermined range and the movement direction is in front of the user. In addition, even if the amount of rotation is within a predetermined range of angles, if the direction of rotation is a direction in which the user turns around, the motion determination unit 213 may determine that the motion is not a peek motion.

[0064] Whether or not a specific action including a peering action has been performed can be determined by comparing at least one of the movement direction, the movement amount, the rotation direction, and the rotation amount with a preset threshold value for each of the movement direction, the movement amount, the rotation direction, and the rotation amount. The threshold values ​​for the movement direction, the movement amount, the rotation direction, and the rotation amount are set according to the specific action to be determined.

[0065] (Metadata indicating display control method) The display control of the control object will be described with reference to Fig. 7. In response to a predetermined action by the user, the control object is subjected to one of the following controls (display control): movement, reduction, transparency, and non-display. The correspondence between the type of object and the type of display control to be applied can be determined in advance. The type of object is set according to, for example, whether it is movable or not, whether it is reducible or not, and so on.

[0066] The type of display control to be applied may differ depending on the type of control object. When different display controls are applied to each control object, which display control among movement, reduction, transparency, and non-display is applied can be added to the data of each control object as metadata. In response to a predetermined action by the user, the control object is stored in the object holding unit 208 in association with metadata indicating which display control among movement, reduction, transparency, and non-display is applied. The object control unit 215 can control the display of the control object based on the metadata.

[0067] 7 is a diagram illustrating metadata assigned to an object. Depending on the type, a virtual object may or may not be expected to move. Therefore, by having a display control method as metadata for a virtual object, the object control unit 215 can appropriately control the display of the control object by referring to the metadata of the control object.

[0068] In the example of Fig. 7, the objects "doll" and "car" can be moved in real space as well. Therefore, the display control method for "movement" is assigned to the objects "doll" and "car" as metadata. The assigned metadata is stored in the object storage unit 208 in association with the data of the virtual object.

[0069] Since the objects "Building" and "Wall" are not expected to move in real space, the user may feel uncomfortable when the objects "Building" and "Wall" are moved. Therefore, the objects "Building" and "Wall" are assigned a display control method of processing "Transparent" as metadata. The assigned metadata is stored in the object storage unit 208 in association with the data of the virtual object. The display control method is not limited to "Movement" and "Transparent" shown in FIG. 7. The display control method may be any control that allows the user to view the rear of the control object, and may include controls such as shrinking, rotating, and hiding.

[0070] (Display control of control object) Display control of a control object will be described with reference to Fig. 8, Fig. 9(A) to (C), and Fig. 10(A) to (D). Fig. 8 is a flowchart illustrating a display control process of a control object. In the example shown in Fig. 8, the movement detection unit 202 includes a three-axis acceleration sensor, and the rotation detection unit 203 includes a three-axis angular velocity sensor.

[0071] The process shown in Fig. 8 is a specific example of the process of step S306 in Fig. 3. The HMD 100 executes the same processes as steps S300 to S304 in Fig. 3, and in step S305, as an example of a predetermined action, determines whether or not the user has performed a peering action. The display control process of the control object shown in Fig. 8 is executed when it is determined that the user has performed a peering action.

[0072] In step S801, the motion determination unit 213 acquires the movement direction of the HMD 100 (user's head) from the movement amount calculation unit 211. The movement direction of the HMD 100 is used to control the display of the control object. The motion determination unit 213 transfers the acquired movement direction of the HMD 100 to the object control unit 215.

[0073] In step S802, the motion determination unit 213 acquires the amount of movement of the HMD 100 from the movement amount calculation unit 211. The amount of movement of the HMD 100 is used to control the display of the control object. The motion determination unit 213 transfers the acquired amount of movement of the HMD 100 to the object control unit 215.

[0074] In step S803, the object control unit 215 determines whether the control method for displaying the control object is "movement". The object control unit 215 can acquire the control method for displaying the control object from the metadata described with reference to Fig. 7. If the control method for displaying the control object is "movement", the process proceeds to step S804.

[0075] If the control method for displaying the control object is not "movement", the process proceeds to step S814. In step S814, the object control unit 215 controls the control object to be transparent so that the rear can be seen. The object control unit 215 can make the control object transparent by increasing the transparency of the control object. The object control unit 215 may set the transparency of the control object within a range that allows the user to see behind. Note that, although the example of FIG. 10 shows an example of making the control object transparent, the control method for the control object may be a control other than the transparency specified by the metadata.

[0076] In step S804, the object control unit 215 calculates the amount of movement of the control object. The object control unit 215 can determine the distance to move the control object based on at least one of the amount of movement and the amount of rotation of the user's head, for example. The object control unit 215 may calculate the amount of movement of the control object so that the control object is moved appropriately according to the display angle size of the display device 102.

[0077] The determination of the movement direction of the control object in steps S805 to S807 will be described with reference to Figures 9(A) to 9(C). Figures 9(A) to 9(C) are diagrams for explaining the movement direction of the control object. Figure 9(A) shows a display example in which a user wearing an HMD 100 is attempting to visually recognize the rear of a control object 900 (a display on the back). Figure 9(A) shows a state in which the user's gaze is directed toward a gaze position 901 for a period of time longer than a predetermined period of time t.

[0078] In steps S805 to S807, the object control unit 215 determines the direction in which to move the control object 900 based on the movement direction of the HMD 100 acquired in step S801. In the example of Fig. 8, the peering action performed by the user is either a peering action from the right side or a peering action from the left side.

[0079] In step S805, the object control unit 215 It is determined whether the user has performed a motion of peeking in from the right side of the control object. If the user has performed a motion of peeking in from the right side of the control object, the process proceeds to step S806. If the user's motion is not a motion of peeking in from the right side, the process proceeds to step S807.

[0080] In step S806, the object control unit 215 determines the moving direction of the control object to the left. When the moving direction of the control object is determined to be to the left, the object control unit 215 controls the control object 900 to move to the left (in the direction indicated by the arrow) as shown in Fig. 9(B). By moving the control object 900 to the left, the user becomes able to view the virtual object 902 and real space behind.

[0081] In step S807, it is assumed that the user wearing the HMD 100 has performed an action of peering in from the left side of the control object, and the object control unit 215 determines the movement direction of the control object to the right. When the movement direction of the control object is determined to be to the right, the object control unit 215 controls the control object 900 to move to the right (in the direction indicated by the arrow) as shown in Fig. 9(C). By moving the control object 900 to the right, the user becomes able to view the virtual object 902 and real space behind.

[0082] In the example of steps S805 to S807, the predetermined action has been described as an action of peeking in from the right or left side, but is not limited to this. For example, if the predetermined action is an action of peeking in from above, the object control unit 215 can determine the movement direction of the control object to be downward. If the predetermined action is an action of peeking in from below, the object control unit 215 can determine the movement direction of the control object to be upward.

[0083] In the example of steps S805 to S807, the moving direction of the control object is determined based on the moving direction of the HMD 100, but may be determined taking into consideration the rotation direction of the HMD 100. By taking into consideration the rotation direction of the HMD 100, the object control unit 215 can accurately detect from which direction the user is peering into the rear of the control object. The object control unit 215 may perform control so as to move the control object in the direction opposite to the direction the user is peering into the control object. In this way, the object control unit 215 can determine the direction in which to move the control object based on at least one of the moving direction and the rotating direction of the HMD 100 (user's head).

[0084] The display control of the control object in steps S808 to S814 will be described with reference to Figures 10(A) to 10(D). Figures 10(A) to 10(D) are diagrams for explaining the display control of the control object. Specifically, when another virtual object exists in the direction in which the control object is moved, the object control unit 215 performs control to reduce the control object so that it does not overlap with the other virtual object.

[0085] Fig. 10(A) shows a display example in which a user wearing an HMD 100 is trying to view the rear (a display on the back) of a control object 1000. Fig. 10(A) shows a state in which the user's gaze is directed to a gaze position 1001 for a time longer than a predetermined time t. The HMD 100 displays a virtual object 1002 different from the control object 1000.

[0086] In steps S808 to S814, object control unit 215 controls the display of the control object. Object control unit 215 changes the control method of the control object depending on whether or not another virtual object exists in the direction in which the control object is moved.

[0087] In step S808, the object control unit 215 determines whether or not another virtual object exists in the direction in which the control object is moved. If another virtual object exists in the moving direction, the process proceeds to step S809. If another virtual object does not exist in the moving direction, the process proceeds to step S813.

[0088] In step S813, since there is no other virtual object in the direction in which the control object is to be moved, the object control unit 215 moves the control object based on the movement amount calculated in step S804. In the example of Fig. 10(B), when the user wearing the HMD 100 peers into the control object 1000 from the left side, the control object 1000 is moved to the right. Since there is no other virtual object to the right of the control object 1000, the control object 1000 does not overlap with other virtual objects such as the virtual object 1002 even if it is moved.

[0089] Steps S809 to S812 are processes performed when another virtual object exists in the direction in which the control object is moved. When another virtual object exists in the direction in which the control object is moved, the object control unit 215 reduces the size of the control object at the destination of the movement. In the example of FIG. 10C, when the user wearing the HMD 100 peers into the control object 1000 from the right side, the control object 1000 moves to the left and overlaps with the other virtual object 1002.

[0090] 10(D), the object control unit 215 reduces the size of the control object 1000 so that the control object 1000 does not overlap with other virtual objects 1002 even when the control object 1000 is moved. However, in order to prevent the reduction from making the object hard to see, a threshold value for the reduction ratio is set in the example of the process shown in FIG.

[0091] In step S809, the object control unit 215 sets a reduction ratio threshold value α (a predetermined reduction ratio). The threshold value α is set within a range in which the control object is visible, based on, for example, the type of the control object (whether or not it includes text) or the size of the display screen. In step S810, the object control unit 215 calculates a reduction ratio β for preventing the control object from overlapping with other virtual objects at the destination.

[0092] In step S811, the object control unit 215 determines whether the reduction rate β is equal to or greater than the threshold value α. If the reduction rate β is equal to or greater than the threshold value α, the control object will not overlap with other virtual objects if it is reduced at the reduction rate of the threshold value α. In this case, the process proceeds to step S812. On the other hand, if the reduction rate β is less than the threshold value α, the control object will overlap with other virtual objects even if it is reduced at the reduction rate of the threshold value α. In this case, the process proceeds to step S814.

[0093] In step S812, the control object is visible even if it is reduced at the reduction ratio β at the movement destination, and the object control unit 215 controls the control object 1000 to be reduced and moved, as shown in FIG. 10(D).

[0094] In step S814, the reduction ratio β is smaller than the threshold value α, and the control object becomes difficult to see when reduced at the reduction ratio β. That is, even if the control object is reduced at the reduction ratio of the threshold value α so that it becomes visible, the control object will overlap with other virtual objects. Therefore, the object control unit 215 controls the control object not to move but to be transparent so that the rear can be seen.

[0095] In steps S812 to S814, the object control unit 215 Controlling the display allows the user to see behind the control object.

[0096] The object control unit 215 transmits data of the virtual object selected by the object selection unit 214, including data of the virtual object whose display has been controlled, to the display image generation unit 216. The display image generation unit 216 generates a composite image in which the virtual object is superimposed on the captured image. The display unit 206 displays the composite image generated by the display image generation unit 216 on the display device 102.

[0097] When the display control process of the control object shown in Fig. 8 is completed, it is determined whether the user's line of sight has changed or a predetermined action has been completed, similarly to steps S307 and S308 in Fig. 3. If the user's line of sight has changed or a predetermined action has been completed, object control unit 215 returns the control of the display of the control object performed in steps 812 to S814 to the original state, similarly to S309.

[0098] (Highlighting Control Objects) The highlighting of the control object will be described with reference to Figures 11(A) and 11(B). The control object is a virtual object that exists at the user's gaze position, and the display is controlled by the user performing a predetermined action. The object control unit 215 highlights the control object selected in step S302 of Figure 3, thereby allowing the user to recognize the virtual object to be controlled.

[0099] In Fig. 11(A), when the gaze of the user who is wearing the HMD 100 is directed to gaze position 1101 for a period of time longer than a predetermined time t, the object control unit 215 controls to select a control object 1100 as a control target and to highlight the control object. For example, as shown in Fig. 11(B), the object control unit 215 can perform highlighting 1102 by lighting the outline of the control object 1100. Note that the method of highlighting is not limited to lighting the outline of the control object. The object control unit 215 only needs to control the display so that the user can easily recognize it, and may control to blink the control object itself, for example.

[0100] The object control unit 215 may perform control so as to highlight the control object from the time the control object is selected in step S302 in Fig. 3 until control of the display of the control object is started. Also, the object control unit 215 may perform control so as to highlight the control object from the time the control object is selected until a predetermined operation is completed.

[0101] According to the above embodiment, the HMD 100 controls the display of a virtual object present at the user's line of sight in response to the user performing a predetermined action, thereby enabling the user to view the rear of the virtual object. That is, the HMD 100 controls the display of the virtual object so that the display of the virtual object is different before and after the user wearing the HMD 100 performs a predetermined action for viewing the rear of the virtual object present at the user's line of sight. The user can easily view the rear of the virtual object by, for example, looking into the rear of the virtual object as the predetermined action.

[0102] The above-described embodiment is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0103] Each functional unit in the above embodiment (FIG. 2) may be an individual piece of hardware. This need not be the case. The functions of two or more functional units may be realized by common hardware. Each of the multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware. For example, the device may have a processor and a memory in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0104] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0105] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) A selection means for selecting a virtual object present at a gaze position of a user wearing the head-mounted display device as a control object; a determination means for determining whether or not the user has performed a predetermined action to visually check the rear of the control object; a control means for controlling display of the control object in response to the user performing the predetermined action so that the rear of the control object can be seen; 13. An information processing device comprising: (Configuration 2) The determination means determines whether or not the user has performed the predetermined motion based on at least one of a moving direction, a moving amount, a rotating direction, and a rotating amount of the user's head. 2. The information processing device according to configuration 1. (Configuration 3) The determining means determines that the user has performed the predetermined motion when the amount of movement of the user's head is within a predetermined range. 3. The information processing device according to configuration 2. (Configuration 4) The determining means determines that the user has performed the predetermined motion when an amount of rotation of the user's head is within a predetermined range of angles. 4. The information processing device according to configuration 2 or 3. (Configuration 5) The control means performs at least one of display control of the control object among movement, reduction, transparency, and non-display in response to the user performing the predetermined action. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The control means determines a direction in which to move the control object based on at least one of a moving direction and a rotating direction of the user's head. 6. The information processing device according to configuration 5. (Configuration 7) The control means determines a distance to move the control object based on at least one of an amount of movement and an amount of rotation of the user's head. 7. The information processing device according to configuration 5 or 6. (Configuration 8) The control means is configured to detect whether another virtual object exists in a direction in which the control object is moved. If there is, the control object is controlled to be reduced so that it does not overlap with the other virtual objects. 8. The information processing device according to any one of configurations 5 to 7. (Configuration 9) The control means controls the control object so as to be transparent when the control object overlaps with the other virtual object even if the control object is reduced at a predetermined reduction rate. 9. The information processing device according to configuration 8. (Configuration 10) A correspondence relationship between the type of object and the type of display control is determined in advance; The control means performs a display control on the control object, which corresponds to the type of the control object, among movement, reduction, transparency, and non-display. 10. The information processing device according to any one of configurations 5 to 9. (Configuration 11) The control means controls to highlight the control object selected by the selection means. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) The control means performs control so as to highlight the control object during a period from when the control object is selected by the selection means until display control of the control object is started or until the predetermined operation is completed. 12. The information processing device according to configuration 11. (Configuration 13) A control means for controlling display of a virtual object visually recognized by wearing the head-mounted display device, The display of a virtual object that is present at a line of sight of a user wearing the head-mounted display device is different before and after the user performs a predetermined action to visually recognize the rear of the virtual object that is present at the user's line of sight. 23. An information processing apparatus comprising: (Configuration 14) An information processing device according to any one of configurations 1 to 12, a display means for displaying an image in which the display of the control object is controlled; A head-mounted display device comprising: (method) selecting a virtual object present at a gaze position of a user wearing the head-mounted display device as a control object; determining whether the user has performed a predetermined action to view behind the control object; In response to the user performing the predetermined action, controlling display of the control object so that an area behind the control object is visible; 13. An information processing method comprising: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 13. (medium) 14. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 13. [Explanation of symbols]

[0106] 100: Head-mounted display device (HMD), 210: Gaze information acquisition unit, 213: Motion determination Part 215: Object control part

Claims

1. A selection means for selecting a virtual object present at a gaze position of a user wearing the head-mounted display device as a control object; a determination means for determining whether or not the user has performed a predetermined action to visually check the rear of the control object; a control means for controlling display of the control object in response to the user performing the predetermined action so that the rear of the control object can be seen; 13. An information processing device comprising:

2. The determination means determines whether or not the user has performed the predetermined motion based on at least one of a moving direction, a moving amount, a rotating direction, and a rotating amount of the user's head.

2. The information processing apparatus according to claim 1,

3. The determining means determines that the user has performed the predetermined motion when the amount of movement of the user's head is within a predetermined range.

3. The information processing apparatus according to claim 2.

4. The determining means determines that the user has performed the predetermined motion when an amount of rotation of the user's head is within a predetermined range of angles.

3. The information processing apparatus according to claim 2.

5. The control means performs at least one of display control of the control object among movement, reduction, transparency, and non-display in response to the user performing the predetermined action.

2. The information processing apparatus according to claim 1,

6. The control means determines a direction in which to move the control object based on at least one of a moving direction and a rotating direction of the user's head.

6. The information processing apparatus according to claim 5,

7. The control means determines a distance to move the control object based on at least one of an amount of movement and an amount of rotation of the user's head.

6. The information processing apparatus according to claim 5,

8. When another virtual object exists in a direction in which the control object is moved, the control means performs control to reduce the control object so that the control object does not overlap with the other virtual object.

6. The information processing apparatus according to claim 5,

9. The control means controls the control object so as to be transparent when the control object overlaps with the other virtual object even if the control object is reduced at a predetermined reduction rate.

9. The information processing apparatus according to claim 8,

10. A correspondence relationship between the type of object and the type of display control is determined in advance; The control means performs a display control on the control object, which corresponds to the type of the control object, among movement, reduction, transparency, and non-display.

6. The information processing apparatus according to claim 5,

11. The control means controls to highlight the control object selected by the selection means.

2. The information processing apparatus according to claim 1,

12. The control means performs control so as to highlight the control object during a period from when the control object is selected by the selection means until display control of the control object is started or until the predetermined operation is completed.

12. The information processing apparatus according to claim 11 .

13. A control means for controlling display of a virtual object visually recognized by wearing the head-mounted display device, The display of a virtual object that is present at a line of sight of a user wearing the head-mounted display device is different before and after the user performs a predetermined action to visually recognize the rear of the virtual object that is present at the user's line of sight.

23. An information processing apparatus comprising:

14. An information processing device according to any one of claims 1 to 12, a display means for displaying an image in which the display of the control object is controlled; A head-mounted display device comprising:

15. selecting a virtual object present at a gaze position of a user wearing the head-mounted display device as a control object; determining whether the user has performed a predetermined action to view behind the control object; In response to the user performing the predetermined action, controlling display of the control object so that an area behind the control object is visible; 13. An information processing method comprising:

16. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 13.

17. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 13.

Images