Information processing apparatus, head-mounted display, information processing method, computer program product, and computer-readable storage medium

Abstract

The invention relates to an information processing apparatus, a head-mounted display, an information processing method, a computer program product, and a computer readable storage medium. The information processing apparatus includes: a selection unit configured to select, as a control object, a virtual object present at a line-of-sight position of a user; a determination unit configured to determine whether or not a predetermined operation for visually recognizing a rear of the control object is performed by the user; and a control unit configured to perform control for moving and displaying the control object such that a rear portion of the control object can be visually recognized, in which the control unit determines a direction in which the control object is to be moved based on at least any one of a moving direction and a rotating direction of a head of the user, and displays the control object in the direction in which the control object is to be moved based on the determined direction. Or a distance at which the control object is to be moved is determined on the basis of at least any one of the amount of movement and the amount of rotation of the head of the user.

Description

Technical Field

[0001] The invention relates to an information processing device, a head-mounted information display and an information processing method. Background Art

[0002] Recently, XR (Extended Reality) technology, which creates new experiences by fusing the real world and the virtual world, has become known. XR technology is a general term for technologies of virtual reality, mixed reality, augmented reality, and alternative reality.

[0003] Virtual reality (VR) technology is a technology that enables users to experience virtual space as if it were real. Mixed reality (MR) technology is a technology that enables users to experience a mixed sense of reality by seamlessly blending reality and virtual reality. Augmented reality (AR) is a technology that shows real space by overlapping the virtual world and projecting it. Alternative reality (SR) is a technology that superimposes past images on the real world and projects them for viewing.

[0004] As one of the XR technologies, there is an XR system using a head mounted display (HMD). The XR system can experience an XR space by superimposing and displaying virtual objects and the like on an actual outdoor landscape and presenting it to a user wearing the HMD.

[0005] The HMD has a function for detecting the user's line of sight position, and can perform display control using information on the line of sight position. For example, Japanese Patent Laid-Open No. 2016-81339 discloses that a virtual object related to an actual object performing a gaze operation is displayed according to the continuation of the gaze operation within a predetermined reference time. Japanese Patent Laid-Open No. 2016-82466 discloses that an image visually recognized by a user changes according to a combination of the direction of the user's head and the line of sight direction.

[0006] When the user wearing the HMD wants to temporarily visually recognize the rear of the virtual object, the user moves the virtual object through a user interface such as a gesture, or the user actually moves to a position where the rear of the virtual object can be seen. However, there are cases where it is difficult to confirm the rear of the virtual object, such as when the range of movement is limited, when it is desired to perform control without changing the standing position, or when the hands and feet cannot move freely. Summary of the invention

[0007] The present invention provides an information processing device that enables a user wearing a head mounted display to easily visually recognize what is behind a virtual object.

[0008] According to the present invention, the information processing device includes: a selection unit, which is configured to select a virtual object existing at the line of sight of a user wearing a head-mounted display as a control object; a judgment unit, which is configured to judge whether the user has performed a predetermined operation for visually identifying the rear of the control object; and a control unit, which is configured to execute control for moving and displaying the control object in response to the user's performance of the predetermined operation, so that the rear of the control object can be visually identified, wherein the control unit determines the direction in which the control object is to be moved based on at least any one of the movement direction and the rotation direction of the user's head, or determines the distance by which the control object is to be moved based on at least any one of the movement amount and the rotation amount of the user's head.

[0009] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram illustrating the appearance of an HMD;

[0011] Figure 2 is a block diagram showing a configuration example of an HMD;

[0012] Figure 3 is a flowchart illustrating the processing of the HMD;

[0013] Figure 4A is a diagram for explaining a three-axis acceleration sensor;

[0014] Figure 4B is a diagram for explaining a three-axis angular velocity sensor;

[0015] Figure 5 is a flowchart illustrating a determination process of an observation operation;

[0016] FIG. 6A to FIG. 6C It is a diagram for explaining the judgment of the observation operation;

[0017] Figure 7 is a diagram used to illustrate metadata given to an object;

[0018] Figure 8 is a flowchart illustrating a display control process of a control object;

[0019] 9A to 9C It is a diagram used to illustrate the direction of movement of the controlled object;

[0020] FIG. 10A to FIG. 10D is a diagram for explaining display control of a control object; and

[0021] Fig.11A and Fig. 11B It is a diagram for explaining highlighting of a control object. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0023] (Device Configuration)

[0024] Will be used by Figure 1 and Figure 2 A configuration example of the HMD 100 as the head mounted display according to the present embodiment will be described. Figure 1 2 is a diagram illustrating the appearance of the HMD 100 . Figure 1 The HMD 100 shown in FIG. 1 is a display device worn on the head of the user. The HMD 100 includes a power switch and an operation member such as a button for controlling the settings of the device. The user can operate the HMD 100 via the operation member or a controller capable of communicating with the HMD 100. The HMD 100 has an image processing unit for performing generation of a virtual object, synthesis of an image, and the like.

[0025] The head mounted member 101 is a member for stably fixing the HMD 100 on the user's head. The head mounted member 101 fixes the HMD 100 to the user's head so that the HMD 100 can be operated without being displaced according to the movement of the user's head. Note that the head mounted member 101 is not limited to a member for fixing the HMD 100 to the user's head, but may be a member for fixing the HMD 100 by hanging the HMD 100 on the user's ear.

[0026] The display device 102 displays a virtual object or a composite image of an image of a real space and a virtual object in front of the eyes of the user who is the wearer of the HMD 100. The display device 102 is a liquid crystal display or an organic EL (electroluminescence) display. The display device 102 includes, for example, a liquid crystal panel, a drive circuit for controlling the liquid crystal panel, and a memory for holding an image to be displayed. The display device 102 may be a non-transmissive display unit, or may be an optically transmissive display unit capable of visually recognizing a real space through a lens.

[0027] The photographing device 103 is a device (camera unit) for photographing the environment around the user. The photographing device 103 can photograph the environment in front of the user as the environment around the user. The front of the user can be, for example, the front of the user's head.

[0028] Figure 2 is a block diagram showing a configuration example of the HMD 100 . Figure 2The HMD 100 shown in FIG. 1 includes an imaging unit 200, a sight line detection unit 201, a movement detection unit 202, a rotation detection unit 203, an information processing unit 204, an image processing unit 205, and a display unit 206. Note that the information processing unit 204 and the image processing unit 205 will be described as processing units provided in the HMD 100, but the processing of the information processing unit 204 and the image processing unit 205 or part of the processing may be performed by an external information processing device capable of communicating with the HMD 100. In this case, the information processing device performs the processing of the information processing unit 204 and the image processing unit 205, and controls the display of a virtual object displayed by the display unit 206 of the HMD 100.

[0029] The information processing unit 204 includes a camera image acquisition unit 207, an object holding unit 208, an object generation unit 209, a line of sight information acquisition unit 210, a movement amount calculation unit 211, a rotation amount calculation unit 212, and an operation determination unit 213. The image processing unit 205 includes an object selection unit 214, an object control unit 215, and a display image generation unit 216.

[0030] The imaging unit 200 has an optical system, an image sensor, a drive circuit for controlling the image sensor, an A / D conversion circuit for converting a signal acquired by the image sensor into a digital signal, and a development circuit for developing the acquired digital signal as an image. The imaging unit 200 transmits image data (camera image) acquired by imaging to the imaging image acquisition unit 207 of the information processing unit 204. The imaging image acquisition unit 207 transmits the image data received from the imaging unit 200 to the display image generation unit 216 of the image processing unit 205. In addition, the imaging image acquisition unit 207 transmits the imaging image to the object generation unit 209.

[0031] The sight line detection unit 201 detects the sight line direction of the user who is the wearer of the HMD 100. The sight line detection unit 201 is, for example, a device used for a single-lens reflex camera or the like, and may be a device that irradiates the user's eyeball with an infrared light emitting diode and detects the sight line direction based on the relationship between a reflected image reflected by the cornea of ​​the light source and the pupil.

[0032] The line-of-sight information acquisition unit 210 of the information processing unit 204 acquires information of the line of sight detected by the line-of-sight detection unit 201 (hereinafter also described as line-of-sight information). The line-of-sight information acquisition unit 210 transmits the acquired line-of-sight information to the object control unit 215 .

[0033] The movement detection unit 202 detects the movement of the HMD 100. The movement detection unit 202 can detect the movement of the HMD 100 by using position information of a GPS (Global Positioning System) and an output of an acceleration sensor, for example.

[0034] The movement amount calculation unit 211 of the information processing unit 204 acquires information about the movement of the HMD 100 detected by the movement detection unit 202, and calculates the movement direction and movement amount of the HMD 100. The movement amount calculation unit 211 transmits the calculated movement direction and movement amount of the HMD 100 (hereinafter also described as movement information) to the operation judgment unit 213.

[0035] The rotation detection unit 203 detects the rotation of the HMD 100. For example, the rotation detection unit 203 can detect the rotation of the HMD 100 by using the output of a rotation displacement sensor or the like. The rotation amount calculation unit 212 of the information processing unit 204 acquires information about the rotation detected by the rotation detection unit 203, and calculates the rotation direction and the rotation amount (hereinafter also described as rotation information) of the HMD 100. The rotation amount calculation unit 212 transmits the calculated rotation direction and the rotation amount of the HMD 100 to the operation judgment unit 213.

[0036] The operation judgment unit 213 judges the operation of the HMD 100 based on the movement direction and movement amount of the HMD 100 received from the movement amount calculation unit 211 and the rotation direction and rotation amount of the HMD 100 received from the rotation amount calculation unit 212. The operation judgment unit 213 transmits information related to the judged operation (hereinafter also described as operation information) to the object control unit 215 of the image processing unit 205.

[0037] The object holding unit 208 holds data related to the virtual space, such as data (shape information and position posture information, etc.) of virtual objects (virtual objects) constituting the virtual space and data of light sources emitted into the virtual space, in a storage unit such as a memory, etc. The object selection unit 214 of the image processing unit 205 selects a virtual object to be displayed from the object holding unit 208, and transmits the data of the selected virtual object to the object control unit 215. In addition, the object holding unit 208 may transmit the data of the virtual object to the object generation unit 209 in order to generate other virtual objects.

[0038] The object generation unit 209 generates a virtual object based on the image data transmitted from the photographic image acquisition unit 207 and the data of the virtual object transmitted from the object holding unit 208. The object generation unit 209 transmits the generated virtual object to the object selection unit 214 of the image processing unit 205.

[0039] The object selection unit 214 selects a virtual object to be displayed to the user from the object holding unit 208 and the object generation unit 209. The object selection unit 214 transmits the data of the selected virtual object to be displayed to the object control unit 215. The data of the virtual object to be displayed transmitted to the object control unit 215 includes display image data, configuration position data in the display viewing angle, and overlapping order data with other virtual objects, etc.

[0040] The object control unit 215 controls the display of the virtual object selected by the object selection unit 214 based on the line of sight information received from the line of sight information acquisition unit 210 and the operation information of the HMD 100 received from the operation judgment unit 213. The control of the display of the virtual object includes the movement, size reduction, transparency, and non-display of the virtual object. The object control unit 215 controls the display of the virtual object by changing the data of the virtual object based on the line of sight information and the operation information. The object control unit 215 transmits the data of the virtual object to which the display control is applied to the display image generation unit 216.

[0041] The display image generation unit 216 generates a composite image of the image data obtained by imaging the real space transmitted from the imaging image acquisition unit 207 and the virtual object received from the object control unit 215. The display image generation unit 216 transmits the generated composite image to the display unit 206.

[0042] (HMD processing)

[0043] Will be used by Figure 3 The processing of the HMD 100 according to the present embodiment will be described. Figure 3 100 is a flowchart illustrating the processing of the HMD 100. When the user wearing the HMD 100 turns on the power switch of the HMD 100 and displays the virtual object on the display device 102, the processing starts. Figure 3 The process shown in .

[0044] In step S300, the line of sight information acquisition unit 210 acquires the line of sight position of the user in the display screen of the display device 102 based on the line of sight information of the user detected by the line of sight detection unit 201. The line of sight information acquisition unit 210 transmits the information of the line of sight position of the user to the object control unit 215.

[0045] In step S301, the object control unit 215 determines whether there is a virtual object at the user's line of sight position based on the data of the virtual object selected by the object selection unit 214 and the line of sight information received from the line of sight information acquisition unit 210. If there is a virtual object at the user's line of sight position, the process proceeds to step S302. If there is no virtual object at the user's line of sight position, the process returns to step S300.

[0046] Note that the object control unit 215 may determine whether the virtual object at the user's line of sight is the virtual object displayed in the foremost position. If the virtual object at the user's line of sight is the foremost virtual object, the process proceeds to step S302. If the virtual object at the user's line of sight is not the foremost virtual object, the process returns to step S300.

[0047] In step S302, the object control unit 215 selects the virtual object existing at the sight line position as the control object. In step S303, the sight line information acquisition unit 210 determines whether the visual recognition time for the control object selected in step S302 is longer than the predetermined time t (time). That is, the sight line information acquisition unit 210 determines whether the sight line position toward the control object has changed during the predetermined time t. The predetermined time t may be, for example, a time that can be determined as the user is looking at the control object.

[0048] When the visual recognition time for the control object is longer than the predetermined time t, the process proceeds to step S304. When the visual recognition time for the control object is equal to or shorter than the predetermined time t, the process returns to step S300.

[0049] In step S304, the operation judgment unit 213 judges the operation of the HMD 100. The HMD 100 is fixed to the head of the user, and the operation judgment unit 213 can judge the operation of the user's head by judging the operation of the HMD 100. Specifically, the operation judgment unit 213 acquires the moving direction and the moving amount of the HMD 100 (the user's head) from the movement detection unit 202. In addition, the operation judgment unit 213 acquires the rotation direction and the rotation amount of the HMD 100 (the user's head) from the rotation detection unit 203. The operation judgment unit 213 judges the operation of the HMD 100 (the user) based on at least any one of the moving direction, the moving amount, the rotation direction, and the rotation amount of the HMD 100 (the user's head). Note that the judgment processing of the operation judgment unit 213 at step S304 may be performed before starting the processing in step S303.

[0050] In step S305, the operation determination unit 213 determines whether the user has performed a predetermined operation. The predetermined operation is an operation for the user wearing the HMD 100 to visually recognize the rear of the control object selected in step S302, and is, for example, an operation of actually observing the rear of the control object. Note that the predetermined operation is not limited to the observation operation, but may be an operation predetermined as an operation for visually recognizing the rear. For example, it may be such an operation for turning the face toward the direction in which the control object is to be moved while looking at the control object.

[0051] The operation judgment unit 213 can judge whether the user has performed a predetermined operation based on at least any one of the movement direction, movement amount, rotation direction, and rotation amount of the HMD 100. For example, the operation judgment unit 213 can judge that the user has performed a predetermined operation when the movement amount of the HMD 100 (the user's head) is a distance within a predetermined range. In addition, the operation judgment unit 213 can judge that the user has performed a predetermined operation when the rotation amount of the HMD 100 (the user's head) is an angle within a predetermined range. The operation judgment unit 213 can judge that the user has performed a predetermined operation when both conditions that the movement amount of the user's head is a distance within a predetermined range and the rotation amount is an angle within a predetermined range are satisfied.

[0052] Whether the movement of the user's head is a distance within the predetermined range can be determined by setting a first threshold and a second threshold (> the first threshold) for determining the predetermined range, and by determining whether the movement of the user's head is at least the first threshold and not greater than the second threshold. Whether the rotation of the user's head is an angle within the predetermined range can be determined by setting a third threshold and a fourth threshold (> the third threshold) for determining the predetermined range, and by determining whether the rotation of the user's head is at least the third threshold and not greater than the fourth threshold.

[0053] When the user performs a predetermined operation, the process proceeds to step S306. When the user does not perform a predetermined operation, the process returns to step S300.

[0054] In step S306, the object control unit 215 controls the display of the control object selected in step S302 in response to the user's performance of a predetermined operation. The object control unit 215 controls the display of the control object so that the user can visually recognize the rear of the control object. The object control unit 215 controls the display of the control object by, for example, performing at least any one of the display controls of moving, reducing the size, making it transparent, and not displaying the control object.

[0055] The object control unit 215 transmits the data of the control object as the control target to be displayed and the data of the virtual object selected by the object selection unit 214 to the display image generation unit 216. The display image generation unit 216 generates a composite image including the virtual object of the control object and the camera image, and transmits the generated composite image to the display unit 206. The display unit 206 displays the composite image received from the display image generation unit 216 on the display device 102.

[0056] In step S307, the line of sight information acquisition unit 210 determines whether the line of sight position of the user has changed. For example, the line of sight information acquisition unit 210 may determine that the line of sight position of the user has changed when the amount of change relative to the line of sight position acquired in step S300 is greater than a preset threshold. If the line of sight position of the user has changed, the process proceeds to step S309. If the line of sight position of the user has not changed, the process proceeds to step S308.

[0057] In step S308, the operation judgment unit 213 judges whether the predetermined operation of the user is completed. For example, the operation judgment unit 213 judges whether the predetermined operation is completed based on the movement amount and the rotation amount relative to the position and posture of the HMD 100 (the position and posture of the user's head) when it is judged that the predetermined operation is performed in step S305. When the movement amount is larger than a preset threshold of the movement amount, when the rotation amount is larger than a preset threshold of the rotation amount, or when both the movement amount and the rotation amount are larger than their respective thresholds, the operation judgment unit 213 can judge that the predetermined operation is completed. In the case where the predetermined operation is completed, the process proceeds to step S309. In the case where the predetermined operation has not been completed, the display control of the control object does not return to the original control, but the process returns to step S307.

[0058] In step S309 , the predetermined operation of the user wearing the HMD 100 actually observing the control object is completed, and the object control unit 215 returns the control of the display performed at step S306 to the original control.

[0059] (Judgment example of observation operation)

[0060] Will be used by Figure 4A , Figure 4B , Figure 5 as well as FIG. 6A to FIG. 6C To illustrate an example of judging the observation operation. The observation operation is an example of a predetermined operation. In the following example, the movement detection unit 202 includes an accelerometer (3-axis acceleration sensor) for detecting acceleration in the 3-axis direction, and the rotation detection unit 203 includes an angular velocity meter (3-axis angular velocity sensor, gyro sensor) for detecting angular velocity in the 3-axis direction.

[0061] Figure 4A This is a diagram for explaining a three-axis acceleration sensor. Figure 4B 2 is a diagram for explaining a 3-axis angular velocity sensor. The 3-axis acceleration sensor of the movement detection unit 202 is mounted on the HMD 100, and detects the acceleration of the HMD 100 in the 3-axis directions. Figure 4A As shown in FIG. 1 , the 3-axis acceleration sensor can detect the movement direction on the X-axis (left and right), the Y-axis (up and down), and the Z-axis (front and back) and the movement amount in each direction. The output of the 3-axis acceleration sensor is transmitted to the movement amount calculation unit 211. The movement amount calculation unit 211 can obtain the movement direction and movement amount of the HMD 100 based on the output data of the 3-axis acceleration sensor.

[0062] The 3-axis angular velocity sensor of the rotation detection unit 203 is mounted on the HMD 100, and detects the angular velocity of the HMD 100 in the 3-axis directions. Figure 4B As shown, the 3-axis angular velocity sensor can detect the rotation direction and the rotation amount in each direction of pitch (rotation with the left and right direction as the axis), yaw (rotation with the up and down direction as the axis), and roll (rotation with the front and back direction as the axis). The output of the 3-axis angular velocity sensor is transmitted to the rotation amount calculation unit 212. The rotation amount calculation unit 212 can obtain the rotation direction and rotation amount of the HMD 100 based on the output data of the 3-axis angular velocity sensor.

[0063] Figure 5 : is a flowchart illustrating the determination process of the observation operation. Figure 5 The process shown in is for Figure 3 A specific example of the process of determining whether a predetermined operation has been performed at steps S304 and S305 in FIG. FIG. 6A to FIG. 6C 2 is a diagram for explaining determination of observation operation. In the following description, a 3-axis acceleration sensor and a 3-axis angular velocity sensor are described as an acceleration sensor and an angular velocity sensor, respectively.

[0064] exist Figure 5 At step S500 in the above, the movement detection unit 202 acquires the output of the acceleration sensor. The movement detection unit 202 transmits the acquired output of the acceleration sensor to the movement amount calculation unit 211. At step S501, the movement amount calculation unit 211 determines whether to maintain the initial value of the acceleration. The initial value of the acceleration is used to calculate the movement direction and movement amount of the HMD 100.

[0065] If the initial value of acceleration is maintained, the process proceeds to step S503. If the initial value of acceleration is not maintained, the process proceeds to step S502. At step S502, the movement amount calculation unit 211 maintains the output value of the acceleration sensor acquired at step S500 as the initial value of acceleration. The process returns to step S500.

[0066] At step S503, the rotation detection unit 203 acquires the output of the angular velocity sensor. The rotation detection unit 203 transmits the acquired output of the angular velocity sensor to the rotation amount calculation unit 212. At step S504, the rotation amount calculation unit 212 determines whether to maintain the initial value of the angular velocity. The initial value of the angular velocity is used to calculate the rotation direction and rotation amount of the HMD 100.

[0067] If the initial value of the angular velocity is maintained, the process proceeds to step S506. If the initial value of the angular velocity is not maintained, the process proceeds to step S505. At step S505, the rotation amount calculation unit 212 maintains the output value of the angular velocity sensor acquired at step S503 as the initial value of the angular velocity. The process returns to step S504.

[0068] At step S506, the movement amount calculation unit 211 acquires the movement direction and movement amount as movement information based on the change amount relative to the initial value of acceleration. In addition, the rotation amount calculation unit 212 acquires the rotation direction and rotation amount as rotation information based on the change amount relative to the initial value of angular velocity.

[0069] At step S507, the operation judgment unit 213 judges whether the movement amount of the HMD 100 is a distance within a predetermined range. The threshold for determining the distance within the predetermined range only needs to be specified by the upper limit value and the lower limit value of the distance in the moving direction. In addition, the threshold for determining the distance within the predetermined range can be set in the up-down direction, the left-right direction, and the front-back direction, respectively.

[0070] exist Fig. 6A In the example of , the threshold value of the amount of movement of the user's head in the upper front direction is set to the amount of movement A. Figure 6B In the example of FIG. 5 , the threshold of the amount of movement of the user's head in the upper left direction or the upper right direction is set to the amount of movement B. Regarding the amount of movement acquired at step S506, if the amount of movement in the upper front direction is within the range from 0 to the amount of movement A and the amount of movement in the upper left direction or the upper right direction is within the range from 0 to the amount of movement B, the operation judgment unit 213 can judge that the amount of movement is a distance within the predetermined range. Note that the threshold for judging the distance of the amount of movement is not limited to Fig. 6A and Figure 6BIn the example in, it can be set according to what the predetermined operation is. In addition, the threshold value for judging the distance of the movement amount can be set for each user according to the characteristics of the user's movement.

[0071] At step S507, if the movement amount of the HMD 100 is a distance within the predetermined range, the process proceeds to step S510. In the case where the movement amount of the HMD 100 is not a distance within the predetermined range, the process proceeds to step S508.

[0072] At step S510, the operation determination unit 213 determines whether the rotation amount of the HMD 100 is an angle within a predetermined range. The threshold for determining the predetermined range of angles only needs to be specified by the upper limit value and the lower limit value of the rotation angle.

[0073] exist Figure 6C In the example of FIG. 5 , the threshold of the amount of rotation of the user's head in the left front direction or the right front direction is set to the rotation amount C. Regarding the rotation amount acquired at step S506, if the amount of rotation in the left front direction or the right front direction is within the range from 0 to the rotation amount C, the operation judgment unit 213 can judge that the rotation amount is an angle within a predetermined range. Note that the threshold for judging the angle of the rotation amount is not limited to Figure 6C In addition, the threshold value of the angle used to determine the rotation amount can be set for each user according to the characteristics of the user's action.

[0074] At step S510, if the rotation amount of the HMD 100 is an angle within the predetermined range, the process proceeds to step S511. In a case where the rotation amount of the HMD 100 is not an angle within the predetermined range, the process proceeds to step S508.

[0075] At step S508, the rotation amount calculation unit 212 updates the initial value of the angular velocity to the output value of the angular velocity sensor acquired at step S503. At step S509, the movement amount calculation unit 211 updates the initial value of the acceleration to the output value of the acceleration sensor acquired at step S500. When the initial value of the angular velocity and the initial value of the acceleration are updated, the process returns to step S500.

[0076] At step S511, the operation determination unit 213 determines that the operation of the user's head is an observation operation. Figure 5 In the process shown in , when both the movement amount and the rotation amount of the HMD 100 are within respective predetermined ranges, the operation judgment unit 213 judges that the observation operation is performed, but this is not restrictive. The operation judgment unit 213 may judge that the observation operation is performed when the movement amount is a distance within the predetermined range, or may judge that the observation operation is performed when the rotation amount is an angle within the predetermined range.

[0077] In addition, Figure 5 In the process shown in , the operation judgment unit 213 judges whether the operation of the user's head is an observation operation by whether the movement amount and the rotation amount of the HMD 100 satisfy a predetermined condition, but the judgment can be made by further using conditions related to the movement direction and the rotation direction. For example, if the movement amount is a distance within a predetermined range and the movement direction is in front of the user, the operation judgment unit 213 judges that this is an observation operation, and if the movement amount is a distance within a predetermined range and the movement direction is behind the user, the operation judgment unit 213 judges that this is not an observation operation. In addition, even if the rotation amount is an angle within a predetermined range, in the case where the rotation direction is the direction when the user faces backward, the operation judgment unit 213 can judge that this is not an observation operation.

[0078] By comparing at least any one of the moving direction, the moving amount, the rotating direction, and the rotating amount with a threshold value pre-set for each of them, it is possible to determine whether a predetermined operation including an observation operation is performed. The threshold value for each of the moving direction, the moving amount, the rotating direction, and the rotating amount is set according to the predetermined operation of the determination target.

[0079] (Metadata indicating the display control method)

[0080] Will be used by Figure 7 The display control of the control object is described as follows. In response to the user's performance of a predetermined operation, any one of the controls (display control) of moving, reducing the size, being transparent, and not displaying is applied to the control object. The correspondence between the type of the object and the type of display control to be applied can be predetermined. The type of the object is, for example, a type set according to whether the object is movable and whether the size can be reduced, etc.

[0081] What display control is to be applied may differ depending on the type of the control object. When display control that differs depending on the control object is to be applied, which display control among movement, size reduction, transparency, and non-display is to be applied may be given as metadata to the data of each control object. The control object is stored in the object holding unit 208 in correspondence with the metadata, which indicates which display control among movement, size reduction, transparency, and non-display is to be applied in response to the user's performance of a predetermined operation. The object control unit 215 may control the display of the control object based on the metadata.

[0082] Figure 72 is a diagram for explaining metadata given to an object. Depending on the type of virtual object, there are cases where it is assumed to move and cases where it is assumed not to move. Therefore, when a virtual object has a display control method as metadata, the object control unit 215 can appropriately control the display of the control object by referring to the metadata of the control object.

[0083] exist Figure 7 In the example in , the objects "doll" and "car" can also move in the real space. Therefore, the display control method of "movement" is given to the objects "doll" and "car" as metadata. The given metadata is associated with the data of the virtual object and is stored in the object holding unit 208.

[0084] Assuming that the "building" and "wall" of the object do not move in the real space, there is a possibility that the user will feel strange if the "building" and "wall" of the object are moved. Therefore, the display control method of the processing method "transparent" is given to the "building" and "wall" of the object as metadata. The given metadata is associated with the data of the virtual object and is held in the object holding unit 208. Note that the display control method is not limited to Figure 7 The display control method only needs to be such control that the user can visually recognize the rear of the control object, and may include control by size reduction, rotation, non-display, and the like.

[0085] (Display control of control objects)

[0086] Will be used by Figure 8 , 9A to 9C as well as FIG. 10A to FIG. 10D To illustrate the display control of the control object. Figure 8 FIG. 1 is a flowchart illustrating a display control process of a control object. Figure 8 In the example shown in , the movement detection unit 202 includes a 3-axis acceleration sensor, and the rotation detection unit 203 includes a 3-axis angular velocity sensor.

[0087] Figure 8 The processing shown in Figure 3 An example of specific processing of the processing at step S306 in FIG. Figure 3 The process is similar to steps S300 to S304 in the above, and at step S305, it is determined whether the user has performed an observation operation as an example of a predetermined operation. When it is determined that the user has performed an observation operation, execution Figure 8 Display control processing of the control object shown in FIG.

[0088] At step S801, the operation judgment unit 213 acquires the moving direction of the HMD 100 (the user's head) from the movement amount calculation unit 211. The moving direction of the HMD 100 is used to control the display of the control object. The operation judgment unit 213 transmits the acquired moving direction of the HMD 100 to the object control unit 215.

[0089] At step S802, the operation judgment unit 213 acquires the movement amount of the HMD 100 from the movement amount calculation unit 211. The movement amount of the HMD 100 is used to control the display of the control object. The operation judgment unit 213 transmits the acquired movement amount of the HMD 100 to the object control unit 215.

[0090] At step S803, the object control unit 215 determines whether the control method when displaying the control object is "move". Figure 7 The control method when displaying the control object is acquired from the metadata of the description. When the control method for displaying the control object is "move", the process proceeds to step S804.

[0091] When the control method for displaying the control object is not "Move", the process proceeds to step S814. At step S814, the object control unit 215 controls so that the control object is transparent and the rear can be visually identified. The object control unit 215 can make the control object transparent by increasing the transparency of the control object. The object control unit 215 only needs to set the transparency of the control object within the range in which the user can visually identify the rear. Note that FIG. 10A to FIG. 10D An example of making a control object transparent is shown in the example of , but the control method of the control object may be control other than transparency specified by metadata.

[0092] At step S804, the object control unit 215 calculates the movement amount of the control object. The object control unit 215 may determine the distance the control object is moved based on at least one of the movement amount and the rotation amount of the user's head, for example. The object control unit 215 may calculate the movement amount of the control object so that the control object is appropriately moved according to the size of the display viewing angle of the display device 102.

[0093] Will be used by 9A to 9C The determination of the moving direction of the control object at steps S805 to S807 is explained. 9A to 9C A diagram for explaining the moving direction of a control object. Fig.9A A display example when the user wearing the HMD 100 is to visually recognize the rear (display on the back surface) of the control object 900 is shown. Fig.9AThis shows a state in which the user's line of sight is directed toward the line of sight position 901 for a period of time longer than the predetermined time t.

[0094] At steps S805 to S807, the object control unit 215 determines the direction in which the control object 900 is to be moved based on the moving direction of the HMD 100 acquired at step S801. Figure 8 In the example of FIG. 1 , it is assumed that the observation operation performed by the user is any one of an operation of observing from the right side and an operation of observing from the left side.

[0095] At step S805, the object control unit 215 determines whether the user wearing the HMD 100 performs an operation viewed from the right side of the control object. If the user performs an operation viewed from the right side of the control object, the process proceeds to step S806. If the user's operation is not an operation viewed from the right side, the process proceeds to step S807.

[0096] At step S806, the object control unit 215 determines the moving direction of the control object to be the left side. When the moving direction of the control object is determined to be the left side, the object control unit 215 performs control so that Fig. 9B As shown, the control object 900 moves to the left (in the direction indicated by the arrow). Since the control object 900 moves to the left, the user can visually recognize the virtual object 902 and the real space behind.

[0097] At step S807, it is assumed that the user wearing the HMD 100 performs a viewing operation from the left side of the control object, and the object control unit 215 determines the moving direction of the control object to be the right side. When the moving direction of the control object is determined to be the right side, the object control unit 215 performs control so that Fig. 9C As shown, the control object 900 moves to the right (in the direction indicated by the arrow). When the control object 900 moves to the right, the user can visually recognize the virtual object 902 and the real space behind.

[0098] Note that in the examples at steps S805 to S807, it is explained that the predetermined operation is an observation operation from the right side or the left side, but this is not restrictive. For example, when the predetermined operation is an observation operation from the upper side, the object control unit 215 can determine the moving direction of the control object to be the lower side. When the predetermined operation is an observation operation from the lower side, the object control unit 215 can determine the moving direction of the control object to be the upper side.

[0099] Furthermore, in the example at steps S805 to S807, the moving direction of the control object is determined based on the moving direction of the HMD 100, but the moving direction of the control object can be determined by considering the rotation direction of the HMD 100. By considering the rotation direction of the HMD 100, the object control unit 215 can accurately detect the direction from which the user will observe the rear of the control object. The object control unit 215 only needs to control so that the control object moves to the side opposite to the direction in which the user is observing the control object. As described above, the object control unit 215 can determine the direction in which the control object is to be moved based on at least any one of the moving direction and the rotation direction of the HMD 100 (the user's head).

[0100] Will be used by FIG. 10A to FIG. 10D The display control of the control object at step S808 to step S814 is explained. FIG. 10A to FIG. 10D 2 is a diagram for explaining display control of a control object. Specifically, when there are other virtual objects in the direction in which the control object is to be moved, the object control unit 215 performs control to reduce the size so that the control object does not overlap with the other virtual objects.

[0101] Fig. 10A A display example is shown in a case where the user wearing the HMD 100 is to visually recognize the rear of the control object 1000 (display on the back surface). Fig. 10A The state in which the line of sight of the user is directed toward the line of sight position 1001 for a period of time longer than the predetermined time t is shown. The HMD 100 displays another virtual object 1002 different from the control object 1000 .

[0102] At step S808 to step S814, the object control unit 215 controls the display of the control object. The object control unit 215 changes the control method of the control object according to whether there is another virtual object in the direction in which the control object is to be moved.

[0103] At step S808, the object control unit 215 determines whether there are other virtual objects in the direction in which the control object is to be moved. If there are other virtual objects in the moving direction, the process proceeds to step S809. If there are no other virtual objects in the moving direction, the process proceeds to step S813.

[0104] At step S813, since there are no other virtual objects in the direction in which the control object is to be moved, the object control unit 215 moves the control object based on the movement amount calculated at step S804. Fig. 10BIn the example of FIG. 1 , when the user wearing the HMD 100 performs a viewing operation from the left side of the control object 1000, the control object 1000 is moved to the right side. Since there is no other virtual object on the right side of the control object 1000, even if the control object 1000 is moved, the control object 1000 will not overlap with other virtual objects such as the virtual object 1002.

[0105] Steps S809 to S812 are processes when there are other virtual objects in the direction in which the control object is to be moved. If there are other virtual objects in the direction in which the control object is to be moved, the object control unit 215 reduces the size of the control object at the movement destination. Fig. 10C In the example of FIG. 1 , if the user wearing the HMD 100 performs a viewing operation from the right side of the control object 1000 , the control object 1000 moves to the left side and overlaps with other virtual objects 1002 .

[0106] Therefore, if Fig. 10D As shown, the object control unit 215 reduces the size of the control object 1000 so that even if the control object 1000 is moved, the control object 1000 does not overlap with other virtual objects 1002. Figure 8 In the example of the process shown in , in order not to deteriorate visibility due to size reduction, a threshold value for the size reduction rate is set.

[0107] At step S809, the object control unit 215 sets a threshold value α of the size reduction rate (predetermined size reduction rate). For example, the threshold value α is set within such a range that the control object becomes visually recognizable based on the type of the control object (whether the control object includes characters, etc.) or the size of the display screen, etc. At step S810, the object control unit 215 calculates such a size reduction rate β that the control object does not overlap with other virtual objects at the movement destination.

[0108] At step S811, the object control unit 215 determines whether the size reduction rate β is greater than or equal to the threshold value α. In the case where the size reduction rate β is greater than or equal to the threshold value α, the control object does not overlap with other virtual objects by reducing the size by the size reduction rate of the threshold value α. In this case, the process proceeds to step S812. On the other hand, in the case where the size reduction rate β is less than the threshold value α, the control object overlaps with other virtual objects even if it is reduced in size by the size reduction rate of the threshold value α. In this case, the process proceeds to step S814.

[0109] At step S812, even if the size of the control object is reduced at the size reduction rate β at the movement destination, the control object can be visually recognized, and Fig. 10DAs shown, the object control unit 215 performs control so that the size of the control object 1000 is reduced and moved.

[0110] At step S814, the size reduction rate β is less than the threshold value α, and if the size of the control object is reduced by the size reduction rate β, visual recognition becomes difficult. That is, even if the size of the control object is reduced by the size reduction rate of the threshold value α so that the control object becomes visually recognizable, the control object overlaps with other virtual objects. Therefore, the object control unit 215 performs control so that the control object does not move and the control object is made transparent so that the rear can be visually recognized.

[0111] At step S812 to step S814 , the object control unit 215 controls the display of the control object so that the user can visually recognize the rear of the control object.

[0112] The object control unit 215 transmits the data of the virtual object selected by the object selection unit 214 including the data of the virtual object whose display is controlled to the display image generation unit 216. The display image generation unit 216 generates a composite image in which the virtual object is superimposed and displayed on the captured image. The display unit 206 displays the composite image generated by the display image generation unit 216 on the display device 102.

[0113] exist Figure 8 When the control process of the control object shown in Figure 3 Similarly to steps S307 and S308 in , it is determined whether the user's line of sight position has changed or whether the predetermined operation has been completed. If the user's line of sight position has changed or the predetermined operation has been completed, similar to step S309, the object control unit 215 returns the control of the display of the control object executed at steps S812 to S814 to the original control.

[0114] (Controls the highlighting of objects)

[0115] Will be used by Fig.11A and Fig. 11B The control object is a virtual object that exists at the user's sight line position, and the display is controlled by the user performing a predetermined operation. The object control unit 215 can Figure 3 The control object selected at step S302 in the process is highlighted so that the virtual object as the control target can be recognized by the user.

[0116] exist Fig.11AIn the embodiment, if the line of sight of the user who is the wearer of the HMD 100 is directed toward the line of sight position 1111 for a period of time longer than the predetermined time t, the object control unit 215 performs control so that the control object 1110 is selected as the control target and is highlighted. The object control unit 215 may perform the following operations, for example: Fig. 11B The highlight display 1112 is shown to illuminate the outline of the control object 1110. Note that the highlight display method is not limited to the illumination of the outline of the control object. The object control unit 215 only needs to control the display so that the user can easily identify the control object, and for example, control can be performed so that the control object itself flashes.

[0117] The object control unit 215 can perform control so that Figure 3 The control object is highlighted during the period from when the control object is selected at step S302 until display control of the control object is started. Alternatively, the object control unit 215 may perform control so that the control object is highlighted during the period from when the control object is selected until a predetermined operation is completed.

[0118] According to the above-described embodiment, the HMD 100 can make the rear of the virtual object visually recognizable by controlling the display of the virtual object existing at the sight position of the user in response to a predetermined operation performed by the user. That is, the HMD 100 performs control so that the display of the virtual object existing at the sight position of the user himself / herself wearing the HMD 100 is different before and after the performance of the predetermined operation, thereby making the rear of the virtual object visually recognizable. The user can easily visually recognize the rear of the virtual object by performing, for example, an observation operation of the rear of the virtual object as a predetermined operation.

[0119] The above-described embodiments are merely examples, and configurations obtained by appropriately deforming or changing the configurations of the above-described embodiments within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the above-described embodiments are also included in the present invention.

[0120] Note that the functional units ( Figure 2) may be separate hardware, or it does not necessarily have to be so. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a functional unit may be implemented by separate hardware. Two or more functions of a functional unit may be implemented by common hardware. In addition, each functional unit may be implemented by hardware, or it does not necessarily have to be so. For example, the device may have a processor and a memory storing a control program. Furthermore, the functions of at least some of the functional units provided in the device may be implemented by a processor that reads out a control program from the memory and executes the control program.

[0121] According to the present invention, a user wearing a head mounted display can easily visually recognize the rear of a virtual object.

[0122] Note that the various types of control described above may be processing performed by one hardware (e.g., a processor or a circuit) or other situations. Processing may be shared among multiple hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) to perform control of the entire device.

[0123] In addition, the above-mentioned processor is a processor in a broad sense, and includes a general-purpose processor and a special-purpose processor. Examples of general-purpose processors include a central processing unit (CPU), a microprocessing unit (MPU), and a digital signal processor (DSP). Examples of special-purpose processors include a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), and a programmable logic device (PLD). Examples of PLDs include a field programmable gate array (FPGA) and a complex programmable logic device (CPLD).

[0124] Other embodiments

[0125] The embodiments of the present invention may also be implemented by the following method, namely, providing software (program) for performing the functions of the above-mentioned embodiments to a system or installation via a network or various storage media, and the system or installed computer or central processing unit (CPU), microprocessing unit (MPU) reading and executing the program.

[0126] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: a selection unit configured to select, as a control object, a virtual object existing at a sight position of a user wearing the head mounted display; a determination unit configured to determine whether the user has performed a predetermined operation for visually recognizing the rear of the control object; as well as a control unit configured to execute control for moving and displaying the control object so that the rear of the control object can be visually recognized in response to the user performing the predetermined operation, In which, the control unit determines the direction in which the controlled object is to be moved based on at least any one of the movement direction and the rotation direction of the user's head, or determines the distance in which the controlled object is to be moved based on at least any one of the movement amount and the rotation amount of the user's head.

2. The information processing device according to claim 1, wherein: The determination unit determines whether the user has performed the predetermined operation based on at least any one of a movement direction, a movement amount, a rotation direction, and a rotation amount of the user's head.

3. The information processing device according to claim 2, wherein: The determination unit determines that the user has performed the predetermined operation when the amount of movement of the user's head is a distance within a predetermined range.

4. The information processing device according to claim 2 or 3, wherein: The determination unit determines that the user has performed the predetermined operation when the amount of rotation of the user's head is an angle within a predetermined range.

5. The information processing device according to any one of claims 1 to 3, wherein: The control unit performs at least any one of display control of movement, size reduction, transparency, and non-display on the control object in response to the user performing the predetermined operation.

6. The information processing device according to claim 5, wherein: When there is another virtual object in a direction in which the control object is to be moved, the control unit performs a control of reducing the size so that the control object does not overlap with the other virtual object.

7. The information processing device according to claim 6, wherein: The control unit performs control for making the control object transparent when the control object overlaps with the other virtual object even if the control object is reduced in size at a predetermined size reduction rate.

8. The information processing device according to claim 5, wherein: Presetting a correspondence between the type of the object and the type of the display control; as well as The control unit performs display control corresponding to the type of the control object among movement, size reduction, transparency, and non-display on the control object.

9. The information processing device according to any one of claims 1 to 3, wherein: The control unit performs control so that highlighting of the control object selected by the selection unit is performed.

10. The information processing device according to claim 9, wherein: The control unit performs control so that highlighted display of the control object is performed within a period from when the control object is selected by the selection unit until display control of the control object is started or the predetermined operation is completed.

11. A head mounted display, comprising: The information processing device according to any one of claims 1 to 10; as well as A display is used to display an image whose display of the control object is controlled.

12. An information processing method, comprising: A selection step for selecting a virtual object existing at a sight line position of a user wearing the head mounted display as a control object; a determination step for determining whether the user has performed a predetermined operation for visually identifying the rear of the control object; as well as a control step for executing control for moving and displaying the control object so that the rear of the control object can be visually recognized in response to the user performing the predetermined operation, In the control step, the direction in which the controlled object is to be moved is determined based on at least one of the movement direction and the rotation direction of the user's head, or the distance in which the controlled object is to be moved is determined based on at least one of the movement amount and the rotation amount of the user's head.

13. A computer program product comprising a program for causing a computer to execute each step of the information processing method according to claim 12.

14. A computer-readable storage medium storing a program for causing a computer to execute each step of the information processing method according to claim 12.

Images