Image display system, image display program, display control device, and image display method

Abstract

To provide an image display system capable of improving operability of an object placed in a virtual space.SOLUTION: An exemplary image display system includes a goggle device having a display unit. A virtual camera and a user interface is placed in a virtual space. The posture of the virtual camera in the virtual space is controlled according to the posture of the goggle device. If, for example, a line of sight of the virtual camera turns to the left and then turns back to the right in a yaw direction, the user interface is moved to the front of the virtual camera when a displacement to the right becomes or exceeds a given value.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an image display system capable of displaying an image, an image display program, a display control device, and an image display method.

Background Art

[0002] As a prior art, there is an image display system that arranges an object in a virtual space, moves within the virtual space, and displays the object by directing the line of sight of a virtual camera toward the object (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above prior art, in order to display an object, for example, a user may need to perform an operation of directing the line of sight of a virtual camera toward the object, and there is room for improvement from the viewpoint of improving the operability of the object.

[0005] Therefore, an object of the present invention is to provide an image display system, an image display program, a display control device, and an image display method capable of improving the operability of an object arranged in a virtual space.

Means for Solving the Problems

[0006] In order to solve the above problems, the present invention employs the following configuration.

[0007] The image display system of the present invention includes a goggle device, an object placement means, a display control means, an attitude acquisition means, a camera rotation means, a detection means, and a control means. The display control means places an object in a virtual space. The object placement means causes an image captured by a virtual camera in the virtual space to be displayed on a display unit of the goggle device. The attitude acquisition means acquires the attitude of the goggle device. The camera rotation means rotates the virtual camera in the virtual space based on the attitude of the goggle device. The detection means detects that the line of sight of the virtual camera has changed from one direction to another in a predetermined rotation direction. The control means performs at least one of moving the object and controlling the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera based on the change being detected by the detection means.

[0008] According to the above, when the line of sight of the virtual camera changes from one direction to another in a predetermined rotation direction, the object can be positioned within the imaging range of the virtual camera by moving the object or controlling the virtual camera. Thereby, the operability of the object can be improved when viewing the virtual space using the goggle device.

[0009] Also, in another configuration, the image display system may further include a calculation means for calculating the rotation amount of the virtual camera. The control means may perform at least one of moving the object and controlling the virtual camera based on at least the change being detected by the detection means and the rotation amount.

[0010] According to the above, in addition to the change from one direction to another being detected by the detection means, the movement of the object and / or the control of the virtual camera can be performed based on the rotation amount of the virtual camera.

[0011] In another configuration, when the change is detected by the detection means, if the amount of rotation after the detection is equal to or greater than a predetermined value, at least one of the movement of the object and the control of the virtual camera may be performed.

[0012] According to the above, based on the amount of rotation of the virtual camera after the line of sight of the virtual camera has changed from one direction to the other direction in a predetermined rotation direction, the movement of the object and / or the control of the virtual camera can be performed.

[0013] In another configuration, the calculation means may calculate the amount of rotation in the other direction and / or the one direction in the predetermined rotation direction. The control means may perform at least one of the movement of the object and the control of the virtual camera based on the amount of rotation in the other direction and / or the one direction in the predetermined rotation direction.

[0014] According to the above, based on the amount of rotation in the other direction and / or the one direction in the predetermined rotation direction, the movement of the object or the control of the virtual camera can be performed. Thereby, for example, it is possible to prevent the movement of the object and / or the control of the virtual camera from being performed due to false detection.

[0015] In another configuration, when the change is detected by the detection means, if the amount of rotation in the other direction and / or the one direction is equal to or greater than a predetermined value, at least one of the movement of the object and the control of the virtual camera may be performed.

[0016] According to the above, when the line of sight of the virtual camera changes from one direction to the other direction in a predetermined rotation direction, and when the amount of rotation in the other direction and / or the one direction becomes equal to or greater than a predetermined value, the movement of the object and / or the control of the virtual camera can be performed. Thereby, for example, it is possible to prevent the movement of the object and / or the control of the virtual camera from being performed due to false detection.

[0017] In another configuration, when the cumulative value of the rotation amounts in the other direction and / or the one direction after the change is detected by the detection means is equal to or greater than a predetermined value, the control means may perform at least one of the movement of the object and the control of the virtual camera.

[0018] According to the above, when the line of sight of the virtual camera changes from one direction to the other direction, and when the cumulative value of the change amount in the other direction becomes equal to or greater than a predetermined value, the movement of the object and / or the control of the virtual camera can be performed. Thereby, for example, it is possible to prevent performing the movement of the object and / or the control of the virtual camera due to misdetection.

[0019] In another configuration, the calculation means may calculate a first rotation amount in the one direction in the predetermined rotation direction and a second rotation amount in the other direction in the predetermined rotation direction. The image display system may further include a comparison means for comparing the first rotation amount and the second rotation amount. The detection means may detect the change based on the comparison result of the comparison means.

[0020] According to the above, by comparing the rotation amount in one direction with the rotation amount in the other direction, it is possible to detect a change in the line of sight of the virtual camera from one direction to the other direction, and the change can be detected by a simple method.

[0021] In another configuration, the detection means may detect the change based on the smaller one of the first rotation amount and the second rotation amount.

[0022] According to the above, the change can be detected by simple calculation.

[0023] In another configuration, the calculation means may calculate a cumulative value of the first rotation amount and a cumulative value of the second rotation amount. The detection means may detect the change based on the smaller one of the cumulative value of the first rotation amount and the cumulative value of the second rotation amount.

[0024] According to the above, it is possible to detect that the change has occurred from one direction to the other direction in the predetermined rotational direction.

[0025] In another configuration, the camera rotation means may rotate the virtual camera so as to be at least in a first posture in which at least a part of the object is located within the imaging range of the virtual camera and in a second posture in which the object is located outside the imaging range of the virtual camera. The one direction may be a direction in which the virtual camera approaches the second posture, and the other direction may be a direction in which the virtual camera approaches the first posture.

[0026] According to the above, when the virtual camera is rotated so that the object is included within the imaging range of the virtual camera, the movement of the object and / or the control of the virtual camera can be performed.

[0027] In another configuration, the camera rotation means may rotate the virtual camera so as to be at least in a first posture in which the virtual camera faces the object and in a second posture in which the virtual camera does not face the object. The one direction may be a direction in which the virtual camera moves away from the first posture, and the other direction may be a direction in which the virtual camera approaches the first posture.

[0028] According to the above, when the virtual camera is rotated so that the virtual camera faces the object, the movement of the object and / or the control of the virtual camera can be performed.

[0029] In another configuration, when the virtual camera is in the second posture, the detection means detects the change when the line of sight of the virtual camera rotates in the other direction, and when the virtual camera is in the second posture, the line of sight of the virtual camera rotates in the one direction without rotating in the other direction, or when the line of sight of the virtual camera does not rotate, the change may not be detected. When the change is detected, the control means performs at least one of the movement of the object and the control of the virtual camera, and when the change is not detected, at least one of the movement of the object and the control of the virtual camera may not be performed.

[0030] According to the above, when the virtual camera is in the second posture (a posture in which the object is located outside the imaging range, a posture in which the virtual camera does not face the object), the movement of the object and / or the control of the virtual camera can be performed. Further, when the virtual camera is in the second posture, when the line of sight of the virtual camera rotates in the other direction, the movement of the object and / or the control of the virtual camera can be performed.

[0031] In another configuration, the second posture may be a posture in which the object exists in the left-right direction of the line of sight of the virtual camera.

[0032] According to the above, when the line of sight of the virtual camera is facing the object in the left-right direction, when the change is detected, the movement of the object and / or the control of the virtual camera can be performed.

[0033] In another configuration, the predetermined rotation direction may be the yaw direction in the virtual space.

[0034] According to the above, it is detected that the line of sight of the virtual camera changes from one direction (for example, the left direction) to the other direction (for example, the right direction) in the yaw direction, and based on the detection of the change, the movement of the object and / or the control of the virtual camera can be performed.

[0035] In another configuration, the camera rotation means may be capable of rotating the line of sight of the virtual camera in a second rotation direction orthogonal to the predetermined rotation direction. The detection means may further detect that the line of sight of the virtual camera has changed from one direction to the other direction in the second rotation direction.

[0036] According to the above, in addition to the predetermined rotation direction, it is possible to detect a change from one direction to the other direction with respect to the second rotation direction, and based on detecting the change, it is possible to perform the movement of the object and / or the control of the virtual camera.

[0037] In another configuration, the predetermined rotation direction may be the pitch direction in the virtual space.

[0038] According to the above, it is possible to detect that the line of sight of the virtual camera has changed from one direction (for example, the upward direction) to the other direction (for example, the downward direction) in the pitch direction, and based on detecting the change, it is possible to perform the movement of the object and / or the control of the virtual camera.

[0039] In another configuration, the object may be a user interface operable by the user.

[0040] According to the above, for example, the user interface can be moved so that the user interface is located within the imaging range of the virtual camera. Thereby, the operability of the user interface can be improved.

[0041] In another configuration, the image display system may further include pointer setting means for setting a pointer for instructing the user interface. The detection means may detect the change when the pointer is not in a predetermined area with respect to the user interface.

[0042] According to the above, for example, when the pointer is not located in the display area of the user interface, the change can be detected. Thereby, for example, when the pointer is located in the display area of the user interface, based on detecting the change, it is possible not to move the user interface, and the operability can be improved.

[0043] In another configuration, when the pointer is not in the predetermined area, the detection means may calculate the change amounts of the rotation in the one direction and the other direction in the predetermined rotation direction.

[0044] According to the above, for example, when the pointer is not located in the display area of the user interface, the change amount can be calculated, and when the pointer is located in the display area of the user interface, the change amount can be prevented from being calculated, and the load associated with calculating the change amount can be reduced.

[0045] In another configuration, when the pointer enters the predetermined area, the detection means may reset the change amount.

[0046] According to the above, even when the pointer enters or exits the predetermined area, the change can be detected based on the change amount after exiting the predetermined area.

[0047] In another configuration, the one direction may be a direction in which the pointer leaves the predetermined area. The detection means may detect the change when the moving direction of the pointer is reversed while the pointer is moving in the direction of leaving the predetermined area.

[0048] According to the above, when the moving direction of the pointer is reversed while the pointer is moving in the direction of leaving the predetermined area, a change from one direction to the other direction of the viewing line of the virtual camera can be detected, and the movement of the object and / or the control of the virtual camera can be performed.

[0049] In another configuration, the detection means may detect the change when the line of sight of the virtual camera with respect to the object is at a predetermined angle.

[0050] According to the above, when the line of sight of the virtual camera has a predetermined angle with respect to the object, the change can be detected.

[0051] In another configuration, the image display system may further include a selection means for selecting the object according to the posture of the virtual camera. When the object is not selected by the selection means, the control means may perform at least one of the movement of the object and the control of the virtual camera.

[0052] According to the above, when the object is not selected, the movement of the object and / or the control of the virtual camera can be performed.

[0053] In another configuration, even when the change is not detected by the detection means, the image display system may further include a second control means for moving the object and / or controlling the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera according to a predetermined operation by the user.

[0054] According to the above, regardless of whether the change is detected, the movement of the object and / or the control of the virtual camera can be performed according to a predetermined operation.

[0055] In another configuration, the detection means may detect that the number of inversions of the line of sight of the virtual camera from the one direction to the other direction and from the other direction to the one direction has reached a predetermined number of times. When the detection means detects that the number of inversions has reached the predetermined number of times, the control means may perform at least one of the movement of the object and the control of the virtual camera.

[0056] According to the above, based on the number of reversals of the line of sight of the virtual camera, the movement of the object and / or the control of the virtual camera can be performed. Thereby, for example, false detection can be prevented.

[0057] Further, another invention may be an image display program that causes a processor of a device that displays an image on a display unit of a goggle device to function as each of the above means. Further, another invention may be a display control device that displays an image on a display unit of a goggle device, the device including each of the above means. Further, another invention may be an image display method performed in the above image display system including a goggle device.

Advantages of the Invention

[0058] According to the present invention, when the line of sight of the virtual camera changes from one direction to the other direction in a predetermined rotation direction, the object can be positioned within the imaging range of the virtual camera, and the operability of the object can be improved when viewing a virtual space using a goggle device.

Brief Description of the Drawings

[0059]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 7A

Figure 7B

Figure 7C

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0060] (Configuration of the Image Display System) Hereinafter, with reference to the drawings, the image display system 1 of the present embodiment will be described. The image display system 1 of the present embodiment is a system that allows a user to experience virtual reality (VR). FIG. 1 is a diagram showing an example of the image display system 1 in the present embodiment. FIG. 2 is a diagram showing an example of the appearance of the information processing device 2 having the display unit 21 included in the image display system 1.

[0061] As shown in FIG. 1, the image display system 1 includes a goggle device 10. The goggle device 10 is held by both or one hand of the user and is applied to the user's face so as to cover the left and right eyes of the user.

[0062] The goggle device 10 includes an upper surface 11, a right side surface 12, a lower surface 13, a left side surface 14, and a bottom surface 15. The goggle body is formed by these surfaces 11 to 15. Further, on the bottom surface 15 of the goggle device 10, a substantially circular left opening 16L is provided at a position corresponding to the user's left eye, and a substantially circular right opening 16R is provided at a position corresponding to the user's right eye. Lenses are fitted into the left opening 16L and the right opening 16R, respectively. Further, a partition surface 17 that divides the inner space of the goggle device 10 into left and right is provided in the inner space surrounded by the surfaces 11 to 15 of the goggle device 10.

[0063] The goggle device 10 also includes a display unit 21. Specifically, an information processing device 2 having the display unit 21 is provided on the back side of the bottom surface 15 of the goggle body. A part of the display unit 21 of the information processing device 2 provided on the back side of the bottom surface 15 is visually recognized by the user through the lenses from the left opening 16L and the right opening 16R. As shown in FIG. 2, the display unit 21 is a horizontally long substantially rectangular display screen. The left region 21L surrounded by the dashed line on the left side of FIG. 2 is visually recognized by the user's left eye through the lens provided in the left opening 16L, and the right region 21R surrounded by the dashed line on the right side is visually recognized by the user's right eye through the lens provided in the right opening 16R. When the user holds the goggle device 10 and applies it to the face, the user's left eye is substantially surrounded by the upper surface 11, the lower surface 13, the left side surface 14, and the partition surface 17, and the user's right eye is substantially surrounded by the upper surface 11, the lower surface 13, the right side surface 12, and the partition surface 17. Therefore, while the left eye image displayed in the left region 21L of the display unit 21 is visually recognized by the user's left eye, it is difficult for the left eye to visually recognize the surrounding environment and the right eye image displayed in the right region 21R. Also, while the right eye image displayed in the right region 21R of the display unit 21 is visually recognized by the user's right eye, it is difficult for the right eye to visually recognize the surrounding environment and the left eye image displayed in the left region 21L.

[0064] The goggle device 10 is composed of the goggle body and the display unit 21. The information processing device 2 having the display unit 21 is detachably attached to the goggle body. As the information processing device 2 that can be detachably attached to the goggle body, a tablet terminal, a smartphone, a portable game device, etc. may be used.

[0065] Although details will be described later, the information processing device 2 generates a left eye image of the virtual space as seen from the left virtual camera and a right eye image of the virtual space as seen from the right virtual camera. The information processing device 2 displays the generated left eye image and right eye image in the left region 21L and the right region 21R of the display unit 21, respectively. Thereby, the user can visually recognize a stereoscopic image and experience virtual reality (VR) as if he / she exists in the virtual space.

[0066] FIG. 3 is a diagram showing an example of the functional configuration of the information processing apparatus 2. As shown in FIG. 3, in addition to the display unit 21, the information processing apparatus 2 includes a processor 20, an inertial sensor 22, a DRAM 23, and a nonvolatile memory 24. The processor 20 includes a CPU and a GPU. The CPU performs processing described later in cooperation with the DRAM 23, and the GPU generates images (left-eye image and right-eye image) according to instructions from the CPU. The generated left-eye image and right-eye image are respectively displayed in the left area 21L and the right area 21R of the display unit 21. Note that the CPU and the GPU may be mounted on separate chips, or they may be mounted on one chip as an SoC (System-on-a-chip).

[0067] Also, the inertial sensor 22 is a sensor for detecting the posture of the information processing apparatus 2. Specifically, the inertial sensor 22 includes an angular velocity sensor and an acceleration sensor. The angular velocity sensor detects the angular velocity around a predetermined three axes (for example, the XYZ axes shown in FIG. 2). Also, the acceleration sensor detects the acceleration around a predetermined three axes (for example, the XYZ axes shown in FIG. 2).

[0068] Also, the nonvolatile memory 24 is a storage device that stores a program for performing processing described later, and may be, for example, a flash memory. Note that the nonvolatile memory 24 may be any storage device such as a magnetic disk or an optical disk.

[0069] FIG. 4 is a diagram showing a state when the user uses the goggle device 10. As shown in FIG. 4, the user holds the goggle device 10 on which the information processing apparatus 2 is mounted and looks into the display unit 21. When the user turns the face or the whole body in the left-right direction (horizontal direction, or also referred to as the "yaw direction") or in the up-down direction (vertical direction, or also referred to as the "pitch direction") in this state, the posture of the goggle device 10 (information processing apparatus 2) changes from the reference posture. The reference posture is, for example, the posture of the goggle device 10 when the user faces forward. For example, the reference posture may be a posture in which the downward direction (negative Y-axis direction in FIG. 2) along the display unit 21 (bottom surface 15) of the goggle device 10 is parallel to the direction of gravity.

[0070] The information processing device 2 calculates the attitude of the information processing device 2 (the goggle device 10) based on the angular velocity value and / or the acceleration value detected by the inertial sensor 22. According to the attitude of the information processing device 2 (the goggle device 10) in the real space, the attitudes of the left virtual camera and the right virtual camera in the virtual space are controlled.

[0071] In the virtual space, virtual objects are arranged, and the user can view a stereoscopic image of the virtual space including the virtual objects. In the present embodiment, a user interface (UI) object is arranged in the virtual space as an example of the virtual object.

[0072] FIG. 5 is a diagram showing an example of the virtual space VS. As shown in FIG. 5, an xyz orthogonal coordinate system is set in the virtual space VS. The x-axis is the horizontal axis of the virtual space VS. The y-axis is the height axis of the virtual space VS. The z-axis is an axis perpendicular to the x-axis and the y-axis and is the depth axis of the virtual space.

[0073] In the virtual space VS, a left virtual camera VCL and a right virtual camera VCR are arranged. The left virtual camera VCL and the right virtual camera VCR are arranged in the virtual space at an interval similar to the average distance between the left and right eyes of a user. The left-eye image and the right-eye image of the virtual space seen from the left virtual camera VCL and the right virtual camera VCR are respectively viewed by the left eye and the right eye of the user, so that the user can view a stereoscopic image of the virtual space. Hereinafter, the left virtual camera VCL and the right virtual camera VCR may be collectively referred to as the "virtual camera VC".

[0074] In addition, a UI object 30 is arranged in the virtual space VS. The UI object 30 is a virtual object for the user to operate or display information presented to the user. For example, the UI object 30 provides a menu function for the user. For example, the UI object 30 is displayed during the execution of a predetermined game application, and the user can perform a predetermined operation in the game using the UI object 30. The UI object 30 is, for example, a plate-shaped object and is arranged in a posture perpendicular to the xz plane at a position of a predetermined height in the virtual space.

[0075] The UI object 30 includes icons 31 to 34 that can be selected by the user. For example, the icon 34 selected by the user may be displayed in a manner different from other icons. The method by which the user selects any one of the icons 31 to 34 is arbitrary. For example, a pointer 37 (see FIG. 7A) is displayed at a predetermined position (for example, the center of the image) of the stereoscopic image displayed on the display unit 21, and when the pointer is within the display area of the icon, the icon may be processed as being selected by the user. In this case, when the user selects a desired icon, the user changes the posture of the goggle device 10 so that the pointer enters the display area of the desired icon. Also, for example, the user may use a handheld operation device connected to the information processing device 2 wirelessly or by wire to control the position of the pointer displayed in the stereoscopic image. For example, the operation device includes a direction indicating unit (for example, an analog stick, a cross key, a button, etc.) capable of indicating a direction, and the position of the pointer may be controlled based on an operation on the direction indicating unit.

[0076] Each icon may be, for example, an icon for determining an operation in a game or an icon for using a predetermined item. Also, each icon may be an icon for performing an operation (for example, an operation to return to the previous screen, an operation for determination) on the menu screen provided by the UI object 30. Also, each icon may be an icon for selecting and starting a predetermined application.

[0077] In addition, the UI object 30 includes an information display object 35. The information display object 35 is an object for providing information to the user using characters, images, and the like.

[0078] Note that in the virtual space, any activity other than games may be performed. For example, a simulation of a specific task (for example, the task of driving an airplane or a car) may be performed. Depending on the activity performed in the virtual space, the functions of each icon included in the UI object 30 (operations performed using the UI object 30) are different.

[0079] (Movement control of UI object 30) Next, the control of the UI object 30 when the user turns the bottom surface 15 of the goggle device 10 in the left - right direction (yaw direction) or the up - down direction (pitch direction) while looking at the display unit 21 of the goggle device 10 will be described.

[0080] FIG. 6A is a view of the virtual space when the goggle device 10 is maintained in the reference posture, as seen from above. FIG. 6B is a view of the virtual space when the goggle device 10 is rotated in the yaw direction (left direction) from the reference posture, as seen from above. FIG. 6C is a view of the virtual space when the goggle device 10 is further rotated in the yaw direction (left direction) from the state of FIG. 6B, as seen from above.

[0081] FIG. 7A is an example of an image visible to the user, and shows an example of an image of the virtual space seen from the virtual camera VC (VCL or VCR) in the posture shown in FIG. 6A. FIG. 7B is an example of an image visible to the user, and shows an example of an image of the virtual space seen from the virtual camera VC in the posture shown in FIG. 6B. FIG. 7C is an example of an image visible to the user, and shows an example of an image of the virtual space seen from the virtual camera VC in the posture shown in FIG. 6C. Note that in FIGS. 7A to 7C, planar rectangular images are shown, but actually, the images shown in FIGS. 7A to 7C are stereoscopic images and are substantially circular in shape like the left region 21L and the right display region 21R.

[0082] A camera coordinate system (XYZ coordinate system) fixed to the virtual camera VC is set for the virtual camera VC. The X-axis is the axis in the right direction of the virtual camera VC, the Y-axis is the axis in the upward direction of the virtual camera VC, and the Z-axis is the axis in the line-of-sight direction of the virtual camera VC. As shown in FIG. 6A, when the goggle device 10 (information processing device 2) is maintained in the reference posture, the virtual camera VC faces the UI object 30.

[0083] In the state shown in FIG. 6A, an image as shown in FIG. 7A is displayed on the display unit 21. The user visually recognizes a stereoscopic image as shown in FIG. 7A by visually recognizing the left-eye image and the right-eye image displayed on the display unit 21. As shown in FIG. 7A, the UI object 30 is displayed in the front, and the background of the virtual space is displayed behind the UI object 30. Further, for example, a pointer 37 is set and displayed at a position that is the center of the stereoscopic image. The user controls the posture of the goggle device 10 so that the pointer 37 is located in the display area of the UI object 30, and uses the pointer 37 to select selectable icons 31 to 34 in the UI object 30. Note that the pointer 37 may not be displayed on the screen and may be set inside the information processing device 2.

[0084] When the goggle device 10 is rotated in the yaw direction (for example, the left direction) in the real space, as shown in FIG. 6B, the virtual camera VC also rotates in the yaw direction (for example, the left direction) in the virtual space. At this time, an image as shown in FIG. 7B is displayed. Specifically, when the goggle device 10 is rotated to the left, the UI object 30 is displayed so as to be located on the right side of the front, and a part of the UI object 30 is not displayed.

[0085] When the goggle device 10 is further rotated leftward from the state shown in FIG. 6B, as shown in FIG. 6C, the virtual camera VC further rotates leftward. At this time, an image as shown in FIG. 7C is displayed on the display unit 21. Specifically, when the goggle device 10 is further rotated leftward from the state shown in FIG. 6B without inverting it rightward, the UI object 30 exits the imaging range of the virtual camera VC and is no longer displayed. In this case, the background of the virtual space other than the UI object 30 is displayed.

[0086] When the user further rotates the goggle device 10 leftward from the state shown in FIG. 6C, the user can visually recognize the virtual space further to the left.

[0087] Here, for example, in the posture shown in FIG. 6C, when the user inverts the goggle device 10 rightward by a predetermined angle, the UI object 30 moves to be positioned in front of the virtual camera VC.

[0088] FIG. 8 is a diagram showing how the UI object 30 moves to the front of the virtual camera VC when the goggle device 10 is inverted rightward by a predetermined angle in the posture shown in FIG. 6C. In FIG. 8, the dashed line 30' indicates the UI object before movement, and the solid line 30 indicates the UI object after movement.

[0089] As shown in FIG. 8, when the goggle device 10 rotates, for example, to the left and then reverses by a predetermined angle to the right, the UI object 30 moves to the front (front in the left - right direction) of the virtual camera VC. For example, while changing its orientation, the UI object 30 moves leftward along a circle centered on the intermediate position between the left virtual camera VCL and the right virtual camera VCR. That is, the UI object 30 moves leftward on a plane parallel to the xz - plane while maintaining its distance from the virtual camera VC. The UI object 30 moves while changing its orientation so as to face the virtual camera VC. Therefore, when the virtual camera VC is not rotating in the pitch direction in the virtual space, the line of sight of the virtual camera VC becomes perpendicular to the moved UI object 30. When the UI object 30 moves to the front of the virtual camera VC, a stereoscopic image as shown in FIG. 7A is displayed on the display unit 21.

[0090] The UI object 30 moves a predetermined distance per frame time (for example, 1 / 60 second). Therefore, when the goggle device 10 rotates, for example, to the left and then reverses by a predetermined angle to the right, the UI object 30 does not move instantaneously to the front of the virtual camera VC, but moves over a certain period of time (for example, several to several tens of frame times). Note that the UI object 30 may be moved instantaneously (in one frame time) to the front of the virtual camera VC.

[0091] Note that even when a part of the UI object 30 is displayed and the other part is not displayed as shown in FIG. 7B, when the goggle device 10 reverses by a predetermined angle to the right, the UI object 30 may be moved to the front of the virtual camera VC. Also, even when the UI object 30 is displayed at the edge of the screen and the entire UI object 30 is displayed, when the goggle device 10 reverses by a predetermined angle to the right, the UI object 30 may be moved to the front of the virtual camera VC.

[0092] Also, when the pointer 37 is located within the display area of the UI object 30, the UI object 30 is not moved. When the pointer 37 is not located within the display area of the UI object 30 and the Google device 10 is inverted by a predetermined angle, the UI object 30 may be moved.

[0093] Thus, in this embodiment, when the virtual camera VC simply rotates in one direction (e.g., the left direction) in the yaw direction, and for example, the UI object 30 only goes out of the imaging range of the virtual camera VC, the UI object 30 is not moved to the front of the virtual camera VC. After the virtual camera VC rotates in one direction (e.g., the left direction) in the yaw direction, when the virtual camera VC rotates (inverts) by a predetermined angle in the other direction (e.g., the right direction) in the yaw direction, the UI object 30 is moved to the front of the virtual camera VC. Hereinafter, the movement of the UI object 30 will be described in detail.

[0094] (Details of the movement of the UI object 30) FIG. 9 is a diagram showing an example of the area where the UI object 30 is moved. In FIG. 9, when the UI object 30 is located in front of the virtual camera VC, a diagram conceptually dividing the virtual space into a plurality of areas with respect to the virtual camera VC is shown.

[0095] In this embodiment, the processing related to the movement of the UI object 30 differs depending on whether the line of sight of the virtual camera VC is in any of the "UI area", "area B", and "area A" shown in FIG. 9.

[0096] When the line of sight of the virtual camera VC is in the "UI area" shown in FIG. 9, the UI object 30 is displayed on the display unit 21, and the UI object 30 does not move in the virtual space according to the posture change of the goggle device 10. The "UI area" is an area based on the UI object 30 and is an area corresponding to the UI object 30. Specifically, taking the angle of the line of sight of the virtual camera VC when the UI object 30 is located in front of the virtual camera VC (front in the left-right and up-down directions) as 0 degrees, when the angles of the line of sight of the virtual camera VC in the yaw direction and pitch direction in the virtual space are within a predetermined threshold, it is determined that the line of sight of the virtual camera VC is in the "UI area" shown in FIG. 9. More specifically, when the absolute value of the rotation angle of the line of sight of the virtual camera VC in the yaw direction is less than the first threshold (Ty), and the absolute value of the rotation angle of the line of sight of the virtual camera VC in the pitch direction is less than the second threshold (Tx), it is determined that the line of sight of the virtual camera VC is in the "UI area" shown in FIG. 9.

[0097] Here, the UI area is defined by the first threshold and the second threshold. For example, the UI area may coincide with the display area of the UI object 30. In this case, for example, when the rotation angle of the line of sight of the virtual camera VC in the yaw direction reaches the first threshold (Ty), the entire UI object 30 is not displayed as shown in FIG. 7C. Also, the UI area may be smaller than the display area of the UI object 30. In this case, when the rotation angle of the line of sight of the virtual camera VC in the yaw direction is at the first threshold (Ty), a part of the UI object 30 is displayed as shown in FIG. 7B, for example. Note that even when the rotation angle of the line of sight of the virtual camera VC in the yaw direction is at the first threshold (Ty), all of the UI object 30 may be displayed. Also, the UI area defined by the first threshold and the second threshold may be larger than the display area of the UI object 30.

[0098] Also, the UI area may be an area where the pointer 37 in the stereoscopic image is located within the display area of the UI object 30. That is, the UI area may be an area formed by the trajectory when the pointer 37 moves along the outer periphery of the display area of the UI object 30.

[0099] When the line of sight of the virtual camera VC is in the B area shown in FIG. 9, "movement processing in the B area" is performed. That is, when the absolute value of the rotation angle in the yaw direction of the line of sight of the virtual camera VC is greater than or equal to the first threshold value (Ty), and the absolute value of the rotation angle in the pitch direction of the line of sight of the virtual camera VC is less than the second threshold value (Tx), "movement processing in the B area" is performed. In the "movement processing in the B area", "determination of whether to move the UI object 30" is performed. Hereinafter, this determination will be specifically described with reference to FIGS. 10 and 11.

[0100] FIG. 10 is a view of the virtual camera VC as seen from above the virtual space, and shows an example when the virtual camera VC is rotated to the left and then to the right. FIG. 11 is a view of the virtual camera VC as seen from the lateral direction of the virtual space, and shows an example when the virtual camera VC is rotated downward and then upward.

[0101] As shown in FIG. 10, when the line of sight of the virtual camera VC rotates beyond a first threshold value "Ty" in the left direction (positive direction) in the yaw direction, for example, the line of sight of the virtual camera VC enters the B region. After the line of sight of the virtual camera VC enters the B region, a change amount "y1" in the left direction and a change amount "y2" in the right direction are calculated. These change amounts "y1" in the left direction and "y2" in the right direction are relative rotation amounts after the line of sight of the virtual camera VC enters the B region. When the line of sight of the virtual camera reciprocates in the left-right direction, the change amount "y1" in the left direction and the change amount "y2" in the right direction are respectively accumulated. For example, after the line of sight of the virtual camera VC enters the B region, if the line of sight of the virtual camera VC rotates in the order of "10" degrees in the left direction, "2" degrees in the right direction, "5" degrees in the left direction, and "3" degrees in the right direction, the cumulative value of the change amount "y1" in the left direction is "10 + 5 = 15" degrees, and the cumulative value of the change amount "y2" in the right direction is "2 + 3 = 5" degrees. Among the cumulative value of the change amount "y1" in the left direction and the cumulative value of the change amount "y2" in the right direction, the smaller value (the cumulative value of the change amount y2 in the right direction) is stored as the change amount Ry in the left-right direction. In this case, the cumulative value of "5" degrees of the change amount "y2" in the right direction is stored as the change amount Ry in the left-right direction.

[0102] The same applies to the rotation in the pitch direction. As shown in FIG. 11, after the line of sight of the virtual camera VC enters the B region (that is, after the line of sight of the virtual camera VC rotates by a first threshold value or more in the yaw direction), it is assumed that it rotates "x1" degrees in the downward direction (negative direction) in the pitch direction, and then rotates "x2 (<x1)" degrees in the upward direction.

[0103] Note that in FIG. 11, a case where the rotation angle in the pitch direction in the virtual space when the line of sight of the virtual camera VC enters the B region is "0" degrees is shown. When the line of sight of the virtual camera VC enters the B region, the line of sight of the virtual camera VC may be rotated by a predetermined angle in the pitch direction in the virtual space. In this case, the change amount in the downward direction from the time when the line of sight of the virtual camera VC enters the B region is "x1".

[0104] Among the cumulative value of the downward change amount “x1” and the cumulative value of the upward change amount “x2”, the smaller value (the cumulative value of the upward change amount x2) is stored as the vertical change amount Rx. For example, after the line of sight of the virtual camera VC enters the B region, if the line of sight of the virtual camera VC rotates downward by “10” degrees, upward by “2” degrees, downward by “3” degrees, and upward by “1” degree in this order, the cumulative value of the downward change amount “x1” is “10 + 3 = 13” degrees, and the cumulative value of the upward change amount “x2” is “2 + 1 = 3” degrees. Among the cumulative value of the downward change amount “x1” and the cumulative value of the upward change amount “x2”, the smaller value (the cumulative value of the upward change amount x2) is stored as the vertical change amount Rx. In this case, the cumulative value of the upward change amount “x2” of “3” degrees is stored as the vertical change amount Rx.

[0105] Here, regarding the yaw direction or the pitch direction, when the line of sight of the virtual camera VC continues to rotate in one direction (for example, the left direction in the yaw direction), the change amount in the other direction (for example, the right direction in the yaw direction) is “0”. Therefore, if the smaller value among the cumulative value of the rotation amount in one direction and the cumulative value of the rotation amount in the other direction exceeds “0” degrees, it means that the rotation of the line of sight of the virtual camera VC has changed from one direction to the other direction. On the contrary, if the smaller value among the cumulative value of the rotation amount in one direction and the cumulative value of the rotation amount in the other direction is “0” degrees, it means that the line of sight of the virtual camera continues to rotate in one direction, or the line of sight of the virtual camera has not rotated.

[0106] Therefore, in this embodiment, based on the smaller values Rx and Ry, it is detected whether the rotation of the line of sight of the virtual camera VC has changed (i.e., reversed) from one direction to the other direction. When a change from one direction to the other direction is detected, the UI object 30 is moved to the front of the virtual camera VC.

[0107] Specifically, when the sum of the change amount Ry in the left - right direction and the change amount Rx in the up - down direction is equal to or greater than a predetermined value (for example, "6" degrees), the UI object 30 is moved to the front of the virtual camera VC. On the other hand, when this sum is less than the predetermined value, the UI object 30 is not moved. For example, when the line of sight of the virtual camera VC is in the B region, if the line of sight of the virtual camera VC rotates in the order of "10 degrees to the left" → "3 degrees to the right" → "20 degrees to the left", the change amount Ry in the left - right direction is "3" degrees, and the change amount Rx in the up - down direction is "0" degrees. In this case, since "Rx + Ry" is less than the predetermined value, the UI object 30 is not moved to the front of the virtual camera VC.

[0108] On the other hand, when the line of sight of the virtual camera VC is in the B region, if the line of sight of the virtual camera VC rotates in the order of "10 degrees to the left" → "3 degrees to the right" → "20 degrees to the left" → "3 degrees to the right", the change amount Ry in the left - right direction is "6" degrees. In this case, since "Rx + Ry" is equal to or greater than the predetermined value, the UI object 30 is moved to the front of the virtual camera VC. Also, when the line of sight of the virtual camera VC rotates in an oblique direction, the rotation direction is divided into components in the left - right direction (yaw direction) and the up - down direction (pitch direction), and the above - mentioned Rx and Ry are calculated respectively. For example, when the line of sight of the virtual camera VC rotates to the "upper - left direction (10 degrees in the left - hand component, 10 degrees in the up - hand component)", and then rotates to the "lower - right direction (3 degrees in the right - hand component, 3 degrees in the down - hand component)", Rx is "3" degrees and Ry is "3" degrees. In this case, since "Rx + Ry" is equal to or greater than the predetermined value, the UI object 30 is moved to the front of the virtual camera VC.

[0109] Thus, in the "movement processing in area B", after the virtual camera VC rotates in one direction in the yaw or pitch direction, it is detected that the virtual camera VC rotates in the opposite direction. When the virtual camera VC rotates in the opposite direction and the accumulated value of the rotation angle in the opposite direction becomes equal to or greater than a predetermined value, the UI object 30 moves to the front of the virtual camera VC. That is, when the line of sight of the virtual camera VC changes from one direction to the other direction in a predetermined rotation direction (yaw direction or pitch direction), and the rotation angle in the other direction becomes equal to or greater than a predetermined value, the UI object 30 is moved.

[0110] Note that when the line of sight of the virtual camera VC is in area B, even if "Rx + Ry" is less than the predetermined value, if a predetermined operation is performed by the user, the UI object 30 may be moved to the front of the virtual camera VC. For example, when a predetermined button or location on the goggle device 10 is pressed, tapped, or struck, the UI object 30 may be moved to the front of the virtual camera VC. Also, for example, when the acceleration sensor in the inertial sensor 22 of the goggle device 10 detects an acceleration value equal to or greater than a predetermined value, the UI object 30 may be moved to the front of the virtual camera VC. <(

[0111] Returning to FIG. 9, when the line of sight of the virtual camera VC is in area A, that is, when the absolute value of the rotation angle in the pitch direction of the line of sight of the virtual camera VC exceeds the second threshold value (Tx), the "movement processing in area A" is performed. In the "movement processing in area A", the determination in area B described above is not performed, and the UI object 30 is moved so that the UI object 30 is always located in front of the virtual camera VC in the left-right direction. Here, it is assumed that the "movement processing in area A" is performed when the depression angle or elevation angle of the virtual camera VC (goggle device 10) exceeds the second threshold value, but the threshold value for the depression angle and the threshold value for the elevation angle do not have to be the same value, and they may be different.

[0112] FIG. 12 is a diagram showing how the UI object 30 moves when the line of sight of the virtual camera VC is in area A.

[0113] In FIG. 12, 30' indicated by a dashed line is the UI object before movement, and 30 indicated by a solid line is the UI object after movement. As shown in FIG. 12, for example, when the line of sight of the virtual camera VC is in area A above the UI object 30, the UI object 30 moves so as to be positioned in front of the virtual camera VC in the left-right direction. The rotation angle of the virtual camera VC in the yaw direction (left-right direction) with respect to the UI object 30 after movement is "0" degrees. That is, with reference to the line-of-sight direction of the virtual camera VC, the UI object 30 after movement is not shifted in the left-right direction. For this reason, when the line of sight of the virtual camera VC is rotated downward (when the virtual camera VC is rotated so that the rotation angle in the pitch direction of the virtual camera VC becomes "0" degrees), the UI object 30 is positioned in front of the virtual camera VC.

[0114] Here, assume that after the line of sight of the virtual camera VC enters area B, the line of sight of the virtual camera VC enters area A without the UI object 30 moving in the "movement process in area B". In this case, when the line of sight of the virtual camera VC enters area A, the UI object 30 starts to move by the "movement process in area A". In this case, the UI object 30 may move to the front of the virtual camera VC in the left-right direction over a certain period of time (for example, several to several tens of frame times), or may move instantaneously (in 1 frame time).

[0115] Also, when the line of sight of the virtual camera VC enters area A from a state where it is in the "UI area", the UI object 30 moves by the "movement process in area A". When the virtual camera VC rotates in the left-right direction when the line of sight of the virtual camera VC is in area A, the UI object 30 moves according to the rotation of the virtual camera VC in the left-right direction. That is, the UI object 30 always moves so as to follow the rotation of the virtual camera VC in the left-right direction while the virtual camera is rotating in the left-right direction.

[0116] FIG. 13A is a diagram showing an example of an image displayed on the display unit 21 when the line of sight of the virtual camera VC rotates downward and enters the A region. FIG. 13B is a diagram showing an example of an image displayed on the display unit 21 when the virtual camera VC rotates leftward from the state of FIG. 13A.

[0117] As shown in FIG. 13A, when the line of sight of the virtual camera VC rotates downward (i.e., when the goggle device 10 rotates downward), a part of the lower side of the UI object 30 is displayed in the upper region of the screen. The UI object 30 is displayed at the center of the screen in the left-right direction. Also, in the background of the virtual space, there is, for example, a virtual object 40, and the virtual object 40 is displayed at the left end of the screen.

[0118] In this state, when the line of sight of the virtual camera VC rotates leftward (i.e., when the goggle device 10 rotates leftward), as shown in FIG. 13B, the position of the UI object 30 in the left-right direction does not change. This is because, by the movement process in the A region described above, the UI object 30 moves within the virtual space according to the rotation of the virtual camera VC in the yaw direction. On the other hand, the virtual object 40 existing in the background of the virtual space is displayed on the right side of the position in FIG. 13A. While the user continues to rotate the goggle device 10 leftward, the position of the UI object 30 on the screen does not change, and the background of the virtual space moves rightward.

[0119] Note that the sizes and shapes of the "UI area", "A area", and "B area" shown in FIG. 9 may vary according to the size and shape of the displayed UI object 30. Also, there are multiple types of UI objects 30, and the sizes and shapes of the respective areas may vary according to the type of UI object 30. Further, the sizes and shapes of the respective areas may vary according to the scene (background scene) of the virtual space. In this case, the above-described first threshold value (Ty) and second threshold value (Tx) will vary according to the sizes and shapes of the respective areas. Also, the "UI area" does not necessarily have to match the size of the UI object 30. For example, there may be a margin around the UI object 30, and an area including the area where the UI object 30 is displayed and the surrounding margin may be set as the "UI area". Conversely, the "UI area" may be smaller than the area where the UI object 30 is displayed.

[0120] As described above, in the present embodiment, when the rotation angles in the yaw direction and pitch direction of the virtual camera VC in the virtual space (the rotation angles in the yaw direction and pitch direction of the goggle device 10) exceed a predetermined threshold value, movement processing of the UI object 30 is performed.

[0121] Specifically, when the absolute value of the rotation angle in the yaw direction of the virtual camera VC in the virtual space is greater than the first threshold value (Ty) and the absolute value of the rotation angle in the pitch direction is less than the second threshold value (Tx) (when the line of sight of the virtual camera VC is in the B area), "movement processing in the B area" is performed. In the "movement processing in the B area", a determination is made as to whether to move the UI object 30 based on the rotation angles in the yaw direction and pitch direction of the virtual camera VC. Specifically, when the rotation in the yaw direction of the virtual camera VC changes from one direction to the other direction, a value Ry obtained by accumulating the rotation angle in the other direction is calculated. Also, when the rotation in the pitch direction of the virtual camera VC changes from one direction to the other direction, a value Rx obtained by accumulating the rotation angle in the other direction is calculated. Then, when the sum of Rx and Ry is equal to or greater than a predetermined value, it is determined that the UI object 30 is to be moved, and the UI object 30 is moved to the front of the virtual camera VC.

[0122] Sometimes, the user may want to view the UI object 30 in the virtual space, and sometimes, the user may want to view objects in the virtual space that are different from the UI object 30 (background or other objects in the virtual space). When the line of sight of the virtual camera VC (the posture of the goggle device 10) continues to rotate in one direction, it is considered that the user wants to view a virtual space different from the UI object 30. Therefore, in this embodiment, when the line of sight of the virtual camera VC continues to rotate in one direction (for example, to the left in the yaw direction or upward in the pitch direction) or is stationary, the UI object 30 is not moved to the front of the virtual camera VC. Therefore, the user can overlook the virtual space other than the UI object 30 by continuously rotating the goggle device 10, for example, to the left.

[0123] On the other hand, when the line of sight of the virtual camera VC reverses from the direction away from the UI object 30 to the direction approaching the UI object 30, the user may want to view the UI object 30 more than other objects in the virtual space. Therefore, in this embodiment, when the line of sight of the virtual camera VC reverses from one direction (the direction away from the UI object 30) to the other direction, and further, when the rotation amount in the other direction becomes equal to or greater than a predetermined value, the UI object 30 is moved to the front of the virtual camera VC.

[0124] By performing such control, in this embodiment, the user can be made to view a virtual space different from the UI object 30, and the UI object 30 existing in the virtual space can be moved to the front to facilitate the operation (viewing) of the UI object 30.

[0125] For example, when the virtual camera VC faces a direction different from that of the UI object 30 and the UI object 30 is out of the imaging range of the virtual camera VC, it is also conceivable to move the UI object 30 to the front of the virtual camera VC. However, in this case, since the UI object 30 moves to the front of the user when it is out of the imaging range of the virtual camera VC, the user cannot see the virtual space other than the UI object 30.

[0126] In contrast, in this embodiment, the goggle device 10 is continuously rotated in one direction in the yaw direction (the direction away from the UI object 30), or when it is stationary, the UI object 30 does not move to the front of the virtual camera VC, so the user can see the virtual space other than the UI object 30. When the user wants to view or operate the UI object 30 while looking around the virtual space, the user can move the UI object 30 to the front of the user, for example, by reversing the goggle device 10 in the other direction in the yaw direction or by rocking the goggle device 10 up and down. Thereby, for example, an icon within the UI object 30 can be easily selected using a pointer, and the operability of the UI object 30 can be improved in the VR space. Even when at least a part of the UI object 30 is displayed and the UI object 30 is displayed at the edge of the screen, the UI object 30 can be moved to the front and the operability can be improved only by reversing the goggle device 10 by a predetermined angle.

[0127] Also, in this embodiment, since the UI object 30 is moved within the virtual space, it may be possible to reduce VR sickness.

[0128] Also, in this embodiment, when the rotation angle of the virtual camera VC in the virtual space in the pitch direction is greater than the second threshold value (Tx) (when the line of sight of the virtual camera VC is in the A region), "movement processing in the A region" is performed. In the "movement processing in the A region", the UI object 30 always moves to the front (front in the left-right direction) of the virtual camera VC according to the rotation of the virtual camera VC in the yaw direction. Therefore, after the line of sight of the virtual camera VC enters the A region, when the display unit 21 is viewed with the goggle device 10 in the reference posture, the UI object 30 always comes to be located in front of the user. For example, after the user places the goggle device 10 on a plane in the real space (a plane parallel to the ground, such as a table) so that the bottom surface 15 of the goggle device 10 is on the lower side or the upper side, and then holds the goggle device 10 again and views the display unit 21, the UI object 30 always appears to be located in front of the user.

[0129] Also, for example, assume that two users experience VR using one goggle device 10. After one user experiences VR using the goggle device 10, the goggle device 10 is passed to the other user. At this time, the posture of the goggle device 10 changes, for example, so that the bottom surface 15 faces downward. In this case, if the UI object 30 does not move by the movement processing in the A region, when the other user holds the goggle device 10 and views the display unit 21, the UI object 30 is not necessarily in front of the other user. If the UI object 30 does not move by the movement processing in the A region, the UI object 30 maintains the position when the goggle device 10 was passed to the other user. Therefore, when the other user views the display unit 21 of the goggle device 10, the UI object 30 may be located behind the other user or may be located to the right. However, in this embodiment, since the UI object 30 moves by the movement processing in the A region, when the other user views the display unit 21 of the goggle device 10, the UI object 30 comes to be located in front of the other user. Therefore, the operability is improved for the other user.

[0130] (Details of the process) Next, a specific example of the process performed in the information processing apparatus 2 will be described. First, the data stored in the information processing apparatus 2 will be described.

[0131] FIG. 14 is a diagram showing an example of the data stored in the information processing apparatus 2 (DRAM 23 thereof). As shown in FIG. 14, the information processing apparatus 2 stores a predetermined program, angular velocity data, virtual camera data, UI data, cumulative values X1 and X2, cumulative values Y1 and Y2, change amounts Rx and Ry. In addition to these, various data such as data related to objects arranged in the virtual space and operation data according to the user's operations are stored.

[0132] The program is a program for executing the processes described later. The program is stored, for example, in the non-volatile memory 24 or an external storage medium, and is read from the non-volatile memory 24 or the external storage medium into the DRAM 23. Note that the program may be acquired from another device via a network (e.g., LAN, WAN, Internet, etc.).

[0133] The angular velocity data is data related to the angular velocity output by the inertial sensor 22.

[0134] The virtual camera data is data related to the positions and postures of the left virtual camera VCL and the right virtual camera VCR.

[0135] The UI data is data related to the position, type of the UI object 30, icons 31 to 34 included in the UI object 30, the information display object 35, etc.

[0136] The cumulative value X1 is a value used in the movement process in the B region, and is a value obtained by accumulating the downward change amount x1 (see FIG. 11). The cumulative value X2 is a value used in the movement process in the B region, and is a value obtained by accumulating the upward change amount x2 (see FIG. 11).

[0137] The cumulative value Y1 is a value used in the movement process in the B region and is a value obtained by accumulating the leftward change amount y1 (see FIG. 10). Further, the cumulative value Y2 is a value used in the movement process in the B region and is a value obtained by accumulating the rightward change amount y2 (see FIG. 10).

[0138] The change amount Rx is a value used in the movement process in the B region and is the smaller value of the cumulative values X1 and X2 of the change amount in the vertical direction (pitch direction) of the line of sight of the virtual camera VC. Further, the change amount Ry is a value used in the movement process in the B region and is the smaller value of the cumulative values Y1 and Y2 of the change amount in the horizontal direction (yaw direction) of the line of sight of the virtual camera VC. Rx and Ry are initially set to "0".

[0139] (Description of the main flow) Next, the details of the main process performed in the information processing apparatus 2 will be described. FIG. 15 is a flowchart showing an example of the main process performed in the processor 20 of the information processing apparatus 2. The process shown in FIG. 15 is performed by the CPU or GPU of the information processing apparatus 2 executing a predetermined program. Note that in FIG. 15, only the process related to the movement of the above-described UI object 30 is shown, and other processes (for example, processes according to the operations of the user in the virtual space) are omitted.

[0140] As shown in FIG. 15, the processor 20 first performs an initial process (step S100). In the initial process, first, initialization regarding the posture of the goggle device 10 (information processing apparatus 2) is performed. For example, the user is instructed to place the goggle device 10 including the information processing apparatus 2 on a table or the like, and the posture of the goggle device 10 is initialized. Further, in the initial process, an xyz coordinate system is set in the virtual space, and the UI object 30, the left virtual camera VCL, the right virtual camera VCR, the background, and other objects are arranged in the virtual space. After the process of step S100, the processor 20 repeatedly executes the processes of steps S101 to S109 at predetermined frame intervals (for example, 1 / 60 second).

[0141] After the process of step S100, the processor 20 acquires data (for example, an angular velocity value) from the inertial sensor 22 (step S101).

[0142] Next, the processor 20 performs an attitude calculation process based on the data from the inertial sensor 22 acquired in step S101 (step S102). Specifically, the processor 20 calculates the attitude of the goggle device 10 (information processing device 2) based on the angular velocity value from the inertial sensor 22. The processor 20 calculates the change in attitude from the initialization in step S100 by integrating the angular velocity value from the inertial sensor 22, and acquires the attitude of the goggle device 10. Further, the processor 20 sets the attitude of the virtual camera VC (left virtual camera CVL and right virtual camera VCR) in the virtual space according to the calculated attitude of the goggle device 10. For example, the processor 20 sets the attitude of each virtual camera VC so that the attitudes of the left virtual camera VCL and the right virtual camera VCR in the virtual space coincide with the attitude of the goggle device 10 in the real space. Thereby, for example, when the goggle device 10 rotates 5 degrees to the left, the left virtual camera VCL and the right virtual camera VCR also rotate 5 degrees to the left.

[0143] Subsequently, the processor 20 determines whether or not the rotation angle in the pitch direction of the line of sight of the virtual camera VC is equal to or greater than a second threshold value (Tx) based on the attitude calculated in step S102 (step S103). When it is determined that the rotation angle in the pitch direction of the line of sight of the virtual camera VC is equal to or greater than the second threshold value (Tx) (step S103: YES), the processor 20 performs a movement process in the A region (step S104). When the process of step S104 is executed, the processor 20 then executes the process of step S108.

[0144] Specifically, in step S104, the processor 20 moves the UI object 30 so as to be in front in the left - right direction of the virtual camera VC. The processor 20 moves the UI object 30 on a plane parallel to the xz plane of the virtual space while maintaining the height of the UI object 30 in the virtual space. For example, the processor 20 moves the UI object 30 along a circle centered on the position of the virtual camera VC (for example, the middle of the left virtual camera VCL and the right virtual camera VCR) so that the UI object 30 is positioned in front in the left - right direction of the virtual camera VC. Thereby, the UI object 30 moves as shown in FIG. 12. Note that the processor 20 may move the UI object 30 by a predetermined distance within one frame time. In this case, the process of step S104 is repeatedly executed for each frame time, and the state in which the UI object 30 moves to the front in the left - right direction of the virtual camera VC is displayed on the display unit 21.

[0145] On the other hand, when it is determined that the rotation angle of the line of sight of the virtual camera VC in the pitch direction is less than the second threshold value (Tx) (step S103: NO), the processor 20 determines whether the rotation angle of the line of sight of the virtual camera VC in the yaw direction is greater than or equal to the first threshold value (Ty) (step S105).

[0146] When it is determined that the rotation angle of the line of sight of the virtual camera VC in the yaw direction is greater than or equal to the first threshold value (Ty) (step S105: YES), the processor 20 performs a movement process in region B (step S106). Details of the movement process in region B will be described later. When the process of step S106 is executed, the processor 20 then executes the process of step S108.

[0147] When it is determined that the rotation angle in the yaw direction of the line of sight of the virtual camera VC is less than the first threshold value (Ty) (step S105: NO), the processor 20 performs UI processing (step S107). In this UI processing, selection processing of each icon in the UI object 30 and processing after selection are performed. For example, when a pointer 37 located at a predetermined position (for example, the center) of the stereoscopic image is within the display area of the icon 31 in the UI object 30, the icon 31 is selected. Each icon is assigned processing (for example, screen switching, operations in a game, activation of other applications, etc.) when selected. The processor 20 performs processing according to the selected icon. Note that in step S107, the processor 20 resets the change amounts Rx, Ry, and the cumulative values X1, X2, Y1, Y2 calculated in step S106 described later to "0" respectively. When the processing of step S107 is executed, the processor 20 then executes the processing of step S108.

[0148] In step S108, the processor 20 performs image generation processing. Specifically, the processor 20 generates a left-eye image by imaging the virtual space using the left virtual camera VCL, and generates a right-eye image by imaging the virtual space using the right virtual camera VCR.

[0149] Then, the processor 20 causes the display unit 21 to display the left-eye image and the right-eye image generated in step S108 (step S109). Specifically, the processor 20 causes the left-eye image to be displayed in the left region 21L and the right-eye image to be displayed in the right region 21R. When the processing of step S109 is executed, the processor 20 executes the processing of step S101 again.

[0150] (Movement processing in the B region) Next, the movement processing in the B region of step S106 will be described. FIG. 16 is a flowchart showing an example of the movement processing in the B region of step S106.

[0151] As shown in FIG. 16, the processor 20 first calculates an upward change amount x1 and a downward change amount x2 with respect to the rotation angle in the pitch direction in the virtual space of the line of sight of the virtual camera VC (step S130). For example, the processor 20 determines whether the line of sight has changed upward or downward based on the posture of the virtual camera VC in the previous processing loop and the posture of the virtual camera VC in the current processing loop, and calculates the change amount x1 or x2. Then, the calculated change amount x1 or x2 is added to the cumulative value X1 or X2. For example, when the line of sight of the virtual camera VC continues to change upward, the cumulative value X1 increases, while the downward change amount x2 becomes "0" degrees, and the cumulative value X2 also becomes "0" degrees. Also, for example, as in the example of FIG. 11, when the virtual camera VC rotates downward by x1 degrees and then rotates upward by x2 degrees, "x1" is set for the cumulative value X1 and "x2" is set for the cumulative value X2, respectively.

[0152] Next, the processor 20 sets the smaller value of the cumulative values X1 and X2 updated in step S130 as the change amount Rx in the vertical direction (step S131).

[0153] Subsequently, the processor 20 calculates a leftward change amount y1 and a rightward change amount y2 with respect to the rotation angle in the yaw direction in the virtual space of the line of sight of the virtual camera VC (step S132). For example, the processor 20 determines whether the line of sight has changed leftward or rightward based on the posture of the virtual camera VC in the previous processing loop and the posture of the virtual camera VC in the current processing loop, and calculates the change amount y1 or y2. Then, the calculated change amount y1 or y2 is added to the cumulative value Y1 or Y2. For example, when the line of sight of the virtual camera VC continues to change leftward, the cumulative value Y1 increases, while the rightward change amount y2 becomes "0" degrees, and the cumulative value Y2 also becomes "0" degrees. Also, for example, as in the example of FIG. 10, when the virtual camera VC rotates leftward by y1 degrees and then rotates rightward by y2 degrees, "y1" is set for the cumulative value Y1 and "y2" is set for the cumulative value Y2, respectively.

[0154] Next, the processor 20 sets the smaller value of the cumulative values Y1 and Y2 updated in step S132 as the amount of change Ry in the horizontal direction (step S133).

[0155] Subsequently, the processor 20 determines whether the sum of Rx set in step S131 and Ry set in step S133 is greater than or equal to a predetermined value (step S134).

[0156] When it is determined that "Rx + Ry" is greater than or equal to the predetermined value (step S134: YES), the processor 20 moves the UI object 30 so as to be in front of the virtual camera VC (step S135). Specifically, the processor 20 moves the UI object 30 on a plane parallel to the xz plane of the virtual space while maintaining the height of the UI object 30 in the virtual space. For example, the processor 20 moves the UI object 30 along a circle centered on the position of the virtual camera VC (for example, the middle between the left virtual camera VCL and the right virtual camera VCR) so that the UI object 30 is located in front of the virtual camera VC in the horizontal direction. As a result, the UI object 30 moves as shown in FIG. 8.

[0157] When the process of step S135 is executed, or when it is determined as NO in step S134, the processor 20 determines whether a predetermined operation has been performed (step S136). Here, the "predetermined operation" may be an operation such as pressing, tapping, or striking a predetermined button or location on the goggle device 10.

[0158] When it is determined that a predetermined operation has been performed (step S136: YES), the processor 20 executes the process of step S137. The process of step S137 is the same as the process of step S135.

[0159] When the process of step S137 is executed, or when it is determined as NO in step S136, the processor 20 ends the process shown in FIG. 16 and returns to the process in FIG. 15.

[0160] Note that the processes shown in the above flowchart are merely examples, and the order and content of the processes, as well as the threshold values used for determination, may be changed as appropriate. Also, a threshold value may or may not be included in the determination. For example, the determination of "whether a certain value is greater than or equal to a threshold value" may be replaced with the determination of "whether a certain value is greater than a threshold value".

[0161] As described above, when the line of sight of the virtual camera VC is in the B region, in the movement process in the B region, it is detected that the line of sight of the virtual camera VC has changed from one direction to the other direction with respect to the yaw direction or the pitch direction. Specifically, when the value of Rx or Ry set in step S131 or step S133 exceeds "0" degrees, it means that it has changed from one direction to the other direction. When the line of sight of the virtual camera VC changes from one direction to the other direction, and the accumulated value (Rx and / or Ry) of the rotation angle in the other direction becomes equal to or greater than a predetermined value, the UI object 30 is moved to the front of the virtual camera VC. Thereby, the user can look around the virtual space other than the UI object 30, and when the user wants to view the UI object 30, the UI object 30 can be moved to the front.

[0162] Also, when the line of sight of the virtual camera VC is in the A region, in the movement process in the A region, the UI object 30 is always moved so as to be in front with respect to the left - right direction of the virtual camera VC. Thereby, for example, by turning the goggle device 10 upward or downward, the UI object 30 can be moved to the front.

[0163] (Modification example) The image processing of the present embodiment has been described above. However, the above embodiment is merely an example, and for example, the following modifications may be added.

[0164] For example, in the above embodiment, in the movement process in area A and the movement process in area B, the UI object 30 was moved so that the UI object 30 was located in front of the virtual camera VC. In other embodiments, the virtual camera VC may be controlled (change of posture and / or movement) so that the UI object 30 is located in front of the virtual camera VC. For example, the UI object 30 may be fixed and the posture of the virtual camera VC may be changed, or the virtual camera VC may be moved, so that the UI object 30 is located in front of the virtual camera VC. Also, in other embodiments, both the movement of the UI object 30 and the control of the virtual camera VC may be performed so that the UI object 30 is located in front of the virtual camera VC. That is, at least either one of the movement of the UI object 30 and the control of the virtual camera VC (change of posture and / or movement) may be performed so that the UI object 30 is located in front of the virtual camera VC. When the movement of the UI object 30 is performed, as shown in the above embodiment, the UI object 30 may be moved and rotated so that the UI object 30 faces the virtual camera VC.

[0165] Also, in the above embodiment, the UI object 30 was moved in front of the virtual camera VC when the condition was satisfied. In other embodiments, the UI object 30 may be moved, the virtual camera VC may be controlled (for example, movement and / or rotation of the virtual camera), or both the movement of the UI object 30 and the control of the virtual camera VC may be performed so that at least a part of the UI object 30 is located within the imaging range of the virtual camera VC when the condition is satisfied.

[0166] In the above embodiment, in the movement process in the B region, when the rotation of the virtual camera VC in the yaw direction or the pitch direction changes from one direction to the other direction, it is determined whether the smaller value among the value obtained by accumulating the rotation angle in the other direction and the value obtained by accumulating the rotation angle in one direction is equal to or greater than a predetermined value. Specifically, it is determined whether the sum of the change amounts Ry and Rx is equal to or greater than a predetermined value. The determination of whether to move the UI object 30 may be performed by other methods. For example, in another embodiment, it may be determined whether either one of the change amounts Ry and Rx is equal to or greater than a predetermined value. Further, when the rotation of the virtual camera VC in the yaw direction or the pitch direction changes from one direction to the other direction, and the value obtained by accumulating the rotation angle in the other direction and / or the value obtained by accumulating the rotation angle in one direction is equal to or greater than a predetermined value, the UI object 30 may be moved.

[0167] In another embodiment, without accumulating the rotation angle in the other direction and the rotation angle in one direction, the UI object 30 may be moved when the rotation angle in the other direction and / or the rotation angle in one direction becomes equal to or greater than a predetermined value. In this case, when the rotation of the virtual camera VC in one direction and the rotation in the other direction are repeated a plurality of times, if the rotation angle in the other direction or the rotation angle in one direction in each rotation is less than the predetermined value, the UI object 30 is not moved. On the other hand, when the rotation of the virtual camera VC in one direction and the rotation in the other direction are repeated a plurality of times, if the rotation angle in the other direction or the rotation angle in one direction in one rotation becomes equal to or greater than a predetermined value, the UI object 30 is moved.

[0168] Also, the determination of whether the change amount Ry regarding the yaw direction is equal to or greater than a predetermined value and the determination of whether the change amount Rx regarding the pitch direction is equal to or greater than a predetermined value may be performed separately. In this case, when either one of Rx and Ry becomes equal to or greater than a predetermined value, the UI object 30 is moved.

[0169] Also, the threshold values may be made different between the change amount Ry related to the yaw direction and the change amount Rx related to the pitch direction. For example, for the yaw direction, if Ry is equal to or greater than a first predetermined value (for example, 4 to 8 degrees), the UI object 30 may be moved. For the pitch direction, if Rx is equal to or greater than a second predetermined value (for example, 10 to 20 degrees), which is greater than the first predetermined value, the UI object 30 may be moved. Further, these threshold values may be made different according to the type of the displayed UI object 30 and the scene (background scene) of the virtual space.

[0170] Also, in the above embodiment, regarding the yaw direction and the pitch direction, taking the fact that the line of sight of the virtual camera VC has reversed from one direction to the other direction as the first condition, when the first condition is satisfied, and further when a second condition is satisfied (specifically, when the cumulative value of the rotation angle in the other direction becomes equal to or greater than a predetermined value), the UI object 30 is moved to the front of the virtual camera VC. In other embodiments, when the first condition is satisfied, that is, when the line of sight of the virtual camera VC has reversed from one direction to the other direction, the UI object 30 may be moved to the front of the virtual camera VC. In this case, regardless of the cumulative value, the UI object 30 moves to the front of the virtual camera VC at the moment when the line of sight of the virtual camera VC reverses.

[0171] Also, in other embodiments, when the number of reversals of the line of sight of the virtual camera VC becomes equal to or greater than a predetermined number of times, the UI object 30 may be moved to the front of the virtual camera VC. For example, when the virtual camera VC rotates to the left direction and enters the B area, and then repeats the rotation to the right direction and the rotation to the left direction, the UI object 30 may be moved when the number of reversals (the reversal to the right direction and the reversal to the left direction) becomes equal to or greater than a predetermined number of times.

[0172] Also, in other embodiments, when the line of sight of the virtual camera VC reverses from one direction (for example, the left direction) to the other direction (for example, the right direction), the UI object 30 may be moved based on the angles in the other direction (the right direction) and the one direction (the left direction) after the reversal.

[0173] That is, in the movement process in the B area, the UI object 30 may be moved to the front of the virtual camera VC based on the fact that the line of sight of the virtual camera VC is reversed from one direction to the other direction. Here, "based on the fact that the line of sight of the virtual camera VC is reversed from one direction to the other direction" includes, for example, the following matters. · When the line of sight of the virtual camera VC is reversed from one direction to the other direction in a predetermined rotation direction, and further, the rotation angle in the other direction or its cumulative value (the above Rx or Ry) becomes a predetermined value or more. · The number of reversals of the line of sight of the virtual camera VC in a predetermined rotation direction becomes a predetermined value. · The line of sight of the virtual camera VC changes from one direction to the other direction in a predetermined rotation direction.

[0174] Also, in the above embodiment, regarding whether the line of sight of the virtual camera VC is in either the "A area" or the "B area" shown in FIG. 9, it is determined based on the angle of the line of sight of the virtual camera VC, and the UI object 30 is moved according to each area. In other embodiments, not limited to the angle of the line of sight of the virtual camera VC, for example, the UI object 30 may be moved according to whether the position of the pointer is in either the "A area" or the "B area". For example, a pointer is displayed at a predetermined position (for example, the center) of the stereoscopic image, and when the pointer is in the A area, the above "movement process in the A area" may be performed, and when the pointer is in the B area, the above "movement process in the B area" may be performed.

[0175] Also, in the above embodiment, when the condition is satisfied, the UI object 30 is moved. The object to be moved is not limited to the UI object 30, and may be any object. For example, when the condition is satisfied, a character object that conveys predetermined information to the user may be moved to the front of the virtual camera VC.

[0176] In the above-described embodiment, the posture of the goggle device 10 (information processing device 2) is detected based on data from the inertial sensor 22 built into the information processing device 2. In other embodiments, the posture of the goggle device 10 may be detected by other methods. For example, the image display system 1 includes a camera that images the goggle device 10 from the outside, and the camera images the goggle device 10 or a marker attached to the goggle device 10, and the posture of the goggle device 10 may be obtained based on the captured image. Further, the goggle device 10 may include a camera, and the posture of the goggle device 10 may be obtained from changes in the image captured by the camera.

[0177] Also, in the above-described embodiment, the rotation direction of the line of sight of the virtual camera VC is divided into a component in the left-right direction (yaw direction) and a component in the up-down direction (pitch direction) in the virtual space, and the UI object 30 is moved based on the fact that the line of sight of the virtual camera VC has changed from one direction to the other direction with respect to the component in the left-right direction or the up-down direction. For example, when the line of sight of the virtual camera VC changes from one direction (e.g., the left direction) to the other direction (e.g., the right direction) in the component in the left-right direction, the rotation amount in the other direction and / or the one direction in the component in the left-right direction is calculated, and when the calculated rotation amount is equal to or greater than a predetermined value (the predetermined value may or may not be included), the UI object 30 is moved. In other embodiments, the UI object 30 may be moved based on the fact that the line of sight of the virtual camera VC has changed from one direction (e.g., the direction from the lower left to the upper right) to the other direction (e.g., the direction from the upper right to the lower left) in the diagonal direction.

[0178] That is, the UI object 30 may be moved based on the fact that a predetermined component (e.g., the left-right direction component, the up-down direction component, the diagonal direction component) of the rotation direction of the line of sight of the virtual camera VC has changed from one direction to the other direction.

[0179] In the above-described embodiment, when the line of sight of the virtual camera VC is in the A region, the UI object 30 is moved following the rotation of the virtual camera VC in the left-right direction. In other embodiments, after the line of sight of the virtual camera VC exits the UI region and enters the A region, the UI object 30 may be moved to the front of the virtual camera VC in the left-right direction (yaw direction) (or the virtual camera VC may be controlled) at the timing when the line of sight of the virtual camera VC returns to the UI region. That is, if the UI object 30 is controlled to be visible in the front when the user removes the goggle device 10 (for example, after turning the bottom surface 15 of the goggle device 10 upward or downward) and then places the goggle device 10 on the face to view the display unit, the timing for moving the UI object 30 may be anytime. For example, when the line of sight of the virtual camera VC enters the A region, the UI object 30 may be moved to follow the rotation of the virtual camera VC in the left-right direction, or when the line of sight of the virtual camera VC returns to the UI region after entering the A region, the UI object 30 may be moved to the front of the virtual camera VC in the left-right direction.

[0180] In the above-described embodiment, when the line of sight of the virtual camera VC is within the A region, control is performed such that the UI object 30 is positioned in front of the virtual camera VC in the yaw direction. Further, when the line of sight of the virtual camera VC is within the B region, control is performed such that the UI object 30 is positioned in front of the virtual camera VC on the condition that the line of sight of the virtual camera VC is reversed. The "front" as used herein includes not only the exact front but also the substantial front. The exact front means that the straight line extending the line of sight of the virtual camera VC intersects a straight line parallel to the y-axis passing through the center in the left-right direction of the UI object 30. In other words, the exact front means that the line of sight of the virtual camera VC is not shifted in the left-right direction of the UI object 30. The substantial front includes cases where the line of sight of the virtual camera VC is shifted in the left-right direction of the UI object 30. For example, "the UI object 30 is positioned in the substantial front of the virtual camera VC" may mean that a plane perpendicular to the xz plane passing through the line of sight of the virtual camera VC intersects the UI object 30. In this case, when the rotation angle of the virtual camera VC in the pitch direction with respect to the UI object 30 is set to "0" degrees, the straight line extending the line of sight of the virtual camera VC hits the UI object 30. Further, "moving the UI object so that the UI object 30 is positioned in the substantial front of the virtual camera VC" means that when the rotation angle of the virtual camera VC in the pitch direction with respect to the UI object 30 is set to "0" degrees, at least a part of the UI object 30 enters the field of view of the virtual camera VC, and the rotation angle of the virtual camera VC in the yaw direction with respect to the UI object 30 becomes smaller than before the movement of the UI object. That is, the "front" in this specification means that the rotation angle of the virtual camera in the yaw direction with respect to the UI object (predetermined object) is a predetermined angle. The "predetermined angle" may be an angle at which at least a part of the predetermined object is included in the field of view of the virtual camera when the rotation angle of the virtual camera in the pitch direction with respect to the predetermined object is set to "0" degrees.

[0181] Also, in the above embodiment, when the line of sight of the virtual camera VC is in the upper A region or the lower A region of the UI region, the movement process in the above A region is performed. In other embodiments, the movement process in the above A region may be performed only when the line of sight of the virtual camera VC is in the upper A region of the UI region. Conversely, the movement process in the above A region may be performed only when the line of sight of the virtual camera VC is in the lower A region of the UI region.

[0182] Also, the sizes and shapes of the respective regions shown in FIG. 9 are merely examples, and each region may have any other size and shape. Also, when the line of sight of the virtual camera VC is at the boundary of each region, it may be determined that the line of sight of the virtual camera VC is in the UI region, or it may be determined that it is in a region different from the UI region (that is, the A region or the B region).

[0183] Also, in the above embodiment, based on the two left virtual cameras VCL and the right virtual camera VCR, left-eye images and right-eye images having parallax with each other are generated. In other embodiments, an image may be generated based on one virtual camera, and the generated image may be deformed to generate left-eye images and right-eye images having parallax with each other. That is, in this specification, "generating left-eye images and right-eye images based on a virtual camera" includes both generating left-eye images and right-eye images based on a pair of left virtual cameras VCL and right virtual camera VCR (that is, based on a plurality of virtual cameras) and generating left-eye images and right-eye images based on one virtual camera.

[0184] In the above-described embodiment, the display unit 21 of the goggle device 10 is detachable from the goggle body. In other embodiments, the display unit 21 of the goggle device 10 may be fixed to the goggle body. That is, the goggle device 10 may be configured such that a display unit for displaying a left-eye image and a right-eye image and an information processing device for generating the left-eye image and the right-eye image are integrated. Further, the display unit of the goggle device 10 is not limited to the display unit 21 as described above. For example, the display unit of the goggle device 10 may have two display units (a left-eye display unit visually recognized by the user's left eye and a right-eye display unit visually recognized by the user's right eye). Further, the display unit of the goggle device 10 may have any shape. For example, the display unit itself of the goggle device 10 may be formed in a substantially circular shape (circular or elliptical) like the left opening 16L and the right opening 16R. Further, two left and right display units formed in a square or rectangular shape may be used as the display unit of the goggle device 10. Further, the display unit of the goggle device 10 may be a display device such as a liquid crystal display device or an organic EL display device, or may be a projection-type display device that projects an image onto a projection surface.

[0185] Also, the configuration of the image display system 1 of the above embodiment is merely an example and is not limited to the above. For example, in the above embodiment, the goggle device 10 is configured by an information processing device 2 having a display unit 21 for displaying an image and a processor 20 for performing processing for generating an image, and a goggle body. That is, the image display system 1 is configured by the goggle device 10 including the goggle body and the information processing device 2. In other embodiments, a goggle device having a display unit (a goggle device integrated with the display unit) and an information processing device for performing processing such as generating an image may be configured as separate devices, and the image display system 1 may be configured by these multiple devices. In this case, the goggle device and the information processing device may be connected by wire or wirelessly, and the left-eye image and the right-eye image generated in the information processing device may be transmitted to the goggle device and visually recognized by the user. Further, the information processing device may perform processing related to the movement of the UI object 30 described above, transmit the result of the processing to the goggle device, and the goggle device may generate the left-eye image and the right-eye image and cause the user to visually recognize them. Also, the goggle device and the information processing device may be connected by a network (LAN, WAN, Internet, etc.).

[0186] In the present embodiment, a goggle device in which the user peeks into the display unit while holding it by hand is used. However, in other embodiments, the goggle device may be a head-mounted display that does not require the user to hold it by hand and is fixedly worn on the user's head.

[0187] Although the present invention has been described above, the above description is merely an exemplification of the present invention, and various improvements and modifications may be added.

Explanation of Reference Numerals

[0188] 1 Image display system 2 Information processing device 10 Goggle device 11 Upper surface 12 Right side surface 13 Lower surface 14 Left side surface 15 Bottom surface 16L Left opening 16R Right opening 17 Partition surface 20 Processor 21 Display unit 22 Inertial sensor 30 UI object 31 - 34 Icons 35 Information display object 37 Pointer VC Virtual camera VCL Left virtual camera VCR Right virtual camera

Claims

1. A goggle device, object placement means for placing an object in a virtual space, display control means for causing an image captured by a virtual camera in the virtual space to be displayed on a display unit of the goggle device, posture acquisition means for acquiring the posture of the goggle device, camera rotation means for rotating the virtual camera in the virtual space based on the posture of the goggle device, first detection means for detecting that the line of sight of the virtual camera has changed from one direction to the other direction in a predetermined rotation direction, second detection means for detecting that a predetermined location of the goggle device has been tapped or struck, first control means for performing at least one of moving the object and controlling the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera based on the change detected by the first detection means, second control means for performing at least one of moving the object and controlling the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera based on the tapping or striking of a predetermined location of the goggle device. An image display system comprising:

2. When the rotation of the virtual camera is within a predetermined range, the second control means performs at least one of moving the object and controlling the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera even when the change is not detected by the first detection means, based on the tapping or striking of a predetermined location of the goggle device. The image display system according to claim 1.

3. Further comprising calculation means for calculating the rotation amount of the virtual camera, The first control means performs at least one of moving the object and controlling the virtual camera based on at least the change detected by the first detection means and the rotation amount. The image display system according to claim 1 or 2.

4. When the change is detected by the first detection means, the first control means performs at least one of moving the object and controlling the virtual camera when the rotation amount after the detection is equal to or greater than a predetermined value. The image display system according to claim 3.

5. The calculating means calculates the amount of rotation in the other direction and / or the one direction in the predetermined rotational direction. The first control means performs at least one of the movement of the object and the control of the virtual camera based on the amount of rotation in the other direction and / or the one direction in the predetermined rotational direction. The image display system according to claim 3 or 4.

6. When the change is detected by the first detection means, the first control means performs at least one of the movement of the object and the control of the virtual camera when the amount of rotation in the other direction and / or the one direction is equal to or greater than a predetermined value. The image display system according to claim 5.

7. When the cumulative value of the amount of rotation in the other direction and / or the one direction after the change is detected by the first detection means is equal to or greater than a predetermined value, the first control means performs at least one of the movement of the object and the control of the virtual camera. The image display system according to claim 6.

8. The calculating means calculates a first amount of rotation in the one direction in the predetermined rotational direction and a second amount of rotation in the other direction in the predetermined rotational direction. The image display system further includes a comparing means for comparing the first amount of rotation with the second amount of rotation. The first detection means detects the change based on the comparison result of the comparing means. The image display system according to any one of claims 3 to 7.

9. The first detection means detects the change based on the smaller one of the first amount of rotation and the second amount of rotation. The image display system according to claim 8.

10. The calculating means calculates a cumulative value of the first amount of rotation and a cumulative value of the second amount of rotation. The first detection means detects the change based on the smaller one of the cumulative value of the first amount of rotation and the cumulative value of the second amount of rotation. The image display system according to claim 8.

11. The camera rotation means rotates the virtual camera so as to be at least in a first posture in which at least a part of the object is located within the imaging range of the virtual camera and a second posture in which the object is located outside the imaging range of the virtual camera. The one direction is a direction in which the virtual camera approaches the second posture. The other direction is a direction in which the virtual camera approaches the first posture. The image display system according to any one of claims 1 to 10.

12. The camera rotation means rotates the virtual camera so as to be at least in a first posture in which the virtual camera faces the object and a second posture in which the virtual camera does not face the object, wherein the one direction is a direction in which the virtual camera moves away from the first posture, and the other direction is a direction in which the virtual camera approaches the first posture. The image display system according to any one of claims 1 to 10. **Claim 13** The first detection means detects the change when the line of sight of the virtual camera rotates in the other direction when the virtual camera is in the second posture, and does not detect the change when the line of sight of the virtual camera rotates in the one direction without rotating in the other direction or when the line of sight of the virtual camera does not rotate when the virtual camera is in the second posture, wherein the first control means performs at least one of the movement of the object and the control of the virtual camera when the change is detected, and does not perform at least one of the movement of the object and the control of the virtual camera when the change is not detected. The image display system according to claim 11 or 12. **Claim 14** The second posture is a posture in which the object exists in the left-right direction of the line of sight of the virtual camera. The image display system according to any one of claims 11 to 13. **Claim 15** The predetermined rotation direction is the yaw direction in the virtual space. The image display system according to any one of claims 1 to 14. **Claim 16** The camera rotation means can rotate the line of sight of the virtual camera in a second rotation direction orthogonal to the predetermined rotation direction, and the first detection means further detects that the line of sight of the virtual camera has changed from one direction to the other direction in the second rotation direction. The image display system according to any one of claims 1 to 15. **Claim 17** The predetermined rotation direction is the pitch direction in the virtual space. The image display system according to any one of claims 1 to 16. **Claim 18** The object is a user interface operable by a user. The image display system according to any one of claims 1 to 17. **Claim 19** The image display system further includes pointer setting means for setting a pointer for instructing the user interface. The image display system according to claim 18, wherein the first detection means detects the change when the pointer is not in a predetermined area with reference to the user interface.

20. The image display system according to claim 19, wherein the first detection means calculates a change amount of rotation in the one direction and the other direction in the predetermined rotation direction when the pointer is not in the predetermined area.

21. The image display system according to claim 20, wherein the first detection means resets the change amount when the pointer enters the predetermined area.

22. The one direction is a direction in which the pointer leaves the predetermined area, The image display system according to any one of claims 19 to 21, wherein the first detection means detects the change when the moving direction of the pointer is reversed while the pointer is moving in a direction away from the predetermined area.

23. The image display system according to any one of claims 1 to 22, wherein the first detection means detects the change when the line of sight of the virtual camera with reference to the object is at a predetermined angle.

24. The image display system further comprises selection means for selecting the object according to the posture of the virtual camera, The first control means performs at least one of movement of the object and control of the virtual camera when the object is not selected by the selection means, according to any one of claims 1 to 23.

25. Even when the change is not detected by the first detection means, the image display system according to any one of claims 1 to 24 further comprises third control means for moving the object or controlling the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera in response to a predetermined operation by the user.

26. The first detection means detects that the number of reversals of the line of sight of the virtual camera from the one direction to the other direction and from the other direction to the one direction has reached a predetermined number, The first control means performs at least one of movement of the object and control of the virtual camera when it is detected by the first detection means that the number of reversals has reached the predetermined number, according to any one of claims 1 to 25.

27. An image display program executed by a processor of a device that causes a display unit of a goggle device to display an image, the processor being configured to: Object placement means for placing an object in a virtual space; Display control means for causing the display unit to display an image captured by a virtual camera in the virtual space; Posture acquisition means for acquiring the posture of the goggle device; Camera rotation means for rotating the virtual camera in the virtual space based on the posture of the goggle device; First detection means for detecting that the line of sight of the virtual camera has changed from one direction to another in a predetermined rotation direction; Second detection means for detecting that a predetermined location of the goggle device has been tapped or struck; First control means for performing at least one of moving the object and controlling the virtual camera so that at least a part of the object is within the imaging range of the virtual camera based on the change detected by the first detection means, and An image display program that functions as second control means for performing at least one of moving the object and controlling the virtual camera so that at least a part of the object is within the imaging range of the virtual camera based on the fact that a predetermined location of the goggle device has been tapped or struck.

28. A display control device that causes a display unit of a goggle device to display an image, the display control device comprising: Object placement means for placing an object in a virtual space; Display control means for causing the display unit to display an image captured by a virtual camera in the virtual space; Posture acquisition means for acquiring the posture of the goggle device; Camera rotation means for rotating the virtual camera in the virtual space based on the posture of the goggle device; First detection means for detecting that the line of sight of the virtual camera has changed from one direction to another in a predetermined rotation direction; Second detection means for detecting that a predetermined location of the goggle device has been tapped or struck; First control means for performing at least one of moving the object and controlling the virtual camera so that at least a part of the object is within the imaging range of the virtual camera based on the change detected by the first detection means; Based on a predetermined location of the goggle device being tapped or struck, second control means for performing at least one of the movement of the object and the control of the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera, a display control device comprising the same.

29. An image display method performed in an image display system including a goggle device, the image display system An object placement step of placing an object in a virtual space; A display control step of displaying an image captured by a virtual camera in the virtual space on a display unit of the goggle device; An attitude acquisition step of acquiring the attitude of the goggle device; A camera rotation step of rotating the virtual camera in the virtual space based on the attitude of the goggle device; A first detection step of detecting that the line of sight of the virtual camera has changed from one direction to the other direction in a predetermined rotation direction; A second detection step of detecting that a predetermined location of the goggle device has been tapped or struck; A first control step of performing at least one of the movement of the object and the control of the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera based on the change being detected in the first detection step; A second control step of performing at least one of the movement of the object and the control of the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera based on a predetermined location of the goggle device being tapped or struck, an image display method for executing the same.

30. A goggle device, Object placement means for placing an object in a virtual space; Display control means for displaying an image captured by a virtual camera in the virtual space on a display unit of the goggle device; Attitude acquisition means for acquiring the attitude of the goggle device; Camera rotation means for rotating the virtual camera in the virtual space based on the attitude of the goggle device; Detection means for detecting that a predetermined location of the goggle device has been tapped or struck When the rotation of the virtual camera exceeds a predetermined value, based on the fact that a predetermined location of the goggle device has been tapped or struck, control means for performing at least one of the movement of the object and the control of the virtual camera so that at least a part of the object is located within the imaging range of the virtual camera. An image display system comprising:

31. An image display program executed in a processor of a device that displays an image on a display unit of a goggle device, the processor being Object placement means for placing an object in a virtual space, Display control means for displaying an image captured by a virtual camera in the virtual space on the display unit, Posture acquisition means for acquiring the posture of the goggle device, Camera rotation means for rotating the virtual camera in the virtual space based on the posture of the goggle device, Detection means for detecting that a predetermined location of the goggle device has been tapped or struck, and When the rotation of the virtual camera exceeds a predetermined value, based on the fact that a predetermined location of the goggle device has been tapped or struck, at least one of the movement of the object and the control of the virtual camera is performed so that at least a part of the object is located within the imaging range of the virtual camera. An image display program that functions as control means.

32. A display control device for displaying an image on a display unit of a goggle device, Object placement means for placing an object in a virtual space, Display control means for displaying an image captured by a virtual camera in the virtual space on the display unit, Posture acquisition means for acquiring the posture of the goggle device, Camera rotation means for rotating the virtual camera in the virtual space based on the posture of the goggle device, Detection means for detecting that a predetermined location of the goggle device has been tapped or struck, and When the rotation of the virtual camera exceeds a predetermined value, based on the fact that a predetermined location of the goggle device has been tapped or struck, at least one of the movement of the object and the control of the virtual camera is performed so that at least a part of the object is located within the imaging range of the virtual camera. A display control device comprising control means.

33. An image display method performed in an image display system including a goggle device, the image display system an object placement step of placing an object in a virtual space; a display control step of causing an image captured by a virtual camera in the virtual space to be displayed on a display unit of the goggle device; a posture acquisition step of acquiring a posture of the goggle device; a camera rotation step of rotating the virtual camera in the virtual space based on the posture of the goggle device; a second detection step of detecting that a predetermined location of the goggle device has been tapped or struck; a control step of performing at least one of movement of the object and control of the virtual camera so that at least a part of the object is located within an imaging range of the virtual camera based on the fact that a predetermined location of the goggle device has been tapped or struck when rotation of the virtual camera exceeds a predetermined value. An image display method for executing the above steps.

Images