Information processing system, information processing program, information processing device, and information processing method

Abstract

To provide an information processing system in which a virtual object can be arranged in a position that a user wants the virtual object to be.SOLUTION: An example of the information processing system sets the direction of imaging of a virtual camera according to the direction of imaging of the camera, and moves a virtual door in a virtual space according to the direction of imaging of the virtual camera if the direction of the imaging of the camera is in a predetermined range. If the direction of imaging of the camera is more upward than a first threshold value, the virtual door is not moved further upward. If the direction of imaging of the camera is more downward than a second threshold value, the virtual door is not moved further downward.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing program, an information processing device, and an information processing method that are capable of synthesizing images of real space and virtual space. [Background technology]

[0002] As a prior art, there is an image processing system that arranges a virtual object in a virtual space and displays a composite image of the real space and the virtual object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-071877 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is room for improvement in placing virtual objects in appropriate positions.

[0005] Therefore, an object of the present invention is to provide an information processing system, an information processing program, an information processing device, and an information processing method that are capable of placing a virtual object at an appropriate position. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention employs the following configuration.

[0007] One example of the present invention is an information processing system that outputs to a display device a composite image that is a combination of real images captured sequentially of a real space by an imaging device and virtual images captured sequentially of a virtual object by a virtual camera placed in a virtual space, and includes a control means that controls at least one of the imaging direction of the virtual camera and the position of the virtual object according to the imaging direction of the imaging device so that the imaging direction of the virtual camera and the virtual object direction, which is the direction from the virtual camera to the virtual object, are linked with a first degree of linkage, and when the imaging direction of the imaging device is pointing upward than a first threshold value or downward than a second threshold value in the real space, the control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object so that the imaging direction of the virtual camera and the virtual object direction are linked with a second degree of linkage that is smaller than the first degree of linkage. Another example of the present invention is an information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space with virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object. The information processing system includes a first control means, a control state change means, and a second control means. In a first control state, the first control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object in accordance with the imaging direction of the imaging device so that the imaging direction of the virtual camera and the virtual object direction, which is the direction from the virtual camera to the virtual object, are linked with a first degree of linkage. The control state change means changes the first control state to a second control state based on a user operation. In the second control state, the second control means controls at least one of the position, imaging direction, and position of the virtual camera in accordance with the position or imaging direction of the imaging device so that the virtual object viewed from the composite image is pseudo-fixed to a position in the real space. When, in the first control state, the imaging direction of the imaging device is directed upwardly relative to a first threshold value or downwardly relative to a second threshold value in the real space, the first control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object so that the imaging direction of the virtual camera and the virtual object direction are linked with a second degree of linkage that is smaller than the first degree of linkage.

[0008] According to the above, in the first control state, the imaging direction of the virtual camera and the virtual object direction can be linked in response to a change in the imaging direction of the imaging device. The control state is changed to the second control state in response to a user operation, and in the second control state, the virtual object is pseudo-fixed in real space. This allows the virtual object to be pseudo-fixed at a desired position in real space. Furthermore, in the first control state, when the imaging direction of the imaging device is pointing upward more than a first threshold, the degree of linkage between the imaging direction of the virtual camera and the virtual object direction can be made smaller than when the imaging direction of the imaging device is not pointing upward more than the first threshold. Similarly, in the first control state, when the imaging direction of the imaging device is pointing downward more than a second threshold, the degree of linkage between the imaging direction of the virtual camera and the virtual object direction can be made smaller than when the imaging direction of the imaging device is not pointing downward more than the second threshold. This makes it possible to prevent a virtual object from being placed in an inappropriate position.

[0009] The information processing system may further include an event processing execution means that executes event processing related to the virtual object when position conditions related to the position of the virtual camera and the position of the virtual object are satisfied in the second control state.

[0010] Based on the above, when a position condition regarding the position of the virtual camera and the position of the virtual object is satisfied, it is possible to execute event processing regarding the virtual object.

[0011] Furthermore, the position condition may be satisfied when the virtual camera approaches within a predetermined distance from the virtual object. Furthermore, a second virtual space may be set based on the position of the virtual object. The event processing may be processing for determining that the virtual camera has entered the second virtual space, and when the virtual camera has entered the second virtual space, the position or imaging direction of the virtual camera in the second virtual space may be controlled according to the position or imaging direction of the imaging device.

[0012] According to the above, when the virtual camera approaches a virtual object, it is determined that the virtual camera has entered the second virtual space. When the virtual camera has entered the second virtual space, the position or imaging direction of the virtual camera can be changed in the second virtual space in accordance with a change in the position or imaging direction of the imaging device. This allows the user to move the imaging device closer to the virtual object, enter the second virtual space, and look around the second virtual space.

[0013] Furthermore, when the imaging direction of the imaging device changes in an up-down direction in the first control state, the first control means may move the virtual object in an up-down direction while maintaining a vertical orientation of the virtual object in the virtual space.

[0014] According to the above, even if the imaging direction of the imaging device changes in the vertical direction in the first control state, the vertical orientation of the virtual object can be maintained, and for example, the virtual object can be moved in the vertical direction while the upward direction of the virtual object is aligned with the upward direction of the virtual space.

[0015] The information processing system may further include a state change unit that, when the virtual object is pseudo-fixed by the second control unit, changes a state of the virtual object from a first state to a second state based on a user operation. When the virtual object is changed to the second state, the second virtual space may be visible from the virtual camera located in the virtual space.

[0016] Based on the above, in a state in which the virtual object is pseudo-fixed, the virtual object can be changed to the second state, and the second virtual space can be made visible in the second state.

[0017] Furthermore, the first control means may control at least one of the imaging direction of the virtual camera and the position of the virtual object in accordance with a change in the left-right imaging direction of the imaging device so that the imaging direction of the virtual camera and the virtual object direction are linked with the first degree of linkage, even when the imaging direction of the imaging device is directed above the first threshold value or below the second threshold value.

[0018] According to the above, even if the imaging direction of the imaging device is facing too far upward or too far downward, by turning the imaging direction of the imaging device left or right, it is possible to, for example, turn the virtual camera left or right, and link the imaging device and the virtual object, thereby improving usability.

[0019] In addition, in the first control state, when the imaging device moves, the first control means may control the position of the virtual camera and the position of the virtual object so that a constant distance is maintained between the virtual camera and the virtual object.

[0020] Based on the above, in the first control state, even when the imaging device moves, the distance between the virtual camera and the virtual object can be kept constant, and the imaging device and the virtual object can be linked.

[0021] In addition, in the first control state, the first control means may set an imaging direction of the virtual camera in accordance with an imaging direction of the imaging device, and control the position of the virtual object so as to follow changes in the imaging direction of the virtual camera.

[0022] Based on the above, when the imaging direction of the imaging device (virtual camera) changes, the position of the virtual object can be controlled in accordance with the change in the imaging direction, and the imaging device and the virtual object can be linked.

[0023] The information processing system may further include a display mode control means for changing a display mode of the virtual object when the virtual object is pseudo-fixed in the real space.

[0024] Based on the above, it is possible to make the user more easily aware that the virtual object has been pseudo-fixed in real space.

[0025] The first control means may also place the virtual object at a predetermined position in the imaging direction of the virtual camera without detecting a predetermined plane in the real space based on an actual image captured by the imaging device.

[0026] Based on the above, it is possible to place a virtual object in a virtual space without detecting a predetermined surface in a real space.

[0027] Another example of the present invention is an information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured of a real space by an imaging device and virtual images sequentially captured of a virtual object by a virtual camera disposed in a virtual space, the information processing system including: a virtual camera direction control means; and an object position control means. The virtual camera direction control means controls the imaging direction of the virtual camera according to the imaging direction of the imaging device. The object position control means controls the position of the virtual object in a manner that follows the imaging direction of the virtual camera. When the imaging direction of the imaging device is directed upwardly relative to a first threshold value or downwardly relative to a second threshold value in the real space, the object position control means controls the position of the virtual object so that the degree to which the virtual object follows the imaging direction of the virtual camera decreases. Another example of the present invention is an information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured of a real space by an imaging device and virtual images sequentially captured of a virtual object by a virtual camera disposed in a virtual space, the information processing system including: a virtual camera direction control means; an object position control means; an object fixing means; and a virtual camera position control means. The virtual camera direction control means controls the imaging direction of the virtual camera according to the imaging direction of the imaging device. The object position control means controls the position of the virtual object so that the virtual object follows the imaging direction of the virtual camera. The object fixing means controls the position of the virtual object based on a user operation so that the position of the virtual object is pseudo-fixed to a position in the real space. The virtual camera position control means controls the position of the virtual camera according to the position of the imaging device while the position of the virtual object is fixed. When the imaging direction of the imaging device is directed upwardly relative to a first threshold value or downwardly relative to a second threshold value in the real space, the object position control means controls the position of the virtual object so that the degree to which the virtual object follows the imaging direction of the virtual camera decreases.

[0028] According to the above, the imaging direction of the virtual camera can be set in response to a change in the imaging direction of the imaging device, and the virtual object can be virtually fixed in real space in response to a user operation. When the imaging direction of the imaging device is facing upward more than a first threshold value or downward more than a second threshold value, the degree to which the virtual object follows the imaging direction of the virtual camera can be reduced. This makes it possible to prevent the virtual object from being placed in an inappropriate position.

[0029] Another invention may be an information processing program executed on one or more computers included in the information processing system, an information processing device, or an information processing method performed in the information processing system. [Effects of the Invention]

[0030] According to the present invention, it is possible to place a virtual object at a desired position and prevent the virtual object from being placed at an inappropriate position. [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 1 is a diagram showing an example of the internal configuration of a smartphone 1 according to an embodiment of the present invention. [Figure 2] A diagram showing an example of real space [Figure 3] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is placed at a position P1. [Figure 4] FIG. 1 is a diagram showing an example of the positional relationship between various objects in real space and virtual space. [Figure 5] FIG. 10 is a diagram showing an example of a change in the virtual space when the imaging direction of the camera 13 is changed to face upward. [Figure 6] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 when the angle θ of the imaging direction of the camera 13 is “0”; [Figure 7] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 when the angle θ of the imaging direction of the camera 13 is a first threshold value “a.” [Figure 8] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 when the angle θ of the imaging direction of the camera 13 is greater than a first threshold value “a.” [Figure 9] FIG. 10 is a diagram showing an example of a change in the virtual space when the imaging direction of the camera 13 is changed downward. [Figure 10] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 when −b<θ<0. [Figure 11] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 when θ=−b. [Figure 12] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 when θ<−b. [Figure 13] FIG. 10 is a diagram showing the virtual space as viewed from above, illustrating an example of the operation of the virtual camera VC and the virtual door VD when the imaging direction of the camera 13 is changed to the right. [Figure 14]FIG. 11 shows an example of an image displayed on the display device 15 when the smartphone 1 is rotated rightward by c degrees from the state of FIG. 10. [Figure 15] FIG. 13 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is rotated rightward by c degrees from the state of FIG. 12. [Figure 16] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 after the virtual door VD is pseudo-fixed in real space. [Figure 17] FIG. 17 shows an example of an image displayed when the camera 13 is translated leftward from the state of FIG. 16. [Figure 18] FIG. 10 is a diagram showing an example of a portion of the second virtual space seen through a virtual door VD. [Figure 19] FIG. 10 is a diagram showing an example of the positional relationship between a virtual door VD, a virtual camera VC, and a second virtual space VS2. [Figure 20] FIG. 20 is a diagram showing an example of an image displayed when a virtual camera VC is placed at a position P2 shown in FIG. 19. [Figure 21] A side view of the real space when the virtual door VD is moved in conjunction with the imaging direction of the virtual camera VC without any restrictions. [Figure 22] FIG. 1 is a diagram showing an example of data stored in a memory 11 of a smartphone 1. [Figure 23] 1 is a flowchart showing an example of a main process executed by the processor 10 of the smartphone 1. [Figure 24] A flowchart showing details of the virtual door fixing process in step S105. DETAILED DESCRIPTION OF THE INVENTION

[0032] The information processing system of this embodiment is a system that allows a user to view a composite image obtained by combining an image of a virtual space with an image of a real space, and is a system that provides the user with augmented reality (AR). The information processing system may be, for example, a smartphone, a tablet terminal, a portable game device, a portable personal computer, or the like. Furthermore, the information processing system is not limited to a portable device, and may also be a stationary device. In this embodiment, the information processing system is assumed to be a smartphone.

[0033] Fig. 1 is a diagram showing an example of the internal configuration of a smartphone 1 according to this embodiment. As shown in Fig. 1, the smartphone 1 includes, for example, a processor 10, a memory 11, an input device 12, a camera 13, an attitude detection sensor 14, and a display device 15. The smartphone 1 also includes a storage device (for example, a non-volatile memory) not shown.

[0034] The processor 10, in cooperation with the memory 11, processes input information from the input device 12, processes real images from the camera 13, and processes data from the attitude detection sensor 14 to calculate the attitude of the smartphone 1 (camera 13). The processor 10 also performs processing related to various objects in the virtual space (described later), generates virtual images based on the virtual camera, and generates composite images by combining the real images and the virtual images. The processor 10 outputs the generated composite images to the display device 15. The processor 10 may also include a GPU (Graphics Processing Unit) for performing image processing.

[0035] The input device 12 receives input from a user. The input device 12 may be, for example, a touch panel or buttons.

[0036] The camera 13 (imaging device) is provided on the rear surface of the smartphone 1, and captures an image of the real space in the rear direction.

[0037] The attitude detection sensor 14 is a sensor for detecting the attitude of the smartphone 1 (camera 13), and is, for example, an acceleration sensor and / or an angular velocity sensor.

[0038] Display device 15 is provided on the front surface of smartphone 1 and displays images from camera 13 and images of the virtual space. Display device 15 may be any display device, such as a liquid crystal display or an organic EL display.

[0039] Next, a description will be given of an image displayed on the display device 15 of the smartphone 1. Fig. 2 is a diagram showing an example of real space. Fig. 3 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is placed at position P1.

[0040] As shown in Fig. 2, the real space includes, for example, a floor RF, a cylindrical real object RO placed on the floor RF, and a wall RW. When the smartphone 1 is placed at a position P1 in the real space shown in Fig. 2, an image as shown in Fig. 3 is displayed on the display device 15.

[0041] An image of real space (real image) captured by the camera 13 is displayed on the display device 15 of the smartphone 1. For example, an image of a floor RF and an image of a real object RO are displayed on the display device 15 as real images. A virtual door VD is also displayed on the display device 15. The virtual door VD is an example of a virtual object placed in the virtual space and does not exist in the real space. A virtual space is defined inside the smartphone 1, and the virtual door VD is placed in the virtual space. A virtual camera is placed in the virtual space, and a virtual image is generated by capturing an image of the virtual door VD using the virtual camera. The generated virtual image is superimposed on the real image from the camera 13 to generate a composite image, and the composite image is output to the display device 15. As a result, a composite image such as that shown in FIG. 3 is displayed.

[0042] Fig. 4 is a diagram showing an example of the positional relationship between various objects in the real space and the virtual space, showing a side view of the real space and the virtual space when the image shown in Fig. 3 is displayed.

[0043] As shown in Fig. 4, a fixed XYZ coordinate system is defined on the smartphone 1. The Z axis is an axis toward the rear of the smartphone 1 and points in the imaging direction of the camera 13. The Y axis is an axis perpendicular to the Z axis and points upward relative to the smartphone 1 (camera 13). The X axis is an axis perpendicular to the Z axis and Y axis and points leftward relative to the smartphone 1 (camera 13).

[0044] The imaging direction of the camera 13 (attitude of the smartphone 1) is calculated based on data from the attitude detection sensor 14. Because the camera 13 is fixed to the smartphone 1, the attitude calculated based on the data from the attitude detection sensor 14 is the attitude of both the smartphone 1 and the camera 13. For example, the smartphone 1 calculates the angle (θ) of the Z axis with respect to the horizontal plane (floor RF) based on gravity G detected by the attitude detection sensor 14.

[0045] A virtual camera VC is placed within the virtual space VS. An xyz coordinate system is set for the virtual camera VC. The z-axis of the virtual camera VC is an axis pointing in the imaging direction of the virtual camera VC. The y-axis of the virtual camera VC is an axis perpendicular to the z-axis and points upward relative to the virtual camera VC. The x-axis of the virtual camera VC is an axis perpendicular to the z-axis and y-axis and points leftward relative to the virtual camera VC.

[0046] The orientation of the camera 13 in the real space and the orientation of the virtual camera VC in the virtual space VS are linked. Specifically, the orientation of the virtual camera VC is set so as to match the orientation of the smartphone 1 calculated based on data from an orientation detection sensor 14 (e.g., an acceleration sensor and / or an angular velocity sensor). For example, when the camera 13 in the real space changes from a horizontal state to a state facing upward by 10 degrees, the virtual camera VC also changes from a horizontal state to a state facing upward by 10 degrees in the virtual space VS.

[0047] The position of the virtual door VD is controlled in a manner that follows the imaging direction of the virtual camera VC. Specifically, as shown in FIG. 4, the virtual door VD is placed in the imaging direction (z-axis direction) of the virtual camera VC. The virtual door VD is a virtual object that resembles a door, and is a plate-shaped virtual object. The virtual door VD is placed at a position a certain distance D away from the position of the virtual camera VC. For example, the center of the virtual door VD may be set at a position D away from the virtual camera VC in the z-axis direction.

[0048] A composite image obtained by combining a real image of the real space captured by the camera 13 and a virtual image of the virtual door VD captured by the virtual camera VC is displayed on the display device 15 of the smartphone 1. Therefore, when a user looks at the composite image, it appears as if the virtual door VD exists in the real space.

[0049] The virtual door VD is set with an xd axis, a yd axis, and a zd axis. The xd axis is an axis perpendicular to the virtual door VD, the yd axis is an axis pointing upward relative to the virtual door VD, and the xd axis is an axis pointing left relative to the virtual door VD. The virtual door VD is placed in the virtual space so that it is perpendicular to the horizontal plane of the virtual space VS (the horizontal plane of the real space). In other words, the virtual door VD is placed in the virtual space so that the upward yd axis of the virtual door VD and the upward axis of the virtual space are oriented in the same direction.

[0050] Although details will be described later, the virtual door VD is a virtual object that serves as an entrance to a virtual room (second virtual space).

[0051] Next, an image displayed on the display device 15 when the attitude of the camera 13 (smartphone 1) changes from the state shown in FIG. 3 will be described.

[0052] FIG. 5 is a diagram showing an example of a change in the virtual space when the imaging direction of the camera 13 is changed to face upward.

[0053] As shown in Fig. 5, when the angle θ between the z axis of the virtual camera VC (camera 13) and the horizontal plane is "0" degrees, the center of the virtual door VD is positioned at a position a certain distance D away from the z axis of the virtual camera VC (camera 13). In this case, the virtual door VD appears to exist, for example, at a position of height H1 from the horizontal plane (floor RF). Note that, hereinafter, when the camera 13 (virtual camera VC) is facing upward, the angle θ is assumed to be a positive value, and when the camera 13 (virtual camera VC) is facing downward, the angle θ is assumed to be a negative value.

[0054] When the imaging direction of the camera 13 is changed upward so that the angle θ ranges from 0 to the first threshold value a (a is a positive value), the position of the virtual door VD changes following the change in the imaging direction of the camera 13. That is, the imaging direction of the virtual camera VC changes according to the imaging direction of the camera 13 so that the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, are interlocked. Accordingly, the position of the virtual door VD also changes. For this reason, to the user, it seems that the camera 13 and the virtual door VD are interlocked, and it seems that the virtual door VD moves following the movement of the camera 13. Specifically, the virtual door VD moves upward in the virtual space while maintaining a state perpendicular to the horizontal plane. That is, the virtual door VD moves upward while maintaining a state where the yd axis in the upward direction of the virtual door VD is parallel to the axis in the upward direction of the virtual space. For example, from the state where θ = 0 to the state where θ = a, the virtual door VD is arranged in the virtual space so that the center of the virtual door VD is located at a position separated from the z-axis of the virtual camera VC by a fixed distance D. Also, during this period, the yd axis in the upward direction of the virtual door VD is controlled to face the direction opposite to the direction of gravity (the upward direction of the virtual space). When θ = a, when looking at the display device 15, the virtual door VD seems to exist, for example, at a position at a height H2 (> H1) from the horizontal plane (floor RF).

[0055] When the imaging direction of the camera 13 is further changed upward from the state where θ = a, the imaging direction of the virtual camera VC changes so as to coincide with the imaging direction of the camera 13. However, the position of the virtual door VD does not change following the change in the imaging direction of the camera 13. That is, in the state where θ > a, the virtual door VD does not move further upward, and the upward movement of the virtual door VD is restricted.

[0056] Thus, on the premise that the imaging direction of the virtual camera VC changes according to the change in the imaging direction of the camera 13, in the range of 0 < θ < a, the virtual door VD moves upward in conjunction with the change in the imaging direction of the virtual camera VC. On the other hand, when θ > a, that is, when the imaging direction of the virtual camera VC faces upward from the first threshold value a, the upward movement of the virtual door VD is restricted.

[0057] When the camera 13 (smartphone 1) moves in parallel in the up / down, left / right, or front / back directions, the relative positional relationship between the virtual camera VC and the virtual door VD does not change. For example, when the camera 13 moves in parallel in the left / right direction while keeping the imaging direction fixed, the virtual camera VC also moves in parallel in the left / right direction in the virtual space, and the virtual door VD also moves in parallel in the left / right direction in the virtual space. The same is true when the camera 13 moves in parallel in the up / down direction while keeping the imaging direction fixed.

[0058] In addition, when the camera 13 moves in parallel in the up / down, left / right, or front / rear directions while keeping the imaging direction fixed, the virtual camera VC and the virtual door VD do not need to move in the virtual space. Even in this case, the relative positional relationship between the virtual camera VC and the virtual door VD does not change, so the virtual door VD remains displayed in approximately the center of the display device 15, and it appears that the virtual door VD moves in accordance with the movement of the camera 13.

[0059] Fig. 6 is a diagram showing an example of an image displayed on display device 15 when angle θ of the imaging direction of camera 13 is "0". Fig. 7 is a diagram showing an example of an image displayed on display device 15 when angle θ of the imaging direction of camera 13 is a first threshold value "a". Fig. 8 is a diagram showing an example of an image displayed on display device 15 when angle θ of the imaging direction of camera 13 is greater than first threshold value "a".

[0060] In Fig. 6, the imaging direction of the camera 13 is more upward than in Fig. 3, so the image of the real object RO is displayed lower on the screen than in Fig. 3. Also, an image of the wall RW is not displayed in Fig. 3, but an image of the wall RW is displayed in Fig. 6. Also, as shown in Fig. 6, a virtual door VD is positioned in front of the virtual camera VC, and the virtual door VD is displayed approximately in the center of the display device 15.

[0061] As shown in FIG. 7, even when θ=a, the virtual door VD is positioned in front of the virtual camera VC, and the virtual door VD is displayed approximately in the center of the display device 15. While θ changes from 0 to a, the virtual door VD moves upward following the change in the imaging direction of the virtual camera VC (camera 13). In other words, the virtual door VD viewed from the composite image displayed on the display device 15 moves upward in conjunction with the change in the imaging direction of the camera 13. On the other hand, as shown in FIG. 7, the image of the real object RO is displayed further below than in FIG. 6. In addition, the area of ​​the image of the wall RW is also larger than in FIG. 6.

[0062] Furthermore, as shown in FIG. 8, when θ>a, that is, when the camera 13 is facing upward more than the first threshold, the virtual door VD is displayed on the lower side of the display screen of the display device 15 without following changes in the imaging direction of the camera 13 (virtual camera VC). In FIG. 8, the lower part of the virtual door VD is out of the imaging range of the virtual camera VC, and this lower part is not displayed on the display device 15. If the camera 13 is pointed further upward than in FIG. 8, the virtual door VD will move out of the imaging range of the virtual camera VC and will no longer be displayed on the screen. In contrast, as shown in FIG. 8, the real image moves further downward than in FIG. 7. The images of the real object RO and floor RF are displayed further downward than in FIG. 7, and the area of ​​the image of the wall RW will be larger than in FIG. 7.

[0063] Next, a case where camera 13 is pointed downward will be described. Fig. 9 is a diagram showing an example of a change in the virtual space when the imaging direction of camera 13 is changed to face downward.

[0064] As shown in FIG. 9, when the angle θ is smaller than 0 (i.e., the camera 13 is facing downward) and larger than "-b (b is a positive value)" degrees, the virtual door VD is positioned so that its center is located at a certain distance D from the z-axis of the virtual camera VC (camera 13). In this case, the virtual door VD viewed from the composite image displayed on the display device 15 is positioned at a height H3 (

[0065] ​ When θ is changed to the second threshold value "-b" (that is, when the camera 13 is changed in angle downward by b), the virtual door VD moves downward following the change in the imaging direction of the camera 13. That is, the imaging direction of the virtual camera VC changes according to the imaging direction of the camera 13, and the position of the virtual door VD also changes so that the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, are interlocked. For this reason, to the user, it seems that the camera 13 and the virtual door VD are interlocked, and it seems that the virtual door VD is moving following the movement of the camera 13. Specifically, the virtual door VD moves downward in the virtual space while maintaining a state perpendicular to the horizontal plane. During this time, the upward yd-axis of the virtual door VD is controlled to face the direction opposite to the gravitational direction (the upward direction in the virtual space). In the state of θ = -b, the virtual door VD seems to exist, for example, at a position of height H4 (<H3) from the horizontal plane (floor RF).

[0066] When the imaging direction of the camera 13 is further changed downward from the state of θ = -b, the imaging direction of the virtual camera VC changes to coincide with the imaging direction of the camera 13. However, the position of the virtual door VD does not change following the change in the imaging direction of the camera 13. That is, in the state of θ < -b, the virtual door VD does not move further downward, and the downward movement of the virtual door VD is restricted.

[0067] As described above, in the range of -b < θ < 0, the virtual door VD moves downward in conjunction with the change in the imaging direction of the virtual camera VC. On the other hand, when θ < -b, that is, when the imaging direction of the virtual camera VC is facing downward more than the second threshold value b, the downward movement of the virtual door VD is restricted.

[0068] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 when -b < θ < 0. FIG. 11 is a diagram showing an example of an image displayed on the display device 15 when θ = -b. FIG. 12 is a diagram showing an example of an image displayed on the display device 15 when θ < -b.

[0069] In Fig. 10, the imaging direction of the camera 13 is more downward than in Fig. 6, so the image of the real object RO is displayed higher on the screen than in Fig. 6, and part of the real object RO is not displayed. Also, the image of the wall RW is not displayed in Fig. 10. Also, as shown in Fig. 10, the virtual door VD is positioned directly in front of the virtual camera VC, and the virtual door VD is displayed approximately in the center of the display device 15.

[0070] As shown in Fig. 11, when θ = -b, the virtual door VD is located in front of the virtual camera VC, and the virtual door VD is displayed approximately in the center of the display device 15. While θ changes from 0 to -b, the virtual door VD moves downward following the change in the imaging direction of the virtual camera VC (camera 13). In other words, the virtual door VD viewed from the composite image displayed on the display device 15 moves downward in conjunction with the change in the imaging direction of the camera 13. Meanwhile, the image of the real object RO moves further upward than in Fig. 10 and is almost no longer displayed.

[0071] 12, when θ<-b, that is, when the camera 13 is facing downward more than the second threshold, the virtual door VD is displayed on the upper side of the display screen of the display device 15 without following the change in the imaging direction of the camera 13 (virtual camera VC). In FIG. 12, the upper part of the virtual door VD is outside the imaging range of the virtual camera VC, and this upper part is not displayed on the display device 15. If the camera 13 is facing further downward than in FIG. 12, the virtual door VD will move outside the imaging range of the virtual camera VC and will no longer be displayed on the screen. In contrast, as shown in FIG. 12, the real image moves further downward than in FIG. 11.

[0072] Next, a case where the imaging direction of camera 13 is changed to the left or right will be described. Fig. 13 is a diagram showing the virtual space as viewed from above, illustrating an example of the operation of virtual camera VC and virtual door VD when the imaging direction of camera 13 is changed to the right.

[0073] When the imaging direction of the camera 13 changes left or right, the imaging direction of the virtual camera VC also changes left or right. For example, when the rotation angle φ of the camera 13 (smartphone 1) in the right direction is 10 degrees, the virtual camera VC also rotates right by 10 degrees. As shown in FIG. 13 , the virtual door VD also moves left or right in conjunction with the left or right rotation of the imaging direction of the virtual camera VC. Specifically, the virtual door VD moves left or right while changing its orientation so as to face the virtual camera VC. For example, when the virtual camera VC rotates rightward by a predetermined angle (when the camera 13 rotates rightward by a predetermined angle), the virtual door VD moves rightward while maintaining the distance D from the virtual camera VC. Even when the virtual door VD moves left or right in response to a change in the imaging direction of the camera 13 left or right, the virtual door VD maintains its orientation perpendicular to the horizontal plane. That is, the virtual door VD moves left or right in the virtual space while maintaining the upward yd axis of the virtual door VD parallel to the upward axis of the virtual space.

[0074] In addition, when the imaging direction of the camera 13 changes left and right, the virtual door VD moves left and right in conjunction with the left and right movement of the imaging direction of the virtual camera VC, regardless of the value of the left and right rotation angle φ.

[0075] Fig. 14 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is rotated rightward by c degrees from the state of Fig. 10. Fig. 15 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is rotated rightward by c degrees from the state of Fig. 12.

[0076] 14, when the smartphone 1 is turned to the right from the state shown in FIG. 10, the real object RO in the real image moves to the left side of the screen, and a part of the right wall RW in the real space is displayed on the screen. Meanwhile, the virtual door VD is positioned in front of the virtual camera VC and is displayed approximately in the center of the display device 15.

[0077] 15, when the smartphone 1 is turned to the right from the state shown in FIG. 12, an image of the right side of the real space is displayed on the screen. The virtual door VD is located in front of the virtual camera VC and is displayed approximately in the center of the display device 15. In this way, even when the camera 13 is facing downwards relative to the second threshold, the virtual door VD also moves left and right in response to the camera 13 being turned left and right. The same is true when the camera 13 is facing upwards relative to the first threshold. In other words, even when the imaging direction of the camera 13 is facing upwards relative to the first threshold or downwards relative to the second threshold, if the imaging direction of the camera 13 changes left and right, the position of the virtual door VD also changes left and right in response to the change in the imaging direction of the camera 13 (virtual camera VC).

[0078] (Fixing virtual door VD) Next, a process after the virtual door VD is pseudo-fixed in real space will be described. As described above, when the imaging direction of the camera 13 is changed while the virtual door VD is displayed on the display device 15, the virtual door VD also moves in the virtual space accordingly. Furthermore, when the imaging direction of the camera 13 is kept fixed and the camera 13 is translated, the virtual door VD also moves in accordance with the camera 13. As a result, the virtual door VD appears to be linked to the camera 13, and the virtual door VD appears to be moving in real space. In this state, when a predetermined user operation is performed on the input device 12, the virtual door VD is pseudo-fixed in real space.

[0079] 16 is a diagram showing an example of an image displayed on the display device 15 after the virtual door VD is virtually fixed in real space. As shown in FIG. 16, when a predetermined user operation (e.g., a touch input on the virtual door VD) is performed on the input device 12, the virtual door VD is virtually fixed and the display mode of the virtual door VD changes. Before the predetermined user operation, the virtual door VD is displayed in a first display mode (e.g., the mode shown by the dashed line in FIG. 3), and in this state, the virtual door VD moves according to a change in the imaging direction of the camera 13. Note that the predetermined user operation is not limited to a touch input, and may be a button input, a non-touch input (e.g., a gesture input), a voice input, or the like.

[0080] After a predetermined user operation, the virtual door VD is displayed in a second display mode (for example, the solid line mode as shown in FIG. 16). In this state, the virtual door VD is pseudo-fixed in real space. When the virtual door VD is pseudo-fixed, the position of the virtual door VD does not change even if the imaging direction of the camera 13 changes. Furthermore, when the camera 13 is translated, the virtual door VD does not move to follow the camera 13.

[0081] FIG. 17 is a diagram showing an example of an image displayed when the camera 13 is translated leftward from the state shown in FIG. 16. As shown in FIG. 17, when the virtual door VD is pseudo-fixed in real space, the virtual door VD does not translate leftward even when the camera 13 translates leftward. Specifically, in FIG. 17, the real object RO moves rightward on the screen in response to the translation of the camera 13 leftward, and a part of the real object RO is displayed at the right edge. As the real object RO moves within the screen, the virtual door VD also moves rightward on the screen. The amount of movement of the real object RO and the amount of movement of the virtual door VD within the screen are the same. This makes the virtual door VD appear pseudo-fixed in real space.

[0082] Specifically, a fixed vx-vy-vz Cartesian coordinate system is set in the virtual space, and when a predetermined user operation is performed, the vx-vy-vz coordinate values ​​of the virtual door VD are fixed. Meanwhile, the virtual camera VC moves in the virtual space in accordance with the movement of the camera 13. The imaging direction of the virtual camera VC is set to match the imaging direction of the camera 13. While the virtual camera VC moves in the virtual space and changes its imaging direction following the camera 13, the virtual door VD is fixed in the virtual space. Therefore, the virtual door VD displayed on the display device 15 appears to be pseudo-fixed in the real space.

[0083] The position of the virtual camera VC is calculated based on real images from the camera 13 acquired at predetermined time intervals (for example, 1 / 60 second intervals). The movement (movement) of the camera 13 is calculated based on the real images acquired successively, and the position of the virtual camera VC is set so as to match the movement (movement) of the camera 13. Note that the movement (movement) of the camera 13 may be calculated based on the output from the attitude detection sensor 14 in addition to (or instead of) the real images from the camera 13. For example, the amount and direction of movement of the camera 13 may be calculated based on the acceleration output from the attitude detection sensor 14. Then, the virtual camera VC may be moved within the virtual space based on the calculated amount and direction of movement. When the virtual door VD is fixed, the relative positional relationship between the virtual camera VC and the virtual door VD changes by moving the camera 13 (virtual camera VC).

[0084] The vx-vy-vz coordinate system fixed in the virtual space may be set based on the virtual door VD when the virtual door VD is fixed. For example, the origin of the vx-vy-vz coordinate system may be set to the position of the virtual door VD, the vy axis may be set in the direction opposite to the direction of gravity, and the vx axis may be set parallel to the xd axis of the virtual door VD. The vx-vy-vz coordinate system may also be fixed in the virtual space before the virtual door VD is fixed. For example, when a real image is acquired from the camera 13, a plane in the real space may be detected based on the real image and the direction of gravity, and the origin of the vx-vy-vz coordinate system may be set on the plane. The vy axis may be set in the direction opposite to the direction of gravity, and the detected plane may be set as the vx-vz plane. Alternatively, a marker may be placed in the real space, and the vx-vy-vz coordinate system may be set based on a marker image included in the real image from the camera 13.

[0085] (Second virtual space) As described above, when the virtual door VD is fixed in a pseudo manner in the real space (when the virtual door VD is fixed in the virtual space), a second virtual space is generated based on the virtual door VD.

[0086] FIG. 18 is a diagram showing an example of a portion of the second virtual space seen through the virtual door VD. After the virtual door VD is fixed, for example, in response to a user's input to the input device 12, the virtual door VD changes from a closed state (first state) shown in FIG. 16 to an open state (second state) shown in FIG. 18. When the virtual door VD changes to the open state, the inside of the second virtual space VS2 becomes visible. As shown in FIG. 18, an image is displayed in which a real image of the real space, an image of the virtual door VD, and an image of the second virtual space VS2 are synthesized. Specifically, as shown in FIG. 18, an image of the real space is displayed in the area other than the virtual door VD, and images of the real object RO and floor RF are displayed. An image of the inside of a virtual room representing the second virtual space VS2 is displayed at the opening of the virtual door VD. In the second virtual space VS2, a virtual table VT and a virtual character C1 are placed on a virtual floor VF.

[0087] 19 is a diagram showing an example of the positional relationship between the virtual door VD, the virtual camera VC, and the second virtual space VS2. In FIG. 19, a diagram showing the entire virtual space including the second virtual space VS2 as seen from above is shown.

[0088] As shown in Fig. 19, when the virtual camera VC is located at position P1 and the virtual door VD is open, the image shown in Fig. 18 is displayed. In this state, the virtual camera VC is outside the second virtual space VS2 and cannot move freely within the second virtual space VS2.

[0089] The second virtual space VS2 is set according to the position of the virtual door VD. The second virtual space VS2 is a virtual room surrounded by a virtual floor VF, four virtual walls VW, and a virtual ceiling. Outside the second virtual space VS2 is a virtual space VS, and the virtual door VD is located between the inside and outside of the second virtual space VS2. The second virtual space VS2 corresponds to a predetermined range in real space. For example, the second virtual space VS2 corresponds to an area of ​​several tens of centimeters to several meters square in real space, giving the user the sensation that the second virtual space VS2 exists within that area.

[0090] A virtual table VT, a virtual character C1, and a virtual character C2 are placed in the second virtual space VS2. In addition to these, various other objects (e.g., furniture objects) may be placed in the second virtual space VS2. The virtual floor VF of the second virtual space VS2 is set perpendicular to the virtual door VD. That is, the virtual floor VF and the yd axis of the virtual door VD intersect perpendicularly.

[0091] As shown in FIG. 16 or 18, when the camera 13 is moved in the imaging direction from a state in which the virtual camera VC and the virtual door VD are a predetermined distance apart, the virtual camera VC also moves in the z-axis direction, and the virtual camera VC approaches the virtual door VD. When the virtual camera VC and the virtual door VD satisfy a predetermined positional condition (when the virtual camera VC approaches the virtual door VD within a predetermined distance), the virtual camera VC can enter the second virtual space VS2. For example, when the distance between the virtual camera VC and the virtual door VD is less than a predetermined value, the left-right position of the virtual camera VC is within the width of the virtual door VD, and the up-down position of the virtual camera VC is within the height of the virtual door VD, the predetermined positional condition between the virtual camera VC and the virtual door VD is satisfied. In other words, when the virtual camera VC (camera 13) approaches the position of the virtual door VD and the virtual camera VC is not displaced from the position of the virtual door VD in either the up-down or left-right directions, the predetermined positional condition is satisfied.

[0092] Note that even if the virtual camera VC approaches the second virtual space VS2 from a position other than the virtual door VD, it cannot enter the second virtual space VS2. That is, if the camera 13 is moved within a predetermined range in real space through the position of the virtual door VD, the virtual camera VC enters the second virtual space VS2, and an image within the second virtual space VS2 is displayed. However, if the camera 13 is moved within the predetermined range in real space through a position other than the virtual door VD, the virtual camera VC does not enter the second virtual space VS2. In this case, a real image from the camera 13 is displayed.

[0093] When the virtual camera VC reaches the position of the virtual door VD and further moves in the imaging direction, the virtual camera VC (camera 13) moves to position P2 in the second virtual space VS2. Fig. 20 is a diagram showing an example of an image displayed when the virtual camera VC is placed at position P2 shown in Fig. 19.

[0094] As shown in FIG. 20, when the virtual camera VC enters the second virtual space VS2, the real image from the camera 13 is not displayed, and the virtual image of the second virtual space VS2 based on the virtual camera VC is displayed on the entire screen. In the example shown in FIG. 20, a virtual table VT and a virtual character C1 are displayed. If the camera 13 is further moved in this state, the virtual camera VC moves within the second virtual space VS2. The movement of the camera 13 is detected based on the real images successively acquired from the camera 13. That is, when the virtual camera VC enters the second virtual space VS2, the real image is not displayed on the screen, but the real images are successively acquired from the camera 13, and the movement of the camera 13 (smartphone 1) is detected based on the acquired real images. Furthermore, if the imaging direction of the camera 13 is changed, the imaging direction of the virtual camera VC also changes within the second virtual space VS2 in response to the change.

[0095] As described above, in this embodiment, the virtual camera VC is operated in the virtual space in accordance with the movement of the camera 13, and the virtual door VD in the virtual space is displayed superimposed on the real image from the camera 13, thereby making it possible to display the virtual door VD as if it were in real space. By performing a predetermined user operation, the user can fix the virtual door VD in the virtual space and pseudo-fix the virtual door VD in real space. In addition, the user can enter the second virtual space VS2 using the virtual door VD as an entrance and view the second virtual space VS2.

[0096] Before a specified user operation is performed, the virtual door VD can be moved within the virtual space by moving the camera 13 or changing the imaging direction of the camera 13, allowing the user to place the entrance to the second virtual space VS2 at the desired position in the real space.

[0097] Here, for example, if the virtual door VD is always placed at a predetermined position in the imaging direction of the camera 13 (virtual camera VC), if the camera 13 is pointed too far upward, the virtual door VD will be located at a high position in the real space.

[0098] FIG. 21 is a diagram showing a real space viewed from the side when the virtual door VD is moved in conjunction with the imaging direction of the virtual camera VC without any restrictions.

[0099] 21, when θ=0, virtual door VD is placed at a predetermined position in the imaging direction of virtual camera VC, and when the user looks at display device 15, the bottom of virtual door VD appears to be located at height H1 in real space. When virtual door VD is fixed at that position, second virtual space VS2 is set so that the virtual floor VF of second virtual space VS2 is located at height H1. When the user brings camera 13 close to virtual door VD fixed in real space in a pseudo-like manner, the user can enter virtual camera VC into second virtual space VS2, and by moving camera 13, the user can look around inside second virtual space VS2.

[0100] On the other hand, when θ is greater than the first threshold a, if the upward movement of the virtual door VD is not restricted, the position of the bottom surface of the virtual door VD is at a height H5. In this case, the user is looking up at the real space. If the virtual door VD is fixed in this state, the virtual floor VF of the second virtual space VS2 is fixed at an upward position in the real space. If the virtual floor VF of the second virtual space VS2 is fixed at a position that the user cannot reach, it becomes difficult for the user to move the camera 13 (smartphone 1) close to the virtual door VD, making it difficult for the user to move the virtual camera VC into the second virtual space VS2. Even if the camera 13 can pass through the virtual door VD, it is difficult to move the camera 13 so that it overlooks the second virtual space VS2 from a high position.

[0101] Therefore, in this embodiment, when θ is greater than the first threshold value a, that is, when the imaging direction of the camera 13 is pointing upward more than the first threshold value a, the upward movement of the virtual door VD is restricted. This prevents the virtual door VD from being fixed at a position that is too high in the real space, and allows the virtual camera VC to enter the second virtual space VS2, making it easier to view the second virtual space VS2.

[0102] The same applies when the virtual door VD moves downward without restriction. If the camera 13 is pointing too far downward and the downward movement of the virtual door VD is not restricted, the bottom of the virtual door VD may be located below the floor of the real space. If the virtual door VD is fixed in this state, the virtual floor VF of the second virtual space VS2 will be fixed at a position below the floor of the real space. This makes it impossible for the user to move the camera 13 (smartphone 1) close to the position of the virtual door VD and therefore impossible to enter the second virtual space VS2.

[0103] Therefore, in this embodiment, when θ is smaller than the second threshold value −b, that is, when the imaging direction of the camera 13 is facing downward more than the second threshold value b, downward movement of the virtual door VD is restricted. This prevents the virtual door VD from being positioned lower than the floor of the real space, and allows entry into the second virtual space VS2.

[0104] (Data stored on the smartphone) Next, a description will be given of data stored in the smartphone 1. Fig. 22 is a diagram showing an example of data stored in the memory 11 of the smartphone 1.

[0105] 22, a program for performing information processing shown in a flowchart described below is stored in the smartphone 1. The smartphone 1 also stores real image data, posture data, virtual camera data, virtual image data, composite image data, virtual door data, virtual character data, and other object data.

[0106] The real image data is data representing a real image captured by the camera 13. The processor 10 acquires real images from the camera 13 at predetermined time intervals (for example, 1 / 60 second intervals) and stores the real image data representing the real images in the memory 11.

[0107] The attitude data is data that represents the attitude of the smartphone 1 in real space, and is data that represents the attitude of the camera 13. The attitude data is calculated based on the output (e.g., acceleration value and / or angular velocity value) from the attitude detection sensor 14. For example, the attitude data may include data (angle θ) related to the up-down rotation of the imaging direction of the camera 13 and data (angle φ) related to the left-right rotation.

[0108] The virtual camera data is data relating to the virtual camera VC, and includes data representing the attitude (imaging direction) of the virtual camera VC and data representing the position of the virtual camera VC in the virtual space.

[0109] The virtual image data is data representing a virtual image captured by the virtual camera VC. The virtual image is repeatedly generated at predetermined time intervals (for example, every 1 / 60 seconds).

[0110] The composite image data is data representing a composite image obtained by combining a real image and a virtual image. The composite image is generated by superimposing a virtual image on a real image. The composite image is repeatedly generated at predetermined time intervals (for example, every 1 / 60 seconds).

[0111] The virtual door data is data related to the virtual door VD, and includes data representing the posture and position of the virtual door VD.

[0112] The virtual character data is data related to virtual characters C1 and C2 placed in the second virtual space VS2. The virtual character data includes data representing the position and posture of each virtual character. The other object data is data related to other virtual objects (e.g., a virtual table VT) placed in the second virtual space VS2, and includes data representing the position and posture of the object.

[0113] (Details of processing on smartphones) Next, a detailed description will be given of the processing performed by the smartphone 1. Fig. 23 is a flowchart showing an example of main processing executed by the processor 10 of the smartphone 1. Note that the main processing shown in Fig. 23 is repeatedly executed at predetermined time intervals (for example, 1 / 60 second intervals).

[0114] 23, the processor 10 acquires a real image from the camera 13 (step S101). The acquired real image is stored in the memory 11 as real image data.

[0115] Next, the processor 10 acquires the attitude of the camera 13 (smartphone 1) (step S102). For example, the processor 10 calculates the attitude of the camera 13 (smartphone 1) based on the acceleration value and / or angular velocity value detected by the attitude detection sensor 14, and stores the calculated attitude in the memory 11 as attitude data.

[0116] Next, the processor 10 sets a virtual camera VC in the virtual space (step S103). Specifically, the processor 10 sets the attitude of the camera 13 acquired in step S102 as the attitude of the virtual camera VC. Note that the attitude of the virtual camera VC only needs to follow the attitude of the camera 13, and may be set to approach the attitude of the camera 13 over a predetermined time.

[0117] Next, the processor 10 determines whether the virtual door VD is fixed (step S104). In step S104, it is determined whether the virtual door VD is fixed in the virtual space (i.e., whether the virtual door VD is pseudo-fixed in the real space). If the processing of step S205 described later has already been performed, the determination in step S104 is YES.

[0118] If the virtual door VD is not fixed (step S104: NO), the processor 10 performs a virtual door fixing process (step S105). The process of step S105 is a process before the virtual door VD is fixed in the virtual space, and includes a process for fixing the virtual door VD. Details of the virtual door fixing process will be described below.

[0119] (Virtual door fixing process) 24 is a flowchart showing details of the virtual door locking process of step S105. As shown in FIG. 24, the processor 10 determines whether θ is greater than a first threshold value "a" or whether θ is less than a second threshold value "-b" (step S201). The process of step S201 is a process of determining whether the imaging direction of the camera 13 is facing upward more than the first threshold value "a" or whether the imaging direction of the camera 13 is facing downward more than the second threshold value "b."

[0120] If the determination in step S201 is NO, the processor 10 positions the virtual door VD so that it follows the camera 13 (step S202). For example, the processor 10 positions the virtual door VD at a position a certain distance D away from the position of the virtual camera VC in the imaging direction. As a result, when the camera 13 is pointed upward, the virtual door VD moves upward, and when the camera 13 is pointed downward, the virtual door VD moves downward. Furthermore, when the camera 13 is pointed left or right, the virtual door VD moves left or right. Furthermore, the processor 10 positions the virtual door VD so that the upward yd axis of the virtual door VD faces in the direction opposite to the direction of gravity. Note that in step S202, the virtual door VD may be positioned so that the virtual door VD moves over a predetermined time to a position a certain distance D away from the position of the virtual camera VC in the imaging direction.

[0121] On the other hand, if the determination in step S201 is YES, the processor 10 places the virtual door VD by restricting its position in the up-down direction (step S203). For example, if θ>a, that is, if the imaging direction of the camera 13 is pointing upward more than the first threshold value a, the processor 10 calculates, as the position of the virtual camera VC, a position that is a certain distance D away from the position of the virtual camera VC in the imaging direction assuming that θ=a. Also, if θ<-b, that is, if the imaging direction of the camera 13 is pointing downward more than the second threshold value b, the processor 10 calculates, as the position of the virtual camera VC, a position that is a certain distance D away from the position of the virtual camera VC in the imaging direction assuming that θ=-b. Note that in step S203, the virtual door VD may be placed so that the virtual door VD moves to the calculated position over a predetermined time.

[0122] When the process of step S202 or step S203 is executed, the processor 10 determines whether or not a predetermined user operation for fixing the virtual door VD has been performed (step S204).

[0123] When a predetermined user operation is performed (step S204: YES), the processor 10 pseudo-fixes the virtual door VD in the real space (step S205). The real space and the virtual space are associated with each other, and the virtual door VD is fixed in the virtual space, thereby pseudo-fixing the virtual door VD in the real space. For example, if a fixed vx-vy-vz coordinate system is set in the virtual space corresponding to the real space, the position of the virtual door VD is expressed as coordinate values ​​of the vx-vy-vz coordinate system. The virtual door VD is fixed in the virtual space by storing the coordinate values ​​of the virtual door VD at the time when the predetermined user operation is performed and preventing the coordinate values ​​from being updated thereafter. Alternatively, the position of the virtual door VD at the time when the predetermined user operation is performed may be set as the origin of the vx-vy-vz coordinate system. At that time, the coordinate values ​​of the vx-vy-vz coordinate system of the virtual camera VC are determined. Thereafter, if the camera 13 moves, the coordinate values ​​of the vx-vy-vz coordinate system of the virtual camera VC also change in accordance with the movement, but the origin of the vx-vy-vz coordinate system (i.e., the position of the virtual door VD) does not change.

[0124] Furthermore, in step S205, a second virtual space VS2 is set based on the position of the virtual door VD. Specifically, the virtual floor VF is set so that the height direction positions of the bottom surface of the virtual door VD and the virtual floor VF of the second virtual space VS2 coincide. Furthermore, the virtual floor VF is set so as to be perpendicular to the virtual door VD (so that the virtual floor VF is perpendicular to gravity). Furthermore, virtual characters C1 and C2 and a virtual table VT are placed within the second virtual space VS2. Note that even though the second virtual space VS2 is set in step S205, the second virtual space VS2 is set to be transparent. For this reason, the user cannot see the second virtual space VS2.

[0125] Next, the processor 10 changes the display mode of the virtual door VD from the first display mode to the second display mode (step S206). For example, the display mode of the virtual door VD may be changed from a dashed line mode to a solid line mode. Also, in step S206, the color of the virtual door VD may be changed, or the virtual door VD may be changed from a translucent mode to an opaque mode.

[0126] If the determination in step S204 is NO, or if the process of step S206 is performed, the processor 10 ends the process shown in FIG. 24 and returns to the process of FIG.

[0127] 23, if the virtual door VD is fixed (step S104: YES), the processor 10 calculates the position of the virtual camera VC (step S106). Specifically, the processor 10 calculates the position of the virtual camera VC in the virtual space based on the real image from the camera 13. The processor 10 calculates the movement direction and movement amount of the camera 13 (smartphone 1) based on the real image acquired from the camera 13, calculates the position of the virtual camera VC according to the calculated movement direction and movement amount, and stores the position as virtual camera data.

[0128] Next, the processor 10 performs a virtual door opening / closing process (step S107). Specifically, the processor 10 determines whether or not the user has performed an opening / closing operation on the virtual door VD based on an input from the input device 12. For example, the processor 10 determines whether or not the virtual door VD has been touched. For example, if the virtual door VD is in a closed state and an opening / closing operation on the virtual door VD is performed, the processor 10 changes the virtual door VD to an open state. Also, if the virtual door VD is in an open state and an opening / closing operation on the virtual door VD is performed, the processor 10 changes the virtual door VD to a closed state. If the virtual door VD is in an open state, the second virtual space VS2 corresponding to the opening of the virtual door VD becomes opaque, and the second virtual space VS2 can be seen through the opening.

[0129] Next, the processor 10 determines whether the position conditions for the virtual camera VC and the virtual door VD are satisfied. Specifically, if the distance in the virtual space between the virtual camera VC and the virtual door VD is less than a predetermined value, the horizontal position of the virtual camera VC is within the width range of the virtual door VD, and the vertical position of the virtual camera VC is within the height range of the virtual door VD, the processor 10 determines that the position conditions are satisfied.

[0130] If the position condition is satisfied (step S108: YES), the processor 10 performs processing for entering the second virtual space (step S109). Specifically, the processor 10 sets ON a flag indicating that the virtual camera VC exists in the second virtual space VS2. The virtual camera VC is placed in the second virtual space VS2. Various virtual objects in the second virtual space VS2 (such as the virtual floor VF, virtual wall VW, virtual ceiling, virtual characters C1 and C2, and virtual table VT) are set to be opaque. As a result, a virtual image of the second virtual space VS2 as viewed from the virtual camera VC is displayed on the display device 15 of the smartphone 1. The virtual door VD is a virtual object that connects the second virtual space VS2 with the external virtual space VS and is visible from within the second virtual space VS2. After the virtual camera VC enters the second virtual space VS2, the virtual camera VC remains in the second virtual space VS2 until the virtual camera VC passes through the virtual door VD and exits the second virtual space VS2. Furthermore, when the process of entering the second virtual space is performed, a virtual boundary is set between the inside and outside of the second virtual space VS2. The second virtual space VS2 corresponds to a predetermined range in the real space, but after the virtual camera VC enters the second virtual space VS2, even if the camera 13 exits the predetermined range in the real space from a position other than the virtual door VD, the virtual camera VC does not exit the second virtual space VS2 but remains at the virtual boundary. At this time, a virtual image (e.g., an entirely black image) capturing the virtual boundary is displayed on the display device 15. Before the virtual camera VC enters the second virtual space VS2, even if the camera 13 enters the predetermined range in the real space from a position other than the virtual door VD, a composite image combining the real image from the camera 13 and the virtual image from the virtual camera VC continues to be displayed. In other words, after it is determined that the virtual camera VC has entered the second virtual space VS2, a process different from that before it is determined that the virtual camera VC has entered the second virtual space VS2 is performed.

[0131] When the process of step S109 has been performed, or when the result of step S108 is NO, the processor 10 determines whether or not the virtual camera VC exists in the second virtual space (step S110).

[0132] If the virtual camera VC exists in the second virtual space (step S110: YES), the processor 10 performs second virtual space processing (step S111). The processing of step S111 is processing when the virtual camera VC exists in the second virtual space VS2, and is processing for controlling the virtual camera VC within the second virtual space VS2 based on the position of the virtual camera VC calculated in step S106. For example, when the virtual camera VC approaches the virtual door VD within the second virtual space VS2 (i.e., when the virtual camera VC and the virtual door VD satisfy the above-mentioned predetermined positional condition), an exit processing is performed to move the virtual camera VC out of the second virtual space VS2. In the exit processing, a flag indicating that the virtual camera VC exists in the second virtual space VS2 is set to OFF. When the exit processing is performed, the virtual camera VC leaves the second virtual space VS2. When the virtual camera VC leaves the second virtual space VS2, the virtual camera VC returns to the state it was in before entering the second virtual space VC2. As a result, a composite image obtained by combining the real image from camera 13 and the virtual image of virtual space VS viewed from virtual camera VC is displayed on display device 15 of smartphone 1. On the other hand, when virtual camera VC moves to the boundary of second virtual space VS2 other than virtual door VD (when camera 13 moves to the boundary of the above-mentioned predetermined range in real space), the movement of virtual camera VC is restricted. Specifically, even if camera 13 moves beyond the above-mentioned predetermined range in real space, virtual camera VC does not go outside the second virtual space VS2 but is kept at the virtual boundary set between the inside and outside of second virtual space VS2.

[0133] When the processing of step S111 is performed, when the determination in step S110 is NO, or when the processing of step S105 is performed, processor 10 performs image output processing (step S112). Specifically, when virtual camera VC is located outside second virtual space VS2 (i.e., virtual space VS), processor 10 generates a composite image by superimposing a virtual image captured by virtual camera VC on a real image captured by camera 13. Then, processor 10 outputs the generated composite image to display device 15. When virtual camera VC is located within second virtual space VS2, processor 10 generates a virtual image of second virtual space VS2 viewed from virtual camera VC and outputs the virtual image to display device 15.

[0134] After the virtual door VD is fixed in step S205, if the user performs a predetermined release operation, the virtual door VD may be released from the fixed state. In this case, the virtual door VD becomes movable again within the virtual space. The processor 10 then ends the processing shown in FIG. 23.

[0135] The processing shown in the above flowchart is merely an example, and the order and content of the processing, values ​​used for the determination, etc. may be changed as appropriate.

[0136] As described above, in this embodiment, the virtual camera VC moves in the virtual space and the imaging direction of the virtual camera VC changes in response to the movement of the camera 13 and the change in the imaging direction of the camera 13. The virtual door VD moves in the virtual space in response to the movement of the virtual camera VC and the change in the imaging direction, and the virtual door VD is fixed in response to a predetermined user operation.

[0137] Specifically, before the virtual door VD is fixed, when the imaging direction of the camera 13 is within a predetermined range in the vertical direction (when -b < θ < a), the virtual door VD moves in the vertical direction in conjunction with the imaging direction of the camera 13 (virtual camera VC). Also, when the camera 13 moves parallel in the front-back, left-right, and up-down directions, the virtual door VD also moves parallel in the same direction within the virtual space. Then, in response to a predetermined user operation, the virtual door VD is fixed within the virtual space. On the other hand, when the imaging direction of the camera 13 is facing upward beyond the first threshold value a or facing downward beyond the second threshold value b, the virtual door VD does not move in the vertical direction in conjunction with the imaging direction of the camera 13.

[0138] Thereby, the virtual door VD can be moved and pseudo-fixed at a desired position in the real space, and it can be prevented that the virtual door VD is fixed at a position that is too high or too low. The user can enter the second virtual space VS2 through the virtual door VD, and it can be made easier for the user to enter the second virtual space VS2.

[0139] Also, the bottom surface of the virtual door VD coincides with the virtual floor VF of the second virtual space VS2, and the second virtual space VS2 is formed continuously with the virtual door VD. That is, the virtual door VD is located at the boundary between the virtual space VS and the second virtual space VS2, and the height of the virtual door VD in the real space coincides with the height of the second virtual space VS2 in the real space. Thereby, the virtual door VD can be configured as an entrance to the second virtual space VS2, and after the user pseudo-fixes the virtual door VD in the real space, the user can enter the second virtual space VS2 from the virtual door VD without a sense of incongruity.

[0140] Also, by moving the camera 13, the virtual camera VC is brought closer to the virtual door VD, and when the virtual camera VC reaches the position of the virtual door VD, the virtual camera VC enters the second virtual space VS2. Thereby, it is possible to enter the second virtual space VS2 through the virtual door VD, and it is possible to give the user a feeling of entering the second virtual space VS2 from the virtual door VD existing in the real space.

[0141] In addition, a virtual door VD is placed in the virtual space as a three-dimensional virtual object, and the virtual door VD is moved within the virtual space while maintaining a vertical orientation relative to the horizontal plane of the virtual space. That is, the virtual door VD moves within the virtual space while maintaining a vertically standing position. This allows the virtual floor VF to be horizontal without the second virtual space VS2 becoming tilted when the virtual door VD is fixed.

[0142] In this embodiment, the virtual door VD is placed at a predetermined position in the imaging direction of the virtual camera VC. This allows the virtual door VD to be placed in the virtual space without detecting a predetermined surface in the real space (a horizontal surface such as a floor or table surface, or a surface such as a marker). Even if the camera 13 moves or the imaging direction changes, the position of the virtual door VD can be controlled in conjunction with the camera 13, and a virtual image of the virtual door VD captured by the virtual camera VC can be displayed.

[0143] (Variation) Although the present embodiment has been described above, the above embodiment is merely an example, and the following modifications may be made, for example.

[0144] For example, in the above embodiment, when -b < θ < a, that is, when the angle θ of the imaging direction of the camera 13 with respect to the horizontal plane is greater than -b and less than a, these are interlocked so that the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, coincide. Specifically, in accordance with the imaging direction of the camera 13, the imaging direction of the virtual camera VC is set and the virtual door VD is moved in the vertical direction. In another embodiment, when -b < θ < a, the imaging direction of the virtual camera VC and the vertical position of the virtual door VD may be set in accordance with the imaging direction of the camera 13 so that the imaging direction of the virtual camera VC and the virtual object direction are interlocked with a first interlocking degree. Here, "being interlocked with a first interlocking degree" between two directions includes the case where the two directions always completely coincide. In this case, when the imaging direction of the virtual camera VC changes, the virtual object direction changes by the same amount as the change amount. Also, "being interlocked with a first interlocking degree" between two directions includes the case where the two directions do not always completely coincide but coincide over a predetermined time. Also, "being interlocked with a first interlocking degree" between two directions may include the case where the two directions do not completely coincide. Also, "being interlocked with a first interlocking degree" between two directions includes the case where the imaging direction of the virtual camera VC and the virtual object direction do not change at all.

[0145] In addition, in the above-described embodiment, when θ < -b or θ > a, the virtual door VD is not moved in accordance with the change in the imaging direction of the camera 13. In other embodiments, when θ < -b or θ > a, the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, may be set in accordance with the imaging direction of the camera 13 so as to interlock with each other at a second interlocking degree smaller than the first interlocking degree. For example, even when θ < -b or θ > a, the virtual door VD may be moved. Here, "the imaging direction of the virtual camera VC and the virtual object direction interlock with each other at a second interlocking degree smaller than the first interlocking degree" means that when the imaging direction of the camera 13 changes as compared with the case where -b < θ < a, the virtual door VD viewed from the display device 15 seems not to move at all in the real space, the moving range of the virtual door VD becomes smaller, the moving speed of the virtual door VD becomes smaller, and the like. Here, "the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, interlock with each other at a second interlocking degree smaller than the first interlocking degree" means that when the imaging direction of the virtual camera VC changes, the virtual object direction does not change at all, that is, even when the imaging direction of the virtual camera VC changes, the position of the virtual door VD does not change. Further, "the imaging direction of the virtual camera VC and the virtual object direction interlock with each other at a second interlocking degree smaller than the first interlocking degree" means that when the imaging direction of the virtual camera VC changes, the degree of change in the virtual object direction becomes smaller (the moving amount of the virtual door VD becomes smaller) as compared with the case where -b < θ < a. For example, when θ > a and the camera 13 changes by 10 degrees upward, the imaging direction of the virtual camera VC also changes by 10 degrees upward, but the virtual object direction may change by only an angle smaller than 10 degrees (for example, 5 degrees) (in this case, the virtual door VD moves upward by a distance corresponding to "5 degrees").

[0146] Further, in the above embodiment, on the premise that the imaging directions of the camera 13 and the virtual camera VC are made to coincide, the position of the virtual door VD is controlled in the virtual space according to the angle θ indicating the imaging direction of the camera 13 (virtual camera VC). In another embodiment, when the angle θ indicating the imaging direction of the camera 13 satisfies -b < θ < a, the imaging direction of the virtual camera VC and the position of the virtual door VD may not be changed according to the imaging direction of the camera 13. Also, when θ < -b or θ > a, only either the imaging direction of the virtual camera VC or the position of the virtual door VD may be changed according to the imaging direction of the camera 13. By controlling in this way, when the camera 13 is not pointing too much upward or downward (-b < θ < a), the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, coincide, and it appears as if the virtual door VD is moving vertically following the vertical change in the imaging direction of the camera 13. On the other hand, when the camera 13 is pointing too much upward or downward (θ < -b or θ > a), the imaging direction of the virtual camera VC and the virtual object direction do not coincide, and it appears as if the virtual door VD does not move vertically without following the vertical change in the imaging direction of the camera 13.

[0147] In the above embodiment, the posture of the virtual camera VC is set so that the posture of the camera 13 and the posture of the virtual camera VC coincide. When -b < θ < a, the imaging direction of the virtual camera VC is set according to the imaging direction of the camera 13, and the virtual door VD is moved so that the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, are interlocked. In other embodiments, when -b < θ < a and the imaging direction of the camera 13 changes, the imaging direction of the virtual camera VC may be changed and the position of the virtual camera VC may be changed without moving the virtual door VD. In this case, the relative positional relationship between the virtual camera VC and the virtual door VD does not change, and the imaging direction of the virtual camera VC and the virtual object direction do not change. Therefore, when the imaging direction of the camera 13 changes, the virtual door VD appears to move following the movement of the camera 13. Also, when -b < θ < a and the imaging direction of the camera 13 changes, both the imaging direction of the virtual camera VC and the position of the virtual door VD may be changed. That is, when -b < θ < a and the imaging direction of the camera 13 changes, at least one of the imaging direction of the virtual camera VC and the position of the virtual door VD may be controlled so that the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, are interlocked.

[0148] In the above embodiment, when the virtual door VD is not pseudo-fixed, the virtual camera VC is moved within the virtual space and the virtual door VD is moved within the virtual space in accordance with the movement of the camera 13 in the real space. Thereby, when the virtual door VD is not pseudo-fixed, the virtual camera VC and the virtual door VD are kept at a constant distance D. In other embodiments, when the virtual door VD is not pseudo-fixed, even if the camera 13 moves, the virtual camera VC and the virtual door VD may not be moved within the virtual space, and the distance between the virtual camera VC and the virtual door VD may be kept constant.

[0149] In the above embodiment, the virtual door VD is fixed in the virtual space, assuming that the virtual camera VC moves in the virtual space in accordance with the movement of the camera 13 in the real space. In other embodiments, when the camera 13 moves, the virtual door VD may be moved while keeping the virtual camera VC fixed in the virtual space, thereby pseudo-fixing the virtual door VD in the real space. For example, when the camera 13 moves leftward, the virtual door VD is moved rightward in the virtual space by the same amount, so that the virtual camera VC moves leftward relative to the virtual door VD. At this time, the virtual door VD in the composite image displayed on the display device 15 moves rightward on the screen. In this way, even if the virtual door VD moves in the virtual space when the camera 13 moves, the user can perceive the virtual door VD as being fixed at a predetermined position in the real space. Furthermore, when the camera 13 moves, both the virtual camera VC and the virtual door VD may be moved. Furthermore, when the imaging direction of the camera 13 changes, the virtual door VD may be moved so that the virtual door VD appears fixed in real space. For example, when the camera 13 is pointed to the right from the front, the virtual door VD may be moved to the left so that the virtual door VD moves to the left as seen from the virtual camera VC in the virtual space. Furthermore, when the camera 13 is pointed to the right from the front, the virtual camera VC may be pointed to the right while keeping the virtual door VD fixed so that the virtual door VD moves to the left as seen from the virtual camera VC in the virtual space. In this case, the virtual door VD displayed on the display device 15 moves to the left as the imaging direction of the camera 13 changes to the right, so that the virtual door VD appears fixed in real space. In other words, at least one of the position of the virtual camera, the imaging direction, and the position of the virtual door VD may be controlled according to the position or imaging direction of the camera 13 so that the virtual door VD viewed from the composite image displayed on the display device 15 appears to be fixed in real space.

[0150] Even after the virtual door VD is virtually fixed in the real space, the virtual door VD may move slightly in the virtual space. For example, even after the virtual door VD is virtually fixed in the real space, the virtual door VD may be moved slightly in the virtual space in response to a slight movement of the camera 13 in order to correct for camera shake of the camera 13.

[0151] In the above embodiment, the above restriction (restriction on vertical movement of the virtual door VD) is performed both when the imaging direction of the camera 13 faces upwards beyond the first threshold and when the imaging direction of the camera 13 faces downwards beyond the second threshold. In other embodiments, the above restriction may be performed at least in either the case where the imaging direction of the camera 13 faces upwards beyond the first threshold or the case where the imaging direction of the camera 13 faces downwards beyond the second threshold.

[0152] In other embodiments, any virtual object other than the virtual door VD may be placed in the virtual space, and the same processing as that for the virtual door VD described above may be performed on the virtual object.

[0153] That is, the information processing system generates a composite image by combining real images sequentially captured by a camera of a real space with virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, and outputs the composite image to a display device. In a first control state, at least one of the imaging direction of the virtual camera and the position of the virtual object may be controlled according to the imaging direction of the camera so that the imaging direction of the virtual camera and the virtual object direction, which is the direction from the virtual camera to the virtual object, are linked with a first degree of linkage. Furthermore, the control state may be changed from the first control state to a second control state based on a user operation, and in the second control state, at least one of the position of the virtual camera, the imaging direction, and the position of the virtual object may be controlled according to the position or the imaging direction of the camera so that the virtual object viewed from the composite image is pseudo-fixed in real space. When the imaging direction of the camera is pointing upwards above the first threshold or downwards above the second threshold, at least one of the virtual camera and the virtual object may be controlled so that the imaging direction of the virtual camera and the direction of the virtual object are linked with a second degree of linkage that is smaller than the first degree of linkage.

[0154] In the above embodiment, when the virtual door VD is fixed, the entry process is performed when the virtual camera VC moves to the position of the virtual door VD. In other embodiments, the entry process may be performed when a predetermined position condition regarding the position of the virtual camera VC and the position of the virtual object is satisfied.

[0155] In the above embodiment, the entry process is a process of determining that the virtual camera VC has entered the second virtual space VS2 through the virtual door VD, and includes a process of setting a flag indicating that the virtual camera VC is present in the second virtual space VS2 to ON. In other embodiments, when a predetermined positional condition regarding the positions of the virtual camera VC and the virtual object is satisfied, not only the above-described entry process but also an event process regarding the virtual object may be performed. The "event process regarding the virtual object" may include a process of setting the flag to ON, a process of having the virtual camera VC enter the second virtual space VS2 to display the inside of the second virtual space VS2, a process of setting a virtual boundary between the inside and outside of the second virtual space VS2, a process of generating the second virtual space VS2 based on the virtual object, a process of restricting the virtual camera VC from leaving the second virtual space VS2 from a position other than the virtual object, and the like. Furthermore, the "event process regarding the virtual object" may be a process of changing the display mode of the virtual object, a process of having a predetermined positional relationship between the virtual object and another object (e.g., a process of another object coming into contact with the virtual object), and the like.

[0156] In the above embodiment, the second virtual space VS2 is a small room that can be entered through the virtual door VD, and the virtual character exists in the second virtual space. In other embodiments, the second virtual space VS2 may be any space, such as a virtual space representing a specific location on Earth or outer space.

[0157] In the above embodiment, the attitude of the camera 13 (smartphone 1) is detected using an acceleration sensor and / or an angular velocity sensor, and the position of the camera 13 is detected based on the actual image captured by the camera 13. Then, the position and attitude of the virtual camera VC are set according to the detected position and attitude of the camera 13. In other embodiments, the position and attitude of the camera 13 (smartphone 1) may be detected using another method. For example, the position of the camera 13 may be detected using a global positioning system (GPS), and the attitude (imaging direction) of the camera 13 may be detected using a geomagnetic sensor. The position and attitude of the camera 13 may also be detected based on the actual image captured by the camera 13. For example, markers may be detected from the actual image captured by the camera 13, and the position and attitude of the camera 13 may be detected based on the marker detection result. Furthermore, the position and attitude of the smartphone 1 may be detected by capturing an image of the smartphone 1 using an external camera, not limited to the camera 13 fixed to the smartphone 1.

[0158] In the above embodiment, a composite image obtained by combining real images sequentially captured by the camera 13 of the real space with virtual images sequentially captured by the virtual camera VC of the virtual space is displayed on the display device 15. In other embodiments, the display device may be a video see-through display device. The user views the real space through a transparent display device and also views the virtual image displayed on the transparent display device. In this way, the transparent display device may be used to allow the user to view a composite image obtained by combining the real space and the virtual image.

[0159] In the above embodiment, a smartphone 1 is used in which a camera, a display device, and a processor that performs the above-described processing are integrated. The smartphone 1 is merely one example of an information processing system. For example, an information processing system in which a camera, a display device, and a processor are each separate may be used. Alternatively, an information processing system may be configured by a device in which a camera and a processor are integrated and a display device that is separate from the device. Alternatively, an information processing system may be configured by a device in which a camera and a display device are integrated and a device having a processor that is separate from the device. Alternatively, an information processing system may be configured by connecting a camera, a display device, and a processor via a network (such as a LAN or the Internet).

[0160] Furthermore, the configurations according to the above-described embodiments and their modifications can be combined in any manner as long as they are not inconsistent with each other. Furthermore, the above is merely an example of the present invention, and various other improvements and modifications may be made thereto. [Explanation of symbols]

[0161] 1. Smartphone 10 processors 13 Camera 14 Attitude detection sensor 15 Display device VC Virtual Camera VD Virtual Door

Claims

1. An information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the composite image comprising: a control means for controlling at least one of an imaging direction of the virtual camera and a position of the virtual object in accordance with an imaging direction of the imaging device so that the imaging direction of the virtual camera and a virtual object direction, which is a direction from the virtual camera to the virtual object, are linked with each other at a first degree of linkage; The control means when the imaging direction of the imaging device is directed upwardly relative to a first threshold value or downwardly relative to a second threshold value in the real space, at least one of the imaging direction of the virtual camera and the position of the virtual object is controlled so that the imaging direction of the virtual camera and the direction of the virtual object are linked with each other at a second degree of linkage that is smaller than the first degree of linkage; When the distance between the virtual camera and the virtual object is within a predetermined distance, the virtual camera is placed in a second virtual space based on the position of the virtual object, and the position or imaging direction of the virtual camera in the second virtual space is controlled according to the position or imaging direction of the imaging device.

2. 2. The information processing system according to claim 1, wherein when the imaging direction of the imaging device changes in the vertical direction, the control means moves the virtual object in the vertical direction while maintaining a vertical orientation of the virtual object in the virtual space.

3. 3. The information processing system according to claim 1, wherein the control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object in accordance with a change in the left-right direction of the imaging direction of the imaging device so that the imaging direction of the virtual camera and the virtual object direction are linked with the first degree of linkage even when the imaging direction of the imaging device is directed above the first threshold or below the second threshold.

4. 4. The information processing system according to claim 1, wherein the control means controls the position of the virtual camera and the position of the virtual object so that a constant distance is maintained between the virtual camera and the virtual object when the imaging device moves.

5. 5. The information processing system according to claim 1, wherein the control means sets an imaging direction of the virtual camera in accordance with an imaging direction of the imaging device, and controls a position of the virtual object so as to follow a change in the imaging direction of the virtual camera.

6. 6. The information processing system according to claim 1, wherein the control means places the virtual object at a predetermined position in the imaging direction of the virtual camera without detecting a predetermined plane in the real space based on the real image captured by the imaging device.

7. 1. An information processing program executed by a computer of an information processing device that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the information processing program comprising: a control unit that controls at least one of the imaging direction of the virtual camera and the position of the virtual object in accordance with the imaging direction of the imaging device so that the imaging direction of the virtual camera and the virtual object direction, which is the direction from the virtual camera to the virtual object, are linked with a first degree of linkage; The control means when the imaging direction of the imaging device is directed upwardly relative to a first threshold value or downwardly relative to a second threshold value in the real space, at least one of the imaging direction of the virtual camera and the position of the virtual object is controlled so that the imaging direction of the virtual camera and the direction of the virtual object are linked with each other at a second degree of linkage that is smaller than the first degree of linkage; an information processing program that, when a distance between the virtual camera and the virtual object is within a predetermined distance, places the virtual camera in a second virtual space based on the position of the virtual object, and controls the position or imaging direction of the virtual camera in the second virtual space according to the position or imaging direction of the imaging device.

8. An information processing device that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, a control means for controlling at least one of an imaging direction of the virtual camera and a position of the virtual object in accordance with an imaging direction of the imaging device so that the imaging direction of the virtual camera and a virtual object direction, which is a direction from the virtual camera to the virtual object, are linked with each other at a first degree of linkage; The control means when the imaging direction of the imaging device is directed upwardly relative to a first threshold value or downwardly relative to a second threshold value in the real space, at least one of the imaging direction of the virtual camera and the position of the virtual object is controlled so that the imaging direction of the virtual camera and the direction of the virtual object are linked with each other at a second degree of linkage that is smaller than the first degree of linkage; When the distance between the virtual camera and the virtual object is within a predetermined distance, the information processing device places the virtual camera in a second virtual space based on the position of the virtual object, and controls the position or imaging direction of the virtual camera in the second virtual space according to the position or imaging direction of the imaging device.

9. 1. An information processing method executed in an information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the method comprising: a control step of controlling at least one of an imaging direction of the virtual camera and a position of the virtual object in accordance with the imaging direction of the imaging device so that the imaging direction of the virtual camera and a virtual object direction, which is a direction from the virtual camera to the virtual object, are linked with a first degree of linkage, In the control step, when the imaging direction of the imaging device is directed upwardly relative to a first threshold value or downwardly relative to a second threshold value in the real space, at least one of the imaging direction of the virtual camera and the position of the virtual object is controlled so that the imaging direction of the virtual camera and the direction of the virtual object are linked with each other at a second degree of linkage that is smaller than the first degree of linkage; An information processing method, wherein, when the distance between the virtual camera and the virtual object is within a predetermined distance, the virtual camera is placed in a second virtual space based on the position of the virtual object, and the position or imaging direction of the virtual camera in the second virtual space is controlled according to the position or imaging direction of the imaging device.

10. An information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the composite image comprising: a virtual camera control means for controlling at least the imaging direction of the virtual camera in accordance with the imaging direction of the imaging device; an object position control means for controlling the position of the virtual object in a manner that follows the imaging direction of the virtual camera, the object position control means controls a position of the virtual object so that a degree to which the virtual object follows the imaging direction of the virtual camera decreases when the imaging direction of the imaging device is oriented above a first threshold value or below a second threshold value in the real space; The virtual camera control means, when the distance between the virtual camera and the virtual object is within a predetermined distance, places the virtual camera in a second virtual space based on the position of the virtual object, and controls the position or imaging direction of the virtual camera in the second virtual space according to the position or imaging direction of the imaging device.

11. 1. An information processing program executed by a computer of an information processing device that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the information processing program comprising: a virtual camera control means for controlling at least the imaging direction of the virtual camera in accordance with the imaging direction of the imaging device; the virtual camera is caused to function as an object position control means for controlling the position of the virtual object in a manner that follows the imaging direction of the virtual camera; the object position control means controls a position of the virtual object so that a degree to which the virtual object follows the imaging direction of the virtual camera decreases when the imaging direction of the imaging device is oriented above a first threshold value or below a second threshold value in the real space; The virtual camera control means, when the distance between the virtual camera and the virtual object is within a predetermined distance, places the virtual camera in a second virtual space based on the position of the virtual object, and controls the position or imaging direction of the virtual camera in the second virtual space according to the position or imaging direction of the imaging device.

12. An information processing device that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, a virtual camera control means for controlling at least the imaging direction of the virtual camera in accordance with the imaging direction of the imaging device; an object position control means for controlling the position of the virtual object in a manner that follows the imaging direction of the virtual camera, the object position control means controls a position of the virtual object so that a degree to which the virtual object follows the imaging direction of the virtual camera decreases when the imaging direction of the imaging device is oriented above a first threshold value or below a second threshold value in the real space; The virtual camera control means, when the distance between the virtual camera and the virtual object is within a predetermined distance, places the virtual camera in a second virtual space based on the position of the virtual object, and controls the position or imaging direction of the virtual camera in the second virtual space depending on the position or imaging direction of the imaging device.

13. 1. An information processing method executed in an information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the method comprising: a virtual camera control step of controlling at least the imaging direction of the virtual camera in accordance with the imaging direction of the imaging device; an object position control step of controlling a position of the virtual object in a manner that follows the imaging direction of the virtual camera, in the object position control step, when the imaging direction of the imaging device is directed upwardly relative to a first threshold value or downwardly relative to a second threshold value in the real space, the position of the virtual object is controlled so that a degree to which the virtual object follows the imaging direction of the virtual camera is reduced; In the virtual camera control step, if the distance between the virtual camera and the virtual object is within a predetermined distance, the virtual camera is placed in a second virtual space based on the position of the virtual object, and the position or imaging direction of the virtual camera in the second virtual space is controlled according to the position or imaging direction of the imaging device.

Images