Information processing system, information processing program, information processor, and method for processing information

Abstract

To provide an information processing system in which a virtual object can be arranged in a position that a user wants the virtual object to be.SOLUTION: An example of the information processing system sets the direction of imaging of a virtual camera according to the direction of imaging of the camera, and moves a virtual door in a virtual space according to the direction of imaging of the virtual camera if the direction of the imaging of the camera is in a predetermined range. If the direction of imaging of the camera is more upward than a first threshold value, the virtual door is not moved further upward. If the direction of imaging of the camera is more downward than a second threshold value, the virtual door is not moved further downward.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing program, an information processing device, and an information processing method capable of synthesizing images of a real space and a virtual space. [Background technology]

[0002] As a prior art, there is an image processing system that arranges a virtual object in a virtual space and displays a composite image of the real space and the virtual object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-071877 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is room for improvement in placing a virtual object in an appropriate position.

[0005] Therefore, an object of the present invention is to provide an information processing system, an information processing program, an information processing device, and an information processing method that are capable of placing a virtual object at an appropriate position. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention employs the following configuration.

[0007] One example of the present invention is an information processing system that outputs to a display device a composite image that is a combination of real images captured sequentially by an imaging device of a real space and virtual images captured sequentially by a virtual camera disposed in a virtual space of a virtual object, and includes a control means that controls at least one of the imaging direction of the virtual camera and the position of the virtual object in accordance with the imaging direction of the imaging device so that the imaging direction of the virtual camera and the virtual object direction, which is the direction from the virtual camera to the virtual object, are linked with a first degree of linkage, and when the imaging direction of the imaging device is facing upward than a first threshold value or downward than a second threshold value in the real space, the control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object so that the imaging direction of the virtual camera and the virtual object direction are linked with a second degree of linkage that is smaller than the first degree of linkage. Another example of the present invention is an information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera arranged in a virtual space of a virtual object. The information processing system includes a first control means, a control state changing means, and a second control means. In a first control state, the first control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object according to the imaging direction of the imaging device so that the imaging direction of the virtual camera and the virtual object direction, which is the direction from the virtual camera to the virtual object, are linked with a first degree of linkage. The control state changing means changes the first control state to a second control state based on a user operation. In the second control state, the second control means controls at least one of the position, imaging direction, and position of the virtual camera according to the position or imaging direction of the imaging device so that the virtual object viewed from the composite image is pseudo-fixed to a position in the real space. When, in the first control state, the imaging direction of the imaging device is directed upwardly above a first threshold value or downwardly below a second threshold value in the real space, the first control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object so that the imaging direction of the virtual camera and the virtual object direction are linked with a second linkage degree smaller than the first linkage degree.

[0008] According to the above, in the first control state, the imaging direction of the virtual camera and the virtual object direction can be linked in response to a change in the imaging direction of the imaging device. The control state is changed to the second control state in response to a user operation, and in the second control state, the virtual object is pseudo-fixed in the real space. This allows the virtual object to be pseudo-fixed at a desired position in the real space. Also, in the first control state, when the imaging direction of the imaging device faces upward more than the first threshold, the degree of linkage between the imaging direction of the virtual camera and the virtual object direction can be made smaller than when the imaging direction of the imaging device does not face upward more than the first threshold. Similarly, in the first control state, when the imaging direction of the imaging device faces downward more than the second threshold, the degree of linkage between the imaging direction of the virtual camera and the virtual object direction can be made smaller than when the imaging direction of the imaging device does not face downward more than the second threshold. This makes it possible to prevent the virtual object from being placed in an inappropriate position.

[0009] The information processing system may further include an event processing execution means that, when a position condition related to a position of the virtual camera and a position of the virtual object is satisfied in the second control state, executes an event processing related to the virtual object.

[0010] Based on the above, when a position condition related to the position of the virtual camera and the position of the virtual object is satisfied, it is possible to execute event processing related to the virtual object.

[0011] The position condition may be satisfied when the virtual camera approaches the virtual object within a predetermined distance. A second virtual space may be set based on the position of the virtual object. The event processing is processing for determining that the virtual camera has entered the second virtual space, and when the virtual camera has entered the second virtual space, the position or imaging direction of the virtual camera in the second virtual space may be controlled according to the position or imaging direction of the imaging device.

[0012] According to the above, when the virtual camera approaches the virtual object, it is determined that the virtual camera has entered the second virtual space. When the virtual camera has entered the second virtual space, the position or imaging direction of the virtual camera can be changed in the second virtual space according to a change in the position or imaging direction of the imaging device. This allows the user to approach the imaging device to the virtual object, enter the second virtual space, and look around the second virtual space.

[0013] In addition, when, in the first control state, the imaging direction of the imaging device changes in an up-down direction, the first control means may move the virtual object in an up-down direction while maintaining a vertical orientation of the virtual object in the virtual space.

[0014] According to the above, even if the imaging direction of the imaging device changes in the vertical direction in the first control state, the vertical orientation of the virtual object can be maintained. For example, the virtual object can be moved in the vertical direction while the upward direction of the virtual object is aligned with the upward direction of the virtual space.

[0015] The information processing system may further include a state change means for changing a state of the virtual object from a first state to a second state based on a user operation when the virtual object is pseudo-fixed by the second control means. When the virtual object is changed to the second state, the second virtual space may be visible from the virtual camera located in the virtual space.

[0016] Based on the above, in a state in which the virtual object is pseudo-fixed, the virtual object can be changed to the second state, and the second virtual space can be made visible in the second state.

[0017] In addition, the first control means may control at least one of the imaging direction of the virtual camera and the position of the virtual object in accordance with a change in the imaging direction of the imaging device in the left-right direction so that the imaging direction of the virtual camera and the virtual object direction are linked with the first degree of linkage even when the imaging direction of the imaging device is directed above the first threshold value or below the second threshold value.

[0018] According to the above, even if the imaging direction of the imaging device is facing too far up or too far down, by facing the imaging direction of the imaging device to the left or right, for example, the virtual camera can be directed to the left or right, and the imaging device and the virtual object can be linked, thereby improving usability.

[0019] In addition, in the first control state, when the imaging device moves, the first control means may control a position of the virtual camera and a position of the virtual object so that the virtual camera and the virtual object are kept at a constant distance from each other.

[0020] Based on the above, in the first control state, even when the imaging device moves, the distance between the virtual camera and the virtual object can be kept constant, and the imaging device and the virtual object can be linked.

[0021] In addition, in the first control state, the first control means may set an imaging direction of the virtual camera in accordance with an imaging direction of the imaging device, and control a position of the virtual object so as to follow a change in the imaging direction of the virtual camera.

[0022] Based on the above, when the imaging direction of the imaging device (virtual camera) changes, the position of the virtual object can be controlled in accordance with the change in the imaging direction, and the imaging device and the virtual object can be linked to each other.

[0023] The information processing system may further include a display mode control means for changing a display mode of the virtual object when the virtual object is pseudo-fixed in the real space.

[0024] Based on the above, it is possible to make the user easily recognize that the virtual object is pseudo-fixed in real space.

[0025] Furthermore, the first control means may place the virtual object at a predetermined position in an imaging direction of the virtual camera, without detecting a predetermined plane in the real space based on an actual image captured by the imaging device.

[0026] Based on the above, it is possible to place a virtual object in a virtual space without detecting a predetermined surface in a real space.

[0027] Another example of the present invention is an information processing system that outputs to a display device a composite image obtained by combining real images captured sequentially by an imaging device of a real space with virtual images captured sequentially by a virtual camera arranged in a virtual space of a virtual object, the information processing system including a virtual camera direction control means and an object position control means. The virtual camera direction control means controls the imaging direction of the virtual camera according to the imaging direction of the imaging device. The object position control means controls the position of the virtual object in a manner that follows the imaging direction of the virtual camera. When the imaging direction of the imaging device is directed upwardly above a first threshold value or downwardly below a second threshold value in the real space, the object position control means controls the position of the virtual object so that the degree to which the virtual object follows the imaging direction of the virtual camera decreases. Another example of the present invention is an information processing system that outputs a composite image, which is a composite image of real images captured sequentially by an imaging device of a real space and virtual images captured sequentially by a virtual camera arranged in a virtual space, to a display device, and includes a virtual camera direction control means, an object position control means, an object fixing means, and a virtual camera position control means. The virtual camera direction control means controls the imaging direction of the virtual camera according to the imaging direction of the imaging device. The object position control means controls the position of the virtual object in a manner that follows the imaging direction of the virtual camera. The object fixing means controls the position of the virtual object so that the position of the virtual object is pseudo-fixed to a position in the real space based on a user operation. The virtual camera position control means controls the position of the virtual camera according to the position of the imaging device in a state in which the position of the virtual object is fixed. When the imaging direction of the imaging device is directed upwards above a first threshold value or downwards below a second threshold value in the real space, the object position control means controls the position of the virtual object so that the degree to which the virtual object follows the imaging direction of the virtual camera decreases.

[0028] According to the above, the imaging direction of the virtual camera can be set according to a change in the imaging direction of the imaging device, and the virtual object can be fixed in a pseudo manner in real space according to a user operation. When the imaging direction of the imaging device faces upwards beyond a first threshold value or faces downwards beyond a second threshold value, the degree to which the virtual object follows the imaging direction of the virtual camera can be reduced. This makes it possible to prevent the virtual object from being placed in an inappropriate position.

[0029] Another invention may be an information processing program executed in one or more computers included in the information processing system, or an information processing device. Another invention may be an information processing method performed in the information processing system. Effect of the Invention

[0030] According to the present invention, it is possible to place a virtual object at a desired position and prevent the virtual object from being placed at an inappropriate position. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a diagram showing an example of the internal configuration of a smartphone 1 according to an embodiment of the present invention. [Diagram 2] A diagram showing an example of a real space [Diagram 3] FIG. 1 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is placed at a position P1. [Figure 4] FIG. 1 is a diagram showing an example of the positional relationship between various objects in a real space and a virtual space; [Diagram 5] FIG. 13 is a diagram showing an example of a change in the virtual space when the imaging direction of the camera 13 is changed to an upward direction. [Figure 6] FIG. 13 is a diagram showing an example of an image displayed on the display device 15 when the angle θ of the imaging direction of the camera 13 is “0” [Figure 7] FIG. 13 is a diagram showing an example of an image displayed on the display device 15 when the angle θ of the imaging direction of the camera 13 is a first threshold value “a”; [Figure 8] FIG. 13 is a diagram showing an example of an image displayed on the display device 15 when the angle θ of the imaging direction of the camera 13 is greater than a first threshold value “a”; [Figure 9] FIG. 13 is a diagram showing an example of a change in the virtual space when the imaging direction of the camera 13 is changed downward. [Figure 10] FIG. 13 is a diagram showing an example of an image displayed on the display device 15 when −b<θ<0. [Figure 11] FIG. 13 is a diagram showing an example of an image displayed on the display device 15 when θ=−b. [Figure 12] FIG. 13 is a diagram showing an example of an image displayed on the display device 15 when θ<-b. [Figure 13] FIG. 13 is a diagram showing the virtual space as viewed from above, illustrating an example of the operation of the virtual camera VC and the virtual door VD when the imaging direction of the camera 13 is changed to the right. [Figure 14]FIG. 11 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is rotated rightward by c degrees from the state of FIG. 10. [Figure 15] FIG. 13 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is rotated rightward by c degrees from the state of FIG. 12. [Figure 16] FIG. 13 is a diagram showing an example of an image displayed on the display device 15 after a virtual door VD is pseudo-fixed in real space. [Figure 17] FIG. 17 shows an example of an image displayed when the camera 13 is translated leftward from the state shown in FIG. 16. [Figure 18] FIG. 1 shows an example of a portion of a second virtual space seen through a virtual door VD. [Figure 19] FIG. 1 is a diagram showing an example of the positional relationship between a virtual door VD, a virtual camera VC, and a second virtual space VS2. [Figure 20] FIG. 20 is a diagram showing an example of an image displayed when a virtual camera VC is placed at a position P2 shown in FIG. 19. [Figure 21] A side view of the real space when the virtual door VD is moved in conjunction with the imaging direction of the virtual camera VC without any restrictions. [Figure 22] FIG. 1 is a diagram showing an example of data stored in a memory 11 of a smartphone 1. [Figure 23] 1 is a flowchart showing an example of a main process executed by the processor 10 of the smartphone 1. [Figure 24] A flowchart showing details of the virtual door fixing process in step S105. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The information processing system of this embodiment is a system for allowing a user to view a composite image obtained by combining an image of a virtual space with an image of a real space, and is a system for providing an augmented reality (AR) to a user. The information processing system may be, for example, a smartphone, a tablet terminal, a portable game device, a portable personal computer, or the like. In addition, the information processing system is not limited to a portable device, and may be a stationary device. In this embodiment, the information processing system is a smartphone.

[0033] Fig. 1 is a diagram showing an example of the internal configuration of a smartphone 1 according to the present embodiment. As shown in Fig. 1, the smartphone 1 includes, for example, a processor 10, a memory 11, an input device 12, a camera 13, a posture detection sensor 14, and a display device 15. The smartphone 1 also includes a storage device (for example, a non-volatile memory) not shown.

[0034] The processor 10 cooperates with the memory 11 to process input information from the input device 12, process real images from the camera 13, and calculate the attitude of the smartphone 1 (camera 13) by processing data from the attitude detection sensor 14. The processor 10 also performs processing related to various objects in a virtual space, which will be described later, generates virtual images based on a virtual camera, and generates composite images by combining the real images and the virtual images. The processor 10 outputs the generated composite images to the display device 15. The processor 10 may include a GPU (Graphics Processing Unit) for performing image processing.

[0035] The input device 12 receives input from a user. The input device 12 may be, for example, a touch panel or a button.

[0036] The camera 13 (imaging device) is provided on the rear surface of the smartphone 1 and captures an image of the real space in the rear direction.

[0037] The attitude detection sensor 14 is a sensor for detecting the attitude of the smartphone 1 (camera 13), and is, for example, an acceleration sensor and / or an angular velocity sensor.

[0038] Display device 15 is provided on the front surface of smartphone 1, and displays images from camera 13 and images of a virtual space. Display device 15 may be any display device, such as a liquid crystal display or an organic EL display.

[0039] Next, an image displayed on the display device 15 of the smartphone 1 will be described. Fig. 2 is a diagram showing an example of a real space. Fig. 3 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is placed at a position P1.

[0040] As shown in Fig. 2, the real space includes, for example, a floor RF, a cylindrical real object RO placed on the floor RF, and a wall RW. When the smartphone 1 is placed at a position P1 in the real space shown in Fig. 2, an image as shown in Fig. 3 is displayed on the display device 15.

[0041] An image (real image) of the real space captured by the camera 13 is displayed on the display device 15 of the smartphone 1. For example, an image of a floor RF and an image of a real object RO are displayed on the display device 15 as real images. A virtual door VD is also displayed on the display device 15. The virtual door VD is an example of a virtual object arranged in a virtual space and does not exist in the real space. A virtual space is defined inside the smartphone 1, and the virtual door VD is arranged in the virtual space. A virtual camera is arranged in the virtual space, and a virtual image is generated by capturing an image of the virtual door VD using the virtual camera. The generated virtual image is superimposed on the real image from the camera 13 to generate a composite image, and the composite image is output to the display device 15. As a result, a composite image as shown in FIG. 3 is displayed.

[0042] Fig. 4 is a diagram showing an example of the positional relationship between various objects in a real space and a virtual space, in which the real space and the virtual space are viewed from the side when the image shown in Fig. 3 is displayed.

[0043] 4, an XYZ coordinate system fixed to the smartphone 1 is defined. The Z axis is an axis in the rear direction of the smartphone 1, and is an axis facing the imaging direction of the camera 13. The Y axis is an axis perpendicular to the Z axis, and is an axis in the upward direction with the smartphone 1 (camera 13) as a reference. The X axis is an axis perpendicular to the Z axis and the Y axis, and is an axis in the leftward direction with the smartphone 1 (camera 13) as a reference.

[0044] The imaging direction of the camera 13 (attitude of the smartphone 1) is calculated based on data from the attitude detection sensor 14. Because the camera 13 is fixed to the smartphone 1, the attitude calculated based on the data from the attitude detection sensor 14 is the attitude of both the smartphone 1 and the camera 13. For example, the smartphone 1 calculates the angle (θ) of the Z axis with respect to the horizontal plane (floor RF) based on gravity G detected by the attitude detection sensor 14.

[0045] A virtual camera VC is placed within the virtual space VS. An xyz coordinate system is set for the virtual camera VC. The z-axis of the virtual camera VC is an axis that faces the imaging direction of the virtual camera VC. The y-axis of the virtual camera VC is an axis perpendicular to the z-axis, and is an axis in the upward direction with respect to the virtual camera VC. The x-axis of the virtual camera VC is an axis perpendicular to the z-axis and y-axis, and is an axis in the leftward direction with respect to the virtual camera VC.

[0046] The attitude of the camera 13 in the real space and the attitude of the virtual camera VC in the virtual space VS are linked. Specifically, the attitude of the virtual camera VC is set so as to match the attitude of the smartphone 1 calculated based on data from the attitude detection sensor 14 (e.g., an acceleration sensor and / or an angular velocity sensor). For example, when the camera 13 in the real space changes from a horizontal state to a state facing 10 degrees upward, the virtual camera VC also changes from a horizontal state to a state facing 10 degrees upward in the virtual space VS.

[0047] The position of the virtual door VD is controlled in a manner that follows the imaging direction of the virtual camera VC. Specifically, as shown in Fig. 4, the virtual door VD is placed in the imaging direction (z-axis direction) of the virtual camera VC. The virtual door VD is a virtual object that resembles a door, and is a plate-shaped virtual object. The virtual door VD is placed at a position a certain distance D away from the position of the virtual camera VC. For example, the center of the virtual door VD may be set at a position D away from the virtual camera VC in the z-axis direction.

[0048] A composite image obtained by combining an actual image captured by the camera 13 of the real space and a virtual image captured by the virtual camera VC of the virtual door VD is displayed on the display device 15 of the smartphone 1. Therefore, when a user looks at the composite image, it appears as if the virtual door VD exists in the real space.

[0049] The virtual door VD is set with an xd axis, a yd axis, and a zd axis. The xd axis is an axis perpendicular to the virtual door VD, the yd axis is an axis pointing upward with respect to the virtual door VD, and the xd axis is an axis pointing left with respect to the virtual door VD. The virtual door VD is placed in the virtual space so as to be perpendicular to the horizontal plane of the virtual space VS (the horizontal plane of the real space). In other words, the virtual door VD is placed in the virtual space so that the upward yd axis of the virtual door VD and the upward axis of the virtual space face in the same direction.

[0050] Although details will be described later, the virtual door VD is a virtual object that serves as an entrance to a virtual room (second virtual space).

[0051] Next, an image displayed on the display device 15 when the attitude of the camera 13 (smartphone 1) changes from the state shown in FIG. 3 will be described.

[0052] FIG. 5 is a diagram showing an example of a change in the virtual space when the imaging direction of the camera 13 is changed to face upward.

[0053] 5, when the angle θ between the z-axis of virtual camera VC (camera 13) and the horizontal plane is "0" degrees, the center of virtual door VD is positioned at a certain distance D away from the z-axis of virtual camera VC (camera 13). At this time, virtual door VD appears to exist, for example, at a height H1 from the horizontal plane (floor RF). Note that, hereinafter, when camera 13 (virtual camera VC) faces upward, angle θ is a positive value, and when camera 13 (virtual camera VC) faces downward, angle θ is a negative value.

[0054] When the imaging direction of the camera 13 is changed upward so that the angle θ ranges from 0 to a first threshold value a (a is a positive value), the position of the virtual door VD changes following the change in the imaging direction of the camera 13. That is, the imaging direction of the virtual camera VC changes according to the imaging direction of the camera 13, and the position of the virtual door VD also changes so that the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, are interlocked. Therefore, to the user, it seems that the camera 13 and the virtual door VD are interlocked, and it seems that the virtual door VD moves following the movement of the camera 13. Specifically, the virtual door VD moves upward in the virtual space while maintaining a state perpendicular to the horizontal plane. That is, the virtual door VD moves upward while maintaining a state where the yd axis in the upward direction of the virtual door VD is parallel to the axis in the upward direction of the virtual space. For example, from the state where θ = 0 to the state where θ = a, the virtual door VD is arranged in the virtual space so that the center of the virtual door VD is located at a position a fixed distance D away from the z axis of the virtual camera VC. Also, during this period, the yd axis in the upward direction of the virtual door VD is controlled to face the direction opposite to the gravitational direction (the upward direction of the virtual space). When θ = a, when looking at the display device 15, the virtual door VD seems to exist, for example, at a position at a height H2 (> H1) from the horizontal plane (floor RF).

[0055] When the imaging direction of the camera 13 is further changed upward from the state where θ = a, the imaging direction of the virtual camera VC changes to coincide with the imaging direction of the camera 13. However, the position of the virtual door VD does not change following the change in the imaging direction of the camera 13. That is, in the state where θ > a, the virtual door VD does not move further upward, and the upward movement of the virtual door VD is restricted.

[0056] Thus, on the premise that the imaging direction of the virtual camera VC changes according to the change in the imaging direction of the camera 13, in the range of 0 < θ < a, the virtual door VD moves upward in conjunction with the change in the imaging direction of the virtual camera VC. On the other hand, when θ > a, that is, when the imaging direction of the virtual camera VC faces upward from the first threshold value a, the upward movement of the virtual door VD is restricted.

[0057] When the camera 13 (smartphone 1) translates in the up-down, left-right, or front-back directions, the relative positional relationship between the virtual camera VC and the virtual door VD does not change. For example, when the camera 13 translates left-right while keeping the imaging direction fixed, the virtual camera VC also translates left-right in the virtual space, and the virtual door VD also translates left-right in the virtual space. The same is true when the camera 13 translates up-down while keeping the imaging direction fixed.

[0058] In addition, when the camera 13 moves in parallel in the up-down, left-right, and front-rear directions while keeping the imaging direction fixed, the virtual camera VC and the virtual door VD do not need to move in the virtual space. Even in this case, the relative positional relationship between the virtual camera VC and the virtual door VD does not change, so the virtual door VD remains displayed almost in the center of the display device 15, and it appears that the virtual door VD moves following the movement of the camera 13.

[0059] Fig. 6 is a diagram showing an example of an image displayed on display device 15 when the angle θ of the imaging direction of camera 13 is "0". Fig. 7 is a diagram showing an example of an image displayed on display device 15 when the angle θ of the imaging direction of camera 13 is a first threshold value "a". Fig. 8 is a diagram showing an example of an image displayed on display device 15 when the angle θ of the imaging direction of camera 13 is larger than the first threshold value "a".

[0060] In Fig. 6, the imaging direction of the camera 13 is directed upward compared to Fig. 3, so the image of the real object RO is displayed lower on the screen than in Fig. 3. Also, an image of the wall RW is not displayed in Fig. 3, but is displayed in Fig. 6. Also, as shown in Fig. 6, a virtual door VD is positioned in front of the virtual camera VC, and the virtual door VD is displayed approximately in the center of the display device 15.

[0061] As shown in FIG. 7, even when θ=a, the virtual door VD is located in front of the virtual camera VC, and the virtual door VD is displayed approximately in the center of the display device 15. While θ changes from 0 to a, the virtual door VD moves upward following the change in the imaging direction of the virtual camera VC (camera 13). That is, the virtual door VD viewed from the composite image displayed on the display device 15 moves upward in conjunction with the change in the imaging direction of the camera 13. On the other hand, as shown in FIG. 7, the image of the real object RO is displayed further below than in FIG. 6. Also, the area of ​​the image of the wall RW is larger than in FIG. 6.

[0062] Also, as shown in FIG. 8, when θ>a, that is, when the camera 13 faces upwards beyond the first threshold, the virtual door VD is displayed on the lower side of the display screen of the display device 15 without following the change in the imaging direction of the camera 13 (virtual camera VC). In FIG. 8, the lower part of the virtual door VD is out of the imaging range of the virtual camera VC, and the lower part is not displayed on the display device 15. If the camera 13 is directed further upwards than in FIG. 8, the virtual door VD will be out of the imaging range of the virtual camera VC and will not be displayed on the screen. In contrast, as shown in FIG. 8, the real image moves further downwards than in FIG. 7. The images of the real object RO and floor RF are displayed further downwards than in FIG. 7, and the area of ​​the image of the wall RW is larger than in FIG. 7.

[0063] Next, a case where the camera 13 is pointed downward will be described. Fig. 9 is a diagram showing an example of a change in the virtual space when the imaging direction of the camera 13 is changed to face downward.

[0064] 9, when the angle θ is smaller than 0 (i.e., the camera 13 faces downward) and is larger than "-b (b is a positive value)" degrees, the center of the virtual door VD is positioned at a certain distance D from the z-axis of the virtual camera VC (camera 13). At this time, the virtual door VD viewed from the composite image displayed on the display device 15 is, for example, at a height H3 (

[0065] ​ When θ is changed to the second threshold value "-b" (that is, when the camera 13 is changed in angle downward by b), the virtual door VD moves downward following the change in the imaging direction of the camera 13. That is, so that the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, are interlocked, according to the imaging direction of the camera 13, the imaging direction of the virtual camera VC changes and the position of the virtual door VD also changes. For this reason, to the user, it seems that the camera 13 and the virtual door VD are interlocked, and it seems that the virtual door VD is moving following the movement of the camera 13. Specifically, the virtual door VD moves downward in the virtual space while maintaining a state perpendicular to the horizontal plane. During this time, the upward yd axis of the virtual door VD is controlled to face the direction opposite to the direction of gravity (the upward direction in the virtual space). In the state of θ = -b, the virtual door VD seems to exist, for example, at a position with a height H4 (<H3) from the horizontal plane (floor RF).

[0066] When the imaging direction of the camera 13 is further changed downward from the state of θ = -b, the imaging direction of the virtual camera VC changes so as to coincide with the imaging direction of the camera 13. However, the position of the virtual door VD does not change following the change in the imaging direction of the camera 13. That is, in the state of θ < -b, the virtual door VD does not move further downward, and the downward movement of the virtual door VD is restricted.

[0067] As described above, in the range of -b < θ < 0, the virtual door VD moves downward in conjunction with the change in the imaging direction of the virtual camera VC. On the other hand, when θ < -b, that is, when the imaging direction of the virtual camera VC is directed downward from the second threshold value b, the downward movement of the virtual door VD is restricted.

[0068] FIG. 10 is a diagram showing an example of an image displayed on the display device 15 when -b < θ < 0. FIG. 11 is a diagram showing an example of an image displayed on the display device 15 when θ = -b. FIG. 12 is a diagram showing an example of an image displayed on the display device 15 when θ < -b.

[0069] In Fig. 10, the imaging direction of the camera 13 is more downward than in Fig. 6, so the image of the real object RO is displayed higher on the screen than in Fig. 6, and part of the real object RO is not displayed. Also, the image of the wall RW is not displayed in Fig. 10. Also, as shown in Fig. 10, a virtual door VD is positioned in front of the virtual camera VC, and the virtual door VD is displayed approximately in the center of the display device 15.

[0070] As shown in Fig. 11, when θ = -b, the virtual door VD is located in front of the virtual camera VC, and the virtual door VD is displayed approximately in the center of the display device 15. While θ changes from 0 to -b, the virtual door VD moves downward following the change in the imaging direction of the virtual camera VC (camera 13). In other words, the virtual door VD viewed from the composite image displayed on the display device 15 moves downward in conjunction with the change in the imaging direction of the camera 13. On the other hand, the image of the real object RO moves further upward than in Fig. 10, and is almost no longer displayed.

[0071] Also, as shown in Fig. 12, when θ<-b, that is, when the camera 13 faces downward further than the second threshold, the virtual door VD is displayed on the upper side of the display screen of the display device 15 without following the change in the imaging direction of the camera 13 (virtual camera VC). In Fig. 12, the upper part of the virtual door VD is out of the imaging range of the virtual camera VC, and the upper part is not displayed on the display device 15. If the camera 13 is directed further downward than in Fig. 12, the virtual door VD will be out of the imaging range of the virtual camera VC and will no longer be displayed on the screen. In contrast, as shown in Fig. 12, the real image moves further downward than in Fig. 11.

[0072] Next, a case where the imaging direction of the camera 13 is changed to the left or right will be described. Fig. 13 is a diagram showing the virtual space as viewed from above, illustrating an example of the operation of the virtual camera VC and the virtual door VD when the imaging direction of the camera 13 is changed to the right.

[0073] When the imaging direction of the camera 13 changes in the left-right direction, the imaging direction of the virtual camera VC also changes in the left-right direction. For example, when the rotation angle φ of the camera 13 (smartphone 1) in the right direction is 10 degrees, the virtual camera VC also rotates 10 degrees to the right. As shown in FIG. 13, the virtual door VD also moves in the left-right direction in conjunction with the left-right rotation of the imaging direction of the virtual camera VC. Specifically, the virtual door VD moves in the left-right direction while changing its posture so as to face the virtual camera VC. For example, when the virtual camera VC rotates in the right direction by a predetermined angle (when the camera 13 rotates in the right direction by a predetermined angle), the virtual door VD moves to the right while maintaining the distance D from the virtual camera VC. Even when the virtual door VD moves in the left-right direction in response to the left-right change in the imaging direction of the camera 13, the virtual door VD maintains a posture perpendicular to the horizontal plane. That is, the virtual door VD moves in the left-right direction in the virtual space while maintaining a state in which the upward yd axis of the virtual door VD and the upward axis of the virtual space are parallel to each other.

[0074] In addition, when the imaging direction of the camera 13 changes left and right, the virtual door VD moves left and right in conjunction with the left and right movement of the imaging direction of the virtual camera VC, regardless of the value of the left and right rotation angle φ.

[0075] Fig. 14 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is rotated rightward by c degrees from the state of Fig. 10. Fig. 15 is a diagram showing an example of an image displayed on the display device 15 when the smartphone 1 is rotated rightward by c degrees from the state of Fig. 12.

[0076] 14, when the smartphone 1 is turned to the right from the state of FIG. 10, the real object RO in the real image moves to the left side of the screen, and a part of the right wall RW in the real space is displayed on the screen. On the other hand, the virtual door VD is located in front of the virtual camera VC and is displayed approximately in the center of the display device 15.

[0077] Also, as shown in FIG. 15, when the smartphone 1 is turned to the right from the state of FIG. 12, an image of the right side of the real space is displayed on the screen. The virtual door VD is located in front of the virtual camera VC and is displayed at approximately the center of the display device 15. In this way, even when the camera 13 is facing downward from the second threshold, the virtual door VD also moves in the left-right direction in response to the camera 13 being turned to the left-right direction. The same is true in the case where the camera 13 is facing upward from the first threshold. That is, even when the imaging direction of the camera 13 is facing upward from the first threshold or downward from the second threshold, if the imaging direction of the camera 13 changes in the left-right direction, the position of the virtual door VD also changes in the left-right direction in response to the change in the imaging direction of the camera 13 (virtual camera VC).

[0078] (Fixing virtual door VD) Next, the process after the virtual door VD is fixed in the real space will be described. As described above, when the imaging direction of the camera 13 is changed while the virtual door VD is displayed on the display device 15, the virtual door VD moves in the virtual space accordingly. Also, when the camera 13 is translated while the imaging direction of the camera 13 is fixed, the virtual door VD moves in the real space accordingly. As a result, the virtual door VD appears to be linked to the camera 13, and the virtual door VD appears to be moving in the real space. When a predetermined user operation is performed on the input device 12 in this state, the virtual door VD is fixed in the real space in a pseudo manner.

[0079] Fig. 16 is a diagram showing an example of an image displayed on the display device 15 after the virtual door VD is fixed in a virtual manner in the real space. As shown in Fig. 16, when a predetermined user operation (e.g., a touch input on the virtual door VD) is performed on the input device 12, the virtual door VD is fixed in a virtual manner and the display mode of the virtual door VD changes. Before the predetermined user operation, the virtual door VD is displayed in a first display mode (e.g., a dashed line mode as shown in Fig. 3), and in this state, the virtual door VD moves according to a change in the imaging direction of the camera 13. Note that the predetermined user operation is not limited to a touch input, and may be a button input, a non-touch input (e.g., a gesture input), an input by voice, or the like.

[0080] After a predetermined user operation, the virtual door VD is displayed in a second display mode (for example, a solid line mode as shown in FIG. 16). In this state, the virtual door VD is pseudo-fixed in real space. When the virtual door VD is pseudo-fixed, the position of the virtual door VD does not change even if the imaging direction of the camera 13 changes. Furthermore, when the camera 13 is translated, the virtual door VD does not move following the camera 13.

[0081] FIG. 17 is a diagram showing an example of an image displayed when the camera 13 is translated to the left from the state of FIG. 16. As shown in FIG. 17, when the virtual door VD is pseudo-fixed in the real space, even if the camera 13 translates to the left, the virtual door VD does not translate to the left. Specifically, in FIG. 17, the real object RO moves to the right of the screen in response to the translation of the camera 13 to the left, and a part of the real object RO is displayed at the right end. As the real object RO moves in the screen, the virtual door VD also moves to the right in the screen. The amount of movement of the real object RO in the screen is the same as the amount of movement of the virtual door VD. This makes it appear as if the virtual door VD is pseudo-fixed in the real space.

[0082] Specifically, a fixed vx-vy-vz Cartesian coordinate system is set in the virtual space, and when a predetermined user operation is performed, the vx-vy-vz coordinate value of the virtual door VD is fixed. Meanwhile, the virtual camera VC moves in the virtual space according to the movement of the camera 13. Also, the imaging direction of the virtual camera VC is set to match the imaging direction of the camera 13. While the virtual camera VC moves in the virtual space and changes its imaging direction following the camera 13, the virtual door VD is fixed in the virtual space. Therefore, the virtual door VD displayed on the display device 15 appears to be pseudo-fixed in the real space.

[0083] The position of the virtual camera VC is calculated based on the real images from the camera 13 acquired at a predetermined time interval (for example, 1 / 60 second intervals). The movement (movement) of the camera 13 is calculated based on the real images acquired successively, and the position of the virtual camera VC is set so as to match the movement (movement) of the camera 13. The movement (movement) of the camera 13 may be calculated based on the output from the attitude detection sensor 14 in addition to (or instead of) the real images from the camera 13. For example, the movement amount and movement direction of the camera 13 may be calculated based on the acceleration output from the attitude detection sensor 14. Then, the virtual camera VC may be moved in the virtual space based on the calculated movement amount and movement direction. When the virtual door VD is fixed, the relative positional relationship between the virtual camera VC and the virtual door VD changes by moving the camera 13 (virtual camera VC).

[0084] The vx-vy-vz coordinate system fixed in the virtual space may be set with the virtual door VD as a reference when the virtual door VD is fixed. For example, the origin of the vx-vy-vz coordinate system may be set at the position of the virtual door VD, the vy axis may be set in the direction opposite to the direction of gravity, and the vx axis may be set parallel to the xd axis of the virtual door VD. The vx-vy-vz coordinate system may also be fixed in the virtual space before the virtual door VD is fixed. For example, when a real image is acquired from the camera 13, a plane in the real space may be detected based on the real image and the direction of gravity, and the origin of the vx-vy-vz coordinate system may be set on the plane. The vy axis may be set in the direction opposite to the direction of gravity, and the detected plane may be set as the vx-vz plane. Also, a marker may be placed in the real space, and the vx-vy-vz coordinate system may be set based on a marker image included in the real image from the camera 13.

[0085] (Second virtual space) As described above, when the virtual door VD is virtually fixed in the real space (when the virtual door VD is fixed in the virtual space), a second virtual space is generated based on the virtual door VD.

[0086] FIG. 18 is a diagram showing an example of a part of the second virtual space seen through the virtual door VD. After the virtual door VD is fixed, for example, in response to a user's input to the input device 12, the virtual door VD changes from a closed state (first state) shown in FIG. 16 to an open state (second state) shown in FIG. 18. When the virtual door VD changes to an open state, the inside of the second virtual space VS2 becomes visible. As shown in FIG. 18, an image is displayed in which a real image of the real space, an image of the virtual door VD, and an image of the second virtual space VS2 are synthesized. Specifically, as shown in FIG. 18, an image of the real space is displayed in the area other than the virtual door VD, and images of the real object RO and floor RF are displayed. At the opening of the virtual door VD, an image of the inside of a virtual room showing the second virtual space VS2 is displayed. In the second virtual space VS2, a virtual table VT and a virtual character C1 are arranged on a virtual floor VF.

[0087] Fig. 19 is a diagram showing an example of the positional relationship between the virtual door VD, the virtual camera VC, and the second virtual space VS2. Fig. 19 shows a diagram of the entire virtual space including the second virtual space VS2 as seen from above.

[0088] As shown in Fig. 19, when the virtual camera VC is located at position P1 and the virtual door VD is open, the image shown in Fig. 18 is displayed. In this state, the virtual camera VC is outside the second virtual space VS2 and cannot move freely within the second virtual space VS2.

[0089] The second virtual space VS2 is set according to the position of the virtual door VD. The second virtual space VS2 is a virtual room surrounded by a virtual floor VF, four virtual walls VW, and a virtual ceiling. Outside the second virtual space VS2 is the virtual space VS, and the virtual door VD is located between the inside and outside of the second virtual space VS2. The second virtual space VS2 corresponds to a predetermined range of the real space. For example, the second virtual space VS2 corresponds to a range of several tens of centimeters to several meters square in the real space, and gives the user the sensation that the second virtual space VS2 exists within that range.

[0090] In the second virtual space VS2, a virtual table VT, a virtual character C1, and a virtual character C2 are arranged. In addition to these, various objects (e.g., furniture objects) may be arranged in the second virtual space VS2. The virtual floor VF of the second virtual space VS2 is set perpendicular to the virtual door VD. That is, the virtual floor VF and the yd axis of the virtual door VD intersect perpendicularly.

[0091] As shown in FIG. 16 or FIG. 18, when the camera 13 is moved in the imaging direction from a state where the virtual camera VC and the virtual door VD are separated by a predetermined distance, the virtual camera VC also moves in the z-axis direction, and the virtual camera VC approaches the virtual door VD. When the virtual camera VC and the virtual door VD satisfy a predetermined position condition (when the virtual camera VC approaches the virtual door VD within a predetermined distance), the virtual camera VC can enter the second virtual space VS2. For example, when the distance between the virtual camera VC and the virtual door VD is less than a predetermined value, the position of the virtual camera VC in the left-right direction is within the width of the virtual door VD, and the position of the virtual camera VC in the up-down direction is within the height of the virtual door VD, the predetermined position condition of the virtual camera VC and the virtual door VD is satisfied. In other words, when the virtual camera VC (camera 13) approaches the position of the virtual door VD and the virtual camera VC does not deviate from the position of the virtual door VD in either the up-down direction or the left-right direction, the predetermined position condition is satisfied.

[0092] It should be noted that even if the virtual camera VC approaches the second virtual space VS2 from a position other than the virtual door VD, it cannot enter the second virtual space VS2. That is, when the camera 13 is moved within a predetermined range in the real space through the position of the virtual door VD, the virtual camera VC enters the second virtual space VS2 and an image in the second virtual space VS2 is displayed. However, when the camera 13 is moved within a predetermined range in the real space through a position other than the virtual door VD, the virtual camera VC does not enter the second virtual space VS2. In this case, a real image from the camera 13 is displayed.

[0093] When the virtual camera VC reaches the position of the virtual door VD and further moves in the imaging direction (camera 13), the virtual camera VC moves to a position P2 in the second virtual space VS2. Fig. 20 is a diagram showing an example of an image displayed when the virtual camera VC is placed at the position P2 shown in Fig. 19.

[0094] As shown in FIG. 20, when the virtual camera VC enters the second virtual space VS2, the real image from the camera 13 is not displayed, and the virtual image in the second virtual space VS2 based on the virtual camera VC is displayed on the entire screen. In the example shown in FIG. 20, the virtual table VT and the virtual character C1 are displayed. If the camera 13 is further moved in this state, the virtual camera VC moves in the second virtual space VS2. The movement of the camera 13 is detected based on the real images successively acquired from the camera 13. That is, when the virtual camera VC enters the second virtual space VS2, the real image is not displayed on the screen, but the real images are successively acquired from the camera 13, and the movement of the camera 13 (smartphone 1) is detected based on the acquired real images. In addition, when the imaging direction of the camera 13 is changed, the imaging direction of the virtual camera VC also changes in the second virtual space VS2 according to the change.

[0095] As described above, in this embodiment, the virtual camera VC is operated in the virtual space in response to the movement of the camera 13, and the virtual door VD in the virtual space is displayed superimposed on the real image from the camera 13, so that the virtual door VD can be displayed as if it exists in the real space. By performing a predetermined user operation, the user can fix the virtual door VD in the virtual space and pseudo-fix the virtual door VD in the real space. In addition, the user can enter the second virtual space VS2 using the virtual door VD as an entrance and visually recognize the inside of the second virtual space VS2.

[0096] Before a specified user operation is performed, the virtual door VD can be moved within the virtual space by moving the camera 13 or changing the imaging direction of the camera 13, allowing the user to place the entrance to the second virtual space VS2 at the desired position in the real space.

[0097] Here, for example, if the virtual door VD is always placed at a predetermined position in the imaging direction of the camera 13 (virtual camera VC), if the camera 13 is pointed too far upward, the virtual door VD will be located at a high position in the real space.

[0098] FIG. 21 is a diagram showing a real space viewed from the side when the virtual door VD is moved in conjunction with the imaging direction of the virtual camera VC without any restrictions.

[0099] 21, when θ=0, the virtual door VD is placed at a predetermined position in the imaging direction of the virtual camera VC, and when the user looks at the display device 15, the bottom surface of the virtual door VD appears to be located at a height H1 in the real space. When the virtual door VD is fixed at that position, the second virtual space VS2 is set so that the virtual floor VF of the second virtual space VS2 is located at a height H1. When the user brings the camera 13 close to the virtual door VD fixed in the real space in a pseudo manner, the user can enter the virtual camera VC into the second virtual space VS2, and the user can look around the second virtual space VS2 by moving the camera 13.

[0100] On the other hand, when θ is greater than the first threshold a, if the upward movement of the virtual door VD is not restricted, the position of the bottom surface of the virtual door VD is at a height H5. In this case, the user is in a state of looking up at the real space. If the virtual door VD is fixed in this state, the virtual floor VF of the second virtual space VS2 is fixed at a position in the upward direction of the real space. If the virtual floor VF of the second virtual space VS2 is fixed at a position that the user cannot reach, it becomes difficult for the user to bring the camera 13 (smartphone 1) close to the virtual door VD, and it becomes difficult for the user to enter the virtual camera VC into the second virtual space VS2. Even if the camera 13 can pass through the virtual door VD, it is difficult to move the camera 13 so as to look over the second virtual space VS2 at a high position.

[0101] Therefore, in this embodiment, when θ is greater than the first threshold a, that is, when the imaging direction of the camera 13 faces upward more than the first threshold a, the upward movement of the virtual door VD is restricted. This makes it possible to prevent the virtual door VD from being fixed at a position that is too high in the real space, and makes it easier to enter the virtual camera VC into the second virtual space VS2 and view the second virtual space VS2.

[0102] The same applies when the virtual door VD moves downward without restriction. If the camera 13 is facing too far downward and the downward movement of the virtual door VD is not restricted, the bottom surface of the virtual door VD may be located below the floor of the real space. If the virtual door VD is fixed in this state, the virtual floor VF of the second virtual space VS2 will be fixed at a position below the floor of the real space. This makes it impossible for the user to move the camera 13 (smartphone 1) close to the position of the virtual door VD, and therefore impossible to enter the second virtual space VS2.

[0103] Therefore, in this embodiment, when θ is smaller than the second threshold value −b, that is, when the imaging direction of the camera 13 faces downward more than the second threshold value b, the downward movement of the virtual door VD is restricted. This makes it possible to prevent the virtual door VD from being positioned lower than the floor of the real space, and to enable entry into the second virtual space VS2.

[0104] (Data stored on smartphone) Next, a description will be given of data stored in the smartphone 1. Fig. 22 is a diagram showing an example of data stored in the memory 11 of the smartphone 1.

[0105] 22, a program for performing information processing shown in a flowchart described later is stored in the smartphone 1. In addition, the smartphone 1 stores real image data, posture data, virtual camera data, virtual image data, composite image data, virtual door data, virtual character data, and other object data.

[0106] The real image data is data representing a real image captured by the camera 13. The processor 10 acquires real images from the camera 13 at predetermined time intervals (for example, 1 / 60 second intervals) and stores the real image data representing the real images in the memory 11.

[0107] The attitude data is data representing the attitude of the smartphone 1 in real space, and is data representing the attitude of the camera 13. The attitude data is calculated based on the output (e.g., acceleration value and / or angular velocity value) from the attitude detection sensor 14. For example, the attitude data may include data (angle θ) regarding the up-down rotation of the imaging direction of the camera 13 and data (angle φ) regarding the left-right rotation.

[0108] The virtual camera data is data relating to the virtual camera VC, and includes data representing the attitude (imaging direction) of the virtual camera VC and data representing the position of the virtual camera VC in the virtual space.

[0109] The virtual image data is data representing a virtual image captured by the virtual camera VC. The virtual image is repeatedly generated at a predetermined time interval (for example, every 1 / 60 seconds).

[0110] The composite image data is data representing a composite image obtained by combining a real image and a virtual image. The composite image is generated by superimposing a virtual image on a real image. The composite image is repeatedly generated at a predetermined time interval (for example, every 1 / 60 seconds).

[0111] The virtual door data is data related to the virtual door VD. The virtual door data includes data representing the posture and position of the virtual door VD.

[0112] The virtual character data is data related to the virtual characters C1 and C2 arranged in the second virtual space VS2. The virtual character data includes data representing the position and posture of each virtual character. The other object data is data related to another virtual object (e.g., a virtual table VT) arranged in the second virtual space VS2, and includes data representing the position and posture of the object.

[0113] (Details of processing on smartphones) Next, a detailed description will be given of the processing performed in the smartphone 1. Fig. 23 is a flowchart showing an example of the main processing executed in the processor 10 of the smartphone 1. Note that the main processing shown in Fig. 23 is repeatedly executed at predetermined time intervals (for example, 1 / 60 second intervals).

[0114] 23, the processor 10 acquires a real image from the camera 13 (step S101). The acquired real image is stored in the memory 11 as real image data.

[0115] Next, the processor 10 acquires the attitude of the camera 13 (smartphone 1) (step S102). For example, the processor 10 calculates the attitude of the camera 13 (smartphone 1) based on the acceleration value and / or the angular velocity value detected by the attitude detection sensor 14, and stores the attitude in the memory 11 as attitude data.

[0116] Next, the processor 10 sets a virtual camera VC in the virtual space (step S103). Specifically, the processor 10 sets the attitude of the camera 13 acquired in step S102 as the attitude of the virtual camera VC. Note that the attitude of the virtual camera VC only needs to follow the attitude of the camera 13, and may be set to approach the attitude of the camera 13 over a predetermined time.

[0117] Next, the processor 10 judges whether the virtual door VD is fixed or not (step S104). In step S104, it is judged whether the virtual door VD is fixed in the virtual space (i.e., whether the virtual door VD is pseudo-fixed in the real space or not). If the process of step S205 described later has already been performed, it is judged as YES in step S104.

[0118] If the virtual door VD is not fixed (step S104: NO), the processor 10 performs a virtual door fixing process (step S105). The process of step S105 is a process before the virtual door VD is fixed in the virtual space, and includes a process for fixing the virtual door VD. The virtual door fixing process will be described in detail below.

[0119] (Virtual door fixing process) Fig. 24 is a flowchart showing details of the virtual door fixing process in step S105. As shown in Fig. 24, the processor 10 determines whether θ is greater than the first threshold value "a" or whether θ is less than the second threshold value "-b" (step S201). The process in step S201 is a process for determining whether the imaging direction of the camera 13 faces upward more than the first threshold value "a" or whether the imaging direction of the camera 13 faces downward more than the second threshold value "b".

[0120] If the result of the determination in step S201 is NO, the processor 10 places the virtual door VD so as to follow the camera 13 (step S202). For example, the processor 10 places the virtual door VD at a position a certain distance D away from the position of the virtual camera VC in the imaging direction. As a result, when the camera 13 is directed upward, the virtual door VD moves upward, and when the camera 13 is directed downward, the virtual door VD moves downward. Also, when the camera 13 is directed left and right, the virtual door VD moves left and right. Also, the processor 10 places the virtual door VD so that the upward yd axis of the virtual door VD faces in the opposite direction to the gravity direction. Note that in step S202, the virtual door VD may be placed so that the virtual door VD moves over a predetermined time to a position a certain distance D away from the position of the virtual camera VC in the imaging direction.

[0121] On the other hand, if the result of the determination in step S201 is YES, the processor 10 places the virtual door VD by restricting its position in the up-down direction (step S203). For example, if θ>a, that is, if the imaging direction of the camera 13 faces upward more than the first threshold a, the processor 10 calculates the position of the virtual camera VC as a position that is a certain distance D away from the position of the virtual camera VC in the imaging direction when it is assumed that θ=a. Also, if θ<-b, that is, if the imaging direction of the camera 13 faces downward more than the second threshold b, the processor 10 calculates the position of the virtual camera VC as a position that is a certain distance D away from the position of the virtual camera VC in the imaging direction when it is assumed that θ=-b. Note that in step S203, the virtual door VD may be placed so that the virtual door VD moves to the position calculated in this way over a predetermined time.

[0122] When the process of step S202 or step S203 is executed, the processor 10 determines whether or not a predetermined user operation for fixing the virtual door VD has been performed (step S204).

[0123] When a predetermined user operation is performed (step S204: YES), the processor 10 fixes the virtual door VD in a pseudo manner in the real space (step S205). The real space and the virtual space are associated with each other, and the virtual door VD is fixed in the virtual space, so that the virtual door VD is fixed in the real space in a pseudo manner. For example, when a fixed vx-vy-vz coordinate system is set in the virtual space in correspondence with the real space, the position of the virtual door VD is expressed as a coordinate value of the vx-vy-vz coordinate system. The coordinate value of the virtual door VD at the time when the predetermined user operation is performed is stored, and the coordinate value is not updated thereafter, so that the virtual door VD is fixed in the virtual space. In addition, the position of the virtual door VD at the time when the predetermined user operation is performed may be set as the origin of the vx-vy-vz coordinate system. At that time, the coordinate value of the vx-vy-vz coordinate system of the virtual camera VC is determined. Thereafter, if the camera 13 moves, the coordinate values ​​of the vx-vy-vz coordinate system of the virtual camera VC also change in accordance with the movement, but the origin of the vx-vy-vz coordinate system (i.e., the position of the virtual door VD) does not change.

[0124] In step S205, the second virtual space VS2 is set based on the position of the virtual door VD. Specifically, the virtual floor VF is set so that the bottom surface of the virtual door VD and the virtual floor VF of the second virtual space VS2 are aligned in the height direction. The virtual floor VF is set so as to be perpendicular to the virtual door VD (so that the virtual floor VF is perpendicular to gravity). In addition, virtual characters C1 and C2 and a virtual table VT are placed in the second virtual space VS2. Even if the second virtual space VS2 is set in step S205, the second virtual space VS2 is set to be transparent. For this reason, the user cannot view the second virtual space VS2.

[0125] Next, the processor 10 changes the display mode of the virtual door VD from the first display mode to the second display mode (step S206). For example, the display mode of the virtual door VD may be changed from a dashed line mode to a solid line mode. In addition, in step S206, the color of the virtual door VD may be changed, or the virtual door VD may be changed from a semi-transparent mode to an opaque mode.

[0126] If the result of the determination in step S204 is NO, or if the process of step S206 has been performed, the processor 10 ends the process shown in FIG. 24, and returns the process to FIG.

[0127] 23, if the virtual door VD is fixed (step S104: YES), the processor 10 calculates the position of the virtual camera VC (step S106). Specifically, the processor 10 calculates the position of the virtual camera VC in the virtual space based on the real image from the camera 13. The processor 10 calculates the movement direction and movement amount of the camera 13 (smartphone 1) based on the real image acquired from the camera 13, calculates the position of the virtual camera VC according to the calculated movement direction and movement amount, and stores it as virtual camera data.

[0128] Next, the processor 10 performs a virtual door opening / closing process (step S107). Specifically, the processor 10 determines whether or not the user has performed an opening / closing operation of the virtual door VD based on an input from the input device 12. For example, the processor 10 determines whether or not the virtual door VD has been touched. For example, when the virtual door VD is in a closed state and an opening / closing operation of the virtual door VD is performed, the processor 10 changes the virtual door VD to an open state. Also, when the virtual door VD is in an open state and an opening / closing operation of the virtual door VD is performed, the processor 10 changes the virtual door VD to a closed state. When the virtual door VD is in an open state, the second virtual space VS2 corresponding to the opening of the virtual door VD becomes opaque, and the second virtual space VS2 becomes visible through the opening.

[0129] Next, the processor 10 judges whether or not the position condition of the virtual camera VC and the virtual door VD is satisfied. Specifically, when the distance in the virtual space between the virtual camera VC and the virtual door VD is less than a predetermined value, the position of the virtual camera VC in the left-right direction is within the range of the width of the virtual door VD, and the position of the virtual camera VC in the up-down direction is within the range of the height of the virtual door VD, the processor 10 judges that the position condition is satisfied.

[0130] If the position condition is satisfied (step S108: YES), the processor 10 performs an entry process into the second virtual space (step S109). Specifically, the processor 10 sets a flag indicating that the virtual camera VC exists in the second virtual space VS2 to ON. The virtual camera VC is placed in the second virtual space VS2. Various virtual objects in the second virtual space VS2 (such as the virtual floor VF, the virtual wall VW, the virtual ceiling, the virtual characters C1 and C2, and the virtual table VT) are set to be opaque. As a result, a virtual image of the second virtual space VS2 viewed from the virtual camera VC is displayed on the display device 15 of the smartphone 1. The virtual door VD is a virtual object that connects the second virtual space VS2 and the outside virtual space VS, and is visible from within the second virtual space VS2. After the virtual camera VC enters the second virtual space VS2, the virtual camera VC remains in the second virtual space VS2 until the virtual camera VC passes through the virtual door VD and goes outside the second virtual space VS2. Furthermore, when the process of entering the second virtual space is performed, a virtual boundary is set between the inside and outside of the second virtual space VS2. The second virtual space VS2 is associated with a predetermined range of the real space, but after the virtual camera VC enters the second virtual space VS2, even if the camera 13 goes out of the predetermined range of the real space from a position other than the virtual door VD, the virtual camera VC does not go out of the second virtual space VS2 and remains at the virtual boundary. At this time, a virtual image (for example, an entirely black image) capturing the virtual boundary is displayed on the display device 15. Before the virtual camera VC enters the second virtual space VS2, even if the camera 13 enters within the predetermined range of the real space from a position other than the virtual door VD, a composite image obtained by combining the real image from the camera 13 and the virtual image from the virtual camera VC continues to be displayed. That is, after it is determined that the virtual camera VC has entered the second virtual space VS2, a process different from that before it is determined that the virtual camera VC has entered the second virtual space VS2 is performed.

[0131] When the process of step S109 has been performed, or when the result of step S108 is NO, the processor 10 determines whether or not the virtual camera VC exists in the second virtual space (step S110).

[0132] When the virtual camera VC exists in the second virtual space (step S110: YES), the processor 10 performs processing in the second virtual space (step S111). The processing in step S111 is processing when the virtual camera VC exists in the second virtual space VS2, and is processing to control the virtual camera VC in the second virtual space VS2 based on the position of the virtual camera VC calculated in step S106. For example, when the virtual camera VC approaches the virtual door VD in the second virtual space VS2 (that is, when the virtual camera VC and the virtual door VD satisfy the above-mentioned predetermined position condition), an exit processing is performed to move the virtual camera VC out of the second virtual space VS2. In the exit processing, a flag indicating that the virtual camera VC exists in the second virtual space VS2 is set to OFF. When the exit processing is performed, the virtual camera VC goes out of the second virtual space VS2. When the virtual camera VC goes out of the second virtual space VS2, the virtual camera VC returns to the state before entering the second virtual space VC2. As a result, a composite image obtained by combining the real image from the camera 13 and the virtual image of the virtual space VS viewed from the virtual camera VC is displayed on the display device 15 of the smartphone 1. On the other hand, when the virtual camera VC moves to the boundary of the second virtual space VS2 other than the virtual door VD (when the camera 13 moves to the boundary of the above-mentioned predetermined range of the real space), the movement of the virtual camera VC is restricted. Specifically, even if the camera 13 moves beyond the above-mentioned predetermined range of the real space, the virtual camera VC does not go outside the second virtual space VS2, but is kept at the virtual boundary set between the inside and outside of the second virtual space VS2.

[0133] When the process of step S111 is performed, when the result of the determination in step S110 is NO, or when the process of step S105 is performed, the processor 10 performs an image output process (step S112). Specifically, when the virtual camera VC is present outside the second virtual space VS2 (i.e., in the virtual space VS), the processor 10 generates a composite image by superimposing a virtual image captured by the virtual camera VC on an actual image captured by the camera 13. Then, the processor 10 outputs the generated composite image to the display device 15. When the virtual camera VC is present within the second virtual space VS2, the processor 10 generates a virtual image of the second virtual space VS2 as viewed from the virtual camera VC, and outputs the virtual image to the display device 15.

[0134] After the virtual door VD is fixed in step S205, if a predetermined release operation is performed by the user, the virtual door VD may be released from the fixed state. In this case, the virtual door VD becomes movable again in the virtual space. The processor 10 then ends the process shown in FIG. 23.

[0135] It should be noted that the processes shown in the above flowcharts are merely examples, and the order and content of the processes, values ​​used in the judgments, etc. may be changed as appropriate.

[0136] As described above, in this embodiment, the virtual camera VC moves in the virtual space and the imaging direction of the virtual camera VC changes in response to the movement of the camera 13 and the change in the imaging direction of the camera 13. The virtual door VD moves in the virtual space in response to the movement of the virtual camera VC and the change in the imaging direction, and the virtual door VD is fixed in response to a predetermined user operation.

[0137] Specifically, before the virtual door VD is fixed, when the imaging direction of the camera 13 is within a predetermined range in the vertical direction (when -b < θ < a), the virtual door VD moves in the vertical direction in conjunction with the imaging direction of the camera 13 (virtual camera VC). Also, when the camera 13 moves parallel in the front-back, left-right, and vertical directions, the virtual door VD also moves parallel in the same direction within the virtual space. Then, in response to a predetermined user operation, the virtual door VD is fixed within the virtual space. On the other hand, when the imaging direction of the camera 13 is facing upward beyond the first threshold value a or downward beyond the second threshold value b, the virtual door VD does not move in the vertical direction in conjunction with the imaging direction of the camera 13.

[0138] Thereby, the virtual door VD can be moved and pseudo-fixed at a desired position in the real space, and it can be prevented that the virtual door VD is fixed at a position that is too high or too low. The user can enter the second virtual space VS2 through the virtual door VD, and it can be made easier for the user to enter the second virtual space VS2.

[0139] Also, the bottom surface of the virtual door VD coincides with the virtual floor VF of the second virtual space VS2, and the second virtual space VS2 is formed continuously with the virtual door VD. That is, the virtual door VD is located at the boundary between the virtual space VS and the second virtual space VS2, and the height of the virtual door VD in the real space coincides with the height of the second virtual space VS2 in the real space. Thereby, the virtual door VD can be configured as an entrance to the second virtual space VS2, and after the user pseudo-fixes the virtual door VD in the real space, the user can enter the second virtual space VS2 from the virtual door VD without a sense of incongruity.

[0140] Also, by moving the camera 13, the virtual camera VC is brought closer to the virtual door VD, and when the virtual camera VC reaches the position of the virtual door VD, the virtual camera VC enters the second virtual space VS2. Thereby, it is possible to enter the second virtual space VS2 through the virtual door VD, and it is possible to give the user a feeling of entering the second virtual space VS2 from the virtual door VD existing in the real space.

[0141] In addition, a virtual door VD is placed in the virtual space as a three-dimensional virtual object, and the virtual door VD is moved in the virtual space while maintaining a vertical posture relative to the horizontal plane of the virtual space. That is, the virtual door VD moves in the virtual space while maintaining a vertical standing posture. This allows the virtual floor VF to be horizontal without the second virtual space VS2 becoming slanted when the virtual door VD is fixed.

[0142] In this embodiment, the virtual door VD is placed at a predetermined position in the imaging direction of the virtual camera VC. This allows the virtual door VD to be placed in the virtual space without detecting a predetermined surface in the real space (a horizontal surface such as a floor surface or a table surface, or a surface such as a marker). Even if the camera 13 moves or the imaging direction changes, the position of the virtual door VD can be controlled in conjunction with the camera 13, and a virtual image of the virtual door VD captured by the virtual camera VC can be displayed.

[0143] (Modification) Although the present embodiment has been described above, the above embodiment is merely an example, and the following modifications may be made, for example.

[0144] For example, in the above embodiment, when -b < θ < a, that is, when the angle θ of the imaging direction of the camera 13 with respect to the horizontal plane is greater than -b and less than a, these are interlocked so that the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, coincide. Specifically, in accordance with the imaging direction of the camera 13, the imaging direction of the virtual camera VC is set and the virtual door VD is moved in the vertical direction. In another embodiment, when -b < θ < a, the imaging direction of the virtual camera VC and the vertical position of the virtual door VD may be set in accordance with the imaging direction of the camera 13 so that the imaging direction of the virtual camera VC and the virtual object direction are interlocked with a first interlocking degree. Here, the two directions being "interlocked with a first interlocking degree" includes the case where the two directions always completely coincide. In this case, when the imaging direction of the virtual camera VC changes, the virtual object direction changes by the same amount as the change amount. Also, the two directions being "interlocked with a first interlocking degree" includes the case where the two directions do not always completely coincide but coincide over a predetermined time. Also, the two directions being "interlocked with a first interlocking degree" may include the case where the two directions do not completely coincide. Also, the two directions being "interlocked with a first interlocking degree" includes the case where the imaging direction of the virtual camera VC and the virtual object direction do not change at all.

[0145] Also, in the above embodiment, when θ < -b or θ > a, the virtual door VD is not moved in accordance with the change in the imaging direction of the camera 13. In other embodiments, when θ < -b or θ > a, the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, are interlocked with a second interlock degree smaller than the first interlock degree. Accordingly, the imaging direction of the virtual camera VC and the vertical position of the virtual door VD may be set in accordance with the imaging direction of the camera 13. For example, even when θ < -b or θ > a, the virtual door VD may be moved. Here, "the imaging direction of the virtual camera VC and the virtual object direction are interlocked with a second interlock degree smaller than the first interlock degree" means that when the imaging direction of the camera 13 changes as compared with the case where -b < θ < a, the virtual door VD viewed from the display device 15 seems not to move at all in the real space, the moving range of the virtual door VD becomes smaller, the moving speed of the virtual door VD becomes smaller, and the like. Here, "the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, are interlocked with a second interlock degree smaller than the first interlock degree" means that when the imaging direction of the virtual camera VC changes, the virtual object direction does not change at all, that is, even when the imaging direction of the virtual camera VC changes, the position of the virtual door VD does not change. Also, "the imaging direction of the virtual camera VC and the virtual object direction are interlocked with a second interlock degree smaller than the first interlock degree" means that when the imaging direction of the virtual camera VC changes, the degree of change in the virtual object direction becomes smaller (the moving amount of the virtual door VD becomes smaller) as compared with the case where -b < θ < a. For example, when θ > a, if the camera 13 changes by 10 degrees upward, the imaging direction of the virtual camera VC also changes by 10 degrees upward, but the virtual object direction may change only by an angle smaller than 10 degrees (for example, 5 degrees) (in this case, the virtual door VD moves upward by a distance corresponding to "5 degrees").

[0146] Further, in the above-described embodiment, on the premise that the imaging direction of the camera 13 coincides with the imaging direction of the virtual camera VC, the position of the virtual door VD was controlled in the virtual space according to the angle θ indicating the imaging direction of the camera 13 (virtual camera VC). In another embodiment, when the angle θ indicating the imaging direction of the camera 13 satisfies -b < θ < a, the imaging direction of the virtual camera VC and the position of the virtual door VD may not be changed according to the imaging direction of the camera 13. Further, when θ < -b or θ > a, only one of the imaging direction of the virtual camera VC and the position of the virtual door VD may be changed according to the imaging direction of the camera 13. By controlling in this way, when the camera 13 is not pointing too much upward or downward (-b < θ < a), the imaging direction of the virtual camera VC coincides with the virtual object direction which is the direction from the virtual camera VC to the virtual door VD, and it seems that the virtual door VD moves vertically following the vertical change of the imaging direction of the camera 13. On the other hand, when the camera 13 is pointing too much upward or downward (θ < -b or θ > a), the imaging direction of the virtual camera VC does not coincide with the virtual object direction, and it seems that the virtual door VD does not move vertically without following the vertical change of the imaging direction of the camera 13.

[0147] Also, in the above embodiment, the posture of the virtual camera VC was set so that the posture of the camera 13 and the posture of the virtual camera VC would coincide. When -b < θ < a, in accordance with the imaging direction of the camera 13, the imaging direction of the virtual camera VC was set, and by moving the virtual door VD, the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, were made to be interlocked. In another embodiment, when -b < θ < a, when the imaging direction of the camera 13 changes, without moving the virtual door VD, the imaging direction of the virtual camera VC may be changed and the position of the virtual camera VC may be changed. In this case, the relative positional relationship between the virtual camera VC and the virtual door VD does not change, and the imaging direction of the virtual camera VC and the virtual object direction do not change. Therefore, when the imaging direction of the camera 13 changes, it appears that the virtual door VD moves following the movement of the camera 13. Also, when -b < θ < a, when the imaging direction of the camera 13 changes, both the imaging direction of the virtual camera VC and the position of the virtual door VD may be changed. That is, when -b < θ < a, when the imaging direction of the camera 13 changes, at least either the imaging direction of the virtual camera VC or the position of the virtual door VD may be controlled so that the imaging direction of the virtual camera VC and the virtual object direction, which is the direction from the virtual camera VC to the virtual door VD, are interlocked.

[0148] Also, in the above embodiment, when the virtual door VD is not pseudo-fixed, in accordance with the movement of the camera 13 in the real space, the virtual camera VC was moved within the virtual space and the virtual door VD was moved within the virtual space. Thereby, when the virtual door VD is not pseudo-fixed, the virtual camera VC and the virtual door VD were made to maintain a certain distance D. In another embodiment, when the virtual door VD is not pseudo-fixed, even if the camera 13 moves, the virtual camera VC and the virtual door VD may not be moved within the virtual space and the distance between the virtual camera VC and the virtual door VD may be kept constant.

[0149] In the above embodiment, the virtual door VD is fixed in the virtual space on the premise that the virtual camera VC is moved in the virtual space according to the movement of the camera 13 in the real space, so that the virtual door VD is fixed in the real space. In another embodiment, when the camera 13 moves, the virtual door VD may be moved in the real space while keeping the virtual camera VC fixed in the virtual space. For example, when the camera 13 moves leftward, the virtual door VD is moved rightward in the virtual space by the same amount of movement, so that the virtual camera VC moves relatively leftward with the virtual door VD as a reference. At this time, the virtual door VD in the composite image displayed on the display device 15 moves rightward in the screen. In this way, even if the virtual door VD is moved in the virtual space when the camera 13 moves, the user can recognize that the virtual door VD is fixed at a predetermined position in the real space. Also, when the camera 13 moves, both the virtual camera VC and the virtual door VD may be moved. Also, when the imaging direction of the camera 13 changes, the virtual door VD may be moved so that the virtual door VD appears fixed in the real space. For example, when the camera 13 is directed to the right from the front, the virtual door VD may be moved to the left so that the virtual door VD moves to the left as seen from the virtual camera VC in the virtual space. Also, when the camera 13 is directed to the right from the front, the virtual camera VC may be directed to the right while keeping the virtual door VD fixed so that the virtual door VD moves to the left as seen from the virtual camera VC in the virtual space. In this case, the virtual door VD displayed on the display device 15 moves to the left in response to a change in the imaging direction of the camera 13 to the right, so that the virtual door VD appears fixed in the real space. That is, at least one of the position of the virtual camera, the imaging direction, and the position of the virtual door VD may be controlled in response to the position or imaging direction of the camera 13 so that the virtual door VD viewed from the composite image displayed on the display device 15 is pseudo-fixed in the real space.

[0150] Note that even after the virtual door VD is virtually fixed in the real space, the virtual door VD may move slightly in the virtual space. For example, even after the virtual door VD is virtually fixed in the real space, the virtual door VD may be moved slightly in the virtual space in response to a slight movement of the camera 13 in order to correct the shaking of the camera 13.

[0151] In the above embodiment, the above restriction (restriction on vertical movement of the virtual door VD) is performed both when the imaging direction of the camera 13 faces upwards beyond the first threshold and when the imaging direction of the camera 13 faces downwards beyond the second threshold. In other embodiments, the above restriction may be performed at least in either case of when the imaging direction of the camera 13 faces upwards beyond the first threshold and when the imaging direction of the camera 13 faces downwards beyond the second threshold.

[0152] In another embodiment, any virtual object other than the virtual door VD may be placed in the virtual space, and the same processing as that for the virtual door VD described above may be performed on the virtual object.

[0153] That is, the information processing system generates a composite image by combining real images sequentially captured by a camera of a real space and virtual images sequentially captured by a virtual camera arranged in a virtual space of a virtual object, and outputs the composite image to a display device. In the first control state, at least one of the imaging direction of the virtual camera and the position of the virtual object may be controlled according to the imaging direction of the camera so that the imaging direction of the virtual camera and the virtual object direction, which is the direction from the virtual camera to the virtual object, are linked with a first degree of linkage. In addition, the first control state may be changed to a second control state based on a user operation, and at least one of the position of the virtual camera, the imaging direction, and the position of the virtual object may be controlled according to the position or imaging direction of the camera so that the virtual object viewed from the composite image is pseudo-fixed in the real space in the second control state. When the imaging direction of the camera is directed upwardly relative to the first threshold value or downwardly relative to the second threshold value, at least one of the virtual camera and the virtual object may be controlled so that the imaging direction of the virtual camera and the virtual object direction are linked with a second degree of linkage that is smaller than the first degree of linkage.

[0154] In the above embodiment, when the virtual door VD is fixed and the virtual camera VC moves to the position of the virtual door VD, the entry process is performed. In other embodiments, the entry process may be performed when a predetermined position condition regarding the position of the virtual camera VC and the position of the virtual object is satisfied.

[0155] In the above embodiment, the entry process is a process of determining that the virtual camera VC has entered the second virtual space VS2 through the virtual door VD, and includes a process of setting a flag indicating that the virtual camera VC is in the second virtual space VS2 to ON. In other embodiments, when a predetermined position condition regarding the position of the virtual camera VC and the position of the virtual object is satisfied, not only the above-mentioned entry process but also an event process regarding the virtual object may be performed. The "event process regarding the virtual object" may include a process of setting the flag to ON, a process of making the virtual camera VC enter the second virtual space VS2 to display the inside of the second virtual space VS2, a process of setting a virtual boundary between the inside and outside of the second virtual space VS2, a process of generating the second virtual space VS2 based on the virtual object, a process of restricting the virtual camera VC from going outside the second virtual space VS2 from a position other than the virtual object, and the like. The "event process regarding the virtual object" may also be a process of changing the display mode of the virtual object, a process of making the virtual object and another object have a predetermined positional relationship (for example, a process of making the other object come into contact with the virtual object), and the like.

[0156] In the above embodiment, the second virtual space VS2 is a small room that can be entered through the virtual door VD, and the virtual character exists in the second virtual space. In other embodiments, the second virtual space VS2 may be any space, for example, a virtual space representing a specific place on the earth or outer space.

[0157] In the above embodiment, the attitude of the camera 13 (smartphone 1) is detected using an acceleration sensor and / or an angular velocity sensor, and the position of the camera 13 is detected based on the actual image from the camera 13. Then, the position and attitude of the virtual camera VC are set according to the detected position and attitude of the camera 13. In other embodiments, the position and attitude of the camera 13 (smartphone 1) may be detected by another method. For example, the position of the camera 13 may be detected using a global positioning system (GPS), and the attitude (imaging direction) of the camera 13 may be detected using a geomagnetic sensor. Also, the position and attitude of the camera 13 may be detected based on the actual image captured by the camera 13. For example, a marker may be detected from the actual image captured by the camera 13, and the position and attitude of the camera 13 may be detected based on the detection result of the marker. Also, the position and attitude of the smartphone 1 may be detected by capturing an image of the smartphone 1 with an external camera, not limited to the camera 13 fixed to the smartphone 1.

[0158] In the above embodiment, a composite image obtained by combining real images sequentially captured by the camera 13 of the real space and virtual images sequentially captured by the virtual camera VC of the virtual space is displayed on the display device 15. In other embodiments, the display device may be a video see-through display device. The user visually recognizes the real space through a transmissive display device and also visually recognizes the virtual image displayed on the transmissive display device. In this manner, the transmissive display device may be used to allow the user to visually recognize a composite image obtained by combining the real space and the virtual image.

[0159] In the above embodiment, the smartphone 1 is used in which the camera, the display device, and the processor that performs the above-mentioned processing are integrated. The smartphone 1 is merely one example of an information processing system. For example, an information processing system in which the camera, the display device, and the processor are each separate may be used. The information processing system may be configured by a device in which the camera and the processor are integrated, and a display device separate from the device. The information processing system may be configured by a device in which the camera and the display device are integrated, and a device having a processor separate from the device. The information processing system may be configured by connecting the camera, the display device, and the processor via a network (such as a LAN or the Internet).

[0160] In addition, the configurations according to the above-described embodiments and their modifications can be combined in any manner as long as they are not inconsistent with each other. Furthermore, the above is merely an example of the present invention, and various improvements and modifications other than those described above may be made. [Explanation of symbols]

[0161] 1. Smartphone 10 Processors 13. Camera 14 Attitude detection sensor 15 Display device VC Virtual Camera VD Virtual Door

Claims

1. An information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the composite image comprising: a control means for controlling at least one of an imaging direction of the virtual camera and a position of the virtual object in accordance with an imaging direction of the imaging device so that the imaging direction of the virtual camera and a virtual object direction, which is a direction from the virtual camera to the virtual object, are linked with a first degree of linkage; When the imaging direction of the imaging device is directed above a first threshold value or below a second threshold value in the real space, the control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object so that the imaging direction of the virtual camera and the virtual object direction are linked with a second degree of linkage that is smaller than the first degree of linkage.

2. The information processing system according to claim 1 , further comprising an event processing execution means for executing an event process related to the virtual object when a position condition related to a position of the virtual camera and a position of the virtual object is satisfied.

3. the position condition is satisfied when the virtual camera approaches the virtual object within a predetermined distance; A second virtual space is set based on the position of the virtual object; the event processing is processing for determining that the virtual camera has entered the second virtual space, The information processing system according to claim 2 , wherein, when the virtual camera enters the second virtual space, a position or an imaging direction of the virtual camera in the second virtual space is controlled in accordance with a position or an imaging direction of the imaging device.

4. The information processing system according to claim 1 , wherein the control means, when an imaging direction of the imaging device is changed in an up-down direction, moves the virtual object in an up-down direction while maintaining a vertical orientation of the virtual object in the virtual space.

5. 5. The information processing system according to claim 1, wherein the control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object in accordance with a change in the left-right direction of the imaging direction of the imaging device so that the imaging direction of the virtual camera and the virtual object direction are linked with the first degree of linkage even when the imaging direction of the imaging device is directed above the first threshold value or below the second threshold value.

6. 6. The information processing system according to claim 1, wherein the control means controls a position of the virtual camera and a position of the virtual object so that the virtual camera and the virtual object are kept at a constant distance when the imaging device is moved.

7. 7. The information processing system according to claim 1, wherein the control means sets an imaging direction of the virtual camera in accordance with an imaging direction of the imaging device, and controls a position of the virtual object so as to follow a change in the imaging direction of the virtual camera.

8. 8. The information processing system according to claim 1, wherein the control means places the virtual object at a predetermined position in an imaging direction of the virtual camera without detecting a predetermined surface in the real space based on an actual image captured by the imaging device.

9. An information processing program executed by a computer of an information processing device that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the information processing program comprising: a control unit that controls at least one of an imaging direction of the virtual camera and a position of the virtual object in accordance with an imaging direction of the imaging device so that the imaging direction of the virtual camera and a virtual object direction, which is a direction from the virtual camera to the virtual object, are linked with a first degree of linkage; The control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object so that the imaging direction of the virtual camera and the virtual object direction are linked with a second degree of linkage smaller than the first degree of linkage when the imaging direction of the imaging device is directed upward than a first threshold value or downward than a second threshold value in the real space.

10. An information processing device that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the composite image being obtained by combining real images sequentially captured by an imaging device of a real space and a virtual camera disposed in a virtual space, the a control means for controlling at least one of an imaging direction of the virtual camera and a position of the virtual object in accordance with an imaging direction of the imaging device so that the imaging direction of the virtual camera and a virtual object direction, which is a direction from the virtual camera to the virtual object, are linked with a first degree of linkage; The control means controls at least one of the imaging direction of the virtual camera and the position of the virtual object so that the imaging direction of the virtual camera and the virtual object direction are linked with a second degree of linkage smaller than the first degree of linkage when the imaging direction of the imaging device is directed upward than a first threshold value or downward than a second threshold value in the real space.

11. 1. An information processing method executed in an information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the method comprising: a control step of controlling at least one of an imaging direction of the virtual camera and a position of the virtual object in accordance with an imaging direction of the imaging device so that an imaging direction of the virtual camera and a virtual object direction, which is a direction from the virtual camera to the virtual object, are linked with a first degree of linkage, In the control step, when the imaging direction of the imaging device is directed above a first threshold value or below a second threshold value in the real space, at least one of the imaging direction of the virtual camera and the position of the virtual object is controlled so that the imaging direction of the virtual camera and the virtual object direction are linked with a second linkage degree smaller than the first linkage degree.

12. An information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the composite image comprising: a virtual camera direction control means for controlling an imaging direction of the virtual camera in accordance with an imaging direction of the imaging device; an object position control means for controlling a position of the virtual object in a manner that follows an imaging direction of the virtual camera; The object position control means controls the position of the virtual object so that a degree to which the virtual object follows the imaging direction of the virtual camera is reduced when the imaging direction of the imaging device is directed upward above a first threshold value or downward above a second threshold value in the real space.

13. An information processing program executed by a computer of an information processing device that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the information processing program comprising: a virtual camera direction control means for controlling an imaging direction of the virtual camera in accordance with an imaging direction of the imaging device; the virtual camera is caused to function as an object position control means for controlling a position of the virtual object in a manner that follows the imaging direction of the virtual camera; The object position control means controls the position of the virtual object so that a degree to which the virtual object follows the imaging direction of the virtual camera is reduced when the imaging direction of the imaging device is directed upward above a first threshold value or downward above a second threshold value in the real space.

14. An information processing device that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the composite image being obtained by combining real images sequentially captured by an imaging device of a real space and a virtual camera disposed in a virtual space, the a virtual camera direction control means for controlling an imaging direction of the virtual camera in accordance with an imaging direction of the imaging device; an object position control means for controlling a position of the virtual object in a manner that follows an imaging direction of the virtual camera; The object position control means controls the position of the virtual object so that a degree to which the virtual object follows the imaging direction of the virtual camera is reduced when the imaging direction of the imaging device is directed upward above a first threshold value or downward above a second threshold value in the real space.

15. 1. An information processing method executed in an information processing system that outputs to a display device a composite image obtained by combining real images sequentially captured by an imaging device of a real space and virtual images sequentially captured by a virtual camera disposed in a virtual space of a virtual object, the method comprising: a virtual camera direction control step of controlling an imaging direction of the virtual camera in accordance with an imaging direction of the imaging device; an object position control step of controlling a position of the virtual object in a manner to follow an imaging direction of the virtual camera, In the object position control step, when the imaging direction of the imaging device is directed above a first threshold value or below a second threshold value in the real space, the position of the virtual object is controlled so that the degree to which the virtual object follows the imaging direction of the virtual camera is reduced.

Images