Information processing device and image generating method

Abstract

To provide a technique for accurately estimating at least one of the position and the posture of a head-mounted display on the basis of a taken image.SOLUTION: An estimation processing unit 222 executes processing of estimating at least one of the position and the posture of a head-mounted display on the basis of an image of regions around the head-mounted display. An image generation unit 230 generates a display image to display in a display device that is not the head-mounted display. The image generation unit 230 generates a display image at least partially including a pattern as a motionless picture.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technique for estimating at least one of the position and orientation of a head-mounted display based on an image captured around the head-mounted display. [Background technology]

[0002] In recent years, users have been wearing a head-mounted display (hereinafter referred to as "HMD") on their heads and operating a game controller to play games while viewing game images displayed on the HMD. By implementing HMD tracking processing and linking the image of the game space displayed on the HMD to the user's head movements, the sense of immersion in the image can be improved, and the entertainment value of the game can be enhanced. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-95045 Summary of the Invention [Problem to be solved by the invention]

[0004] SLAM (Simultaneous Localization And Mapping) is a well-known technology for estimating the position and / or orientation of an HMD. SLAM is a technology that simultaneously estimates self-location and creates an environmental map, and it makes it possible to estimate the position and / or orientation of the HMD by tracking feature points using images captured by a camera mounted on the HMD.

[0005] In feature point tracking processing, if there is a television displaying video near the HMD and the television is included in the image captured by the camera, valid feature points cannot be extracted, and tracking loss may occur. In particular, in game systems that display game images on both the HMD and the television, tracking accuracy may deteriorate when the camera mounted on the HMD captures game images displayed on the television.

[0006] Therefore, an object of the present invention is to provide a technique for estimating at least one of the position and orientation of an HMD with high accuracy in an environment where a display device different from the HMD is present. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing device according to one aspect of the present invention includes an estimation processing unit that performs processing to estimate at least one of the position and the orientation of the head-mounted display based on an image captured around the head-mounted display, and an image generation unit that generates a display image to be displayed on a display device other than the head-mounted display. The image generation unit generates a display image that includes, at least in part, a picture that is a still image.

[0008] Another aspect of the image generation method of the present invention includes the steps of performing a process to estimate at least one of the position or posture of the head-mounted display based on an image captured around the head-mounted display, and generating a display image to be displayed on a display device other than the head-mounted display, the display image including at least a portion of a still image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 2] 1A and 1B are diagrams illustrating examples of the external shape of an HMD. [Figure 3]FIG. 2 is a diagram illustrating functional blocks of an HMD. [Figure 4] FIG. 2 is a diagram illustrating functional blocks of the information processing device. [Figure 5] 10A and 10B are diagrams illustrating examples of display images displayed on a display panel. [Figure 6] 10(a) and 10(b) are diagrams showing examples of images captured by an imaging device. [Figure 7] FIG. 10 is a diagram illustrating an example of an image displayed on an output device. [Figure 8] FIG. 10 is a diagram showing an example of a setting screen for turning on / off a picture display function. [Figure 9] FIG. 10 is a diagram illustrating an example of a setting screen displayed on a display panel. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1 shows an example of the configuration of an information processing system 1 according to an embodiment. The information processing system 1 includes an information processing device 10, a recording device 11, a head-mounted display (HMD) 100, an input device 16 operated by a user's fingers, and an output device 15 that outputs images and sounds. The output device 15 is a display device different from the HMD 100 and may be a stationary television. The information processing device 10 is connected to an external network 2 such as the Internet via an access point (AP) 17. The AP 17 has the functions of a wireless access point and a router, and the information processing device 10 may be connected to the AP 17 by a cable or by a known wireless communication protocol.

[0011] The recording device 11 records applications such as system software and game software. The information processing device 10 may download game software to the recording device 11 from a content server via the network 2. The information processing device 10 executes a game program and supplies game image data and audio data to the HMD 100. The information processing device 10 and HMD 100 may be connected using a known wireless communication protocol, or may be connected by cable.

[0012] The HMD 100 is a display device worn by a user on the head, which displays images on display panels positioned in front of the user's eyes. The HMD 100 separately displays an image for the left eye on the left-eye display panel and an image for the right eye on the right-eye display panel. These images form parallax images seen from left and right viewpoints, realizing stereoscopic vision. Because the user views the display panel through optical lenses, the information processing device 10 supplies the HMD 100 with parallax image data that has been corrected for optical distortion caused by the lenses.

[0013] The output device 15 is not necessary for a user wearing the HMD 100, but providing the output device 15 allows another user to view the image displayed on the output device 15. The information processing device 10 may display on the output device 15 the same image as the image viewed by the user wearing the HMD 100, or may display a different image. For example, when a user wearing an HMD and another user play a game together, the output device 15 may display a game image from the viewpoint of the character of the other user.

[0014] The information processing device 10 and the input device 16 may be connected via a known wireless communication protocol or via a cable. The input device 16 has a plurality of operation members such as operation buttons, and a user operates the operation members with their fingers while holding the input device 16. When the information processing device 10 executes a game program, the input device 16 is used as a game controller.

[0015] The HMD 100 is equipped with a plurality of image capturing devices 14. The plurality of image capturing devices 14 are attached at different positions on the front surface of the HMD 100. The image capturing devices 14 may have a visible light sensor used in general digital video cameras, such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The plurality of image capturing devices 14 capture images of the area in front of the user at a predetermined cycle (for example, 60 frames per second) in a synchronized manner, and transmit the captured images to the information processing device 10.

[0016] The information processing device 10 has a function of estimating at least one of the position and orientation of the HMD 100 based on an image captured around the HMD 100. In the following, the information processing device 10 will be described as having a function of estimating both the position and orientation of the HMD 100, but it is sufficient if the information processing device 10 has a function of estimating at least one of the position and orientation.

[0017] The information processing device 10 estimates the position and orientation of the HMD 100 at any time (t) from the position and orientation of the HMD 100 at the time (t-1) immediately before the time (t) using captured images captured in time series. The information processing device 10 may derive the position of the HMD 100 as position coordinates in a coordinate system defined in real space, and derive the orientation of the HMD 100 as orientation information in the coordinate system defined in real space. Note that the information processing device 10 may further use sensor data detected by an orientation sensor provided in the HMD 100 to derive the position information and orientation information of the HMD 100 with high accuracy.

[0018] 2 shows an example of the external shape of the HMD 100. The HMD 100 is composed of an output mechanism unit 102 and a wearing mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that, when worn by the user, goes around the head and secures the HMD 100 to the head. The wearing band 106 is made of a material or has a structure that allows its length to be adjusted to fit the user's head circumference.

[0019] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes of a user wearing the HMD 100, and includes a display panel inside that faces the eyes when worn. The display panel may be a liquid crystal panel, an organic EL panel, or the like. The housing 108 also includes a pair of optical lenses, one on each side, that are positioned between the display panel and the user's eyes and expand the user's field of view. The HMD 100 may also include speakers or earphones at positions corresponding to the user's ears, and may be configured to allow external headphones to be connected.

[0020] A plurality of image capturing devices 14a, 14b, 14c, and 14d are provided on the front outer surface of the housing 108. With respect to the front direction of the housing 108 as a reference, image capturing device 14a is attached to the upper right corner of the front outer surface so that the camera's optical axis faces diagonally upward to the right, and image capturing device 14b is attached to the upper left corner of the front outer surface so that the camera's optical axis faces diagonally upward to the left. Image capturing device 14c is attached to the lower right corner of the front outer surface so that the camera's optical axis faces forward, and image capturing device 14d is attached to the lower left corner of the front outer surface so that the camera's optical axis faces forward, and image capturing devices 14c and 14d form a stereo camera.

[0021] The HMD 100 transmits images captured by the imaging device 14 and sensor data detected by the orientation sensor to the information processing device 10, and also receives game image data and game audio data generated by the information processing device 10.

[0022] 3 shows functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various data such as image data, audio data, and sensor data, as well as commands. The storage unit 122 temporarily stores the data and commands processed by the control unit 120. The orientation sensor 124 acquires sensor data related to the movement of the HMD 100. The orientation sensor 124 includes at least a three-axis acceleration sensor and a three-axis gyro sensor, and detects the values ​​of each axial component (sensor data) at a predetermined cycle (for example, 1600 Hz).

[0023] The communication control unit 128 transmits data output from the control unit 120 to the external information processing device 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 128 also receives data from the information processing device 10 and outputs it to the control unit 120.

[0024] When the control unit 120 receives game image data and game audio data from the information processing device 10, it supplies the data to the display panel 130 for display and to the audio output unit 132 for audio output. The display panel 130 is composed of a left-eye display panel 130a and a right-eye display panel 130b, and a pair of parallax images is displayed on each display panel. The control unit 120 also causes the communication control unit 128 to transmit sensor data acquired by the orientation sensor 124, audio data acquired by the microphone 126, and captured images acquired by the imaging device 14 to the information processing device 10.

[0025] 4 shows functional blocks of the information processing device 10. The information processing device 10 includes a processing unit 200 and a communication unit 202. The processing unit 200 includes an acquisition unit 210, a setting unit 220, an estimation processing unit 222, a game execution unit 224, and an image generation unit 230. The acquisition unit 210 includes a captured image acquisition unit 212, a sensor data acquisition unit 214, and an operation information acquisition unit 216. The image generation unit 230 includes an HMD image generation unit 232 that generates a display image to be displayed on the HMD 100, and a TV image generation unit 234 that generates a display image to be displayed on the output device 15.

[0026] The communication unit 202 receives operation information transmitted from the input device 16 and supplies it to the acquisition unit 210. The communication unit 202 also receives captured images and sensor data transmitted from the HMD 100 and supplies it to the acquisition unit 210.

[0027] These configurations can be realized in hardware terms using any processor, memory, or other LSI, and in software terms using programs loaded into memory, but here we are depicting functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms using only hardware, only software, or a combination of both.

[0028] The captured image acquisition unit 212 acquires an image of the surroundings of the HMD 100 and supplies it to the estimation processing unit 222. The estimation processing unit 222 performs processing to estimate the position and orientation of the HMD 100 based on the captured image, and supplies the position information and orientation information that are the estimation results to the game execution unit 224. The sensor data acquisition unit 214 acquires sensor data detected by the orientation sensor 124 of the HMD 100 and supplies it to the estimation processing unit 222. It is preferable that the estimation processing unit 222 use the sensor data to improve the estimation accuracy of the position information and orientation information of the HMD 100.

[0029] Before starting game play, a user wearing the HMD 100 performs initial setup, in which the user captures and registers an image of the environment around them with the imaging device 14. During initial setup, the user defines the area in which they will play (the area in which they can move) to ensure safety while playing. During game play, if the user is about to leave the play area, the information processing device 10 warns the user that they are about to leave the play area. During game play, the image of the surrounding environment registered during initial setup may be periodically updated to create an updated environmental map.

[0030] The estimation processing unit 222 acquires images captured by the imaging device 14 of the HMD 100 in time series, divides each image into a grid, and detects feature points for each section. The estimation processing unit 222 associates feature points between an image captured at time (t-1) and an image captured at time (t), and extracts the movement amount (dx, dy) of the feature point between the two images. For example, the estimation processing unit 222 may perform feature point tracking processing by predicting the movement of the feature points and performing block matching of the feature points through a peripheral search. Note that the estimation processing unit 222 may perform feature point tracking processing using a known method other than this method. By performing tracking processing with high accuracy, the estimation processing unit 222 can estimate the position and orientation of the HMD 100 with high accuracy. The position information and orientation information of the HMD 100 are supplied to the game execution unit 224 and used as input for the game.

[0031] The operation information acquisition unit 216 acquires operation information transmitted from the input device 16 and supplies it to the game execution unit 224. The game execution unit 224 executes a game program based on the operation information of the input device 16 and the position information and orientation information of the HMD 100, and performs calculation processing to move a game character in a virtual space. The image generation unit 230 includes a GPU (Graphics Processing Unit) that executes rendering processing and the like, and generates game images.

[0032] In the image generation unit 230, the HMD image generation unit 232 generates a display image to be displayed on the display panel 130 of the HMD 100, and the TV image generation unit 234 generates a display image to be displayed on the output device 15. Although not shown, the information processing device 10 is configured to include an audio generation unit that generates game audio.

[0033] 5 shows an example of a display image displayed on the display panel 130. The HMD image generation unit 232 generates a game image and supplies it to the display panel 130 of the HMD 100. As described above, the display panel 130 has a left-eye display panel 130a and a right-eye display panel 130b, and the HMD image generation unit 232 generates a game image for the left eye and a game image for the right eye, and supplies them to the left-eye display panel 130a and the right-eye display panel 130b. The left-eye display panel 130a and the right-eye display panel 130b each display a game image.

[0034] The TV image generation unit 234 generates a game image that is the same as the image generated by the HMD image generation unit 232, and supplies the image to the output device 15. Note that in the information processing system 1, since a user wearing the HMD 100 does not see the game image displayed on the output device 15, the TV image generation unit 234 may generate a different game image for another user and supply the image to the output device 15.

[0035] When a user wearing the HMD 100 on their head is playing a game while facing the direction in which the output device 15 is installed, if the screen size of the output device 15 is large and the distance between the user and the output device 15 is short, the area of ​​the output device 15 included in the image captured by the imaging device 14 will be large. In this case, the estimation processing unit 222 will extract many feature points from the game image displayed on the output device 15, and since it will be unable to associate these with previously extracted feature points, there is a high possibility that tracking loss will occur.

[0036] 6(a) shows an example of an image captured by imaging device 14c, and FIG. 6(b) shows an example of an image captured by imaging device 14d. Because imaging devices 14c and 14d form a stereo camera, the images captured by the two devices are slightly shifted horizontally. As shown in FIG. 6, the captured image largely includes the game image displayed on output device 15.

[0037] 6(a) and 6(b), a very large proportion of the captured images is the game image displayed on the output device 15. Because the game image changes every moment, it becomes difficult for the estimation processing unit 222 to extract valid feature points from the captured images, and the estimation processing of the position and orientation of the HMD 100 may fail.

[0038] Therefore, in the information processing device 10 of the embodiment, the TV image generating unit 234 generates a display image that includes at least a part of a still image pattern, so that the estimation processing unit 222 can extract feature points that are effective for tracking.

[0039] 7 shows an example of an image displayed on the output device 15. The TV image generation unit 234 generates a display image that includes, as a part thereof, a picture 70, which is a still image. In this example, the TV image generation unit 234 generates a display image that includes the picture 70 that forms a frame and a game video 72 inside the frame, and displays it on the output device 15. By displaying the picture 70 that forms the rectangular frame as a still image, the estimation processing unit 222 can extract feature points that are effective for tracking from the picture 70 included in the image captured of the output device 15.

[0040] The TV image generation unit 234 may display the game video 72 with a reduced image size inside a frame formed by the picture 70. Alternatively, the TV image generation unit 234 may generate a display image in which a rectangular frame formed by the picture 70 is superimposed on the game video 72 without reducing the image size. To avoid impairing the visibility of the game video 72, it is preferable that the area occupied by the frame be set to 50% or less of the screen size of the output device 15. Instead of displaying the picture 70 as a frame, the TV image generation unit 234 may display the picture 70 as a block in, for example, one of the four corners of the screen of the output device 15. That is, the TV image generation unit 234 may generate a display image including the picture 70 as a block image in a corner. The TV image generation unit 234 may also generate a display image including the picture 70 on one or more of the top, bottom, left, right, or multiple sides, and display the picture 70 on one or more of the top, bottom, left, or right sides of the screen of the output device 15. In either case, the picture 70 is displayed as a still image to support tracking processing by the estimation processing unit 222. The TV image generating unit 234 may not include the game moving image 72 in the display image, and may generate a display image that includes, for example, the picture 70, which is a still image, on the entire screen.

[0041] Since the estimation processing unit 222 extracts corners (angular parts) included in the captured image as feature points, it is preferable that the picture 70 be configured to have a pattern that includes many corners. Note that, in order to improve tracking accuracy, it is preferable that the picture 70 be configured so that the same pattern is not repeated in nearby positions.

[0042] The setting unit 220 sets the picture display function by the TV image generation unit 234 to on or off, and sets whether the function is enabled or disabled when the picture display function is set to on. When the setting unit 220 sets the picture display function to be enabled, the TV image generation unit 234 generates a display image that includes, at least in part, the picture 70, which is a still image, and displays it on the output device 15. When the setting unit 220 sets the picture display function to be disabled, the TV image generation unit 234 does not generate a display image that includes the picture 70.

[0043] The setting unit 220 may enable the picture display function by the TV image generation unit 234 when the estimation processing unit 222 is performing estimation processing of the position and orientation of the HMD 100, and may disable the picture display function by the TV image generation unit 234 when the estimation processing unit 222 is not performing estimation processing. Therefore, when the estimation processing unit 222 is performing estimation processing, the TV image generation unit 234 generates a display image including the picture 70 to support tracking processing by the estimation processing unit 222. On the other hand, when the estimation processing unit 222 is not performing estimation processing, there is no need to support tracking processing by the estimation processing unit 222, and therefore the TV image generation unit 234 does not generate a display image including the picture 70. At this time, the TV image generation unit 234 may generate a display image that includes the game video 72 in its entirety and display it on the output device 15.

[0044] The setting unit 220 may enable the picture display function by the TV image generation unit 234 when the user is wearing the HMD 100 on their head, and may disable the picture display function by the TV image generation unit 234 when the user is not wearing the HMD 100 on their head. Whether the HMD 100 is being worn is determined, for example, based on the detection result of a proximity sensor (not shown) provided in the HMD 100. Therefore, when the user is wearing the HMD 100 on their head, the TV image generation unit 234 generates a display image including the picture 70 to support the tracking process by the estimation processing unit 222. On the other hand, when the user is not wearing the HMD 100 on their head, there is no need to support the tracking process by the estimation processing unit 222, and therefore the TV image generation unit 234 does not generate a display image including the picture 70. In this case, the TV image generation unit 234 may generate a display image including the game video 72 in its entirety and display it on the output device 15.

[0045] The setting unit 220 may enable the picture display function by the TV image generation unit 234 on the condition that the estimation processing unit 222 is performing estimation processing of the position and orientation of the HMD 100 and the user is wearing the HMD 100 on their head.

[0046] When the estimation processing unit 222 recognizes a frame made up of the picture 70 in the captured image, it may control the tracking process so as not to extract feature points from the game video 72 arranged inside the frame. Note that, if a frame made up of the picture 70 is displayed on the output device 15 at the time of initial setup and an image of the frame is registered, the estimation processing unit 222 may control the tracking process so as not to use feature points at the position of the frame during game play.

[0047] The picture display function by the TV image generating unit 234 may be set to on or off by the user at the time of initial setup. 8 shows an example of a setting screen for turning the picture display function on and off. This setting screen is displayed on the display panel 130 during initial setup. When the user sets the picture display function to on, the setting unit 220 enables the picture display function by the TV image generation unit 234, on the condition that the estimation processing unit 222 is performing estimation processing of the position and orientation of the HMD 100 and the user is wearing the HMD 100 on their head. This causes the TV image generation unit 234 to generate a display image including the picture 70. On the other hand, when the user sets the picture display function to off, the setting unit 220 disables the picture display function by the TV image generation unit 234.

[0048] When the picture display function is set to OFF, the estimation processing unit 222 may fail to estimate the position and orientation of the HMD 100 by performing tracking processing using an image captured of a game image displayed on the entire surface of the output device 15. When the estimation processing fails, the setting unit 220 may automatically change the picture display function by the TV image generation unit 234 to ON, thereby enabling the picture display function.

[0049] It should be noted that instead of the setting unit 220 automatically changing the picture display function to an on setting, the HMD image generation unit 232 may generate a display image including an option to enable the picture display function and display it on the display panel 130.

[0050] 9 shows an example of a setting screen displayed on the display panel 130 during gameplay. When the user selects the "Turn on auxiliary display function" option, the setting unit 220 turns on the picture display function of the TV image generation unit 234, enabling the picture display function. This improves tracking accuracy.

[0051] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and those skilled in the art will understand that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention. In the embodiments, the estimation process is performed by the information processing device 10, but some or all of the functions of the information processing device 10 may be provided in the HMD 100, and the HMD 100 may perform the estimation process. In this case, the HMD 100 functions as an information processing device.

[0052] In the embodiment, the imaging device 14 is attached to the HMD 100, but the imaging device 14 may be attached to a position other than the HMD 100 as long as it can capture an image of the space around the HMD 100.

[0053] In a modified example, the setting unit 220 may dynamically enable or disable the picture display function. For example, the setting unit 220 may enable or disable the picture display function based on the distance between the output device 15 and the HMD 100. Specifically, the setting unit 220 may enable the picture display function if the distance between the output device 15 and the HMD 100 is equal to or shorter than a predetermined distance, and disable the picture display function if the distance is longer than the predetermined distance.

[0054] The setting unit 220 may enable or disable the picture display function based on the proportion of the area of ​​the output device 15 included in the captured image. Specifically, the setting unit 220 may enable the picture display function if the proportion of the area of ​​the output device 15 included in the captured image is equal to or greater than a predetermined value, and may disable the picture display function if the proportion is less than the predetermined value.

[0055] When setting the play area, the setting unit 220 may detect the screen size of the output device 15 and the distance between the output device 15 and the HMD 100, and if the screen size is large and the distance is short, recommend the user to turn on the picture display function. The screen size may be detected by HDMI (registered trademark). A large screen size and a short distance may be determined based on the proportion of the area of ​​the output device 15 included in the image captured by the imaging device 14.

[0056] The setting unit 220 may also select the picture 70 to be displayed depending on the screen size of the output device 15. Specifically, if the screen size is small, a picture 70 including a large pattern may be selected, and if the screen size is large, a picture 70 including a small pattern may be selected. Note that the setting unit 220 may also select the picture 70 based on the proportion of the area of ​​the output device 15 that is included in the captured image. [Explanation of symbols]

[0057] 1···Information processing system, 10···Information processing device, 14···Imaging device, 15···Output device, 70···Image, 72···Game video, 100···HMD, 200···Processing unit, 202···Communication unit, 210···Acquisition unit, 212···Captured image acquisition unit, 214···Sensor data acquisition unit, 216···Operation information acquisition unit, 220···Setting unit, 222···Estimation processing unit, 224···Game execution unit, 230···Image generation unit, 232···HMD image generation unit, 234···TV image generation unit.

Claims

1. an estimation processing unit that extracts feature points from an image captured around the head mounted display and performs a tracking process of the feature points to estimate at least one of the position and the orientation of the head mounted display; an image generation unit that generates a display image to be displayed on a display device different from the head-mounted display, the image generation unit generates a display image including a frame configured by a picture that is a still image and a moving image displayed inside the frame; the estimation processing unit extracts feature points to be tracked from the pattern included in the captured image and performs tracking processing of the feature points.

1. An information processing device comprising:

2. the image generation unit generates a display image including the same moving image as the moving image to be displayed on the head-mounted display; 2. The information processing apparatus according to claim 1, wherein:

3. the image generating unit generates the image in such a way that the same pattern is not repeated at nearby positions; 2. The information processing apparatus according to claim 1, wherein:

4. The image generation unit generates the image including a plurality of corners.

2. The information processing apparatus according to claim 1, wherein:

5. The image generation unit generates a display image including the picture, which is a still image, on one or more sides of the picture, above, below, left, or right.

2. The information processing apparatus according to claim 1, wherein:

6. When the estimation processing unit is performing the estimation processing, the image generation unit generates a display image including the picture, When the estimation processing unit is not performing the estimation process, the image generation unit does not generate a display image including the picture.

6. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

7. When a user wears a head-mounted display on his / her head, the image generation unit generates a display image including the picture, When the user is not wearing a head-mounted display on his / her head, the image generation unit does not generate a display image including the picture.

7. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

8. Further, a setting unit is provided for setting whether to enable or disable the picture display function of the image generating unit.

8. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

9. When the estimation processing unit fails in the estimation processing, the setting unit enables a picture display function by the image generation unit, or the image generation unit generates a display image to be displayed on a head-mounted display, the display image including an option to enable the picture display function.

9. The information processing apparatus according to claim 8,

10. a first step of extracting feature points from an image of the surroundings of a head-mounted display and performing a tracking process of the feature points to estimate at least one of a position and an orientation of the head-mounted display; a second step of generating a display image to be displayed on a display device different from the head-mounted display, the display image including a frame made up of a still image and a moving image to be displayed inside the frame; The first step extracts feature points to be tracked from the image included in the captured image and performs tracking processing of the feature points. An image generating method comprising:

11. On the computer, a first function of extracting feature points from an image of the surroundings of the head-mounted display and performing a process of tracking the feature points to estimate at least one of the position and the orientation of the head-mounted display; A program for realizing a second function of generating a display image to be displayed on a display device different from the head-mounted display, the display image including a frame made up of a still image pattern and a moving image to be displayed inside the frame, The first function is a program including a function of extracting feature points to be tracked from the image included in the captured image and performing tracking processing of the feature points.

Images