Head-mounted display device, control method for head-mounted display device, and program

Abstract

To provide a technique that can accurately estimate (determine) at least one of the position and the attitude of a head-mounted display device.SOLUTION: A head-mounted display device has: first imaging means; projection means that projects a pattern image within an imaging range of the first imaging means; second imaging means that picks up an image of a range of the pattern image not overlapping the projection range; first acquisition means that, based on a first image picked up by the first imaging means, acquires first information related to a state including at least one of the position and the attitude of the head-mounted display device; second acquisition means that, based on a second image picked up by the second imaging means, acquires second information related to the state of the head-mounted display device; and determination means that, based on the first information and the second information, determines the state of the head-mounted display device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a head-mounted display device, a state determination device, a control method for a head-mounted display device, a control method for a state determination device, and a program. [Background technology]

[0002] As a head-mounted display device (for example, an HMD (head-mounted display)), a display device capable of estimating its own position and orientation based on an image captured by a camera (captured image) has been proposed. Technology relating to such a head-mounted display device is disclosed, for example, in Patent Document 1. By using the estimated position and orientation, it becomes possible to suitably display images to the user. For example, it becomes possible to seamlessly synthesize an image of a virtual space or a virtual object into a real space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-009557 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in a conventional head-mounted display device, the position and orientation of the head-mounted display device cannot be estimated with high accuracy. For example, when a user wearing a head-mounted display device moves his / her head quickly, the feature points in the real space tracked by the camera may go outside the imaging range of the camera, or the captured image may be blurred due to the motion of the feature points. As a result, the feature points cannot be tracked, and the position and orientation of the head-mounted display device cannot be estimated with high accuracy.

[0005] An object of the present invention is to provide a technique that can estimate (determine) at least one of the position and orientation of a head-mounted display device with high accuracy. [Means for solving the problem]

[0006] A first aspect of the present invention is a head-mounted display device characterized in having a first imaging means, a projection means for projecting a pattern image into the imaging range of the first imaging means, a second imaging means for imaging a range that does not overlap with the projection range of the pattern image, a first acquisition means for acquiring first information regarding a state including at least one of a position and an orientation of the head-mounted display device based on a first image captured by the first imaging means, a second acquisition means for acquiring second information regarding the state of the head-mounted display device based on a second image captured by the second imaging means, and a judgment means for judging the state of the head-mounted display device based on the first information and the second information.

[0007] A second aspect of the present invention is a status determination device that can be used together with a head-mounted display device having a first imaging means and a second imaging means, characterized in that the status determination device has a first estimation means for estimating a state including at least one of a position and an orientation of the head-mounted display device based on a first image captured by the first imaging means, a second estimation means for estimating a change in the state of the head-mounted display device per specified time based on a second image captured by the second imaging means, and a determination means for determining the state of the head-mounted display device based on the estimation result of the first estimation means and the estimation result of the second estimation means.

[0008] A third aspect of the present invention is a method for detecting a state of a vehicle comprising the first imaging means, the second imaging means, and the above-mentioned state determination device. The present invention relates to a head-mounted display device.

[0009] A fourth aspect of the present invention is a control method for a head-mounted display device having a first imaging means, a projection means for projecting a pattern image into the imaging range of the first imaging means, and a second imaging means for imaging a range that does not overlap with the projection range of the pattern image, characterized in having a first acquisition step of acquiring first information regarding a state including at least one of a position and an orientation of the head-mounted display device based on a first image captured by the first imaging means, a second acquisition step of acquiring second information regarding the state of the head-mounted display device based on a second image captured by the second imaging means, and a judgment step of judging the state of the head-mounted display device based on the first information and the second information.

[0010] A fifth aspect of the present invention is a control method for a state determination device that can be used together with a head-mounted display device having a first imaging means and a second imaging means, characterized in having a first estimation step of estimating a state including at least one of a position and an orientation of the head-mounted display device based on a first image captured by the first imaging means, a second estimation step of estimating a change in the state of the head-mounted display device per specified time based on a second image captured by the second imaging means, and a judgment step of judging the state of the head-mounted display device based on the estimation result of the first estimation step and the estimation result of the second estimation step.

[0011] A sixth aspect of the present invention is a program for causing a computer to execute each step of the above-mentioned control method. Effect of the Invention

[0012] According to the present invention, at least one of the position and the orientation of a head-mounted display device can be estimated (determined) with high accuracy. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is an external view of an HMD according to the present embodiment. [Diagram 2]FIG. 2 is a block diagram showing the functional configuration of a CPU according to the present embodiment. [Diagram 3] FIG. 4 is a block diagram showing a functional configuration of a state estimation unit according to the present embodiment. [Figure 4] 4 is a flowchart of a state estimation process according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] An embodiment of the present invention will be described. In this embodiment, an example in which the present invention is applied to an HMD (head mounted display) will be described, but the present invention can also be applied to other head mounted display devices. For example, a handheld display device that a user holds in his / her hand and wears (places against) on the head is one type of head mounted display device, and the present invention can also be applied to a handheld display device. The present invention can also be applied to a head mounted display device in which a user views an image with both eyes, and a head mounted display device in which a user views an image with one eye.

[0015] The present invention is applicable to both the video see-through method and the optical see-through method. In the case of the video see-through method, the head-mounted display device displays an image of the outside world (real space) by synthesizing graphics (e.g., virtual objects) as necessary. In this case, the user cannot see the real space directly, but can indirectly see the real space or see graphics synthesized with the image of the real space by looking at the displayed image. In the case of the optical see-through method, the head-mounted display device has, for example, a lens similar to the lens of ordinary glasses, and projects graphics onto the lens as necessary. In this case, the user can directly see the real space through the lens or see graphics projected onto the lens. The present invention is a method for displaying an image of a virtual space without displaying an image of the real space. That is, the present invention is applicable to various XR (Cross Reality), such as AR (Augmented Reality), MR (Mixed Reality), and VR (Virtual Reality).

[0016] The present invention can be applied to electronic devices other than the head-mounted display device as long as the electronic devices can be used together with the head-mounted display device. The state determination device to which the present invention is applied may be provided in the head-mounted display device, or may be provided in an electronic device separate from the head-mounted display device. For example, the present invention can be applied to a controller or a personal computer (PC) connected to the HMD by wire or wirelessly.

[0017] 1(A) and 1(B) are external views of an HMD 100 according to this embodiment. FIG. 1(A) is a front perspective view of the HMD 100, and FIG. 1(B) is a rear perspective view of the HMD 100. As shown in FIG. 1(A), the HMD 100 is provided with a headband 110. The user places the HMD 100 in contact with the eyes and fixes it to the head with the headband 110. The HMD 100 has a stereo camera 101, a projection unit 102, a side camera 103, a left display unit 104L, and a right display unit 104R. The HMD 100 has a CPU (Central The internal memory has a processing unit (CPU) 105, a read only memory (ROM) 106, and a random access memory (RAM) 107.

[0018] The stereo camera 101 is a camera (imaging device) that captures the outside world, and has a left imaging unit 101L and a right imaging unit 101R. For example, the stereo camera 101 is a global shutter type camera (each of the left imaging unit 101L and the right imaging unit 101R is, for example, a global shutter type imaging unit). The images obtained by the stereo camera 101 (each of the left imaging unit 101L and the right imaging unit 101R) are, for example, grayscale images. The projection unit 102 is directed in the same direction as the stereo camera 101, and projects a pattern image into the imaging range of the stereo camera 101. In this embodiment, the projection unit 102 projects a pattern image of invisible light (for example, a dot pattern of infrared light). The side camera 103 is also a camera that captures the outside world, and is, for example, a global shutter type camera. The images obtained by the side camera 103 are also, for example, grayscale images. The left display unit 104L is disposed so as to face the left eye of the user wearing the HMD 100, and displays an image to be viewed by the user's left eye (display image, left display image). The right display unit 104R is disposed so as to face the right eye of the user wearing the HMD 100, and displays an image to be viewed by the user's right eye (display image, right display image).

[0019] The CPU 105 is an information processing device (image processing device) that controls the entire HMD 100. The ROM 106 stores various data (for example, various programs and various parameters). The RAM 107 also stores various data (including images captured by the stereo camera 101 and images captured by the side camera 103). For example, the CPU 105 loads a program stored in the ROM 106 into the RAM 107 and executes it.

[0020] 2 is a block diagram showing the functional configuration of the CPU 105. The CPU 105 has a distance measurement unit 201, a state estimation unit 202, a graphics generation unit 203, and a display control unit 204.

[0021] The distance measuring unit 201 acquires two images having a parallax from the stereo camera 101 (an image captured by the left imaging unit 101L and an image captured by the right imaging unit 101R). The distance measuring unit 201 then generates a distance map (distance information, depth map, depth information) from the two acquired images. The image used by the distance measuring unit 201 is an image that includes a pattern image projected by the projection unit 102. The distance map is generated by the stereo camera 101. The distance map is information indicating the distribution of distance (depth) from the camera 1 to the subject, and is, for example, an image having depth values ​​as pixel values. Note that a monocular camera may be used instead of the stereo camera 101. For example, a ToF (Time of Flight) camera may be used. In that case, an image captured by the ToF camera may be used as the distance map. However, from the viewpoint of stability of various processes, the stereo camera 101 is more preferable than the monocular camera.

[0022] The state estimation unit 202 estimates the state of the HMD 100 based on the captured image of the stereo camera 101 (image captured by the stereo camera 101) and the captured image of the side camera 103 (image captured by the side camera 103). In this embodiment, the state of the HMD 100 includes at least one of the position and the orientation of the HMD 100.

[0023] The graphics generation unit 203 generates graphics. For example, the graphics generation unit 203 generates an image of a virtual object such that the virtual object is placed in a three-dimensional position based on the distance map generated by the distance measurement unit 201 and the state of the HMD 100 estimated by the state estimation unit 202.

[0024] The display control unit 204 generates a left display image based on an image captured by the left imaging unit 101L, and displays the left display image on the left display unit 104L. The display control unit 204 synthesizes a graphic (e.g., a virtual object) generated by the graphic generation unit 203 as necessary, and generates a left display image in which the graphic is arranged. Similarly, the display control unit 204 generates a right display image based on an image captured by the right imaging unit 101R, and displays the right display image on the right display unit 104R. The display control unit 204 synthesizes a graphic (e.g., a virtual object) generated by the graphic generation unit 203 as necessary, and generates a right display image in which the graphic is arranged. It is preferable that the image used by the display control unit 204 is an image in which the pattern image projected by the projection unit 102 is not reflected.

[0025] Here, consider a configuration in which the state of the HMD 100 is estimated based only on the images captured by the stereo camera 101. In such a configuration, the state of the HMD 100 may not be estimated with high accuracy. For example, when a user wearing the HMD 100 moves his / her head quickly, the feature points in the real space tracked by the stereo camera 101 may go outside the imaging range of the stereo camera 101, or motion blur of the feature points may occur in the images captured by the stereo camera 101. As a result, the feature points cannot be tracked, and the state of the HMD 100 cannot be estimated with high accuracy.

[0026] Therefore, in this embodiment, an image captured by the side camera 103 is further used. The side camera 103 is disposed so as to capture an image of a range that does not overlap with the projection range of the pattern image. The projection range of the pattern image may be a part of the imaging range of the stereo camera 101, or may be the entire imaging range of the stereo camera 101. In this embodiment, the projection unit 102 projects the pattern image onto the entire imaging range of the stereo camera 101. Therefore, the side camera 103 captures an image of a range that does not overlap with the imaging range of the stereo camera 101.

[0027] As a result, the range that can be referred to in order to estimate the state of the HMD 100 is expanded by the imaging range of the side camera 103, so that the state of the HMD 100 can be estimated with high accuracy. For example, even if a feature point is outside the imaging range of the stereo camera 101, as long as the feature point is imaged by the side camera 103, the state of the HMD 100 can be estimated with high accuracy. Furthermore, even if motion blur of the feature point occurs in the image captured by the stereo camera 101, as long as motion blur of the feature point does not occur in the image captured by the side camera 103, the state of the HMD 100 can be estimated with high accuracy.

[0028] In the present embodiment, each of the stereo camera 101 and the side camera 103 is a global shutter camera, but the present invention is not limited to this. For example, a rolling shutter camera may be used as the stereo camera 101. A rolling shutter camera is less expensive than a global shutter camera and is easy to increase the resolution. Therefore, by using a rolling shutter camera as the stereo camera 101, the price of the HMD 100 can be reduced and the resolution of the displayed image can be increased. A rolling shutter camera may also be used as the side camera 103. However, since the global shutter system is less likely to cause distortion in the captured image than the rolling shutter system, it is preferable to use a global shutter camera from the viewpoint of suppressing distortion. For example, it is preferable that the side camera 103 can capture an image with less distortion than the image captured by the stereo camera 101, so it is preferable to use a global shutter camera as the side camera 103.

[0029] Although there are no particular limitations on the frame rate of imaging by the stereo camera 101 and the frame rate of imaging by the side camera 103, they are preferably the same (for example, 60 fps, 90 fps, or 120 fps). If the frame rate of imaging by the stereo camera 101 is the same as the frame rate of imaging by the side camera 103, the difference in timing between these images can be reduced, thereby enabling more suitable processing.

[0030] 3 is a block diagram showing the functional configuration of the state estimation unit 202. The state estimation unit 202 has a first estimation unit 301, a feature detection unit 302, a second estimation unit 303, and a state determination unit 304.

[0031] The first estimation unit 301 acquires first information on the state of the HMD 100 based on the captured image of the stereo camera 101. In this embodiment, the first estimation unit 301 acquires the first information indicating the estimated state by estimating the state of the HMD 100 based on the captured image of the stereo camera 101. For example, the first estimation unit 301 estimates the state of the HMD 100 by Simultaneous Localization and Mapping (SLAM).

[0032] The first information may be information about the state of the HMD 100, and may be, for example, information indicating an amount of change from a predetermined state. The first estimation unit 301 may use distance information (a distance map generated by the distance measurement unit 201) corresponding to an image captured by the stereo camera 101 to estimate the state of the HMD 100. For example, the first estimation unit 301 may estimate the state of the HMD 100 by RGB-D SLAM. The first estimation unit 301 may estimate the state of the HMD 100 using only the distance information.

[0033] If the pattern image projected by the projection unit 102 appears in the captured image of the stereo camera 101, the features of the pattern image (e.g., dots) may be erroneously detected as features of the real space (e.g., feature points), and the state of the HMD 100 may not be estimated with high accuracy. For this reason, the projection unit 102 may repeat projecting and not projecting the pattern image. Then, the first estimation unit 301 may estimate the state of the HMD 100 based on the captured image of the stereo camera 101 obtained during the period when the pattern image is not projected.

[0034] Note that even during the period when the pattern image is projected, the state of the HMD 100 can be estimated by odometry using the image captured by the side camera 103. For example, the current state of the HMD 100 can be estimated by adding a state change of the HMD 100 corresponding to a time change of the image captured by the side camera 103 (for example, a motion vector of a feature point) to the previous state of the HMD 100 (estimated state).

[0035] The feature detection unit 302 detects features from the image captured by the side camera 103. The features to be detected may be features related to real space, and in this embodiment, the feature detection unit 302 detects feature points from the image captured by the side camera 103. The feature points are, for example, points (part or whole of an object) that exist in real space. A specific object indicating a feature point may be installed, or a part or whole of an object that already exists may be detected as a feature point. Various known techniques (for example, KLT (Kanade-Lucas-Tomasi) Tracker or goodFeatures To Track) can be used to detect the features.

[0036] The second estimation unit 303 acquires second information on the state of the HMD 100 based on the image captured by the side camera 103. Note that the second information only needs to be information on the state of the HMD 100, and may or may not be information indicating the state of the HMD 100.

[0037] As described above, by further using the image captured by the side camera 103, the state of the HMD 100 can be estimated with high accuracy. However, since the range in which the side camera 103 can be arranged is limited, the side camera 103 may be provided in a position with low rigidity. If the side camera 103 is provided in a position with low rigidity, the relative position of the side camera 103 with respect to the stereo camera 101 may change due to temperature or the passage of time. In a configuration in which the state of the HMD 100 is estimated based on an image captured by the side camera 103, if the relative position of the side camera 103 changes, the accuracy of the state estimation decreases. If the side camera 103 is provided in a member different from the member in which the stereo camera 101 is provided, the relative position of the side camera 103 may also change due to a misalignment between these members.

[0038] In this embodiment (FIG. 1), the stereo camera 101 is provided on the front of the HMD 100, and the side camera 103 is provided on a side of the HMD 100. The side of the HMD 100 is the surface that the user holds when wearing the HMD 100, and is often made of a material with lower rigidity than the front in order to reduce the overall weight of the HMD 100. Even if the material of the front and the material of the side are the same, the rigidity of the edge part of the material is generally lower than the rigidity of the central part of the material. Therefore, in this embodiment, the side camera 103 is provided at a position with lower rigidity than the position where the stereo camera 101 is provided. The arrangement of the stereo camera 101 and the side camera 103 is not limited to the above arrangement.

[0039] Therefore, in this embodiment, the second estimation unit 303 estimates the state change of the HMD 100 per predetermined time based on the captured image of the side camera 103, thereby acquiring second information indicating the estimated state change. For example, the second estimation unit 303 estimates the state change of the HMD 100 according to the time change of the feature detected by the feature detection unit 302 (for example, the motion vector of the feature point). Since it is considered that the change in the relative position of the side camera 103 does not occur (or is small) within the predetermined time (short time), the state change of the HMD 100 can be estimated with high accuracy. Furthermore, since the change in the relative position of the side camera 103 can be ignored, even if the relative position of the side camera 103 changes, calibration of the state estimation of the HMD 100 is not necessary, and high convenience can be realized.

[0040] In this embodiment (FIG. 1), side cameras 103 are provided on both the left and right sides of HMD 100, but the number of side cameras 103 may be one or three or more. When multiple side cameras 103 are provided, for example, second estimation unit 303 estimates multiple state changes using images captured by each side camera 103 individually, and determines the average of the multiple state changes as the final state change.

[0041] Also, as long as the change in the relative position of the side camera 103 can be substantially ignored, the predetermined time is not particularly limited, but it is preferable that the predetermined time is short. Therefore, in this embodiment, the second estimation unit 303 estimates a change in the state of the HMD 100 from the previous image capture by the side camera 103 based on the image captured by the side camera 103.

[0042] The state determination unit 304 determines the state of the HMD 100 based on the first information and the second information. In general, in a configuration in which the state of the HMD 100 is estimated based only on the captured image of the stereo camera 101, the greater the number of feature points detected from the captured image of the stereo camera 101, the higher the accuracy of state estimation. Therefore, in this embodiment, when the number of feature points detected from the captured image of the stereo camera 101 is greater than a threshold, the state determination unit 304 determines the state indicated by the first information (the state estimated by the first estimation unit 301) as the state of the HMD 100. Then, when the number of feature points detected from the captured image of the stereo camera 101 is less than the threshold, the state determination unit 304 determines the state of the HMD 100 based on the first information and the second information. In this embodiment, the state determination unit 304 determines the current state of the HMD 100 by adding the state change estimated by the second estimation unit 303 (the state change indicated by the current second information) to the state determined at the timing before the predetermined time from the present. The state determined (by the state determination unit 304) at the timing the predetermined time before the present is, for example, the state indicated by the first information obtained at that timing.

[0043] Note that the method of determining the state of the HMD 100 is not limited to the above method. For example, the state determination unit 304 may weight and combine the state change estimated by the second estimation unit 303 and the state change estimated by the first estimation unit 301 during the period from the timing before the predetermined time to the present (the change in the state indicated by the first information during that period). Then, the state determination unit 304 may determine the current state of the HMD 100 by adding the change obtained by the weighted combination to the state determined at the timing before the predetermined time from the present. By performing the weighted combination, it is possible to suppress a sudden change in the result of state estimation before and after using the second information.

[0044] In the weighted synthesis, the state determination unit 304 may use weights based on the number of feature points detected from the images captured by the stereo camera 101 and the number of feature points detected from the images captured by the side camera 103. For example, the state determination unit 304 increases the weight of the state change estimated by the first estimation unit 301 as the difference value obtained by subtracting the number of feature points of the images captured by the stereo camera 101 from the number of feature points of the side camera 103 increases. The state determination unit 304 increases the weight of the state change estimated by the second estimation unit 303 as the difference value decreases.

[0045] Fig. 4 is a flowchart of the state estimation process of the HMD 100. The state estimation process of Fig. 4 is realized by the CPU 105 expanding a program stored in the ROM 106 into the RAM 107 and executing it. For example, when the HMD 100 is started, the state estimation process of Fig. 4 starts. The CPU 105 repeatedly executes the state estimation process of Fig. 4. In the first state estimation process, for example, the state of the HMD 100 is determined by the process of step S409.

[0046] In step S401, the CPU 105 determines whether or not a pattern image is projected by the projection unit 102. If the CPU 105 determines that a pattern image is projected, the process proceeds to step S402, and if the CPU 105 determines that a pattern image is not projected, the process proceeds to step S405.

[0047] In step S402, the CPU 105 (feature detection unit 302) acquires an image captured by the side camera 103. In step S403, the CPU 105 (feature detection unit 302) Feature points are detected from the captured image (image captured by the side camera 103) acquired in step S402. In step S404, the CPU 105 (second estimation unit 303) estimates a state change of the HMD 100 based on the time change (motion vector) of the feature points detected in step S403. Thereafter, the CPU 105 advances the process to step S412.

[0048] In step S405, the CPU 105 (first estimation unit 301) acquires an image captured by the stereo camera 101. In step S406, the CPU 105 (feature detection unit 302) acquires an image captured by the side camera 103. In step S407, the CPU 105 (first estimation unit 301) estimates the state of the HMD 100 by SLAM using the captured image (image captured by the stereo camera 101) acquired in step S405. The process of step S407 includes a process of detecting feature points from the captured image of the stereo camera 101. In step S408, the CPU 105 (feature detection unit 302) detects feature points from the captured image (image captured by the side camera 103) acquired in step S406. In step S409, the CPU 105 (second estimation unit 303) estimates a state change of the HMD 100 based on the time change (motion vector) of the feature points detected in step S408. Thereafter, the CPU 105 advances the process to step S410.

[0049] In step S410, the CPU 105 (state determination unit 304) determines whether or not the number of feature points detected from the image captured by the stereo camera 101 in step S407 is equal to or greater than a threshold. If the CPU 105 determines that the number of feature points is equal to or greater than the threshold, the process proceeds to step S411, whereas if the CPU 105 determines that the number of feature points is less than the threshold, the process proceeds to step S412. Note that if the number of feature points is equal to the threshold, the process may proceed to step S412.

[0050] In step S411, the CPU 105 (state determination unit 304) determines the state estimated by the first estimation unit 301 in step S407 as the state of the HMD 100.

[0051] In step S412, the CPU 105 (state determination unit 304) determines the state of the HMD 100 by adding the state change estimated by the second estimation unit 303 in step S404 or step S409 to the state previously determined by the state determination unit 304.

[0052] As described above, according to this embodiment, at least one of the position and the orientation of the head mounted display device can be estimated (determined) with high accuracy. For example, by using the side camera 103 arranged so as to capture an image of a range that does not overlap with the projection range of the pattern image in addition to the stereo camera 101, the range that can be referred to for estimating the state of the HMD 100 can be expanded. As a result, the state of the HMD 100 can be estimated with high accuracy.

[0053] Furthermore, according to this embodiment, the state of the HMD 100 is estimated based on the images captured by the stereo camera 101, but the state change of the HMD 100 per predetermined time is estimated based on the images captured by the side camera 103. Then, the state of the HMD 100 is determined based on these two estimation results. By estimating the state change of the HMD 100 per predetermined time based on the images captured by the side camera 103, it is possible to suppress deterioration in the accuracy of the estimation of the state of the HMD 100 caused by a change in the relative position of the side camera 103.

[0054] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the above-described configurations.

[0055] <Other embodiments> The present invention relates to a program for implementing one or more functions of the above-described embodiments, which is transmitted via a network or The program may be provided to a system or device via a storage medium, and one or more processors in the computer of the system or device may read and execute the program. The program may also be implemented by a circuit (e.g., ASIC) that implements one or more functions.

[0056] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A head-mounted display device, A first imaging means; a projection means for projecting a pattern image onto an imaging range of the first imaging means; a second imaging means for imaging an area that does not overlap with the projection area of ​​the pattern image; a first acquisition means for acquiring first information regarding a state including at least one of a position and an orientation of the head mounted display device based on a first image captured by the first imaging means; A second acquisition means for acquiring second information regarding the state of the head mounted display device based on a second image captured by the second imaging means; a determination means for determining the state of the head-mounted display device based on the first information and the second information; A head-mounted display device comprising: (Configuration 2) The first acquisition means estimates the state of the head-mounted display device based on the first image, and thereby acquires the first information indicating the estimated state. 2. A head-mounted display device according to configuration 1. (Configuration 3) The second acquisition means estimates a change in the state of the head-mounted display device per predetermined time based on the second image, and acquires the second information indicating the estimated change. 3. The head-mounted display device according to configuration 2. (Configuration 4) The second acquisition means estimates a change in the state of the head mounted display device from a previous image captured by the second imaging means based on the second image, and acquires the second information indicating the estimated change. 4. The head-mounted display device according to configuration 3. (Configuration 5) The determination means determines the current state by adding the change indicated by the current second information to the state indicated by the first information obtained at the timing before the predetermined time from the present. 5. The head-mounted display device according to configuration 3 or 4. (Configuration 6) The determination means determines the current state by adding a weighted combination of the change indicated by the current second information and the change in the state indicated by the first information during the period from the timing before the predetermined time to the present to the state indicated by the first information obtained at the timing before the present. 5. The head-mounted display device according to configuration 3 or 4. (Configuration 7) The determining means uses weights based on the number of feature points detected from the first image and the number of feature points detected from the second image in the weighted synthesis. 7. A head-mounted display device according to configuration 6. (Configuration 8) When the number of feature points detected from the first image is greater than a threshold, the determination means determines the state indicated by the first information as the state of the head-mounted display device. 8. The head-mounted display device according to any one of configurations 2 to 7, wherein: (Configuration 9) The head-mounted display device described in any one of configurations 2 to 8, characterized in that the judgment means judges the state of the head-mounted display device based on the first information and the second information when the number of feature points detected from the first image is smaller than a threshold value. (Configuration 10) The first acquisition means estimates the state of the head mounted display device based on distance information corresponding to the first image. 10. The head-mounted display device according to any one of configurations 2 to 9, (Configuration 11) the projection means repeats projecting and not projecting the pattern image; The first acquisition means estimates the state of the head mounted display device based on the first image acquired during a period in which the pattern image is not projected. 10. The head-mounted display device according to any one of configurations 2 to 9, (Configuration 12) 12. The head-mounted display device according to any one of configurations 2 to 11, wherein the first acquisition means estimates the state of the head-mounted display device by SLAM. (Configuration 13) The pattern image represents a dot pattern. 13. The head-mounted display device according to any one of configurations 1 to 12. (Configuration 14) The first imaging means is a stereo camera. 14. The head-mounted display device according to any one of configurations 1 to 13. (Configuration 15) The second imaging means is provided at a position having lower rigidity than a position at which the first imaging means is provided. 15. The head-mounted display device according to any one of configurations 1 to 14. (Configuration 16) The first imaging means is provided in front of the head-mounted display device. 16. The head-mounted display device according to any one of configurations 1 to 15. (Configuration 17) The second imaging means is provided on a side surface of the head-mounted display device. 17. The head-mounted display device according to any one of configurations 1 to 16. (Configuration 18) The projection means projects the pattern image over the entire imaging range of the first imaging means, The second imaging means captures an image of a range that does not overlap with the imaging range of the first imaging means. 18. The head-mounted display device according to any one of configurations 1 to 17. (Configuration 19) The frame rate of the image captured by the first imaging means is the same as the frame rate of the image captured by the second imaging means. 19. The head-mounted display device according to any one of configurations 1 to 18. (Configuration 20) The first imaging means is a global shutter type imaging means. 20. The head-mounted display device according to any one of configurations 1 to 19. (Configuration 21) The first imaging means is a rolling shutter type imaging means. 21. The head-mounted display device according to any one of configurations 1 to 20. (Configuration 22) The second imaging means is a global shutter type imaging means. 22. The head-mounted display device according to any one of configurations 1 to 21, (Configuration 23) A state determination device usable with a head-mounted display device having a first imaging means and a second imaging means, a first estimation means for estimating a state including at least one of a position and an orientation of the head mounted display device based on a first image captured by the first imaging means; a second estimation means for estimating a change in the state of the head mounted display device per predetermined time based on a second image captured by the second imaging means; a determination means for determining the state of the head-mounted display device based on an estimation result of the first estimation means and an estimation result of the second estimation means; A state determination device comprising: (Configuration 24) The first imaging means; The second imaging means; A state determination device according to claim 23, A head-mounted display device comprising: (Method 1) A method for controlling a head-mounted display device having a first imaging means, a projection means for projecting a pattern image into an imaging range of the first imaging means, and a second imaging means for imaging an area not overlapping with the projection range of the pattern image, comprising: a first acquisition step of acquiring first information regarding a state including at least one of a position and an orientation of the head-mounted display device based on a first image captured by the first imaging means; a second acquisition step of acquiring second information regarding the state of the head mounted display device based on a second image captured by the second imaging means; a determination step of determining the state of the head-mounted display device based on the first information and the second information; A control method comprising the steps of: (Method 2) A control method for a state determination device usable with a head-mounted display device having a first imaging means and a second imaging means, comprising: a first estimation step of estimating a state including at least one of a position and an orientation of the head-mounted display device based on a first image captured by the first imaging means; a second estimation step of estimating a change in the state of the head-mounted display device per predetermined time based on a second image captured by the second imaging means; a determination step of determining the state of the head-mounted display device based on an estimation result of the first estimation step and an estimation result of the second estimation step; A control method comprising the steps of: (program) A program for causing a computer to execute each step of the control method according to Method 1 or 2. [Explanation of symbols]

[0057] 100: HMD 101: Stereo camera 102: Projection unit 103: Side camera 105: CPU

Claims

**Claim 1**: A head-mounted display device, comprising: a first imaging means provided on the front; a second imaging means provided on the side; a first acquisition means for acquiring first information regarding a state including at least one of the position and orientation of the head-mounted display device based on an image captured by the first imaging means; a second acquisition means for acquiring second information regarding the state based on an image captured by the second imaging means; a generation means for generating an image of a virtual object; and when the first acquisition means cannot acquire the first information with an accuracy higher than a predetermined accuracy, the generation means generates the image of the virtual object based on the second information. A head-mounted display device characterized by the above. **Claim 2**: The first acquisition means acquires the first information by detecting features of the real space from an image captured by the first imaging means. The head-mounted display device according to claim 1, characterized by the above. **Claim 3**: The case where the first acquisition means cannot acquire the first information with an accuracy higher than a predetermined accuracy means that the first acquisition means cannot track the features of the real space or misdetects the features of the real space. The head-mounted display device according to claim 2, characterized by the above. **Claim 4**: The device has a projection means provided on the front, and the projection means projects invisible light. The head-mounted display device according to any one of claims 1 to 3, characterized by the above. **Claim 5**: The first acquisition means acquires the first information based on an image captured by the first imaging means during a period when the invisible light is not projected. The head-mounted display device according to claim 4, characterized by the above. **Claim 6**: The device has a distance measuring means for acquiring distance information of a subject, and the distance measuring means acquires the distance information based on an image captured by the first imaging means during a period when the invisible light is projected. The head-mounted display device according to claim 4 or 5, characterized by the above. **Claim 7**: The first acquisition means acquires the first information based on an image captured by the first imaging means in which the invisible light does not appear. The head-mounted display device according to claim 4, characterized by the above. **Claim 8**: The device has a distance measuring means for acquiring distance information of a subject, and the distance measuring means acquires the distance information based on an image captured by the first imaging means in which the invisible light appears. The head-mounted display device according to claim 4 or 7, characterized in that...

9. The second imaging means includes a plurality of side cameras, and the second acquisition means acquires information regarding the state by individually using the captured images of the plurality of side cameras. The head-mounted display device according to any one of claims 1 to 8, characterized in that...

10. The plurality of side cameras include a side camera provided on the left side surface and a side camera provided on the right side surface. The head-mounted display device according to claim 9, characterized in that...

11. A head-mounted display device, imaging means provided on the front, projection means provided on the front and projecting invisible light, acquisition means for acquiring position and orientation information of the head-mounted display device based on an image captured by the imaging means, distance measurement means for acquiring distance information of a subject based on an image captured by the imaging means, comprising: the acquisition means acquires the position and orientation information based on an image captured by the imaging means during a period when the invisible light is not projected; the distance measurement means acquires the distance information based on an image captured by the imaging means during a period when the invisible light is projected. The head-mounted display device, characterized in that...

12. A head-mounted display device, imaging means provided on the front, projection means provided on the front and projecting invisible light, acquisition means for acquiring position and orientation information of the head-mounted display device based on an image captured by the imaging means, distance measurement means for acquiring distance information of a subject based on an image captured by the imaging means, comprising: the acquisition means acquires the position and orientation information based on an image captured by the imaging means in which the invisible light does not appear; the distance measurement means acquires the distance information based on an image captured by the imaging means in which the invisible light appears. The head-mounted display device, characterized in that...

13. A control method for a head-mounted display device having first imaging means provided on the front and second imaging means provided on the side, a first acquisition step of acquiring first information regarding a state including at least one of the position and orientation of the head-mounted display device based on an image captured by the first imaging means. A second acquisition step of acquiring second information regarding the state based on an image captured by the second imaging means; A generation step of generating an image of a virtual object; comprising: In the generation step, when the first information cannot be acquired with higher accuracy than a predetermined accuracy in the first acquisition step, an image of the virtual object is generated based on the second information A control method characterized by the above.

14. A control method for a head-mounted display device having an imaging means provided on the front and a projection means provided on the front for projecting invisible light, An acquisition step of acquiring position and orientation information of the head-mounted display device based on an image captured by the imaging means; A distance measurement step of acquiring distance information of a subject based on an image captured by the imaging means; comprising: In the acquisition step, the position and orientation information is acquired based on an image captured by the imaging means during a period when the invisible light is not projected. In the distance measurement step, the distance information is acquired based on an image captured by the imaging means during a period when the invisible light is projected. A control method characterized by the above.

15. A control method for a head-mounted display device having an imaging means provided on the front and a projection means provided on the front for projecting invisible light, An acquisition step of acquiring position and orientation information of the head-mounted display device based on an image captured by the imaging means; A distance measurement step of acquiring distance information of a subject based on an image captured by the imaging means; comprising: In the acquisition step, the position and orientation information is acquired based on an image captured by the imaging means in which the invisible light does not appear. In the distance measurement step, the distance information is acquired based on an image captured by the imaging means in which the invisible light appears. A control method characterized by the above.

16. A program for causing a computer to execute each step of the control method according to any one of Claims 13 to 15.

Images