Head-mounted display device, state determination device, method for controlling head-mounted display device, method for controlling state determination device, and program

Abstract

The head-mounted display device according to the present invention comprises: a first imaging unit; a projection unit configured to project a pattern image onto an imaging range of the first imaging unit; a second image capturing unit configured to capture a range that does not overlap with a projection range of the pattern image; a first acquisition unit configured to acquire first information related to a state including at least one of a position and a posture of the head-mounted display device based on a first image captured by the first imaging unit; a second acquisition unit configured to acquire second information related to a state of the head-mounted display device based on a second image captured by the second imaging unit; and a determination unit configured to determine a state of the head-mounted display device based on the first information and the second information.

Description

Technical Field

[0001] The present invention relates to a head mounted display device, a state determination device, a control method of the head mounted display device, a control method of the state determination device, and a program. Background Art

[0002] As a head-mounted display device (for example, HMD (head-mounted display)), a display device capable of estimating the position and posture of the head-mounted display device based on an image (captured image) captured by a camera has been proposed. For example, a technology related to such a head-mounted display device is disclosed in Patent Document 1. By using the estimated position and posture, an image can be appropriately displayed to the user. For example, an image in a virtual space or a virtual object in a real space can be synthesized without causing a sense of strangeness.

[0003] [Citation list]

[0004] [Patent Document]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2021-009557 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] However, in the case of a conventional head-mounted display device, it may not be possible to estimate the position and posture of the head-mounted display device with high accuracy. For example, when a user wearing the head-mounted display device quickly moves his / her head, a feature point in the real space tracked by a camera may exceed the camera's imaging range, or motion blur of the feature point may occur in the captured image. As a result, the feature point cannot be tracked, and the position and posture of the head-mounted display device cannot be estimated with high accuracy.

[0008] The present invention provides a technology capable of estimating (judging) at least one of the position and posture of a head mounted display device with high accuracy.

[0009] Solutions for solving problems

[0010] In its first aspect, the present invention provides a head-mounted display device, which includes: a first camera unit; a projection unit, which is configured to project a pattern image onto a camera range of the first camera unit; a second camera unit, which is configured to capture a range that does not overlap with the projection range of the pattern image; a first acquisition unit, which is configured to acquire first information related to a state of the head-mounted display device including at least one of a position and a posture based on a first image captured by the first camera unit; a second acquisition unit, which is configured to acquire second information related to the state of the head-mounted display device based on a second image captured by the second camera unit; and a judgment unit, which is configured to judge the state of the head-mounted display device based on the first information and the second information.

[0011] In its second aspect, the present invention provides a state judgment device that can be used together with a head-mounted display device including a first camera unit and a second camera unit, the state judgment device including: a first estimation unit, which is configured to estimate the state of the head-mounted display device including at least one of a position and a posture based on a first image taken by the first camera unit; a second estimation unit, which is configured to estimate the change in the state of the head-mounted display device within a predetermined time based on a second image taken by the second camera unit; and a judgment unit, which is configured to judge the state of the head-mounted display device based on the estimation result of the first estimation unit and the estimation result of the second estimation unit.

[0012] In a third aspect, the present invention provides a head mounted display device, which includes a first camera unit, a second camera unit and the above-mentioned state determination device.

[0013] In its fourth aspect, the present invention provides a control method for a head-mounted display device, wherein the head-mounted display device includes a first camera unit, a projection unit configured to project a pattern image onto a camera range of the first camera unit, and a second camera unit configured to capture a range that does not overlap with the projection range of the pattern image. The control method includes: a first acquisition step for acquiring first information related to a state of the head-mounted display device including at least one of a position and a posture based on a first image captured by the first camera unit; a second acquisition step for acquiring second information related to the state of the head-mounted display device based on a second image captured by the second camera unit; and a judgment step for judging the state of the head-mounted display device based on the first information and the second information.

[0014] In its fifth aspect, the present invention provides a control method for a state judgment device, which can be used together with a head-mounted display device including a first camera unit and a second camera unit, and the control method includes: a first estimation step for estimating the state of the head-mounted display device including at least one of a position and a posture based on a first image taken by the first camera unit; a second estimation step for estimating the change in the state of the head-mounted display device within a predetermined time based on a second image taken by the second camera unit; and a judgment step for judging the state of the head-mounted display device based on the estimation result of the first estimation step and the estimation result of the second estimation step.

[0015] In a sixth aspect, the present invention provides a program that causes a computer to execute the steps of the above-mentioned control method.

[0016] Effects of the Invention

[0017] According to the present invention, at least one of the position and the posture of the head mounted display device can be estimated (judged) with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A and Figure 1 B is an external view of the HMD according to this embodiment.

[0019] Figure 2 is a block diagram showing the functional configuration of the CPU according to the present embodiment.

[0020] Figure 3 is a block diagram showing a functional configuration of a state estimation unit according to the present embodiment.

[0021] Figure 4 is a flowchart of the state estimation process according to the present embodiment. DETAILED DESCRIPTION

[0022] An embodiment of the present invention will be described. In this embodiment, an example in which the present invention is applied to an HMD (head mounted display) will be described, but the present invention can also be applied to other head mounted display devices. For example, a handheld display device that a user holds in his / her hand and wears (places) on his / her head is a head mounted display device, and the present invention is also applicable to the handheld display device. The present invention is also applicable to a head mounted display device in which a user views an image with both eyes and a head mounted display device in which a user views an image with one eye.

[0023] The present invention can be applied to both video see-through type and optical see-through type. In the case of the video see-through type, the head-mounted display device displays an image of the external world (real space) by synthesizing graphics (e.g., virtual objects) as needed. In this case, the user cannot see the real space directly, but can indirectly see the real space or see graphics synthesized with the image of the real space by watching the displayed image. In the case of the optical see-through type, the head-mounted display device has lenses similar to those of ordinary glasses, and projects graphics onto the lenses as needed. In this case, the user can see the real space directly through the lenses, or see graphics projected onto the lenses. The present invention is also applicable to types that display images of virtual space without displaying images of real space. In other words, the present invention is applicable to various XR (cross reality) such as AR (augmented reality), MR (mixed reality) and VR (virtual reality).

[0024] The present invention is also applicable to electronic devices other than the head-mounted display device as long as the electronic device can be used together with the head-mounted display device. The image processing device to which the present invention is applied may be provided in the head-mounted display device or in an electronic device separate from the head-mounted display device. For example, the present invention is also applicable to a controller or a personal computer (PC) connected to the HMD by wire or wirelessly.

[0025] Figure 1 A and Figure 1 B is an external view of the HMD 100 according to the present embodiment. Figure 1 A is a front perspective view of the HMD 100, and Figure 1 FIG. B is a rear perspective view of the HMD 100. Figure 1 As shown in FIG. 1A , the HMD 100 is provided with a headband 110. The user places the HMD 100 in contact with the eyes and fixes the HMD 100 to the head with the headband 110. The HMD 100 has a stereo camera 101, a projection unit 102, a side camera 103, a left display unit 104L, and a right display unit 104R. The HMD 100 has a CPU (Central Processing Unit) 105, a ROM (Read Only Memory) 106, and a RAM (Random Access Memory) 107.

[0026] The stereo camera 101 is a camera (imaging device) that captures images of the outside world, and has a left imaging unit 101L and a right imaging unit 101R. For example, the stereo camera 101 is a global shutter camera (the left imaging unit 101L and the right imaging unit 101R are, for example, global shutter imaging units). The image obtained by the stereo camera 101 (each of the left imaging unit 101L and the right imaging unit 101R) is, for example, a grayscale image. The projection unit 102 faces the same direction as the stereo camera 101, and projects a pattern image to the imaging range of the stereo camera 101. In the present embodiment, the projection unit 102 projects a pattern image of invisible light (for example, a dot pattern of infrared light). The side camera 103 is also a camera that captures the outside world, and is, for example, a global shutter camera. The image obtained by the side camera 103 is also, for example, a grayscale image. The left display unit 104L is arranged to face the left eye of the user wearing the HMD 100, and displays an image to be viewed by the left eye of the user (display image, left display image). The right display unit 104R is arranged to face the right eye of the user wearing the HMD 100, and displays an image to be viewed by the right eye of the user (display image, right display image).

[0027] The CPU 105 is an information processing device (image processing device) that controls the entire HMD 100. The ROM 106 stores various data (e.g., various programs and various parameters). The RAM 107 also stores various data (including images captured by the stereo camera 101 and images captured by the side cameras 103). For example, the CPU 105 loads the program stored in the ROM 106 into the RAM 107 and executes the program.

[0028] Figure 2 20 is a block diagram showing a functional configuration of the CPU 105. The CPU 105 has a distance measurement unit 201, a state estimation unit 202, a graphic generation unit 203, and a display control unit 204.

[0029] The distance measurement unit 201 acquires two images with parallax (an image captured by the left camera unit 101L and an image captured by the right camera unit 101R) from the stereo camera 101. The distance measurement unit 201 then generates a distance map (distance information, depth map, depth information) from the two acquired images. The image used by the distance measurement unit 201 is an image in which the pattern image projected by the projection unit 102 is captured. The distance map is information indicating the distribution of the distance (depth) from the stereo camera 101 to the object, and is, for example, an image having depth values ​​as pixel values. Note that a monocular camera may be used instead of the stereo camera 101. For example, a ToF (Time of Flight) camera may be used. In this case, the image captured by the ToF camera may be used as a distance map. However, from the viewpoint of stability of various processes, the stereo camera 101 is more preferable than a monocular camera.

[0030] The state estimation unit 202 estimates the state of the HMD 100 based on the image captured by the stereo camera 101 (image captured by the stereo camera 101) and the image captured by the side camera 103 (image captured by the side camera 103). In the present embodiment, the state of the HMD 100 includes at least one of the position and posture of the HMD 100.

[0031] The graphic generation unit 203 generates a graphic. For example, the graphic generation unit 203 generates an image of a virtual object so that the virtual object is placed at a three-dimensional position based on the distance map generated by the distance measurement unit 201 and the state of the HMD 100 estimated by the state estimation unit 202 .

[0032] The display control unit 204 generates a left display image based on the image captured by the left camera unit 101L, and displays the left display image on the left display unit 104L. The display control unit 204 synthesizes the graphics (e.g., virtual objects) generated by the graphics generation unit 203 as needed, and generates a left display image in which the graphics are arranged. Similarly, the display control unit 204 generates a right display image based on the image captured by the right camera unit 101R, and displays the right display image on the right display unit 104R. The display control unit 204 synthesizes the graphics (e.g., virtual objects) generated by the graphics generation unit 203 as needed, and generates a right display image in which the graphics are arranged. Preferably, the image used by the display control unit 204 is an image in which the pattern image projected by the projection unit 102 does not appear.

[0033] Here, a configuration is considered in which the state of the HMD 100 is estimated based only on the image captured by the stereo camera 101. In such a configuration, the state of the HMD 100 may not be estimated with high accuracy. For example, when the user wearing the HMD 100 quickly moves his / her head, the feature points in the real space tracked by the stereo camera 101 may exceed the imaging range of the stereo camera 101, or motion blur of the feature points may occur in the image captured by the stereo camera 101. As a result, the feature points cannot be tracked, and the state of the HMD 100 cannot be estimated with high accuracy.

[0034] Therefore, in the present embodiment, the image captured by the side camera 103 is also used. The side camera 103 is arranged to capture an image of a range that does not overlap with the projection range of the pattern image. The projection range of the pattern image may be a part of the imaging range of the stereo camera 101, or may be the entire imaging range of the stereo camera 101. In the present embodiment, the projection unit 102 projects the pattern image onto the entire imaging range of the stereo camera 101. Therefore, the side camera 103 captures an image of a range that does not overlap with the imaging range of the stereo camera 101.

[0035] Therefore, the range that can be referred to when estimating the state of the HMD 100 is expanded by the imaging range of the side camera 103, so that the state of the HMD 100 can be estimated with high accuracy. For example, even if a feature point is outside the imaging range of the stereo camera 101, as long as the image of the feature point is captured by the side camera 103, the state of the HMD 100 can be estimated with high accuracy. In addition, even if motion blur of the feature point occurs in the image captured by the stereo camera 101, as long as motion blur of the feature point does not occur in the image captured by the side camera 103, the state of the HMD 100 can be estimated with high accuracy.

[0036] Note that in the present embodiment, both the stereo camera 101 and the side camera 103 are global shutter type cameras, but the present invention is not limited thereto. For example, a rolling shutter type camera may be used as the stereo camera 101. A rolling shutter type camera is cheaper than a global shutter type camera, and it is easy to increase the resolution. Therefore, by using a rolling shutter type camera as the stereo camera 101, the price of the HMD 100 can be reduced, and the resolution of the displayed image can be improved. A rolling shutter type camera may also be used as the side camera 103. However, since the global shutter type is less likely to cause distortion in the captured image than the rolling shutter type, it is preferable to use a global shutter type camera from the viewpoint of distortion suppression. For example, it is preferable that the side camera 103 can capture an image with less distortion than the image captured by the stereo camera 101, so it is preferable to use a global shutter type camera as the side camera 103.

[0037] The frame rate of the stereo camera 101 and the frame rate of the side camera 103 are not particularly limited, but the two frame rates are preferably the same (for example, 60fps, 90fps, or 120fps). If the frame rate of the stereo camera 101 is the same as the frame rate of the side camera 103, the timing difference between the images taken by these cameras can be reduced, thereby enabling more appropriate processing.

[0038] Figure 3 2 is a block diagram showing a functional configuration of the state estimation unit 202. The state estimation unit 202 has a first estimation unit 301, a feature detection unit 302, a second estimation unit 303, and a state judgment unit 304.

[0039] The first estimation unit 301 acquires first information related to the state of the HMD 100 based on the image captured by the stereo camera 101. In this embodiment, the first estimation unit 301 estimates the state of the HMD 100 based on the image captured by the stereo camera 101, thereby acquiring first information indicating the estimated state. For example, the first estimation unit 301 estimates the state of the HMD 100 through SLAM (Simultaneous Localization and Mapping).

[0040] Note that the first information may be information related to the state of the HMD 100, and may be information indicating, for example, an amount of change from a predetermined state. The first estimation unit 301 may estimate the state of the HMD 100 using distance information corresponding to an image captured by the stereo camera 101 (a distance map generated by the distance measurement unit 201). For example, the first estimation unit 301 may estimate the state of the HMD 100 through RGB-D SLAM. The first estimation unit 301 may estimate the state of the HMD 100 using only the distance information.

[0041] When the pattern image projected by the projection unit 102 appears in the image captured by the stereo camera 101, the features (e.g., points) of the pattern image may be erroneously detected as features (e.g., feature points) of the real space, and the state of the HMD 100 may not be estimated with high accuracy. For this reason, the projection unit 102 may repeatedly perform projection and non-projection of the pattern image. Then, the first estimation unit 301 may estimate the state of the HMD 100 based on the image captured by the stereo camera 101 obtained during the period in which the pattern image is not projected.

[0042] Note that even during the period of projecting the pattern image, the state of the HMD 100 can be estimated by odometry using the image captured by the side camera 103. For example, the present state of the HMD 100 can be estimated by adding a change in the state of the HMD 100 corresponding to a change over time in the image captured by the side camera 103 (e.g., a motion vector of a feature point) to the previous state (estimated state) of the HMD 100.

[0043] The feature detection unit 302 detects features from the image captured by the side camera 103. The features to be detected may be features related to the real space. In the present embodiment, the feature detection unit 302 detects feature points from the image captured by the side camera 103. For example, feature points are points (part or all of an object) that exist in the real space. A specific object for indicating feature points may be installed, or part or all of an already existing object may be detected as feature points. Various known techniques (e.g., KLT (Kanade-Lucas-Tomasi) tracker or goodFeatures To Track) may be used to detect features.

[0044] The second estimation unit 303 acquires second information related to the state of the HMD 100 based on the image captured by the side camera 103. The second information may be information related to the state of the HMD 100, and may or may not be information indicating the state of the HMD 100.

[0045] As described above, by further using the image captured by the side camera 103, the state of the HMD 100 can be estimated with high accuracy. However, since the range in which the side camera 103 can be arranged is limited, the side camera 103 may be set at a position with low rigidity. If the side camera 103 is set at a position with low rigidity, the relative position of the side camera 103 with respect to the stereo camera 101 may change due to temperature or the passage of time. In a configuration in which the state of the HMD 100 is estimated based on the image captured by the side camera 103, if the relative position of the side camera 103 changes, the accuracy of the state estimation decreases. If the side camera 103 is set in a member different from the member in which the stereo camera 101 is set, the relative position of the side camera 103 may also change due to the misalignment between these members.

[0046] In this embodiment ( Figure 1), the stereo camera 101 is provided on the front side of the HMD 100, and the side camera 103 is provided on the side side of the HMD 100. The side side of the HMD 100 is a side that the user holds when wearing the HMD 100, and is generally made of a material having lower rigidity than the front side to reduce the overall weight of the HMD 100. Even if the material of the front side and the material of the side side are the same, the rigidity of the end of the material is generally lower than the rigidity of the center of the material. Therefore, in the present embodiment, the side camera 103 is provided at a position having lower rigidity than the position where the stereo camera 101 is provided. Note that the arrangement of the stereo camera 101 and the side camera 103 is not limited to the above arrangement.

[0047] Therefore, in the present embodiment, the second estimation unit 303 estimates the state change of the HMD 100 in a predetermined time based on the image captured by the side camera 103, and acquires the second information indicating the estimated state change. For example, the second estimation unit 303 estimates the state change of the HMD 100 based on the change of the feature detected by the feature detection unit 302 over time (for example, the motion vector of the feature point). Since the change of the relative position of the side camera 103 is considered not to occur (or is small) within a predetermined time (short time), the state change of the HMD 100 can be estimated with high accuracy. In addition, since the change of the relative position of the side camera 103 can be ignored, even if the relative position of the side camera 103 changes, it is not necessary to calibrate the state estimation of the HMD 100, and high convenience can be achieved.

[0048] Note that in this embodiment ( Figure 1 A and Figure 1 B), the side cameras 103 are provided on each of the left side and the right side of the HMD 100, but the number of the side cameras 103 may be one, three, or more than three. For example, when a plurality of side cameras 103 are provided, the second estimation unit 303 estimates a plurality of state changes using images respectively captured by the respective side cameras 103, and determines the average of the plurality of state changes as the final state change.

[0049] In addition, as long as the relative position change of the side camera 103 can be basically ignored, the predetermined time is not particularly limited, but it is preferable that the predetermined time is short. Therefore, in this embodiment, the second estimation unit 303 estimates the state change of the HMD 100 from the previous imaging of the side camera 103 based on the image captured by the side camera 103.

[0050] The state judgment unit 304 judges the state of the HMD 100 based on the first information and the second information. In general, in a configuration in which the state of the HMD 100 is estimated based on only the image captured by the stereo camera 101, the more feature points are detected from the image captured by the stereo camera 101, the higher the accuracy of the state estimation. Therefore, in the present embodiment, when the number of feature points detected from the image captured by the stereo camera 101 is greater than a threshold value, the state judgment unit 304 judges the state indicated by the first information (the state estimated by the first estimation unit 301) as the state of the HMD 100. Then, when the number of feature points detected from the image captured by the stereo camera 101 is less than the threshold value, the state judgment unit 304 judges the state of the HMD 100 based on the first information and the second information. In the present embodiment, the state judgment unit 304 judges the current state of the HMD 100 by adding the state change estimated by the second estimation unit 303 (the state change indicated by the current second information) to the state judged at a timing earlier than the current by a predetermined time. The state determined at a timing that is a predetermined time earlier than the current state (by the state determination unit 304 ) is, for example, a state indicated by the first information obtained at the timing.

[0051] Note that the method of judging the state of the HMD 100 is not limited to the above method. For example, the state judgment unit 304 may weight and synthesize the state change estimated by the second estimation unit 303 and the state change estimated by the first estimation unit 301 during the period from the timing advanced by a predetermined time until the present (the state change indicated by the first information during the period). The state judgment unit 304 can judge the present state of the HMD 100 by adding the change obtained by weighted synthesis to the state judged at the timing advanced by a predetermined time from the present. By performing weighted synthesis, a sudden change in the state estimation result before and after using the second information can be suppressed.

[0052] In the weighted synthesis, the state judgment unit 304 may use weights based on the number of feature points detected from the image captured by the stereo camera 101 and the number of feature points detected from the image captured by the side camera 103. For example, as the difference value obtained by subtracting the number of feature points of the image captured by the stereo camera 101 from the number of feature points of the image captured by the side camera 103 increases, the state judgment unit 304 increases the weight of the state change estimated by the first estimation unit 301. As the difference value decreases, the state judgment unit 304 increases the weight of the state change estimated by the second estimation unit 303.

[0053] Figure 4 : is a flowchart of the state estimation process of the HMD 100. Figure 4The state estimation process in is implemented by the CPU 105 loading the program stored in the ROM 106 into the RAM 107 and executing the program. For example, when the HMD 100 is started, Figure 4 The CPU 105 repeatedly executes Figure 4 In the first state estimation process, for example, the state of the HMD 100 is determined by the process of step S409.

[0054] In step S401, the CPU 105 determines whether a pattern image is projected by the projection unit 102. If the CPU 105 determines that a pattern image is projected, the process proceeds to step S402, and if the CPU 105 determines that a pattern image is not projected, the process proceeds to step S405.

[0055] In step S402, the CPU 105 (feature detection unit 302) acquires an image captured by the side camera 103. In step S403, the CPU 105 (feature detection unit 302) detects feature points from the captured image acquired in step S402 (the image captured by the side camera 103). In step S404, the CPU 105 (second estimation unit 303) estimates a state change of the HMD 100 based on a change over time (motion vector) of the feature point detected in step S403. The CPU 105 then advances the process to step S412.

[0056] In step S405, the CPU 105 (first estimation unit 301) acquires an image captured by the stereo camera 101. In step S406, the CPU 105 (feature detection unit 302) acquires an image captured by the side camera 103. In step S407, the CPU 105 (first estimation unit 301) estimates the state of the HMD 100 by SLAM using the captured image acquired in step S405 (the image captured by the stereo camera 101). The process of step S407 includes a process of detecting feature points from the image captured by the stereo camera 101. In step S408, the CPU 105 (feature detection unit 302) detects feature points from the captured image acquired in step S406 (the image captured by the side camera 103). In step S409, the CPU 105 (second estimation unit 303) estimates the state change of the HMD 100 based on the change over time (motion vector) of the feature point detected in step S408. Thereafter, the CPU 105 advances the process to step S410.

[0057] In step S410, the CPU 105 (state determination unit 304) determines whether the number of feature points detected in step S407 from the image captured by the stereo camera 101 is equal to or greater than a threshold value. If the CPU 105 determines that the number of feature points is equal to or greater than the threshold value, the process proceeds to step S411. If the CPU 105 determines that the number of feature points is less than the threshold value, the process proceeds to step S412. Note that if the number of feature points is equal to the threshold value, the process may proceed to step S412.

[0058] In step S411 , the CPU 105 (state determination unit 304 ) determines that the state estimated by the first estimation unit 301 in step S407 is the state of the HMD 100 .

[0059] In step S412 , the CPU 105 (state determination unit 304 ) determines the state of the HMD 100 by adding the state change estimated by the second estimation unit 303 in step S404 or step S409 to the state determined last by the state determination unit 304 .

[0060] As described above, according to the present embodiment, at least one of the position and posture of the head mounted display device can be estimated (judged) with high accuracy. For example, by using the side camera 103 arranged so as to capture an image with a range that does not overlap with the projection range of the pattern image in addition to the stereo camera 101, the range that can be referred to when estimating the state of the HMD 100 can be expanded. As a result, the state of the HMD 100 can be estimated with high accuracy.

[0061] Furthermore, according to the present embodiment, the state of the HMD 100 is estimated based on the images captured by the stereo camera 101, but the change in the state of the HMD 100 in a predetermined time is estimated based on the images captured by the side cameras 103. The state of the HMD 100 is then determined based on these two estimation results. By estimating the change in the state of the HMD 100 in a predetermined time based on the images captured by the side cameras 103, it is possible to suppress the degradation of the state estimation accuracy of the HMD 100 caused by the change in the relative position of the side cameras 103.

[0062] The above-described embodiments (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.

[0063] <Other embodiments>

[0064] The present invention can also be implemented by supplying a program for implementing one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium and causing one or more processors in a computer of the system or device to read and execute the program. In addition, the present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0065] The disclosure of this embodiment includes the following configurations, methods, and programs.

[0066] (Configuration 1)

[0067] A head mounted display device, comprising:

[0068] A first camera unit;

[0069] a projection unit, configured to project the pattern image onto a photographing range of the first photographing unit;

[0070] a second camera unit configured to capture a range that does not overlap with a projection range of the pattern image;

[0071] a first acquisition unit configured to acquire first information related to a state including at least one of a position and a posture of the head mounted display device based on a first image captured by the first camera unit;

[0072] a second acquiring unit configured to acquire second information related to a state of the head mounted display device based on a second image captured by the second camera unit; and

[0073] A determination unit configured to determine a state of the head mounted display device based on the first information and the second information.

[0074] (Configuration 2)

[0075] The head mounted display device according to configuration 1, wherein:

[0076] The first acquisition unit acquires the first information indicating the estimated state by estimating the state of the head mounted display device based on the first image.

[0077] (Configuration 3)

[0078] The head mounted display device according to configuration 2, wherein:

[0079] The second acquisition unit acquires the second information indicating the estimated change by estimating a change in the state of the head mounted display device in a predetermined time based on the second image.

[0080] (Configuration 4)

[0081] The head mounted display device according to configuration 3, wherein:

[0082] The second acquisition unit acquires the second information indicating the estimated change by estimating a change in the state of the head mounted display device from a previous photographing by the second imaging unit based on the second image.

[0083] (Configuration 5)

[0084] The head mounted display device according to configuration 3 or 4, wherein:

[0085] The determination unit determines the state at the current timing by adding the change indicated by the current second information to the state indicated by the first information acquired at a timing earlier than the current timing by the predetermined time.

[0086] (Configuration 6)

[0087] The head mounted display device according to configuration 3 or 4, wherein:

[0088] The judgment unit judges the state of the current timing by adding the following change to the state indicated by the first information obtained at a timing earlier than the current timing by the predetermined time, wherein the change is obtained by weighting and synthesizing the change indicated by the second information of the current timing and the change of the state indicated by the first information during the period from the timing earlier than the predetermined time until the current timing.

[0089] (Configuration 7)

[0090] The head mounted display device according to configuration 6, wherein:

[0091] The determination unit uses a weight based on the number of feature points detected from the first image and the number of feature points detected from the second image in weighted sum synthesis.

[0092] (Configuration 8)

[0093] A head mounted display device according to any one of configurations 2 to 7, wherein:

[0094] The determination unit determines that the state indicated by the first information is the state of the head mounted display device when the number of feature points detected from the first image is greater than a threshold value.

[0095] (Configuration 9)

[0096] A head mounted display device according to any one of configurations 2 to 8, wherein:

[0097] The determination unit determines the state of the head mounted display device based on the first information and the second information when the number of feature points detected from the first image is smaller than a threshold value.

[0098] (Configuration 10)

[0099] A head mounted display device according to any one of configurations 2 to 9, wherein:

[0100] The first acquisition unit estimates a state of the head mounted display device based on distance information corresponding to the first image.

[0101] (Configuration 11)

[0102] A head mounted display device according to any one of configurations 2 to 9, wherein:

[0103] The projection unit repeatedly projects and does not project the pattern image, and

[0104] The first acquisition unit estimates a state of the head mounted display device based on the first image captured during a period in which the pattern image is not projected.

[0105] (Configuration 12)

[0106] A head mounted display device according to any one of configurations 2 to 11, wherein:

[0107] The first acquisition unit estimates the state of the head mounted display device through SLAM.

[0108] (Configuration 13)

[0109] A head mounted display device according to any one of configurations 1 to 12, wherein:

[0110] The pattern image represents a dot pattern.

[0111] (Configuration 14)

[0112] A head mounted display device according to any one of configurations 1 to 13, wherein:

[0113] The first camera unit is a stereo camera.

[0114] (Configuration 15)

[0115] A head mounted display device according to any one of configurations 1 to 14, wherein:

[0116] The second imaging unit is provided at a position having lower rigidity than a position at which the first imaging unit is provided.

[0117] (Configuration 16)

[0118] A head mounted display device according to any one of configurations 1 to 15, wherein:

[0119] The first camera unit is arranged on the front side of the head mounted display device.

[0120] (Configuration 17)

[0121] A head mounted display device according to any one of configurations 1 to 16, wherein:

[0122] The second camera unit is arranged on a side of the head mounted display device.

[0123] (Configuration 18)

[0124] A head mounted display device according to any one of configurations 1 to 17, wherein:

[0125] The projection unit projects the pattern image onto the entire imaging range of the first imaging unit, and

[0126] The second camera unit captures a range that does not overlap with a range captured by the first camera unit.

[0127] (Configuration 19)

[0128] A head mounted display device according to any one of configurations 1 to 18, wherein:

[0129] The frame rate of the photography performed by the first camera unit is the same as the frame rate of the photography performed by the second camera unit.

[0130] (Configuration 20)

[0131] A head mounted display device according to any one of configurations 1 to 19, wherein:

[0132] The first imaging unit is a global shutter type imaging unit.

[0133] (Configuration 21)

[0134] A head mounted display device according to any one of configurations 1 to 20, wherein:

[0135] The first camera unit is a rolling shutter type camera unit.

[0136] (Configuration 22)

[0137] A head mounted display device according to any one of configurations 1 to 21, wherein:

[0138] The second imaging unit is a global shutter type imaging unit.

[0139] (Configuration 23)

[0140] A state determination device, which can be used together with a head mounted display device including a first camera unit and a second camera unit, the state determination device comprising:

[0141] a first estimating unit configured to estimate a state including at least one of a position and a posture of the head mounted display device based on the first image captured by the first camera unit;

[0142] a second estimating unit configured to estimate a change in a state of the head mounted display device in a predetermined time based on a second image captured by the second camera unit; and

[0143] A determination unit configured to determine a state of the head mounted display device based on an estimation result of the first estimation unit and an estimation result of the second estimation unit.

[0144] (Configuration 24)

[0145] A head mounted display device, comprising:

[0146] the first camera unit;

[0147] the second camera unit; and

[0148] A state judgment device according to configuration 23.

[0149] (Method 1)

[0150] A control method for a head mounted display device, the head mounted display device comprising a first camera unit, a projection unit configured to project a pattern image onto a camera range of the first camera unit, and a second camera unit configured to capture a range that does not overlap with the projection range of the pattern image, the control method comprising:

[0151] A first acquisition step for acquiring first information related to a state of at least one of a position and a posture of the head mounted display device based on a first image captured by the first camera unit;

[0152] a second acquisition step for acquiring second information related to the state of the head mounted display device based on the second image captured by the second camera unit; and

[0153] A determination step for determining a state of the head mounted display device based on the first information and the second information.

[0154] (Method 2)

[0155] A control method of a state judgment device, wherein the state judgment device can be used together with a head mounted display device including a first camera unit and a second camera unit, the control method comprising:

[0156] A first estimating step, for estimating a state of the head mounted display device including at least one of a position and a posture based on a first image captured by the first camera unit;

[0157] a second estimating step for estimating a change in the state of the head mounted display device in a predetermined time based on a second image captured by the second camera unit; and

[0158] A determination step for determining a state of the head mounted display device based on an estimation result of the first estimation step and an estimation result of the second estimation step.

[0159] (program)

[0160] A program that causes a computer to execute each step of the control method according to method 1 or 2.

[0161] The present invention is not limited to the above-described embodiments, and various modifications and corrections may be made thereto without departing from the spirit and scope of the present invention. Therefore, the appended claims are attached to disclose the scope of the present invention.

[0162] This application claims priority based on Japanese Patent Application No. 2022-105685 filed on Jun. 30, 2022, the disclosure of which is hereby incorporated by reference in its entirety.

[0163] [reference numerals list]

[0164] 100 HMD

[0165] 101 Stereo Camera

[0166] 102 projection unit

[0167] 103 Side Camera

[0168] 105 CPU

Claims

1. A head mounted display device, comprising: A first camera unit; a projection unit, configured to project the pattern image onto a photographing range of the first photographing unit; a second camera unit configured to capture a range that does not overlap with a projection range of the pattern image; a first acquisition unit configured to acquire first information related to a state including at least one of a position and a posture of the head mounted display device based on a first image captured by the first camera unit; a second acquisition unit configured to acquire second information related to a state of the head mounted display device based on a second image captured by the second camera unit; as well as A determination unit configured to determine a state of the head mounted display device based on the first information and the second information.

2. The head mounted display device according to claim 1, wherein: The first acquisition unit acquires the first information indicating the estimated state by estimating the state of the head mounted display device based on the first image.

3. The head mounted display device according to claim 2, wherein: The second acquisition unit acquires the second information indicating the estimated change by estimating a change in the state of the head mounted display device in a predetermined time based on the second image.

4. The head mounted display device according to claim 3, wherein: The second acquisition unit acquires the second information indicating the estimated change by estimating a change in the state of the head mounted display device from a previous photographing by the second imaging unit based on the second image.

5. The head mounted display device according to claim 3 or 4, wherein: The determination unit determines the state at the current timing by adding the change indicated by the current second information to the state indicated by the first information acquired at a timing earlier than the current timing by the predetermined time.

6. The head mounted display device according to claim 3 or 4, wherein: The judgment unit judges the state of the current timing by adding the following change to the state indicated by the first information obtained at a timing earlier than the current timing by the predetermined time, wherein the change is obtained by weighting and synthesizing the change indicated by the second information of the current timing and the change of the state indicated by the first information during the period from the timing earlier than the predetermined time until the current timing.

7. The head mounted display device according to claim 6, wherein: The determination unit uses a weight based on the number of feature points detected from the first image and the number of feature points detected from the second image in weighted sum synthesis.

8. The head mounted display device according to claim 2, wherein: The determination unit determines that the state indicated by the first information is the state of the head mounted display device when the number of feature points detected from the first image is greater than a threshold value.

9. The head mounted display device according to claim 2, wherein: The determination unit determines the state of the head mounted display device based on the first information and the second information when the number of feature points detected from the first image is smaller than a threshold value.

10. The head mounted display device according to claim 2, wherein: The first acquisition unit estimates a state of the head mounted display device based on distance information corresponding to the first image.

11. The head mounted display device according to claim 2, wherein: The projection unit repeatedly projects and does not project the pattern image, and The first acquisition unit estimates a state of the head mounted display device based on the first image captured during a period in which the pattern image is not projected.

12. The head mounted display device according to claim 2, wherein: The first acquisition unit estimates the state of the head mounted display device through SLAM.

13. The head mounted display device according to claim 1 or 2, wherein: The pattern image represents a dot pattern.

14. The head mounted display device according to claim 1 or 2, wherein: The first camera unit is a stereo camera.

15. The head mounted display device according to claim 1 or 2, wherein: The second imaging unit is provided at a position having lower rigidity than a position at which the first imaging unit is provided.

16. The head mounted display device according to claim 1 or 2, wherein: The first camera unit is arranged on the front side of the head mounted display device.

17. The head mounted display device according to claim 1 or 2, wherein: The second camera unit is arranged on a side of the head mounted display device.

18. The head mounted display device according to claim 1 or 2, wherein: The projection unit projects the pattern image onto the entire imaging range of the first imaging unit, and The second camera unit captures a range that does not overlap with a range captured by the first camera unit.

19. The head mounted display device according to claim 1 or 2, wherein: The frame rate of the photography performed by the first camera unit is the same as the frame rate of the photography performed by the second camera unit.

20. The head mounted display device according to claim 1 or 2, wherein: The first imaging unit is a global shutter type imaging unit.

21. The head mounted display device according to claim 1 or 2, wherein: The first camera unit is a rolling shutter type camera unit.

22. The head mounted display device according to claim 1 or 2, wherein: The second imaging unit is a global shutter type imaging unit.

23. A state determination device, which can be used together with a head mounted display device including a first camera unit and a second camera unit, the state determination device comprising: a first estimating unit configured to estimate a state including at least one of a position and a posture of the head mounted display device based on the first image captured by the first camera unit; a second estimating unit configured to estimate a change in a state of the head mounted display device in a predetermined time based on a second image captured by the second camera unit; as well as A determination unit configured to determine a state of the head mounted display device based on an estimation result of the first estimation unit and an estimation result of the second estimation unit.

24. A head mounted display device, comprising: the first camera unit; the second camera unit; as well as A state determination device according to claim 23.

25. A control method for a head mounted display device, the head mounted display device comprising a first camera unit, a projection unit configured to project a pattern image onto a camera range of the first camera unit, and a second camera unit configured to capture a range that does not overlap with the projection range of the pattern image, the control method comprising: A first acquisition step for acquiring first information related to a state of at least one of a position and a posture of the head mounted display device based on a first image captured by the first camera unit; a second acquisition step for acquiring second information related to the state of the head mounted display device based on the second image captured by the second camera unit; as well as A determination step for determining a state of the head mounted display device based on the first information and the second information.

26. A control method for a state determination device, wherein the state determination device can be used together with a head mounted display device including a first camera unit and a second camera unit, the control method comprising: A first estimating step, for estimating a state of the head mounted display device including at least one of a position and a posture based on a first image captured by the first camera unit; a second estimating step for estimating a change in the state of the head mounted display device in a predetermined time based on a second image captured by the second camera unit; as well as A determination step for determining a state of the head mounted display device based on an estimation result of the first estimation step and an estimation result of the second estimation step.

27. A program causing a computer to execute each step of the control method according to claim 25 or 26.

Images