Information processing device

Abstract

A technique is provided that can determine (set) a coordinate system so that a virtual object is placed at a desired position. [Solution] The information processing device of the present invention is characterized by having a setting means for setting an origin selected by the user in three-dimensional space and setting a coordinate system including the origin and a first axis parallel to the direction of gravity as a coordinate system when presenting a virtual image to the user via a head-mounted display device.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, and more particularly to the presentation (display) of a virtual image. [Background technology]

[0002] In recent years, mixed reality (MR) technology and augmented reality (AR) technology have become known as technologies that seamlessly combine real space and virtual space in real time. There is also virtual reality (VR) technology, which provides only virtual space. These technologies are used, for example, in head-mounted displays (HMDs).

[0003] When presenting a virtual image to a user (the person wearing the HMD) via an HMD, a coordinate system is required to generate the virtual image. One method for determining this coordinate system has been proposed, in which the HMD detects a pattern (marker) on a real object and determines the coordinate system based on that pattern. Another method has been proposed, in which the HMD detects a plane in real space and determines the coordinate system based on that plane. Another method has been proposed, in which the HMD generates a map of real space and determines the coordinate system based on that map.

[0004] Furthermore, Patent Documents 1 and 2 disclose techniques for determining a coordinate system using the direction of gravity. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-40893 [Patent Document 2] International Publication No. 2020 / 157955 Summary of the Invention [Problem to be solved by the invention]

[0006] However, even if the techniques disclosed in Patent Documents 1 and 2 are used, only a coordinate system including an axis parallel to the direction of gravity is determined, and it may not be possible to present a virtual image so that a virtual object is positioned at a desired position.

[0007] An object of the present invention is to provide a technique that can determine (set) a coordinate system so that a virtual object is placed at a desired position. [Means for solving the problem]

[0008] A first aspect of the present invention is an information processing device characterized by having a setting means for setting an origin selected arbitrarily by a user in three-dimensional space, and setting a coordinate system including the origin and a first axis parallel to the direction of gravity as a coordinate system when presenting a virtual image to the user via a head-mounted display device.

[0009] A second aspect of the present invention is an information processing method characterized by having the steps of setting an origin selected by the user in three-dimensional space, and setting a coordinate system including the origin and a first axis parallel to the direction of gravity as a coordinate system when presenting a virtual image to the user via a head-mounted display device.

[0010] A third aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device. [Effects of the Invention]

[0011] According to the present invention, it is possible to determine (set) a coordinate system so that a virtual object is placed at a desired position. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an HMD. [Figure 2] 10 is a flowchart of the operation of the HMD. [Figure 3] FIG. 2 is a schematic diagram of a reference coordinate system according to the first embodiment. [Figure 4]FIG. 10 is a schematic diagram of a reference coordinate system according to the second embodiment. [Figure 5] FIG. 10 is a schematic diagram of a reference coordinate system according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] (First embodiment) A first embodiment of the present invention will be described below. FIG. 1 is a block diagram showing an example of the configuration of a head-mounted display (HMD) 100, which is an example of an information processing device according to the first embodiment and also an example of a head-mounted display device. The HMD 100 may be an HMD using mixed reality (MR) technology or augmented reality (AR) technology, or an HMD using virtual reality (VR) technology. The HMD 100 may be an HMD using a video see-through system or an HMD using an optical see-through system. When using MR technology or AR technology, the HMD 100 presents images of virtual objects as virtual images to a user (a person wearing the HMD 100). When using VR technology, the HMD 100 presents images of a virtual space and virtual objects as virtual images to a user.

[0014] In the following description, multiple processes performed by one component may be performed by multiple components. For example, multiple calculations performed by the calculation unit 12 may be performed by multiple calculation units (e.g., multiple calculation circuits).

[0015] The detection unit 11 detects information for calculating (estimating) the position and orientation of the HMD 100. For example, an acceleration sensor or an angular velocity sensor called an IMU (Inertial Measurement Unit) can be used as the detection unit 11. A camera that captures images of real space can also be used as the detection unit 11. A communication interface that acquires captured images from an external camera can also be used as the detection unit 11.

[0016] The calculation unit 12 determines a reference coordinate system, which is a coordinate system when a virtual image is presented to the user, and estimates the position and orientation of the HMD 100 relative to the reference coordinate system (the virtual image presented based on the reference coordinate system), using the information obtained by the detection unit 11. For example, the position and orientation of the HMD 100 are estimated by a self-position estimation method called SLAM (Simultaneous Localization And Mapping), which is a type of VIO (Visual Inertial Odometry).

[0017] The generation unit 13 generates a virtual image based on the calculation result by the calculation unit 12. For example, the generated virtual image is an image that represents how a virtual space or a virtual object at a predetermined position and orientation relative to a reference coordinate system looks from the estimated position and orientation of the HMD 100. The generation unit 13 is, for example, a GPU (Graphics Processing Unit) and renders a CG (Computer Graphics) virtual image.

[0018] The display unit 14 presents (displays) to the user the virtual image generated by the generation unit 13. When the user moves their head, the detection unit 11 detects the head movement, and the calculation unit 12 calculates the virtual image based on the detected head movement. Based on the results of the movement detection, a new position and orientation of the HMD 100 are estimated, and the generation unit 13 generates a new virtual image based on the new position and orientation of the HMD 100. The display unit 14 updates the displayed virtual image to the new virtual image, thereby providing the user with the sensation that the virtual space and virtual objects are actually present in that location.

[0019] In the case of the video see-through method, the generation unit 13 generates a composite image by combining a captured image of real space with a virtual image, and the display unit 14 presents the composite image. The captured image of real space used to generate the composite image may or may not be obtained by the detection unit 11. The HMD 100 may have an imaging unit (camera) separate from the detection unit 11 for obtaining the captured image of real space used to generate the composite image. The display unit 14 may be a liquid crystal display or an organic EL display placed in front of the eyes of the user (the person wearing the HMD 100), a projection device that projects an image onto a projection surface placed in front of the user's eyes, or a laser device that projects an image directly onto the user's retina.

[0020] The HMD 100 may have one display unit 14 corresponding to one eye, or two display units 14 corresponding to both eyes. When the HMD 100 has two display units 14, two images with parallax are displayed on the two display units 14, allowing the user to see the images in stereoscopic view.

[0021] FIG. 2 is a flowchart showing an example of the operation of the HMD 100, and FIG. 3 is a schematic diagram showing an example of the reference coordinate axes set (determined) by the operation of FIG.

[0022] In S21 of Fig. 2, the calculation unit 12 sets an origin (origin 300 of Fig. 3) arbitrarily selected by the user in three-dimensional space. The three-dimensional space may be a real space or a virtual space. When the distance from the user to the origin changes, the user's convergence angle also changes.

[0023] The timing for setting the origin is not particularly limited, but in the first embodiment, the origin is set at a timing instructed by the user. The user instructs the setting of the origin by, for example, pressing a physical button provided on the HMD 100, pressing a virtual button presented by the HMD 100, using voice, gestures, or the like. Also, in the first embodiment, the calculation unit 12 sets the origin at a position designated by the user. The method for designating the position is not particularly limited, and for example, the user designates the position of the origin by, for example, pressing a physical button provided on the HMD 100, pressing a virtual button presented by the HMD 100, or the like.

[0024] The method for setting the origin is not particularly limited. For example, the three-dimensional coordinates of the origin based on a predetermined three-dimensional coordinate system may be set as the information about the origin, or a combination of the two-dimensional coordinates of the origin in the image presented to the user and information about the distance from the user to the origin may be set.

[0025] Furthermore, in order to immediately present the origin to the user, it is preferable to set the origin in the three-dimensional space within the range perceived by the user via the HMD 100. In order to present the origin to the user with good visibility, it is preferable to set the origin on the focal plane of the display unit 14 (within the range in three-dimensional space where the image presented by the HMD 100 is in focus).

[0026] In S22, the calculation unit 12 detects the direction of gravity using the information obtained by the detection unit 11, and sets a first axis parallel to the direction of gravity. For example, the calculation unit 12 detects the 1G direction of the acceleration sensor as the direction of gravity. In order to reduce the influence of disturbances immediately after a user operation, the calculation unit 12 may detect the average direction of the 1G direction over a predetermined time as the direction of gravity. Then, the calculation unit 12 sets, as the first axis, an axis extending from the origin in the direction opposite to the direction of gravity (axis 301 in FIG. 3), or an axis extending from the origin in the direction of gravity. Axis 1 corresponds to the Y axis in standards such as OpenGL and OpenXR, or the Z axis in a coordinate system called Z-up.

[0027] In S23, the calculation unit 12 sets a second axis that is perpendicular to the first axis set in S22 and is parallel to a direction passing through the origin and the HMD 100 on a plane perpendicular to the first axis. For example, the calculation unit 12 sets, as the second axis, an axis (axis 302 in FIG. 3) that is tilted 90 degrees from the first axis and extends from the origin toward the user (HMD 100), or an axis that is tilted 90 degrees from the first axis and extends from the origin toward the opposite side from the user. The second axis corresponds to the Z axis in standards such as OpenGL and OpenXR.

[0028] In S24, the calculation unit 12 sets a third axis that is an axis perpendicular to the first axis set in S22 and the second axis set in S23. For example, the calculation unit 12 sets the third axis so that a right-handed coordinate system is defined by the origin, the first axis, the second axis, and the third axis. The calculation unit 12 may also set the third axis so that a left-handed coordinate system is defined by the origin, the first axis, the second axis, and the third axis. The third axis corresponds to the X-axis in the OpenGL or OpenXR standard.

[0029] The above process sets a reference coordinate system that includes the origin, the first axis, the second axis, and the third axis. This reference coordinate system defines six degrees of freedom of movement: translation parallel to the first axis, translation parallel to the second axis, translation parallel to the third axis, rotation around the first axis, rotation around the second axis, and rotation around the third axis.

[0030] The generation unit 13 generates a virtual image using the reference coordinate system set in this manner and the result of self-position estimation. For example, as described above, an image is generated as the virtual image, showing how a virtual space or a virtual object at a predetermined position and orientation relative to the reference coordinate system looks from the estimated position and orientation of the HMD 100. Note that the calculation unit 12 may start self-position estimation before setting the reference coordinate system, or may start self-position estimation after setting the reference coordinate system. As a result of the self-position estimation, information indicating the position and orientation of the HMD 100 relative to the predetermined coordinate system may be acquired, a reference coordinate system may be set, and a difference between the predetermined coordinate system and the reference coordinate system may be acquired (calculated). Using the calculated difference, the position and orientation of the HMD 100 relative to the predetermined coordinate system can be converted into the position and orientation of the HMD 100 relative to the reference coordinate system. After setting the reference coordinate system, information indicating the position and orientation of the HMD 100 relative to the reference coordinate system may be acquired by self-position estimation.

[0031] The calculation unit 12 may delete or reset the reference coordinate system at a timing instructed by the user. In the HMD 100, the virtual image may be hidden (not displayed) in response to the deletion of the reference coordinate system. This can improve convenience. The user instructs deletion or reset of the reference coordinate system, for example, by pressing a physical button provided on the HMD 100, pressing a virtual button presented by the HMD 100, using voice or gesture. For example, when the user points upward to instruct reset, the virtual image can be presented so that the upward direction is the front direction of the virtual image, allowing the user to view a lower part of the virtual image. When the user points downward to instruct reset, the virtual image can be presented so that the downward direction is the front direction of the virtual image, allowing the user to view a higher part of the virtual image.

[0032] (Second embodiment) The second embodiment of the present invention will be described below. Note that, in the following, the description of the configuration and processing that are the same as those of the first embodiment will be omitted, and only the configuration and processing that are different from those of the first embodiment will be described.

[0033] 4 is a schematic diagram showing an example of the reference coordinate axes set (determined) by the operation of FIG. 2. In the second embodiment, in S21 of FIG. 2, the calculation unit 12 detects the hand of the user (the person wearing the HMD 100) and sets the origin (origin 400 in FIG. 4) at the position indicated by the hand. The position indicated by the hand is, for example, the tip of the index finger or the center of the palm. The position indicated by the hand may be a position a predetermined distance away from the tip of the index finger in the direction in which the index finger extends, or the intersection of a line extending from the tip of the index finger in the direction in which the index finger extends and an object.

[0034] A user's hands can be detected in various ways. For example, hands can be detected using color and depth information from real space. Color information from real space can be acquired using a camera, and depth information (depth map) from real space can be acquired using a stereo camera or TOF (Time Of Flight) sensor. It is also possible to use AI (Artificial Intelligence) to detect skeletons (bone models) from real space and detect hands based on the detection results.

[0035] Thereafter, the first axis 401, the second axis 402, and the third axis 403 are set in the same manner as in the first embodiment.

[0036] (Third embodiment) The third embodiment of the present invention will be described below. Note that, in the following, descriptions of the configurations and processes that are the same as those of the first embodiment will be omitted, and only configurations and processes that are different from those of the first embodiment will be described.

[0037] FIG. 5 is a schematic diagram showing an example of the reference coordinate axes set (determined) by the operation of FIG. 2. In the third embodiment, in S21 of FIG. 2, the calculation unit 12 detects a plurality of feature points in real space, allows the user to arbitrarily select one of the plurality of feature points, and sets the origin (origin 500 in FIG. 5) at the position of the selected feature point. The method for detecting feature points is not particularly limited, and for example, feature points are detected from a captured image of real space. Edge positions are often detected as feature points. For example, the detected plurality of feature points are presented to the user, and the user designates (selects) one of the plurality of feature points by pressing a physical button provided on the HMD 100, pressing a virtual button presented by the HMD 100, using voice, gestures, or the like.

[0038] Thereafter, a first axis 501, a second axis 502, and a third axis 503 are set in the same manner as in the first embodiment.

[0039] According to the first to third embodiments described above, after the user arbitrarily selects the origin of the reference coordinate system, the axes of the reference coordinate system are set based on the direction of gravity, etc. In this way, the user can determine (set) the reference coordinate system so that a virtual object is placed at a desired position simply by selecting the origin.

[0040] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0041] The processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units). , ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), etc. Programmable logic devices include FPGA (Field Programmable Gate Array), CPLD (Complex Programmable Logic Device), etc.

[0042] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, the above-described embodiments merely illustrate one embodiment of the present invention, and the embodiments can be appropriately combined. For example, the present invention is not limited to HMDs, but can also be applied to various information processing devices such as personal computers, smartphones, tablet terminals, and HMD controllers.

[0043] (Other embodiments) The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0044] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A setting means for setting an origin arbitrarily selected by the user in three-dimensional space and setting a coordinate system including the origin and a first axis parallel to the direction of gravity as a coordinate system when presenting a virtual image to the user via a head-mounted display device. An information processing device comprising: (Configuration 2) The coordinate system set by the setting means further includes a second axis that is an axis perpendicular to the first axis and is parallel to a direction passing through the origin and the head-mounted display device on a plane perpendicular to the first axis. 2. The information processing device according to configuration 1, (Configuration 3) The coordinate system set by the setting means further includes a third axis perpendicular to the first axis and the second axis. 3. The information processing device according to configuration 2. (Configuration 4) The coordinate system set by the setting means defines six degrees of freedom of movement, namely, translational movement parallel to the first axis, translational movement parallel to the second axis, translational movement parallel to the third axis, rotational movement around the first axis, rotational movement around the second axis, and rotational movement around the third axis. 4. The information processing device according to configuration 3. (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, wherein the setting means sets the coordinate system at a timing designated by the user. (Configuration 6) The setting means resets the coordinate system at a timing instructed by the user. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The setting means sets the origin within a range in which an image presented by the head-mounted display device is in focus in the three-dimensional space. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The setting means sets the origin within a range perceived by the user via the head-mounted display device in the three-dimensional space. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The setting means sets the origin at a position designated by the user. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 10) further comprising a detection means for detecting the user's hand; The setting means sets the origin at a position indicated by the user's hand. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 11) further comprising a selection means for allowing a user to arbitrarily select any one of a plurality of feature points in real space; The setting means sets the origin at the position of the feature point selected by the selection means. 9. The information processing device according to any one of configurations 1 to 8. (method) establishing an origin in three-dimensional space selected by the user; setting a coordinate system including the origin and a first axis parallel to the direction of gravity as a coordinate system when presenting a virtual image to a user via a head-mounted display device; An information processing method comprising: (program) 12. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0045] 100: Head-mounted display (HMD) 12: Calculation unit

Claims

1. a setting means for setting an origin arbitrarily selected by the user in a three-dimensional space and setting a coordinate system including the origin and a first axis parallel to the direction of gravity as a coordinate system when presenting a virtual image to the user via a head-mounted display device; An information processing device comprising:

2. The coordinate system set by the setting means further includes a second axis that is an axis perpendicular to the first axis and is parallel to a direction passing through the origin and the head-mounted display device on a plane perpendicular to the first axis.

2. The information processing apparatus according to claim 1, wherein:

3. The coordinate system set by the setting means further includes a third axis perpendicular to the first axis and the second axis.

3. The information processing apparatus according to claim 2, wherein:

4. The coordinate system set by the setting means defines six degrees of freedom of movement, namely, translational movement parallel to the first axis, translational movement parallel to the second axis, translational movement parallel to the third axis, rotational movement about the first axis, rotational movement about the second axis, and rotational movement about the third axis.

4. The information processing apparatus according to claim 3,

5. 2. The information processing apparatus according to claim 1, wherein the setting means sets the coordinate system at a timing designated by the user.

6. The setting means resets the coordinate system at a timing instructed by the user.

2. The information processing apparatus according to claim 1, wherein:

7. The setting means sets the origin within a range in which an image presented by the head-mounted display device is in focus in the three-dimensional space.

2. The information processing apparatus according to claim 1, wherein:

8. The setting means sets the origin within a range perceived by the user via the head-mounted display device in the three-dimensional space.

2. The information processing apparatus according to claim 1, wherein:

9. The setting means sets the origin at a position designated by the user.

2. The information processing apparatus according to claim 1, wherein:

10. further comprising a detection means for detecting the user's hand; The setting means sets the origin at a position indicated by the user's hand.

2. The information processing apparatus according to claim 1, wherein:

11. further comprising a selection means for allowing a user to arbitrarily select any one of a plurality of feature points in real space; The setting means sets the origin at the position of the feature point selected by the selection means.

2. The information processing apparatus according to claim 1, wherein:

12. setting an origin in three-dimensional space selected by the user; setting a coordinate system including the origin and a first axis parallel to the direction of gravity as a coordinate system when presenting a virtual image to a user via a head-mounted display device; An information processing method comprising:

13. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 11.

Images