Information processing device, display device, method executed by the information processing device, and program

Abstract

This technology provides a way to improve the visibility of the real space in a display that overlays the real and virtual spaces. [Solution] The information processing device 101 superimposes virtual content 202 onto the real space visible to the user 201 via an HMD 102 that can be worn by the user 201. The information processing device determines the priority of the display of the real space over the display of the virtual content, and controls at least one display of the real space and the virtual content based on the determination result of the priority of the display of the real space.

Description

Technical Field

[0001] The present disclosure relates to technologies that use a head-mounted display to superimpose and display real space and virtual content, such as AR (Augmented Reality) and MR (Mixed Reality).

Background Art

[0002] In recent years, technologies such as AR and MR that superimpose and display real space and virtual space have been actively developed. As an example of AR technology, there is a virtual monitor function that uses an HMD (Head-Mounted Display) to display the screen of a PC or smartphone in space. Since the virtual monitor function superimposes and displays objects in the virtual space on the real space, there is also a problem that the visibility of the real space is reduced.

[0003] For example, Patent Document 1 discloses a technology that, when real objects moving relative to the user exist in the surroundings, transparently displays the objects in the virtual space to improve the visibility of the real objects.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to improve the visibility of real objects, there is still room for improvement in the technology described in Patent Document 1.

[0006] The present disclosure provides a technology for improving the visibility of real space.

Means for Solving the Problems

[0007] An information processing device according to one embodiment of the present disclosure is an information processing device that superimposes a virtual content image onto a real space visible to the user via a display device that can be worn by the user. The information processing device includes determination means for determining the priority of the display of the real space over the display of the virtual content image, and control means for controlling at least one display of the real space and the virtual content image based on the determination result of the priority of the display of the real space. [Effects of the Invention]

[0008] According to this disclosure, visibility in real space can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] Configuration diagram of the information processing device according to the first embodiment [Figure 2] This figure shows an example of a usage scenario in the first embodiment. [Figure 3] This figure shows an example of a usage scenario in the first embodiment. [Figure 4] This figure shows an example of a usage scenario in the first embodiment. [Figure 5] This is a block diagram showing the logical configuration of the information processing device of the first embodiment. [Figure 6] This is a flowchart showing the flow of the information processing method according to the first embodiment. [Figure 7] This flowchart shows the flow of the determination process in the first embodiment. [Figure 8] This is a diagram illustrating a usage scenario in the second embodiment. [Figure 9] This is a diagram illustrating a usage scenario in the second embodiment. [Figure 10] This is a diagram illustrating a usage scenario in the second embodiment. [Figure 11] This is a block diagram showing the logical configuration of the information processing device of the second embodiment. [Figure 12] This is a flowchart showing the flow of the information processing method according to the second embodiment. [Figure 13] This is a flowchart showing the flow of the determination process of the second embodiment. [Figure 14] This is a diagram explaining the usage scenarios in other embodiments.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Not all of the plurality of features in the embodiments of the present disclosure are essential, and the plurality of features may be arbitrarily combined. Also, the configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the illustrated configurations. Duplicate descriptions are omitted by attaching the same reference numerals to the same or similar configurations in the drawings.

[0011] [First Embodiment] (Configuration of Information Processing Apparatus) FIG. 1 is a diagram showing a physical configuration example of an information processing apparatus 101 and an HMD (head-mounted display) 102 according to this embodiment. The information processing apparatus 101 has the functions of a general PC device and includes a CPU 111, a ROM 112, a RAM 113, a storage unit 114, a communication unit 115, and a system bus 116.

[0012] The CPU 111 uses the RAM 113 as a work memory, executes an operating system (OS) and various computer programs stored in the ROM 112, the storage unit 114, etc., and controls each part via the system bus 116. For example, the programs executed by the CPU 111 include programs for executing the processes described later. The HMD 102 is connected to the information processing apparatus 101 by wire or wirelessly via the communication unit 115. The storage unit 114 is composed of an HDD, an SSD, etc. The storage unit 114 may be composed of a device using an optical storage medium. The information processing apparatus 101 may include a GPU separately from the CPU 111.

[0013] The HMD102 is a video see-through type AR HMD. The HMD102 is an example of a display device that can be worn by a user. Video see-through type AR is an AR system that displays an image (video) captured by a camera as a real image representing the real space on a display and superimposes virtual content on the real image. As a different type from this video see-through type AR device, there is also an optical see-through type AR device, which will be described later.

[0014] The HMD102 includes a display 121, a camera 122, an IMU 123, a magnetic sensor 124, earphones 125, a microphone 126, a gaze measurement sensor 127, a communication unit 128, and a system bus 129.

[0015] The display 121 displays an image in the field of view of the user wearing the HMD102. The display 121 is an example of display means. The camera 122 captures an image of a region corresponding to the field of view of the user. The IMU 123 is an Inertial Measurement Unit including, for example, an acceleration sensor and an angular velocity sensor. The gaze measurement sensor 127 measures the gaze direction of the user wearing the HMD. The gaze measurement sensor 127 includes, for example, one or more infrared sensors and measures the gaze direction by irradiating infrared rays on the user's eyeball and receiving the reflected light. The communication unit 128 makes a connection with the information processing device 101.

[0016] The information processing device 101 and the HMD102 synthesize an image of the real space (real image) captured by the camera 122 and an image of virtual content (virtual content image) and present an AR image to the user by displaying it on the display 121.

[0017] (Usage scenario) Figures 2-4 show an example of a usage scenario in this embodiment. In Figure 2, user 201 is wearing an HMD 102 and viewing virtual content 202 displayed on an AR image using the display 121. The virtual content 202 is, for example, an image (virtual monitor) of the UI screen of the information processing device 101. The field of view 203 shows the field of view of user 201 through the HMD 102. Within the user's field of view 203, areas where the virtual content 202 is not displayed show images of the real world captured by the camera 122. In this embodiment, the virtual content 202 follows the movement of the head of user 201 wearing the HMD 102 and is displayed at a fixed position relative to the position of the HMD 102. In this embodiment, the virtual content 202 is displayed, for example, in the center of the field of view 203, and is displayed at a fixed size. Person 204 is an example of an object in the real world (real object), and in this case, it is a person speaking to user 201.

[0018] Figure 3 shows the state from Figure 2, where user 201 has turned towards person 204 in response to person 204's voice. At this time, because the virtual content 202 is superimposed on person 204, user 201 cannot see person 204 in its entirety.

[0019] Figure 4 shows the state in Figure 3, but with the display of the virtual content 202 made transparent. In this state, user 201 can see the person 204 behind the virtual content 202.

[0020] (Logical configuration of information processing device) The logical configuration of the information processing device 101 according to this embodiment will be described below. The processing of each part shown below is carried out as software by the CPU 111 loading a computer program from ROM 112 or the like onto RAM 113 and then executing the program.

[0021] Figure 5 is a block diagram mainly showing the logical configuration of the information processing device 101. The virtual content acquisition unit 501 acquires virtual content data for display on the HMD 102. In this embodiment, the virtual content data is two-dimensional data placed in a virtual space, and is, for example, an image of the UI screen of the information processing device 101, as will be described later. The virtual content data is stored in, for example, the storage unit 114 or an external storage device (not shown). The virtual content data is also stored in the storage unit 114, etc., associated with a predetermined position and orientation relative to the HMD 102. There may be multiple relative positions and orientations of the virtual content data with respect to the HMD 102. In that case, the user may select one of the multiple positions and orientations and specify it, or the information processing device 101 may specify one of the multiple positions and orientations based on predetermined conditions.

[0022] The determination unit 502 determines the priority of displaying the real space based on data obtained from the camera 122, IMU 123, and magnetic sensor 124 on the HMD 102. The determination unit 502 is an example of a determination means for determining the priority of displaying the real space over the display of the virtual content image.

[0023] The image generation unit 503, based on the determination of the determination unit 502, synthesizes an image of the real space captured by the camera 122 and an image of the virtual content acquired by the virtual content acquisition unit 501 to generate an image to be displayed on the HMD 102. In the information processing device 101, the image generation unit 503 and the CPU (or GPU) that controls the display data of the display 121 are examples of control means. These control means control the display of at least one of the real space and the virtual content image based on the determination result of the priority of displaying the real space.

[0024] The flowchart-based processing procedures described hereafter are not limited to this embodiment. Any combination of procedures, grouping of multiple processes, or subdivision of processes is possible as long as the results of this disclosure are satisfied. Furthermore, each process can be individually separated and function as a single functional element, and can be used in combination with processes other than those shown.

[0025] (Process flow of information processing methods) Next, an information processing method using the information processing device 101 will be described. Figure 6 is a flowchart showing the processing flow of the information processing method according to this embodiment. The series of processes starts when user 201 operates the information processing device 101 and starts a program that displays virtual content 202 on the HMD 102. The processes from S602 to S605 are repeated at a rate of, for example, 60 times per second, depending on the frame rate of the display 121.

[0026] In step S601, the information processing device 101 performs its own initialization. Here, the CPU 111 loads a program from ROM 112 or the like into RAM 113, and then starts executing the program.

[0027] In step S602, the virtual content acquisition unit 501 acquires data for the virtual content 202 to be displayed on the HMD 102. Here, the virtual content acquisition unit 501 acquires the image of the UI screen of the information processing device 101 as the content of the virtual content 202. The virtual content acquisition unit 501 also acquires the relative position and orientation values ​​of the virtual content 202 with respect to the HMD 102.

[0028] In step S603, the determination unit 502 determines the priority of displaying in the real space. Details of the process in step S603 will be described later.

[0029] Step S604 is a branch based on the result of the determination in step S603. If it is determined in step S603 that displaying the real space has a higher priority within a certain time (for example, within 3 seconds), the process proceeds to step S605. Otherwise, the process proceeds to step S606.

[0030] In step S605, the image generation unit 503 combines the image of the real space captured by the camera 122 with the image of the virtual content 202 acquired by the virtual content acquisition unit 501 to generate an image to be displayed on the HMD 102. Here, the image generation unit 503 superimposes the image of the real space with the transparency of the virtual content 202 image set to, for example, 50%.

[0031] In step S606, the image generation unit 503 combines the image of the real space captured by the camera 122 with the image of the virtual content 202 acquired by the virtual content acquisition unit 501 to generate an image to be displayed on the HMD 102. Here, the image generation unit 503 superimposes the image of the real space onto the image of the virtual content 202, setting the transparency of the image to, for example, 0%.

[0032] As a result of the process in step S605, the virtual content 202 is displayed transparently for a certain period of time after the determination unit 502 determines that displaying in the real space has a higher priority. Therefore, even when the real-world person 204 and the virtual content 202 are superimposed, as shown in Figure 4, the user can still see the person 204.

[0033] In step S607, the information processing device 101 determines whether or not to terminate the series of processes. If processing is to continue, after providing an appropriate waiting time to match the frame rate of the display 121, the process returns to step S602.

[0034] (Details of the decision-making process) The details of the determination process in step S603 will be explained below. Figure 7 is a flowchart showing the flow of the determination process. Here, we will explain an example in which the determination unit 502 determines the priority of display in the real space based on the movements of the user 201, that is, the movements of the HMD 102.

[0035] In step S701, the determination unit 502 obtains angular velocity values ​​from the IMU 123. The angular velocity values ​​obtained here include three values: a yaw component with the vertical direction of rotation as the axis of rotation of the HMD 102, a pitch component with the horizontal direction of rotation as the axis of rotation, and a roll component with the front-to-back direction of rotation as the axis of rotation.

[0036] In step S702, the determination unit 502 calculates the turning speed of the HMD 102 based on the angular velocity value obtained in step S701. The turning speed is calculated as the square root of the sum of the squares of the yaw component and pitch component of the angular velocity value.

[0037] The processing in steps S701 and S702 is an example of processing by the calculation means. That is, the calculation means includes, for example, an IMU 123 and / or a magnetic sensor 124, and calculates the movement of the HMD 102.

[0038] In step S703, the determination unit 502 determines whether the turning speed calculated in step S702 exceeds a predetermined threshold. If the turning speed exceeds the threshold, the determination unit 502 determines that the priority of displaying the real space is high. Otherwise, the determination unit 502 determines that the priority of displaying the real space is low.

[0039] In this embodiment, since the virtual content 202 is displayed in a fixed position relative to the HMD 102, the user 201 does not need to move their head to view the virtual content 202. Therefore, if the user 201 moves their head significantly, it is highly likely that the user 201 is trying to look at the real world rather than the virtual content 202. From this, the determination unit 502 can determine whether the display of the real world has a higher priority by evaluating whether the rotation speed of the HMD 102 exceeds a threshold.

[0040] Through the processing described above, for example, if user 201 turns around in response to a call from person 204, as shown in Figures 2-4, in step S703, the determination unit 502 determines that displaying the real space has a higher priority based on the movement of the HMD 102. Subsequently, for a certain period of time, the virtual content 202 is displayed transparently, as described in step S604. As a result, as shown in Figure 4, with user 201 facing person 204, the virtual content 202 remains transparent for a certain period of time. This allows user 201 to see person 204 behind the virtual content 202.

[0041] (Effects of the invention in this embodiment) In this embodiment, the visibility of the real space can be improved by determining the priority of displaying the real space.

[0042] (Modified version of the first embodiment) In the first embodiment, in step S702, the determination unit 502 calculated the rotation speed of the HMD 102 based on the angular velocity value obtained from the IMU 123. However, the method for calculating the rotation speed is not limited to this. For example, the determination unit 502 may determine the attitude of the HMD 102 by obtaining the direction of gravity from the measurement value of the acceleration sensor of the IMU 123 and the direction of magnetic north from the measurement value of the magnetic sensor 124. The determination unit 502 can then calculate the rotation speed of the HMD 102 by measuring the displacement of the attitude per unit time. Alternatively, the determination unit 502 may calculate the position and attitude of the HMD 102 using a known SLAM (Simultaneous Localization and Mapping) technique with respect to the image from the camera 122, and then determine the rotation speed from the displacement. Furthermore, these methods may be used in combination.

[0043] Alternatively, the determination unit 502 may determine the priority of displaying in real space based on the amount of rotation angle within a certain period of time, rather than the rotation speed of the HMD 102. In this case as well, the determination unit 502 may calculate the value using the displacement of the HMD 102's posture obtained from the measurement data of the camera 122 and the magnetic sensor 124, similar to the angular velocity value. Note that both the rotation speed and the amount of rotation angle within a certain period of time are essentially "angular velocity". As described above, the determination unit 502 may calculate the angular velocity based on one or more angular velocity sensors. Alternatively, as described above, the determination unit 502 may calculate the angular velocity as the displacement of posture per unit time based on the respective measurement values ​​of the acceleration sensor and the magnetic sensor 124.

[0044] The determination unit 502 may calculate the speed of the HMD 102 based on the measured values ​​of one or more acceleration sensors instead of angular velocity, and determine the priority of displaying in the real world based on the calculated speed. In this case, the determination unit 502 only needs to determine that the priority of displaying in the real world is high if the calculated speed exceeds a threshold.

[0045] Alternatively, the determination unit 502 may determine the priority of displaying in the real space based on the audio data acquired from the microphone 126. For example, the determination unit 502 can determine that displaying in the real space has a high priority if it detects a sound that exceeds a predetermined threshold in magnitude (sound pressure level) or if it detects that the audio includes (or matches) a predetermined pattern, such as a warning sound or a call.

[0046] In the first embodiment, an example was shown in which the virtual content 202 is fixed in the field of view and follows the movement of the HMD 102. However, the virtual content 202 may not follow the movement of the HMD 102 and may be fixed in a fixed position in real space.

[0047] The determination method of the determination unit 502 and the image generation method of the image generation unit 503, as described above, may be selectable by the user 201 through prior settings. Furthermore, multiple methods may be used in combination.

[0048] [Second Embodiment] In the first embodiment, a method for improving visibility in the real world by determining the priority of display in the real world based on the movement of the HMD 102 was described. In the second embodiment, a method for improving visibility by determining the direction in the real world that has a high display priority will be described. In the second embodiment, an example will be given in which the virtual content 202 does not follow the movement of the HMD 102 but is fixed in a fixed position such as the center of the field of view.

[0049] (Configuration of information processing device) The physical configuration of the information processing device 101 according to this embodiment is the same as that of the first embodiment, so a detailed explanation is omitted here. The configuration of the HMD 102 dealt with in this embodiment is also the same as that of the first embodiment, except that the microphone 126 is a stereo configuration with a pair of microphones mounted on the left and right sides of the HMD. Note that the microphone 126 in the first embodiment may also be a stereo configuration.

[0050] (Usage scenarios) Figures 8-10 show an example of a usage scenario for this disclosure in this embodiment. In Figure 8, user 201 is wearing an HMD 102 and viewing virtual content 801a-801c displayed on an AR image using the display 121. The virtual content 801a-801c are, for example, a group of images (virtual monitors) of multiple UI screens of an information processing device 101. Virtual content 801a is on the left, virtual content 801b is in the front, and virtual content 801c is on the right. Person 204 is behind virtual content 801c on the right.

[0051] Virtual content 801a to 801c are displayed in fixed positions relative to real space. In Figure 8, user 201 is looking in the direction of virtual content 801b in front of them. The HMD 102 displays the entirety of virtual content 801b within the field of view 203, as well as parts of virtual content 801a and 801c at both the left and right edges.

[0052] Figure 9 shows the state after user 201 has turned towards the virtual content 801c on the right, following the state in Figure 8. The HMD 102 displays the entirety of virtual content 801c within the field of view 203, and a portion of the virtual content 801b in front at the far left.

[0053] In these usage scenarios, the virtual content 801a-801c is not always displayed in the center of the field of view 203. However, even if the virtual content is fixed relative to the real space, as shown in Figure 9, the virtual content may be superimposed on an object in the real space (person 204). In such cases, the user 201 is still unable to see the person 204.

[0054] Figure 10 shows the state in Figure 9 with the display of virtual content 801c made transparent. This improves the visibility of person 204 while maintaining the visibility of virtual content 801a and 801b, which do not interfere with person 204.

[0055] (Logical configuration of information processing device) The logical configuration of the information processing device according to this embodiment will be described below. The processing of each part shown below is carried out as software by the CPU 111 loading a computer program from ROM 112 or the like onto RAM 113 and then executing the program.

[0056] Figure 11 is a block diagram mainly showing the logical configuration of the information processing device 101. The virtual content acquisition unit 501 and the image generation unit 503 are the same as those described in the first embodiment, so their explanation is omitted here.

[0057] The determination unit 502 determines the priority of displaying information in the real world based on data obtained from the camera 122, microphone 126, and gaze measurement sensor 127. It also determines the direction in the real world with a higher display priority. Hereinafter, the direction in the real world with a higher display priority will be referred to as the "priority direction". The camera 122, microphone 126, and / or gaze measurement sensor 127 are examples of acquisition means for acquiring information about the surroundings of the display device. In addition to these devices, the acquisition means may also include an IMU 123 and / or magnetic sensor 124.

[0058] The position and orientation estimation unit 1101 acquires images from the camera 122, IMU 123, or measured values ​​from the magnetic sensor 124, and calculates the position and orientation of the HMD 102, for example, using known SLAM technology. The position and orientation estimation unit 1101 is an example of an estimation means for estimating the position and orientation of the display device in real space.

[0059] (Process flow of information processing methods) Next, an information processing method using the information processing device of this embodiment will be described. Figure 12 is a flowchart showing the processing flow of the information processing method according to this embodiment. The series of processes starts when user 201 operates the information processing device 101 and starts a program that displays virtual content 202 on the HMD 102. The processes from S1202 to S1206 are repeated at a rate of, for example, 60 times per second, depending on the frame rate of the display 121.

[0060] In step S1201, the information processing device 101 performs its own initialization. Here, the CPU 111 reads a program from ROM 112 or the like onto RAM 113 and then starts executing the program. The CPU 111 also initializes the SLAM electronic map data used by the position and attitude estimation unit 1101 and sets the initial position and attitude of the HMD 102 relative to the real world.

[0061] In step S1202, the virtual content acquisition unit 501 acquires data for the virtual content 801a to 801c to be displayed on the HMD. Here, the virtual content acquisition unit 501 acquires multiple images of the UI screen of the information processing device 101 as the content of the virtual content. These multiple images correspond to the virtual content 801a to 801c, respectively. The virtual content acquisition unit 501 also acquires the position and orientation values ​​in real space where the virtual content 801a to 801c are placed.

[0062] In step S1203, the position and attitude estimation unit 1101 calculates (estimates) the position and attitude of the HMD 102 relative to the real world using SLAM technology. Specifically, the position and attitude estimation unit 1101 acquires images from the camera 122 and acceleration and angular velocity values ​​measured by the IMU 123, updates the position and attitude of the HMD 102, and updates the electronic map data.

[0063] In step S1204, the determination unit 502 determines the priority direction. Details of the process in step S1204 will be described later.

[0064] In step S1205, the image generation unit 503 combines the image of the real space captured by the camera 122 with the images of the virtual content 801a to 801c acquired by the virtual content acquisition unit 501 to generate an image to be displayed on the HMD 102. Specifically, the image generation unit 503 calculates the relative position and orientation of each virtual content 801a to 801c from the HMD 102. Here, the image generation unit 503 uses the processing results of steps S1202 and S1203 to calculate the relative position and orientation of the virtual content 801a to 801c. Specifically, the image generation unit 503 calculates the relative position and orientation of the virtual content 801a to 801c based on the position and orientation of the virtual content 801a to 801c in the real space where it is located and the position and orientation of the HMD 102. Subsequently, the image generation unit 503 renders images of each virtual content 801a to 801c as seen from the HMD 102, based on the calculated relative position and orientation. Finally, the image generation unit 503 generates a display image by superimposing the rendered images onto the image of the real space captured by the camera 122.

[0065] At this time, the image generation unit 503 superimposes the image of the virtual content 202 included in the field of view in the direction that was determined in S1204 to have a high priority for displaying the real space, with the transparency of the image set to 50%. For virtual content in other directions, the transparency of the image of the virtual content is set to 0% and superimposed on the image of the real space.

[0066] The process in step S607 is the same as that described in the first embodiment, so its explanation is omitted here.

[0067] (Details of the decision-making process) The details of the process in step S1204 will be described below. Figure 13 is a flowchart showing the flow of the determination process. Here, we will describe an example of determining the priority direction based on the sound and direction acquired from the microphone 126.

[0068] In step S1301, the determination unit 502 acquires ambient sound from multiple microphones 126 attached to the HMD 102.

[0069] In step S1302, the information processing device 101 detects or acquires an object that attracts the user 201's attention. For example, in step S1302, the determination unit 502 acquires whether there is one or more candidate objects in the real world that have a high display priority, and the direction in which such objects are located as seen from the HMD 102. Specifically, the determination unit 502 determines whether the volume (sound pressure level) of the sound acquired in step S1301 is above a predetermined threshold. If the determination unit 502 determines that a sound volume above the threshold has occurred, it determines that there is an object in the real world that has a high display priority. Then, based on the phase difference of the sound acquired by the multiple microphones 126 and the orientation of the HMD 102, the determination unit 502 estimates the direction in which the sound originated in the real world (for example, the direction of the sound source).

[0070] In step S1303, the determination unit 502 obtains the direction in real space that user 201 is looking at. Here, the direction in front of the HMD 102 is simply considered to be the direction that user 201 is looking at.

[0071] In step S1304, the determination unit 502 determines whether the angle difference between the direction of the object with high display priority and the direction the user 201 is looking is below a threshold. Specifically, in step S1302, the determination unit 502 calculates the angle between the direction of the object acquired within a certain past time period (for example, within 3 seconds) and the direction the user 201 is looking, acquired in step S1303. If the angle calculated for the candidate object is below a predetermined threshold, the determination unit 502 determines that the direction of one or more objects that meet this condition is the priority direction. The angle difference threshold is set to, for example, 0° to 15°, but is not limited to this range. The information processing device 101 may have a function that allows the user to customize the setting of the angle threshold.

[0072] As described above, for example, when person 204 calls out to user 201 as shown in Figure 8, the determination unit 502 detects the voice of the call as a candidate for an object with high display priority. Subsequently, when user 201 turns towards person 204 as shown in Figure 10, the direction of the voice of the call (direction of the sound source) and the direction the user is looking approximately coincide, so the determination unit 502 determines that the direction of the voice of the call is the priority direction. Due to the processing in step S1205, virtual content in the priority direction is made transparent, so virtual content 801c is made transparent, and user 201 can see person 204.

[0073] (Effects of the invention in this embodiment) In this embodiment as well, when displaying virtual content fixed to the real space, even if the HMD 102 and the person 204 are not moving relative to each other, it is possible to improve the visibility of the real space by determining the priority of displaying the real space.

[0074] (Modified version of the second embodiment) In the second embodiment, in step S1302, the determination unit 502 acquired candidate objects with high display priority and their directions based on ambient sound acquired from the microphone 126. However, the method for acquiring candidate objects and their directions is not limited to this when implementing the present disclosure. The determination unit 502 may also determine the direction of a detected real object as the priority direction if it detects a real object with pre-set attributes or a real object performing a pre-set action from the image of the camera 122. Real objects with pre-set attributes include, for example, pre-set target people or objects, or objects that have caused some kind of pre-defined anomaly. Pre-set actions include a person performing a specific motion such as beckoning, or an animal performing a specific action.

[0075] In the second embodiment, in step S1303, the forward direction of the HMD 102 was obtained as the direction the user 201 was looking. However, the method for obtaining the direction in real space that the user 201 is looking is not limited to this. For example, the direction of the user 201's gaze in real space may be obtained based on the orientation of the HMD 102 and the measurement results of the gaze measurement sensor 127.

[0076] In the second embodiment, a method for determining a "direction" with a high priority for display in real space was described. The information processing device 101 may also determine a "position" with a high priority for display in real space. In other words, these are examples of the determination means determining a position or direction with a high priority for display in real space.

[0077] In the second embodiment, a method was described as in which the virtual content 202 does not follow the movement of the HMD 102 but is fixed in a fixed position, such as the center of the field of view. However, in the second embodiment as well, a method may be adopted in which the virtual content 202 follows the movement of the HMD 102, that is, is fixed to the HMD 102, similar to the first embodiment.

[0078] [Other embodiments] In the first and second embodiments, as a means of improving visibility in the real space, the transparency of the image of virtual content 202 or virtual content 801c was set to 50% and superimposed on the image in the real space. However, the numerical value of the transparency is not limited to this. The HMD or information processing device 101 may also have a function that allows the user to customize the transparency. The means of improving visibility in the real space are not limited to this. For example, as shown in Figure 14, the information processing device 101 may reduce the size of the virtual content 202 or move it to the edge of the field of view 203 so that the virtual content 202 does not overlap with real objects (e.g., person 204). Alternatively, the information processing device 101 may adjust the degree of reduction or movement so that the virtual content 801c does not overlap with objects with a high display priority.

[0079] In the first and second embodiments described above, virtual monitors displaying the UI screen of the information processing device 101 were used as examples of virtual content 202 and virtual content 801a-c. The content of the virtual content to be composited with the displayed image is not limited to this, and for example, 3D objects (3D data) represented by polygons may be used. In this case, one method to improve the visibility in real space is to make the 2D image after rendering the 3D object transparent. Alternatively, the information processing device 101 may reduce the 3D object in three dimensions and / or move it to the edge of the field of view 203, and then render it to generate a 2D image.

[0080] In the first and second embodiments described above, a video see-through type HMD was used as an example. However, the technology described in the first and second embodiments can also be applied to optical see-through type AR devices. An optical see-through type AR device is a device that uses the optical image incident on the device via an optical lens, etc., rather than the image captured by the camera 122, as the background of the device's display, and displays virtual content 202 on that display. In the case of an optical see-through type device, the real space is visible to the naked eye, so the image generation unit 503 does not perform synthesis of the image of the real space and the image of the virtual content. However, as described above, the visibility of the real space can be improved by increasing the transparency of the virtual content 202, reducing its size, and / or moving it towards the edge of the field of view 203. Alternatively, there are optical see-through type AR devices that can reduce the visibility of the real space using a light-shielding filter whose transmittance can be controlled electrically. Such a device can improve the visibility of the real space by increasing the transmittance of the light-shielding filter according to the determination of the determination unit 502.

[0081] The information processing device 101 may perform processing related to the visibility of the real space based on the determination result of the priority of displaying the real space. For example, as an HMD device according to the comparative example of this disclosure, processing is performed in advance to reduce the visibility of the real space, such as by darkening or blurring the display of the real space, for the purpose of improving immersion in the virtual content 202. In this case, the information processing device 101 of this disclosure may increase the visibility of the real space by stopping or weakening the processing that reduces the visibility of the real space. For example, suppose that while the information processing device 101 determines that the priority of displaying the real space is low, the image generation unit 503 synthesizes an image with the brightness value of the camera image reduced to 50% into the background area of ​​the field of view. Then, when the determination unit 502 determines that the priority of the real space is high, the information processing device 101 may switch processing to increase the brightness value of the image acquired by the camera 122 and synthesize it. This increases the visibility of the real space. Such technology is applicable to both the first and second embodiments described above. In the case where it applies to the second embodiment, if the information processing device 101 has acquired a field of view in a direction with high priority for displaying the real space, it performs processing to weaken the processing that reduces visibility for the area of ​​the real space corresponding to the field of view in that direction.

[0082] Alternatively, the information processing device 101 may perform processing to accept adjustment operations from the user 201 regarding the visibility of the real space via a GUI (Graphical User Interface). For example, the information processing device 101 may display a GUI for improving the visibility of the real space on the display 121 in accordance with the determination of the determination unit 502, and the user 201 can perform processing to improve the visibility of the real space by operating the GUI.

[0083] The information processing device 101 may control the display of both the real space and the virtual content 202 based on the result of determining the priority of displaying the real space. An example of controlling the display of the real space is the processing related to the visibility of the real space, as described above. An example of controlling the display of the virtual content 202 is the adjustment, reduction, or movement of the transparency of the virtual content 202, as described above.

[0084] In the first and second embodiments, an information processing device 101 connected to the HMD 102 was described as an example. However, the technology of this disclosure can also be applied to standalone HMD devices in which the HMD is equipped with the functionality of an information processing device.

[0085] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., ASIC or FPGA) that implements one or more functions. The network is a communication network implemented by one or more of the following, such as a LAN, WAN, telephone line, dedicated digital line, ATM, frame relay line, cable television line, wireless line for data broadcasting, etc.

[0086] Although the present disclosure has been described in detail above based on its preferred embodiments, the present disclosure is not limited to the above embodiments, and various modifications are possible in accordance with the spirit of the present disclosure, and these modifications are not excluded from the scope of the present disclosure.

[0087] Embodiments in this disclosure include the following configurations, methods, and programs. (Composition 1) An information processing device that superimposes virtual content images onto a real space visible to the user via a display device that can be worn by the user, A determination means for determining the priority of displaying the real space over displaying the virtual content image, The system includes a control means that controls the display of the real space and at least one of the virtual content images based on the result of determining the priority of display of the real space. An information processing device characterized by the following: (Configuration 2) The display device has a calculation means for calculating the movement of the display device, The determination means determines that the priority of displaying the real space is high when the calculated value of the movement of the display device exceeds a threshold. The information processing device according to configuration 1, characterized by the above. (Composition 3) The calculated value of the aforementioned motion is either a velocity value or an angular velocity value. The information processing apparatus according to configuration 2, characterized in that... (Composition 4) The display device has one or more microphones, The determination means determines that the priority of displaying the real space is high if the sound pressure level of the sound acquired by the one or more microphones exceeds a threshold. The information processing device according to configuration 1, characterized by the above. (Composition 5) The display device has one or more microphones, The determination means determines that the priority of displaying the real space is high if the sound acquired by the one or more microphones includes a predetermined pattern of sound. The information processing device according to configuration 1, characterized by the above. (Composition 6) If the control means determines that the priority of displaying the real space is high, it increases the transparency of the virtual content image. An information processing device according to any one of configurations 2 to 5, characterized in that it is an information processing device. (Composition 7) If the control means determines that displaying the real space has a high priority, it reduces the size of the virtual content image. An information processing device according to any one of configurations 2 to 5, characterized in that it is an information processing device. (Composition 8) If the control means determines that displaying the real space has a high priority, it moves the virtual content image to the edge of the field of view displayed on the display device. An information processing device according to any one of configurations 2 to 5, characterized in that it is an information processing device. (Composition 9) The determination means determines the priority direction, which is the direction in the real space with a higher display priority, from the display device. The control means controls the display of at least one of the real space and the virtual content image included in the field of view of the determined preferred direction. The information processing device according to configuration 1, characterized by the above. (Composition 10) The system further includes estimation means for estimating the position and orientation of the display device in the real space, The display device has acquisition means for acquiring information about the surroundings of the display device, The determination means determines the direction with the highest priority for displaying in the real space, based on the estimated position and orientation information of the display device and the acquired information about the surroundings of the display device. The information processing apparatus according to configuration 9, characterized by the features described therein. (Composition 11) The determination means determines that the direction of the object is the preferred direction if the angular difference between the estimated orientation of the display device and the direction of the object in real space obtained based on the information surrounding the display device is less than or equal to a threshold. The information processing apparatus according to configuration 10, characterized by the above. (Composition 12) The acquisition means includes multiple microphones, The determination means determines the direction with the highest priority for displaying in the real space, based on the direction of the sound source identified based on the sound acquired by the plurality of microphones. The information processing apparatus according to configuration 10, characterized by the above. (Composition 13) The acquisition means includes one or more cameras, The determination means determines the preferred direction based on one or more images acquired by the one or more cameras. The information processing apparatus according to configuration 10, characterized by the above. (Composition 14) The determination means, when it detects an object in the real space having pre-set attributes or an object in the real space performing a pre-set action from one or more images, determines the preferred direction based on the direction of the detected object. The information processing device according to configuration 13, characterized by the above. (Composition 15) The acquisition means detects the user's gaze direction, The determination means determines the preferred direction based on the detected information regarding the user's gaze direction. The information processing apparatus according to configuration 10, characterized by the above. (Composition 16) When the preferred direction is determined, the control means increases the transparency of the virtual content image included in the field of view of that preferred direction. An information processing apparatus according to any one of configurations 9 to 15, characterized by the features described herein. (Composition 17) When the preferred direction is determined, the control means reduces the size of the virtual content image included in the field of view of that preferred direction. An information processing apparatus according to any one of configurations 9 to 15, characterized by the features described herein. (Composition 18) When the preferred direction is determined, the control means moves the virtual content image included in the field of view of the preferred direction toward the edge of the field of view displayed on the display device. An information processing apparatus according to any one of configurations 9 to 15, characterized by the features described herein. (Composition 19) The determination means determines the position in the real space that has a high priority for display, The control means controls the display of at least one of the real space and the virtual content image that is located in a position where the display of the real space has a high priority, as determined. The information processing device according to configuration 1, characterized by the above. (Composition 20) The control means performs processing to receive adjustment operations from the user regarding the visibility of the real space via a graphical user interface. The information processing device according to configuration 1, characterized by the above. (Composition 21) The control means performs processing related to the visibility of the real space based on the determination result of the priority of displaying the real space. An information processing apparatus according to configuration 1 or 9, characterized by the above. (Composition 22) The real space visible to the user is a real image obtained by photographing the real space. The determination means determines the priority of displaying the real image over displaying the virtual content image, The control means controls the display of at least one of the real image and the virtual content image based on the determination result of the priority of displaying the real image. An information processing device according to any one of configurations 1 to 21, characterized by the above. (Composition 23) A display device that can be worn by a user and superimposes virtual content images onto a real space visible to the user, A determination means for determining the priority of displaying the real space over displaying the virtual content image, The system includes a control means that controls the display of the real space and at least one of the virtual content images based on the result of determining the priority of display of the real space. A display device characterized by the following features. (method) A method performed by an information processing device that superimposes a virtual content image onto a real space visible to the user via a display device that can be worn by the user, A determination step for determining the priority of displaying the real space over displaying the virtual content image, The control step includes controlling the display of at least one of the real space and the virtual content image based on the result of determining the priority of display of the real space. A method characterized by the following: (program) A program that causes a computer to perform the above method. [Explanation of symbols]

[0088] 101: Information Processing Device 102: HMD 121: Display 122: Camera 126: Microphone 127: Eye-tracking sensor 201: User 202, 801a, 801b, 801c: Virtual content 203: Visibility 502: Judgment Department 503: Image generation unit 1101:Position and orientation estimation unit

Claims

1. An information processing device that superimposes virtual content images onto a real space visible to the user via a display device that can be worn by the user, A determination means for determining the priority of displaying the real space over displaying the virtual content image, The system includes a control means that controls the display of the real space and at least one of the virtual content images based on the result of determining the priority of display of the real space. An information processing device characterized by the following:

2. The display device has a calculation means for calculating the movement of the display device, The determination means determines that the priority of displaying the real space is high when the calculated value of the movement of the display device exceeds a threshold. The information processing apparatus according to feature 1.

3. The calculated value of the aforementioned motion is either a velocity value or an angular velocity value. The information processing apparatus according to feature 2.

4. The display device has one or more microphones, The determination means determines that the priority of displaying the real space is high if the sound pressure level of the sound acquired by the one or more microphones exceeds a threshold. The information processing apparatus according to feature 1.

5. The display device has one or more microphones, The determination means determines that the priority of displaying the real space is high if the sound acquired by the one or more microphones includes a predetermined pattern of sound. The information processing apparatus according to feature 1.

6. If the control means determines that the priority of displaying the real space is high, it increases the transparency of the virtual content image. The information processing apparatus according to feature 2.

7. If the control means determines that displaying the real space has a high priority, it reduces the size of the virtual content image. The information processing apparatus according to feature 2.

8. If the control means determines that displaying the real space has a high priority, it moves the virtual content image to the edge of the field of view displayed on the display device. The information processing apparatus according to feature 2.

9. The determination means determines the priority direction, which is the direction in the real space with a higher display priority, from the display device. The control means controls the display of at least one of the real space and the virtual content image included in the field of view of the determined preferred direction. The information processing apparatus according to feature 1.

10. The system further includes estimation means for estimating the position and orientation of the display device in the real space, The display device has acquisition means for acquiring information about the surroundings of the display device, The determination means determines the direction with the highest priority for displaying in the real space, based on the estimated position and orientation information of the display device and the acquired information about the surroundings of the display device. The information processing apparatus according to feature 9.

11. The determination means determines that the direction of the object is the preferred direction if the angular difference between the estimated orientation of the display device and the direction of the object in real space obtained based on the information surrounding the display device is less than or equal to a threshold. The information processing apparatus according to feature 10.

12. The acquisition means includes multiple microphones, The determination means determines the preferred direction based on the direction of the sound source identified based on the sound acquired by the plurality of microphones. The information processing apparatus according to feature 10.

13. The acquisition means includes one or more cameras, The determination means determines the preferred direction based on one or more images acquired by the one or more cameras. The information processing apparatus according to feature 10.

14. The determination means, when it detects an object in the real space having pre-set attributes or an object in the real space performing a pre-set action from one or more images, determines the preferred direction based on the orientation of the detected object. The information processing apparatus according to feature 13.

15. The acquisition means detects the user's gaze direction, The determination means determines the preferred direction based on the detected information regarding the user's gaze direction. The information processing apparatus according to feature 10.

16. When the preferred direction is determined, the control means increases the transparency of the virtual content image included in the field of view of that preferred direction. The information processing apparatus according to feature 9.

17. When the preferred direction is determined, the control means reduces the size of the virtual content image included in the field of view of that preferred direction. The information processing apparatus according to feature 9.

18. When the preferred direction is determined, the control means moves the virtual content image included in the field of view of that preferred direction toward the edge of the screen displayed on the display device. The information processing apparatus according to feature 9.

19. The determination means determines the position in the real space that has a high priority for display, The control means controls the display of at least one of the real space and the virtual content image that is located in a position where the display of the real space has a high priority, as determined. The information processing apparatus according to feature 1.

20. The control means performs processing to receive adjustment operations from the user regarding the visibility of the real space via a graphical user interface. The information processing apparatus according to feature 1.

21. The control means performs processing related to the visibility of the real space based on the determination result of the priority of displaying the real space. The information processing apparatus according to feature 1.

22. The real space visible to the user is a real image obtained by photographing the real space. The determination means determines the priority of displaying the real image over displaying the virtual content image, The control means controls the display of at least one of the real image and the virtual content image based on the determination result of the priority of displaying the real image. The information processing apparatus according to feature 1.

23. A display device that can be worn by a user and superimposes virtual content images onto a real space visible to the user, A determination means for determining the priority of displaying the real space over displaying the virtual content image, The system includes a control means that controls the display of the real space and at least one of the virtual content images based on the result of determining the priority of display of the real space. A display device characterized by the following features.

24. A method performed by an information processing device that superimposes a virtual content image onto a real space visible to the user via a display device that can be worn by the user, A determination step for determining the priority of displaying the real space over displaying the virtual content image, The control step includes controlling the display of at least one of the real space and the virtual content image based on the result of determining the priority of display of the real space. A method characterized by the following:

25. A program that causes a computer to perform the method described in claim 24.

Images