Information processing device, information processing method, and recording medium
The information processing device addresses the challenge of transmittance control in VR and AR by determining scene-specific transmittance adjustments based on user actions and environmental factors, preventing collisions and improving virtual object visibility.
Patent Information
- Application Number
- PCT/JP2024/015844
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-04-23
- Publication Date
- 2024-11-14
AI Technical Summary
Existing VR and AR technologies face challenges in effectively controlling the transmittance of real objects when they are between the viewpoint position and virtual objects, leading to potential collisions or difficulties in grasping virtual objects, as current methods either obscure virtual objects or fail to provide appropriate transparency adjustments based on user actions and positional relationships.
An information processing device and method that performs scene determination to control the transmittance of real objects based on user actions, positional relationships, and environmental factors such as distance and movement speed, ensuring appropriate transparency adjustments to prevent collisions and enhance virtual object visibility.
The solution effectively prevents collisions by reducing real object transmittance when collision risks are high and improves virtual object visibility by making real objects transparent when risks are low, thereby enhancing user safety and interaction with virtual environments.
Smart Images

Figure JP2024015844_14112024_PF_FP_ABST
Abstract
Description
Information processing device, information processing method, and recording medium
[0001] The present technology relates to an information processing device, an information processing method, and a recording medium, and more particularly to a technical field related to transmittance control of a real object when the real object is present between a viewpoint position and a virtual object.
[0002] In recent years, virtual reality (VR) technology and augmented reality (AR) technology have become widely used. Patent Document 1 listed below discloses a technology for hiding a real object (increasing the transmittance of the real object) when the real object is in front of a virtual object in the line of sight of a user (observer) observing the virtual object. This prevents the virtual object from being obscured by the real object and becoming invisible when the real object is in front of a virtual object that the user is focusing on.
[0003] International Publication No. 2022 / 044124
[0004] However, there may be cases where it is not desirable to increase the transmittance of a real object simply because a real object exists in front of a virtual object, in other words, because a real object exists between the viewpoint and the virtual object. For example, when the distance between the user and the real object is short or the user is moving fast, increasing the transmittance of the real object in front of the virtual object may make it difficult for the user to notice the approach of the real object, which could result in the user colliding with the real object. Furthermore, when the real object is the user's hand and the virtual object is a virtual object intended to be grasped by the user's hand, the real object exists in front of the virtual object. In this case, increasing the transmittance of the hand as a real object may make it difficult for the user to grasp the sense of distance from the virtual object as the grasped object, which could result in the user being unable to grasp the grasped object as desired. Furthermore, if we imagine a scene in which some kind of operation button is assumed as the virtual object and the user's hand, which is the real object, is to operate this operation button, then the real object will exist between the user's position and the virtual object. However, even in this case, if the transparency of the hand as a real object is increased simply because the real object is in front of the virtual object, it will be difficult for the user to grasp the positional relationship between the button to be operated and the hand, and there is a risk that the user will not be able to operate the button as desired.
[0005] This technology was developed in consideration of the above circumstances, and aims to realize appropriate control of the transmittance of real objects according to the scene when a real object is located between the viewpoint and a virtual object.
[0006] An information processing device according to the present technology includes a transmittance control unit that, when a real object is present between a viewpoint position and a virtual object, performs scene determination related to at least one of a user's behavior and a positional relationship between the user and the real object, and controls the transmittance of the real object based on the result of the scene determination. According to the above configuration, the transmittance of a real object present in front of a virtual object is controlled based on the result of scene determination related to a user's behavior, such as the user's moving speed, whether the user is attempting to virtually grasp a virtual object, or whether the user is attempting to operate a virtual object as a button, or the like, and the result of scene determination related to a positional relationship between the user and the real object, such as whether the user is close to the real object.
[0007] An information processing method according to the present technology is an information processing method for, when a real object exists between a viewpoint position and a virtual object, performing scene determination related to at least one of a user's behavior and a positional relationship between the user and the real object, and controlling the transmittance of the real object based on the result of the scene determination. Furthermore, a recording medium according to the present technology is a recording medium on which a computer-readable program is recorded, the recording medium on which the program is recorded to cause the computer to realize a function of, when a real object exists between a viewpoint position and a virtual object, performing scene determination related to at least one of a user's behavior and a positional relationship between the user and the real object, and controlling the transmittance of the real object based on the result of the scene determination. These information processing methods and recording media make it possible to realize an information processing device according to the present technology.
[0008] 1 is a block diagram showing an example configuration of an information processing device as a first embodiment according to the present technology. FIG. 1 is a diagram schematically showing a conventional transmittance control method. FIG. 2 is a flowchart showing a specific processing procedure example for realizing the transmittance control method as the first embodiment. FIG. 3 is an explanatory diagram of a light adjustment unit used in another example of the first embodiment. FIG. 4 is a block diagram showing an example configuration of an information processing device as another example of the first embodiment. FIG. 5 is an explanatory diagram of a transmittance control method as a second embodiment. FIG. 6 is a block diagram showing an example configuration of an information processing device corresponding to a case where a first pattern is adopted in the second embodiment. FIG. 7 is a flowchart showing a specific processing procedure example for realizing the transmittance control method corresponding to a case where the first pattern is adopted in the second embodiment. FIG. 8 is a block diagram showing an example configuration of an information processing device corresponding to a case where a second pattern is adopted in the second embodiment. FIG. 9 is a flowchart showing a specific processing procedure example for realizing the transmittance control method corresponding to a case where the second pattern is adopted in the second embodiment. FIG. 10 is an explanatory diagram of a transmittance control method as a third embodiment. FIG. 11 is a block diagram showing an example configuration of an information processing device as the third embodiment. FIG. 12 is a flowchart showing a specific processing procedure example for realizing the transmittance control method as the third embodiment. FIG. 13 is an explanatory diagram of an AR system assumed in a fourth embodiment. FIG. 14 is an explanatory diagram of transmittance control as the fourth embodiment. FIG. 15 is a block diagram showing an example configuration of a display device in the fourth embodiment. Fig. 10 is a block diagram showing a configuration example of an information processing device as a fourth embodiment; Fig. 11 is a flowchart showing a specific processing procedure example for realizing the transmittance control method as the fourth embodiment; Fig. 12 is an explanatory diagram of the transmittance control method as a fifth embodiment; Fig. 13 is a block diagram showing a configuration example of an information processing device as a fifth embodiment; Fig. 14 is a flowchart showing a specific processing procedure example for realizing the transmittance control method as the fifth embodiment.
[0009] Hereinafter, with reference to the accompanying drawings, embodiments according to the present technology will be described in the following order: <1. First embodiment> [1-1. Configuration example of information processing device] [1-2. Transmittance control method as first embodiment] [1-3. Processing procedure] [1-4. Another example of the first embodiment] <2. Second embodiment> <3. Third embodiment> <4. Fourth embodiment> <5. Fifth embodiment> <6. Modified example> <7. Program and recording medium> <8. Summary of embodiments> <9. The present technology>
[0010] 1. First Embodiment [1-1. Configuration Example of Information Processing Device] FIG. 1 is a block diagram showing a configuration example of an information processing device 1 according to a first embodiment of the present technology. The information processing device 1 is configured as a device for allowing a user to experience a virtual space. Specifically, the information processing device 1 in this example is configured to allow a user to experience an AR (Augmented Reality) space. Methods for experiencing an AR space include a VST (Video See-Through) method that presents the real world using captured images, and an OST (Optical See-Through) method that allows the user to directly view the real world without using captured images. The information processing device 1 in this example is configured to support the former VST method. The information processing device 1 can take a variety of device forms, but here, a case where it is configured as an HMD (Head Mounted Display) is illustrated.
[0011] In this specification, "AR" also includes the concept of "MR (Mixed Reality)."
[0012] As shown in the figure, the information processing device 1 includes an IMU (Inertial Measurement Unit) 2, an image sensor 3, a ranging sensor 4, a self-position and orientation estimation unit 5, a distance image generation unit 6, a virtual object drawing unit 7, a transmittance control unit 8, a synthesis unit 9, a display control unit 10, and a display unit 11.
[0013] The IMU 2 has a three-axis acceleration sensor and an angular velocity sensor, and outputs three-axis acceleration information and angular velocity information.
[0014] The image sensor 3 is configured to have, for example, a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) solid-state imaging element, and obtains a captured image of the real world. Although not shown in the figures, in the information processing device 1 as an HMD in this example, an imaging lens is provided at a position near the eyes of a user wearing the information processing device 1, and the image sensor 3 receives light from a subject that is input through the imaging lens and performs photoelectric conversion, thereby obtaining a captured image of the real world from the user's viewpoint.
[0015] The ranging sensor 4 measures distances to objects in the real world. In this example, a sensor capable of measuring distances on a pixel-by-pixel basis is used as the ranging sensor 4 to obtain a distance image (depth map) as a three-dimensional measurement result of the real world. Specifically, in this example, a sensor capable of measuring distances on a pixel-by-pixel basis in accordance with the ToF method is used as the ranging sensor 4. As is well known, the ToF (Time of Flight) method measures distances based on the length of time it takes for light irradiated onto an object to be measured and reflected from the object to return. For this reason, the ranging sensor 4 includes a light source for irradiating the object with light (e.g., irradiating invisible light such as IR light) and a light receiving unit in which multiple pixels having light receiving elements are two-dimensionally arranged. Although not shown, the ranging sensor 4 is also provided with an illumination optical system for irradiating the object with light emitted from the light source and a light receiving optical system for guiding reflected light from the object to the light receiving unit. The distance measurement unit, which is configured to include the above-described irradiation optical system, light receiving optical system, and distance measurement sensor 4, is provided in the information processing device 1 so as to measure distances to objects within the imaging range of the image sensor 3.
[0016] The distance image generating unit 6 generates a distance image based on the distance measurement results for each pixel by the distance measuring sensor 4 .
[0017] The configuration for obtaining the distance image is not limited to the above configuration. For example, it is also possible to adopt a configuration in which distance estimation is performed by machine learning such as a convolutional neural network (CNN) based on an image captured by an image sensor, or a configuration in which distance measurement is performed based on parallax using a stereo camera (e.g., a semi-global matching method).
[0018] The self-position and orientation estimation unit 5, the virtual object rendering unit 7, the transmittance control unit 8, and the synthesis unit 9 constitute a calculation unit 15 for generating an image for allowing the user to experience the AR space, based on the detection information from the IMU 2, the captured image from the image sensor 3, and the distance image obtained by the distance image generation unit 6. The calculation unit 15 can be constituted by one or more computers.
[0019] In the calculation unit 15, the self-position and attitude estimation unit 5 estimates the self-position and attitude of the information processing device 1 based on the detection information of the IMU 2 (in this example, triaxial acceleration information and angular velocity information) and the image captured by the image sensor 3. Examples of methods for estimating the self-position and attitude include SLAM (Simultaneous Localization and Mapping) and VIO (Visual Inertial Odometry). For example, VIO assumes that the output of the IMU 2, which has a higher output rate than the image sensor 3, is used to estimate the self-position and attitude using a technology such as INS (Inertial Navigation System). Note that methods such as VO (Visual Odometry) may also be employed as methods that do not use the detection information of the IMU 2.
[0020] The self-position and orientation estimation unit 5 estimates the self-position and orientation of the information processing device 1 in the world coordinate system, for example, by the method exemplified above. For the self-position and orientation estimation unit 5, a general CPU (Central Processing Unit) or GPU (Graphics Processing Unit) is often used for the self-position and orientation estimation process, but a processor specialized for image processing or machine learning processing can also be used.
[0021] The virtual object rendering unit 7 renders a virtual object using 3DCG (Computer Graphics) technology based on the self-position and orientation estimated by the self-position and orientation estimation unit 5, and outputs the rendering result and its depth information. Here, as the depth information of the virtual object, the depth of the virtual object in world coordinates (distance from the self-position) is output as a depth map. Hereinafter, the rendering result of the virtual object may be referred to as a "virtual object image."
[0022] Hereinafter, virtual objects will be referred to as "Vo." Also, objects that actually exist in the real world will be referred to as real objects Ro.
[0023] There are various ways to arrange the virtual object Vo in the AR space. For example, 1) a position that has a predetermined positional relationship with a certain reference position is set as the arrangement position. For example, the reference position may be set to the self-position, and a position a predetermined distance away from the self-position in a predetermined direction may be set as the arrangement position of the virtual object Vo. Alternatively, a marker may be placed in the real space, the reference position may be set to the marker position, and a position a predetermined distance away from the position where the marker is detected in a predetermined direction may be set as the arrangement position of the virtual object Vo. 2) Superimposing on the real object Ro. For example, the virtual object is placed at a position superimposed on a predetermined real object Ro captured in an image captured by the image sensor 3. 3) At a predetermined position in the AR space. For example, the arrangement position of the virtual object Vo is set in advance on a map. For example, a predetermined position of a real object such as a wall or a window may be set as the arrangement position of the virtual object Vo.
[0024] In this example, for the sake of explanation, it is assumed that the virtual object Vo is to be placed at a predetermined position in the AR space as described in 3) above. In this case, placement instruction information is input to the virtual object rendering unit 7 as shown in the figure. This placement instruction information includes information on the placement position and placement orientation (posture) of the virtual object Vo in the world coordinate system. The virtual object rendering unit 7 in this example draws the virtual object Vo in accordance with this placement position information. At this time, the position and orientation of the two-dimensional plane onto which the virtual object Vo is projected in the drawing are determined based on the self-position and orientation estimated by the self-position and orientation estimation unit 5.
[0025] In this example, a GPU is used for the rendering process of the virtual object Vo by the virtual object rendering unit 7. However, this is just an example, and it is also possible to use a CPU for the rendering process of the virtual object Vo.
[0026] When a real object Ro is present between the viewpoint position and the virtual object Vo, the transmittance control unit 8 performs scene determination related to at least one of the user's behavior and the positional relationship between the user and the real object Ro, and controls the transmittance of the real object Ro based on the result of the scene determination. As will be described in detail later, the transmittance control unit 8 in this embodiment generates an alpha map for controlling the transmittance of the real object Ro based on the distance image output from the distance image generation unit 6, depth information (depth information for each pixel of the virtual object image) regarding the rendering result of the virtual object Vo (virtual object image) output from the virtual object rendering unit 7, and information on the self-position and orientation estimated by the self-position and orientation estimation unit 5, and outputs the alpha map to the synthesis unit 9. The alpha map is information indicating the opacity (alpha value) of each pixel of the VST image (image captured by the image sensor 3). In this example, 0 represents complete transparency and 1 represents complete opacity.
[0027] The synthesis unit 9 synthesizes the image captured by the image sensor 3 (VST image) with the virtual object image based on the rendering result (virtual object image) and depth information of the virtual object Vo output by the virtual object rendering unit 7 and the distance image output by the distance image generation unit 6. Specifically, in this case, synthesis is performed by leaving pixels in the captured image that do not overlap with the virtual object image, and for pixels that overlap with the virtual object image, leaving the pixel with the smaller depth value, i.e., the closer pixel, based on the depth of the pixel identified from the distance image and the depth of the pixel in the virtual object image that overlaps with the pixel. As a result of this synthesis process, if the real object Ro overlaps behind the virtual object Vo in the field of view of the VST image, the pixels of the virtual object Vo that are in the foreground are left, and conversely, if the virtual object Vo overlaps behind the real object Ro, the pixels of the real object Ro that are in the background are left.
[0028] Furthermore, the composition unit 9 adjusts the transmittance of the composited image based on the alpha map output by the transmittance control unit 8. Specifically, for pixels of the captured image whose opacity is specified as a value less than 1 by the alpha map, the composition unit 9 adjusts the transmittance by alpha blending the pixels of the virtual object image whose positions overlap with those pixels.
[0029] In this example, a GPU is used for the above-mentioned composition process by the composition unit 9. However, this is just an example, and it is also possible to use a CPU for the composition process.
[0030] The output image from the synthesis unit 9 is displayed to the user via a display system configured with a display control unit 10 and a display unit 11. The display unit 11 has a display panel such as an LCD (Liquid Crystal Display) panel or an organic EL (Electro Luminescence) panel, and displays an image to the user. In the case of an HMD, the display unit 11 also includes an optical system that allows the user to view the image displayed by the display panel.
[0031] The display control unit 10 is configured to have a driver circuit for the display panel in the display unit 11, and drives the display panel based on the output image of the synthesis unit 9. As a result, the output image of the synthesis unit 9, i.e., the image for experiencing the AR space, is displayed to the user.
[0032] [1-2. Transmittance Control Method as First Embodiment] As mentioned above, Patent Document 1 discloses a technique for hiding a real object (increasing the transmittance of a real object) Ro when the real object Ro is in front of a virtual object Vo in the line of sight of a user (observer) observing the virtual object Vo. This makes it possible to prevent the virtual object Vo from being blocked by the real object Ro and becoming invisible when the real object Ro is in front of the virtual object Vo that the user is focusing on.
[0033] 2 is a schematic diagram showing a conventional transmittance control method as the transmittance control method described in Patent Document 1. When a real object Ro is present in front of a virtual object Vo as shown in Fig. 2A, the real object Ro present in front is made transparent as shown in Fig. 2B.
[0034] However, if the real object Ro in front of the virtual object Vo is made transparent as in Patent Document 1, when the distance between the user and the real object Ro is short or when the user is moving fast, the user may have difficulty noticing the approach of the real object Ro, and there is a risk that the user may collide with the real object Ro.
[0035] Therefore, in the first embodiment, the transmittance control for avoiding the above-described collision between the user and the real object Ro is performed by the above-described transmittance control unit 8. Specifically, when a real object is present between the viewpoint position and the virtual object, the transmittance control unit 8 determines whether or not there is a risk of a collision between the user and the real object Ro based on at least one of the distance between the user and the real object Ro and the user's moving speed, and if it determines that there is such a risk, performs control to reduce the transmittance of the real object Ro.
[0036] Here, the "viewpoint position" refers to the position of the user's viewpoint in the AR space. The "viewpoint position" referred to here does not necessarily have to be the exact position of the user's eyes, but may be, for example, the center position of the user's face, the center position of the entire user, or the position of the user's own device (information processing device 1) as the user's viewpoint position. The "viewpoint position" may be any position where the user's viewpoint can be considered to exist.
[0037] In this example, the transmittance control unit 8 determines whether the distance between the user and the real object Ro is equal to or less than a threshold, and if the distance is equal to or less than the threshold, performs control to reduce the transmittance of the real object Ro. Furthermore, in this example, the transmittance control unit 8 determines whether the scene involves a collision risk by determining whether the user's moving speed is equal to or greater than a threshold, and if the moving speed is equal to or greater than the threshold, performs control to reduce the transmittance of the real object Ro. Hereinafter, the distance between the user and the real object Ro in front of the virtual object Vo will be referred to as "distance D," and the user's moving speed will be referred to as "moving speed S."
[0038] Furthermore, in controlling the transmittance of the real object Ro based on the distance between the user and the real object Ro, the transmittance control unit 8 in this example gradually decreases the transmittance of the real object Ro according to the magnitude of the distance D. Specifically, in the information processing device 1 in this example, a first threshold THd1 and a second threshold THd2 smaller than the first threshold THd1 are defined as thresholds for the distance D, and the transmittance control unit 8 gradually decreases the transmittance of the real object Ro according to the magnitude of the distance D when the distance D is equal to or less than the first threshold THd1 and exceeds the second threshold THd2, and controls the transmittance of the real object Ro to 0 when the distance D is equal to or less than the second threshold. More specifically, in a region where the distance D is equal to or less than the first threshold THd1 and exceeds the second threshold THd2, the transmittance control unit 8 in this example determines the opacity (denoted as α) as follows, for example: α=1-(D / 1.5-1)
[0039] In addition, the transmittance control unit 8 sets the opacity of the real object Ro to 1 when the distance D is equal to or less than the second threshold value THd2, and sets the opacity of the real object Ro to 0 when the distance D exceeds the first threshold value THd1.
[0040] That is, in this case, in the region where the distance D between the user and the real object Ro present between the viewpoint position and the virtual object Vo exceeds the first threshold value THd1 (when it is estimated that there is no risk of collision), the opacity becomes 0 and the real object Ro becomes completely transparent, and in the region where the distance D is equal to or less than the first threshold value THd1, the opacity becomes greater than 0 and the transmittance of the real object Ro is reduced. Then, in the region where the distance D is equal to or less than the first threshold value THd1 and exceeds the second threshold value THd2, the opacity increases as the distance D decreases and the transmittance of the real object Ro gradually decreases, and when the distance D becomes equal to or less than the second threshold value THd2, the opacity becomes 1 and the real object Ro becomes completely opaque.
[0041] For confirmation, when a real object Ro exists in front of the virtual object Vo, the composition unit 9 performs processing to leave pixels on the virtual object Vo side, while for pixels of the captured image (pixels of the real object Ro) whose opacity has been specified by the transmittance control unit 8 to be a value less than 1, the composition unit 9 adjusts the transmittance by alpha blending the pixels with pixels of the virtual object image that overlap with the pixels. Therefore, when a real object Ro exists in front of the virtual object Vo, the transmittance control of the real object Ro in the region where THd1≧D>THd2 as described above is realized.
[0042] Also, just to confirm, in this example, in the area where the distance D exceeds the first threshold value THd1, the real object Ro located in front of the virtual object Vo becomes transparent, and therefore, in this area, as in the case of Patent Document 1, the effect of improving the visibility of the virtual object Vo located behind the real object Ro is obtained.
[0043] By controlling the transmittance according to the magnitude of the distance D as described above, it is possible to increase the transparency of the real object Ro when the risk of collision with the real object Ro is low, and decrease the transparency of the real object Ro as the risk of collision with the real object Ro increases. Therefore, it is possible to achieve both the effect of improving the visibility of the virtual object Vo located behind the real object Ro and the effect of preventing collision with the real object Ro.
[0044] Here, the transmittance control unit 8 calculates the distance D based on the depth information of the corresponding real object Ro in the distance image input from the distance image generation unit 6. For example, the smallest depth value among the depth values of each pixel of the corresponding real object Ro may be set as the distance D. Alternatively, the average value of the depth values of each pixel of the corresponding real object Ro may be calculated as the distance D. The distance D may be calculated based on the depth information of the corresponding real object Ro in the distance image input from the distance image generation unit 6.
[0045] Furthermore, the transmittance control unit 8 calculates the moving speed S based on the self-position estimated by the self-position and orientation estimation unit 5. For example, the moving speed S can be calculated as the change ΔP in the self-position between frames. In this case, when the self-position in the current frame is P1(x, y, z) and the self-position in the previous frame is P0(x, y, z), the change ΔP(x, y, z) can be calculated as the norm of P1 and P0 as follows: ΔP = ∥P1 - P0∥ Note that, in order to suppress the influence of sensor noise, the moving speed S may be calculated as a moving average of the change ΔP over N frames. Alternatively, the moving speed S can be calculated as a speed obtained by first-order differentiation of the self-position.
[0046] Furthermore, the transmittance control unit 8 in this example determines whether the position and ratio of the virtual object Vo in the field of view satisfy predetermined conditions. If it determines that the predetermined conditions are not satisfied, it does not perform scene determination and controls the real object Ro to be opaque. Here, even if a real object Ro is present in front of the virtual object Vo, if the virtual object Vo is small or located at the edge of the field of view, it is highly likely that the user is not paying attention to the virtual object Vo. Therefore, making the real object Ro transparent in such a case may cause the user to feel uncomfortable. Therefore, as described above, it determines whether the position and ratio of the virtual object Vo in the field of view satisfy predetermined conditions. If it determines that the predetermined conditions are not satisfied, it controls the real object Ro in front of the virtual object Vo to be opaque. As a result, even if a real object Ro is present in front of the virtual object Vo, if it is estimated that the user is not paying attention to the virtual object Vo, the real object Ro in front of the virtual object Vo can be made opaque and not be transmitted, thereby preventing the user from feeling uncomfortable.
[0047] Specifically, in this example, the transmittance control unit 8 determines whether the above-mentioned specified conditions are met by determining whether the proportion of the virtual object Vo (virtual object image) in the field of view (in this example, the range of the captured image when the VST method is adopted) is equal to or greater than a specified threshold value THr, and whether the position of the virtual object Vo in the field of view is within a specified area Ar that includes the center of the field of view.
[0048] The threshold value THr and the size of the predetermined area Ar may be variably set according to the user operation, the type of AR content, and the like.
[0049] 3 is a flowchart showing an example of a specific processing procedure for realizing the transmittance control method according to the first embodiment described above. Specifically, the flowchart shows an example of a specific processing procedure to be executed by the transmittance control unit 8.
[0050] First, in step S101, the transmittance control unit 8 determines whether or not there is a virtual object whose position and ratio in the field of view satisfy predetermined conditions. Specifically, based on the rendering result of the virtual object Vo input from the virtual object rendering unit 7, it determines whether or not there is a virtual object Vo whose ratio to the field of view is equal to or greater than a predetermined threshold value THr and whose position in the field of view is within a predetermined area Ar.
[0051] If it is determined in step S101 that there is no virtual object Vo that satisfies the above-described predetermined condition, the transmittance control unit 8 proceeds to step S106 and determines whether or not the process has ended. That is, it determines whether or not a predetermined processing end condition that indicates that the process shown in FIG. 3 should be ended has been met, such as an external instruction to stop the process. If the processing end condition has not been met and it is determined that the process has not ended, the transmittance control unit 8 returns to step S101. Here, the loop from step S106 to step S101 is performed at a frame cycle.
[0052] As described above, when it is determined in step S101 that no virtual object Vo satisfies the predetermined condition, the transmittance control unit 8 does not output the opacity to the synthesis unit 9. Therefore, when no virtual object Vo satisfies the predetermined condition in step S101, even if a real object Ro exists in front of the virtual object Vo, the real object Ro is not transparent and becomes opaque. In other words, even if a real object Ro exists in front of the virtual object Vo, if it is estimated that the user is not paying attention to the virtual object Vo, the real object Ro in front is not transparent and becomes opaque.
[0053] If it is estimated in step S101 that a virtual object Vo that satisfies the above-described predetermined condition exists, the transmittance control unit 8 proceeds to step S102 and determines whether or not a real object Ro exists in front of the virtual object Vo. That is, it determines whether or not a real object Ro exists in front of the virtual object Vo that satisfies the predetermined condition in step S101.
[0054] If it is determined in step S102 that the real object Ro does not exist in front of the virtual object Vo, the transmittance control unit 8 proceeds to step S106. As described above, the composition unit 9 performs composition processing so as to leave the pixels on the foreground, so in this case the real object Ro is not transparent and is displayed completely opaque.
[0055] On the other hand, if it is determined in step S102 that the real object Ro is in front of the virtual object Vo, the transmittance control unit 8 proceeds to step S103 and determines whether the user's moving speed is equal to or greater than a threshold value, i.e., whether the above-mentioned moving speed S is equal to or greater than a predetermined threshold value.
[0056] If it is determined in step S103 that the user's moving speed is equal to or greater than the threshold, the transmittance control unit 8 proceeds to step S104, outputs 1 as the opacity of the real object Ro, and proceeds to step S106. As a result, even if the real object Ro is present in front of the virtual object Vo, if the user's moving speed is equal to or greater than the threshold, that is, if it is estimated that there is a risk of collision between the user and the real object Ro, the real object Ro is not made transparent and becomes opaque.
[0057] On the other hand, if it is determined in step S103 that the user's moving speed is equal to or greater than the threshold value, the transmittance control unit 8 proceeds to step S105, where it determines and outputs the opacity of the real object Ro according to the distance D between the user position and the real object Ro. Specifically, in this example, the transmittance control unit 8 determines the opacity of the real object Ro using the first threshold value THd1 and the second threshold value THd2 as described above, and outputs the determined opacity to the synthesis unit 9. This realizes the transmittance control of the real object Ro according to the magnitude of the distance D as described above. Specifically, in a region where the distance D exceeds the first threshold value THd1 (when it is estimated that there is no collision risk), the opacity becomes 0 and the real object Ro is in a completely transparent state, whereas in a region where the distance D is equal to or less than the first threshold value THd1, the opacity becomes greater than 0 and the transmittance of the real object Ro is reduced. In the region where the distance D is less than the first threshold value THd1 and exceeds the second threshold value THd2, the opacity increases as the distance D decreases, and the transmittance of the real object Ro gradually decreases. When the distance D becomes less than the second threshold value THd2, the opacity becomes 1 and the real object Ro becomes completely opaque.
[0058] After executing the process of step S105, the transmittance control unit 8 advances the process to step S106.
[0059] If it is determined in step S106 that the process has ended, the transmittance control unit 8 ends the series of processes shown in FIG.
[0060] While the above describes an example in which a scene with a collision risk is determined based on both the distance D and the movement speed S, it is also possible to determine a scene with a collision risk based on only either the distance D or the movement speed S. In this case, when performing scene determination based only on the distance D of the distance D and the movement speed S, it is also possible to determine whether the movement speed S is equal to or greater than a threshold value regardless of the determination result of whether the real object Ro is in front of the virtual object Vo, and to set the opacity of the real object Ro to a value less than 1 (for example, 0) if the movement speed S is equal to or greater than the threshold. Conversely, when performing scene determination based only on the movement speed S of the distance D and the movement speed S, it is also possible to determine whether the distance D is equal to or less than a threshold value regardless of the determination result of whether the real object Ro is in front of the virtual object Vo, and to set the opacity of the real object Ro to a value less than 1 (for example, 0) if the distance D is equal to or less than the threshold.
[0061] Furthermore, although the example in which only the user's moving speed is used has been given, the moving speed of the real object Ro as well as the user may be used. In this case, it is also possible to use the relative speed between the user and the real object Ro. Furthermore, the direction of movement may also be taken into consideration. For example, it is possible to consider whether the user and the real object Ro are moving in a direction that brings them closer to each other.
[0062] [1-4. Another example of the first embodiment] Although the example has been described in which the transmittance control method of the embodiment is applied when the user experiences an AR space using the VST method, the transmittance control method of the embodiment can also be applied when the user experiences an AR space using the OST method.
[0063] In the OST system, unlike the VST system, in which the real object Ro is shown to the user through a captured image, the real object Ro is shown to the user directly without the intervention of a captured image. Therefore, in order to control the transmittance of the real object Ro, a dimming unit 16 is provided in addition to a display unit 11 used to display the virtual object Vo, as shown in FIG. 4 . The dimming unit 16 is configured, for example, with a liquid crystal panel or the like, and performs dimming of light emitted from objects in the real world and directed to the user's eyes. Specifically, the dimming unit 16 is configured to perform so-called segmented dimming, which performs dimming for each region that divides the user's field of view, and is capable of dimming only a specific real object Ro.
[0064] 5 is a block diagram showing an example of the configuration of an information processing device 1A as another example of the first embodiment that supports the OST system. In the following explanation, parts that are the same as parts that have already been explained will be assigned the same reference numerals and explanations thereof will be omitted.
[0065] The information processing device 1A differs from the information processing device 1 in that a calculation unit 15A is provided instead of the calculation unit 15 and that a light adjustment unit 16 is added. The calculation unit 15A differs from the calculation unit 15 in that a combination unit 9A is provided instead of the combination unit 9. The combination unit 9A differs from the combination unit 9 in that the function of alpha blending the pixels of the real object Ro and the pixels of the virtual object image in accordance with the opacity instructed by the transmittance control unit 8 is omitted.
[0066] 3, but differs from the case of the information processing device 1 in that the output destination of the opacity in steps S104 and S105 is the light adjustment unit 16. In this case, the light adjustment unit 16 minimizes the amount of light reduction for pixels whose opacity is 1, and increases the amount of light reduction for pixels whose opacity is less than 1 as the opacity approaches 0.
[0067] When the OST method is adopted, the information processing device 1A as described above can realize the transmittance control method as the first embodiment.
[0068] Here, in the second to fourth embodiments described below, by adopting a configuration for controlling the dimming unit 16 in the same manner as described above, it is possible to appropriately control the transmittance of the real object Ro in response to the case where the OST method is adopted.
[0069] In the first embodiment, the process flow is such that when the moving speed S is equal to or less than a threshold value, the transmittance is controlled to correspond to the magnitude of the distance D. However, conversely, it is also possible to adopt a process flow in which when the distance D is equal to or greater than a threshold value, the transmittance is controlled to correspond to the magnitude of the moving speed S. In this case, as in the case of the distance D, an area in which the transmittance gradually changes according to the magnitude of the moving speed S may be provided.
[0070] Although the above example illustrates a configuration in which only one display unit 11 is provided to display an AR space image to the user, a configuration in which a display unit 11 for the user's right eye and a display unit 11 for the user's left eye are provided may also be considered. In that case, transmittance control according to the embodiment may be performed for each of the right eye and the left eye. This also applies to the embodiments described below.
[0071] 2. Second Embodiment Next, a description will be given of a second embodiment. In the second and third embodiments, scene determination is performed by determining whether or not a user has performed a predetermined action on a virtual object Vo.
[0072] In the second embodiment, the virtual object Vo is a graspable object that is expected to be grasped by a user, and the real object Ro is a graspable object used to simulate grasping of the graspable object, and the transmittance of the real object Ro is controlled.
[0073] Here, the following two patterns can be broadly considered as methods for allowing a user to virtually grasp a virtual object Vo as a graspable object in an AR space. The first pattern is a method in which the virtual object Vo as a graspable object is superimposed on a real object Ro, and the user actually grasps the real object Ro with the superimposed graspable object, thereby appearing to grasp the graspable object. The second pattern is a method in which the virtual object Vo as the graspable object is displayed, and the user makes a predetermined gesture with their hand near the object, or performs a predetermined operation using an operation device while holding the operation device for playing AR content, thereby appearing to grasp the object. Here, if the object used to virtually grasp the graspable object is referred to as a "graspable object," in the first pattern, the graspable object corresponds to the user's hand, and in the second pattern, the graspable object corresponds to the operation device.
[0074] In the second embodiment, when a real object Ro as a grasped object is present in front of a virtual object Vo as a grasped object, scene determination is made to determine whether or not the user has pseudo-grasped the grasped object, and if it is determined that the user has pseudo-grasped the grasped object, control is performed to increase the transmittance of the grasped object.
[0075] 6 is an explanatory diagram of a transmittance control method according to a second embodiment. FIG. 6A illustrates a scene in which a user attempts to virtually grasp a virtual object Vo (a graspable object) with a hand using a grasping object (a hand) in the case where the first pattern described above is employed. When the user attempts to virtually grasp the graspable object, a real object Ro (a grasping object: in this case, the hand) is present in front of the virtual object Vo (the graspable object). In this case, if the transmittance of the real object Ro is increased due to the presence of the real object Ro in front of the virtual object Vo, as in Patent Document 1, it may be difficult for the user to grasp the sense of distance from the virtual object Vo (the graspable object), and the user may not be able to grasp the graspable object as desired.
[0076] For this reason, in the second embodiment, when it is determined that the user has pseudo-held the object to be grasped as described above, control is performed to increase the transmittance of the object to be grasped (see FIG. 6B ). That is, in a situation where a real object Ro as the object to be grasped is present in front of a virtual object Vo as the object to be grasped, the transmittance of the real object Ro as the object to be grasped is not increased and the real object Ro as the object to be grasped is kept in an opaque state until it is determined that the user has pseudo-held the object to be grasped.
[0077] This makes it easier for the user to grasp the sense of distance from the virtual object Vo as the graspable object, thereby preventing it from becoming difficult to pseudo-grasp the graspable object. Furthermore, since the transmittance of the graspable object is increased in response to a determination that the graspable object has been pseudo-grabbed, the user can see through the virtual object Vo (grabbable object) covered by the graspable object. For example, if the graspable object has descriptive information such as letters or marks, the descriptive information can be visually recognized by the user. Therefore, the transmittance control method of the second embodiment can achieve both improved ease for the user to pseudo-grasp the graspable object and improved visibility of the virtual object Vo.
[0078] 7 is a block diagram showing an example of the configuration of an information processing device 1B in the second embodiment, which corresponds to the case where the first pattern described above is adopted. The difference from the information processing device 1 shown in FIG. 3 is that a calculation unit 15B is provided instead of the calculation unit 15. The calculation unit 15A differs from the calculation unit 15 in that an object recognition unit 17 is added and that a transmittance control unit 8B is provided instead of the transmittance control unit 8.
[0079] The object recognition unit 17 recognizes the real object Ro (hereinafter referred to as the "superimposed object") on which the grasped object is to be superimposed, and recognizes the grasped object (the user's hand in this example). Specifically, the object recognition unit 17 in this example performs object recognition processing based on the image captured by the image sensor 3 and the distance image obtained by the distance image generation unit 6. The object recognition processing here also includes processing to identify the three-dimensional position and orientation of the recognized object in a world coordinate system. Note that the object recognition processing can be realized, for example, using object recognition AI (artificial intelligence) such as a CNN that is machine-learned based on the image captured by the image sensor.
[0080] In this case, the virtual object rendering unit 7 performs processing to render the virtual object Vo as the object to be grasped, based on information about the position and orientation of the real object Ro as the superimposed object recognized by the object recognition unit 17. Specifically, the object to be grasped is rendered so that the object to be grasped is superimposed on the superimposed object. In this case, the output information of the virtual object rendering unit 7 includes information about the position and orientation of the virtual object Vo as the object to be grasped.
[0081] The transmittance control unit 8B performs transmittance control as the second embodiment described above, i.e., when a real object Ro as a grasped object is present in front of a virtual object Vo as a grasped object, as a scene determination, it determines whether the user has pseudo-grasped the grasped object, and if it determines that the user has pseudo-grasped the grasped object, it performs control to increase the transmittance of the grasped object.
[0082] Specifically, in this example, the transmittance control unit 8B determines whether the user has pseudo-grabbed the object based on the distance between the user's hand and the object recognized by the object recognition unit 17 and the amount of movement of the object. More specifically, the transmittance control unit 8B determines whether the user has pseudo-grabbed the object by determining whether the distance between the user's hand and the object is equal to or less than a certain value and the amount of movement of the object is equal to or greater than a certain value. When the first pattern is adopted, when the user pseudo-grabs the object, the user moves their hand toward the object (superimposed object) and then actually grasps the object. In response to the pseudo-grabbing of the object, the user lifts the object (superimposed object), for example, to move the object. Therefore, by making a determination based on the distance between the user's hand and the object and the amount of movement of the object as described above, it is possible to appropriately determine whether the object has been pseudo-grabbed.
[0083] 8 is a flowchart showing an example of a specific processing procedure for implementing the transmittance control method corresponding to the case where the first pattern described above is adopted. In FIG. 8, in step S201, the transmittance control unit 8B determines whether a hand and a graspable object are being detected within the field of view. This determination can be made based on, for example, the recognition result of the user's hand by the object recognition unit 17 and the position information of the virtual object Vo as the graspable object output by the virtual object drawing unit 7.
[0084] If it is determined in step S201 that a hand and an object to be grasped are not being detected within the field of view, the transmittance control unit 8B proceeds to step S205, where it determines whether or not the process has ended. The process of step S205, like the process of step S106, is a process of determining whether or not a predetermined process end condition is met. If it is determined in step S205 that the process has not ended, the transmittance control unit 8B returns to step S201. In this case as well, if it is determined that the process has not ended, the loop is performed at a frame cycle.
[0085] On the other hand, if it is determined in step S201 that the hand and the object to be grasped are being detected within the field of view, the transmittance control unit 8B proceeds to step S202, where it determines whether the distance between the hand and the object to be grasped is equal to or less than a certain value. If it is determined in step S202 that the distance between the hand and the object to be grasped is not equal to or less than the certain value, the transmittance control unit 8B proceeds to step S205. As a result, when the distance between the hand and the object to be grasped is not equal to or less than the certain value, the opacity of the real object Ro is not output to the synthesis unit 9. Therefore, if the hand is located in front of the virtual object Vo as the object to be grasped, the hand is displayed in a completely opaque state.
[0086] On the other hand, if it is determined in step S202 that the distance between the hand and the object to be grasped is equal to or less than a certain value, the transmittance control unit 8B proceeds to step S203, where it determines whether the amount of movement of the object to be grasped is equal to or greater than a certain value. If it is determined in step S203 that the amount of movement of the object to be grasped is not equal to or greater than the certain value, the transmittance control unit 8B proceeds to step S205. As a result, if it is estimated that the user is not virtually grasping the object to be grasped, the opacity of the real object Ro is not output to the synthesis unit 9. Therefore, if the hand is located in front of the virtual object Vo as the object to be grasped, the hand is displayed in a completely opaque state.
[0087] If it is determined in step S203 that the amount of movement of the object to be grasped is equal to or greater than a certain value, the transmittance control unit 8B proceeds to step S204, performs processing to output a predetermined value less than 1 as the opacity of the hand, and proceeds to step S205. This makes it possible to reduce the transmittance of the hand in response to cases where it is estimated that the user has pseudo-grasped the object to be grasped.
[0088] The numerical value output as the opacity of the hand in step S204 is not particularly limited, but for example, if the effect of improving the visibility of the grasped object is to be enhanced, it is conceivable to output 0 (complete transparency).
[0089] If it is determined in step S205 that the process has ended, the transmittance control unit 8B ends the series of processes shown in FIG.
[0090] When the first pattern is adopted, the distance between the hand and the object to be grasped can also be calculated as the distance between the hand and the superimposed object.
[0091] Next, a case where the above-mentioned second pattern is adopted will be described. Fig. 9 is a block diagram showing an example of the configuration of an information processing device 1C corresponding to the case where the second pattern is adopted. The difference from the information processing device 1B shown in Fig. 7 is that a calculation unit 15C is provided instead of the calculation unit 15B. The calculation unit 15C differs from the calculation unit 15B in that a transmittance control unit 8C is provided instead of the transmittance control unit 8B.
[0092] Note that the object recognition unit 17 in the calculation unit 15C does not need an object recognition function for the superimposed object. Furthermore, the virtual object rendering unit 7 in this case does not need a function for rendering the virtual object Vo as the graspable object so as to be superimposed on the real object Ro as the superimposed object. When a pseudo-grasp operation of the virtual object Vo as the graspable object is not performed, the virtual object Vo as the graspable object is rendered as a virtual object Vo existing at a predetermined position in the AR space. When a pseudo-grasp operation is performed, the virtual object Vo is rendered as a virtual object Vo existing at a position that has a predetermined positional relationship with the position of the grasped object (the user's hand in this example). In this case, too, the output information of the virtual object rendering unit 7 includes information on the position and orientation of the virtual object Vo as the graspable object.
[0093] The transmittance control unit 8C controls the transmittance of the object to be grasped (the user's hand in this example) in accordance with the second pattern. Specifically, in this example, the transmittance control unit 8C determines whether the user has pseudo-grasped the object to be grasped based on a gesture operation using the hand. More specifically, the transmittance control unit 8C determines whether a predetermined gesture operation using the hand has been performed as a determination of whether the user has pseudo-grasped the object to be grasped. In this case, too, when it is determined that the user has pseudo-grasped the object to be grasped, the transmittance of the user's hand as the object to be grasped is controlled to be reduced.
[0094] 10 is a flowchart showing an example of a specific processing procedure for implementing the transmittance control method corresponding to the case where the second pattern described above is adopted. In step S301, the transmittance control unit 8C determines whether a hand and a virtual object Vo as a graspable object are being detected within the field of view. This determination can be made, for example, based on the object recognition result of the hand by the object recognition unit 17 and the position information of the virtual object Vo as a graspable object output by the virtual object drawing unit 7.
[0095] If it is determined in step S301 that the hand and the virtual object Vo as the graspable object are not being detected within the field of view, the transmittance control unit 8C proceeds to step S304 and determines whether or not the processing has ended. The processing in step S304, like the processing in steps S106 and S205, is processing for determining whether or not a predetermined processing end condition is met. If it is determined in step S304 that the processing has not ended, the transmittance control unit 8C returns to step S301. In this case as well, if it is determined that the processing has not ended, the loop is performed at a frame cycle.
[0096] On the other hand, if it is determined in step S301 that a hand and a virtual object Vo as a graspable object are being detected within the field of view, the transmittance control unit 8C proceeds to step S302 and determines whether a pinch gesture of the hand has been detected. Note that pinch gestures are merely an example of the detection target, and it is of course possible to assign other gestures using the hand as pseudo-grasp operations when the second pattern is adopted. Gesture detection (gesture recognition) here can be achieved, for example, using recognition AI such as a CNN that has been machine-learned based on images captured by an image sensor.
[0097] If it is determined in step S302 that a pinch gesture of the hand has not been detected, the transmittance control unit 8C proceeds to step S304. As a result, in a state in which it is estimated that an operation of pseudo-grasping the object to be grasped has not been performed, the opacity of the real object Ro is not output to the synthesis unit 9. Therefore, if the hand is positioned in front of the virtual object Vo as the object to be grasped, the hand is displayed in a completely opaque state.
[0098] On the other hand, if it is determined in step S302 that a pinch gesture of the hand has been detected, the transmittance control unit 8C proceeds to step S303, performs processing to output a predetermined value less than 1 as the opacity of the hand, and proceeds to step S304. This makes it possible to reduce the transmittance of the hand in response to a case in which it is estimated that the user has pseudo-grasped the object to be grasped.
[0099] Note that the numerical value output as the opacity of the hand in step S303 is not particularly limited, but in this case too, it is possible to output 0 (complete transparency), for example, to improve the visibility of the grasped object.
[0100] If it is determined in step S304 that the process is to be completed, the transmittance control unit 8C ends the series of processes shown in FIG.
[0101] As described above, when the second pattern is adopted, an operation using an operation device may be assigned to the operation of pseudo-grasping the object to be grasped. In this case, the transmittance control unit 8C may determine whether the user has pseudo-grasped the object to be grasped based on the operation using the operation device. Furthermore, when an operation using an operation device is assigned to the operation of pseudo-grasping the object to be grasped in this way, it is also possible to perform control to increase the transmittance of the real object Ro serving as the operation device when it is determined that an operation of pseudo-grasping the object to be grasped has been performed.
[0102] 3. Third Embodiment The third embodiment aims to achieve both improved ease of grasping the sense of distance to the virtual object Vo and improved visibility of the virtual object Vo in a scene in which the user tries to touch the virtual object Vo with their fingertip, for example, when the virtual object Vo is an operation button or the like and the user operates the operation button.
[0103] Specifically, in the third embodiment, when a real object Ro as a user's hand is present in front of a virtual object Vo, a scene determination is made based on the determination result of whether or not the user's fingertips and the user's line of sight overlap, and if it is determined that the fingertips and the line of sight overlap, control is performed to reduce the transmittance of the real object Ro as the hand.
[0104] 11A is an explanatory diagram of a transmittance control method according to a third embodiment. For example, as illustrated in FIG. 11A , when a scene is assumed in which a virtual object Vo is some kind of operation button and a user's hand, which is a real object Ro, is about to operate this operation button, the real object Ro exists between the user and the virtual object Vo. However, even in this case, if the transmittance of the hand, which is the real object Ro, is increased simply because the real object Ro is in front of the virtual object Vo, it becomes difficult for the user to grasp the positional relationship between the button to be operated and the hand, which may prevent the user from operating the button as desired.
[0105] Therefore, in the third embodiment, scene determination is performed by determining whether the user's fingertip and the user's line of sight overlap, as described above. When a user attempts to touch a virtual object Vo with their fingertip, such as when operating a button, they typically gaze at the target virtual object Vo while bringing their pointing hand toward the virtual object Vo. When touching the virtual object Vo, the user's fingertip and the user's line of sight typically overlap. Therefore, the above-described scene determination can appropriately determine whether the user has touched the virtual object Vo with their fingertip (whether the operation button has been operated). Furthermore, when the user's fingertip and the user's line of sight overlap in a situation where the user is attempting to touch the virtual object Vo with their fingertip, as described above, the user's line of sight also overlaps the virtual object Vo. Therefore, the above-described scene determination also serves as a determination of whether the user wants to view information depicted on the virtual object Vo, such as a button.
[0106] In the third embodiment, when it is determined as a result of the scene determination that the user's fingertip and the user's line of sight overlap, control is performed to reduce the transmittance of the real object Ro as the user's hand (see FIG. 11B ). That is, in a situation where the real object Ro as the user's hand is present in front of the virtual object Vo, the transmittance of the real object Ro as the user's hand is not increased and the real object Ro as the user's hand is kept in an opaque state until it is estimated that the user has touched the virtual object Vo with their fingertip.
[0107] This makes it easier for the user to grasp the sense of distance to the virtual object Vo when the user is about to touch the virtual object Vo with their fingertips, for example, when they are about to operate an operation button serving as the virtual object Vo. Furthermore, when it is estimated that the user has touched the virtual object Vo with their fingertips, the transmittance of the user's hand is increased, allowing the user to see through the virtual object Vo covered by the user's fingertips. For example, when the virtual object Vo has written information such as letters or marks, the written information can be made visible to the user. Therefore, the transmittance control method of the third embodiment can achieve both improved ease of grasping the sense of distance to the virtual object Vo and improved visibility of the virtual object Vo.
[0108] Fig. 12 is a block diagram showing an example of the configuration of an information processing device 1D according to a third embodiment. The differences from the information processing device 1C shown in Fig. 9 are that a gaze estimation unit 18 is added and that a calculation unit 15D is provided instead of the calculation unit 15C.
[0109] The gaze estimation unit 18 detects the user's gaze direction. Various configurations are possible for estimating the user's gaze direction, but the gaze estimation unit 18 in this example is configured to include a camera (inner camera) that is configured to capture an image of the user's eyes and estimate the user's gaze direction through image analysis of the image captured by the camera. In this case, the gaze estimation unit 18 first estimates the user's gaze direction in a camera coordinate system and converts the user's gaze direction in the camera coordinate system into directional information in a world coordinate system based on the above-mentioned inner camera placement position information, etc.
[0110] The calculation unit 15D differs from the calculation unit 15C in that a transmittance control unit 8D is provided instead of the transmittance control unit 8C. Similarly to the virtual object drawing unit 7 in the information processing device 1, the calculation unit 15D draws a virtual object Vo in accordance with drawing instruction information from the outside.
[0111] The transmittance control unit 8D performs the transmittance control of the third embodiment described above, i.e., when a real object Ro as the user's hand is present in front of the virtual object Vo, the transmittance control unit 8D performs scene determination based on the determination result of whether the user's fingertip and the user's line of sight overlap, and if it determines that the fingertip and the line of sight overlap, performs control to lower the transmittance of the real object Ro as the hand. Specifically, the transmittance control unit 8D in this example performs the scene determination based on the determination result of whether the user's fingertip and the user's line of sight overlap, as well as the determination result of whether the user's hand is in a pointing state.
[0112] 13 is a flowchart showing a specific example of a processing procedure for implementing the transmittance control method according to the third embodiment described above. In FIG. 13 , in step S401, the transmittance control unit 8D determines whether the user is looking at the virtual object Vo. This determination can be made based on information about the user's gaze direction estimated by the gaze estimation unit 18 and output information from the virtual object rendering unit 7. Specifically, this determination can be made as a determination of whether the gaze direction overlaps with the virtual object Vo.
[0113] If it is determined in step S401 that the user is not looking at the virtual object Vo, the transmittance control unit 8D proceeds to step S406, where it determines whether or not the process has ended. The process of step S406, like the processes of steps S106 and S205, determines whether or not a predetermined process end condition has been met. If it is determined in step S406 that the process has not ended, the transmittance control unit 8D returns to step S401 (also in this case, if it is determined that the process has not ended, the loop is repeated at the frame cycle).
[0114] On the other hand, if it is determined in step S401 that the user is looking at the virtual object Vo, the transmittance control unit 8D proceeds to step S402 and determines whether or not a hand is present in front of the virtual object Vo that the user is looking at. This determination can be made based on the recognition result information of the user's hand by the object recognition unit 17 and the output information of the virtual object drawing unit 7.
[0115] In step S402, if it is determined that the hand does not exist in front of the virtual object Vo that the user is looking at, the transmittance control unit 8D proceeds to step S406. As a result, if the hand does not exist in front of the virtual object Vo that the user is looking at, the opacity is not output to the synthesis unit 9, and the hand is displayed in a completely opaque state.
[0116] On the other hand, if it is determined in step S402 that a hand is present in front of the virtual object Vo that the user is looking at, the transmittance control unit 8D proceeds to step S403, where it determines whether a pointing gesture has been detected. Note that, as in the case of the information processing device 1C, it is possible to use AI or the like for hand gesture detection (recognition).
[0117] If it is determined in step S403 that a pointing gesture has not been detected, the transmittance control unit 8D advances the process to step S406. That is, in this case as well, the hand is displayed in a completely opaque state.
[0118] On the other hand, if it is determined in step S403 that a pointing gesture has been detected, the transmittance control unit 8D proceeds to step S404, where it determines whether the fingertip and the line of sight overlap. If it is determined in step S404 that the fingertip and the line of sight do not overlap, the transmittance control unit 8D proceeds to step S406. In other words, if it is estimated that the user is not touching the virtual object Vo that the user is looking at with their fingertip, the user's hand is displayed in a completely opaque state.
[0119] If it is determined in step S404 that the fingertip and the line of sight overlap, the transmittance control unit 8D proceeds to step S405, performs processing to output a predetermined value less than 1 as the opacity of the hand, and proceeds to step S406. This makes it possible to reduce the transmittance of the user's hand in response to a case in which it is estimated that the user has come into contact with the virtual object Vo that the user is looking at with his or her fingertip.
[0120] Note that the numerical value output as the opacity of the hand in step S405 is not particularly limited, but in this case too, it is possible to output 0 (complete transparency), for example, to improve the visibility of the object being grasped.
[0121] If it is determined in step S406 that the process has ended, the transmittance control unit 8D ends the series of processes shown in FIG.
[0122] 4. Fourth Embodiment The fourth embodiment provides a measure for the case where there are multiple users sharing an AR space. Fig. 14 is an explanatory diagram of an AR system assumed in the fourth embodiment. For the purpose of explanation, two users, user A and user B, are assumed to share the AR space.
[0123] As shown in the figure, it is assumed that user A and user B are located remotely. In this AR system, user A wears an information processing device 1E according to the fourth embodiment, and user B wears a display device 30. In this example, the display device 30 is assumed to be in the form of an HMD, but the form of the display device 30 is not limited to this and may also be in the form of a mobile computer device such as a smartphone or tablet terminal.
[0124] In the fourth embodiment, an AR space image generated by superimposing a virtual object Vo on an image captured from the viewpoint position of user A (one user) is displayed to user B (another user), allowing user A and user B to share the AR space. In this case, an AR space image based on an image captured from the viewpoint position of user A, generated in the information processing device 1E, is displayed on the display device 30 to user B, so that user B sees the same AR space as the AR space seen by user A. In this case, the imaging range of the captured image used to generate the AR space image is the vicinity of the information processing device 1E, and therefore parts of user B, such as the hands and feet of user B, are not displayed in the AR space image displayed to user B.
[0125] Here, in the case where an AR space image based on an image captured from the viewpoint position of user A is displayed to user B as described above, thereby allowing user A and user B to share the AR space, if the information processing device 1E controls the transmittance of the real object Ro based on user A's line of sight as the AR space image generated for user B, there is a risk that an AR space image that feels unnatural to user B will be generated. Specifically, as a control for controlling the transmittance of the real object Ro, when at least a part of user A's body part, such as a hand, is present in front of the virtual object Vo that user A is looking at, control may be performed to increase the transmittance of that part. However, in this case, it may happen that user B is not looking at the virtual object Vo that user A is looking at. In this case, if the transmittance of user A's body part is similarly increased in the AR space image for user B, at least a part of user A's body part will be visible even though at least a part of user A is not present in front of the virtual object Vo that user B is looking at, which may result in an unnatural feeling for user B.
[0126] Therefore, in the fourth embodiment, as will be described later, the display device 30 is provided with a gaze estimation unit 34 that estimates the gaze direction of user B, and the information processing device 1E controls the transmittance of the part of user A in the AR space image in generating the AR space image to be displayed for user B based on the result of determining whether or not at least a part of user A exists between the viewpoint position of user B and the virtual object Vo that user B is looking at. Specifically, if it is not determined that at least a part of user A exists between the viewpoint position of user B and the virtual object Vo that user B is looking at, the transmittance of the corresponding part of user A in the AR space image for user B is not increased.
[0127] 15A is an explanatory diagram of transmittance control according to a fourth embodiment, and Fig. 15A illustrates a state in which a real object Ro is displayed as user A's hand in a display image for user B (for user B). In the fourth embodiment, if it is determined in the state of Fig. 15A that user A's hand is not in front of the virtual object Vo that user B is looking at, the display state of Fig. 15A is maintained, that is, user A's hand is maintained in a completely opaque state. On the other hand, if it is determined that user A's hand is in front of the virtual object Vo that user B is looking at, the transmittance of user A's hand is increased as shown in Fig. 15B.
[0128] By controlling the transmittance in this manner, it is possible to prevent at least a part of user A from being visible in the image displayed for user B, even though at least a part of user A is not in front of the virtual object Vo that user B is looking at, thereby preventing user B from feeling uncomfortable.
[0129] 16 is a block diagram showing an example of the configuration of a display device 30 according to the fourth embodiment. As shown in the figure, the display device 30 includes a receiving unit 31, a display control unit 32, a display unit 33, a line-of-sight estimation unit 34, an IMU 35, an image sensor 36, a self-position and orientation estimation unit 37, and a transmitting unit 38.
[0130] The receiving unit 31 receives an AR space image for user B, which is generated by the information processing device 1E as described below. The display control unit 32 causes the display unit 33, which has a display panel such as an LCD panel or an organic EL panel and displays images, to display the AR space image received by the receiving unit 31.
[0131] The gaze estimation unit 34 estimates the gaze direction of the user B. Note that the configuration of the gaze estimation unit 34 may be the same as the configuration of the gaze estimation unit 18 described above, and therefore detailed description thereof will be omitted. As for the information on this gaze direction, the IMU 35 outputs three-axis acceleration information and angular velocity information, similar to the IMU 2 described above, and the image sensor 36 obtains a captured image of the outside world of the display device 30.
[0132] In the display device 30, the self-position and orientation estimation unit 37 performs self-position and orientation estimation processing using, for example, SLAM, similar to the self-position and orientation estimation unit 5 described above, based on the acceleration information and angular velocity information from the IMU 35 and the image captured by the image sensor 36, thereby estimating the self-position and orientation of the display device 30.
[0133] The transmission unit 38 transmits information on the gaze direction of the user B estimated by the gaze estimation unit 34 and information on the self-position and orientation of the display device 30 estimated by the self-position and orientation estimation unit 37 to the information processing device 1E.
[0134] Here, it is assumed that a world coordinate system is shared between the information processing device 1E and the display device 30 via markers or a 3D map created in advance. Therefore, the information on the line of sight of user B and the information on the self-position and attitude of the display device 30 transmitted by the transmission unit 38 as described above are generated as information on the line of sight and the self-position and attitude in the world coordinate system on the information processing device 1E side.
[0135] Fig. 17 is a block diagram showing an example of the configuration of an information processing device 1E according to a fourth embodiment. Compared to the information processing device 1D shown in Fig. 12, the information processing device 1E differs in that a calculation unit 15E is provided instead of the calculation unit 15D, and in that a receiving unit 19 and a transmitting unit 21 are added. The receiving unit 19 receives information on the line of sight of user B and information on the self-position and attitude of the display device 30 transmitted by the transmitting unit 38 of the display device 30.
[0136] The calculation unit 15E differs from the calculation unit 15D in that a transmittance control unit 8E is provided instead of the transmittance control unit 8D, and in that a transmission image generation unit 20 is added.
[0137] In the calculation unit 15E, the transmission image generation unit 20 generates an AR space image for user B based on the image captured by the image sensor 3. Furthermore, the transmission image generation unit 20 performs the transmittance control of the AR space image for user B as described in the fourth embodiment, based on information on the gaze direction of user B received by the receiving unit 19, information on the self-position and attitude of the display device 30, information on the object recognition processing result by the object recognition unit 17, and output information from the virtual object rendering unit 7. Specifically, the transmission image generation unit 20 determines whether at least a part of user A's body part is between the viewpoint position of user B and the virtual object Vo viewed by user B, and controls the transmittance of the part of user A in the AR space image for user B based on the result of the determination. Specifically, if it is not determined that at least a part of user A's body part is between the virtual object Vo viewed by user B, the transmittance of the part of user A in the AR space image for user B is not increased. When it is determined that at least a part of user A exists between the virtual object Vo that user B is looking at and the display device 30, the transmittance of the relevant part of user A is increased. Here, in this example, the position of user B's viewpoint is determined based on the information on the self-position of the display device 30 received by the receiving unit 19. Furthermore, the virtual object Vo that user B is looking at can be detected as the virtual object Vo that exists in user B's line of sight.
[0138] The transmission unit 21 transmits the AR space image for user B generated by the transmission image generation unit 20 to the display device 30.
[0139] The transmittance control unit 8E performs transmittance control for the AR space image for user A similar to the transmittance control of the fourth embodiment described above, based on information on the gaze direction of user A estimated by the gaze estimation unit 18, information on the self-position and orientation estimated by the self-position and orientation estimation unit 5, information on the object recognition processing result by the object recognition unit 17, and output information from the virtual object drawing unit 7. Specifically, the transmittance control unit 8E determines whether at least a part of user A's body part is present between the viewpoint position of user A and the virtual object Vo viewed by user A, and controls the transmittance of the part of user A in the AR space image for user A (the composite image obtained by the composition unit 9) based on the result of the determination. Specifically, if it is not determined that at least a part of user A's body part is present between the virtual object Vo viewed by user A, the transmittance of the part of user A in the AR space image for user A is not increased. If it is determined that at least a part of user A's body part is present between the virtual object Vo viewed by user A, the transmittance control unit 8E controls the transmittance of the part of user A to be increased.
[0140] 18 is a flowchart showing a specific example of a processing procedure for realizing the transmittance control method according to the fourth embodiment. Here, an example of the processing procedure performed by the transmission image generation unit 20 will be described. The processing performed by the transmittance control unit 8E is the same as the processing shown in FIG. 18 except that the reference user is user A instead of user B, the opacity determined in step S504 is output to the synthesis unit 9, and the synthesis-related processing from steps S505 to S507 is unnecessary. Therefore, a duplicated description will be avoided.
[0141] In step S501, the transmission image generation unit 20 determines whether the user B is looking at the virtual object Vo. This determination can be made based on information on the line of sight direction of the user B received by the receiving unit 31, information on the self-position and orientation of the display device 30, and output information from the virtual object drawing unit 7.
[0142] If it is determined in step S501 that user B is looking at virtual object Vo, the transmission image generation unit 20 proceeds to step S502 and determines whether a hand (user A's hand) is present in front of the virtual object Vo that user B is looking at. This determination can be made based on information on the object recognition processing result by the object recognition unit 17 and output information from the virtual object drawing unit 7.
[0143] If it is determined in step S502 that a hand is present in front of the virtual object Vo that user B is looking at, the transmission image generation unit 20 proceeds to step S503 and determines whether or not a hand is present in the line of sight of user B. This determination can be made based on information on the line of sight direction of user B received by the receiving unit 19, information on the self-position and orientation of the display device 30, and information on the object recognition processing result by the object recognition unit 17.
[0144] If it is determined in step S503 that a hand is present in the line of sight of user B, the transmission image generation unit 20 proceeds to step S504 and determines the opacity of the hand to a predetermined value less than 1. That is, the transmission image generation unit 20 determines the opacity of the hand of user A, which is determined to be present in the line of sight of user B in step S503, to a predetermined value less than 1.
[0145] In response to having performed the determination process in step S504, the transmission image generation unit 20 proceeds to step S506 and performs a process of combining the captured image and the virtual object Vo in accordance with the determined opacity. The combination here is performed using the same method as the combination in the combination unit 9. Therefore, when the process in step S504 above is performed, the transmittance of the hand in the AR space image for user B is increased on the condition that user A's hand is in front of the virtual object Vo that user B is looking at and that the hand is in user B's line of sight.
[0146] In step S507 following step S506, the transmission image generating unit 20 performs processing to output the composite image to the transmitting unit 21.
[0147] In step S508 following step S507, the transmission image generation unit 20 determines whether or not the processing has ended. The processing in step S508 is processing for determining whether or not a predetermined processing end condition is met, similar to the processing in steps S106 and S205. If the transmission image generation unit 20 determines in step S508 that the processing has not ended, it returns to step S501 (also in this case, if it is determined that the processing has not ended, the loop is performed at the frame period).
[0148] Furthermore, in each of the cases where the transmission image generation unit 20 determines in the previous step S501 that user B is not looking at the virtual object Vo, where the transmission image generation unit 20 determines in the previous step S502 that a hand is not present in front of the virtual object Vo that user B is looking at, and where the transmission image generation unit 20 determines in the previous step S503 that a hand is not present in user B's line of sight, the processing proceeds to step S505, where the transmission image generation unit 20 determines the hand's opacity to be 1. Note that, in step S505, if user A's hand is not recognized in the captured image, the transmission image generation unit 20 does not determine the hand's opacity. Then, in response to having executed the processing of step S505, the transmission image generation unit 20 proceeds to the previously described step S506. As a result, if user A's hand is recognized in the captured image but is not present in front of the virtual object Vo that user B is looking at, or if a hand is present in front of the virtual object Vo that user B is looking at but is not present in user B's line of sight, the hand is displayed in a completely opaque state.
[0149] If it is determined in step S508 that the processing has ended, the transmission image generating unit 20 ends the series of processing steps shown in FIG.
[0150] In the fourth embodiment, a condition for reducing the transparency of the hand of user A may be a pointing gesture of the hand as in the third embodiment.
[0151] In addition, in the above example, the AR space image for user B is an image from the same viewpoint as the AR space image for user A. However, it is also possible for the transmission image generation unit 20 to perform, for example, two-dimensional homography transformation or three-dimensional transformation using depth information on the image captured by the image sensor 3, so that the AR space image for user B is an image from the viewpoint position of user B.
[0152] Furthermore, the processing of the transmittance control unit 8E can be similar to the processing by the transmittance control unit 8B described in the second embodiment or the processing by the transmittance control unit 8D described in the third embodiment.
[0153] 5. Fifth Embodiment The fifth embodiment relates to a technique for increasing the ease with which a user can interact with the real world when experiencing a VR space (a space in which everything visible is a virtual object Vo).
[0154] In the case of a VR space in which the real object Ro is not displayed, it is conceivable that the user may want to temporarily interact with the real object Ro behind the virtual object Vo, for example, to eat or drink, answer a phone call, etc. In the fifth embodiment, to accommodate such a case, the transmittance of the virtual object Vo is reduced so that the real object Ro behind the virtual object Vo is visible to the user.
[0155] 19 is an explanatory diagram of a transmittance control method according to a fifth embodiment. In the fifth embodiment, as shown in FIG. 19A , a specific virtual object Vo, such as a virtual wall in a VR space, is defined as a target virtual object Vo. When a user gazes at the target virtual object Vo, the transmittance of at least a portion of the virtual object Vo is increased. This allows the user to see a real object Ro behind the virtual object Vo, enabling the user to interact with the real object Ro ( FIG. 19B ).
[0156] 20 is a block diagram showing an example of the configuration of an information processing device 1F according to a fifth embodiment. The information processing device 1F differs from the information processing device 1 shown in FIG. 1 in that a line-of-sight estimation unit 18 is added and that a calculation unit 15F is provided instead of the calculation unit 15. The calculation unit 15F differs from the calculation unit 15 in that a transmittance control unit 8F is provided instead of the transmittance control unit 8 and that a combination unit 9F is provided instead of the combination unit 9.
[0157] The transmittance control unit 8F performs processing for realizing the transmittance control of the virtual object Vo according to the fifth embodiment described above. Specifically, the transmittance control unit 8F instructs the synthesis unit 9F to increase the transmittance of at least a portion of a specific virtual object Vo (the target virtual object Vo described above) on the condition that the user is gazing at the specific virtual object Vo, based on information on the user's gaze direction estimated by the gaze estimation unit 18, information on the self-position and orientation estimated by the self-position and orientation estimation unit 5, and output information from the virtual object rendering unit 7. Specifically, the transmittance control unit 8F outputs an opacity of less than 1 (for example, 0) to the synthesis unit 9F as the opacity of the virtual object Vo.
[0158] The transmittance control unit 8F in this example imposes the following additional condition as a condition for increasing the transmittance of the target virtual object Vo. That is, the transmittance control unit 8F further imposes a condition that the distance between a predetermined area of the target virtual object Vo and the user is equal to or less than a certain value. In this example, the transmittance of a predetermined area of the target virtual object Vo is increased in response to the user gazing at the predetermined area when the user approaches within a certain distance of the predetermined area of the target virtual object Vo. Here, "gazing" means looking at the target for at least a certain period of time.
[0159] In order to generate an image that reproduces the VR space, the synthesis unit 9F basically outputs the virtual object image drawn by the virtual object drawing unit 7, but when an opacity of less than 1 is specified for a specific virtual object Vo by the transmittance control unit 8F as described above, it performs alpha blending in accordance with the specified opacity for pixels in the virtual object image of the virtual object Vo for which an opacity of less than 1 is specified and corresponding pixels in the captured image input from the image sensor 3, that is, pixels whose positions overlap with pixels in the virtual object image for which an opacity of less than 1 is specified, and outputs the synthesis image obtained by performing the alpha blending.
[0160] Here, the transmittance control unit 8F in this example also controls transmittance based on the distance between the user and the real object Ro. Specifically, the transmittance control unit 8F in this example controls to increase the transmittance of the entire virtual space when the distance between the user and the real object Ro falls below a predetermined warning threshold. This makes it possible to avoid a collision between the user and the real object Ro.
[0161] 21 is a flowchart showing a specific example of a processing procedure for implementing the transmittance control method according to the fifth embodiment described above. In step S601, the transmittance control unit 8F determines whether the distance between the user and the real object Ro is equal to or less than the warning threshold. This determination can be made based at least on the distance image generated by the distance image generation unit 6.
[0162] If it is determined in step S601 that the distance between the user and the real object Ro is equal to or less than the warning threshold, the transmittance control unit 8F proceeds to step S605 and performs processing to output a predetermined value less than 1 as the opacity of the entire virtual space to the synthesis unit 9F. This makes it possible to avoid a collision with the real object Ro as described above.
[0163] In response to execution of the process of step S605, the transmittance control unit 8F proceeds to step S606 and determines whether or not the process has ended. The process of step S606 is a process of determining whether or not a predetermined process end condition is met, similar to the processes of steps S106 and S205. If the transmittance control unit 8F determines in step S606 that the process has not ended, it returns to step S601 (also in this case, if it is determined that the process has not ended, the loop is performed at the frame period).
[0164] Also, if it is determined in step S601 that the distance between the user and the real object Ro is not equal to or less than the warning threshold, the transmittance control unit 8F proceeds to step S602 and determines whether the distance between a predetermined area of the virtual object Vo (target virtual object Vo) and the user is equal to or less than a certain value.
[0165] If it is determined in step S602 that the distance between the user and a predetermined area of the virtual object Vo is equal to or less than a certain value, the transmittance control unit 8F proceeds to step S603 and determines whether the user is gazing at the predetermined area.
[0166] If it is determined in step S603 that the user is gazing at the predetermined region, the transmittance control unit 8F proceeds to step S604, where it performs processing to output a predetermined value less than 1 as the opacity of the predetermined region to the synthesis unit 9F. This makes it possible to increase the transmittance of the target virtual object Vo in response to the establishment of the trigger condition, thereby enabling the user to interact with the real object Ro located behind the target virtual object Vo.
[0167] After executing the process of step S604, the transmittance control unit 8F advances the process to step S606.
[0168] Furthermore, if it is determined in step S602 that the distance between the predetermined area of the virtual object Vo and the user is not equal to or less than a certain value, or if it is determined in step S603 that the user is not gazing at the predetermined area, the transmittance control unit 8F proceeds to step S606. As a result, if the trigger condition is not met, control to increase the transmittance of the target virtual object Vo is not performed.
[0169] If it is determined in step S606 that the process is to be completed, the transmittance control unit 8F ends the series of processes shown in FIG.
[0170] In the fifth embodiment, the trigger conditions for the control to increase the transmittance of the target virtual object Vo are not limited to the conditions described above. In the fifth embodiment, the transmittance control unit 8F may determine whether the user has performed a predetermined action with respect to the virtual object Vo when a real object is present behind the virtual object Vo as viewed from the user's viewpoint, and may control the transmittance of the virtual object Vo based on the result of the determination.
[0171] 6. Modifications Although various embodiments according to the present technology have been described above, the present technology is not limited to the specific examples described above and may adopt configurations as various modifications. For example, although the above describes an example in which the information processing device as each embodiment takes the form of an HMD, the information processing device as an embodiment may also take the form of a mobile computer device such as a smartphone or a tablet terminal, and is not limited to the form of an HMD.
[0172] 7. Program and Recording Medium> Here, as an embodiment, a program that causes, for example, a CPU, a DSP (Digital Signal Processor), or a device including these, to realize the functions of the transmittance control units 8, 8B, 8C, and 8D described with reference to Figures 3, 8, 13, etc., can be considered. That is, the program of the embodiment is a program readable by a computer device, and causes the computer device to realize the function of, when a real object is present between the viewpoint position and a virtual object, performing scene determination related to at least one of the user's behavior and the positional relationship between the user and the real object, and controlling the transmittance of the real object based on the results of the scene determination. Furthermore, a recording medium as an embodiment is a recording medium on which the program of the above embodiment is recorded. Such a program or recording medium allows the functions of the transmittance control unit 8 and the like to be realized in an apparatus such as the information processing device 1.
[0173] The above-described programs can be pre-recorded on a hard disk drive (HDD) or a ROM in a microcomputer having a CPU, which serves as a recording medium built into a computer or other device. Alternatively, the programs can be temporarily or permanently stored (recorded) on removable recording media such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a Magneto Optical (MO) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such programs can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a local area network (LAN) or the Internet.
[0174] Furthermore, such a program is suitable for widely providing the transmittance control method of the embodiment. For example, by downloading the program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the personal computer or the like can function as a device that realizes the transmittance control method of the present disclosure.
[0175] 8. Summary of the Embodiments As described above, the information processing devices (1, 1A, 1B, 1C, 1D, 1E) according to the embodiments include a transmittance control unit (8, 8B, 8C, 8D) that, when a real object is present between a viewpoint position and a virtual object, performs scene determination related to at least one of a user's behavior and a positional relationship between the user and the real object, and controls the transmittance of the real object based on the result of the scene determination. According to the above configuration, the transmittance of a real object present in front of a virtual object is controlled based on the result of scene determination related to a user's behavior, such as the speed at which the user is moving, whether the user is attempting to virtually grasp a virtual object, or whether the user is attempting to operate a virtual object as a button, or the like, or the result of scene determination related to a positional relationship between the user and the real object, such as whether the user is close to the real object. Therefore, it is possible to, for a real object that is in front of a virtual object, not increase the transmittance of the real object if there is a high possibility of the real object colliding with the user, not increase the transmittance of the real object as the user's hand if the user is attempting to grasp a virtual object as an object to be grasped, and not increase the transmittance of the hand if the user is attempting to operate an operation button as a virtual object with the real object as the user's hand, etc. Thus, it is possible to realize appropriate control according to the scene for controlling the transmittance of real objects when there is a real object between the viewpoint position and the virtual object.
[0176] Furthermore, in the information processing device (1, 1A) according to the embodiment, the transmittance control unit (8, 8A) determines whether a scene poses a risk of collision between the user and the real object based on at least one of the distance between the user and the real object and the user's moving speed, and if the scene is determined to pose such a risk, controls to reduce the transmittance of the real object. As a result, even if a real object is present in front of a virtual object, if it is estimated that there is a high possibility of the real object colliding with the user, the transmittance of the real object is reduced. This makes it easier for the user to notice the approach of the real object, preventing a collision between the user and the real object and improving safety. Furthermore, if a real object is present in front of a virtual object and it is estimated that there is a low possibility of the real object colliding with the user, the real object can be made transparent, allowing the user to view the virtual object behind the real object. Therefore, the above configuration can achieve both improved visibility of the virtual object and improved safety.
[0177] Furthermore, in the information processing device according to the embodiment, the transmittance control unit determines whether the distance is equal to or less than a threshold value as scene determination, and performs control to reduce the transmittance of the real object when the distance is equal to or less than the threshold value. By determining whether the distance between the user and the real object is equal to or less than a threshold value as described above, it is possible to appropriately determine whether the scene poses a risk of collision between the user and the real object.
[0178] Furthermore, in the information processing device according to the embodiment, a first threshold and a second threshold smaller than the first threshold are defined as thresholds for distance, and the transmittance control unit gradually reduces the transmittance of the real object in accordance with the distance when the distance is equal to or less than the first threshold and exceeds the second threshold, and controls the transmittance of the real object to 0 when the distance is equal to or less than the second threshold. As a result, in a situation where the distance between the user and the real object gradually decreases, the transmittance of the real object gradually decreases in accordance with the increasing risk of collision. Therefore, it is possible to make the user more aware of the approach of the real object as the risk of collision increases, thereby improving safety.
[0179] In addition, in the information processing device according to the embodiment, the transmittance control unit determines whether the moving speed is equal to or greater than a threshold value as scene determination, and performs control to reduce the transmittance of the real object when the moving speed is equal to or greater than the threshold value. By determining whether the moving speed of the user is equal to or greater than a threshold value as described above, it is possible to appropriately determine whether the scene poses a risk of collision between the user and a real object.
[0180] Furthermore, in the information processing device according to the embodiment, the transmittance control unit performs scene determination based on both the distance and the moving speed. This makes it possible to determine that a scene has a collision risk if the user's moving speed is equal to or greater than a threshold even if the distance between the user and the real object is not equal to or less than a threshold, and conversely, to determine that a scene has a collision risk if the user's moving speed is equal to or less than the threshold even if the user's moving speed is not equal to or greater than the threshold. This makes it possible to improve the accuracy of risk determination and further improve safety.
[0181] Furthermore, in an information processing device according to an embodiment, the transmittance control unit determines whether the position and ratio of a virtual object in the field of view satisfy predetermined conditions. If it determines that the predetermined conditions are not satisfied, scene determination is not performed and control is performed so that the real object is opaque. Even if a real object is present in front of a virtual object, if the virtual object is small or located at the edge of the field of view, it is highly likely that the user is not paying attention to the virtual object. Therefore, making the real object transparent in such a case may cause the user to feel uncomfortable. Therefore, as described above, the transmittance control unit determines whether the position and ratio of the virtual object in the field of view satisfy predetermined conditions, and if it determines that the predetermined conditions are not satisfied, control is performed so that the real object in front of the virtual object is opaque. As a result, even if a real object is present in front of a virtual object, if it is estimated that the user is not paying attention to the virtual object, the real object in front of the virtual object can be made opaque and not made transparent, thereby preventing the user from feeling uncomfortable.
[0182] In addition, in the information processing devices (1B, 1C, 1D) according to the embodiments, the transmittance control unit (8B, 8C, 8D) determines whether the user has performed a predetermined action with respect to a virtual object as a scene determination. This controls the transmittance of the real object in response to the user's predetermined action with respect to the virtual object, such as when the user performs an action to virtually grasp a virtual object as a graspable object or when the user performs an action to point at a predetermined virtual object such as an operation button. Therefore, for a real object located in front of the virtual object, it is possible to prevent the transmittance of the real object of the user's hand from increasing when the user attempts to grasp the virtual object as a graspable object, or to prevent the transmittance of the hand from increasing when the user attempts to operate an operation button as a virtual object with the real object of the user's hand. This allows for appropriate control of the transmittance of the real object depending on the scene when a real object is located between the viewpoint and the virtual object. Specifically, this allows for both improved ease of grasping the sense of distance to the virtual object and improved visibility of the virtual object.
[0183] Furthermore, in the information processing device (1B, 1C) according to the embodiment, the virtual object is a virtual object intended to be grasped by the user, and the real object is a gripping object used to simulate the gripping object. The transmittance control unit (8B, 8C) performs scene determination by determining whether the user has simulatedly grasped the gripping object. If it is determined that the user has simulatedly grasped the gripping object, it controls to increase the transmittance of the gripping object. This makes it possible to prevent the transparency of gripping objects in front of the virtual object intended to be grasped by the user until it is determined that the user has simulatedly grasped the virtual object. This makes it easier for the user to grasp the sense of distance from the virtual object intended to be grasped, thereby preventing difficulty in simulatedly grasping the gripping object. Furthermore, since the transparency of the grasped object is increased in response to the determination that the grasped object has been virtually grasped, the user can see through the virtual object covered by the grasped object. For example, if the grasped object has descriptive information such as letters or marks, the descriptive information can be visually recognized by the user. Therefore, with the above configuration, it is possible to improve both the ease with which the user can virtually grasp the grasped object and the visibility of the virtual object.
[0184] Furthermore, in an information processing device (same as 1B) according to an embodiment, the graspable object is a virtual object superimposed on a predetermined real object, the graspable object is the user's hand, and the transmittance control unit (same as 8B) determines whether a pseudo-grasp has occurred based on the distance between the hand and the graspable object and the amount of movement of the graspable object. When a virtual object as the graspable object is superimposed on a real object as the superimposed object, the pseudo-grasp of the graspable object is performed by the user grasping the superimposed object with their hand. Therefore, in this case, when the user pseudo-grasp the graspable object, the user brings their hand closer to the graspable object (superimposed object) and then takes the action of actually grasping the superimposed object. In response to the pseudo-grasp of the graspable object, the user moves the graspable object (superimposed object) by, for example, lifting it. Therefore, by making a judgment based on the distance between the user's hand and the object to be grasped and the amount of movement of the object to be grasped as described above, it is possible to appropriately determine whether or not a pseudo-grasp of the object to be grasped has been performed.
[0185] In an information processing device (1C) according to an embodiment, the object to be grasped is a user's hand or an operation device having an operator, and the transmittance control unit (8C) determines whether or not a pseudo-grasp has been performed based on a gesture operation using the hand or an operation using the operation device. This makes it possible to appropriately determine whether or not a pseudo-grasp of the object to be grasped has been performed, in response to a case where the pseudo-grasp of the object to be grasped is performed by a predetermined gesture operation using the user's hand or a predetermined operation using the operation device.
[0186] Furthermore, in an information processing device (1D) according to an embodiment, the real object is a user's hand, and the transmittance control unit (8D) performs scene determination based on the result of determining whether the user's fingertip overlaps the user's line of sight. If it is determined that the fingertip overlaps the line of sight, the transmittance control unit increases the transmittance of the real object. This prevents the transmittance of the hand from increasing until it is determined that the user's fingertip overlaps the line of sight, i.e., until it is determined that the fingertip is located on the virtual object. This makes it easier to grasp the sense of distance to the virtual object when operating a virtual object such as an operation button with the user's fingertip. Furthermore, the transmittance of the hand is increased in response to determining that the fingertip is located on the virtual object, allowing the user to see through the virtual object covered by the fingertip. For example, if the virtual object contains descriptive information such as letters or marks, the descriptive information can be visually recognized by the user. Therefore, the above configuration achieves both improved ease of grasping the sense of distance to the virtual object and improved visibility of the virtual object.
[0187] Furthermore, in the information processing device according to the embodiment, the transmittance control unit performs scene determination based on the determination result of whether or not the user's hand is in a pointing state, thereby improving the accuracy of scene determination regarding whether or not the virtual object is an operator that is operated with a fingertip, such as a button.
[0188] In addition, in an information processing device (same as 1E) according to an embodiment, when there are multiple users sharing an AR space and an AR space image generated by superimposing a virtual object on an image captured from the viewpoint position of one of the users is displayed to another of the users, the device includes an AR space image control unit (transmission image generation unit 20) that controls the transmittance of a part of the one user in the AR space image based on a result of determining whether at least a part of the one user's body part is between the viewpoint position of the other user and the virtual object being viewed by the other user. By controlling the transmittance of the one user in the AR space image displayed to the other user based on the virtual object being viewed by the other user as described above, even if at least a part of the one user is between the viewpoint position of the one user and the virtual object, if at least a part of the one user is not between the viewpoint position of the other user and the virtual object, it is possible to prevent at least a part of the one user from being transmitted in the AR space image displayed to the other user. In other words, even though at least a part of the first user is not present in front of the virtual object that the other user is looking at, it is possible to prevent at least a part of the first user from being visible in the AR space image displayed to the other user, thereby preventing the other user from feeling uncomfortable.
[0189] An information processing method according to an embodiment is an information processing method that, when a real object exists between a viewpoint position and a virtual object, performs a scene determination related to at least one of the user's behavior and the positional relationship between the user and the real object, and controls the transmittance of the real object based on the result of the scene determination.
[0190] Furthermore, the recording medium as an embodiment is a recording medium on which a program readable by a computer device is recorded, and on which a program is recorded that causes the computer device to realize the function of, when a real object exists between the viewpoint position and a virtual object, making a scene determination relating to at least one of the user's behavior and the positional relationship between the user and the real object, and controlling the transmittance of the real object based on the result of the scene determination.
[0191] The information processing device according to the above embodiment can be realized by the above information processing method and recording medium.
[0192] Another information processing device (1F) according to an embodiment includes a transmittance control unit (8F) that, when a real object is present behind a virtual object as viewed from the user's viewpoint, determines whether the user has taken a predetermined action with respect to the virtual object and controls the transmittance of the virtual object based on the result of the determination. This makes it possible to make a virtual object in front of a real object transparent in response to the user taking a predetermined action with respect to the virtual object. Therefore, in response to the user taking a predetermined action with respect to the virtual object, the user can see through the real object present behind the virtual object, enabling the user to interact with the real world.
[0193] Furthermore, in another information processing device according to an embodiment, the transmittance control unit determines whether a predetermined action has been performed by determining whether the user gazes at a virtual object while the virtual object is within a predetermined distance from the user. This allows the user to interact with the real world by simply approaching the virtual object within the predetermined distance and gazing at it. In other words, the gesture operation required of the user to interact with the real world is simply to approach the virtual object within the predetermined distance and gaze at it. This reduces the operational burden on the user when interacting with the real world.
[0194] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.
[0195] <9. The Present Technology> The present technology may also have the following configuration. (1) An information processing device including a transmittance control unit that, when a real object is present between a viewpoint position and a virtual object, performs scene determination related to at least one of a user's behavior and a positional relationship between the user and the real object, and controls the transmittance of the real object based on the result of the scene determination. (2) The information processing device described in (1), wherein the transmittance control unit determines, as the scene determination, whether or not there is a risk of a collision between the user and the real object based on at least one of a distance between the user and the real object and a moving speed of the user, and performs control to reduce the transmittance of the real object if it is determined that there is such a risk. (3) The information processing device described in (2), wherein the transmittance control unit determines, as the scene determination, whether or not the distance is equal to or less than a threshold, and performs control to reduce the transmittance of the real object if the distance is equal to or less than the threshold. (4) The information processing device according to any of (2) to (4), wherein a first threshold and a second threshold smaller than the first threshold are defined as thresholds for the distance, and the transmittance control unit gradually reduces the transmittance of the real object in accordance with the magnitude of the distance when the distance is equal to or smaller than the first threshold and exceeds the second threshold, and controls the transmittance of the real object to 0 when the distance is equal to or smaller than the second threshold. (5) The information processing device according to any of (2) to (4), wherein the transmittance control unit determines, as the scene determination, whether the moving speed is equal to or larger than a threshold, and performs control to reduce the transmittance of the real object when the moving speed is equal to or larger than the threshold. (6) The information processing device according to any of (2) to (5), wherein the transmittance control unit performs the scene determination based on both the distance and the moving speed. (7) The information processing device according to any one of (2) to (6), wherein the transmittance control unit determines whether or not a position and a ratio of the virtual object in a field of view satisfy a predetermined condition, and when it is determined that the predetermined condition is not satisfied, the transmittance control unit does not perform the scene determination and controls the real object to be opaque. (8) The information processing device according to (1), wherein the transmittance control unit determines whether or not the user has performed a predetermined action with respect to the virtual object as the scene determination.(9) The information processing device described in (8), wherein the virtual object is a graspable object that is expected to be grasped by the user, and the real object is a gripping object used to pseudo-grasp the graspable object, and the transmittance control unit determines as the scene determination whether the user has pseudo-grabbed the graspable object, and when it is determined that the user has pseudo-grabbed the graspable object, performs control to increase the transmittance of the gripping object. (10) The graspable object is a virtual object that is superimposed on a superimposed object that is a predetermined real object, and the gripping object is the user's hand, and the transmittance control unit determines whether the pseudo-grasp has been performed based on a distance between the hand and the graspable object and a movement amount of the graspable object. (11) The information processing device according to (9), wherein the held object is the user's hand or an operation device having an operator, and the transmittance control unit determines whether the pseudo-grasp has been performed based on a gesture operation using the hand or an operation using the operation device. (12) The information processing device according to (8), wherein the real object is the user's hand, and the transmittance control unit performs the scene determination based on a determination result of whether the user's fingertip and the user's line of sight overlap, and performs control to increase the transmittance of the real object when it is determined that the fingertip and the line of sight overlap. (13) The information processing device according to (12), wherein the transmittance control unit performs the scene determination based on a determination result of whether the user's hand is in a pointing state. (14) In a case where there are multiple users sharing an AR space and an AR space image generated by superimposing a virtual object on an image captured from the viewpoint position of one of the users is displayed to another of the users, the information processing device described in any of (1) to (14) is provided with an AR space image control unit that controls the transmittance of a part of the one user in the AR space image based on a result of determining whether or not at least a part of the one user's body part is between the viewpoint position of the other user and the virtual object that the other user is looking at.(15) An information processing method, when a real object exists between a viewpoint position and a virtual object, performing a scene determination related to at least one of a user's action and a positional relationship between the user and a real object, and controlling the transmittance of the real object based on the result of the scene determination. (16) A recording medium having a computer-readable program recorded thereon, the computer realizing a function of performing a scene determination related to at least one of a user's action and a positional relationship between the user and a real object when a real object exists between a viewpoint position and a virtual object, and controlling the transmittance of the real object based on the result of the scene determination. (17) An information processing device, comprising a transmittance control unit that, when a real object exists behind a virtual object as seen from the user's viewpoint position, determines whether the user has taken a predetermined action with respect to the virtual object, and controls the transmittance of the virtual object based on the result of the determination. (18) The information processing device according to (17), wherein the transmittance control unit determines whether the user has performed the predetermined action by determining whether the user has gazed at the virtual object while the distance between the virtual object and the user is within a predetermined distance.
[0196] REFERENCE SIGNS LIST 1, 1A, 1B, 1C, 1D, 1E, 1F Information processing device 2 IMU 3 Image sensor 4 Distance measurement sensor 5 Self-position and orientation estimation unit 6 Distance image generation unit 7 Virtual object drawing unit 8, 8B, 8C, 8D, 8E, 8F Transmittance control unit 9, 9A, 9F Combining unit 10 Display control unit 11 Display unit 15, 15A, 15C, 15D, 15E, 15F Calculation unit Vo Virtual object Ro Real object 16 Light adjustment unit 17 Object recognition unit 18 Gaze estimation unit 19 Receiving unit 20 Transmission image generation unit 21 Transmitting unit
Claims
1. An information processing device having a transmittance control unit that, when a real object exists between a viewpoint position and a virtual object, performs scene determination related to at least one of a user's behavior and a positional relationship between the user and the real object, and controls the transmittance of the real object based on the result of the scene determination.
2. The information processing device of claim 1, wherein the transmittance control unit determines whether or not the scene poses a risk of collision between the user and the real object based on at least one of the distance between the user and the real object and the user's moving speed as the scene determination, and if it determines that the scene poses such a risk, performs control to reduce the transmittance of the real object.
3. The information processing device according to claim 2, wherein the transmittance control unit determines whether the distance is equal to or less than a threshold as the scene determination, and performs control to reduce the transmittance of the real object if the distance is equal to or less than the threshold.
4. The information processing device of claim 3, wherein a first threshold and a second threshold smaller than the first threshold are defined as thresholds for the distance, and the transmittance control unit reduces the transmittance of the real object in accordance with the magnitude of the distance when the distance is equal to or smaller than the first threshold and exceeds the second threshold, and controls the transmittance of the real object to 0 when the distance is equal to or smaller than the second threshold.
5. The information processing device according to claim 2, wherein the transmittance control unit determines whether the moving speed is equal to or greater than a threshold as the scene determination, and performs control to reduce the transmittance of the real object if the moving speed is equal to or greater than the threshold.
6. The information processing device according to claim 2, wherein the transmittance control unit performs the scene determination based on both the distance and the moving speed.
7. The information processing device according to claim 2, wherein the transmittance control unit determines whether or not the position and ratio of the virtual object in the field of view satisfy predetermined conditions, and if it determines that the predetermined conditions are not satisfied, performs control so as not to perform the scene determination and to make the real object opaque.
8. The information processing device according to claim 1, wherein the transmittance control unit determines, as the scene determination, whether or not the user has performed a predetermined action on the virtual object.
9. The information processing device according to claim 8, wherein the virtual object is a virtual object as a graspable object expected to be grasped by the user, the real object is a graspable object used to pseudo-grasp the graspable object, and the transmittance control unit determines whether or not the user has pseudo-grabbed the graspable object as the scene determination, and when it is determined that the user has pseudo-grabbed the graspable object, controls to increase the transmittance of the graspable object.
10. The information processing device according to claim 9, wherein the graspable object is a virtual object superimposed on a superimposed object which is a specified real object, the grasped object is the user's hand, and the transmittance control unit determines whether or not the pseudo-grasping has been performed based on the distance between the hand and the graspable object and the amount of movement of the graspable object.
11. The information processing device of claim 9, wherein the object to be grasped is the user's hand or an operating device having an operator, and the transmittance control unit determines whether or not the pseudo-grasping has been performed based on a gesture operation using the hand or an operation using the operating device.
12. The information processing device of claim 8, wherein the real object is the user's hand, and the transmittance control unit performs the scene determination based on a determination result of whether or not the user's fingertip and the user's line of sight overlap, and when it is determined that the fingertip and the line of sight overlap, performs control to increase the transmittance of the real object.
13. The information processing device according to claim 12, wherein the transmittance control unit performs the scene determination based on a determination result of whether or not the user's hand is in a pointing state.
14. An information processing device as described in claim 1, comprising: an AR space image control unit that controls the transmittance of a part of the one user in the AR space image based on a result of determining whether or not at least a part of the one user is between the viewpoint position of the other user and the virtual object that the other user is looking at, when there are multiple users sharing an AR space and an AR space image generated by superimposing a virtual object on an image captured from the viewpoint position of one of the users is displayed to another of the users.
15. An information processing method, in which, when a real object exists between a viewpoint position and a virtual object, a scene determination is made relating to at least one of a user's behavior and a positional relationship between the user and the real object, and the transmittance of the real object is controlled based on the result of the scene determination.
16. A recording medium having a program recorded thereon that can be read by a computer device, the program causing the computer device to realize a function of performing scene determination relating to at least one of a user's behavior and a positional relationship between the user and a real object when a real object exists between a viewpoint position and a virtual object, and controlling the transmittance of the real object based on the result of the scene determination.
Citation Information
Patent Citations
3D virtual-reality display device, head-mounted display, and 3D virtual-reality display method
WO2022044124A1
Image processing device, image processing method and program
JP2018063567A
Simulation system and program
JP2018171309A
Information processing device, information processing system, control method thereof and program
JP2018190447A
Image processing apparatus and method for controlling the same
JP2021002290A
