Information processing device and control method for the information processing device

Abstract

This improves the visibility of CG objects to the user when displaying them in the foreground so they are not obscured by real-world objects. [Solution] The information processing device according to the present disclosure includes: generation means for generating a CG object to be displayed in the space viewed by the user; determination means for determining the front-to-back relationship between a real object and the CG object in the space; and processing means for performing a process that, when it is determined that the real object is in front of the CG object from the user's perspective, displays the CG object in front of the real object from the user's perspective and reduces the visibility of a predetermined area of ​​the real object, including the boundary where the CG object and the real object appear to be touching.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus using a technique related to the synthesis of CG (Computer Graphics) objects and a control method for the information processing apparatus.

Background Art

[0002] There are technologies that enable user experiences such as virtual reality space (VR: Virtual Reality) and mixed reality space (MR: Mixed Reality). VR is a technology that represents a virtual world by displaying CG (Computer Graphics) objects of virtual objects on a virtual space. MR is a technology that represents a space that combines the real world and the virtual world by superimposing CG objects on an image of the real space.

[0003] In these technologies, a user wears a head-mounted display (HMD), which is a head-mounted device. In the case of MR where a CG object is superimposed on an image of the real space, due to the depth relationship between the real space and the CG object, an object existing in the real space (hereinafter referred to as a real object) may be in front of the CG object. In this case, the HMD may perform occlusion processing so that the CG object is hidden by the real object.

[0004] On the other hand, when a CG object is superimposed on an image of the real space mainly for the purpose of operating a UI (User Interface) or viewing content, it may be preferable to prevent the CG object from being hidden by the real object. For example, Patent Document 1 discloses a technique for changing the display position and display method of a CG object when at least a part of the CG object is hidden by a real object.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2004-234253 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, in the prior art disclosed in the above-mentioned patent documents, it is sometimes preferable to display CG objects in their original display position or method when performing UI operations or viewing content. In this case, the CG object can be displayed in front of the real object without changing its display position or method. However, the part of the CG object that is actually hidden from view by the real object is displayed as being in front of the real object. As a result, there is a discrepancy between the change in the relative position of the real object and the CG object due to the user's viewpoint movement and the sense of distance the user perceives regarding the CG object, which may cause the user to feel a sense of unease when viewing the CG object.

[0007] The object of the present invention has been made in view of the above, and is to provide a technology that can improve the visibility of CG objects to the user when displaying CG objects in the foreground so as not to be hidden by real objects. [Means for solving the problem]

[0008] The information processing device relating to this disclosure includes a generation means for generating CG objects to be displayed in the space viewed by the user, and a means for determining the spatial relationship between real objects and the CG objects in the space. The device is characterized by comprising: determination means for making a determination; and processing means for performing a process that, when it is determined that the real object is in front of the CG object from the user's perspective, displays the CG object in front of the real object from the user's perspective, and reduces the visibility of a predetermined area of ​​the real object, including the boundary where the CG object and the real object appear to be touching. [Effects of the Invention]

[0009] According to the technology disclosed herein, when displaying a CG object in the foreground so that it is not obscured by real-world objects, the visibility of the CG object to the user can be improved. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing an example of the configuration of the HMD in the first embodiment. [Figure 2] A flowchart of the processes performed by the HMD in the first embodiment. [Figure 3] A schematic diagram illustrating image processing in the HMD in the first embodiment. [Figure 4] A flowchart of the process performed by the HMD in one modified example of the first embodiment. [Figure 5] A block diagram showing an example of the HMD configuration in the second embodiment. [Figure 6] A flowchart of the processes performed by the HMD in the second embodiment. [Figure 7] A schematic diagram illustrating image processing in the HMD in the second embodiment. [Figure 8] A flowchart of the process performed by the HMD in one modified example of the second embodiment. [Modes for carrying out the invention]

[0011] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the invention as defined in the claims. Each embodiment describes multiple features, but not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the accompanying drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] [First Embodiment] A head-mounted display (HMD), which is an example of an information processing device according to the first embodiment, will be described. Figure 1 is a block diagram showing an example of the configuration of an HMD in the first embodiment. As shown in Figure 1, the HMD 100 is a wearable information processing device (electronic device) that can be worn on the user's head. The HMD 100 has a control unit 101, an imaging unit 102, an IMU (Inertial Measurement Unit) 103, and a spatial recognition unit 104. The HMD 100 also has a CG generation unit 105, a distance determination unit 106, an image processing unit 107, an image synthesis unit 108, an image display unit 109, a working memory 120, and a non-volatile memory 121. In this embodiment, the processing described later may be performed by a control unit integrated into the HMD, or it may be performed by a personal computer (PC) or smartphone connected to the HMD.

[0013] The control unit 101 controls various parts of the HMD 100. The imaging unit 102 includes two cameras (imaging devices). The two cameras are positioned symmetrically with respect to the center of the HMD so as to capture images of the real space in front of the user when the HMD 100 is worn by the user. The images captured by the two cameras are stored in the working memory 120, which will be described later.

[0014] In the first embodiment, the HMD100 is assumed to have two cameras, but instead, the HMD100 may be configured to have only one camera, or to have three or more cameras. Furthermore, the arrangement of the cameras in the HMD100 is not limited to the above and may be placed in any position.

[0015] The IMU103 is an inertial measurement device equipped with a gyroscope and an accelerometer. The IMU103 acquires the angular velocity and acceleration of the HMD100 as sensor information using its onboard sensors. Note that the sensors comprising the IMU103 are not limited to those mentioned above and may include other sensors such as geomagnetic sensors.

[0016] The spatial recognition unit 104 estimates the position and orientation (hereinafter referred to as the position and orientation) of the HMD 100 in the real space and recognizes the space around the HMD 100. The spatial recognition unit 104 performs position and orientation estimation and surrounding space mapping using the captured image captured by the imaging unit 102 and the sensor information acquired by the IMU 103. Specifically, the spatial recognition unit 104 executes spatial mapping using the Visual Inertial SLAM (Simultaneous Localization and Mapping) method. The spatial mapping process by the spatial recognition unit 104 is not limited to using Visual Inertial SLAM. For example, the spatial recognition unit 104 may execute spatial mapping using other methods such as Visual SLAM that does not use the sensor information output by the IMU 103. Also, the spatial recognition unit 104 may recognize the space around the HMD 100 using technologies such as LiDAR (Light Detection And Rnging).

[0017] The CG generation unit 105 generates CG objects to be displayed in the space that the user wearing the HMD 100 is looking at. The CG objects generated by the CG generation unit 105 are superimposed on the captured images of the real space captured by the imaging unit 102. The CG objects include various virtual objects such as pre-prepared 3D models and windows for displaying the running applications. Also, when the CG object and the real object seem to overlap from the viewpoint of the user wearing the HMD 100, the CG generation unit 105 performs a processing operation on the CG object to eliminate the sense of incongruity in the display of the CG object received by the user. Details of the processing operation of the CG object executed by the CG generation unit 105 will be described later. Also, the CG generation unit 105 renders the CG object and stores the rendered CG object in the working memory 120 described later. The CG objects generated by the CG generation unit 105 are used for synthesis with the captured images captured by the above-described camera of the HMD 100. Also, a predetermined transparency is set for the CG objects generated by the CG generation unit 105 so that the images of the real space are synthesized.

[0018] The distance determination unit 106 determines the front-back relationship between the real object and the CG object in the real space using the information of the space map obtained by the space mapping by the space recognition unit 104. Specifically, when the CG object and the real object overlap when the user wearing the HMD 100 views the image, the distance determination unit 106 determines whether the CG object is located deeper than the real object as viewed from the HMD 100.

[0019] The image processing unit 107 performs various image processes on the captured images captured by the camera of the HMD 100. The image processes executed by the image processing unit 107 include various correction processes and development processes. The image processing unit 107 reads out the image from the working memory 120 in which the captured image obtained by the imaging unit 102 is stored, and outputs the image (hereinafter, background image) obtained by performing various image processes to the working memory 120.

[0020] The image synthesis unit 108 synthesizes the CG object generated by the CG generation unit 105 and the background image generated by the image processing unit 107, and outputs the synthesized image to the working memory 120.

[0021] The image display unit 109 displays the image generated by the image synthesis unit 108. The image display unit 109 has, for example, a liquid crystal panel or an organic EL panel. When the user wears the HMD 100, the image display unit 109 displays the image in front of each of the user's eyes. It will be mounted on the HMD100 so that the number 9 is positioned correctly.

[0022] The working memory 120 stores various data necessary for the control unit 101 to perform the processing described below. For example, the working memory stores sensor information acquired by the IMU 103, position and orientation information estimated by the spatial recognition unit 104, and spatial map information.

[0023] The non-volatile memory 121 is an electrically erasable and recordable non-volatile storage medium that stores the application program executed by the control unit 101 and various settings necessary for the execution of the application.

[0024] (Processing of the control unit) Next, the processes executed by the control unit 101 will be explained with reference to the flowchart in Figure 2. The processes in the flowchart in Figure 2 are those executed by the control unit 101 within one frame after the HMD 100 is started up. Normally, the control unit 101 executes the following processes according to a predetermined frame rate such as 60Hz or 120Hz, thereby updating and displaying the image in the image display unit 109.

[0025] In step S201, the control unit 101 acquires the captured image taken by the imaging unit 102. The control unit 101 controls the image processing unit 107 to perform image processing such as development processing and various correction processing on the acquired captured image, and stores the processed image (background image) in the working memory 120.

[0026] In step S202, the control unit 101 acquires sensor information from the IMU 103 and stores the acquired sensor information in the working memory 120.

[0027] In step S203, the control unit 101 acquires the position and orientation information of the HMD 100 estimated by the spatial recognition unit 104, spatial map information, and other additional information. The additional information includes information on the tracking status of the HMD 100, information on feature points related to the real space, and information on the time when the position and orientation were estimated. The control unit 101 stores the acquired information in the working memory 120.

[0028] In step S204, the control unit 101 acquires data of the CG object generated by the CG generation unit 105. Alternatively, the control unit 101 may acquire data of the CG object stored in the non-volatile memory 121.

[0029] In step S205, the control unit 101 controls the distance determination unit 106 to determine whether the acquired CG object will be positioned in front of the real object as seen from the HMD 100 when it is combined with the background image generated in step S201. For example, the control unit 101 performs collision detection between each real object in the background image and the CG object based on the position of the HMD 100 and the spatial map information. Based on the result of the collision detection, the control unit 101 determines whether the CG object will be positioned in front of the real object that overlaps with it when it is combined with the background image.

[0030] Alternatively, the control unit 101 sets up a virtual window that is always facing forward to the HMD 100. The control unit 101 performs region selection, object detection, feature extraction, segmentation, etc., within the background image using the virtual window, and compares the distance from the HMD 100 to the CG object with the distance from the HMD 100 to the real object within the virtual window. Then, based on the distance comparison result, the control unit 101 determines that when the CG object is composited with the background image, the CG object will be further away from the real object that overlaps with the CG object. It also determines whether or not to position it in the foreground. Based on this, the control unit 101 makes a processing determination as to whether or not to perform processing on the CG object.

[0031] Another example of this method is that the control unit 101 uses so-called ray tracing to project rays (light rays) connecting the user wearing the HMD 100 to each pixel of the image display unit 109, and determines whether the CG object or the real object intersects the ray first. Based on the determination result, the control unit 101 then determines whether the CG object will be placed in front of the real object that overlaps with it when the CG object is composited with the background image.

[0032] If the control unit 101 determines that the CG object is positioned in front of the real object (S205: YES), it proceeds to step S207. On the other hand, if the control unit 101 determines that the real object is positioned in front of the CG object (S205: NO), it proceeds to step S206.

[0033] In step S206, the control unit 101 controls the CG generation unit 105 to perform CG processing on the CG object data acquired in step S204. In this step, if the CG generation unit 105 determines that a real object is in front of the CG object from the user's perspective, it performs processing to display the CG object in front of the real object from the user's perspective. Furthermore, the CG generation unit 105 performs processing to reduce the visibility of a predetermined area of ​​the real object, including the boundary where the CG object and the real object appear to be touching to the user. Details of the processing performed on the CG object in this step will be described later.

[0034] In step 207, the control unit 101 controls the CG generation unit 105 to perform rendering using the data of the CG object after the CG processing generated in step S206, thereby generating a CG image.

[0035] In step S208, the control unit 101 combines the background image acquired in step S201 with the CG image generated by rendering in step S207, and stores the combined image (hereinafter referred to as the combined image) in the working memory 120.

[0036] In step S209, the control unit 101 updates the image to be displayed on the image display unit 109 with the composite image output in step S208, thereby displaying the composite image on the image display unit 109.

[0037] (Regarding CG processing) Referring to Figure 3, the CG processing performed by the CG generation unit 105 in step S206 will be described. Figure 3A shows an example of an image 301 displayed on the image display unit 109 of the HMD 100. Image 301 depicts real objects 302 placed in real space, which have been captured by the HMD 100's camera. Image 301 also depicts CG objects 303 for displaying the application being run on the HMD 100. CG object 303 is, for example, a CG object for an application that displays content such as a video. In image 301, the CG object 303 is positioned at a predetermined distance from the user (HMD 100) so that it is easily visible to the user. It is also assumed that the CG object 303 is positioned further away from the real objects 302 as seen from the user wearing the HMD 100.

[0038] In the image displayed on the image display unit 109, real objects are in front of CG objects. If both real and CG objects exist, it is preferable for the user if the CG composite is designed so that the real object appears in front of the CG object. However, in the composite image displayed on the image display unit 109, the user may prioritize viewing the CG object over the real object, such as for operation UI or content viewing. In this case, it is preferable that the CG object is not hidden by the real object in the composite image displayed on the image display unit 109, even if the real object is in front of the CG object.

[0039] In image 301 in Figure 3A, the real object 302 is positioned in front of the CG object 303 from the perspective of the user wearing the HMD 100, but the CG object 303 is composited to appear in front of the real object 302.

[0040] In such cases, the CG object appears to be closer to the user on display, but the user experiences discomfort because the perceived distance differs from that perceived distance due to the change in the relative position of the CG object and the real object as the user moves their viewpoint. This discomfort experienced by the user wearing the HMD100 is due to the change in the relative position of the real object and the CG object as the user moves their viewpoint. Therefore, in the case of image 301, it is expected that this discomfort can be alleviated by separating the real object 302 and the CG object 303 at the boundary 304 where the real object 302 and the CG object 303 meet. Accordingly, in the HMD100 according to this embodiment, the CG object processing is performed on the region including the part adjacent to the boundary 304, as described below.

[0041] Figure 3B shows Image 311, an example of the result of processing Image 301 with the CG object 303 in this embodiment. In Image 311 of Figure 3B, the processing causes the CG object 303 to appear in front of the real object 302. Furthermore, the processing causes the CG object 305 to be placed in an area that includes the portion adjacent to the boundary 304 and hides a part of the real object 302. The CG object 305 is an object that reduces the visibility of that area. As a result, when a user views Image 311 displayed on the image display unit 109, they will feel less of a change in the relative position between the real object 302 and the CG object 303 due to changes in viewpoint compared to when viewing Image 301. As a result, the discomfort a user may feel when viewing the CG object 303 due to the real object 302 can be mitigated.

[0042] Furthermore, in the example shown in Figure 3B, the CG object 305 generated by the processing is placed only in a portion of the area surrounding the CG object 303, including the boundary 304. However, in this embodiment, a processing operation may be performed to generate a CG object 306 that surrounds the entire circumference of the CG object 303, as illustrated in image 321 in Figure 3C.

[0043] Based on the above, the HMD100 according to this embodiment can reduce the sense of discomfort experienced by the user when viewing CG objects in images displayed on the image display unit 109, thereby improving the visibility of CG objects.

[0044] Here, with reference to Figure 4, the process executed by the control unit 101 in one modified example of this embodiment will be described. In the flowchart of Figure 4, the same reference numerals are used for the same steps as in Figure 2, and detailed explanations are omitted. In this modified example, if a real object displayed on the image display unit 109 is within reach of the user's hand, and the real object is hidden by a CG object, the user may unintentionally come into contact with the real object when performing actions such as moving their hand. Therefore, in step S401, the control unit 101 executes a distance acquisition process to acquire the distance from the HMD 100 to the real object based on the information acquired by the spatial recognition unit 104 and the distance determination unit 106. Then, if the acquired distance is less than or equal to the distance at which the user's hand will come into contact with the real object, the control unit 101 executes (S4 01:YES), assuming the user may come into contact with a real object, the process proceeds to step S207. As a result, the control unit 101 executes a control that does not process or display the CG objects 305 and 306. Alternatively, instead of executing a control that does not process or display the CG objects 305 and 306, the control unit 101 may execute a control that increases the transparency of the CG objects 305 and 306 in the above processing. This is expected to make it easier for the user to recognize the real object 302 when viewing the CG object 303 through the image displayed in the image display unit 109.

[0045] (Second Embodiment) Next, an information processing device according to the second embodiment will be described. In this embodiment as well, an HMD is assumed as an example of an information processing device. In the following description, the same reference numerals are used for components and processes as in the HMD according to the first embodiment, and detailed explanations are omitted.

[0046] The HMD according to the second embodiment will now be described with reference to Figure 5. Figure 5 is a block diagram showing an example of the configuration of the HMD 500 according to this embodiment. As shown in Figure 5, the HMD 500 according to this embodiment has an evaluation value acquisition unit 510 in addition to the configuration of the HMD 100 according to the first embodiment. The processing described later in this embodiment may be performed by a control unit integrated into the HMD, or by a personal computer (PC) or smartphone connected to the HMD.

[0047] The evaluation value acquisition unit 510 uses the camera image output by the image processing unit 107 to acquire evaluation values ​​regarding the visibility of each real-world object depicted in the image. In this embodiment, in addition to the processing described in the first embodiment, the CG generation unit 105 adaptively generates CG objects to be displayed on the image display unit 109 based on the evaluation values ​​acquired by the evaluation value acquisition unit 510. Details of the evaluation value acquisition process and the CG object processing process will be described later.

[0048] (Processing of the control unit) Referring to the flowchart in Figure 6, an example of the processing performed by the control unit 101 in this embodiment will be described.

[0049] Steps S601 and S603 through S610 correspond to steps S201 and S202 through S209 in the flowchart of Figure 2 in the first embodiment, respectively. Therefore, a detailed explanation of the processing in each of those steps will be omitted here.

[0050] In step S602, the control unit 101 controls the evaluation value acquisition unit 510 to acquire an evaluation value regarding the visibility of the real object depicted in the image, using the captured image acquired in step S601.

[0051] Furthermore, in step S608, the control unit 101 controls the CG generation unit 105 to perform processing on the CG object that is composited with the background image generated in step S601 based on the evaluation value acquired in step S602. The control unit 101 stores the processed CG object generated by the processing in this step in the working memory 120.

[0052] (Process for obtaining evaluation values ​​and generating processed CG objects) Referring to Figures 7A to 7D, the evaluation value acquisition process by the evaluation value acquisition unit 510 and the CG object processing process by the CG generation unit 105 in this embodiment will be explained.

[0053] Figure 7A shows image 701, which is the background image generated in step S601. As shown in Figure 7A, a real object 702 is depicted in image 701.

[0054] Figure 7B shows the state in which the image 701 has been divided into blocks of a predetermined size by the evaluation value acquisition unit 510. For each divided block of the image 701, the evaluation value acquisition unit 510 acquires the average value of the pixel values ​​within the block as an evaluation value related to the visibility of real objects in each block. The size and shape of the blocks into which the image 701 is divided may be appropriately determined according to the size of the image 701 and the processing load of each part of the HMD 500.

[0055] In this embodiment, the image captured by the camera is divided into multiple blocks, and the average value of the pixel values ​​calculated for each block is used as the evaluation value. However, the method for obtaining the evaluation value is not limited to this, and a method for obtaining the evaluation value based on the content of the CG object may be adopted. As an example, the control unit 101 divides the display area of ​​the CG object 703 and obtains the average value of the pixel values ​​for each divided area as the evaluation value. In the example of image 701 in Figure 7B, the control unit 101 calculates the average value of the pixel values ​​for each of the multiple divided areas adjacent to the boundary 704 between the CG object 703 and the real object 702. Then, for blocks adjacent to the boundary 704 outside the CG object 703, the control unit 101 obtains the average value of the pixel values ​​of the divided area of ​​the CG object 703 adjacent to the block as the evaluation value. As a result, by applying the block blurring process described below, the image 701 can give the viewer a visual effect that makes it appear as if the area of ​​the CG object 703 extends beyond the boundary 704 and light is leaking out from the CG object 703.

[0056] Figure 7B shows a CG object 703, which is the display area of ​​an application executed using the HMD 500 in Image 701. Figure 7B also shows the boundary 704 where the CG object 703 touches the real-world object 702 in Image 701. The CG object 703 displays a window for the application, and the window displays content such as videos executed by the application. The spatial position of the CG object 703 is determined to be a predetermined distance away from the user wearing the HMD 500, so that the user can easily view the content. The distance between the CG object 703 and the HMD 500 may be appropriately set to a distance that is generally easy for a user of an HMD to view the content. Furthermore, in this embodiment, the display position of the CG object 703 is assumed to be farther away from the real-world object 702 as viewed from the user wearing the HMD 500.

[0057] Figure 7C schematically shows the evaluation values ​​for each block of the image 701, including the block containing the boundary 704 and the region 705 containing the adjacent blocks, among the evaluation values ​​acquired for each block by the evaluation value acquisition unit 510. In Figure 7C, blocks with the same evaluation value are given the same pattern. In the example of Figure 7C, evaluation values ​​that are similar to each other are also considered to be the same evaluation value. Figure 7D shows an example of the result when the control unit 101 controls the CG generation unit 105 to process the image 701 to include the boundary 704 based on the evaluation values ​​shown in the region 705, and the generated CG object 706 is combined with the image 701.

[0058] In this embodiment, the control unit 101 controls the CG generation unit 105 to generate an image of each block using the average value of the pixel values, which are evaluation values. As a result, each block is blurred according to the degree of blurring corresponding to the size of the block.

[0059] Alternatively, instead of blurring the block using the above process, the control unit 101 may determine the pixel value of each pixel within the block by referring to a predetermined range of pixel values ​​centered on the pixel of interest, and then generate the block image using the determined pixel values. This allows the control unit 101 to apply blurring more adaptively according to the image within the block. Alternatively, instead of blurring, a fill process may be used, where the pixel value of each pixel within a block is set to a constant value. This is expected to reduce the processing load on the CG generation unit 105 for processing CG objects.

[0060] As shown in Figure 7D, in image 701, a CG object 706 generated by processing is displayed in the region including the boundary 704 where the real object 702 and the CG object 703 appear to be touching. As a result, when the user views the content displayed on the CG object 703, the user perceives the CG object 703 and the real object 702 as being separated. Consequently, even when the CG object 703 is displayed in the HMD 500 without being hidden by the real object 702, the user will be less likely to perceive changes in the relative position between the real object 702 and the CG object 703 due to changes in viewpoint. Therefore, the discomfort the user experiences when viewing the CG object 703 due to the real object 702 can be mitigated, and the visibility of the CG object 703 can be improved.

[0061] In this embodiment, the control unit 101 may perform image processing on the CG object 706 so that the user perceives a greater affinity between the CG object 706 and the real space in the image 701. For example, the control unit 101 may apply interpolation processing such as bilinear or bicubic interpolation to the CG object 706 to smooth the boundaries between blocks. In addition, various image processing methods that enhance affinity with the real space may be employed for the CG object 706, not limited to interpolation processing.

[0062] Furthermore, in this embodiment, the application range of the blurring process by the CG object 706 may be changed by the control unit 101 according to the distance between the user and the CG object 703 or the real object 702. For example, if the distance from the user to the CG object 703 and the distance from the user to the real object 702 are approximately the same, the difference in parallax when the user views the CG object 703 and the real object 702 with their left and right eyes is considered to be negligible. Therefore, in such cases, the control unit 101 generates the CG object 706 such that the range of the blurring process by the CG object 706 becomes smaller. This expands the area in which the real image is visible in the composite image displayed on the image display unit 109.

[0063] Furthermore, for example, it is conceivable that the difference between the distance from the user to the CG object 703 and the distance from the user to the real object 702 may be large. As an example, in the composite image displayed on the image display unit 109, it is conceivable that the CG object 703 is positioned 1 m in front of the HMD 500 and the real object 702 is positioned 10 cm in front of the HMD 500. In such a case, the parallax when the user views the CG object 703 and the real object 702 with their left and right eyes will also be larger. As a result, the sense of discomfort the user experiences when viewing the CG object 703 due to the real object 702 will also be greater. Therefore, in such cases, the control unit 101 generates the CG object 706 such that the range of blurring by the CG object 706 is larger. This reduces the sense of discomfort the user experiences when viewing the CG object due to the real object 702 and improves the visibility of the CG object.

[0064] Furthermore, as a modified example of this embodiment, for example, when a user reaches out to the CG object 703 and attempts to manipulate the CG object 703, the control unit 101 may perform control to prevent the above-mentioned processing of the CG object 706. Figure 8 shows a flowchart of the process executed by the control unit 101 in this modified example. In the flowchart of Figure 8, the same reference numerals are used for the same steps as in Figure 6, and detailed explanations are omitted. In step S801, the control unit 101 determines whether or not it has detected an operation on the CG object 603 by a user wearing the HMD 400, based on the captured image captured by the imaging unit 102. Note: Method for detecting user operation on CG object 603 Various methods based on prior art may be employed for this. If the control unit 101 determines that it has detected a user operation on the CG object 603 (S801:YES), it proceeds to step S802.

[0065] In step S802, the control unit 101 controls the image synthesis unit 108 and the image display unit 109 to hide the CG object 706 to which blurring is applied, which is displayed on the image display unit 109. Then, in step S803, the control unit 101 updates the image displayed on the image display unit 109 with the synthesized image output in step S802, and displays the updated synthesized image on the image display unit 109. As a result, when the user attempts to manipulate the CG object 603, the control unit 101 cancels the blurring applied to the CG object 706 and restores the original processing. As a result, the user can more easily see the real object 702 in image 701. Consequently, when the user examines the CG object 703, the sense of unease caused by the real object 602 is reduced, and when the user manipulates the CG object 703, the possibility of the user's hand coming into contact with the real object 702 is reduced.

[0066] Furthermore, although the above embodiments assume that HMDs 100 and 500 are video see-through type HMDs, the configuration and processing of the HMDs in each embodiment can also be applied to optical see-through type HMDs. If HMDs 100 and 500 are optical see-through type HMDs, the imaging unit 102 has a camera and acquires captured images of the real space.

[0067] Furthermore, in the above embodiment, the control unit 101 processes a CG object that has been blurred onto an image of real space in one frame. However, the control unit 101 may also apply the CG object generated by the processing to the image of real space in the next frame. This is expected to reduce the processing time of the composite image associated with the generation of CG objects by the processing in the HMDs 100 and 500, while also mitigating the sense of incongruity that users experience when viewing CG objects due to real objects.

[0068] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0069] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0070] Furthermore, the embodiments described above (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.

[0071] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or recording medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0072] This embodiment includes the following configurations, methods, programs, and storage media. (Composition 1) A means for generating CG objects to be displayed in the space viewed by the user, A determination means for determining the spatial relationship between a real object and a CG object, Processing means that, when it is determined from the user's perspective that the real object is in front of the CG object, displays the CG object in front of the real object from the user's perspective and processes it to reduce the visibility of a predetermined area of ​​the real object, including the boundary where the CG object and the real object appear to be touching. An information processing device characterized by having the following features. (Configuration 2) The information processing apparatus according to configuration 1, characterized in that the processing is a process of arranging an object in the space to reduce the visibility of the predetermined area. (Composition 3) The information processing apparatus according to configuration 1, characterized in that the processing is a fill-in process of the predetermined area. (Composition 4) The information processing apparatus according to configuration 1, characterized in that the processing is a blurring process of the predetermined area. (Composition 5) The processing means performs the processing to reduce the visibility of the real object in the predetermined region based on the pixel values ​​in the predetermined region of the image captured from real space. An information processing device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The system further includes means for acquiring the distance from the information processing device to the actual object, The processing means does not perform the processing if the calculated distance is less than or equal to a predetermined distance. An information processing device according to any one of configurations 1 to 5, characterized by the above. (Composition 7) The processing means is characterized in that when the user attempts to perform an operation on the CG object, the processing is restored to its original state, as described in any one of configurations 1 to 6. (method) A method for controlling an information processing device, A generation step that generates CG objects to be displayed in the space the user is viewing, A determination step of determining the spatial relationship between a real object and a CG object, If it is determined that the real object is in front of the CG object from the user's perspective, the processing step involves displaying the CG object in front of the real object from the user's perspective, and reducing the visibility of a predetermined area of ​​the real object, including the boundary where the CG object and the real object appear to be touching. A control method for an information processing device, characterized by having the following features. (program) A program for causing a computer to function as one of the means of an information processing device described in any one of items 1 to 7 of the configuration. (storage medium) A storage medium for storing programs that cause a computer to function as one of the means of the information processing device described in any one of items 1 to 7 of the configuration. [Explanation of symbols]

[0073] 100 HMD, 101 Control unit, 104 Spatial recognition unit, 105 CG generation unit, 106 Distance determination unit, 107 Image processing unit

Claims

1. A means for generating CG objects to be displayed in the space viewed by the user, A determination means for determining the spatial relationship between a real object and a CG object, When it is determined that the real object is in front of the CG object from the user's perspective, a processing means is provided to display the CG object in front of the real object from the user's perspective, and to perform processing that reduces the visibility of a predetermined area of ​​the real object, including the boundary where the CG object and the real object appear to be touching. An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, characterized in that the processing is a process of arranging an object in the space to reduce the visibility of the predetermined area.

3. The information processing apparatus according to claim 1, characterized in that the processing is a fill-in process of the predetermined area.

4. The information processing apparatus according to claim 1, characterized in that the processing is a blurring process of the predetermined area.

5. The processing means performs the processing to reduce the visibility of the real object in the predetermined region based on the pixel values ​​in the predetermined region of the image captured from real space. The information processing apparatus according to feature 1.

6. The system further includes means for acquiring the distance from the information processing device to the actual object, The processing means does not perform the processing if the calculated distance is less than or equal to a predetermined distance. The information processing apparatus according to feature 1.

7. The processing means is characterized in that it can be restored to its original state when the user attempts to perform an operation on the CG object, as described in claim 1.

8. A method for controlling an information processing device, A generation step that generates CG objects to be displayed in the space the user is viewing, A determination step of determining the spatial relationship between a real object and a CG object, If it is determined that the real object is in front of the CG object from the user's perspective, the processing step involves displaying the CG object in front of the real object from the user's perspective, and reducing the visibility of a predetermined area of ​​the real object, including the boundary where the CG object and the real object appear to be touching. A control method for an information processing device, characterized by having the following features.

9. A program for causing a computer to function as one of the means of an information processing apparatus according to any one of claims 1 to 7.

10. A storage medium for storing a program that causes a computer to function as one of the means of an information processing apparatus according to any one of claims 1 to 7.

Images