Image processing device, and image processing method

Abstract

To decrease discomfort of a user caused by difference in appearance of an MR image displayed in an HMD of a video see-through system from a real space.SOLUTION: An image processing device includes: image acquisition unit which acquires a background image; distance map acquisition means which acquires a distance map of the background image; line-of-sight acquisition means which acquires line-of-sight information of a user; visible distance acquisition means which acquires a visible distance being a distance from a photographing device for photographing the back ground image to a position corresponding to a line-of-sight position of the user on the basis of the line-of-sight information of the user; and image processing means which executes a blurring process to the background image on the basis of the visible distance and the distance on the distance map.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] Mixed Reality (MR) technology is known, which combines real and virtual spaces, allowing users to interact with virtual objects. MR systems using MR technology realize interaction between real and virtual spaces by synthesizing and displaying computer graphics (CG) that represent virtual objects with real scenery, and by expressing contact between real and virtual objects. Because the images of the virtual space displayed by MR technology are displayed clearly regardless of the distance from the user, it can be difficult to provide a realistic viewing experience, unlike the scenery in real space that the user actually sees.

[0003] Patent Document 1 discloses a method for capturing an image with a camera lens mounted on an HMD, in which the subject the user is focusing on is not blurred and the background is blurred. Patent Document 2 discloses a method for blurring a CG image displayed on an optical see-through HMD based on the amount of blur associated with the distance of the CG image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-227950 [Patent Document 2] International Publication No. 2020 / 115815 Summary of the Invention [Problem to be solved by the invention]

[0005] Even if blurring is applied to CG images, the background image of the MR image in a video see-through HMD will be in focus regardless of the distance from the user. A background image that is in focus across the entire image may cause discomfort to the user.

[0006] Therefore, an object of the present invention is to reduce the sense of discomfort felt by the user due to the difference in appearance of MR images displayed on a video see-through HMD from the appearance of real space. [Means for solving the problem]

[0007] The image processing device of the present invention is characterized by having an image acquisition means for acquiring a background image, a distance map acquisition means for acquiring a distance map of the background image, a gaze acquisition means for acquiring information on a user's gaze, a visible distance acquisition means for acquiring, based on the information on the user's gaze, a visible distance, which is the distance from an imaging device that captures the background image to a position corresponding to the user's gaze position, and an image processing means for performing a blurring process on the background image based on the visible distance and the distance in the distance map. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce the sense of discomfort felt by the user due to the difference between the appearance of an MR image displayed on a video see-through HMD and the appearance of real space. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Figure 2] 1 is a block diagram of an information processing system according to a first embodiment. [Figure 3] FIG. 10 is a diagram illustrating the relationship between the convergence angle and the visible distance. [Figure 4] 10A and 10B are diagrams illustrating differences in blurring processing due to differences in convergence angles. [Figure 5] FIG. 10 is a diagram illustrating differences in distance between objects. [Figure 6] 10A and 10B are diagrams illustrating differences in blurring processing depending on differences in the distance between objects. [Figure 7] 1 is a flowchart illustrating an example of an image blurring process according to the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating a difference in visibility distance. [Figure 9] 10A and 10B are diagrams illustrating differences in blurring processing due to differences in visible distance. [Figure 10] FIG. 10 is a block diagram of an information processing system according to a second embodiment. [Figure 11] 10 is a flowchart illustrating an example of an image blurring process according to the second embodiment. [Figure 12] FIG. 10 is a block diagram of an information processing system according to a third embodiment. [Figure 13] 10 is a flowchart illustrating an image blurring process according to a third embodiment. [Figure 14] FIG. 1 is a block diagram of an HMD incorporating an image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Embodiment 1> 1 is a diagram illustrating the configuration of an information processing system 1. The information processing system 1 includes an HMD 100 and an image processing device 110. The HMD 100 and the image processing device 110 are shown in FIG.

[0011] The HMD 100 is a video see-through head-mounted display device that is worn on the user's head 130. The HMD 100 displays a composite image that combines, for example, a captured image of the area in front of the user captured by the HMD 100 with an object such as computer graphics (CG) in a form that corresponds to the posture of the HMD 100. The captured image of the area in front of the user captured by the HMD 100 is hereinafter also referred to as a background image.

[0012] The image processing device 110 executes processes such as controlling the HMD 100, processing captured images captured by a camera mounted on the HMD 100, generating objects such as CG, and generating composite images to be displayed on the HMD 100. The image processing device 110 is configured by a computer including a processor such as a CPU (Central Processing Unit) and a memory. The image processing device 110 may be, for example, a smartphone, a tablet terminal, a PC (personal computer), a game console, or the like. The image processing device 110 is connected to the HMD 100 wirelessly or via a wired connection. The image processing device 110 generates a composite image by combining captured images and CG, and transmits the composite image to the HMD 100. Note that some or all of the components of the image processing device 110 may be built into the HMD 100.

[0013] 2 is a block diagram of the information processing system 1 according to embodiment 1. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an alignment imaging unit 203, a display unit 204, and a convergence angle acquisition unit 205.

[0014] The HMD control unit 201 controls each component of the HMD 100, thereby controlling the entire HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by combining a background image and a CG image) from the image processing device 110, the HMD control unit 201 displays the composite image on the display unit 204. By wearing the HMD 100, the user can view the composite image displayed on the display unit 204. The user can experience various mixed realities, such as CG blended into real space.

[0015] The imaging unit 202 may include two cameras (imaging devices). The two cameras are arranged near the positions of the left and right eyes of the user when wearing the HMD 100, in order to capture an image of a space similar to the space the user normally sees. The background image obtained by capturing the subject (surrounding area) is output to the image processing device 110. Furthermore, the two cameras in the imaging unit 202 can acquire information on the distance from the two cameras to the subject as distance information by measuring distances using a stereo camera.

[0016] The positioning imaging unit 203 is a camera (imaging device) provided separately from the camera of the imaging unit 202. The positioning imaging unit 203 acquires images used for SLAM (Simultaneous Localization and Mapping: self-localization and environmental map creation) executed by the SLAM unit 212. The positioning imaging unit 203 also acquires images used by the distance map acquisition unit 214 to generate a distance map.

[0017] The display unit 204 displays the composite image generated by the image processing device 110. The display unit 204 has a display panel formed of, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. When the user is wearing the HMD 100, a display panel is disposed in front of each of the user's left and right eyes.

[0018] The convergence angle acquisition unit 205 acquires the convergence angle formed between the gaze directions of the right eye and the left eye. The convergence angle acquisition unit 205 can acquire gaze information of the left and right eyes using, for example, an eye camera that captures images of the user's eyes, and acquire the convergence angle based on the gaze information. The convergence angle acquisition unit 205 uses the acquired convergence angle to acquire a visible distance, which is the distance from the imaging unit 202 that captures the background image to a position corresponding to the user's gaze position. The visible distance is used by the image processing device 110 for blurring the background image and CG.

[0019] The image processing device 110 includes a control unit 211 , a SLAM unit 212 , a CG generation unit 213 , a distance map acquisition unit 214 , a CG blurring processing unit 215 , a background blurring processing unit 216 , and an image synthesis unit 217 .

[0020] The control unit 211 controls each component of the image processing device 110, thereby controlling the entire image processing device 110. The control unit 211 acquires a background image captured by the imaging unit 202 from the HMD control unit 201, and outputs the image to the background blur processing unit 216. The control unit 211 acquires an image captured by the alignment imaging unit 203 from the HMD control unit 201, and outputs the image to the SLAM unit 212 and the distance map acquisition unit 214. The control unit 211 transmits a composite image synthesized by the image synthesis unit 217 to the HMD control unit 201 of the HMD 100.

[0021] The SLAM unit 212 estimates the self-position and creates an environmental map using the image captured by the positioning imaging unit 203. The CG generation unit 213 generates CG to be combined with the background image captured by the imaging unit 202. The CG generation unit 213 determines the position where the CG is to be placed, using the environmental map created by the SLAM unit 212 and the self-position estimated by the SLAM unit 212.

[0022] The distance map acquisition unit 214 acquires a distance map of the background image captured by the imaging unit 202. For example, the distance map acquisition unit 214 can generate the distance map by using stereo images captured by a stereo camera to acquire the distance from the stereo camera for each pixel. The distance map acquisition unit 214 is not limited to using a stereo camera, and may also generate the distance map using a depth sensor.

[0023] The CG blurring processing unit 215 performs blurring processing on the CG image generated by the CG generating unit 213, using the distance map generated by the distance map acquiring unit 214 and the visible distance acquired by the convergence angle acquiring unit 205. The CG blurring processing unit 215 performs blurring processing on the CG image placed at a distance different from the visible distance. The degree of blurring processing on the CG image (blur amount) depends on the visible distance and the position where the CG is placed in the space captured by the imaging unit 202. The degree of blurring may be determined based on the user's instructions. The CG blurring processor 215 may use at least one of an averaging filter and a median filter to perform blurring of the CG image.

[0024] The background blur processing unit 216 performs blurring on the background image captured by the imaging unit 202 using the distance map generated by the distance map acquisition unit 214 and the visible distance acquired by the convergence angle acquisition unit 205. The background blur processing unit 216 performs blurring on an area on the background image that corresponds to a position in real space at a distance different from the visible distance. The degree of blurring on the background image is determined based on the visible distance and the distance in the distance map. Furthermore, the degree of blurring may be changed based on a user instruction. Like the CG blur processing unit 215, the background blur processing unit 216 can perform blurring on the background image using at least one of an averaging filter and a median filter.

[0025] Image synthesis unit 217 synthesizes the background image blurred by background blur processing unit 216 with the CG image blurred by CG blur processing unit 215. Because blurring is performed on the CG image placed at a distance different from the visible distance and on an area on the background image corresponding to a position in real space at a distance different from the visible distance, the synthesized image resembles the state when viewing real space with the human eye.

[0026] When the background image includes a background image for the left eye and a background image for the right eye, blurring is performed on each of the background images for the left eye and the right eye. CG blurring processor 215 performs blurring on the CG images to be superimposed on each of the background images for the left eye and the right eye. Furthermore, background blurring processor 216 performs blurring on each of the background images for the left eye and the right eye.

[0027] CG blurring processor 215 uses the same visible distance and the same distance map to blur a CG image to be superimposed on a background image for the left eye and a CG image to be superimposed on a background image for the right eye. Background blurring processor 216 uses the same visible distance and the same distance map to blur a background image for the left eye and a background image for the right eye.

[0028] The image synthesis unit 217 synthesizes a blurred background image for the left eye with a corresponding CG image to generate an image for the left eye.The image synthesis unit 217 synthesizes a blurred background image for the right eye with a corresponding CG image to generate an image for the right eye.

[0029] A specific example of image blurring processing using the convergence angle will be described with reference to Figures 3(A), 3(B), 4(A), 4(B), 5(A), 5(B), 6(A), and 6(B). In these figures, CG objects (objects) of a television and a sofa are placed in a space captured by the imaging unit 202. The user's gaze is directed toward the television or sofa, and the user's gaze position is detected in the display area of ​​the television or sofa.

[0030] In the following description, the television and sofa are assumed to be CG objects, but the television and sofa may be objects in real space. The background blurring processor 216 can perform blurring on each object in the same way as when the television and sofa are CG objects.

[0031] 3(A) and 3(B) are diagrams explaining the relationship between the convergence angle and the visible distance. FIG. 3(A) shows a state in which the user's line of sight is directed toward the sofa. FIG. 3(B) shows a state in which the user's line of sight is directed toward the television. If the interpupillary distance between the left and right eyes is L, the visible distance D1 when the user is looking at the sofa is longer than the visible distance D2 when the user is watching the television. In this case, the convergence angle θ1 when the user is looking at the sofa is larger than the convergence angle θ2 when the user is watching television.

[0032] Figures 4(A) and 4(B) are diagrams explaining the difference in blurring processing due to differences in convergence angles. Figures 4(A) and 4(B) show examples of blurring processing in the states of Figures 3(A) and 3(B), respectively. In Figure 4(A), the user's gaze is directed toward the sofa, so the CG blurring processing unit 215 performs blurring processing on the television. In Figure 4(B), the user's gaze is directed toward the television, so the CG blurring processing unit 215 performs blurring processing on the sofa.

[0033] Figures 5(A) and 5(B) are diagrams illustrating the difference in distance between objects. In Figures 5(A) and 5(B), the user's gaze is directed toward the sofa. The interpupillary distance between the left and right eyes is L, and the convergence angle is θ. The distance D3 between the sofa and the television in Figure 5(A) is shorter than the distance D4 between the sofa and the television in Figure 5(B).

[0034] Figures 6(A) and 6(B) are diagrams illustrating differences in blurring processing due to differences in the distance between objects. Figures 6(A) and 6(B) show examples of blurring processing for a television in the states of Figures 5(A) and 5(B), respectively. Because the user's gaze is directed toward the sofa, the CG blurring processing unit 215 performs blurring processing on the television. In Figure 6(B), the distance between the sofa and television is greater than in Figure 6(A), so the degree of blurring processing on the television is greater than in Figure 6(A).

[0035] Image blurring processing according to the first embodiment will be described with reference to Fig. 7. In the example of Fig. 7, the visible distance, which is the distance from an imaging device that captures a background image to a position corresponding to the user's line of sight, is acquired using the convergence angle.

[0036] In step S701, the control unit 211 (image acquisition means) acquires a background image captured by the imaging unit 202. The control unit 211 outputs the background image acquired in step S701 to the background blur processing unit 216.

[0037] In step S702, control unit 211 acquires an image captured by alignment imaging unit 203. The image captured by alignment imaging unit 203 is used for generating a distance map and estimating self-position using SLAM, etc. Control unit 211 outputs the image acquired in step S702 to SLAM unit 212 and distance map acquisition unit 214.

[0038] In step S703, the distance map acquisition unit 214 generates a distance map using the image acquired in step S702. The distance map indicates information on the distance from the image capture unit for alignment 203 for each pixel of the background image.

[0039] In step S704, the SLAM unit 212 estimates the self-position and creates an environmental map using the image acquired in step S702. In step S705, the CG generation unit 213 determines the position where the CG is to be placed using the environmental map created in step S704, and generates a CG image.

[0040] In step S706, the convergence angle acquisition unit 205 (gaze acquisition means) acquires gaze information of the left and right eyes. In step S707, the convergence angle acquisition unit 205 acquires the convergence angle from the gaze information of the left and right eyes acquired in step S706.

[0041] In step S708, the convergence angle acquisition unit 205 determines whether the visible distance is stable. For example, the convergence angle acquisition unit 205 can determine that the visible distance is stable when the convergence angle has not changed for a period longer than a predetermined time. If the visible distance is stable, the process The process proceeds to step S709. If the convergence angle changes before the predetermined time has elapsed and the visible distance is not stabilized, the process proceeds to step S710.

[0042] In step S709, the convergence angle acquisition unit 205 (visible distance acquisition means) acquires the visible distance, which is the distance from the camera of the alignment imaging unit 203 to the gaze position where the left and right eyes are focused, using the convergence angle acquired in step S707. Note that the method of acquiring the visible distance is not limited to using the convergence angle, as long as it is possible to acquire the distance to an object present at the user's gaze position. The convergence angle acquisition unit 205 updates the visible distance stored in the memory of the image processing device 110, etc.

[0043] By determining whether the visible distance is stable, the image processing device 110 can start blurring processing after the visible distance has stabilized without the convergence angle changing for a predetermined period of time or longer.

[0044] If it is determined in step S708 that the visible distance is not stable, the convergence angle acquisition unit 205 does not update the visible distance stored in the memory. In steps S710 and S711, the most recently updated visible distance is used.

[0045] When the interpupillary distance between the left and right eyes is L and the convergence angle is θ, the visible distance D can be calculated using the following formula 1.

number

[0046] In step S710, the CG blurring processing unit 215 performs blurring of the CG image using the distance map generated in step S703, the CG image generated in step S705, and the visible distance acquired in step S709.

[0047] The CG blurring processor 215 focuses on points in the distance map that are approximately the same as the visible distance, and blurs the CG image in areas of other distances. In the example of FIG. 3(A), the CG blurring processor 215 focuses on the position of the sofa. In FIG. 4(A), the CG blurring processor 215 does not blur the position of the sofa so that the sofa is displayed clearly, and blurs the position of the television. In contrast, in the example of FIG. 3(B), the CG blurring processor 215 focuses on the position of the television. In FIG. 4(B), the CG blurring processor 215 does not blur the position of the television so that the television is displayed clearly, and blurs the position of the sofa. The amount of blurring (degree of blurring) is preferably set to be the same as the depth of field of the human eye.

[0048] As shown in Figures 5(A), 5(B) and 6(A), 6(B), even when the gaze is directed toward the sofa, which is the same subject, CG blurring processor 215 changes the degree of blurring of the television based on the distance between the sofa and the television. In Figure 5(B), the distance D4 between the objects (the distance between the sofa and the television) is longer than the distance D3 between the objects in Figure 5(A), so CG blurring processor 215 blurs the television more in Figure 6(B) than in Figure 6(A).

[0049] An example of changing the degree of blurring based on the difference in the visible distance will be described with reference to Figs. 8(A), 8(B) and 9(A), 9(B). Figs. 8(A) and 8(B) are diagrams for explaining the difference in the visible distance. In Figs. 8(A) and 8(B), the user's line of sight is The subject is facing the sofa. The interpupillary distance between the left and right eyes is L. The visible distance D5 in FIG. 8(A) is shorter than the visible distance D6 in FIG. 8(B). Therefore, the convergence angle θ3 in FIG. 8(A) is larger than the convergence angle θ4 in FIG. 8(B).

[0050] 9(A) and 9(B) are diagrams illustrating differences in blurring processing due to differences in viewing distance. FIGS. 9(A) and 9(B) show examples of blurring processing for a television in the states of FIGS. 8(A) and 8(B), respectively. In FIG. 8(B), the distance between objects is the same as in FIG. 8(A), but the convergence angle θ4 is smaller than the convergence angle θ3, and viewing distance D6 is longer than viewing distance D5. In this case, CG blurring processor 215 reduces the amount of blurring of the television in FIG. 9(B) compared to FIG. 9(A).

[0051] 7, the background blurring processor 216 performs blurring of the background image using the background image acquired in step S701, the distance map acquired in step S703, and the visible distance acquired in step S709. As in step S710, the background blurring processor 216 changes the degree of blurring based on at least one of the visible distance and the distance between objects. The distance between objects in the background image is the distance between real objects or the distance from a position corresponding to the user's gaze position to the real object.

[0052] In step S712, the image synthesis unit 217 generates a synthesized image by synthesizing the CG image blurred in step S710 with the background image blurred in step S711. In step S713, the control unit 211 transmits the synthesized image generated in step S712 to the HMD control unit 201. The HMD control unit 201 displays the received synthesized image on the display unit 204.

[0053] In the first embodiment, the image processing device 110 not only performs blurring on the CG image but also performs blurring on the background image based on the visible distance and the distance in the distance map. This allows the image processing device 110 to make the composite image displayed on the display unit 204 of the HMD 100 closer to the state when a person actually views real space, thereby reducing the sense of discomfort felt by the user.

[0054] <Embodiment 2> In the first embodiment, the camera (image capturing unit 203 for positioning) that captures the images for generating the distance map and the images used in SLAM is a camera separate from the camera (image capturing unit 202) that captures the background image. In the second embodiment, the image processing device 110 generates a distance map using the image captured by the camera that captures the background image, and performs self-location estimation and environmental map creation by SLAM.

[0055] Regarding the configuration of the information processing system 1 according to the second embodiment, a description common to that of the first embodiment will be omitted. Fig. 10 is a block diagram of the information processing system 1 according to the second embodiment. The same components as those in Fig. 2 are denoted by the same reference numerals, and a description common to those components will be omitted.

[0056] In the second embodiment, the HMD 100 does not have the alignment imaging unit 203. Therefore, the SLAM unit 212 and the distance map acquisition unit 214 use the image captured by the imaging unit 202 instead of the image captured by the alignment imaging unit 203.

[0057] The image blurring process according to the second embodiment will be described with reference to FIG. 11. The difference from the first embodiment is that the SLAM unit 212 estimates the self-position and creates an environmental map using the image captured by the image capturing unit 202, and the distance map acquisition unit 214 generates a distance map using the image captured by the image capturing unit 202. The blurring process is the same as in the first embodiment, so a detailed description will be omitted.

[0058] In step S1101, the control unit 211 acquires a background image captured by the imaging unit 202. The control unit 211 outputs the background image acquired in step S1101 to the background blur processing unit 216, the SLAM unit 212, and the distance map acquisition unit 214.

[0059] In step S1103, distance map acquisition unit 214 generates a distance map using the background image acquired in step S1101. In step S1104, SLAM unit 212 performs self-localization and environmental map creation using the background image acquired in step S1101.

[0060] Except for the above points, the processes in steps S1103 to S1113 are the same as the processes in steps S703 to S713 in FIG.

[0061] In the second embodiment described above, even if the camera that captures the background image and the camera that captures the image used for SLAM alignment are the same, the image processing device 110 can perform blurring on the background image based on the visible distance and the distance in the distance map. Therefore, the image processing device 110 can make the composite image displayed on the display unit 204 of the HMD 100 closer to the state when a person actually views the real space, thereby reducing the sense of discomfort felt by the user.

[0062] <Embodiment 3> In the first embodiment, the image processing device 110 performs blurring on each of the CG image and the background image, and then combines the CG image and the background image to generate a composite image. In the third embodiment, the image processing device 110 can also combine the CG image and the background image and then perform blurring on the composite image.

[0063] The process of performing blurring on a background image and blurring on a CG image (image of an object) and then compositing the CG image onto the background image is also referred to as a first process. Furthermore, the process of compositing a CG image (image of an object) onto a background image and then blurring the background image and blurring the CG image is also referred to as a second process. In the third embodiment, the image processing device 110 can switch between the first process and the second process in response to a user instruction or automatically.

[0064] Regarding the configuration of the information processing system 1 according to the third embodiment, a description common to that of the first embodiment will be omitted. Fig. 12 is a block diagram of the information processing system 1 according to the third embodiment. The same components as those in Fig. 2 are denoted by the same reference numerals, and a description common to those components will be omitted.

[0065] In the third embodiment, the image processing device 110 has a composite image blurring processing unit 1201 in addition to the configuration of the first embodiment shown in Fig. 2. The composite image blurring processing unit 1201 performs blurring processing on the composite image after the image synthesis unit 217 synthesizes the CG image and the background image in the second processing.

[0066] Image blurring processing according to the third embodiment will be described with reference to Fig. 13. The blurring processing by the CG blurring processing unit 215 and the background blurring processing unit 216 is the same as in the first embodiment, and therefore a detailed description will be omitted.

[0067] The processing in steps S1301 to S1309 is the same as the processing in steps S701 to S709 in FIG.

[0068] In step S1310, control unit 211 determines whether blurring of the background image and CG image should be performed first (first processing) or whether synthesis of the background image and CG image should be performed first (second processing).

[0069] Whether or not to perform blurring processing first may be set by the user, or may be set automatically by the control unit 211. For example, the control unit 211 performs blurring processing first to reduce delay when the number of CGs arranged in the space viewed by the user is greater than a predetermined number (first processing), and performs blurring processing later when the number is equal to or less than the predetermined number (second processing). Furthermore, the control unit 211 performs blurring processing first to reduce delay when the image display mode is a low-delay mode (first processing), and performs blurring processing later when the image display mode is a high-quality mode (second processing).

[0070] If it is determined in step S1310 that blurring processing is to be performed first, the control unit 211 proceeds to step S1311. The processing in steps S1311 to S1313 is the same as the processing in steps S710 to S712 in FIG.

[0071] If it is determined in step S1310 that the compositing process of the background image and the CG image is to be performed first, control unit 211 proceeds to step S1314. In step S1314, image compositing unit 217 generates a composite image by compositing the background image captured in step S1301 with the CG image generated in step S1305. In step S1315, composite image blurring processing unit 1201 performs blurring on the composite image created in step S1314 using the distance map and the visible distance. The blurring method is the same as in steps S1311 to S1312, and the degree of blurring is changed based on the visible distance and the distance between objects.

[0072] In step S1316, the control unit 211 transmits the composite image generated in step S1313 or the composite image subjected to blurring processing in step S1315 to the HMD control unit 201. The HMD control unit 201 displays the received composite image on the display unit 204.

[0073] In the above-described third embodiment, the image processing device 110 can switch between performing blurring of the background image and the CG image first (first processing) and performing synthesis of the background image and the CG image first (second processing) depending on the situation. In either processing, the image processing device 110 can make the synthetic image displayed on the display unit 204 of the HMD 100 closer to the state when a person actually views real space, thereby reducing the sense of discomfort felt by the user. Furthermore, by switching between the first processing and the second processing, the image processing device 110 can reflect the user's preferences and reduce the processing load of the image processing device 110.

[0074] Although the above-described embodiments show examples in which the HMD 100 and the image processing device 110 are configured as separate devices, the image processing device 110 may be incorporated into the HMD 100. Fig. 14 is a block diagram of the HMD 100 incorporating the image processing device 110. The HMD 100 includes the components of the image processing device 110 shown in Fig. 2 except for the control unit 211. The components incorporated into the HMD 100 are the same as those in the first embodiment, and therefore will not be described further.

[0075] The various controls described above as being performed by the control unit 211 and the HMD control unit 201 may or may not be performed by a single piece of hardware (for example, a processor or circuit). The entire device may be controlled by multiple pieces of hardware (for example, multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0076] The above processors are processors in a broad sense, and include general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays), CPLDs (Complex Programmable Logic Devices), and so on.

[0077] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0078] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0079] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) image acquisition means for acquiring a background image; a distance map acquisition means for acquiring a distance map of the background image; A gaze acquisition means for acquiring information on a user's gaze; a visible distance acquisition means for acquiring a visible distance, which is a distance from an imaging device that captures the background image to a position corresponding to the user's line of sight, based on information about the user's line of sight; an image processing means for performing blurring processing on the background image based on the visible distance and the distance in the distance map; 1. An image processing device comprising: (Configuration 2) The image processing means performs blurring processing on the image of the object displayed superimposed on the background image based on the visible distance and the position of the object placed in the space captured by the imaging device. 2. The image processing device according to configuration 1, (Configuration 3) The image processing means a first process of blurring the background image and the image of the object, and then combining the image of the object with the background image; or a second process of combining the image of the object with the background image and then blurring the background image and the image of the object; Run 3. The image processing device according to configuration 2. (Configuration 4) The image processing means executes the first process or the second process in response to an instruction from the user. 4. The image processing device according to configuration 3. (Configuration 5) When the number of the objects is greater than a predetermined number, the image processing means performs the first processing. Execute the process 4. The image processing device according to configuration 3. (Configuration 6) The position where the object is placed is determined based on an environmental map created by SLAM (Simultaneous Localization and Mapping) using the image captured by the imaging device that captures the background image, and the self-position estimated by the SLAM. 6. The image processing device according to any one of configurations 2 to 5, wherein: (Configuration 7) the background image includes a first image for a left eye and a second image for a right eye; The image processing means performs blurring processing on the first image and blurring processing on the second image based on the visible distance and the distance in the distance map. 7. The image processing device according to any one of configurations 1 to 6. (Configuration 8) The image processing means performs blurring on the background image using at least one of an averaging filter and a median filter. 8. The image processing device according to any one of configurations 1 to 7, wherein: (Configuration 9) The image processing means changes the degree of blurring of the background image based on an instruction from the user. 9. The image processing device according to any one of configurations 1 to 8. (Configuration 10) The visible distance acquisition means acquires the visible distance using a convergence angle acquired based on information about the user's line of sight. 10. The image processing device according to any one of configurations 1 to 9, wherein: (Configuration 11) The visible distance acquisition means acquires the visible distance when the convergence angle has not changed for a predetermined period of time. 11. The image processing device according to configuration 10. (Configuration 12) The image processing means starts blurring the background image when the convergence angle has not changed for a period longer than the predetermined time. 12. The image processing device according to configuration 11. (Configuration 13) The distance map acquisition means generates the distance map using at least one of a depth sensor and a stereo camera. 13. The image processing device according to any one of configurations 1 to 12. (method) obtaining a background image; obtaining a distance map of the background image; acquiring information about a user's line of sight; acquiring a visible distance, which is a distance from an imaging device that captures the background image to a position corresponding to the user's line of sight, based on information about the user's line of sight; performing a blurring process on the background image based on the visible distance and the distance in the distance map; An image processing method comprising: (program) 14. A program for causing a computer to function as each means of the image processing device according to any one of configurations 1 to 13. [Explanation of symbols]

[0080] 110: Image processing device, 211: Control unit, 214: Distance map acquisition unit, 216: Background blur processing unit

Claims

1. image acquisition means for acquiring a background image; a distance map acquisition means for acquiring a distance map of the background image; A gaze acquisition means for acquiring information on a user's gaze; a visible distance acquisition means for acquiring a visible distance, which is a distance from an imaging device that captures the background image to a position corresponding to the user's line of sight, based on information about the user's line of sight; an image processing means for performing blurring processing on the background image based on the visible distance and the distance in the distance map; 1. An image processing device comprising:

2. The image processing means performs blurring processing on the image of the object displayed superimposed on the background image based on the visible distance and the position of the object placed in the space captured by the imaging device.

2. The image processing device according to claim 1, wherein:

3. The image processing means a first process of performing a blurring process on the background image and a blurring process on the image of the object, and then combining the image of the object with the background image; or a second process of combining the image of the object with the background image and then blurring the background image and the image of the object; Run 3. The image processing device according to claim 2.

4. The image processing means executes the first process or the second process in response to an instruction from the user.

4. The image processing device according to claim 3.

5. The image processing means executes the first process when the number of the objects is greater than a predetermined number.

4. The image processing device according to claim 3.

6. The position where the object is placed is determined based on an environmental map created by SLAM (Simultaneous Localization and Mapping) using an image captured by the imaging device that captures the background image, and the self-position estimated by the SLAM.

3. The image processing device according to claim 2.

7. the background image includes a first image for a left eye and a second image for a right eye; The image processing means performs blurring processing on the first image and blurring processing on the second image based on the visible distance and the distance in the distance map.

2. The image processing device according to claim 1, wherein:

8. The image processing means performs blurring on the background image using at least one of an averaging filter and a median filter.

2. The image processing device according to claim 1, wherein:

9. The image processing means changes the degree of blurring of the background image based on an instruction from the user.

2. The image processing device according to claim 1, wherein:

10. The visible distance acquisition means acquires the visible distance using a convergence angle acquired based on information about the user's line of sight.

2. The image processing device according to claim 1, wherein:

11. The visible distance acquisition means acquires the visible distance when the convergence angle has not changed for a predetermined period of time.

11. The image processing device according to claim 10.

12. The image processing means starts blurring the background image when the convergence angle has not changed for a period longer than the predetermined time.

12. The image processing device according to claim 11.

13. The distance map acquisition means generates the distance map using at least one of a depth sensor and a stereo camera.

2. The image processing device according to claim 1, wherein:

14. obtaining a background image; obtaining a distance map of the background image; acquiring information about a user's line of sight; acquiring a visible distance, which is a distance from an imaging device that captures the background image to a position corresponding to the user's line of sight, based on information about the user's line of sight; performing a blurring process on the background image based on the visible distance and the distance in the distance map; An image processing method comprising:

15. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 13.

Images