Image processing apparatus and method, and program

Abstract

To provide an image processor for generating a shared image limited only to an object whose sharing with a third person is permitted, and for achieving image processing under the consideration of privacy protection.SOLUTION: The image processor is provided with a first generation means 401 for generating a first display image and a second generation means 402 for generating a second display image including a shared image with the first display image, and the second generation means 402 sets a three dimensional shared area in the first display image and generates the shared image by image information included in the shared area.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, method, and program. [Background technology]

[0002] A device used to experience MR (Mixed Reality) content is, for example, a head-mounted display (HMD), which is worn on the user's head and displays images in front of the user's eyes. By using an HMD, it is possible to provide the user with an experience of moving through a space where reality and CG are fused together, for example, by generating and displaying images (moving images, still images, etc.) according to the user's position and posture.

[0003] One possible use case utilizing MR is remote work support, in which a third party remotely issues instructions to an HMD user. In this use case, the HMD user needs to share the MR image they are observing through the HMD with a third party. However, the image to be shared (i.e., the MR image observed through the HMD) may contain areas or objects that the user does not want to share with third parties. To address this situation, Patent Document 1 discloses a technology that separates a face area and a background area, performs privacy masking processing, and outputs these as separate images. This makes it possible to generate an image in which privacy is protected for areas that the user does not want to share with third parties. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-194687 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 separates the facial area from the background on a two-dimensional image and performs image processing without taking into account the three-dimensional shape of the subject. As a result, for example, if there is an object that the subject wants to share behind or in front of a subject that has been set as not wanting to be shared, the object will also be set as an area that the subject does not want to be shared. In other words, Patent Document 1 has the problem that it is difficult to set privacy protection settings that take spatial arrangement into account.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an image processing device that generates shared images limited to only those objects that are permitted to be shared with third parties, and enables image processing that takes privacy protection into consideration. [Means for solving the problem]

[0007] The image processing device of the present disclosure includes a first generating means for generating a first display image, and a second generating means for generating a second display image including a shared image with the first display image. The second generating means sets a shared area having a three-dimensional shape within the first display image, and generates the shared image using image information included in the shared area. [Effects of the Invention]

[0008] According to the present disclosure, an image processing device is realized that generates a shared image limited to only objects that are permitted to be shared with third parties, and enables image processing that takes privacy protection into consideration. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram showing a schematic configuration of an image display system according to a first embodiment. [Figure 2] FIG. 2 is a schematic diagram illustrating an example of the internal configuration of an HMD. [Figure 3] 1 is a block diagram showing an example of a hardware configuration of an image display system according to a first embodiment. [Figure 4]FIG. 2 is a block diagram showing the functional configuration of an image processing device that is a component of the image display system. [Figure 5] 4 is a flowchart showing an example of an image processing method performed by the image display system according to the first embodiment. [Figure 6] FIG. 10 is a schematic diagram illustrating an example of a mask image. [Figure 7] 3 is a schematic diagram showing an example of a real space and a virtual space observed by a user who is sharing an image through an HMD in the first embodiment. FIG. [Figure 8] 6 is a flowchart showing details of the display image generation process in S501 of FIG. 5. [Figure 9] FIG. 4 is a schematic diagram showing an example of a first display image. [Figure 10] FIG. 2 is a schematic diagram showing an example of a first depth image corresponding to a first display image. [Figure 11] 6 is a flowchart showing details of the shared area setting process in S502 of FIG. 5. [Figure 12] FIG. 10 is a schematic diagram showing an initial shape of a sharing area. [Figure 13] FIG. 10 is a schematic diagram showing an example of a GUI for changing the scale and position of the shape of a shared area. [Figure 14] 10A and 10B are schematic diagrams showing examples of the state of the shape of a shared area after the scale and position have been changed by a user; [Figure 15] 6 is a flowchart showing details of the process of determining whether the shared area is inside or outside based on the mask in S505 of FIG. 5. [Figure 16] FIG. 10 is a schematic diagram showing a shared image generated using a mask image. [Figure 17] 7 is a flowchart illustrating details of the process of generating a depth image of the shared area shape in S506 of FIG. 5. [Figure 18] FIG. 10 is a schematic diagram showing a generated second depth image. [Figure 19] 7 is a flowchart illustrating details of the inside / outside determination process of the shared area based on depth in S507 of FIG. 5. [Figure 20]10A and 10B are schematic diagrams illustrating an example of a shared image generated by inside / outside determination processing of a shared area based on depth to which mask processing is applied. [Figure 21] FIG. 10 is a block diagram showing the functional configuration of an image processing device that is a component of an image display system according to a second embodiment. [Figure 22] 10 is a flowchart showing an example of an image processing method performed by an image display system according to a second embodiment. [Figure 23] FIG. 23 is a schematic diagram showing an example of a first display image generated in S2201 of FIG. 22. DETAILED DESCRIPTION OF THE INVENTION

[0010] -Basic configuration of image processing device in various embodiments- Before specifically disclosing the embodiments, the basic configuration of the image processing device in each embodiment will be described.

[0011] The image processing device of the present disclosure includes a first generating means for generating a first display image and a second generating means for generating a second display image including a shared image of the first display image. The first display image is an image observed on a first display device, and the second display image is an image observed on a second display device different from the first display device. The second generating means sets a three-dimensional shared area within the first display image and generates the shared image using image information included in the shared area. The first display image may include objects that are not desired to be shared. While it is desirable to obtain a shared image excluding these objects, it is difficult to selectively display only the objects desired to be shared using image processing that captures the first display image as a two-dimensional image. In the present disclosure, a three-dimensional shared area that is permitted to be shared with a third party is set, and a shared image limited to only the objects included in this shared area is generated. This makes it possible to generate a second display image including a shared image of only the objects desired to be shared, excluding the objects not desired to be shared from the first display image, thereby enabling desired image processing that takes privacy protection into consideration.

[0012] As a specific example of creating a shared image using image information including only the object desired to be shared, the second generation means generates an inside / outside determination image for determining whether the shared area in the first display image is inside or outside, and then generates the shared image by performing the inside / outside determination using the inside / outside determination image. Examples of the inside / outside determination image include a mask image or a depth image of the shape of the shared area. When a mask image is used, the inside / outside determination process excludes from the shared image objects outside the masked shared area in the first display image, while when a depth image is used, the shared image is created by excluding from the shared image objects outside the shared area within a predetermined depth range. In this case, by using the results of the mask-based inside / outside determination process in the depth-based inside / outside determination process, it is possible to reliably generate a second display image including a shared image containing only image information of the object desired to be shared in the first display image.

[0013] Furthermore, when viewing the first display image, the user who is sharing the image may wish to confirm the shared image shared by a third party. To meet such needs, the image processing device of the present disclosure further includes a presentation means for presenting the shared image together with the first display image. The presentation means, for example, displays the shared image superimposed on the first display image. This allows the user who is sharing the image to view the shared image together with the first display image, which is convenient for the user who wishes to confirm the shared image viewed by three parties.

[0014] -Specific Description of Various Embodiments- Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims. Although multiple features are described in the various embodiments, not all of these multiple features are necessarily essential, and multiple features may be combined arbitrarily. Furthermore, in the drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0015] [First embodiment] A first embodiment of the present disclosure will be described below.

[0016] <Image display system configuration> FIG. 1 is a schematic diagram showing a schematic configuration of an image display system according to this embodiment. The image display system according to this embodiment includes an HMD 101 which is an image processing device, and an image processing device 102.

[0017] The HMD 101 and the image processing device 102 are electrically connected via a predetermined communication path, and are connected so as to be able to mutually transmit and receive various data such as image data and various control signals. In the example shown in FIG. 1 , the HMD 101 and the image processing device 102 are connected via a cable compliant with standards such as HDMI (registered trademark) (High-Definition Multimedia Interface) and USB (Universal Serial Bus). The type of communication path connecting the HMD 101 and the image processing device 102 is not particularly limited. As a specific example, a communication path between the HMD 101 and the image processing device 102 may be established via wireless communication such as Bluetooth (registered trademark). The image processing device 102 used by user A is connected to the image processing device 104 used by user B via a network. With this system, an image observed by user A through the HMD 101 is transmitted to the HMD 103 of user B, allowing user B to observe the image observed by user A through the HMD 101 through the HMD 103.

[0018] 1 is merely an example and does not limit the configuration of the image display system according to this embodiment. As a specific example, an input device such as a controller or keyboard (not shown) for receiving input from a user may be connected to the image processing device 102 via a predetermined communication path.

[0019] FIG. 2 is a schematic diagram showing an example of the internal configuration of the HMD 101. The HDM 101 has a plurality of image capturing devices 201 (for example, RGB cameras) for imaging the real space. The HDM 101 also has an IMU (Inertial Measurement Unit) such as a gyro sensor or acceleration sensor (not shown), an image capturing device, etc., in order to realize its own position tracking.

[0020] The HMD 101 has a configuration for acquiring depth information indicating the distance to an object located in the external environment. For example, in the example shown in Fig. 2, the HMD 101 has a distance sensor 202 such as a LiDAR as a configuration for acquiring depth information. The HMD 101 also has displays 203 corresponding to the left and right eyes, which are realized by display panels such as liquid crystal panels or organic EL panels, as a configuration for displaying images. Eyepieces 204 are disposed between the displays 203 and the left and right eyes of the user wearing the HMD 101. With this configuration, the user wearing the HMD 101 can observe an enlarged virtual image of the display image displayed on the displays 203 through the eyepieces 204.

[0021] The HMD 101 is worn on the head of a user (not shown), and allows the user's left eye to observe a left-eye display image (an enlarged virtual image), and the user's right eye to observe a right-eye display image (an enlarged virtual image). The image processing device 102 generates the left-eye display image and the right-eye display image, and displays these images on the display 203 of the HMD 101. At this time, the image processing device 102 may provide a parallax between the left-eye display image and the right-eye display image according to the distance between the left and right eyes of the user wearing the HMD 101 (for example, the distance between the eyepieces 204 corresponding to the left and right eyes). By applying such control, it is possible to provide the user wearing the HMD 101 with an image perception that has a sense of depth.

[0022] In this embodiment, the description focuses on a system configuration in which the image processing device 102 is realized as a device independent of the HMD 101, but this does not necessarily limit the configuration of the image display system according to this embodiment. As a specific example, the image display system according to this embodiment may be realized by an all-in-one HMD system in which the HMD 101 includes a configuration equivalent to the image processing device 102.

[0023] FIG. 3 is a block diagram showing an example of the hardware configuration of the image display system according to this embodiment. The image processing device 102 has a CPU (Central Processing Unit) 301, a RAM (Random Access Memory) 302, and a ROM (Read Only Memory) 303. The image processing device 102 also includes an HDD (Hard Disk Drive) 304, a general-purpose I / F (Interface) 305, a video output I / F 306, and a network I / F 307. The above-mentioned series of components of the image processing device 102 are connected via a main bus 300 so as to be able to send and receive information to and from each other.

[0024] The CPU 301 is a processor that performs overall control of each unit within the image processing device 102. The RAM 302 functions as the main memory, work area, etc. of the CPU 301. The ROM 303 stores a group of programs executed by the CPU 301. The HDD 304 is a storage area that stores applications executed by the CPU 301, data used in image processing, etc. Note that this storage area is not limited to an HDD, and various storage devices may be applied. As a specific example, an auxiliary storage device such as an SSD (Solid State Drive) may be applied instead of or in addition to the HDD 304.

[0025] The general-purpose I / F 305 is a serial bus interface conforming to standards such as USB and IEEE1394, and is connected to, for example, an IMU or distance sensor provided in the HMD 101. This enables the image processing device 102 to acquire posture information and a depth image of a target object (an image onto which depth information corresponding to the measurement result of the distance to the target object is mapped for each pixel) from the HMD 101. The general-purpose I / F 305 is also used to acquire an image corresponding to the image capture result by the image capture device 201 of the HMD 101.

[0026] The video output I / F 306 is an interface such as HDMI or display port, and is used to transmit to the HMD 101 a display image to be displayed on the display 203 of the HMD 101 .

[0027] The network I / F 307 is an interface for connecting the image processing device 102 to a predetermined network. The configuration of the network I / F 307 may be changed as appropriate depending on the type of network to be connected and the communication method to be applied.

[0028] 4 is a block diagram showing the functional configuration of the image processing device 102, which is a component of the image display system. The following description will focus on the configuration of the image processing device 102 in particular, of the image display system, with reference to FIG.

[0029] The image processing device 102 includes a first generation unit 401 and a second generation unit 402 . The second generation unit 402 includes a setting unit 411 , a determination image generation unit 412 , and a display image generation unit 413 . The determination image generation unit 412 includes a mask image generation unit 4121 , a first depth image generation unit 4122 , and a second depth image generation unit 4123 .

[0030] The first generation unit 401 generates a first display image that is observed by the user who is sharing the image through, for example, the HMD 101. The first display image is generated by, for example, combining a real image and a CG image. The second generation unit 402 generates a second display image including an image to be shared with the first display image. The second display image is an image that is observed by a user (third party) with whom the image is shared in FIG. 1 through, for example, the HMD 103, and is provided from the image processing device 102 to the image processing device 104.

[0031] The setting unit 411 sets a sharing area, which is an area that a user who is to share an image wants to share, as three-dimensional shape information. The determination image generating unit 412 generates an image required for performing the inside / outside determination process for the shared area on the first display image. Second generation unit 413 performs a process of determining whether the first display image is inside or outside the shared area, and generates a shared image.

[0032] The mask image generation unit 4121 projects the three-dimensional shape indicating the shared area onto a two-dimensional plane using the same path as the CG rendering used when generating the display image, and generates a mask image to be used in the inside / outside determination process. The first depth image generation unit 4122 generates a first depth image that indicates depth information from the viewpoint position with respect to the first display image. The second depth image generation unit 4123 generates a second depth image that indicates depth information for the three-dimensional shape that indicates the shared area.

[0033] <Main flow> 5 is a flowchart showing an example of an image processing method by the image display system according to this embodiment. With reference to FIG. 5, an example of processing by the image display system according to this embodiment will be described, focusing particularly on processing executed by the image processing device 102.

[0034] 5 is realized in the image processing apparatus 102 by loading a program stored in the ROM 303 or HDD 304 into the RAM 302 and executing it by the CPU 301. As a result, the CPU 301 fulfills the roles of the components shown in FIG.

[0035] In S501, the first generation unit 401 combines a real-life image and a CG image to generate a first display image that the user sharing the image observes through the HMD 101. During the processing of S501, the first depth image generation unit 4122 generates a first depth image that indicates depth information from the viewpoint position for the real-life object and the CG object that constitute the first display image. Details of the processing will be described later.

[0036] In S502, the setting unit 402 sets a shared area indicating an area that the user who will be sharing the image wishes to share as three-dimensional shape information. Details of this process will be described later. Note that the setting method may involve reading information indicating the shared area that has been created and saved in advance.

[0037] In S503, display image generation unit 413 performs initialization processing for the shared image. Here, the shared image is initialized by copying the first display image generated in S501. That is, at this point, the first display image and the shared image have the same size and pixel values.

[0038] In S504, the mask image generation unit 4121 projects a three-dimensional shape indicating the shared area onto a two-dimensional plane using the same path as the CG rendering in the display image generation process of S501, and generates a mask image to be used in the inside / outside determination process. The mask image is generated with the same size and resolution as the display image. Figure 6 shows an example of a mask image. Here, the mask image is generated so that the pixel value of an area 601 corresponding to the shared area in the mask image is 0, and the pixel values ​​of other areas are non-zero.

[0039] In S505, the display image generation unit 413 performs inside / outside determination processing of the shared area by applying the mask image generated in S504 to the first display image. Details of this processing will be described later. In S506, the second depth image generation unit 4123 generates a second depth image indicating depth information for the three-dimensional shape indicating the shared area. Details of this processing will be described later.

[0040] In S507, the display image generation unit 413 performs inside / outside determination processing of the sharing area based on the first depth image indicating depth information of the display image generated in the processing of S501 and the second depth image indicating depth information for the three-dimensional shape indicating the sharing area generated in S506. Details of the processing will be described later.

[0041] <Display image generation process> FIG. 7 is a schematic diagram showing an example of the real space and virtual space observed by a user who is sharing an image through the HMD 101 in this embodiment. In the real space, there exists a real object 701 that does not want to be shared, and a real object 702 that wants to be shared. In the virtual space, there exists a CG object 703 that wants to be shared. Furthermore, 704 indicates the position of the user in the real space or virtual space, and in the real space, the HMD 101 is located at 704, and in the virtual space, the virtual camera is located at 704.

[0042] The display image generation process of S501 will be described in detail with reference to FIG. In S801, the first generation unit 401 acquires the position and orientation information of the HMD 101 using information acquired from an IMU or a known self-position estimation technique such as SLAM.

[0043] In S802, the first generation unit 401 acquires the display angle of view of the HMD 101 from the device information. In S803, the first generation unit 401 sets the position, orientation, and angle of view of the virtual camera used for CG rendering. The position and orientation of the virtual camera are set to match the position and orientation of the HMD 101 acquired in S801. At this time, the same world coordinate system is used to represent the real space indicating the position of the HMD 101 acquired in S801 and the virtual space indicating the position of the virtual camera. The angle of view of the virtual camera is set to match the display angle of view of the HMD 101 acquired in S802.

[0044] In S804, the first generation unit 401 uses the virtual camera set in S803 to process CG to be superimposed on real space using a known CG rendering method such as rasterization to generate a rendered image. Two rendered images corresponding to the right and left eyes are generated. To allow the CG to be perceived three-dimensionally, it is necessary to show images with parallax to each of the right and left eyes. Therefore, the position of the virtual camera set in step S803 is shifted by a distance corresponding to the distance between the two eyes, and rendering processes corresponding to the right and left eyes are performed, respectively, to generate two rendered images, one for the left and one for the right eye.

[0045] In S805, the first generation unit 401 generates a depth image corresponding to the rendering image based on the distance information for each CG object obtained during the rendering process in S804. In S806, the first generation unit 401 acquires two real-life images corresponding to the left and right eyes from the imaging device 201 provided in the HMD 101. After correcting aberrations such as lens distortion of the imaging device 201, the real-life images are subjected to processing such as cropping to match the angle of view of the display acquired in S802.

[0046] In S807, the first generating unit 401 generates a depth image corresponding to the real image. This may be generated by performing a known stereo depth estimation process using the captured image acquired by the image capturing device 201, or may be generated based on distance information obtained from a distance sensor 202 such as a LiDAR.

[0047] In S808, the first generation unit 401 performs a synthesis process of the rendering image generated in S804 and the real-life image obtained in S806 to generate a first display image. In the synthesis process, a depth image corresponding to the rendering image obtained in S805 and a depth image corresponding to the real-life image obtained in S807 are used to perform processing taking occlusion into consideration. FIG. 9 shows an example of the first display image. A real object 701 that does not want to be shared, a real object 703 that wants to be shared, and a CG object 703 that wants to be shared are all included in the first display image 900.

[0048] Furthermore, a depth image corresponding to the rendered image is synthesized with a depth image corresponding to the actual image to generate a first depth image corresponding to the first display image. FIG. 10 shows an example of the first depth image corresponding to the first display image. The first depth image is one-channel image data, for example, image data with 8-bit gradation. Here, it is shown that the distance increases as the brightness of the pixel values ​​on the image decreases (the pixel values ​​become smaller).

[0049] <Shared area setting process> FIG. 11 is a flowchart showing details of the shared area setting process in S502. In S1101, the setting unit 411 displays the initial shape of the sharing area shape, which is a three-dimensional shape indicating the sharing area, at a predetermined position, such as the center of the screen.

[0050] FIG. 12 is a schematic diagram showing the initial shape of the sharing area. FIG. 12(a) shows an example of the relationship between real objects and CG objects and a shared area shape 1201 in three-dimensional space. Here, an example is shown in which a wireframe showing the shared area shape is displayed. A coordinate system 1203 indicates the same coordinate system as that used in the display image generation process. The shared area shape 1201 is set in virtual space as a CG object. The setting unit 411 renders the shared area shape 1201 using the same rendering path as in the display image generation process, and can display the shared area shape on the screen of the HMD 101 by combining it with the first display image. FIG. 12(b) shows the shared area shape 1202 combined on the first display image.

[0051] In S1102 and S1103, the setting unit 411 changes the size (scale) and position of the shared area shape initially set in S1101. FIG. 13 is a schematic diagram showing an example of a GUI for changing the scale and position of the shared area shape. The user changes the scale and position of the shared area shape by operating a position adjustment UI 1301 and a scale adjustment UI 1302 via an input device such as a controller. Regarding the position, by selecting each of the X, Y, and Z axes of the position adjustment UI 1301 with the controller and stretching or shrinking in the direction along each axis, the shared area shape can be moved along that axis. Regarding the scale, by selecting each of the X, Y, and Z axes of the scale adjustment UI 1302 with the controller and stretching or shrinking in the direction along each axis, the scale of the shared area shape can be changed along that axis.

[0052] FIG. 14 is a schematic diagram showing an example of the state of the sharing area shape 1201 after the scale and position have been changed by the user. FIG. 14(a) shows an example of the relationship between the real object and the CG object in three-dimensional space and the sharing area shape 1201 after the scale and position have been changed. FIG. 14(b) shows the sharing area shape 1201 after the scale and position have been changed and the object has been composited onto the first display image. As shown here, a primitive shape, which is a three-dimensional shape indicating the sharing area, is set so that the real object and the CG object desired to be shared are included in the sharing area. As described above, in this embodiment, the user who will be sharing the image can easily set a three-dimensional sharing area at a desired position and size within the first display image.

[0053] In S1104, the setting unit 411 stores the three-dimensional shape information of the sharing area shape adjusted and changed by the user in S1102 in the RAM 302 as sharing area information.

[0054] In this embodiment, a rectangular parallelepiped is used as the shape of the shared area, but this is not limited to this. For example, a sphere or various other three-dimensional shapes can be used as the shape of the shared area. Furthermore, the method of changing the position and scale of the shared area shape is not limited to the method described here. For example, the coordinates of each vertex of the shared area shape may be changed individually.

[0055] <Mask-based inside / outside shared area determination process> FIG. 15 is a flowchart showing details of the process of determining whether a shared area is inside or outside based on the mask in S505.

[0056] In S1501, the display image generation unit 413 initializes the pixel indexes of the mask image and the shared image.

[0057] In S1502, the display image generation unit 413 determines whether processing has been completed for all pixels of the shared image. If processing has been completed, the inside / outside determination processing of the shared area based on the mask is terminated. If processing has not been completed, the process proceeds to S1503 and continues.

[0058] In S1503, the display image generating unit 413 acquires the pixel value of the pixel corresponding to the currently set pixel index for the mask image generated in S504.

[0059] In S1504, the display image generation unit 413 determines whether the pixel value of the mask image acquired in S1503 is equal to 0. If it is equal to 0, the process proceeds to S1506, and if it is not equal to 0, the process proceeds to S1505.

[0060] In S1505, the display image generating unit 413 rewrites the pixel value of the pixel corresponding to the currently set pixel index in the shared image to (R, G, B)=(0, 0, 0) indicating black.

[0061] In S1506, the display image generating unit 413 increments the pixel index to a value indicating a pixel that has not yet been processed, and the process returns to S1502.

[0062] FIG. 16 is a schematic diagram showing a shared image generated using a mask image. FIG. 16(a) shows the relationship between the first display image and an area 1301 in the mask image that corresponds to the shared area. By performing mask processing on the first display image to generate a shared image, it can be seen that some of the real objects 701 that are not desired to be shared are excluded. This processing is particularly effective in excluding objects that exist spatially above, below, left, and right of the shared area. FIG. 16(b) shows an example of a shared image generated in this way. In this example, it can be seen that the pixel values ​​of the areas that are not desired to be shared are black.

[0063] <Depth image generation process for shared area shape> Fig. 17 is a flowchart illustrating the details of the shared area shape depth image generation process of S506. Fig. 18 is a schematic diagram illustrating the generated second depth image. In this embodiment, the second depth image is composed of a front-side depth image and a back-side depth image.

[0064] In S1701, the second depth image generation unit 4123 generates a surface depth image of the shared area shape. Specifically, the second depth image generation unit 4123 generates surface-side depth images of polygons that make up the shared area shape and whose faces face toward the virtual camera by rendering only the surface polygons of the shared area shape using the same rendering path as in the display image generation process.

[0065] 18(a) shows an example of a surface-side depth image 1801 for the surface polygon of the shared area shape. The surface-side depth image 1801 is one-channel image data, for example, image data with 8-bit gradation. Here, it shows that the distance increases as the brightness of the pixel values ​​on the image decreases (the pixel values ​​become smaller).

[0066] In S1702, the second depth image generation unit 4123 generates a back-side depth image of the shared area shape. Specifically, the second depth image generation unit 4123 generates back-side depth images of polygons that make up the shared area shape, whose faces face back relative to the virtual camera, by rendering only the back-side polygons of the shared area shape using the same rendering path as in the display image generation process. At this time, back-face culling is not performed because processing is performed on the back-side polygons.

[0067] Figure 18(b) shows an example of a back-side depth image 1802 for a back-side polygon of the shared area shape. As in Figure 18(a), the back-side depth image 1802 is one-channel image data, for example, image data with 8-bit gradation. Here, it is shown that the distance increases as the brightness of the pixel values ​​on the image decreases (the pixel values ​​become smaller).

[0068] <Depth-based inside / outside determination process for shared area> 19 is a flowchart illustrating the details of the inside / outside determination process of the sharing area based on the depth in S507. Here, a case will be described in which the result of the depth image generation process of the sharing area shape is applied to the inside / outside determination process of the sharing area based on the depth.

[0069] In S1901, the display image generation unit 413 initializes the value i, which indicates the index of a pixel in the depth image and the shared image, to 0.

[0070] In S1902, the display image generation unit 413 determines whether processing has been completed for all pixels of the shared image. If processing has been completed, the inside / outside determination processing of the shared area based on depth is terminated. If processing has not been completed, the process proceeds to S1903 and continues.

[0071] In S1903, the display image generating unit 413 obtains the pixel value M_i of the pixel for the mask image obtained in S1503.

[0072] In S1904, the display image generation unit 413 determines whether the pixel value M_i of the mask image acquired in S1903 is equal to 0. Here, if it is equal to 0, the process proceeds to S1905; if it is not equal to 0, the process proceeds to S1909.

[0073] In S1905, the display image generation unit 413 acquires the pixel value Dd_i of the first depth image (FIG. 10) corresponding to the first display image generated in S808. In S1906, the display image generation unit 413 acquires the pixel value Df_i of the second depth image for the surface polygon of the shared area shape generated in S1701. In S1907, the display image generation unit 413 acquires the pixel value Db_i of the second depth image for the back surface polygon of the shared area shape generated in S1702.

[0074] In S1908, the display image generation unit 413 determines whether the object shown in the processing target pixel of the shared image is inside or outside the shared area. Specifically, when Dd_i < Df_i and Dd_i > Db_i (when Db_i < Dd_i < Df_i is satisfied), it is determined that the object is located inside the shared area, and the process proceeds to S1910. When the above conditional expression is not satisfied, it is determined that the object is located outside the shared area, and the process proceeds to S1909.

[0075] In S1909, the display image generation unit 413 rewrites the pixel value of the pixel corresponding to the index of the currently set pixel in the shared image to (R, G, B) = (0, 0, 0) indicating black. In S1910, the display image generation unit 413 increments the pixel index i.

[0076] FIG. 20 is a schematic diagram showing an example of a shared image generated by the inside / outside determination process of the shared area based on the depth to which the mask process is applied. Comparing Fig. 20 with the shared image after masking shown in Fig. 16(b), it can be seen that objects that are determined to be located three-dimensionally outside the shape of the shared area based on depth have been further changed to black. Depth processing has the effect of excluding objects that are not desired to be shared, particularly those located in the depth direction, from the shared image.

[0077] As described above, according to this embodiment, an image display system is realized that includes an image processing device 102 that generates shared images limited to only those objects that are permitted to be shared with third parties, and enables image processing that takes privacy protection into consideration.

[0078] In this embodiment, the pixel values ​​of the area not included in the shared area shape are colored black, but the color may be other than black. Also, instead of the coloring process, a known blurring process such as a Gaussian filter may be applied to make it impossible to identify the object.

[0079] Furthermore, the method for setting the shared area is not limited to the method of deforming the initial shape described in this embodiment, and other methods may be used. For example, the user may set two pieces of information: shape information for the area indicating the bottom surface of the shared area and height information for the shared area, and then a three-dimensional shared area shape may be set from these pieces. Alternatively, the three-dimensional shape may be automatically set so as to include all specific objects specified by the user.

[0080] Furthermore, it is not necessary to generate the shared area information each time, but it is also possible to read and use information that has been set and stored in advance.

[0081] In this embodiment, an example of MR in which live action and CG are combined and displayed has been described, but similar effects can be obtained by performing similar processing in VR (Virtual Reality) using only CG. In the display image generation processing shown in the flowchart of Fig. 8, the processing for live action can be skipped, and in S808, a CG rendering image can be used as the display image without combining it with the live action. In this case, the world coordinate system set in the virtual space can be used.

[0082] [Second embodiment] A second embodiment of the present disclosure will be described below. In this embodiment, a configuration for allowing a user who shares an image to recognize a shared image generated by the method described in the first embodiment will be described.

[0083] Fig. 21 is a block diagram showing the functional configuration of an image processing device 102, which is a component of an image display system according to this embodiment. With reference to Fig. 21, the following description will focus on the configuration of the image display system, particularly the image processing device 102. Note that the functional configuration of the image display system according to this embodiment is configured by adding a presentation unit 2101 to the functional configuration of the first embodiment shown in Fig. 4. Therefore, parts common to the functional configuration of the first embodiment are denoted by the same reference numerals, and description thereof will be omitted.

[0084] The presentation unit 2101 superimposes the shared image generated by the second generation unit 402 on the first display image generated by the first generation unit 401. Alternatively, the presentation unit 2101 superimposes the shared area set by the setting unit 402 on the first display image. This allows the user who is sharing the image to recognize what image is actually being shared with a third party.

[0085] 22 is a flowchart showing an example of an image processing method by the image display system according to this embodiment. With reference to FIG. 22, an example of processing by the image display system according to this embodiment will be described, focusing particularly on processing executed by the image processing device 102.

[0086] The series of processes shown in Fig. 22 is realized in the image processing device 102 by loading a program stored in the ROM 303 or HDD 304 into the RAM 302 and executing it by the CPU 301. As a result, the CPU 301 fulfills the roles of the components shown in Fig. 21. Note that steps common to the main flow of the first embodiment are assigned the same reference numerals as in Fig. 5, and description thereof will be omitted. Only S2201, which is different from the first embodiment, will be described here.

[0087] In S2201, the presentation unit 2101 performs a process of superimposing the shared image generated in S507 on the first display image generated in S501, thereby updating the display image. For example, the shared image may be resized and superimposed on a partial area of ​​the display image by performing a synthesis process in the form of picture-in-picture. FIG. 23 is a schematic diagram showing an example of the first display image generated in S2201. Here, the shared image 2301 is shown superimposed in the upper left area of ​​the first display image 700 in the form of picture-in-picture.

[0088] The above processing allows the user sharing the image to recognize the shared image being shared with a third party, which is convenient for the user sharing the image who wants to confirm the shared image being observed by a third party.

[0089] In this embodiment, we have described a process for allowing the user sharing the image to recognize the shared image, but it is also possible to provide a mode for switching this process on and off, and control whether or not to superimpose the shared image on the display image depending on this state.

[0090] In addition, although the present embodiment has described the process of superimposing the shared image itself on the display image, the same effect can be obtained by superimposing the shape of the shared area on the display image. This can be achieved by displaying a wireframe of the shape of the shared area on the display image using a flow similar to that of the shared area setting process, as shown in Figure 14(b).

[0091] [Other embodiments] While various embodiments of the present disclosure have been described above in detail, the present disclosure can be embodied, for example, as a system, an apparatus, a method, a program, or a recording medium (storage medium). Specifically, the present disclosure may be applied to a system consisting of multiple devices (e.g., a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus consisting of a single device. The programs for realizing the functions of the components shown in FIGS. 4 and 21 above, and the programs for causing a computer to execute the steps shown in FIGS. 5, 8, 11, 15, 17, 19, and 22 constitute the present disclosure.

[0092] Needless to say, the object of the present disclosure can be achieved by the following: A recording medium (or storage medium) on which software program code (computer program) that realizes the functions of the above-described embodiments is recorded is supplied to a system or device. This storage medium is, of course, a computer-readable storage medium. A computer (or a CPU or MPU) of the system or device then reads and executes the program code stored on the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiments, and the computer-readable recording medium on which the program code is recorded constitutes the present disclosure.

[0093] The disclosure of the various embodiments includes the following configurations and methods. (Configuration 1) a first generating means for generating a first display image; a second generating means for generating a second display image including a shared image with the first display image; It is equipped with The second generating means setting a three-dimensional shared area within the first display image, and generating the shared image using image information included in the shared area; Image processing device. (Configuration 2) the first display image is an image observed on a first display device, the second display image is an image observed on a second display device different from the first display device; 2. The image processing device according to claim 1. (Configuration 3) The second generating means a setting unit that sets the shared area; 3. The image processing device according to configuration 1 or 2. (Configuration 4) The setting unit setting an initial shape of the shared area within the first display image; modifying the position and size of the initial shape to create the shared area; 4. The image processing device according to configuration 3. (Configuration 5) The second generating means a determination image generation unit that generates an inside / outside determination image for determining inside / outside of the shared area in the first display image; 5. The image processing device according to configuration 3 or 4. (Configuration 6) The second generating means a display image generation unit that performs the inside / outside determination using the inside / outside determination image and generates the shared image; 6. The image processing device according to configuration 5. (Configuration 7) The determination image generation unit a mask image generation unit that generates a mask image of the two-dimensionally projected shared area; The display image generation unit performing the inside / outside determination using the mask image and generating the second display image; 7. The image processing device according to configuration 6. (Configuration 8) The determination image generation unit a first depth image generating unit that generates a first depth image indicating depth information from a viewpoint position with respect to the first display image; a second depth image generating unit that generates a second depth image indicating depth information from a viewpoint position with respect to the shared area; It has The display image generation unit performing the inside / outside determination by comparing the first depth image with the second depth image, and generating the second display image; 8. The image processing device according to configuration 6 or 7. (Configuration 9) The second depth image generation unit generating, as the second depth image, an image obtained by rendering only the front surface polygons of the shape of the shared area, and an image obtained by rendering only the back surface polygons of the shape of the shared area; 9. The image processing device according to configuration 8. (Configuration 10) a presentation means for presenting the shared image together with the first display image; Further comprising: 10. The image processing device according to any one of configurations 1 to 9. (Configuration 11) The presentation means the shared image is displayed superimposed on the first display image. 11. The image processing device according to configuration 10. (Method 1) a first step of generating a first display image; a second step of generating a second display image including a shared image with the first display image; It is equipped with The second step is setting a three-dimensional shared area within the first display image, and generating the shared image using image information included in the shared area; Image processing methods. (Configuration 12) A program for causing a computer to execute each step of method 1. [Explanation of symbols]

[0094] 101 HMD 102 Image processing device 401 1st generation part 402 Second generation part 411 Settings 412 Judgment image generation unit 413 Display image generation unit 4121 Mask Image Generation Unit 4122 First depth image generation unit 4123 Second depth image generation unit

Claims

1. a first generating means for generating a first display image; a second generating means for generating a second display image including a shared image with the first display image; It is equipped with The second generating means a shared area having a three-dimensional shape is set within the first display image, and the shared image is generated using image information included in the shared area; Image processing device.

2. the first display image is an image observed on a first display device, the second display image is an image observed on a second display device different from the first display device; The image processing device according to claim 1 .

3. The second generating means a setting unit that sets the shared area; The image processing device according to claim 1 .

4. The setting unit setting an initial shape of the shared area within the first display image; modifying the position and size of the initial shape to create the shared area; The image processing device according to claim 3 .

5. The second generating means a determination image generating unit that generates an inside / outside determination image for determining inside / outside of the shared area in the first display image; The image processing device according to claim 3 .

6. The second generating means a display image generation unit that performs the inside / outside determination using the inside / outside determination image and generates the shared image; The image processing device according to claim 5 .

7. The determination image generation unit a mask image generation unit that generates a mask image of the two-dimensionally projected shared area; The display image generation unit performing the inside / outside determination using the mask image and generating the second display image; The image processing device according to claim 6 .

8. The determination image generation unit a first depth image generating unit that generates a first depth image indicating depth information from a viewpoint position with respect to the first display image; a second depth image generating unit that generates a second depth image indicating depth information from a viewpoint position with respect to the shared area; It has The display image generation unit performing the inside / outside determination by comparing the first depth image with the second depth image, and generating the second display image; The image processing device according to claim 6 .

9. The second depth image generation unit generating, as the second depth image, an image obtained by rendering only the front surface polygons of the shape of the shared area, and an image obtained by rendering only the back surface polygons of the shape of the shared area; The image processing device according to claim 8 .

10. a presentation means for presenting the shared image together with the first display image; Further comprising: The image processing device according to claim 1 .

11. The presentation means the shared image is displayed superimposed on the first display image; The image processing device according to claim 10.

12. a first step of generating a first display image; a second step of generating a second display image that includes a shared image with the first display image; It is equipped with The second step is a shared area having a three-dimensional shape is set within the first display image, and the shared image is generated using image information included in the shared area; Image processing methods.

13. A program for causing a computer to execute the steps recited in claim 12.

Images