Information processing apparatus, information processing method, and program

Abstract

To generate a more appropriate image including a translucent virtual object while reducing the time delay when displaying a captured real image.SOLUTION: An information processing apparatus includes: first acquisition means for acquiring a first layer including a first image in which a translucent object is drawn and first depth information that corresponds to the first image; second acquisition means for acquiring a second layer including a second image in which an opaque object is drawn and second depth information that corresponds to the second image; third acquisition means for acquiring a third layer including a third image in which a real object, which is placed in a real space, is drawn and third depth information that corresponds to the third image; and composition means for drawing a composite image in which the first image, the second image, and the third image are combined on a basis of the first depth information, the second depth information, and the third depth information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, head-mounted displays (hereinafter referred to as "HMDs") have become popular. Users can easily experience mixed reality (hereinafter referred to as "MR") using HMDs.

[0003] Patent Document 1 discloses a technology that provides a natural image experience by taking into account the anteroposterior relationship between real space and virtual space. In Patent Document 1, an information processing device acquires in advance an image captured of real space (hereinafter referred to as a "real image") and depth information of the real space. The information processing device determines the anteroposterior relationship between the real image and a semi-transparent virtual object using a rendering engine, and then generates an image by rendering processing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-170232 Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology of Patent Document 1, an information processing device creates an object from a real-life image, and then determines the anteroposterior relationship between the object and a translucent virtual object using a rendering engine. This results in a long display delay until the real-life image is displayed on the HMD. Therefore, a method is sometimes adopted in which an image of only the virtual object is generated using a rendering engine and then combined with the latest real-life image to shorten the display delay of the real-life image. However, this method does not properly represent the anteroposterior relationship between the real-life image and the translucent virtual object.

[0006] The present invention aims to generate more appropriate images having semi-transparent virtual objects while reducing the display delay time of real-life images. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. a first acquisition means for acquiring a first layer having a first image in which a semi-transparent object, which is a virtual object having transparency, is drawn and first depth information corresponding to the first image; a second acquisition means for acquiring a second layer having a second image in which an opaque object, which is a virtual object that does not have transparency, is drawn and second depth information corresponding to the second image; a third acquisition means for acquiring a third layer having a third image in which a real object arranged in a real space is drawn and third depth information corresponding to the third image; and a synthesis means for drawing a synthesized image by synthesizing the first image, the second image, and the third image based on the first depth information, the second depth information, and the third depth information.

[0008] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. a first image in which a semi-transparent object, which is a virtual object having transparency, is drawn; a first acquisition step of acquiring a first layer having first depth information corresponding to the first image; a second acquisition step of acquiring a second layer having a second image in which an opaque object, which is a virtual object that does not have transparency, is drawn and second depth information corresponding to the second image; a third acquisition step of acquiring a third layer having a third image in which a real object arranged in real space is drawn and third depth information corresponding to the third image; a compositing step of drawing a composite image by combining the first image, the second image, and the third image based on the first depth information, the second depth information, and the third depth information; The information processing method is characterized by comprising: [Effects of the Invention]

[0009] According to the present invention, it is possible to generate a more appropriate image having a semi-transparent virtual object while reducing the display delay time of a real-life image. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a hardware configuration diagram of an information processing device according to a first embodiment. [Figure 2] 1 is a functional block diagram of an information processing device according to a first embodiment. [Figure 3] FIG. 2 is a diagram illustrating a composite image according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an image according to the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating another composite image according to the first embodiment. [Figure 6] 10 is a flowchart of a CG layer acquisition process according to the first embodiment. [Figure 7] 10 is a flowchart of a layer correction process according to the first embodiment. [Figure 8] 10 is a flowchart of a layer generation process according to the first embodiment. [Figure 9] 4 is a flowchart of image synthesis processing according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, a configuration may be achieved by appropriately combining parts of each of the embodiments described below. In the following embodiments, the same components will be described with the same reference symbols.

[0012] <Embodiment 1> Fig. 1 is a diagram showing an example of the hardware configuration of an information processing device 101 according to embodiment 1. Fig. 2 is a functional block diagram showing the functional configuration of the information processing device 101 according to embodiment 1.

[0013] First, an image (composite image) obtained by combining a real image and a virtual object will be described with reference to Figures 3A and 3B. Here, as an example, a case will be described in which a user's hand (hereinafter referred to as "hand") and a virtual object are displayed on an HMD worn by the user.

[0014] The MR space 300 shown in FIG. 3A is a diagram showing the MR space experienced by a user 301 wearing an information processing device 101, viewed from above. The angle of view 302 is the range that the user 301 can see through the information processing device 101. The semi-transparent object 303 is a virtual object with transparency. The hand 304 is the hand of the user 301. The user 301 sees his / her own hand 304 as part of a real-life image. The opaque object 305 is , are virtual objects that do not have transparency. With the user 301 as the reference, the distance from the user 301 is greatest in the order of semi-transparent object 303, hand 304, and opaque object 305.

[0015] The display image 310 shown in FIG. 3B is an image displayed on the information processing device 101. The display image 310 represents the anteroposterior relationship between a "real object" and a "virtual object including a semi-transparent object 303" in the MR space 300. The background 306 is a real-life image. Here, an example is shown in which the background 306 is a real-life image, but it may also be a background representing a virtual space configured by virtual objects.

[0016] In FIG. 3B, from the front, the objects are arranged in the following order: semi-transparent object 303, hand 304, opaque object 305, and background 306. Semi-transparent object 303 has transparency. Therefore, in the area where "hand 304, opaque object 305, and background 306" overlap with semi-transparent object 303, the transparent color information of semi-transparent object 303 is synthesized. On the other hand, because hand 304 and opaque object 305 do not have transparency, the image information behind these objects is occluded and cannot be seen.

[0017] Therefore, even if the translucent object 303, the hand 304, and the opaque object 305 are aligned in the positional relationship shown in FIG. 3A, when the image 360 ​​shown in FIG. 5B or the image 370 shown in FIG. 5C is displayed, the image is displayed in an inappropriate positional relationship. For example, unlike the first embodiment, assume that the translucent object 303 and the opaque object 305 are collectively represented by a single virtual image 350 shown in FIG. 5A, and depth information corresponding to the virtual image is acquired. In such a case, the depth information indicates the depth corresponding to the opaque object 305, because the depth of the translucent object 303 is usually ignored in the area where the translucent object 303 and the opaque object 305 overlap. Therefore, the depth information does not reflect the appropriate positional relationship between the translucent object 303 and the opaque object 305. Therefore, when the hand 304 is composited into the virtual image 350, an image 360 ​​is generated in which the hand 304 is positioned in the foreground, as shown in FIG. 5B.

[0018] 1 shows the hardware configuration of an information processing device 101, which is an example of an HMD used by a user. The information processing device 101 has a CPU 102, a ROM 103, a RAM 104, a sensing unit 105, an imaging unit 106, a display unit 107, an operation unit 108, and a communication unit 109. The components are connected to each other via a bus 110.

[0019] The CPU 102 is an arithmetic processing unit that performs overall control of the information processing device 101. The CPU 102 executes various programs stored in the ROM 103 or the like to perform various processes.

[0020] The ROM 103 stores programs (such as image processing programs and initial data) and parameters that do not require modification. The ROM 103 is a read-only nonvolatile memory device.

[0021] The RAM 104 temporarily stores input information, calculation results in image processing, etc. The RAM 104 is also a memory device that provides a working area for the CPU 102.

[0022] The sensing unit 105 is a device such as a sensor. The sensing unit 105 acquires information about the position and orientation of the user of the information processing device 101 by detecting the rotation, tilt, and amount of movement of the user's head. The sensing unit 105 may also acquire hand tracking information of the user of the information processing device 101 and information about surrounding real objects (model data, depth information, or position and orientation information) by using an infrared sensor or the like. good.

[0023] The photographing unit (image capturing unit) 106 is a photographing device that captures (photographs) the real space to obtain a captured image. The photographing unit 106 is a built-in camera of the HMD, a web camera connected to a PC, or the like.

[0024] The display unit 107 is a liquid crystal display, etc. The display unit 107 displays captured images, virtual objects, characters, items, etc.

[0025] The operation unit 108 is an operation unit including operation members such as a power button or a dial, etc. The operation unit 108 may include a keyboard or a mouse.

[0026] The communication unit 109 transmits and receives data to and from external devices via wired communication or wireless communication (such as wireless LAN or local 5G). In this embodiment, the communication unit 109 can transmit position and orientation information detected by the user's HMD and receive information about real objects detected by other devices (such as model data and position and orientation information) via a network.

[0027] 2 is a functional block diagram of the information processing device 101. The information processing device 101 includes an image acquisition unit 201, a position and orientation acquisition unit 202, a CG information storage unit 203, a semi-transparent layer acquisition unit 204, an opaque layer acquisition unit 205, a layer storage unit 206, and a layer correction unit 207. The information processing device 101 also includes a real object detection unit 208, a real layer acquisition unit 209, an image synthesis unit 210, and an output unit 211.

[0028] 2 can be realized by the CPU 102 executing a program, but it is not necessary for the CPU 102 to realize all of the functions. For example, the information processing device 101 may be provided with a dedicated processing circuit that realizes one or more functions.

[0029] The image acquisition unit 201 acquires an image of the real space captured by the imaging unit 106 as a real image.

[0030] The position and orientation acquisition unit 202 acquires information such as the position, orientation, speed, and acceleration of the HMD worn by the user 301 from the sensing unit 105 or the like as position and orientation information. Note that the position and orientation acquisition unit 202 may acquire information on the self-position of the HMD calculated using a self-position estimation technology based on a real-life image acquired from the image acquisition unit 201 or the like as position and orientation information.

[0031] The CG information storage unit 203 stores CG information necessary for rendering multiple virtual objects (CG), including semi-transparent objects. The CG information includes model data of the virtual objects, position and orientation information, color information including transparency, and camera viewpoint information (such as position, angle of view, and resolution) when rendering the virtual objects.

[0032] The semi-transparent layer acquisition unit 204 acquires a semi-transparent CG layer from the rendering engine based on the "position and orientation information acquired from the position and orientation acquisition unit 202" and the "CG information acquired from the CG information storage unit 203." The semi-transparent CG layer has an image of a semi-transparent object and depth information corresponding to the image. Details of the processing by the semi-transparent layer acquisition unit 204 will be described later with reference to the flowchart in FIG. 6.

[0033] The opaque layer acquisition unit 205 renders an opaque CG layer based on the "position and orientation information acquired from the position and orientation acquisition unit 202" and the "CG information acquired from the CG information storage unit 203." The opaque CG layer has an image of an opaque object and depth information corresponding to that image. Details of the processing by the opaque layer acquisition unit 205 will be described later with reference to the flowchart in FIG.

[0034] The layer storage unit 206 stores translucent CG layers and opaque CG layers as CG layers. The layer storage unit 206 also stores real object layers acquired by the real layer acquisition unit 209. The real object layers have images of real objects and depth information corresponding to the images.

[0035] The layer correction unit 207 corrects the "image and depth information" stored as a CG layer in the layer storage unit 206 based on the latest position and orientation information of the HMD. Details of the processing by the layer correction unit 207 will be described later with reference to the flowchart in FIG. 7. Note that here, an example will be described in which the layer correction unit 207 corrects the CG layers acquired by the translucent layer acquisition unit 204 and the opaque layer acquisition unit 205. However, the layer correction unit 207 may also correct the real object layer acquired by the real layer acquisition unit 209.

[0036] The real object detection unit 208 acquires information about real objects arranged in the real space based on the real image acquired by the image acquisition unit 201. Details of the processing by the real object detection unit 208 will be described later with reference to the flowchart in FIG.

[0037] The reality layer acquisition unit 209 acquires a reality object layer based on the information about the reality object acquired by the reality object detection unit 208. The reality layer acquisition unit 209 stores the reality object layer in the layer storage unit 206. Details of the processing by the reality layer acquisition unit 209 will be described later with reference to the flowchart in FIG. 8.

[0038] The image synthesis unit 210 generates a synthesized image based on the "CG layer and real object layer" stored in the layer storage unit 206. Details of the processing by the image synthesis unit 210 will be described later with reference to the flowchart of FIG.

[0039] The output unit 211 displays the composite image generated by the image composition unit 210 on the display unit 107. In this way, the output unit 211 presents the composite image to the user 301.

[0040] (Processing of the semi-transparent layer acquisition part and the opaque layer acquisition part) An example of the details of the CG layer acquisition process executed by the semi-transparent layer acquisition unit 204 and the opaque layer acquisition unit 205 will be described with reference to the flowchart of FIG.

[0041] In step S600, the semi-transparent layer acquisition unit 204 determines whether or not to acquire (generate) a semi-transparent CG layer related to a semi-transparent object. If it is determined that a semi-transparent CG layer will be acquired (generated), the process proceeds to step S602. If it is determined that a semi-transparent CG layer will not be acquired (generated), the process proceeds to step S603.

[0042] The semi-transparent layer acquisition unit 204 determines to acquire a semi-transparent CG layer only in the first case, and determines not to acquire a semi-transparent CG layer in the second case other than the first case. The first case is, for example, when a semi-transparent object is located within the range of the drawing range of the composite image. The first case may also be when the transparency of the semi-transparent object (alpha value indicating transparency) exceeds a preset first threshold. The first case may also be when the distance between the semi-transparent object and the user 301 exceeds a preset second threshold. The first case may also be when the ratio of the drawing range of the semi-transparent object to the drawing range of the composite image exceeds a preset third threshold. The first case may also be when a semi-transparent object is located in front of the hand 304. In the first case The first case may be a case where a semi-transparent object and another virtual object are superimposed on each other in a composite image. Note that the first case may be a case where at least one of the cases described as examples above is satisfied, or a case where two or more of the cases are satisfied.

[0043] In both of the first cases illustrated above, it can be said that there is a high need to place the translucent object in an appropriate position in the composite image. Therefore, in such cases, the processing from step S601 onward is performed, making it possible to display the translucent object appropriately in the composite image. On the other hand, in cases other than the first case illustrated above, even if the position of the translucent object is somewhat inaccurate or the translucent object is treated the same as an opaque object, the user is unlikely to feel uncomfortable when viewing the composite image. Therefore, there is little need to acquire a special translucent CG layer. This reduces processing, thereby improving the processing efficiency of generating the composite image.

[0044] In step S601, the translucent layer acquisition unit 204 acquires a CG image and depth information of a translucent object based on the "position and orientation information acquired from the position and orientation acquisition unit 202" and the "CG information acquired from the CG information storage unit 203."

[0045] Specifically, first, the semi-transparent layer acquisition unit 204 acquires a CG image of a semi-transparent object that has been rendered by a rendering engine. CG image 320 shown in Fig. 4A is an example of a CG image of a semi-transparent object that has been rendered, and a semi-transparent object 303 that has transparency is drawn.

[0046] Next, the semi-transparent layer acquisition unit 204 acquires depth information of the semi-transparent object that has been rendered by the rendering engine. The depth information of the semi-transparent object is information that indicates the depth of each pixel (each position) of the CG image of the semi-transparent object, and the depth information and the screen resolution (aspect ratio and number of pixels) of the CG image match each other.

[0047] In step S602, the semi-transparent layer acquisition unit 204 acquires a combination of the CG image and depth information of the semi-transparent object acquired in step S601 as a semi-transparent CG layer. Therefore, the semi-transparent CG layer has the CG image and depth information of the semi-transparent object.

[0048] In step S603, the opaque layer acquisition unit 205 determines, as opaque objects, all objects other than the semi-transparent objects rendered in step S601. For this reason, for example, if it is determined in step S600 that a semi-transparent CG layer is not acquired, a transparent object may be determined to be an opaque object. Then, the opaque layer acquisition unit 205 acquires a CG image and depth information of the opaque object.

[0049] Specifically, first, the opaque layer acquisition unit 205 acquires a CG image of an opaque object that has been rendered by a rendering engine based on the position and orientation information and the CG information. CG image 330 shown in Fig. 4B is an example of a CG image of an opaque object that has been rendered, and depicts an opaque object 305.

[0050] Next, the opaque layer acquisition unit 205 acquires depth information of the opaque object rendered by the rendering engine. The depth information of the opaque object is information that indicates the depth of each pixel (each position) of the CG image of the opaque object, and the depth information and the screen resolution (aspect ratio and number of pixels) of the CG image match each other.

[0051] In step S604, the opaque layer acquisition unit 205 acquires the opaque layer A combination of the CG image of the opaque object and the depth information is acquired (generated) as an opaque CG layer. Therefore, the opaque CG layer has the CG image of the opaque object and the depth information.

[0052] In the processes following this flowchart, the "semi-transparent CG layer generated in step S602" and the "opaque CG layer generated in step S604" are used interchangeably. For this reason, the semi-transparent CG layer and the opaque CG layer are collectively referred to as the "CG layer."

[0053] Furthermore, in the explanation of steps S600 to S602, an example was given in which only one translucent CG layer is generated, but the number of translucent CG layers is not limited to one. For example, a plurality of translucent objects may be divided into any number of groups, and a plurality of translucent CG layers may be generated for each group. Similarly, the number of opaque CG layers is not limited to one, and a plurality of opaque CG layers may be generated.

[0054] (Layer correction processing) An example of the details of the layer correction process executed by the layer correction unit 207 will be described with reference to the flowchart of FIG.

[0055] In step S700, the layer correction unit 207 acquires the position and orientation information of the latest frame or an arbitrary frame from the position and orientation acquisition unit 202.

[0056] In step S701, the layer correction unit 207 determines whether the processing from step S702 onwards has already been performed on all layers held by the layer holding unit 206. If it is determined that the processing from step S702 onwards has already been performed on all layers, the processing of this flowchart ends. If it is determined that the processing from step S702 onwards has not been performed on at least one of all layers, the layer correction unit 207 selects one layer from one or more layers for which it is determined that the processing from step S702 onwards has not been performed. Here, the layer selected by the layer correction unit 207 is referred to as the "selected layer." Then, the process proceeds to step S702.

[0057] In step S702, the layer correction unit 207 estimates the position and orientation of each virtual object based on the position and orientation information acquired in step S700. The layer correction unit 207 corrects the position and orientation of the virtual object corresponding to the selected layer in the CG image of the selected layer based on the estimation result. In this way, the layer correction unit 207 corrects the CG image of the selected layer.

[0058] In step S703, the layer correction unit 207 estimates the position and orientation of each virtual object based on the position and orientation information acquired in step S700. The layer correction unit 207 corrects the depth information of the selected layer based on the estimation result.

[0059] In step S704, the layer correction unit 207 stores the combination of the "CG image corrected in step S702" and the "depth image corrected in step S703" as a "corrected layer" in the layer storage unit 206. At this time, the layer correction unit 207 may overwrite the "selected layer" stored in the layer storage unit 206 with the "corrected layer."

[0060] (Processing of the reality layer detection unit and reality layer acquisition unit) An example of the details of the layer generation process executed by the real object detection unit 208 and the real layer acquisition unit 209 will be described with reference to the flowchart of FIG.

[0061] In step S800, the real object detection unit 208 acquires information about a real object, such as a hand, in the real image acquired by the image acquisition unit 201. In the first embodiment, the real object detection unit 208 detects the area of ​​the hand in the real image by performing image processing or the like on the real image.

[0062] 4C is an example of a real-life image acquired by the image acquisition unit 201, and depicts a hand 304 detected by the real object detection unit 208 and a background 306. Note that in this example, the real object to be detected is the hand 304. However, the real object to be detected may be the body of the user 301, such as the foot, a person other than the user 301, or a piece of furniture, such as a desk or a chair.

[0063] In step S801, the real object detection unit 208 acquires depth information of the "real object detected in step S800." For example, the depth information can be acquired by stereo matching using multiple real-life images captured of a real space from multiple viewpoints corresponding to the right eye and left eye. The real object detection unit 208 may also use the depth information of the real object acquired from the sensing unit 105. The real object detection unit 208 may also acquire depth information of a real object detected by another device via the communication unit 109. The depth information of the real object is information indicating the depth of each pixel (each position) of the real-life image of the real object, and the screen resolution (aspect ratio and number of pixels) of the depth information and the real-life image match each other.

[0064] In step S802, the reality layer acquisition unit 209 generates a reality object layer by combining the real-life image acquired in step S800 and the depth information acquired in step S801. The reality object layer has a real-life image in which a reality object is reflected and the depth information of the reality object.

[0065] In step S803, the reality layer acquisition unit 209 stores the reality object layer generated in step S802 in the layer storage unit 206.

[0066] In the description of steps S800 to S802, an example in which only one real object layer is generated has been shown, but the number of real object layers is not limited to 1. For example, multiple real objects may be divided into arbitrary groups, and multiple real object layers may be generated for each group.

[0067] (Image synthesis processing) An example of the details of the image synthesis process executed by the image synthesis unit 210 will be described with reference to the flowchart in Fig. 9. The image synthesis unit 210 generates a synthesized image by synthesizing an image of a semi-transparent object, an image of an opaque object, and an image of a real object. Here, based on the depth information possessed by each layer, the pixels of each image are drawn (synthesized) in order from back to front, thereby making it possible to generate a synthesized image with an appropriate chronological relationship.

[0068] In step S900, the image synthesis unit 210 acquires multiple layers (hereinafter referred to as a "layer list") to be used for synthesis from the layer storage unit 206. The layer list is composed of CG layers and real object layers. At this time, if the layer correction unit 207 has performed correction, each layer includes a corrected layer instead of the layer before correction. Note that for all layers stored in the layer list, the screen resolutions (aspect ratio and number of pixels) of the images and depth information stored in each layer are consistent with each other.

[0069] In step S901, the image synthesis unit 210 determines whether the processing from step S902 onwards has been executed for all pixels of the synthesized image. If it is determined that the processing from step S902 onwards has been executed for all pixels, the processing of this flowchart ends. If it is determined that the processing from step S902 onwards has not been executed for at least one pixel, the processing from step S902 onwards is executed for one pixel for which the processing from step S902 onwards has not yet been executed. Hereinafter, the pixel (pixel of depth information) that is the target of the processing from step S902 onwards will be referred to as the "selected pixel."

[0070] In step S902, the image synthesis unit 210 acquires pixels (hereinafter referred to as "layer pixels") of images held by each layer in the layer list that are located at the coordinates of the selected pixel in the synthesized image (hereinafter referred to as "selected coordinates"). The image synthesis unit 210 then sorts the acquired layer pixels or pixels in descending order of depth information value. As a result, the layer pixels are arranged in descending order of distance from the user 301.

[0071] For example, assume that the first pixel of the composite image is selected as the selected pixel. In this case, it is further assumed that the second pixel of the semi-transparent object 303, the third pixel of the hand 304, and the fourth pixel of the opaque object 305 are located at the coordinates of the selected pixel. Then, in step S902, the image composition unit 210 arranges these three pixels in order of the fourth pixel, the third pixel, and the second pixel, in order of the largest depth information at the selected coordinates.

[0072] In step S903, the image synthesis unit 210 controls to sequentially execute step S904 for the layer pixels sorted in descending order in step S903. Specifically, step S903 determines whether the processing of step S904 has been executed for all layer pixels. If it is determined that the processing of step S904 has been executed for all layer pixels, the process proceeds to step S901. If not, the processing of step S904 is executed for the layer pixel (hereinafter referred to as the "drawing pixel") with the largest depth information value among one or more layer pixels for which the processing of step S904 has not been executed. Note that this is not the case when it is determined that the processing of step S904 has not been executed for at least any layer pixel.

[0073] In step S904, the image composition unit 210 applies the color information of the drawing pixel to the pixel at the coordinates of the selected pixel in the composite image. At this time, the image composition unit 210 performs alpha blending using a value indicating the transparency of the drawing pixel, thereby enabling image composition that takes transparency into consideration.

[0074] For example, assume that in step S902, a certain selected pixel is rearranged in the order of the fourth pixel of the opaque object 305, the third pixel of the hand 304, and the second pixel of the semi-transparent object 303. In this case, by repeating the processes of steps S903 and S904, the image synthesis unit 210 draws the fourth pixel, the third pixel, and the second pixel in that order. In this way, the image synthesis unit 210 draws the pixel at the coordinates of the selected pixel in the synthesized image.

[0075] As described above, by generating layers of semi-transparent objects separately from layers of opaque objects, it is possible to generate a composite image that properly expresses the context when compositing images that include layers of real objects. Furthermore, layers of virtual objects can be obtained by a process separate from the detection of real objects. This also reduces the display delay time of live-action images.

[0076] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, unless a contradiction arises, "greater than or equal to A" may be read as "greater than (higher; longer; more) than A," and "less than or equal to A" may be read as "less than (lower; shorter; fewer) than A." Furthermore, "greater than (higher; longer; more) than A" may be read as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be read as "less than or equal to A."

[0077] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0078] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0079] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0080] In the above-described embodiment, the present invention has been described as being applied to an information processing device, but the present invention is not limited to this example and can be applied to any electronic device that can display a composite image. The electronic device may be a computer, a smartphone, a tablet terminal, a digital camera, or a home appliance.

[0081] <Other embodiments> The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0082] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) a first acquisition means for acquiring a first layer having a first image in which a semi-transparent object, which is a virtual object having transparency, is drawn and first depth information corresponding to the first image; A second layer is obtained, which has a second image in which an opaque object, which is a virtual object that does not have transparency, is drawn and second depth information corresponding to the second image. Acquisition means; a third acquisition means for acquiring a third layer having a third image in which a real object arranged in a real space is drawn and third depth information corresponding to the third image; and a synthesis means for drawing a synthesized image by synthesizing the first image, the second image, and the third image based on the first depth information, the second depth information, and the third depth information. (Configuration 2) the first acquisition means acquires the first layer when the semi-transparent object is located within a range of a drawing range of the composite image. 2. The information processing device according to configuration 1, (Configuration 3) the first acquisition means acquires the first layer when the transparency of the semi-transparent object exceeds a first threshold. 3. The information processing device according to configuration 1 or 2. (Configuration 4) the first acquisition means acquires the first layer when a distance between the semi-transparent object and the user exceeds a second threshold. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) the first acquisition means acquires the first layer when a ratio of a drawing range of the semi-transparent object to a drawing range of the composite image exceeds a third threshold. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) the first acquisition means acquires the first layer when the semi-transparent object is located in front of the real object. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The device further includes a position and orientation acquisition means for acquiring information on the position and orientation of the user. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) further comprising a correction means for correcting the first layer, the second layer, and the third layer based on the position and posture of the user; 8. The information processing device according to configuration 7. (Configuration 9) further comprising image acquisition means for acquiring a captured image of a real space including the real object; the third acquisition means acquires the third layer based on the captured image. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 10) the combining means draws the combined image obtained by combining the first image, the second image, and the third image based on the first depth information, the second depth information, and the third depth information and based on a value indicating pixel transparency. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) when a first pixel of the first image, a second pixel of the second image, and a third pixel of the third image are located at the same coordinates, the synthesizing means draws the first pixel, the second pixel, and the third pixel in order of the pixel having the largest corresponding depth. 11. The information processing device according to any one of configurations 1 to 10. (method) a first image in which a semi-transparent object, which is a virtual object having transparency, is drawn; a first acquisition step of acquiring a first layer having first depth information corresponding to the first image; a second acquisition step of acquiring a second layer having a second image in which an opaque object, which is a virtual object that does not have transparency, is drawn and second depth information corresponding to the second image; a third acquisition step of acquiring a third layer having a third image in which a real object arranged in real space is drawn and third depth information corresponding to the third image; a compositing step of drawing a composite image by combining the first image, the second image, and the third image based on the first depth information, the second depth information, and the third depth information; An information processing method comprising: (program) 12. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0083] 101: Information processing device, 204: Semi-transparent layer acquisition unit, 205: opaque layer acquisition unit, 209: reality layer acquisition unit, 210: Image synthesis unit

Claims

1. a first acquisition means for acquiring a first layer having a first image in which a semi-transparent object, which is a virtual object having transparency, is drawn and first depth information corresponding to the first image; a second acquisition means for acquiring a second layer having a second image in which an opaque object, which is a virtual object that does not have transparency, is drawn and second depth information corresponding to the second image; a third acquisition means for acquiring a third layer having a third image in which a real object arranged in a real space is drawn and third depth information corresponding to the third image; and a synthesis means for drawing a synthesized image by synthesizing the first image, the second image, and the third image based on the first depth information, the second depth information, and the third depth information.

2. the first acquisition means acquires the first layer when the semi-transparent object is located within a drawing range of the composite image.

2. The information processing apparatus according to claim 1, wherein:

3. the first acquisition means acquires the first layer when the transparency of the semi-transparent object exceeds a first threshold.

2. The information processing apparatus according to claim 1, wherein:

4. the first acquisition means acquires the first layer when a distance between the semi-transparent object and the user exceeds a second threshold.

2. The information processing apparatus according to claim 1, wherein:

5. the first acquisition means acquires the first layer when a ratio of a drawing range of the semi-transparent object to a drawing range of the composite image exceeds a third threshold.

2. The information processing apparatus according to claim 1, wherein:

6. the first acquisition means acquires the first layer when the semi-transparent object is located in front of the real object.

2. The information processing apparatus according to claim 1, wherein:

7. The device further includes a position and orientation acquisition means for acquiring information on the position and orientation of the user.

2. The information processing apparatus according to claim 1, wherein:

8. further comprising a correction means for correcting the first layer, the second layer, and the third layer based on the position and posture of the user; 8. The information processing apparatus according to claim 7,

9. further comprising image acquisition means for acquiring a captured image of a real space including the real object; the third acquisition means acquires the third layer based on the captured image.

2. The information processing apparatus according to claim 1, wherein:

10. the combining means draws the combined image obtained by combining the first image, the second image, and the third image based on the first depth information, the second depth information, and the third depth information and based on a value indicating pixel transparency.

2. The information processing apparatus according to claim 1, wherein:

11. when a first pixel of the first image, a second pixel of the second image, and a third pixel of the third image are located at the same coordinates, the composition means draws the first pixel, the second pixel, and the third pixel in order of corresponding depth from the largest pixel; 2. The information processing apparatus according to claim 1, wherein:

12. a first acquisition step of acquiring a first layer having a first image in which a semi-transparent object, which is a virtual object having transparency, is drawn and first depth information corresponding to the first image; a second acquisition step of acquiring a second layer having a second image in which an opaque object, which is a virtual object that does not have transparency, is drawn and second depth information corresponding to the second image; a third acquisition step of acquiring a third layer having a third image in which a real object arranged in a real space is rendered and third depth information corresponding to the third image; a compositing step of drawing a composite image by combining the first image, the second image, and the third image based on the first depth information, the second depth information, and the third depth information; An information processing method comprising:

13. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 11.

Images