Image processing device, image processing method, and program

Abstract

To reduce a calculation amount of generation processing of an MR static image while improving the quality of the MR static image.SOLUTION: An image processing device 100 acquires photographed image data, generates CG corresponding to the view from a virtual viewpoint of a virtual object arranged in a virtual space on the basis of rendering information indicating a rendering condition, generates an MR moving image by superimposing the CG on a photographed image, and outputs generation information used when generating the MR static image in a case of receiving an acquisition request of data of the MR static image corresponding to a frame of the MR moving image. Here, the generation information includes data and rendering information of the photographed image when the acquisition request is received, and intermediate data generated when the CG is generated, the intermediate data being data when the acquisition request is received. The intermediate data is information used for writing processing when the CG is generated.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to image processing techniques for generating MR images. [Background technology]

[0002] When generating MR (Mixed Reality) moving images and displaying them on a display device, it is common to reduce the amount of computation required to generate CG (Computer Graphics) to be superimposed on the MR images, thereby generating low-quality CG, in order to maintain the real-time nature of the moving images. Therefore, if frame data of a given MR moving image is stored as MR still image data, the MR still image will be of low quality. Patent Document 1 discloses a technology that combines real-time CG generation as a moving image with high-quality still images by using a means for generating CG in real time and a separate means for generating high-quality CG at any given time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-67820 Summary of the Invention [Problem to be solved by the invention]

[0004] A large number of calculations are required for the generation process to generate high-quality CG at any time point using the technology described in Patent Document 1. Therefore, when the technology described in Patent Document 1 is applied to the generation of MR images, a large number of calculations are required for the generation process to generate high-quality MR still images at any time point. [Means for solving the problem]

[0005] The image processing device according to the present disclosure includes an image acquisition means for acquiring data of a captured image, a CG generation means for generating CG corresponding to the appearance of a virtual object placed in a virtual space from a virtual viewpoint based on rendering information indicating rendering conditions, an MR generation means for generating an MR moving image by superimposing the CG on the captured image, a receiving means for receiving a request for acquisition of data of an MR still image corresponding to a frame of the MR moving image, and an output means for outputting generation information used when generating the MR still image, wherein the output means outputs, as the generation information, the data of the captured image at the time the acquisition request is received, the rendering information at the time the acquisition request is received, and intermediate data generated when generating the CG, the intermediate data at the time the acquisition request is received, and the intermediate data is information used for lighting processing when generating the CG. [Effects of the Invention]

[0006] According to the present disclosure, it is possible to improve the quality of an MR still image at any point in time, while reducing the amount of calculation required for generating an MR still image. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram showing an example of a functional configuration of an image processing device 100 according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an image processing device 100 according to a first embodiment. [Figure 3] 1 is a block diagram showing an example of a functional configuration of an image generating apparatus according to a first embodiment. [Figure 4] 4 is a flowchart showing an example of a processing flow of the image processing device according to the first embodiment. [Figure 5] 5 is a flowchart showing an example of a processing flow of a CG generation unit included in the image processing device according to the first embodiment. [Figure 6] FIG. 2 is an explanatory diagram for explaining an example of processing by a CG generating unit included in the image processing device according to the first embodiment. [Figure 7] FIG. 10 is an explanatory diagram for explaining an example of a method for calculating diffuse reflected light. [Figure 8] FIG. 10 is an explanatory diagram illustrating an example of a method for calculating specular reflected light. [Figure 9] 4 is a flowchart showing an example of a processing flow of the image generating apparatus according to the first embodiment. [Figure 10] 10 is a flowchart showing an example of a processing flow of an image processing device according to a second embodiment. [Figure 11] 10 is a flowchart showing an example of a processing flow of an image generating apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments do not necessarily limit the present disclosure. Furthermore, not all combinations of features described in the following embodiments are necessarily essential to the solutions of the present disclosure, and multiple features may be combined arbitrarily. Furthermore, each process (step) in the flowchart is indicated by a reference number beginning with "S."

[0009] [Embodiment 1] An image processing device 100 according to a first embodiment will be described with reference to Figs. 1 to 9. First, the configuration of the image processing device 100 will be described with reference to Figs. 1 and 2. Fig. 1 is a block diagram showing an example of the functional configuration of the image processing device 100 according to the first embodiment. As shown in Fig. 1, the image processing device 100 is applied to, for example, an image processing system 1. The image processing system 1 includes the image processing device 100, an imaging device 110, and an image generation device 120.

[0010] The image processing device 100 and the imaging device 110 are connected to each other so that they can communicate with each other via a communication line 140 such as a dedicated line or a LAN (Local Area Network). The image processing device 100 and the image generation device 120 are connected to each other so that they can communicate with each other via a communication line 150 such as a dedicated line or a LAN. The imaging device 110 is configured with a digital still camera, a digital video camera, or the like, and outputs data (hereinafter also referred to as captured image data) of an image obtained by capturing (hereinafter referred to as captured image).

[0011] The image processing device 100 acquires captured image data output from the imaging device 110 via a communication line 140, and generates an image (hereinafter referred to as an MR image) by superimposing computer graphics (hereinafter referred to as CG) on the captured image. The MR image generated by the image processing device 100 is a moving image (hereinafter referred to as an MR moving image). The image processing device 100 displays the generated MR moving image on a display device not shown in FIG. 1. The image processing device 100 is applied to, for example, an HMD (Head Mounted Display). When the image processing device 100 is applied to an HMD, the imaging device 110 is, for example, disposed in the HMD and captures an image in the direction in which the face of a user wearing the HMD is facing. In this case, the image processing device 100 displays the generated MR moving image on a display device disposed in the HMD.

[0012] The image processing device 100 also receives, from a user or the like, a request to acquire data of a still image of an MR image (hereinafter referred to as an MR still image) corresponding to a frame of an MR moving image (hereinafter simply referred to as an acquisition request). The image processing device 100 outputs information necessary for generating an MR still image corresponding to a frame of the MR moving image when the acquisition request is received. Hereinafter, the information necessary for generating an MR still image will be referred to as "generation information." Specifically, when the acquisition request is received, the image processing device 100 outputs captured image data, depth data, rendering information, and intermediate data generated when generating CG as generation information. In this embodiment, the image processing device 100 will be described as outputting, as generation information, not only the captured image data, rendering information, and intermediate data generated when generating CG, but also the depth data when the acquisition request is received. Here, the depth data is data indicating the depth of a space corresponding to at least a part of the imaging range of the captured image, i.e., data indicating the distance from the imaging device to an object existing in the space. The rendering information is information indicating rendering conditions used when generating CG. The rendering information includes at least viewpoint information indicating the position of the viewpoint. The intermediate data is generated when generating the CG, and is used, for example, for lighting processing when generating the CG. The depth data, rendering information, and intermediate data will be described in detail later.

[0013] The image generation device 120 acquires the generation information output from the image processing device 100 via the communication line 150 and generates an MR still image based on the acquired generation information. Specifically, the image generation device 120 generates CG to be superimposed on the captured image using intermediate data and rendering information included in the generation information. Furthermore, the image generation device 120 generates an MR still image in which the CG is superimposed on the captured image using the generated CG data and the captured image data and depth data included in the generation information. By generating CG using the intermediate data and rendering information, the image generation device 120 can generate high-quality CG with a smaller amount of calculation compared to the quality of the above-mentioned frames. As a result, the image generation device 120 can generate high-quality MR still images with a smaller amount of calculation.

[0014] <Configuration of image processing device> The image processing device 100 has, as its functional configuration, an image acquisition unit 101, a CG generation unit 102, an MR generation unit 103, a reception unit 104, and an output unit 105. The processing of each unit included in the image processing device 100 as a functional configuration is performed by hardware such as an ASIC (Application Specific Integrated Circuit) built into the image processing device 100. The processing may also be performed by hardware such as an FPGA (Field Programmable Gate Array). Furthermore, the processing of each unit included in the image processing device 100 as a functional configuration may also be performed by software using a memory such as a RAM (Random Access Memory) and a processor such as a CPU (Central Processor Unit). Details of the processing of each unit shown in FIG. 1 will be described later.

[0015] The hardware configuration of the image processing device 100 when each unit included as a functional configuration of the image processing device 100 operates as software will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the image processing device 100 according to the first embodiment. The image processing device 100 is configured by a computer, and the computer has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication unit 207, and a bus 208, as shown as an example in Fig. 2.

[0016] The CPU 201 is a processor that controls the computer using programs or data stored in the ROM 202, RAM 203, or the like, to cause the computer to function as each unit included in the functional configuration of the image processing device 100 shown in FIG. 1 . The image processing device 100 may also have one or more dedicated hardware components different from the CPU 201, and at least some of the processing performed by the CPU 201 may be performed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC, an FPGA, and a DSP (digital signal processor). The ROM 202 is a memory that stores programs and the like that do not require modification. The RAM 203 is a memory that temporarily stores programs or data supplied from the auxiliary storage device 204, or data and the like that are supplied from the outside via the communication unit 207. The auxiliary storage device 204 is, for example, a hard disk drive, and stores programs or various data such as image data or audio data.

[0017] The display unit 205 is configured, for example, by a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) or the like that allows the user to operate the image processing device 100 or view the status of processing in the image processing device 100. In this embodiment, the MR moving images generated by the image processing device 100 are displayed on the display unit 205. The operation unit 206 is configured, for example, by a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 201 in response to operations by the user. The CPU 201 also operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206.

[0018] The communication unit 207 is used for communication such as sending and receiving data between the image processing device 100 and an external device. For example, if the image processing device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication unit 207. If the image processing device 100 has a function for wireless communication with an external device, the communication unit 207 is equipped with an antenna. In this embodiment, the image processing device 100 communicates with the imaging device 110, the image generation device 120, etc. via the communication unit 207. The bus 208 connects each unit provided as a hardware configuration of the image processing device 100 to transmit information. In the first embodiment, the display unit 205 and the operation unit 206 are described as being present inside the image processing device 100, but at least one of the display unit 205 and the operation unit 206 may be present as a separate device outside the image processing device 100.

[0019] <Processing in each functional configuration of the image processing device> The processing of each unit included in the functional configuration of the image processing device 100 will be described. The image acquisition unit 101 acquires captured image data. Specifically, the image acquisition unit 101 acquires captured image data output by the imaging device 110 via the communication unit 207. The source of the captured image data is not limited to the imaging device 110, and the image acquisition unit 101 may acquire the captured image data by reading the captured image data from a storage device not shown in FIG. 1. In this case, for example, the imaging device 110 outputs the captured image data to the storage device and stores the captured image data in advance in the storage device.

[0020] The image acquisition unit 101 may acquire depth data in addition to captured image data. For example, the image acquisition unit 101 acquires depth image data output by a depth camera (not shown in FIG. 1) as the depth data. In this case, the image capture device 110 and the depth camera are disposed in close proximity to each other, and it is preferable that the direction of the optical axis of the image capture device 110 and the direction of the optical axis of the depth camera are the same. Note that "same" here is not limited to a case where they are completely the same, but also includes a case where they are approximately the same.

[0021] The depth image data is not limited to that output by a depth camera. For example, if the imaging device 110 is a stereo camera, the image acquisition unit 101 may generate and acquire depth image data using two captured image data corresponding to the left and right sides output by the imaging device 110. A method for generating depth image data using captured image data output by a stereo camera is well known, so a description thereof will be omitted. The depth data is not limited to depth image data, but may also be point cloud data. In this case, for example, the image acquisition unit 101 acquires point cloud data output by a three-dimensional scanner (not shown in FIG. 1) using a laser light time-of-flight method or the like as depth data. In this case, it is preferable that the imaging device 110 and the three-dimensional scanner are located in close proximity to each other. In the following description, the image acquisition unit 101 will be described as acquiring depth data in addition to captured image data.

[0022] The CG generation unit 102 generates CG by performing rendering based on the rendering information. Here, the rendering information is information indicating the rendering conditions when the CG generation unit 102 generates CG. Specifically, the CG generation unit 102 generates CG corresponding to the appearance of a virtual object placed in a virtual space from a virtual viewpoint by performing rendering based on the rendering information. The rendering information includes at least information indicating the position of the virtual viewpoint in the virtual space (hereinafter referred to as viewpoint information). The CG generation process will be described in detail later.

[0023] When generating CG, the CG generation unit 102 generates intermediate data during the generation process. The intermediate data is used, for example, for lighting processing when the CG generation unit 102 generates CG. Here, the intermediate data includes information indicating a base color (hereinafter referred to as color information), information indicating a normal (hereinafter referred to as normal information), information indicating a reflectance (hereinafter referred to as reflectance information), or information indicating the world coordinates of each pixel of the CG (hereinafter referred to as CG coordinate information). The intermediate data includes, for example, at least one of the above information.

[0024] Color information is information that indicates the color or hue of each of one or more elements that make up a virtual object and are displayed in CG. Normal information is information that indicates the orientation of each element that makes up a virtual object and are displayed in CG. Reflectance information is information that indicates the reflectance of each element that makes up a virtual object and are displayed in CG. CG coordinate information is, for example, information that indicates the position in virtual space corresponding to each pixel in CG, and is information expressed using the same coordinate system as real space.

[0025] Specifically, when three-dimensional shape data representing a virtual object is represented by one or more polygons, color information is, for example, information indicating the color or hue of each polygon that constitutes the virtual object as depicted in the CG. In this case, normal information is, for example, information indicating the orientation of each polygon that constitutes the virtual object as depicted in the CG, i.e., information indicating the direction of the normal of each polygon. In this case, reflectance information is, for example, information indicating the reflectance of each polygon that constitutes the virtual object as depicted in the CG.

[0026] The 3D shape data representing a virtual object is not limited to being represented by one or more polygons, and may be represented by, for example, a point cloud or voxels. When the 3D shape data representing a virtual object is represented by a point cloud, the color information is, for example, information indicating the color of each point in the point cloud representing the virtual object depicted in CG. In this case, the normal information is, for example, information indicating the orientation of a plane enclosed by line segments connecting each point in the point cloud representing the virtual object depicted in CG, and is information indicating the direction of the normal to this plane. In this case, the reflectance information is, for example, information indicating the reflectance of a plane enclosed by line segments connecting each point in the point cloud representing the virtual object depicted in CG.

[0027] Similarly, when the three-dimensional shape data representing the virtual object is represented by voxels, the color information is, for example, information indicating the color of each voxel representing the surface of the virtual object, as depicted in CG. In this case, the normal information is, for example, information indicating the orientation of a plane enclosed by lines connecting the voxels representing the surface of the virtual object, as depicted in CG, and is information indicating the direction of the normal to this plane. In this case, the reflectance information is, for example, information indicating the reflectance of a plane enclosed by lines connecting the voxels representing the surface of the virtual object, as depicted in CG. Hereinafter, the three-dimensional shape data representing the virtual object will be described as being represented by one or more polygons.

[0028] The MR generation unit 103 superimposes CG on the captured images to generate frames of the MR moving image. Specifically, the frames of the MR moving image are generated using captured image data, depth data, and CG data. The MR generation unit 103 generates the MR moving image by repeatedly generating frames of the MR moving image. The MR moving image generated by the MR generation unit 103 is output to the display unit 205 and displayed on the display unit 205. Here, since the MR images generated by the MR generation unit 103 are moving images, the MR generation unit 103 generates each frame of the MR moving image in accordance with the display frame rate. Therefore, the CG generation unit 102 generates CG used when generating each frame of the MR moving image in accordance with the display frame rate. A method for generating frames of the MR moving image using captured image data and CG data is well known, so a detailed description thereof will be omitted.

[0029] The receiving unit 104 receives a signal indicating the acquisition request, thereby accepting the acquisition request. The signal is output from the operation unit 206, for example, when the user operates the operation unit 206. The output unit 105 outputs generation information used when generating MR still images corresponding to frames of MR moving images when the acquisition request is accepted. The generation information includes captured image data used by the MR generation unit 103 to generate frames of MR moving images when the acquisition request is accepted. The generation information also includes rendering information used when generating CG used to generate frames of MR moving images when the acquisition request is accepted. The generation information also includes intermediate data generated when generating CG used to generate frames of MR moving images when the acquisition request is accepted.

[0030] Furthermore, the output unit 105 may output, in addition to viewpoint information, rendering information including data of a shader code used as a rendering condition when an acquisition request is received, as part of the generation information. Furthermore, the output unit 105 may output, in addition to viewpoint information, rendering information including information indicating the position of a light source, the angle at which light emitted by the light source penetrates, the color of the light, or the intensity of the light (hereinafter referred to as light source information), as part of the generation information. Furthermore, the output unit 105 may output, in addition to captured image data, depth data used for generating frames of an MR moving image when an acquisition request is received, as part of the generation information. The output unit 105 outputs the generation information to the image generation device 120 via the communication line 150. The output destination of the generation information is not limited to the image generation device 120. The output unit 105 may output the generation information to the auxiliary storage device 204 or a storage device not shown in FIG. 1 , etc., and store the output generation information.

[0031] <Configuration of image generation device> The image generation device 120 uses the generation information output by the image processing device 100 to generate a high-quality MR still image corresponding to the frame of the MR moving image when the acquisition request is received. Specifically, the image generation device 120 performs re-rendering using the rendering information and intermediate data included in the generation information to generate a high-quality CG corresponding to the CG superimposed on the frame of the MR moving image when the acquisition request is received. Furthermore, the image generation device 120 uses the captured image data and depth data included in the generation information and the generated CG data to superimpose the CG on the captured image to generate a high-quality MR still image. The functional configuration of the image generation device 120 will be described with reference to FIG. 3. FIG. 3 is a block diagram showing an example of the functional configuration of the image generation device 120 according to the first embodiment. The image generation device 120 has, as its functional configuration, an acquisition unit 301, a CG generation unit 302, and an MR generation unit 303.

[0032] The image generating device 120 is configured by a computer such as a personal computer, a smartphone, or a tablet terminal. Each unit that the image generating device 120 has as a functional configuration is implemented by software using a processor such as a CPU or GPU built into the image generating device 120 and a memory such as RAM. Note that the image generating device 120 may have one or more dedicated hardware pieces different from the processor, and at least a part of the processing by the processor may be executed by the dedicated hardware. Examples of dedicated hardware include an ASIC, an FPGA, and a DSP.

[0033] <Processing in each functional configuration of the image generation device> The acquisition unit 301 acquires the generation information output by the image processing device 100 via the communication line 150. The source of the generation information is not limited to the image processing device 100, and the acquisition unit 301 may acquire the generation information by reading it from a storage device that has previously stored the generation information. The CG generation unit 302 generates CG by performing re-rendering using the intermediate data and rendering information included in the generation information acquired by the acquisition unit 301. By performing re-rendering using the intermediate data, the CG generation unit 302 can generate high-quality CG with a small amount of calculation.

[0034] The MR generation unit 303 generates an MR still image in which the CG is superimposed on the captured image, using the captured image data and depth data included in the generation information acquired by the acquisition unit 301 and the CG generated by the CG generation unit 302. In this way, the image generation device 120 can generate, with a small amount of calculation, a high-quality MR still image corresponding to the frame of the MR moving image at the time of receiving the acquisition request.

[0035] <Operation of image processing device> The operation of the image processing device 100 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of a processing flow of the image processing device 100 according to the first embodiment. The image processing device 100 repeatedly executes the processing of this flowchart until it receives an instruction to end the display of the MR moving image on the display device, for example, by a user operation. First, in S410, the image acquisition unit 101 acquires captured image data. Next, in S420, the image acquisition unit 101 acquires depth data. Next, in S430, the CG generation unit 102 generates CG. Details of the rendering process when the CG generation unit 102 generates CG will be described later with reference to Figs. 5 and 6.

[0036] Next, in S440, the MR generation unit 103 generates frames of an MR moving image using the captured image data acquired in S410, the depth data acquired in S420, and the CG data generated in S430. For example, the MR generation unit 103 compares the depth data acquired in S420 with information indicating the depth of each pixel of the CG generated in S430, i.e., information indicating the distance from the virtual viewpoint to a point in virtual space corresponding to the CG pixel. Specifically, for example, the MR generation unit 103 compares, for each pixel, a pixel of the captured image with a pixel of the CG to be superimposed on that pixel, and determines which is located closer to the user's viewpoint. If the CG pixel is closer to the user's viewpoint, the MR moving image frame is generated by superimposing the CG pixel on the captured image.

[0037] Next, in S450, the receiving unit 104 determines whether or not an acquisition request has been received. If it is determined in S450 that an acquisition request has been received, in S460, the output unit 105 outputs the generation information. At this time, the output unit 105 may output frame data of the MR moving image generated in S440 in addition to the generation information. Note that the output unit 105 may output the captured image data, rendering information, intermediate data, etc. included in the generation information all together, or may output the captured image data, rendering information, intermediate data, etc. sequentially in any order. After S460, or if it is determined in S450 that an acquisition request has not been received, the image processing device 100 terminates the processing of the flowchart shown in FIG. 4. After the termination, the image processing device 100 returns to the processing of S410 and repeatedly executes the processing of the flowchart, for example, until it receives an instruction to terminate the display of the MR moving image on the display device.

[0038] <Rendering process> The rendering processing executed in S430 will be described with reference to FIGS. 5 and 6. FIG. 5 is a flowchart showing an example of the processing flow of the rendering processing by the CG generation unit 102 included in the image processing device 100 according to the first embodiment. FIG. 6 is an explanatory diagram for explaining an example of the rendering processing by the CG generation unit 102 included in the image processing device 100 according to the first embodiment. FIG. 6(a) shows the positional relationship between a virtual viewpoint 603 and a virtual object 601. First, in S431, the CG generation unit 102 projects each vertex of the virtual object 601, which is represented by three-dimensional shape data such as polygons, onto a two-dimensional plane 602. FIG. 6(b) shows how the virtual object 601 appears as viewed from the virtual viewpoint 603, and illustrates a region 604 surrounded by line segments connecting the vertices of the virtual object 601 projected onto the two-dimensional plane 602. In other words, the region 604 is a region corresponding to the virtual object 601 projected onto the two-dimensional plane 602.

[0039] Fig. 6(c) shows an image area 606 corresponding to area 604 when two-dimensional plane 602 is converted into image 605. After S431, in S432, the CG generation unit 102 rasterizes the shape projected onto two-dimensional plane 602, i.e., area 604, to obtain image 605 including image area 606 corresponding to area 604. Note that in Fig. 6(c), each area expressed using a rectangle in image 605 indicates a pixel in image 605, and image area 606 is expressed by a collection of pixels.

[0040] 6(d) shows a collection of intermediate data 607. After S432, in S433, the CG generation unit 102 generates various data to be used in subsequent processing and stores some or all of the generated data in the RAM 203 or the like as intermediate data 607. The subsequent processing is, for example, lighting processing for the virtual object 601. Data required for lighting processing is, for example, the above-mentioned color information, normal information, and CG coordinate information, but is not limited to these.

[0041] 6(e) shows an image 608 including an image region 609 corresponding to the virtual object 601 after the lighting process. After S433, in S434, the CG generation unit 102 performs lighting process using the data generated in S433. Through the lighting process, the CG generation unit 102 calculates the shadow that appears when light emitted from a light source is irradiated onto the virtual object 601, and generates an image 608 that takes into account the influence of the light source. Details of the lighting process will be described later.

[0042] 6(f) shows an image 610 after effect processing has been performed on the image 608. After S434, in S435, the CG generation unit 102 performs various effect processing as post-processing on the image 608 generated in S434, and generates the final CG image 610 corresponding to the virtual object 601. The effect processing is, for example, exposure processing that adjusts the brightness of the entire image depending on the display content, but is not limited to this.

[0043] <Lighting processing> In this embodiment, as an example of lighting processing, a form in which the intensities of two reflected lights, diffuse reflected light and specular reflected light, are calculated will be described. Hereinafter, the light source in this embodiment will be described as a directional light that exists at infinity and whose light intensity is constant regardless of the distance from the light source. Note that in this embodiment, as an example, the light source is a directional light and there are the above-mentioned two reflected lights, but the lighting processing method is not limited to this.

[0044] Diffuse reflection is a reflection in which reflected light spreads in various directions with roughly the same intensity. A method for calculating diffuse reflected light will be described with reference to FIG. 7. FIG. 7 is an explanatory diagram for explaining an example of a general method for calculating diffuse reflected light. Below, a case will be described in which light from a directional light is irradiated onto a certain plane 701. A ray vector 702 indicates the direction in which the light emitted by the directional light travels. A normal vector 703 indicates the direction of the normal to the plane 701. In this case, the intensity I of the diffuse reflected light is d is calculated using the following equation (1): d is the diffuse reflectance, I i is the intensity of the light emitted by the light source, N is the normal vector 703 of the plane 701, and L is the ray vector 702 of the light emitted by the directional light. I d =k d I i ·(-N·L) ···Formula (1)

[0045] Specular reflection is a reflection that is strongly reflected in the regular reflection direction and is also called a highlight. A method for calculating specular reflected light will be described with reference to FIG. 8. FIG. 8 is an explanatory diagram for explaining an example of a general method for calculating specular reflected light. Below, a case will be described in which light from a directional light is irradiated onto a certain plane 801. A ray vector 802 indicates the direction in which the light emitted by the directional light travels. A normal vector 803 indicates the direction of the normal to the plane 801. A ray vector 804 indicates the direction in which the specularly reflected light of the light from the directional light 802 travels. A viewpoint 806 indicates the position of the viewpoint, and a line-of-sight vector 805 indicates the direction of the line of sight. In this case, the intensity I of the specular reflected light s is calculated using the following equation (2): where k s is the specular reflectance, I i represents the intensity of the light emitted by the light source, R represents the ray vector indicating the direction of the specularly reflected light from the directional light reflected by the plane 801, V represents the line of sight vector, and n represents the parameter that controls the intensity of the specular reflection. I s =k s I i ·(-R·V) n ··· Equation (2)

[0046] The reflected light vector R and the line of sight vector V can be calculated using the following equations (3) and (4): where N is the normal vector of the plane 801, L is the ray vector of the irradiated light of the directional light, (x e ,y e ,z e ) is the coordinate of the viewpoint, (x s ,y s ,z s ) indicates the coordinates of the incident point of the directional light. R=L+2(-N L)×N Equation (3) V=(x s ,y s ,z s )-(x e ,y e ,z e )...Equation (4)

[0047] The lighting process is performed using the two reflected light intensities of diffuse and specular reflections, resulting in a pixel value I L is calculated using the following equation (5): B indicates the base color value. I L =(I d +I s )·I B ··· Equation (5)

[0048] <Operation of the image generating device> The operation of the image generating device 120 will be described with reference to FIG. 9. FIG. 9 is a flowchart showing an example of a processing flow of the image generating device 120 according to the first embodiment. For example, the image generating device 120 executes the processing of the flowchart shown in FIG. 9 when there is new generation information that has not been used before. First, in S910, the acquisition unit 301 acquires the generation information. In this embodiment, the generation information includes, as rendering information, light source information indicating the angle at which light emitted by a light source enters and the intensity of this light, and the CG generation unit 302 generates CG using the generation information indicated by this information. Note that the rendering conditions used when generating CG are not limited to these.

[0049] Next, in S950, the CG generation unit 302 generates CG using intermediate data such as color information, normal information, reflectance information, and CG coordinate information included in the generation information acquired in S910, and rendering information such as viewpoint information and light source information. Specifically, for example, the CG generation unit 302 generates CG by first calculating the intensities of two types of reflected light, diffuse reflected light and specular reflected light, as a lighting process, similar to S434 by the CG generation unit 102. The intensity of the diffuse reflected light is calculated using the diffuse reflectance and normal information indicated by the reflectance information included in the intermediate data, and the ray vector of the illumination light based on the angle of incidence of light emitted by the light source indicated by the light source information included in the rendering information, and the intensity of the light emitted by the light source indicated by the light source information. The intensity of the specular reflected light is calculated using the reflected light vector calculated from the normal information and the ray vector of the illumination light, and the line of sight vector, specular reflectance, and intensity of light emitted by the light source calculated from the viewpoint information and CG coordinate information. Furthermore, the CG generation unit 302 performs lighting processing using the calculated intensities of the two reflected lights and the color information included in the intermediate data, thereby generating the final CG.

[0050] Next, in S960, the MR generation unit 303 generates an MR still image by superimposing CG on the captured image using the captured image data and depth data included in the generation information acquired in S910 and the CG data generated in S950. After S960, the image generation device 120 ends the processing of the flowchart shown in FIG. 9. The image generation device 120 can generate high-quality CG with a small amount of calculation by rendering CG using the intermediate data at the time of receiving the acquisition request. As a result, the image generation device 120 can generate high-quality MR still images corresponding to the frames of the MR moving image at the time of receiving the acquisition request with a small amount of calculation.

[0051] Therefore, according to the image processing device 100 configured as above, it is possible to improve the quality of an MR still image at any point in time, while reducing the amount of calculation required for generating an MR still image.

[0052] As described above, the rendering information may include shader code data used as a rendering condition when an acquisition request is received. When the rendering information includes shader code data, the CG generation unit 302 may perform lighting processing using the shader code in addition to the intermediate data. With this configuration, the CG generation unit 302 can perform lighting processing using the same shader code as the CG generation unit 102. Therefore, the MR generation unit 303 can generate high-quality MR still images that do not look unnatural compared to the frames of the MR moving images generated by the MR generation unit 103, with a small amount of calculation.

[0053] Furthermore, in the first embodiment, the rendering information output as generation information is described as including not only viewpoint information but also light source information used by the CG generation unit 102 for lighting processing, but this is not limited to this. For example, the rendering information output as generation information may not include information other than viewpoint information, such as light source information. If the rendering information output as generation information does not include light source information, the CG generation unit 302 may, for example, perform lighting processing using light source information prepared in advance to generate CG.

[0054] Furthermore, for example, the CG generation unit 302 may generate light source information from an object and its shadow appearing in a captured image using the captured image data or the captured image data and depth data included in the generation information, and perform lighting processing using the generated light source information. Furthermore, for example, the CG generation unit 302 may perform lighting processing using light source information set by a user via the operation unit 206. Note that the light source information is information indicating the angle at which light emitted by a light source penetrates or the intensity of this light, but the light source information is not limited to this. By configuring in this manner, the CG generation unit 302 can generate CG in which lighting processing has been performed under conditions different from those used when the CG generation unit 102 generated the CG.

[0055] In addition, in the first embodiment, the image processing device 100 is described as acquiring depth data and further outputting the depth data as generation information. However, the image processing device 100 does not necessarily need to acquire depth data, and the depth data does not necessarily need to be included in the generation information. When the image processing device 100 does not acquire depth data, the MR generation unit 103 may generate frames of an MR moving image by, for example, a method of superimposing CG on all pixels of a captured image. When the generation information acquired by the image generation device 120 does not include depth data, the MR generation unit 303 may generate an MR still image by, for example, a method of superimposing CG on all pixels of a captured image. Note that the method of superimposing CG on all pixels of a captured image is merely an example, and processing when depth data is not available is not limited to this.

[0056] Furthermore, in the first embodiment, the CG generation unit 302 generates the final CG by performing lighting processing in S950, but the processing in S950 is not limited to this. For example, in addition to the lighting processing in S950, the CG generation unit 302 may also perform post-processing processing such as that performed in S435. In this case, the CG generation unit 302 may perform post-processing processing using intermediate data included in the generation information.

[0057] In addition, in the first embodiment, the output unit 105 outputs the generation information in S460, but the timing of outputting the generation information is not limited to this. For example, in S460, the output unit 105 saves the generation information in a memory such as the RAM 203. Thereafter, when there is room for processing, such as when an instruction to end the display of the MR moving image on the display device is received and the generation processing of the MR moving image frames is no longer necessary, the output unit 105 may output the generation information saved in the memory.

[0058] In the first embodiment, the image processing device 100 and the image generation device 120 are configured as different devices, but the present invention is not limited to this. For example, the image processing device 100 may have the image acquisition unit 101, the CG generation unit 102, the MR generation unit 103, the reception unit 104, and the output unit 105, as well as the acquisition unit 301, the CG generation unit 302, and the MR generation unit 303 that the image generation device 120 has.

[0059] Furthermore, the application range of the image processing device 100 is not limited to HMD. For example, the image processing device 100 can also be applied to a smartphone or a tablet terminal. For example, when the image processing device 100 is applied to a smartphone, the imaging device 110 is placed on the opposite side of the smartphone's display screen. In this case, the image processing device 100 displays the generated MR moving image on the smartphone's display screen.

[0060] [Embodiment 2] An image processing device 100 according to a second embodiment (hereinafter simply referred to as the image processing device 100) will be described with reference to Figures 1 to 3 and Figures 10 and 11. The image processing device 100 according to the first embodiment outputs captured image data as generation information. The image generation device 120 according to the first embodiment generates an MR still image in which CG is superimposed on a captured image, using the captured image data included in the generation information.

[0061] In contrast, the image processing device 100 acquires RAW data corresponding to the captured image data in addition to the captured image data, and outputs the RAW data as generation information instead of the captured image data. Also, the image generation device 120 according to the second embodiment uses the RAW data included in the generation information to generate an MR still image in which CG is superimposed on an image (hereinafter referred to as a developed image) obtained by developing the RAW data. This makes it possible to generate a higher quality MR still image.

[0062] <Configuration of image processing device> The image processing device 100 includes the units shown as an example in Fig. 1, similarly to the image processing device 100 according to the first embodiment. The processing of each unit included in the image processing device 100 is performed by hardware such as an ASIC or FPGA built into the image processing device 100, similarly to the image processing device 100 according to the first embodiment. The processing may also be performed by software using a memory such as a RAM 203 and a processor such as a CPU 201, as shown as an example in Fig. 2. Furthermore, similarly to the image processing device 100 according to the first embodiment, the image processing device 100 is applied to an image processing system 1 shown as an example in Fig. 1.

[0063] <Processing in each functional configuration of the image processing device> The processing of each unit included in the image processing device 100 as a functional configuration will be described below. The CG generation unit 102, MR generation unit 103, and reception unit 104 included in the image processing device 100 are the same as the corresponding units included in the image processing device 100 according to the first embodiment, and therefore detailed description thereof will be omitted. The image acquisition unit 101 acquires captured image data and RAW data corresponding to the captured image data. Specifically, the image acquisition unit 101 acquires captured image data and RAW data output by the imaging device 110. The source of the captured image data and RAW data is not limited to the imaging device 110. For example, the image acquisition unit 101 may acquire the captured image data or RAW data by reading at least one of the captured image data and RAW data from a storage device not shown in FIG. 1. In this case, for example, the imaging device 110 outputs the captured image data or RAW data to the storage device and stores it in advance in the storage device. The image acquisition unit 101 may acquire depth data in addition to the captured image data and RAW data. In the following description, it is assumed that the image acquisition unit 101 acquires depth data in addition to captured image data and RAW data.

[0064] The output unit 105 outputs generation information used to generate MR still images corresponding to the frames of the MR moving image when the acquisition request is received by re-rendering the CG. The generation information includes RAW data corresponding to the captured image data used to generate the frames of the MR moving image when the acquisition request is received. When the image acquisition unit 101 acquires depth data, the output unit 105 may include the depth data in the output generation information. Furthermore, the output unit 105 outputs, as generation information, rendering information used in the CG generation process when the acquisition request is received and intermediate data generated during the CG generation process when the acquisition request is received. Details of the rendering information and the intermediate data have been described above, so a detailed description thereof will be omitted.

[0065] The rendering information includes at least viewpoint information. In addition to viewpoint information, the rendering information may include at least one of light source information at the time of receiving an acquisition request and shader code data used as a rendering condition at the time of receiving an acquisition request. The output unit 105 outputs the generation information to the image generation device 120 via the communication line 150. The output destination of the generation information is not limited to the image generation device 120, and the output unit 105 may output the generation information to the auxiliary storage device 204 or a storage device not shown in FIG. 1 or the like, and store the output generation information in either of these.

[0066] <Configuration of image generation device> The image generating device 120 includes the units shown as an example in Fig. 3, similar to the image generating device 120 according to the first embodiment. The image generating device 120 is configured by a computer, similar to the image processing device 100 according to the first embodiment. Each unit that the image generating device 120 has as a functional configuration is implemented by software using a processor such as a CPU or GPU, and a memory such as a RAM, that are built into the image generating device 120. Note that the image generating device 120 may include one or more pieces of dedicated hardware that are different from the processor, and at least a part of the processing by the processor may be executed by the dedicated hardware.

[0067] <Processing in each functional configuration of the image generation device> The processing of each unit included in the functional configuration of the image generation device 120 will be described below. Note that the acquisition unit 301 and CG generation unit 302 included in the image generation device 120 are similar to the corresponding units included in the image generation device 120 according to the first embodiment, and therefore detailed description thereof will be omitted. The MR generation unit 303 generates an MR still image using RAW data included in the generation information acquired by the acquisition unit 301 and CG data generated by the CG generation unit 302. The MR still image generated by the MR generation unit 303 is a high-quality MR still image corresponding to the frame of the MR moving image when the acquisition request is received.

[0068] Specifically, first, the MR generation unit 303 develops the RAW data to obtain a developed image corresponding to the RAW data. Here, parameters used for development are, for example, stored in advance in a storage device not shown in FIG. 3, and the MR generation unit 303 obtains the parameters used for development by reading them from the storage device. The parameters used for development are not limited to those stored in the storage device, and may be, for example, parameters set by user input. Next, the MR generation unit 303 superimposes the CG generated by the CG generation unit 302 on the developed image to generate an MR still image. If the generation information includes depth data, the MR generation unit 303 may superimpose the CG on the developed image using the depth data.

[0069] With the above configuration, the image generating device 120 can generate high-quality CG with a small amount of calculation by performing rendering using intermediate data, and can therefore generate high-quality MR still images with a small amount of calculation.

[0070] <Operation of image processing device> The operation of the image processing device 100 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of a processing flow of the image processing device 100 according to the second embodiment. The image processing device 100 repeatedly executes the processing of the flowchart until it receives an instruction to end the display of the MR moving image on the display device, for example, by a user operation. In the following description, the same processes as those of the image processing device 100 according to the first embodiment will be designated by the same reference numerals as in Fig. 4, and description thereof will be omitted.

[0071] First, the image processing device 100 executes the process of S410. After S410, in S1010, the image acquisition unit 101 acquires RAW data corresponding to the captured image data acquired in S410. After S1010, the image processing device 100 executes the processes of S420 to S460. Note that the generated information output by the output unit in S460 does not include the captured image data, but includes RAW data corresponding to the captured image data. After S460, or if it is determined in S450 that an acquisition request has not been received, the image processing device 100 terminates the process of the flowchart shown in FIG. 10. After the termination, the image processing device 100 returns to the process of S410 and repeatedly executes the process of the flowchart, for example, until it receives an instruction to terminate the display of the MR moving image on the display device.

[0072] <Operation of the image generating device> The operation of the image generating device 120 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of a processing flow of the image generating device 120 according to the second embodiment. For example, the image generating device 120 executes the processing of the flowchart shown in Fig. 11 when new generation information that has not been used until now exists. In the following description, the same processes as those of the image generating device 120 according to the first embodiment will be designated by the same reference numerals as those in Fig. 9, and description thereof will be omitted.

[0073] First, the image generating device 120 executes the processes of S910 and S950. Note that the generation information acquired by the acquisition unit 301 in S910 includes RAW data. After S950, in S1110, the MR generation unit 303 develops the RAW data included in the generation information to acquire developed image data. After S1110, in S1120, an MR still image in which CG is superimposed on the developed image is generated using the depth data included in the generation information acquired in S910, the CG data generated in S950, and the developed image data acquired in S1110.

[0074] After S1110, the image generation device 120 ends the processing of the flowchart shown in FIG. 11. The image generation device 120 can generate high-quality CG with a small amount of calculation by rendering CG using the intermediate data at the time of receiving the acquisition request. As a result, the image generation device 120 can generate high-quality MR still images corresponding to frames of MR moving images at the time of receiving the acquisition request with a small amount of calculation. Furthermore, the image generation device 120 generates MR still images corresponding to these frames using RAW data corresponding to the captured image data used to generate the frames of the MR moving images at the time of receiving the acquisition request. This allows the image generation device 120 to generate MR still images of higher quality than when captured image data is used.

[0075] Therefore, the image processing device 100 configured as described above can improve the quality of an MR still image at any time while reducing the amount of calculation required for generating the MR still image. Also, the image processing device 100 outputs RAW data as generated information, thereby further improving the quality of the MR still image.

[0076] In the second embodiment, the image processing device 100 has been described as outputting RAW data corresponding to the captured image data as generated information instead of the captured image data, but it may also output both the RAW data and the captured image data as generated information.

[0077] In the second embodiment, the image processing device 100 and the image generation device 120 are configured as different devices, but the present invention is not limited to this. For example, the image processing device 100 may have the image acquisition unit 101, the CG generation unit 102, the MR generation unit 103, the reception unit 104, and the output unit 105, as well as the acquisition unit 301, the CG generation unit 302, and the MR generation unit 303 that the image generation device 120 has.

[0078] [Other embodiments] The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit such as an ASIC that realizes one or more functions.

[0079] Furthermore, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0080] [Configuration of the present disclosure] [Configuration 1] image acquisition means for acquiring captured image data; a CG generating means for generating CG corresponding to an appearance of a virtual object placed in a virtual space from a virtual viewpoint based on rendering information indicating rendering conditions; an MR generating means for generating an MR moving image by superimposing the CG on the captured image; a receiving means for receiving a request to acquire data of an MR still image corresponding to a frame of the MR moving image; an output means for outputting generation information used when generating the MR still image; and the output means outputs, as the generation information, data of the captured image at the time of receiving the acquisition request, the rendering information at the time of receiving the acquisition request, and intermediate data generated when generating the CG, the intermediate data at the time of receiving the acquisition request; The intermediate data is information used in lighting processing when generating the CG. An image processing device comprising:

[0081] [Configuration 2] The intermediate data includes at least one of information indicating the color of each of one or more elements constituting the virtual object, information indicating the distance of each of the one or more elements from the virtual viewpoint, and information indicating the reflectance of each of the one or more elements. 2. The image processing device according to claim 1,

[0082] [Configuration 3] When the virtual object is configured by one or more polygons, the intermediate data includes at least one of information indicating the color of each of the one or more polygons that configure the virtual object, information indicating the distance of each of the one or more polygons from the virtual viewpoint, information indicating the reflectance of each of the one or more polygons, and information indicating the orientation of each of the one or more polygons. 3. The image processing device according to configuration 1 or 2, characterized in that:

[0083] [Configuration 4] The apparatus further includes a display control means for displaying the MR moving image on a display device, the display device is a head-mounted display device, The display control means controls the head-mounted display device to display the MR moving image. 4. The image processing device according to any one of configurations 1 to 3, characterized in that:

[0084] [Configuration 5] The output means outputs, in addition to the generation information, data of the frame of the MR moving image at the time of receiving the acquisition request. 5. The image processing device according to any one of configurations 1 to 4, characterized in that:

[0085] [Configuration 6] The rendering information includes at least viewpoint information indicating the position of the virtual viewpoint, and in addition to the viewpoint information, includes at least one of data of a shader code used as the rendering condition when the acquisition request is received and light source information used as the rendering condition when the acquisition request is received. 6. The image processing device according to any one of configurations 1 to 5, characterized in that:

[0086] [Configuration 7] the image acquisition means further acquires depth data indicating a depth of a space corresponding to at least a part of an imaging range in the captured image; the MR generating means generates the MR moving image using the captured image data and the depth data acquired by the image acquiring means, and the CG data generated by the CG generating means; The output means further outputs the depth data as the generation information. 7. The image processing device according to any one of configurations 1 to 6,

[0087] [Configuration 8] the image acquisition means further acquires RAW data corresponding to the captured image data, The output means outputs the RAW data at the time of receiving the acquisition request as the generation information, instead of the data of the captured image. 8. The image processing device according to any one of configurations 1 to 7, characterized in that:

[0088] [Configuration 9] the CG generating means operates as first CG generating means to generate a first CG; the MR generating means operates as a first MR generating means to generate the MR moving image; an acquisition means for acquiring the generation information output by the output means; a second CG generation means for generating a second CG different from the first CG by performing a lighting process using the intermediate data and the rendering information included in the generation information acquired by the acquisition means; a second MR generating means for generating the MR still image by superimposing the second CG on the captured image using data of the captured image included in the generation information acquired by the acquiring means and data of the second CG generated by the second CG generating means; Having 9. The image processing device according to any one of configurations 1 to 8,

[0089] [Configuration 10] the image acquisition means further acquires depth data indicating a depth of a space corresponding to at least a part of an imaging range in the captured image; the output means further outputs, as the generation information, the depth data at the time of receiving the acquisition request; The second MR generating means generates the MR still image using the captured image data and the depth data included in the generation information acquired by the acquiring means, and the second CG data. 10. The image processing device according to configuration 9,

[0090] [Configuration 11] the image acquisition means further acquires RAW data corresponding to the captured image data, the output means outputs the RAW data at the time of receiving the acquisition request, instead of the captured image data, as the generation information; The second MR generating means generates the MR still image by superimposing the second CG on an image obtained by developing the RAW data included in the generation information acquired by the acquiring means, instead of the captured image. 11. The image processing device according to configuration 9 or 10,

[0091] [Configuration 12] The second CG generating means generates the second CG by further performing post-processing using the intermediate data included in the generation information acquired by the acquiring means. 12. The image processing device according to any one of configurations 9 to 11,

[0092] [Configuration 13] an image acquisition step of acquiring captured image data; a CG generation step of generating CG corresponding to how a virtual object placed in a virtual space appears from a virtual viewpoint, based on rendering information indicating rendering conditions; an MR generating step of generating an MR moving image by superimposing the CG on the captured image; a receiving step of receiving a request to acquire data of an MR still image corresponding to a frame of the MR moving image; an output step of outputting generation information used in generating the MR still image; and the output step outputs, as the generation information, data of the captured image at the time of receiving the acquisition request, the rendering information at the time of receiving the acquisition request, and intermediate data generated when generating the CG, the intermediate data at the time of receiving the acquisition request; The intermediate data is information used in lighting processing when generating the CG. An image processing method comprising:

[0093] [Configuration 14] 13. A program for causing a computer to operate as the image processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0094] 100 Image processing device 104 Reception Department 105 Output section

Claims

1. image acquisition means for acquiring captured image data; a CG generating means for generating CG corresponding to an appearance of a virtual object placed in a virtual space from a virtual viewpoint based on rendering information indicating rendering conditions; an MR generating means for generating an MR moving image by superimposing the CG on the captured image; a receiving means for receiving a request to acquire data of an MR still image corresponding to a frame of the MR moving image; an output means for outputting generation information used when generating the MR still image; and the output means outputs, as the generation information, data of the captured image at the time of receiving the acquisition request, the rendering information at the time of receiving the acquisition request, and intermediate data generated when generating the CG, the intermediate data at the time of receiving the acquisition request; The intermediate data is information used in lighting processing when generating the CG. An image processing device characterized by:

2. The intermediate data includes at least one of information indicating the color of each of one or more elements constituting the virtual object, information indicating the distance of each of the one or more elements from the virtual viewpoint, and information indicating the reflectance of each of the one or more elements.

2. The image processing device according to claim 1, wherein:

3. When the virtual object is configured by one or more polygons, the intermediate data includes at least one of information indicating the color of each of the one or more polygons that configure the virtual object, information indicating the distance of each of the one or more polygons from the virtual viewpoint, information indicating the reflectance of each of the one or more polygons, and information indicating the orientation of each of the one or more polygons.

2. The image processing device according to claim 1, wherein:

4. further comprising a display control means for displaying the MR moving image on a display device; the display device is a head-mounted display device, The display control means controls the head-mounted display device to display the MR moving image.

2. The image processing device according to claim 1, wherein:

5. The output means outputs, in addition to the generated information, data of the frame of the MR moving image when the acquisition request is received.

2. The image processing device according to claim 1, wherein:

6. The rendering information includes at least viewpoint information indicating the position of the virtual viewpoint, and in addition to the viewpoint information, includes at least one of data of a shader code used as the rendering condition when the acquisition request is received and light source information used as the rendering condition when the acquisition request is received.

2. The image processing device according to claim 1, wherein:

7. the image acquisition means further acquires depth data indicating a depth of a space corresponding to at least a part of an imaging range in the captured image; the MR generating means generates the MR moving image using the captured image data and the depth data acquired by the image acquiring means, and the CG data generated by the CG generating means; The output means further outputs the depth data as the generation information.

2. The image processing device according to claim 1, wherein:

8. the image acquisition means further acquires RAW data corresponding to the captured image data, The output means outputs the RAW data at the time of receiving the acquisition request as the generation information, instead of the captured image data.

2. The image processing device according to claim 1, wherein:

9. the CG generating means operates as a first CG generating means to generate a first CG; the MR generating means operates as a first MR generating means to generate the MR motion image; an acquisition means for acquiring the generation information output by the output means; a second CG generation means for generating a second CG different from the first CG by performing a lighting process using the intermediate data and the rendering information included in the generation information acquired by the acquisition means; a second MR generating means for generating the MR still image by superimposing the second CG on the captured image using data of the captured image included in the generation information acquired by the acquiring means and data of the second CG generated by the second CG generating means; Having 2. The image processing device according to claim 1, wherein:

10. the image acquisition means further acquires depth data indicating a depth of a space corresponding to at least a part of an imaging range in the captured image; the output means further outputs, as the generation information, the depth data at the time of receiving the acquisition request; The second MR generating means generates the MR still image using the captured image data and the depth data included in the generation information acquired by the acquiring means, and the second CG data. The image processing device according to claim 9 ,

11. the image acquisition means further acquires RAW data corresponding to the captured image data, the output means outputs the RAW data at the time of receiving the acquisition request, instead of the captured image data, as the generation information; The second MR generating means generates the MR still image by superimposing the second CG on an image obtained by developing the RAW data included in the generation information acquired by the acquiring means, instead of the captured image. The image processing device according to claim 9 ,

12. The second CG generating means generates the second CG by further performing post-processing using the intermediate data included in the generation information acquired by the acquiring means. The image processing device according to claim 9 ,

13. an image acquisition step of acquiring captured image data; a CG generation step of generating CG corresponding to how a virtual object placed in a virtual space appears from a virtual viewpoint, based on rendering information indicating rendering conditions; an MR generating step of generating an MR moving image by superimposing the CG on the captured image; a receiving step of receiving a request to acquire data of an MR still image corresponding to a frame of the MR moving image; an output step of outputting generation information used in generating the MR still image; and the output step outputs, as the generation information, data of the captured image at the time of receiving the acquisition request, the rendering information at the time of receiving the acquisition request, and intermediate data generated when generating the CG, the intermediate data at the time of receiving the acquisition request; The intermediate data is information used in lighting processing when generating the CG. An image processing method comprising:

14. A program for causing a computer to operate as the image processing device according to any one of claims 1 to 12.

Images