Information processing apparatus and method for generating image data

Abstract

An image compositing unit (55) (image generation unit) of a decoding device (50a) (information processing device) generates a composite image (L) by compositing a real image (Ia, Ib) including a subject (90) (first object) captured by a real camera (70) (first image capturing device), a virtual image (V) (first virtual image), which is generated on the basis of a 3D model (90M) corresponding to the subject (90) and a specific virtual viewpoint (Vp) (viewpoint information), and a 3D CG object image (O) (second virtual image), which is generated on the basis of a 3D model (92) corresponding to a 3D CG object (second object) different from the subject (90) and the virtual viewpoint (Vp), in a superimposition ratio (r, r 3DCG) (composition ratio) according to the virtual viewpoint.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and a method for generating image data.

Background Art

[0002] Conventionally, a method has been proposed for generating a 3D object in a viewing space using information obtained by sensing a real 3D space, for example, multi-viewpoint images captured of a subject from different viewpoints, and generating an image (volumetric image) that appears as if the object exists in the viewing space.

[0003] Also, Patent Document 1 discloses an example of superimposing information related to a subject on a generated volumetric image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in Patent Document 1, what is superimposed on the subject is information related to the subject, and different images are not superimposed. Therefore, there has been a problem that it cannot be applied to the generation of an image that switches between a real image of a subject captured by an actual camera and a virtual image of the subject observed from a virtual viewpoint.

[0006] The present disclosure proposes an information processing apparatus and a method for generating image data that can switch between a real image and a virtual image captured by an actual camera.

Means for Solving the Problems

[0007] In order to solve the above problems, an information processing apparatus according to one aspect of the present disclosure includes a real image including a first object imaged by a first imaging device, a first virtual image generated based on the real image, a 3D model corresponding to the first object, and specific viewpoint information, and a second virtual image generated based on a 3D model corresponding to a second object different from the first object and the viewpoint information, and generates a composite image obtained by compositing the first virtual image and the second virtual image at a composite ratio according to the viewpoint information.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] Also, the present disclosure will be described in accordance with the following item order. 1. First Embodiment 1-1. Functional Configuration of Information Processing System 1-2. Processing Flow Performed by Information Processing System 1-3. Hardware Configuration of Information Processing System 1-4. Detailed Configuration of Rendering Unit 1-5. Operation of First Embodiment 1-6. Processing Flow of First Embodiment 1-7. Modification Example of First Embodiment 1-8. Effects of First Embodiment 2. Second Embodiment 2-1. Outline of Information Processing System 2-2. Functional Configuration of Information Processing System 2-3. Operation of Second Embodiment 2-4. Processing Flow of Second Embodiment 2-5. Effects of Second Embodiment 3. Application Examples of the Present Disclosure 3-1. Content Creation 3-2. Experience in Virtual Space 3-3. Application to Communication with Remote Locations 3-4. Others

[0011] (1. First Embodiment) [1-1. Functional Configuration of Information Processing System] First, the outline of the information processing system 10a to which the present disclosure is applied will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the schematic configuration of the information processing system according to the first embodiment.

[0012] As shown in FIG. 1, the information processing system 10a includes a data acquisition unit 11, a 3D model generation unit 12, an encoding unit 13a, a transmission unit 14, a reception unit 15, a decoding unit 16a, a rendering unit 17a, and a display unit 18.

[0013] The data acquisition unit 11 acquires image data for generating a 3D model 90M of a subject 90, which is an object to be imaged. For example, as shown in FIG. 2, a plurality of viewpoint images captured by a plurality of physical cameras 70 (70a, 70b, 70c, 70d, 70e,...) arranged so as to surround the subject 90 are acquired as image data. In this case, it is preferable that the plurality of viewpoint images are images captured by the plurality of physical cameras 70 synchronously. Further, the data acquisition unit 11 may acquire, for example, image data obtained by moving one physical camera 70 to image the subject 90 from a plurality of viewpoints. Note that the data acquisition unit 11 may perform calibration based on the image data to acquire the internal parameters and external parameters of each physical camera 70. Further, the data acquisition unit 11 may acquire, for example, a plurality of depth information indicating the distances from a plurality of viewpoints to the subject 90. Note that the physical cameras 70 (70a, 70b, 70c, 70d, 70e,...) are an example of the first imaging device in the present disclosure. Further, the subject 90 is an example of the first object in the present disclosure.

[0014] The 3D model generation unit 12 generates a model having three-dimensional information of the subject based on image data for generating a 3D model 90M of the subject 90. For example, the 3D model generation unit 12 generates a 3D model of the subject by cutting the three-dimensional shape of the subject using images from a plurality of viewpoints (for example, silhouette images from a plurality of viewpoints) using so-called Visual Hull. In this case, the 3D model generation unit 12 can further deform the 3D model 90M generated using Visual Hull with high precision using a plurality of depth information indicating the distances from the plurality of viewpoints to the subject. Further, the 3D model generation unit 12 may generate a 3D model 90M of the subject 90 from a single captured image of the subject 90.

[0015] The 3D model 90M generated by the 3D model generation unit 12 can also be said to be a video of the 3D model by generating it in units of time-series frames. Further, since the 3D model 90M is generated using an image captured by an imaging device, it can also be said to be a real-life 3D model. The 3D model can represent the shape information representing the surface shape of the subject 90 in the form of three-dimensional shape mesh data expressed by, for example, a connection between vertices (Vertex) called a polygon mesh. The three-dimensional shape mesh data has, for example, three-dimensional coordinates of the vertices of the mesh and index information indicating which vertices are combined to form a triangular mesh. Note that the method of expressing the 3D model is not limited to these, and it may be described in a so-called point cloud expression method expressed by point position information. Color information data is also generated as a texture in a form associated with these 3D shape data. There are cases of a View Independent texture that has a constant color no matter from which direction it is viewed and a View Dependent texture whose color changes depending on the viewing direction.

[0016] The symbolization unit 13a converts the data of the 3D model 90M generated by the 3D model generation unit 12 into a format suitable for transmission and storage. For example, the 3D model generated by the 3D model generation unit 12 may be converted into a plurality of two-dimensional images by perspective projection from a plurality of directions. In this case, depth information, which is a two-dimensional depth image from a plurality of viewpoints, may be generated using the 3D model. The depth information and color information in the state of the two-dimensional image are compressed and output to the transmission unit 14. The depth information and color information may be transmitted as one image arranged side by side, or may be transmitted as two separate images. In this case, since it is in the form of two-dimensional image data, it can also be compressed using a two-dimensional compression technique such as AVC (Advanced Video Coding). Also, for example, the 3D data may be converted into a point cloud format, or may be output to the transmission unit 14 as three-dimensional data. In this case, for example, a three-dimensional compression technique of the Geometry-based-Approach discussed in MPEG can be used.

[0017] The transmission unit 14 transmits the transmission data formed by the symbolization unit 13a to the reception unit 15. The transmission unit 14 transmits the transmission data to the reception unit 15 after performing a series of processes of the data acquisition unit 11, the 3D model generation unit 12, and the symbolization unit 13a offline. Also, the transmission unit 14 may transmit the transmission data generated from the above-described series of processes to the reception unit in real time.

[0018] Note that the 3D model generation unit 12, the symbolization unit 13a, and the transmission unit 14 constitute an encoding device 40a. The encoding device 40a is an example of the information processing device in the present disclosure.

[0019] The reception unit 15 receives the transmission data transmitted from the transmission unit 14.

[0020] The decoding unit 16a restores the bitstream received by the reception unit 15 into a two-dimensional image, and restores the mesh and texture information that can be drawn by the rendering unit 17a from the restored two-dimensional image.

[0021] Note that the receiving unit 15, the decoding unit 16a, and the rendering unit 17a constitute a decoding device 50a. The decoding device 50a is an example of the information processing device in the present disclosure.

[0022] The rendering unit 17a performs texture mapping in which it projects the mesh of the 3D model 90M from the viewpoint of the rendering camera and attaches a texture representing color and pattern. The feature of this embodiment is that the rendering at this time can be viewed from an arbitrarily set free viewpoint regardless of the camera position at the time of imaging. For texture mapping, there are so-called methods called View Dependent (VD method) that consider the user's viewing viewpoint and View Independent (VI method) that do not consider the user's viewing viewpoint. The VD method has the advantage that higher-quality rendering can be achieved than the VI method because it changes the texture attached to the 3D model according to the position of the viewing viewpoint. On the other hand, the VI method has the advantage that the processing amount can be less than that of the VD method because it does not consider the position of the viewing viewpoint. Note that the data of the viewing viewpoint is detected by the display device at the user's viewing location (Region of Interest) and input from the display device to the rendering unit 17a.

[0023] The display unit 18 displays the result rendered by the rendering unit 17a on the display unit of the display device. The display device may be, for example, a head-mounted display, a spatial display, a mobile phone, a television, a PC, etc., and may be a 2D monitor or a 3D monitor.

[0024] Note that the information processing system 10a in FIG. 1 shows a series of processes from the data acquisition unit 11 that acquires a captured image, which is material for generating content, to the display unit 18 that controls the display device viewed by the user. However, it does not mean that all functional blocks are required for the implementation of this embodiment, and this embodiment can be implemented for each functional block or a combination of multiple functional blocks. For example, FIG. 1 is provided with a transmission unit 14 and a reception unit 15 to show a series of processes from the side creating the content to the side viewing the content through the distribution of content data. However, when the production to viewing of the content is carried out by the same information processing device (for example, a personal computer), it is not necessary to provide the transmission unit 14 and the reception unit 15.

[0025] When implementing the information processing system 10a, in some cases, the same implementer may implement all of them, and different implementers may implement each functional block. For example, business operator X generates 3D content through the data acquisition unit 11, the 3D model generation unit 12, and the encoding unit 13a. Then, there may be cases where multiple implementers jointly implement such that the 3D content is distributed through the transmission unit 14 (platform) of business operator Y, and the display device of business operator Z receives, renders, and controls the display of the 3D content.

[0026] Also, each of the above-described functional blocks can be implemented on the cloud. For example, the rendering unit 17a may be implemented within the display device or on the server. In that case, information exchange occurs between the display device and the server.

[0027] FIG. 1 has described the data acquisition unit 11, the 3D model generation unit 12, the encoding unit 13a, the transmission unit 14, the reception unit 15, the rendering unit 17a, and the display unit 18 together as the information processing system 10a. However, the information processing system 10a in this specification is referred to as an information processing system if two or more functional blocks are involved. For example, excluding the display unit 18, the data acquisition unit 11, the 3D model generation unit 12, the encoding unit 13a, the transmission unit 14, the reception unit 15, the decoding unit 16a, and the rendering unit 17a can also be collectively referred to as the information processing system 10a.

[0028] [1-2. Flow of Processing Performed by Information Processing System] Next, with reference to FIG. 3, the flow of processing performed by the information processing system 10a will be described. FIG. 3 is a flowchart showing an example of the flow of processing performed by the information processing system.

[0029] When the processing starts, in step S11, the data acquisition unit 11 acquires image data for generating a 3D model of the subject.

[0030] Next, in step S12, the 3D model generation unit 12 generates a 3D model having three-dimensional information of the subject based on the image data for generating the 3D model of the subject.

[0031] In step S13, the encoding unit 13a encodes the shape and texture data of the 3D model generated by the 3D model generation unit 12 into a format suitable for transmission and storage.

[0032] Then, in step S14, the transmission unit 14 transmits the encoded data, and in step S15, the reception unit 15 receives the transmitted data.

[0033] In step S16, the decoding unit 16a performs decoding processing to convert the 3D model into shape and texture data necessary for display. Then, in step S17, the rendering unit 17a performs rendering using the shape and texture data. The rendered result is displayed on the display unit in step S18. Then, when the processing in step S18 ends, the processing of the information processing system 10a ends.

[0034] [1-3. Hardware Configuration of Information Processing System] Next, with reference to FIG. 4, the hardware configuration of the information processing system 10a will be described. FIG. 4 is a hardware block diagram showing an example of the hardware configuration of the information processing system.

[0035] In FIG. 4, the CPU 21, ROM 22, and RAM 23 are interconnected via a bus 24. An input / output interface 25 is also connected to the bus 24. An input unit 26, an output unit 27, a storage unit 28, a communication unit 29, and a drive 30 are connected to the input / output interface 25.

[0036] The input unit 26 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 27 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 28 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 29 includes, for example, a network interface, etc. The drive 30 drives a removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0037] The computer configured as described above performs the above-described series of processes by the CPU 21 loading and executing, for example, a program stored in the storage unit 28 into the RAM 23 via the input / output interface 25 and the bus 24. The RAM 23 also appropriately stores data and the like necessary for the CPU 21 to execute various processes.

[0038] The program executed by the computer can be recorded and applied, for example, on a removable medium such as a package medium. In that case, the program can be installed in the storage unit 28 via the input / output interface by mounting the removable medium on the drive.

[0039] Also, this program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 29 and installed in the storage unit 28.

[0040] [1-4. Detailed Configuration of the Rendering Unit] Next, the detailed configuration of the rendering unit 17a will be described with reference to FIG. 5. FIG. 5 is a functional block diagram showing an example of the functional configuration of the rendering unit.

[0041] As shown in FIG. 5, the rendering unit 17a includes a virtual image generation unit 51, a viewpoint setting unit 52, an overlap ratio calculation unit 53, a 3DCG object image generation unit 54, and an image composition unit 55.

[0042] The virtual image generation unit 51 acquires the real image captured by the real camera 70, the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 (see FIG. 1), and the position of the virtual viewpoint Vp set by the viewpoint setting unit 52, and generates a virtual image V (see FIG. 7) of the subject 90 viewed from the virtual viewpoint Vp (see FIG. 6). The 3D model 90M includes the shape information and texture information of the subject 90 respectively acquired by a plurality of real cameras 70. Note that the subject 90 is an example of an object in the present disclosure.

[0043] The virtual image V generated by the virtual image generation unit 51 is a so-called volumetric image obtained by projecting the 3D model 90M of the subject 90 onto the virtual viewpoint Vp. Note that the method for generating the virtual image V is not limited. For example, a 3D model 90M of the subject 90 generated by a method such as Visual Hull is generated using the real images respectively captured by a plurality of real cameras 70. Then, by projecting the generated 3D model 90M onto the virtual viewpoint Vp, a virtual image V of the subject 90 viewed from the virtual viewpoint Vp is generated. Note that the virtual image V is an example of the first virtual image in the present disclosure.

[0044] The viewpoint setting unit 52 sets the position of the virtual viewpoint Vp and the observation direction from the virtual viewpoint Vp toward the subject 90 (the first object) over time, that is, according to the time t. Note that the virtual viewpoint Vp is an example of the viewpoint information in the present disclosure.

[0045] The overlapping ratio calculation unit 53 receives the installation positions of the real cameras 70a and 70b, the real camera accuracy information, and the 3D model 90M of the subject 90 from the 3D model generation unit 12 (see FIG. 1). Then, when generating the composite image L (see FIG. 9) of the real image Ia, the virtual image V, the 3DCG object image O, and the real image Ib at the virtual viewpoint Vp set by the viewpoint setting unit 52, the overlapping ratio calculation unit 53 calculates the overlapping ratio r of the real image Ia, the virtual image V, the 3DCG object image O, and the real image Ib. Note that the real image Ia is an example of the first real image in the present disclosure. Also, the real image Ib is an example of the second real image in the present disclosure. The overlapping ratio r is an example of the composite ratio in the present disclosure. Further, the real camera accuracy information is calibration information related to the real cameras 70a and 70b (external calibration information related to the installation positions of the real cameras and internal parameter information related to the optical parameters of the real cameras), etc.

[0046] Note that the real images Ia and Ib (see FIG. 7) are images each including the subject 90 actually captured by the real cameras 70a and 70b. And the overlapping ratio r defines the overlapping ratio (ratio of luminance, ratio of color) of each image and the 3DCG object image O to be overlapped when the image composition unit 55 overlaps and composes the real image Ia, the virtual image V, the 3DCG object image O, and the real image Ib to generate the composite image L. Note that the overlapping ratio of the real image Ia is r ca (0 ≤ r ca ≤ 1), the overlapping ratio of the real image Ib is r cb (0 ≤ r cb ≤ 1), the overlapping ratio of the virtual image V is r V (0 ≤ r V ≤ 1), the overlapping ratio of the 3DCG object image O is r 3DCG (0 ≤ r 3DCG ≤ 1), when the overlapping ratios r ca , r cb , r V shall satisfy the conditions of Equation (1). Also, the overlapping ratio r is set according to the position of the virtual viewpoint Vp. Details will be described later.

[0047] r ca + r cb + r V=1···(1)

[0048] Further, the 3DCG object image O is synthesized by alpha blending with respect to an image in which the real image Ia, the virtual image V, and the real image Ib are superimposed. Details will be described later (see FIG. 7).

[0049] The 3DCG object image generation unit 54 generates a 3DCG object image O to be superimposed on the composite image L. Note that the 3DCG object image generation unit 54 may generate an image including a character having 3D information as the 3DCG object image O, or may generate a video effect to act on the entire composite image L. Specifically, the 3DCG object image generation unit 54 projects the 3D model 92 of the 3DCG object onto the virtual viewpoint Vp to generate a 3DCG object image O of the 3DCG object viewed from the virtual viewpoint Vp. Note that the 3DCG object image O is an example of the second virtual image in the present disclosure. Also, the 3DCG object is an example of the second object in the present disclosure.

[0050] The image composite unit 55 combines the real images Ia and Ib including the subject 90 having the 3D model 90M captured by the real camera 70 (first imaging device) from different positions, the virtual image V of the subject 90 observed from the virtual viewpoint Vp set by the viewpoint setting unit 52 according to the time t, and the 3DCG object image O having the 3D model 92 viewed from the virtual viewpoint Vp, with the superimposition ratios r and r calculated by the superimposition ratio calculation unit 53 according to the position of the virtual viewpoint Vp 3DCG to generate a composite image L. Note that the image composite unit 55 is an example of the image generation unit in the present disclosure.

[0051] [1-5. Operations of the First Embodiment] Next, with reference to FIGS. 6 to 9, the operations performed by the decoding device 50a of the first embodiment will be specifically described. FIG. 6 is a diagram showing an example of the arrangement of the real camera and the virtual camera. FIG. 7 is a diagram for explaining a method of superimposing a plurality of images. FIG. 8 is a diagram showing an example of setting the superimposition ratio according to the position of the virtual viewpoint. FIG. 9 is a diagram showing an example of the superimposition result of each image.

[0052] In the information processing system 10a, as shown in FIG. 6 for example, the viewpoint setting unit 52 performs a setting to move the virtual viewpoint Vp along a movement path 88a corresponding to the time t from the actual viewpoint position of the actual camera 70a to the actual viewpoint position of the actual camera 70b.

[0053] More specifically, while continuously capturing the subject 90 having the 3D model 90M, the virtual viewpoint Vp moves to the position of the virtual camera 72a at time t1. Then, thereafter, the virtual viewpoint Vp moves to the position of the virtual camera 72b at time t2, the position of the virtual camera 72c at time t3, the position of the virtual camera 72d at time t4, the position of the virtual camera 72e at time t5, and the position of the actual camera 70b at time t6.

[0054] The superimposition ratio calculation unit 53 calculates the superimposition ratios r of the actual images Ia, Ib, the virtual image V viewed from the virtual viewpoint Vp, and the 3DCG object image O viewed from the virtual viewpoint Vp in the composite image L based on the spatial distance (distance) between the actual camera 70a or the actual camera 70b and the virtual viewpoint Vp, the temporal distance from the time when the virtual viewpoint Vp is at the position of the actual camera 70a or the actual camera 70b, that is, the moving speed of the virtual viewpoint, etc.

[0055] The image composition unit 55 performs the process shown in FIG. 7 to generate the composite image L. Note that the superimposition ratio calculation unit 53 calculates the superimposition ratio r of the actual image Ia ca the superimposition ratio r of the actual image Ib cb the superimposition ratio r of the virtual image V V the superimposition ratio r of the 3DCG object image O 3DCG will be described as having been calculated.

[0056] As shown in FIG. 7, the image composition unit 55 includes multipliers M1, M2, M3, M4, M5, adders A1, A2, a 3D object superimposition unit 56, and a signal selection unit 57.

[0057] The multiplier M1 multiplies the actual image Ia by the superimposition ratio r cais multiplied. The multiplier M2 multiplies the real image Ib and the superimposition ratio r cb is multiplied. The multiplier M3 multiplies the virtual image V and the superimposition ratio r V is multiplied. The multiplier M4 multiplies the 3DCG object image O and the superimposition ratio r 3DCG is multiplied. The multiplier M5 multiplies the addition result of the output of the multiplier M1, the output of the multiplier M2, and the output of the multiplier M3, and the superimposition ratio (1 - r 3DCG ).

[0058] The adder A1 adds the output of the multiplier M1, the output of the multiplier M2, and the output of the multiplier M3.

[0059] The 3D object superimposition unit 56 generates a superimposed image Ix2 by further synthesizing the 3DCG object image O with respect to the superimposed image Ix1 obtained by superimposing the real image Ia, the real image Ib, and the virtual image V. The synthesis of the 3DCG object image O is performed, for example, by alpha blending. Alpha blending is one of the methods for synthesizing a plurality of images and is a method of overlapping images in consideration of the transmittance. That is, the superimposition ratio r 3DCG of the 3DCG object image O represents the transmittance of the 3DCG object image O.

[0060] The signal selection unit 57 is a selector that selects the superimposed image Ix1 (input A) obtained by superimposing the real image Ia, the real image Ib, and the virtual image V, and the superimposed image Ix2 (input B). The signal selection unit 57 selects the superimposed image Ix2 (input B) for the region where the 3DCG object exists. On the other hand, the signal selection unit 57 selects the superimposed image Ix1 (input A) for the region where the 3DCG object does not exist. Whether or not the 3DCG object exists is determined based on the 3DCG object existence map Mo. The 3DCG object existence map Mo is an image indicating the existence position of the 3DCG object. The 3DCG object existence map Mo is used to specify the region used for synthesis by the image synthesis unit 55 from the 3DCG object image O viewed from the virtual viewpoint Vp.

[0061] Then, the image synthesis unit 55 outputs the output of the signal selection unit 57 to the display unit 18 as a composite image L.

[0062] When the 3DCG object image O is an image that applies a video effect to the entire composite image L, the signal selection unit 57 selects the superimposed image Ix2 (input B) for the entire image.

[0063] Although not shown in FIG. 7, in order to realize a more complex superimposition, a foreground area map indicating the existence position of the subject 90 in the real images Ia and Ib may be used. Also, a depth map representing the distance to the subject 90 in the virtual image V may be used.

[0064] Next, with reference to FIG. 8, a setting example of the superimposition ratio r (r ca , r cb , r V , r 3DCG ) corresponding to the position of the virtual viewpoint Vp will be described.

[0065] First, the superimposition ratio r ca of the real image Ia will be described. The superimposition ratio calculation unit 53 gradually decreases the superimposition ratio r ca set to 100% at the position of the real camera 70a from around time t1, and at the same time gradually increases the superimposition ratio r V set to 0% at the position of the real camera 70a from around time t1. Then, when the superimposition ratio r ca reaches 0% at time t, the superimposition ratio r V is set to 100%.

[0066] Then, the superimposition ratio calculation unit 53 gradually decreases the superimposition ratio r V around time t4, and at the same time gradually increases the superimposition ratio r cb of the real image Ib. The superimposition ratio r V is set to 0% around time t5, and the superimposition ratio r cb is set to 100% around time t5. Note that the superimposition ratio r ca maintains 0% until time t6.

[0067] In this way, when the distance between the real camera 70a and the virtual viewpoint Vp is equal to or less than a predetermined value, and when the distance between the real camera 70b and the virtual viewpoint Vp is equal to or less than a predetermined value, the superimposition ratio r is lower than when the distance is greater than the predetermined value. V By doing so, when the virtual viewpoint Vp is in the vicinity of the real cameras 70a and 70b, the contribution degrees of the real images Ia and Ib in the composite image L are increased.

[0068] Qualitatively, when the virtual viewpoint Vp is located between the real camera 70a and the real camera 70b (near time t3 in FIG. 8), the superimposition ratio r of the virtual image V V is set to 100%. However, when the real cameras 70a and 70b are arranged in close proximity, since the superimposition ratio r of the virtual image V is estimated to be low, the superimposition ratio r V may not reach 100%. V

[0069] Also, the superimposition ratio calculation unit 53 sets the superimposition ratio r of the 3DCG object image O. 3DCG In the example of FIG. 8, the superimposition ratio calculation unit 53 sets the superimposition ratio r 3DCG to 100% from around time t1 to around time t5. And at times before and after that, the superimposition ratio r 3DCG is changed to 0%.

[0070] Also, the superimposition ratio calculation unit 53 may set the change rate of the superimposition ratio r (r ca , r cb , r V , r 3DCG ) according to the moving speed of the virtual viewpoint Vp. For example, when the moving speed of the virtual viewpoint Vp is slow, it is desirable to set the change rate of the superimposition ratio r (r ca , r cb , r V , r 3DCG ) per unit time to be small. On the other hand, when the moving speed of the virtual viewpoint Vp is fast, the change rate of the superimposition ratio r (r ca , r cb , r V , r 3DCGIt is desirable to set a large rate of change of (). This can make the change in the composite image L accompanying the movement of the virtual viewpoint Vp less noticeable.

[0071] Note that the setting pattern of the overlapping ratio r is not limited to the example in FIG. 8. Other setting patterns of the overlapping ratio r will be described later (see FIG. 11).

[0072] Further, the overlapping ratio calculation unit 53 may simultaneously control not only the overlapping ratio r of each image but also the color density of each image, the degree of blurring of each image, and the like.

[0073] Next, an example of the composite image generated by the image composite unit 55 will be described with reference to FIG. 9. FIG. 9 shows the real images Ia and Ib observed at each time from time t0 to time t6 shown in FIG. 6, the virtual image V generated from the real images Ia and Ib, and the 3DCG object image O representing the video effect at each time. FIG. 9 also shows the composite image L obtained by compositing these images at each time. Note that the background is shown in the real images Ia and Ib, but the virtual image V is generated only for the region of the subject 90 extracted from the real images Ia and Ib.

[0074] The virtual image V is generated based on the 3D model 90M of the subject 90 shown in the real images Ia and Ib. Therefore, although not fully shown in FIG. 9, strictly speaking, the virtual image V is an image in which the real images Ia and Ib overlap.

[0075] Also, in FIG. 9, the 3DCG object image O is a video effect that draws particles randomly pouring over the entire image.

[0076] The composite image L is generated based on the overlapping ratio r set by the overlapping ratio calculation unit 53. FIG. 9 shows an example in which an overlapping ratio r approximately equal to that in FIG. 8 is set.

[0077] [1-6. Flow of processing of the first embodiment] Next, with reference to FIG. 10, the flow of processing performed by the information processing system 10a will be described. FIG. 10 is a flowchart showing an example of the flow of processing performed by the information processing system according to the first embodiment.

[0078] The viewpoint setting unit 52 sets a virtual viewpoint Vp for each time (step S21).

[0079] The image synthesis unit 55 acquires the real image Ia captured by the real camera 70a and the real image Ib captured by the real camera 70b (step S22).

[0080] The virtual image generation unit 51 generates a virtual image V based on the real image Ia captured by the real camera 70a, the real image Ib captured by the real camera 70b, the 3D model 90M (shape information and texture information) of the subject 90 generated by the 3D model generation unit 12, and the position of the virtual viewpoint Vp set by the viewpoint setting unit 52 (step S23).

[0081] The 3DCG object image generation unit 54 generates a 3DCG object image O viewed from the virtual viewpoint Vp (step S24).

[0082] The superimposition ratio calculation unit 53 sets a superimposition ratio r (r ca , r cb , r V , r 3DCG ) (step S25).

[0083] The image synthesis unit 55 generates a composite image L (step S26).

[0084] The image synthesis unit 55 outputs the composite image L to the display unit 18 (step S27).

[0085] The viewpoint setting unit 52 determines whether the virtual viewpoint Vp has moved to the final position (step S28). If it is determined that the virtual viewpoint Vp has moved to the final position (step S28: Yes), the information processing system 10a ends the processing of FIG. 10. On the other hand, if it is determined that the virtual viewpoint Vp has not moved to the final position (step S28: No), the process returns to step S21.

[0086] Note that the above operation example is an example of generating the composite image L after converting the Volumetric video and the 3DCG object into two-dimensional images respectively. Of course, the composite image L may be generated by synthesizing the Volumetric video and the 3DCG object as 3D models and projecting the synthesized 3D model onto a two-dimensional image.

[0087] For example, the 3D model 90M of the subject 90 and the 3D model 92 of the 3DCG object may be three-dimensionally synthesized based on the three-dimensional position information of each 3D model and the three-dimensional position information of the virtual viewpoint Vp. Then, the composite image L may be generated by projecting the synthesized three-dimensional information onto a two-dimensional virtual image V observed from the virtual viewpoint Vp.

[0088] [1-7. Modification Example of the First Embodiment] The overlapping ratio r set by the overlapping ratio calculation unit 53 is not limited to the setting example shown in FIG. 8. Hereinafter, another setting example of the overlapping ratio r will be described with reference to FIG. 11. FIG. 11 is a diagram showing another setting example of the overlapping ratio according to the position of the virtual viewpoint.

[0089] The overlapping ratio r1 is a setting example of instantaneously switching between the real image Ia and the virtual image V, and between the virtual image V and the real image Ib at a predetermined time. That is, the overlapping ratio r of the real image Ia is switched from 100% to 0% at time ta. And the overlapping ratio r of the virtual image V is switched from 0% to 100% at time ta. Also, the overlapping ratio r of the real image Ib is switched from 0% to 100% at time tb. And the overlapping ratio r of the virtual image V is switched from 100% to 0% at time tb. Note that each waveform showing the overlapping ratio r1 shown in FIG. 11 is strictly a rectangular wave, but is drawn with a steep slope because it is difficult to see. ca of the real image Ia is switched from 100% to 0% at time ta. And the overlapping ratio r of the virtual image V is switched from 0% to 100% at time ta. Also, the overlapping ratio r of the real image Ib is switched from 0% to 100% at time tb. And the overlapping ratio r of the virtual image V is switched from 100% to 0% at time tb. Note that each waveform showing the overlapping ratio r1 shown in FIG. 11 is strictly a rectangular wave, but is drawn with a steep slope because it is difficult to see. V of the virtual image V is switched from 0% to 100% at time ta. Also, the overlapping ratio r of the real image Ib is switched from 0% to 100% at time tb. And the overlapping ratio r of the virtual image V is switched from 100% to 0% at time tb. Note that each waveform showing the overlapping ratio r1 shown in FIG. 11 is strictly a rectangular wave, but is drawn with a steep slope because it is difficult to see. cb of the real image Ib is switched from 0% to 100% at time tb. And the overlapping ratio r of the virtual image V is switched from 100% to 0% at time tb. Note that each waveform showing the overlapping ratio r1 shown in FIG. 11 is strictly a rectangular wave, but is drawn with a steep slope because it is difficult to see. V of the virtual image V is switched from 100% to 0% at time tb. Note that each waveform showing the overlapping ratio r1 shown in FIG. 11 is strictly a rectangular wave, but is drawn with a steep slope because it is difficult to see.

[0090] In this way, a cut (switching) effect can be obtained by instantaneously switching between two images. Note that the superimposition ratio r1 shown in FIG. 11 is also set to instantaneously switch whether or not to superimpose the 3DCG object image O on the superimposed image Ix1.

[0091] The superimposition ratio r2 is an example of changing the superimposition ratio r 3DCG of the real image Ia and the superimposition ratio r ca of the virtual image V in a state where the superimposition ratio r V of the 3DCG object image O is set to 100% (maximum value). That is, at the time tc when the superimposition ratio r 3DCG of the 3DCG object image O is set to 100%, the superimposition ratio r ca of the real image Ia gradually decreases from 100%. Also, the superimposition ratio r cb of the real image Ib gradually increases to 100% at the time td, and then the superimposition ratio r 3DCG of the 3DCG object image O gradually decreases from 100%.

[0092] By setting the superimposition ratio r2 in this way, when the superimposition ratios r2 of the real image Ia, the virtual image V, and the real image Ib change, the switching from the real image Ia to the virtual image V and the switching from the virtual image V to the real image Ib can be made less noticeable. And it has the effect of making the viewer of the composite image L recognize that the virtual image V and the real images Ia and Ib are in the same space.

[0093] The superimposition ratio r3 is a setting example in which the rising timing of the superimposition ratio r V of the virtual image V is delayed compared to the falling of the superimposition ratio r ca of the real image Ia, and the falling timing of the superimposition ratio r V is advanced compared to the rising of the superimposition ratio r cb of the real image Ib.

[0094] Specifically, from time te to time tf, the superimposition ratio r caAfter gradually decreasing from 100% to 0%, at time tf, the superimposition ratio r of the virtual image V V gradually increases from 0% to 100%. Then, the superimposition ratio r of the virtual image V V is gradually decreased to 0% at time tg, and then, from time tg to time th, the superimposition ratio r of the real image Ib cb gradually increases from 0% to 100%. Note that the superimposition ratio r of the 3DCG object image O 3DCG starts to gradually increase from 0% at time te, and after the superimposition ratio r 3DCG is set to 100%, it is gradually decreased to 0% at time th.

[0095] By setting the superimposition ratio r3 in this way, an image effect can be obtained in which the subject 90 disappears once and then the virtual image V emerges.

[0096] [1-8. Effects of the First Embodiment] As described above, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) includes real images Ia, Ib including the subject 90 (first object) captured by the real camera 70 (first imaging device), a virtual image V (first virtual image) generated based on the 3D model 90M corresponding to the subject 90, a specific virtual viewpoint Vp (viewpoint information), and a 3DCG object image O (second virtual image) generated based on the 3D model 92 corresponding to a 3DCG object (second object) different from the subject 90 and the virtual viewpoint Vp, and generates a composite image L synthesized at a superimposition ratio r, r 3DCG (synthesis ratio) according to the viewpoint information.

[0097] Thereby, the real images Ia, Ib captured by the real camera 70 and the virtual image V can be seamlessly switched.

[0098] Also, in the decoding device 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates the composite image L at a superimposition ratio r (synthesis ratio) according to the distance between the virtual viewpoint Vp and the real camera 70 (first imaging device).

[0099] As a result, by simple arithmetic processing, it is possible to set the superimposition ratio r (composition ratio) between the real images Ia and Ib captured by the real cameras 70a and 70b and the virtual image V.

[0100] Also, in the decoding device 50a (information processing device) of the first embodiment, when the distance between the real camera 70 (first imaging device) and the virtual viewpoint Vp is equal to or less than a predetermined value, the image synthesizing unit 55 (image generating unit) synthesizes the virtual image V at a lower superimposition ratio r V (composition ratio) than when the distance is greater than the predetermined value.

[0101] As a result, in the vicinity of the real cameras 70a and 70b, a higher-quality composite image L can be obtained by increasing the contribution degrees of the real images Ia and Ib in the composite image L.

[0102] Also, in the decoding device 50a (information processing device) of the first embodiment, the viewpoint setting unit 52 sets the virtual viewpoint Vp, which is viewpoint information, over time.

[0103] As a result, by moving the virtual viewpoint Vp, free viewpoint movement can be realized.

[0104] Also, in the decoding device 50a (information processing device) of the first embodiment, the viewpoint information is the position of the virtual viewpoint Vp and the observation direction at the virtual viewpoint Vp.

[0105] As a result, it is possible to move the virtual viewpoint Vp while capturing the subject 90.

[0106] Also, in the decoding device 50a (information processing device) of the first embodiment, the image synthesizing unit 55 (image generating unit) generates the composite image L at a superimposition ratio r (composition ratio) corresponding to the moving speed of the virtual viewpoint Vp.

[0107] As a result, it is possible to make the switching between the real images Ia and Ib and the virtual image V less noticeable.

[0108] Also, in the decoder 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) uses the real image Ia (first real image) captured by two different real cameras 70a (first imaging devices), the real image Ib (second real image) captured by the real camera 70b (first imaging device), the virtual image V (first virtual image) generated by projecting the 3D model 90M of the subject 90 (first object) onto a virtual viewpoint Vp located between the two real cameras 70a and 70b, and the 3DCG object image O (second virtual image) generated from the 3DCG object (second object) to generate a composite image L.

[0109] Thereby, the real image Ia captured by the real camera 70a, the real image Ib captured by the real camera 70b, and the virtual image V can be seamlessly switched.

[0110] Also, in the decoder 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) generates the composite image L by switching the superimposition ratio r (synthesis ratio) between the minimum value (0%) and the maximum value (100%) at a predetermined time.

[0111] Thereby, since two images can be instantaneously switched, a cut (switching) effect can be obtained.

[0112] Also, in the decoder 50a (information processing device) of the first embodiment, the image synthesis unit 55 (image generation unit) changes the superimposition ratio r 3DCG (synthesis ratio) of the real images Ia and Ib and the virtual image V in a state where the superimposition ratio r (synthesis ratio) of the 3DCG object image O (CG object) is set to the maximum value (100%) to generate the composite image L.

[0113] Thereby, the transition from the real image Ia to the virtual image V and the transition from the virtual image V to the real image Ib can be made less noticeable, so that the viewer of the composite image L can be made to recognize that the virtual image V and the real images Ia and Ib are in the same space.

[0114] Also, in the decoder 50a (information processing device) of the first embodiment, the image synthesizing unit 55 (image generating unit) adjusts the timing of changing the superimposition ratio r V of the virtual image V to be later than the falling edge timing of the superimposition ratio r ca of the real image Ia, or earlier than the rising edge timing of the superimposition ratio r cb of the real image Ib to generate the composite image L.

[0115] As a result, an image effect can be obtained in which the subject 90 disappears once and then the virtual image V floats up.

[0116] Also, in the decoder 50a (information processing device) of the first embodiment, the superimposition ratio r 3DCG (synthesis ratio) of the 3DCG object image O (CG object) is the transmittance when superimposing the real images Ia and V and the 3DCG object image O, or the real images Ib and V and the 3DCG object image O.

[0117] As a result, the 3DCG object image O can be synthesized at an arbitrary transmittance, so the range of expression of the composite image L can be widened.

[0118] Also, in the decoder 50a (information processing device) of the first embodiment, the 3DCG object image O (CG object) includes a video effect having 3D information.

[0119] As a result, when switching between the real images Ia and Ib and the virtual image V, by imparting a video effect, the sense of strangeness of the switch can be reduced.

[0120] Also, in the decoder 50a (information processing device) of the first embodiment, the 3DCG object image O (CG object) includes a character.

[0121] As a result, since the composite image L can have a character superimposed thereon, the range of expression of the composite image L can be widened.

[0122] Also, in the decoding device 50a (information processing device) of the first embodiment, the observation direction is the direction from the virtual viewpoint Vp toward the subject 90 (object).

[0123] As a result, the real images Ia, Ib and the virtual image V can be switched while always capturing the subject 90.

[0124] Also, in the decoding device 50a (information processing device) of the first embodiment, an image synthesizing unit 55 (image generating unit) synthesizes real images Ia, Ib including the subject 90 (first object) imaged by a real camera 70 (first imaging device), a 3D model 90M corresponding to the subject 90, and a 3D model 92 corresponding to a 3DCG object (second object) different from the subject 90 at superimposition ratios r, r 3DCG (synthesis ratio) to generate a composite image L.

[0125] As a result, the real images Ia, Ib imaged by the real camera 70 and the virtual image V can be seamlessly switched.

[0126] (2. Second Embodiment) The information processing system 10a (see FIG. 1) described in the first embodiment is configured using only a real camera 70 installed, for example, at a live venue or the like as an imaging device. Therefore, since the imaging position is limited, the images (real images Ia, Ib, virtual image V) capturing the subject 90 are uniform. The information processing system 10b (see FIG. 12) of the second embodiment enhances the production effect even further by expanding the variations of the real images capturing the subject 90 by adding, for example, a real camera provided in a mobile terminal held by a spectator to the configuration of the information processing system 10a.

[0127] [2-1. Outline of Information Processing System] First, the outline of the information processing system 10b will be described with reference to FIG. 12. FIG. 12 is a diagram for explaining the outline of the information processing system according to the second embodiment.

[0128] In addition to the real cameras 70 (70a, 70b) installed at a live venue or the like that image the subject 90, the information processing system 10b includes real cameras 76 (76a, 76b,...) built into a plurality of mobile terminals 74 (74a, 74b,...) held by the audience at the live venue. The real camera 76a images a real image Ic including the subject 90. Also, the real camera 76b images a real image Id including the subject 90. The real camera 76 is a real camera different from the above-described real camera 70 (the first imaging device). That is, while the real images Ia and Ib captured by the real camera 70 are used for generating the 3D model 90M of the subject 90, the real images Ic and Id captured by the real camera 76 are not used for generating the 3D model 90M of the subject 90. Note that the real camera 76 (76a, 76b,...) is an example of the second imaging device in the present disclosure.

[0129] Then, the viewpoint setting unit 52 (viewpoint setting unit) of the information processing system 10b acquires the current position of the real camera 76 and sets a virtual viewpoint Vp based on the position of the real camera 70. Thereby, the viewpoint setting unit 52 sets, for example, a movement path for moving the virtual viewpoint Vp to the current position of the real camera 76.

[0130] Then, the information processing system 10b moves the virtual viewpoint Vp gradually away from a position near the real camera 70 to the position of the real camera 76 (76a, 76b,...) built into the mobile terminal 74 (74a, 74b,...) held by the audience.

[0131] At this time, while moving the virtual viewpoint Vp, the information processing system 10b generates a composite image L of the virtual image V (two-dimensional image) generated from the Volumetric video and the real image captured by the real camera 76 at the position of the moving virtual viewpoint Vp. Then, the information processing system 10b causes the generated composite image L to be displayed on the mobile terminal 74 held by the audience.

[0132] Note that since the real cameras 76 (76a, 76b, …) built into the mobile terminals 74 (74a, 74b, …) held by the audience are not used for generating the 3D model 90M of the subject 90, the 3D model 90M of the subject 90 uses the one generated by the real camera 70.

[0133] As a result, for example, the audience holding the mobile terminals 74a, 74b can experience the situation where the video of the subject 90 captured by the real camera 70 switches to the real images Ic, Id captured by the real cameras 76a, 76b of their own mobile terminals 74a, 74b over time.

[0134] Note that the real cameras 76 (76a, 76b, …) are not limited to the cameras built into the mobile terminal 74. That is, the real cameras 76 (76a, 76b, …) may be fixed cameras or movable cameras provided far from the subject 90 at the live venue.

[0135] The movement path of the virtual viewpoint Vp can be set arbitrarily. However, in the example shown in FIG. 12, the virtual viewpoint Vp is moved from the position of the real camera 70a at time t0 to the position of the real camera 70b at time t6, and then to the position of the real camera 76a at time t10 (movement path 88b). Also, the virtual viewpoint Vp is moved from the position of the real camera 70a at time t0 to the position of the virtual camera 72c at time t3, and then to the position of the real camera 76b at time t14 (movement path 88c). In this way, the information processing system 10b sets the movement path of the virtual viewpoint Vp based on the position of the mobile terminal 74 (74a, 74b, …) for a plurality of mobile terminals 74 (74a, 74b, …).

[0136] [2-2. Functional Configuration of Information Processing System] The information processing system 10b includes an encoding device 40a described in the first embodiment and a decoding device 50b (not shown) that replaces the decoding device 50a. Further, the information processing system 10b includes the above-described mobile terminal 74 (74a, 74b,...). Note that the decoding device 50b is an example of the information processing device in the present disclosure.

[0137] The decoding device 50b is different from the decoding device 50a in that it has a function of receiving an image captured by the real camera 76 built in the mobile terminal 74, a function of receiving the current position of the mobile terminal 74, and a function of transmitting the composite image L to the mobile terminal 74. Accordingly, the decoding device 50b includes a rendering unit 17b (not shown) instead of the rendering unit 17a. The rendering unit 17b has a function of receiving an image captured by the real camera 76, a function of receiving the current position of the mobile terminal 74, and a function of transmitting the composite image L to the mobile terminal 74, with respect to the rendering unit 17a. Note that since the functional block configuration of the rendering unit 17b is the same as that of the rendering unit 17a, illustration thereof is omitted, and the reference numerals of the respective functional blocks are described using the same ones as in FIG. 5. That is, the image synthesis unit 55 and the virtual image generation unit 51 receive an image captured by the real camera 76. Also, the superimposition ratio calculation unit 53 receives the current position of the mobile terminal 74. Also, the image synthesis unit 55 transmits the composite image L to the mobile terminal 74.

[0138] The mobile terminal 74 has the functional configuration shown in FIG. 13. FIG. 13 is a functional block diagram showing an example of the functional configuration of the mobile terminal.

[0139] The mobile terminal 74 (74a, 74b,...) includes an operation control unit 75 having a computer configuration. The operation control unit 75 includes an imaging control unit 80, a positioning unit 81, an application control unit 82, a display control unit 83, an operation control unit 84, and a communication control unit 85.

[0140] The imaging control unit 80 controls the imaging operation of the real camera 76 (76a, 76b,...) built in the mobile terminal 74 (74a, 74b,...).

[0141] The positioning unit 81 detects the current position of the mobile terminal 74 (74a, 74b, …) by, for example, a GPS receiver 77 built in the mobile terminal 74. Note that the positioning unit 81 may detect the current position of the mobile terminal 74 (74a, 74b, …) by functions other than the GPS receiver 77, for example, Wi-Fi (registered trademark) positioning, Bluetooth (registered trademark) positioning, etc. Further, the positioning unit 81 may detect the position and orientation of the mobile terminal 74 (74a, 74b, …) based on an image of a marker installed at the venue captured by the actual camera 76 (76a, 76b, …).

[0142] The application control unit 82 controls the operation of a video display application that operates on the mobile terminal 74 (74a, 74b, …).

[0143] The display control unit 83 controls the display state of the display 78 provided in the mobile terminal 74 (74a, 74b, …).

[0144] The operation control unit 84 receives an operation signal of an operation device such as a touch panel 79 provided in the mobile terminal 74 (74a, 74b, …) and recognizes the operation content.

[0145] The communication control unit 85 controls the communication state between the mobile terminal 74 (74a, 74b, …) and the decoding device 50b.

[0146] [2-3. Operation of the Second Embodiment] Next, the operation of the information processing system 10b of the second embodiment will be described with reference to FIG. 14. FIG. 14 is a diagram showing an example of setting the superimposition ratio according to the position of the virtual viewpoint in the second embodiment. Note that, for simplicity of explanation, FIG. 14 shows only the superimposition ratios of the real images Ia, Ib, Ic, Id and the virtual image V, and the superimposition ratio of the 3DCG object image O is omitted.

[0147] When the viewpoint setting unit 52 makes a setting to move the virtual viewpoint Vp along the movement path 88b, the superimposition ratio calculation unit 53 sets the superimposition ratio r4 shown in FIG. 14.

[0148] The superimposition ratio calculation unit 53 sets the superimposition ratio r of the real image Ia set to 100% ca to gradually decrease from time t1 (the position of the virtual camera 72a) and become 0% before reaching time t3 (the position of the virtual camera 72c). Then, until reaching time t10 (the position of the real camera 76a), r ca = 0% is maintained.

[0149] Also, the superimposition ratio calculation unit 53 sets the superimposition ratio r of the virtual image V set to 0% V 1 to gradually increase from near time t1 (the position of the virtual camera 72a) and become 100% before reaching time t3 (the position of the virtual camera 72c). Then, the superimposition ratio calculation unit 53 sets the superimposition ratio r V 1 of the virtual image V to gradually decrease to 0% from time t4 (the position of the virtual camera 72d) to time t5 (the position of the virtual camera 72e). Note that the superimposition ratio r V 1 is the superimposition ratio of the virtual image V generated based on the real image Ia captured by the real camera 70a and the real image Ib captured by the real camera 70b.

[0150] Also, the superimposition ratio calculation unit 53 sets the superimposition ratio r of the real image Ib set to 0% cb to gradually increase from near time t4 (the position of the virtual camera 72d) and become 100% at time t5 (the position of the virtual camera 72e). Then, the superimposition ratio calculation unit 53 sets the superimposition ratio r cb of the real image Ib to gradually decrease from time t7 (the position of the virtual camera 72f) and set it to 0% before reaching time t8 (the position of the virtual camera 72g).

[0151] Furthermore, the superimposition ratio calculation unit 53 sets the superimposition ratio r of the virtual image V set to 0% V 2 to gradually increase from near time t7 (the position of the virtual camera 72f) and become 100% before reaching time t8 (the position of the virtual camera 72g). Then, the superimposition ratio calculation unit 53 sets the superimposition ratio r V 2 of the virtual image V to gradually decrease to 0% from time t9 (the position of the virtual camera 72h) to time t10 (the position of the real camera 76a). Note that the superimposition ratio rV 2 is the superimposition ratio of the virtual image V generated based on the real image Ib captured by the real camera 70b and the real image Ic captured by the real camera 76a.

[0152] Then, the superimposition ratio calculation unit 53 sets the superimposition ratio r of the real image Ic set to 0% cc to gradually increase from near the time t9 (position of the virtual camera 72h) and reach 100% at the time t10 (position of the real camera 76a).

[0153] Note that in the vicinity of the real camera 76a, for example, from the time t9 to the time t10, the superimposition ratio r V 2 is gradually decreased (changed) gently. This is because the real image Ic captured by the real camera 76a, that is, the second imaging device, has low calibration and synchronization accuracy with respect to the real images Ia and Ib captured by the real cameras 70 (70a, 70b), that is, the first imaging device. That is, when switching the virtual image V to the real image Ic, there may be a positional deviation of the subject 90, and this is to make this positional deviation less noticeable.

[0154] Also, when the viewpoint setting unit 52 makes a setting to move the virtual viewpoint Vp along the movement path 88c, the superimposition ratio calculation unit 53 sets the superimposition ratio r5 shown in FIG. 14.

[0155] The superimposition ratio calculation unit 53 sets the superimposition ratio r of the real image Ia set to 100% ca to gradually decrease from the time t1 (position of the virtual camera 72a) and become 0% before reaching the time t3 (position of the virtual camera 72c). And until the time t10 (position of the real camera 76a) is reached, r ca = 0% is maintained.

[0156] Also, the superimposition ratio calculation unit 53 sets the superimposition ratio r of the virtual image V set to 0% V 1 to gradually increase from near the time t1 (position of the virtual camera 72a) and become 100% between the time t2 (position of the virtual camera 72b) and the time t3 (position of the virtual camera 72c). And the superimposition ratio calculation unit 53 calculates the superimposition ratio r of the virtual image VV Set 1 to gradually decrease to 0% between time t3 (the position of the virtual camera 72c) and time t11 (the position of the virtual camera 72i). Note that the superimposition ratio r V 1 is the superimposition ratio of the virtual image V generated based on the real image Ia captured by the real camera 70a and the real image Ib captured by the real camera 70b.

[0157] Furthermore, the superimposition ratio calculation unit 53 sets the superimposition ratio r V of the virtual image V, which is set to 0%, to gradually increase from near time t3 (the position of the virtual camera 72c) and reach 100% before reaching time t11 (the position of the virtual camera 72i). Then, the superimposition ratio calculation unit 53 sets the superimposition ratio r V 2 of the virtual image V to gradually decrease to 0% from time t13 (the position of the virtual camera 72k) to time t14 (the position of the real camera 76b). Note that the superimposition ratio r V 2 is the superimposition ratio of the virtual image V generated based on the virtual image V generated at time t3 (the position of the virtual camera 72c) and the real image Id captured by the real camera 76b.

[0158] Then, the superimposition ratio calculation unit 53 sets the superimposition ratio r cd of the real image Id, which is set to 0%, to gradually increase from near time t13 (the position of the virtual camera 72k) and reach 100% at time t14 (the position of the real camera 76b).

[0159] Note that in the vicinity of the real camera 76b, for example, from time t13 to time t14, the superimposition ratio r V 2 is gently gradually decreased (changed). This is because the calibration and synchronization accuracy of the real image Id captured by the real camera 76b, that is, the second imaging device, is low compared to the real images Ia and Ib captured by the real cameras 70 (70a, 70b), that is, the first imaging device. That is, when switching the virtual image V to the real image Id, there may be a positional deviation of the subject 90, and this is to make this positional deviation less noticeable.

[0160] In addition, the overlap ratio calculation unit 53 sets the overlap ratio r of the real images Ic and Id according to the distance between the virtual viewpoint Vp and the real cameras 76 (76a, 76b). cc ,r cd (synthesis ratio). For example, when the virtual viewpoint Vp is closer to the real cameras 76 (76a, 76b) than a predetermined distance, the overlap ratio r of the real images Ic and Id captured by the real camera 76 cc ,r cd is set higher than the overlap ratio r of the real images Ic and Id captured by the real camera 76 when the virtual viewpoint Vp is farther from the real camera 76 than a predetermined distance. cc ,r cd It may be set higher.

[0161] In addition, the overlap ratio calculation unit 53 may simultaneously control not only the overlap ratio r of each image but also the color density of each image, the degree of blurring of each image, and the like.

[0162] [2-4. Flow of processing in the second embodiment] Next, with reference to FIG. 15, the flow of processing performed by the information processing system 10b will be described. FIG. 15 is a flowchart showing an example of the flow of processing performed by the information processing system according to the second embodiment. As an example, the operation in the state where the movement path 88b shown in FIG. 12 is set will be described.

[0163] First, the flow of processing performed by the rendering unit 17b (not shown) will be described. After the information processing system 10b starts an application that operates in the second embodiment, the rendering unit 17b determines whether it has received the position information of the mobile terminal 74a (step S31). If it is determined that the position information of the mobile terminal 74a has been received (step S31: Yes), the process proceeds to step S32. On the other hand, if it is determined that the position information of the mobile terminal 74a has not been received (step S31: No), the determination in step S31 is repeated.

[0164] When it is determined Yes in step S31, the viewpoint setting unit 52 sets the virtual viewpoint Vp for each time (step S32).

[0165] The image synthesis unit 55 acquires the real image Ia captured by the real camera 70a, the real image Ib captured by the real camera 70b, and the real image Ic captured by the mobile terminal 74a (real camera 76a) (step S33).

[0166] The virtual image generation unit 51 generates a virtual image V based on the real image Ia captured by the real camera 70a, the real image Ib captured by the real camera 70b, the 3D model 90M (shape information and texture information) of the subject 90 generated by the 3D model generation unit 12, and the position of the virtual viewpoint Vp set by the viewpoint setting unit 52. Also, the virtual image generation unit 51 generates a virtual image V based on the real image Ib captured by the real camera 70b, the real image Ib captured by the real camera 70b, the real image Ic captured by the mobile terminal 74a (real camera 76a), the 3D model 90M (shape information and texture information) of the subject 90 generated by the 3D model generation unit 12, and the position of the virtual viewpoint Vp set by the viewpoint setting unit 52. (Step S34).

[0167] The superimposition ratio calculation unit 53 sets the superimposition ratio r (r ca , r cb , r V 1, r V 2) (step S35).

[0168] The image synthesis unit 55 generates a composite image L (step S36).

[0169] The image synthesis unit 55 transmits the composite image L and the position of the virtual viewpoint Vp to the mobile terminal 74a (step S37).

[0170] The rendering unit 17b determines whether the virtual viewpoint Vp has been moved to the position of the mobile terminal 74a (step S38). If it is determined that the virtual viewpoint Vp has been moved to the position of the mobile terminal 74a (step S38: Yes), the rendering unit 17b ends the process of FIG. 15. On the other hand, if it is not determined that the virtual viewpoint Vp has been moved to the position of the mobile terminal 74a (step S38: No), the process returns to step S33.

[0171] Next, the process flow performed by the mobile terminal 74a will be described. The application control unit 82 determines whether an application for operating the second embodiment has been launched based on the operation signal detected by the operation control unit 84 (step S41). If it is determined that the application has been launched (step S41: Yes), the process proceeds to step S42. On the other hand, if it is determined that the application has not been launched (step S41: No), the determination in step S41 is repeated.

[0172] When it is determined Yes in step S41, the communication control unit 85 transmits the current position of the mobile terminal 74a calculated by the positioning unit 81 to the rendering unit 17b (step S42).

[0173] The communication control unit 85 transmits the real image Ic captured by the real camera 76a by the imaging control unit 80 to the rendering unit 17b (step S43).

[0174] The communication control unit 85 receives the composite image L and the position of the virtual viewpoint Vp from the rendering unit 17b (step S44).

[0175] The display control unit 83 displays the composite image L on the display 78 (step S45).

[0176] The application control unit 82 determines whether the virtual viewpoint Vp coincides with the position of the mobile terminal 74a (step S46). If it is determined that the virtual viewpoint Vp coincides with the position of the mobile terminal 74a (step S46: Yes), the rendering unit 17b ends the process of FIG. 15. On the other hand, if it is determined that the virtual viewpoint Vp does not coincide with the position of the mobile terminal 74a (step S46: No), the process returns to step S43.

[0177] Note that in FIG. 15, it has been described that the application is launched on the mobile terminal 74a side, but an image may be transmitted from the decoder device 50b side to the customer's mobile terminal 74 as an effect in a certain scene.

[0178] [2-5. Effects of the Second Embodiment] As described above, in the decoding device 50b (information processing device) of the second embodiment, the viewpoint setting unit 52 sets the position of the virtual viewpoint Vp based on the positions of the actual cameras 76 (76a, 76b) (second imaging devices) different from the plurality of actual cameras 70 (first imaging devices) used for generating the 3D model 90M of the subject 90 (object).

[0179] Thereby, since the actual images Ic and Id captured by the actual cameras 76 (76a, 76b) (second imaging devices) not used for generating the 3D model 90M of the subject 90 can be used as materials for the composite image L, the range of expression of the composite image L can be widened.

[0180] Also, in the decoding device 50b (information processing device) of the second embodiment, the image compositing unit 55 (image generation unit) determines the overlapping ratio r cc ,r cd (composite ratio) of the actual images Ic and Id captured by the actual camera 76 when the virtual viewpoint Vp is closer to the actual camera 76 (76a, 76b) (second imaging device) than a predetermined distance, and the overlapping ratio r cc ,r cd of the actual images Ic and Id captured by the actual camera 76 when the virtual viewpoint Vp is farther from the actual camera 76 than a predetermined distance, and sets the former higher than the latter.

[0181] Thereby, as the virtual viewpoint Vp approaches the actual cameras 76 (76a, 76b), the overlapping ratio r cc ,r cd of the actual images Ic and Id is set higher, so that the switching from the virtual image V to the actual images Ic and Id can be made less noticeable.

[0182] Also, in the decoding device 50b (information processing device) of the second embodiment, the image compositing unit 55 (image generation unit) determines the overlapping ratio r cc ,r cd(Combination ratio) is made to change more gently than the overlapping ratios r ca , r cb (Combination ratio) of the real images Ia and Ib captured by the real cameras 70(70a, 70b) when the position of the virtual viewpoint Vp approaches the real cameras 70(70a, 70b) (first imaging devices).

[0183] This makes it possible to make the switch from the virtual image V to the real images Ic and Id less noticeable when using the real images Ic and Id captured by the real cameras 76(76a, 76b) with poor calibration and synchronization accuracy.

[0184] Also, in the decoder device 50b (information processing device) of the second embodiment, the second imaging device is provided in the mobile terminal 74(74a, 74b).

[0185] This enables the real images Ic and Id captured by the real cameras 76(76a, 76b) (second imaging devices) provided in the mobile terminal 74(74a, 74b) to be transmitted to the decoder device 50b (information processing device), and a customer possessing the mobile terminal 74(74a, 74b) can view the composite image L transmitted from the decoder device 50b (information processing device). That is, a two-way service can be realized between the decoder device 50b (information processing device) and the mobile terminal 74(74a, 74b).

[0186] (3. Application Examples of the Present Disclosure) [3-1. Production of Content] For example, a video content may be produced by synthesizing the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 and a 3D model managed by another server. Also, for example, when there is background data acquired by an imaging device such as Lidar, by combining the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 and the background data, content can be produced such that the subject 90 appears to be at the location indicated by the background data. Note that the video content may be three-dimensional video content or two-dimensional video content converted into two dimensions. The 3D model 90M of the subject 90 generated in the present embodiment is, for example, a 3D model generated by the 3D model generation unit 12 or a 3D model reconstructed by the rendering units 17a and 17b, etc.

[0187] [3-2. Experience in Virtual Space] For example, the subject 90 generated in the present embodiment can be placed in a virtual space which is a place where a user communicates as an avatar. In this case, the user can view the real-life subject 90 in the virtual space as an avatar.

[0188] [3-3. Application to Communication with Remote Locations] For example, by transmitting the 3D model 90M of the subject 90 generated by the 3D model generation unit 12 from the transmission unit 14 to a remote location, a user at the remote location can view the 3D model 90M of the subject 90 through a playback device at the remote location. For example, by transmitting the 3D model 90M of the subject 90 in real time, the subject 90 and the user at the remote location can communicate in real time. For example, it can be assumed that the subject 90 is a teacher and the user is a student, or the subject 90 is a doctor and the user is a patient, etc.

[0189] [3-4. Others] For example, based on the 3D models 90M of a plurality of subjects 90 generated by the 3D model generation unit 12, it is also possible to generate a free viewpoint video such as a sports video. In addition, an individual can also distribute himself / herself generated by the 3D model generation unit 12 to a distribution platform. Thus, the content in the embodiments described in this specification can be applied to various technologies and services.

[0190] As described above, the present disclosure has been described using several embodiments, but these embodiments may be executed in any device. In that case, it is sufficient that the device has necessary functional blocks and can obtain necessary information.

[0191] Also, for example, each step of a flowchart may be executed by one device, or may be executed by a plurality of devices in cooperation. Further, when a plurality of processes are included in one step, the plurality of processes may be executed by one device, or may be executed by a plurality of devices in cooperation. In other words, the plurality of processes included in one step can also be executed as processes of a plurality of steps. Conversely, the processes described as a plurality of steps can also be executed together as one step.

[0192] Also, for example, the program executed by a computer may be such that the processes of the steps of describing the program are executed in time series along the order described in this specification, or may be executed in parallel, or individually executed at necessary timings such as when a call is made. That is, as long as there is no contradiction, the processes of each step may be executed in an order different from the order described above. Further, the processes of the steps of describing the program may be executed in parallel with the processes of other programs, or may be executed in combination with the processes of other programs.

[0193] In addition, for example, a plurality of technologies related to the present technology can be implemented independently and individually as long as there is no conflict. Of course, any plurality of the present technologies can also be applied and implemented. For example, part or all of the present technology described in any one embodiment can also be implemented in combination with part or all of the present technology described in other embodiments. Further, part or all of any of the above-described present technologies can also be implemented in combination with other technologies not described above.

[0194] Note that the effects described in this specification are merely illustrative and not limiting, and there may be other effects. Also, the embodiments of the present disclosure are not limited to the above-described embodiments, and various changes can be made without departing from the gist of the present disclosure.

[0195] For example, the present disclosure can have the following configurations.

[0196] (1) An information processing apparatus including an image generation unit that generates a composite image by combining a real image including a first object imaged by a first imaging device, a first virtual image generated based on a 3D model corresponding to the first object and specific viewpoint information, and a second virtual image generated based on a 3D model corresponding to a second object different from the first object and the viewpoint information, at a composite ratio corresponding to the viewpoint information. (2) The image generation unit generates the composite image at a composite ratio corresponding to the distance between a virtual viewpoint based on the viewpoint information and the first imaging device. The information processing apparatus according to (1) above. (3) When the distance between the first imaging device and the virtual viewpoint based on the viewpoint information is equal to or less than a predetermined value, the image generation unit synthesizes the first virtual image at a lower composite ratio than when the distance is greater than the predetermined value. The information processing apparatus according to (1) or (2) above. (4) The information processing apparatus further includes a viewpoint setting unit that sets the virtual viewpoint, which is the viewpoint information, over time. The information processing apparatus according to any one of (1) to (3) above. (5) The viewpoint information is the position of the virtual viewpoint and the observation direction at the virtual viewpoint. The information processing apparatus according to any one of (1) to (4) above. (6) The image generation unit generates the composite image at a composite ratio corresponding to the moving speed of the virtual viewpoint. The information processing apparatus according to (4) or (5) above. (7) The image generation unit generates a composite image using a first real image and a second real image respectively captured by two different first imaging devices, a first virtual image generated by projecting a 3D model of the first object onto a virtual viewpoint located between the two first imaging devices, and a second virtual image generated from the second object. The information processing apparatus according to any one of (2) to (6) above. (8) The image generation unit generates the composite image by switching the composite ratio between a minimum value and a maximum value at a predetermined time. The information processing apparatus according to (6) or (7) above. (9) The image generation unit generates the composite image by changing the composite ratios of the real image and the first virtual image in a state where the composite ratio of the second object is set to the maximum value. The information processing apparatus according to any one of (6) to (8) above. (10) The image generation unit generates the composite image by delaying the change timing of the composite ratio of the first virtual image compared to the falling timing of the composite ratio of the real image or advancing it compared to the rising timing of the composite ratio of the real image. The information processing apparatus according to any one of (6) to (9) above. (11) The composite ratio of the second object is the transmittance when superimposing the real image and the first virtual image on the second object. The information processing apparatus according to any one of (1) to (10) above. (12) The second object includes a video effect. The information processing apparatus according to any one of (1) to (11) above. (13) The second object includes a character. The information processing apparatus according to any one of (1) to (11) above. (14) The observation direction is the direction from the virtual viewpoint toward the first object. The information processing apparatus according to any one of (5) to (13) above. (15) The viewpoint setting unit sets the virtual viewpoint based on the position of a second imaging device different from the plurality of first imaging devices used for generating the 3D model of the first object. The information processing apparatus according to any one of (5) to (14) above. (16) The image generation unit sets the composite ratio of the real image captured by the second imaging device when the position of the virtual viewpoint is closer than a predetermined distance to the second imaging device to be higher than the composite ratio of the real image captured by the second imaging device when the position of the virtual viewpoint is farther than the predetermined distance from the second imaging device. The information processing apparatus according to (15) above. (17) When the position of the virtual viewpoint approaches the second imaging device, the image generation unit changes the composite ratio of the real image captured by the second imaging device more gently than the composite ratio of the real image captured by the first imaging device when the position of the virtual viewpoint approaches the first imaging device. The information processing apparatus according to (16) above. (18) The second imaging device is provided in a mobile terminal. The information processing apparatus according to any one of (15) to (17) above. (19) An information processing apparatus comprising an image generation unit that generates a composite image by combining a real image including a first object captured by a first imaging device, a 3D model corresponding to the first object, and a 3D model corresponding to a second object different from the first object at a composite ratio according to specific viewpoint information. (20) A method for generating image data, which generates a composite image by combining a real image including a first object captured by a first imaging device, a 3D model corresponding to the first object, a first virtual image generated based on specific viewpoint information, a 3D model corresponding to a second object different from the first object, and a second virtual image generated based on the viewpoint information at a composite ratio according to the viewpoint information.

Explanation of Signs

[0197] 10a... Information processing system, 11... Data acquisition unit, 12... 3D model generation unit, 13a... Encoding unit, 14... Transmission unit, 15... Reception unit, 16a... Decoding unit, 17a, 17b... Rendering unit, 18... Display unit, 40a... Encoding device (information processing device), 50a, 50b... Decoding device (information processing device), 51... Virtual image generation unit, 52... Viewpoint setting unit, 53... Superimposition ratio calculation unit, 54... 3DCG object image generation unit, 55... Image synthesis unit (image generation unit), 56... 3D object superimposition unit, 57... Signal selection unit, 70, 70a, 70b... Real camera (first imaging device), 72a, 72b, 72c, 72d, 72e, 72f, 72g, 72h, 72i, 72k... Virtual camera, 74, 74a, 74b... Mobile terminal, 75... Operation control unit, 76, 76a, 76b... Real camera (second imaging device), 80... Imaging control unit, 81... Positioning unit, 82... Application control unit, 83... Display control unit, 84... Operation control unit, 85... Communication control unit, 88a, 88b, 88c... Movement path, 90... Subject (first object), 90M, 92... 3D model, A1, A2... Adder, Ia... Real image (first real image), Ib... Real image (second real image), Ic, Id... Real image, Ix1, Ix2... Superimposed image, L... Composite image, M1, M2, M3, M4, M5... Multiplier, Mo... 3DCG object existence map, O... 3DCG object image (second virtual image), r, r1, r2, r3, r ca ,r cb ,r cc ,r cd ,r V ,r V 1,r V 2,r 3DCG … Superimposition ratio (synthesis ratio), V... Virtual image (first virtual image), Vp... Virtual viewpoint (viewpoint information)

Claims

1. A real image including a first object captured by a first imaging device, a first virtual image generated based on a 3D model corresponding to the first object and specific viewpoint information, a second virtual image generated based on a 3D model corresponding to a second object different from the first object and the viewpoint information, and an image generation unit that generates a composite image by compositing the above at a composite ratio according to the viewpoint information. An information processing apparatus.

2. The image generation unit generates the composite image at a composite ratio according to the distance between a virtual viewpoint based on the viewpoint information and the first imaging device. The information processing apparatus according to Claim 1.

3. The image generation unit when the distance between the first imaging device and a virtual viewpoint based on the viewpoint information is equal to or less than a predetermined value, composites the first virtual image at a lower composite ratio than when the distance is greater than the predetermined value. The information processing apparatus according to Claim 1.

4. Further comprising a viewpoint setting unit that sets a virtual viewpoint, which is the viewpoint information, over time. The information processing apparatus according to Claim 1.

5. The viewpoint information is the position of a virtual viewpoint and the observation direction at the virtual viewpoint. The information processing apparatus according to Claim 4.

6. The image generation unit generates the composite image at a composite ratio according to the moving speed of the virtual viewpoint. The information processing apparatus according to Claim 5.

7. The image generation unit a first real image and a second real image respectively captured by two different first imaging devices, a first virtual image generated by projecting the 3D model of the first object onto a virtual viewpoint located between the two first imaging devices, and a second virtual image generated from the second object, and uses them to generate a composite image. The information processing apparatus according to Claim 2.

8. The image generation unit generates the composite image by switching the composite ratio between a minimum value and a maximum value at a predetermined time. The information processing apparatus according to Claim 6.

9. The image generation unit generates the composite image by changing the composite ratio of the real image and the first virtual image in a state where the composite ratio of the second object is set to the maximum value. The information processing apparatus according to Claim 6.

10. The image generation unit Generate the composite image by delaying the change timing of the composite ratio of the first virtual image relative to the fall timing of the composite ratio of the real image, or advancing it relative to the rise timing of the composite ratio of the real image. The information processing apparatus according to claim 6.

11. The composite ratio of the second object is The transmittance when superimposing the real image and the first virtual image and the second object. The information processing apparatus according to claim 1.

12. The second object includes a video effect. The information processing apparatus according to claim 1.

13. The second object includes a character. The information processing apparatus according to claim 1.

14. The observation direction is the direction from the virtual viewpoint toward the first object. The information processing apparatus according to claim 5.

15. The viewpoint setting unit Sets the virtual viewpoint based on the position of a second imaging device different from the plurality of first imaging devices used to generate the 3D model of the first object. The information processing apparatus according to claim 5.

16. The image generation unit Sets the composite ratio of the real image captured by the second imaging device when the virtual viewpoint is closer to the second imaging device than a predetermined distance to be higher than the composite ratio of the real image captured by the second imaging device when the virtual viewpoint is farther from the second imaging device than a predetermined distance. The information processing apparatus according to claim 15.

17. The image generation unit When the position of the virtual viewpoint approaches the second imaging device, changes the composite ratio of the real image captured by the second imaging device more gently than when the position of the virtual viewpoint approaches the first imaging device and the composite ratio of the real image captured by the first imaging device. The information processing apparatus according to claim 16.

18. The second imaging device is provided in a mobile terminal. The information processing apparatus according to claim 15.

19. An image generation unit that generates a composite image by synthesizing a real image including a first object captured by a first imaging device, a 3D model corresponding to the first object, and a 3D model corresponding to a second object different from the first object at a composite ratio according to specific viewpoint information. Information processing apparatus.

20. A real image including a first object captured by a first imaging device, A first virtual image generated based on a 3D model corresponding to the first object and specific viewpoint information, and a second virtual image generated based on a 3D model corresponding to a second object different from the first object and the viewpoint information, are combined at a combination ratio corresponding to the viewpoint information to generate a combined image, A method for generating image data.

Images