Image processing device, image processing method, and program

Abstract

The present technology relates to an image processing device, an image processing method, and a program capable of broadening the representation range of a motion image generated by using a 3D model. The image processing device is provided with: a virtual camera device setting unit that sets a virtual camera device; a virtual viewpoint image generation unit that generates a virtual viewpoint image, which is an image of a 3D object viewed from the viewpoint of the virtual camera device; a synthesis unit that generates a synthetic image based on a plurality of virtual viewpoint images; and a motion image generation unit that generates a motion image including a frame based on the virtual viewpoint image or the synthetic image, wherein, after arranging a plurality of virtual cameras, the virtual camera device setting unit integrates the plurality of virtual cameras by moving at least one of the plurality of virtual cameras. The present technology can be applied to, for example, a device for editing motion images.

Description

Technical Field

[0001] The present technology relates to an image processing device, an image processing method, and a program, and more particularly, to an image processing device, an image processing method, and a program suitable for use in the case of generating a moving image by using a 3D model. Background Art

[0002] In recent years, the following technology is becoming widespread: a 3D model of an object (subject) is generated based on images captured by multiple cameras arranged around the object, and an image of the object viewed from a desired viewpoint is reproduced by using the generated 3D model (for example, see Patent Document 1).

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: WO 2018 / 150933 A Summary of the invention

[0006] Problems to be solved by the present invention

[0007] An image of an object based on a desired viewpoint can be reproduced by using the above-mentioned technique. Therefore, it is expected that the above-mentioned technique will expand the range of representation in moving images.

[0008] The present technology has been made in view of such circumstances, and aims to be able to expand the range of representation in a moving image generated by using a 3D model.

[0009] Solution to the problem

[0010] According to one aspect of the present technology, an image processing device includes: a virtual camera device setting unit, which sets the virtual camera device; a virtual viewpoint image generation unit, which generates a virtual viewpoint image, which is an image of a 3D object viewed from the viewpoint of the virtual camera device; a synthesis unit, which generates a synthetic image based on a plurality of virtual viewpoint images; and a motion image generation unit, which generates a motion image including frames based on the virtual viewpoint image or the synthetic image, wherein, after arranging a plurality of virtual cameras, the virtual camera device setting unit moves at least one of the plurality of virtual cameras to integrate the plurality of virtual cameras.

[0011] According to one aspect of the present technology, an image processing method includes: after arranging multiple virtual cameras, causing an image processing device to move at least one of the multiple virtual cameras to integrate the multiple virtual cameras, and the image processing device is configured to: set the virtual camera; generate a virtual viewpoint image, which is an image of a 3D object viewed from the viewpoint of the virtual camera; generate a composite image based on the multiple virtual viewpoint images; and generate a motion image including frames generated based on the virtual viewpoint image or the composite image.

[0012] According to one aspect of the present technology, a program causes a computer of an image processing device to perform the following processing: after arranging a plurality of virtual cameras, moving at least one of the plurality of virtual cameras to integrate the plurality of virtual cameras, the image processing device being configured to: set up the virtual cameras; generate a virtual viewpoint image, which is an image of a 3D object viewed from a viewpoint of the virtual camera; generate a composite image based on the plurality of virtual viewpoint images; and generate a motion image including frames based on the virtual viewpoint image or the composite image.

[0013] In one aspect of the present technology, after arranging a plurality of virtual cameras, at least one of the plurality of virtual cameras is moved to integrate the plurality of virtual cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram showing a configuration example of an image processing device to which the present technology is applied.

[0015] Figure 2 is a flowchart for describing the moving image generation process.

[0016] Figure 3 is a diagram for describing a first specific example of the moving image generating method.

[0017] Figure 4 is a diagram for describing a first specific example of the moving image generating method.

[0018] Figure 5 is a diagram for describing a first specific example of the moving image generating method.

[0019] Figure 6 is a diagram for describing a first specific example of the moving image generating method.

[0020] Figure 7 is a diagram for describing a first specific example of the moving image generating method.

[0021] Figure 8 is a diagram for describing a first specific example of the moving image generating method.

[0022] Fig. 9 is a diagram for describing a second specific example of the moving image generating method.

[0023] Fig.10 It is a diagram for describing a third specific example of the moving image generating method.

[0024] Fig.11 is a diagram showing a configuration example of a computer. DETAILED DESCRIPTION

[0025] A mode for carrying out the present technology will be described below. The description will be provided in the following order.

[0026] 1. Implementation Method

[0027] 2. Modification

[0028] 3. Others

[0029] <<1. Implementation Method>>

[0030] Will refer to Figures 1 to 10 Embodiments of the present technology are described.

[0031] <Configuration Example of Image Processing Apparatus>

[0032] Figure 1 : is a block diagram showing an embodiment of an image processing device 11 to which the present technology is applied.

[0033] The image processing device 11 is a device that generates a moving image by using a 3D model.

[0034] Note that the type of 3D model and the method for generating the 3D model are not particularly limited. For example, the 3D model to be used may be a 3D model generated based on images captured from multiple viewpoints around a real object (subject), or may be a 3D model artificially generated by using computer graphics or the like. Furthermore, in the former case, for example, the 3D model may be a model in which texture information has been superimposed on polygons, or a model in which texture information has been superimposed on a point cloud.

[0035] In addition, the number of objects (hereinafter referred to as 3D objects) in the 3D model is not particularly limited. That is, there may be only a single 3D object (eg, a single person) in the 3D model, or there may be multiple 3D objects (eg, multiple people) in the 3D model.

[0036] The image processing device 11 includes an input unit 21 , an image processing unit 22 , a storage unit 23 , and a display unit 24 .

[0037] For example, the input unit 21 includes various input devices, and is used to input instructions (e.g., virtual camera setting instructions), data required for processing to be performed by the input image processing device 11 (e.g., 3D model data), etc. The input unit 21 provides the input data, instructions, etc. to the image processing unit 22, and stores the input data, instructions, etc. in the storage unit 23.

[0038] The virtual camera device setting unit 31 sets the virtual camera device to be used for generating the virtual viewpoint image based on the setting instruction or the like input by the user via the input unit 21. For example, the virtual camera device setting unit 31 sets the position, orientation, focal length (viewing angle), etc. of the virtual camera device in the 3D space in which the 3D model exists. The virtual camera device setting unit 31 provides the setting information on the virtual camera device to the virtual viewpoint image generating unit 32.

[0039] The virtual viewpoint image generating unit 32 generates a virtual viewpoint image based on the 3D model data input via the input unit 21 or stored in the storage unit 23 and the virtual camera set by the virtual camera setting unit 31 .

[0040] For example, a virtual viewpoint image is a virtual image of a 3D object in a 3D model viewed from a viewpoint of a virtual camera (hereinafter, referred to as a virtual viewpoint). More specifically, a virtual viewpoint image is a virtual image assumed to be obtained when an image of a 3D object in a 3D model is captured by a virtual camera with a set position, orientation, and focal length.

[0041] The virtual viewpoint image generating unit 32 supplies the generated virtual viewpoint image to the synthesizing unit 33 and the moving image generating unit 34 as necessary, or stores the generated virtual viewpoint image in the storage unit 23 .

[0042] The synthesis unit 33 generates a synthesized image based on the virtual viewpoint images generated based on a plurality of different virtual cameras, and provides the generated synthesized image to the motion image generation unit 34 or stores the generated synthesized image in the storage unit 23 as needed.

[0043] The moving image generation unit 34 generates a frame based on a virtual viewpoint image or a synthesized image, and generates a moving image including the generated frame. The moving image generation unit 34 stores the generated moving image in the storage unit 23 as needed, or supplies the generated moving image to the display unit 24.

[0044] For example, the display unit 24 includes a display including a liquid crystal display (LCD) or an organic EL display. The display unit 24 displays the moving image supplied from the moving image generation unit 34, the moving image stored in the storage unit 23, the virtual viewpoint image, the synthesized image, and the like.

[0045] <Moving Image Generation Processing>

[0046] Next, we will refer to Figure 2 The flowchart of FIG. 1 describes the moving image generation process to be performed by the image processing device 11.

[0047] This process starts, for example, when a user inputs an instruction to generate a moving image via the input unit 21 .

[0048] In step S1, the virtual camera setting unit 31 sets the virtual camera. For example, the virtual camera setting unit 31 sets the position, orientation, focal length, etc. of the virtual camera in the 3D space based on the setting instruction input by the user via the input unit 21. Alternatively, for example, the virtual camera setting unit 31 sets the position, orientation, focal length, etc. of the virtual camera in the 3D space according to a preset algorithm, rule, etc.

[0049] At this time, for example, multiple virtual cameras can be arranged for a single 3D object. In addition, for example, when the process is to be performed on multiple 3D objects, a virtual camera can be placed separately for each 3D object, or a common virtual camera can be placed for multiple 3D objects.

[0050] The virtual camera setting unit 31 generates setting information about the virtual camera and provides the setting information to the virtual viewpoint image generating unit 32. The setting information about the virtual camera includes, for example, parameters such as position, orientation, and focal length of each virtual camera in 3D space, and information about a 3D object to be imaged by each virtual camera.

[0051] In step S2 , the virtual viewpoint image generation unit 32 generates a virtual viewpoint image based on the set virtual camera.

[0052] Specifically, the virtual viewpoint image generation unit 32 acquires data on a 3D model including a 3D object for which a virtual viewpoint image is to be generated. For example, the 3D model data may be input from the outside via the input unit 21 or may be pre-stored in the storage unit 23.

[0053] Then, the virtual viewpoint image generation unit 32 generates a virtual viewpoint image for each combination of a virtual camera and a 3D object. That is, the virtual viewpoint image generation unit 32 generates a virtual viewpoint image of the target 3D object viewed from the viewpoint (virtual viewpoint) of each virtual camera for each virtual camera.

[0054] In step S3, the virtual viewpoint image generation unit 32 determines whether a plurality of virtual viewpoint images have been generated. In the case where it is determined that a plurality of virtual viewpoint images have been generated, the process proceeds to step S4.

[0055] In step S4 , the synthesis unit 33 generates a synthesized image.

[0056] Specifically, the virtual viewpoint image generating unit 32 supplies the plurality of virtual viewpoint images generated in the process of step S2 to the synthesizing unit 33 .

[0057] The synthesizing unit 33 generates a synthesized image based on the plurality of acquired virtual viewpoint images. The synthesizing unit 33 supplies the generated synthesized image to the moving image generating unit 34.

[0058] Note that the method for generating the synthesized image is not particularly limited. For example, the synthesizing unit 33 generates the synthesized image by arranging a plurality of virtual viewpoint images. For example, the synthesizing unit 33 generates the synthesized image by superimposing a plurality of virtual viewpoint images one on top of another. For example, the synthesizing unit 33 generates the synthesized image by extracting an area including a 3D object from each virtual viewpoint image and arranging the area in a single image.

[0059] In step S5, the motion image generation unit 34 adds the frame based on the synthesized image to the motion image. Specifically, the motion image generation unit 34 generates a frame to be included in the motion image based on the acquired synthesized image. At this time, for example, the motion image generation unit 34 can combine an image of a background, etc. with the synthesized image as needed. The motion image generation unit 34 adds the generated frame as the latest frame to the generated motion image.

[0060] Thereafter, the process proceeds to step S7.

[0061] On the other hand, in the event that it is determined in step S3 that a single virtual viewpoint image has been generated, the process proceeds to step S6 .

[0062] In step S6 , the moving image generation unit 34 adds a frame based on the virtual viewpoint image to the moving image.

[0063] Specifically, the virtual viewpoint image generating unit 32 supplies the virtual viewpoint image generated in the process of step S2 to the moving image generating unit 34 .

[0064] The motion image generation unit 34 generates a frame to be included in the motion image based on the acquired virtual viewpoint image. At this time, for example, the motion image generation unit 34 can combine an image of a background, etc. with the virtual viewpoint image as needed. The motion image generation unit 34 adds the generated frame as the latest frame to the motion image being generated.

[0065] Thereafter, the process proceeds to step S7.

[0066] In step S7, the moving image generation unit 34 determines whether to end the process. In the case where it is determined that the process should not be ended, the process returns to step S1.

[0067] Thereafter, in step S7, the processing in steps S1 to S7 is repeatedly performed until it is determined that the processing should be terminated. Therefore, the following processing is repeated: in this processing, a virtual viewpoint image is generated based on the virtual camera device, a synthetic image is generated as needed, and a frame based on the virtual viewpoint image or the synthetic image is added to the motion image.

[0068] On the other hand, in step S7 , for example, in the case where an instruction to stop generating a moving image is input by the user via the input unit 21 , the moving image generating unit 34 determines that the process should be ended.

[0069] Thereafter, the moving image generation process ends.

[0070] As described above, a moving image is generated by using a 3D model.

[0071] <Specific Example of Moving Image Generation Method>

[0072] Next, we will refer to Figures 3 to 10 A specific example of the moving image generation method is described.

[0073] First, refer to Figures 3 to 8 A first specific example of the moving image generating method is described.

[0074] exist Figure 3 In the example of FIG. 1 , the 3D objects 101 a to 101 e are arranged in a lateral direction in a 3D space.

[0075] Hereinafter, in the case where the 3D objects 101a to 101e do not need to be individually distinguished, these objects are simply referred to as 3D objects 101. Furthermore, hereinafter, the surface of each 3D object 101 on which letters are written is defined as the front surface of each 3D object 101.

[0076] Note that the 3D object 101 may be included in different 3D models, or may be included in the same 3D model.

[0077] In the former case, for example, an image of each object is captured separately, and a plurality of 3D models are generated so that a 3D object 101 corresponding to each object is included in a corresponding one of the plurality of 3D models. Then, the 3D models including the corresponding 3D objects 101 are arranged in a 3D space, respectively. Therefore, in this case, the positions of the 3D objects 101 relative to each other can be changed in the 3D space.

[0078] In the latter case, for example, images of the object are captured simultaneously and a single 3D model is generated such that all 3D objects 101 corresponding to the object are included in the single 3D model. The 3D model including all 3D objects 101 is then placed in a 3D space. Therefore, in this case, the positions of the 3D objects 101 relative to each other are fixed in the 3D space.

[0079] Then, the virtual camera setting unit 31 individually arranges the virtual cameras 102 a to 102 e for the 3D objects 101 a to 101 e , respectively.

[0080] Hereinafter, in a case where the virtual cameras 102 a to 102 e do not need to be individually distinguished, these virtual cameras are simply referred to as virtual cameras 102 .

[0081] Each virtual camera 102 is placed at a predetermined distance from the front surface of the corresponding 3D object 101 in such a manner as to face the front surface of the corresponding 3D object 101 .

[0082] Next, the virtual viewpoint image generation unit 32 generates a virtual viewpoint image including the corresponding 3D object 101 based on each virtual camera 102 .

[0083] Next, if Figure 4 As shown, the synthesis unit 33 generates a synthesized image by combining the images of the rectangular areas extracted from the virtual viewpoint image. The rectangular areas are equal in size and each includes a corresponding 3D object 101. Specifically, the synthesis unit 33 generates a synthesized image 121-1 in which images 122a-1 to 122e-1 are arranged side by side.

[0084] Note that image 122a-1 is an image of a rectangular area including 3D object 101a extracted from a virtual viewpoint image generated based on virtual camera 102a. Image 122b-1 is an image of a rectangular area including 3D object 101b extracted from a virtual viewpoint image generated based on virtual camera 102b. Image 122c-1 is an image of a rectangular area including 3D object 101c extracted from a virtual viewpoint image generated based on virtual camera 102c. Image 122d-1 is an image of a rectangular area including 3D object 101d extracted from a virtual viewpoint image generated based on virtual camera 102d. Image 122e-1 is an image of a rectangular area including 3D object 101e extracted from a virtual viewpoint image generated based on virtual camera 102e.

[0085] Then, the moving image generation unit 34 generates a frame based on the synthesized image 121 - 1 , and adds the frame to the moving image.

[0086] Next, if Figure 5 As shown, the virtual camera setting unit 31 moves the virtual camera 102 in the corresponding direction in such a manner as to integrate the virtual camera 102 into a single virtual camera.

[0087] Specifically, the virtual camera 102c moves backward (in a direction away from the 3D object 101c) while maintaining the front surface facing the 3D object 101c. As indicated by arrow 103 in the figure, the virtual camera 102a and the virtual camera 102b move in a direction in which the virtual camera 102a and the virtual camera 102b approach the virtual camera 102c (diagonally to the right and rear direction) without changing their respective orientations. As indicated by arrow 104 in the figure, the virtual camera 102d and the virtual camera 102e move in a direction in which the virtual camera 102d and the virtual camera 102e approach the virtual camera 102c (diagonally to the rear and left direction) without changing their respective orientations.

[0088] Next, before integrating the virtual camera 102 , the virtual viewpoint image generating unit 32 generates a plurality of virtual viewpoint images including the corresponding 3D object 101 based on the respective virtual camera 102 .

[0089] Next, if Figure 6 As shown, the synthesis unit 33 generates a synthesized image by combining corresponding images of rectangular areas extracted from the virtual viewpoint image. The rectangular areas are equal in size and each includes a corresponding 3D object 101. Specifically, the synthesis unit 33 generates a synthesized image 121-2 in which images 122a-2 to 122e-2 are arranged side by side.

[0090] Note that image 122a-2 is an image of a rectangular area including 3D object 101a extracted from a virtual viewpoint image generated based on virtual camera 102a. Image 122b-2 is an image of a rectangular area including 3D object 101b extracted from a virtual viewpoint image generated based on virtual camera 102b. Image 122c-2 is an image of a rectangular area including 3D object 101c extracted from a virtual viewpoint image generated based on virtual camera 102c. Image 122d-2 is an image of a rectangular area including 3D object 101d extracted from a virtual viewpoint image generated based on virtual camera 102d. Image 122e-2 is an image of a rectangular area including 3D object 101e extracted from a virtual viewpoint image generated based on virtual camera 102e.

[0091] Then, the moving image generation unit 34 generates a frame based on the synthesized image 121 - 2 , and adds the frame to the moving image.

[0092] Afterwards, if Figure 7 As shown, the virtual camera setting unit 31 finally integrates each virtual camera 102 into a virtual camera 102f.

[0093] Note that before each virtual camera 102 is integrated into the virtual camera 102f, the number of frames of the moving image to be generated can be freely set. That is, at any desired point on the trajectory where each virtual camera 102 moves, a virtual viewpoint image can be generated based on each virtual camera, a synthetic image can be generated based on the virtual viewpoint image, and a frame of the moving image can be generated based on the synthetic image. As a result of repeating this process, a virtual viewpoint image, a synthetic image, and a frame of the moving image are generated at each point on the trajectory where each virtual camera 102 moves. That is, two or more frames of the virtual viewpoint image, two or more frames of the synthetic image, and two or more frames of the moving image are generated.

[0094] Then, the virtual viewpoint image generation unit 32 generates a virtual viewpoint image including Figure 8 101a to 101e of the virtual viewpoint images 121-N shown in FIG.

[0095] In addition, the motion image generation unit 34 generates frames based on the virtual viewpoint images 121-N and adds the frames to the motion image. Thus, frames based on a synthetic image based on a plurality of virtual viewpoint images corresponding to the corresponding virtual camera devices before integration and frames based on virtual viewpoint images corresponding to the integrated virtual camera devices are obtained as continuous frames.

[0096] Note that while each virtual camera 102 is moving, each 3D object 101 may be moving or stationary.

[0097] As described above, different virtual cameras are arranged for the corresponding 3D objects 101, and when each virtual camera moves in an appropriate direction in a manner integrated into a single virtual camera, there is a frame of a motion image generated based on a composite image based on a plurality of virtual viewpoint images corresponding to each virtual camera. A frame of a motion image is generated based on a virtual viewpoint image corresponding to the virtual camera into which each virtual camera has been integrated.

[0098] Therefore, for example, it is possible to produce a visual effect such that a user viewing a moving image feels as if a plurality of objects (eg, persons) that respectively exist in different spaces naturally move to a single space.

[0099] Note that the number of 3D objects 101 can be freely set. For example, the number of 3D objects 101 can be set to 1.

[0100] For example, when the number of 3D objects is set to 1, a plurality of virtual cameras 102 having different virtual viewpoints are arranged for a single 3D object 101. Then, each virtual camera 102 moves in a manner of finally being integrated into a single virtual camera. In this case, the same 3D object is included in each virtual viewpoint image.

[0101] Furthermore, the number of virtual cameras 102 may be set to any number equal to or greater than 2. Furthermore, the number of 3D objects 101 and virtual cameras 102 does not necessarily have to be equal.

[0102] In a case where the number of virtual cameras 102 is smaller than the number of 3D objects 101 , for example, one or more virtual cameras 102 are arranged in such a manner as to include a plurality of 3D objects 101 within a viewing angle.

[0103] In addition, for example, each virtual camera 102 may be placed in a manner including all 3D objects 101 within a viewing angle, and then moved in a manner integrated into a single virtual camera. In this case, each virtual viewpoint image includes a plurality of identical 3D objects.

[0104] Furthermore, not all virtual cameras 102 need to move. For example, the virtual cameras 102 may be configured such that when a single virtual camera 102 is fixed, the other virtual cameras 102 move toward the fixed virtual camera 102 to be integrated.

[0105] In addition, not all of the virtual cameras 102 need to be integrated. Therefore, the virtual cameras 102 may be configured such that only some of the virtual cameras 102 are integrated.

[0106] Furthermore, not all of the virtual cameras 102 need to be integrated at the same time, and, for example, the virtual cameras 102 may be integrated sequentially.

[0107] Furthermore, for example, each virtual camera 102 may be integrated into a virtual camera 102f and then separated again.

[0108] Next, we will refer to Fig. 9 A second specific example of the moving image generating method is described.

[0109] In this example, for example, the virtual camera setting unit 31 arranges three virtual cameras (not shown) so that the 3D object 142 located at the center is surrounded by the three virtual cameras. For example, the three virtual cameras are arranged around the 3D object 142 located at the center at equal angular intervals (i.e., at intervals of 120 degrees).

[0110] The virtual viewpoint image generation unit 32 generates three virtual viewpoint images including the same 3D object based on the respective virtual cameras. The virtual viewpoint images show the 3D object viewed from respective virtual viewpoints spaced 120 degrees apart.

[0111] The synthesis unit 33 generates a synthesized image by extracting an image of the 3D object from each virtual viewpoint image and combining the images to form a single image. Fig. 9 As shown, the synthesis unit 33 combines (the images of) the 3D objects 142 a to 142 c extracted from the respective virtual viewpoint images into a synthesized image 141 .

[0112] The synthesizing unit 33 arranges the 3D objects 142 a to 142 c at regular intervals (ie, at intervals of 120 degrees) around a predetermined axis extending vertically in the synthesized image 141 .

[0113] The moving image generation unit 34 generates a frame based on the synthesized image 141 , and adds the frame to the moving image.

[0114] Thereafter, the above-mentioned processing is repeatedly performed.

[0115] At this time, for example, the virtual camera setting unit 31 fixes the position and orientation of each virtual camera.

[0116] Furthermore, for example, the synthesizing unit 33 combines the 3D objects 142 a to 142 c in the synthesized image so that the 3D objects 142 a to 142 c rotate around the above-mentioned axis while maintaining a constant 120-degree interval when transitioning from one frame to another.

[0117] Therefore, an effect is produced that causes the 3D objects 142 a to 142 a representing the same 3D object 142 viewed from different virtual viewpoints to appear to rotate around a predetermined axis in the moving image like a kaleidoscope.

[0118] Note that the number of virtual cameras may be set to any number equal to or greater than two.

[0119] Furthermore, the virtual cameras may be arranged in any desired manner. For example, it is not always necessary to arrange the virtual cameras at regular intervals around the 3D object.

[0120] In addition, the virtual cameras can be moved. For example, multiple virtual cameras can be rotated around a 3D object located in the center.

[0121] Next, we will refer to Fig.10 A third specific example of the moving image generating method is described.

[0122] In this example, for example, the virtual camera setting unit 31 arranges different virtual cameras for a plurality of 3D objects so that the positional relationship between the virtual cameras is similar to the positional relationship between the plurality of 3D objects. That is, the virtual camera setting unit 31 arranges the virtual cameras at similar positions and orientations relative to the corresponding 3D objects.

[0123] The virtual viewpoint image generating unit 32 repeats a process of generating a plurality of virtual viewpoint images individually including corresponding 3D objects at the same time based on the respective virtual cameras.

[0124] The motion image generation unit 34 repeats the following processing: selecting one of the virtual viewpoint images including different 3D objects, generating frames based on the selected virtual viewpoint image, and adding the frames to the motion image. At this time, for example, the motion image generation unit 34 generates two or more consecutive frames based on the virtual viewpoint image including a certain 3D object, and then generates two or more consecutive frames based on the virtual viewpoint image including another 3D object.

[0125] Here, as described above, the virtual camera is arranged at a similar position and orientation relative to the corresponding 3D object. Therefore, the switching of the virtual viewpoint images used to generate the moving image produces an effect that makes the 3D object appear to be instantaneously replaced by another 3D object in the moving image. Fig.10The person 162 that is a 3D object in the frame 161 appears to be instantaneously replaced by a different person in the next frame.

[0126] Next, a fourth specific example of the moving image generating method will be described.

[0127] For example, in the case where two or more persons are arranged as 3D objects in a 3D space, the virtual camera setting unit 31 arranges the virtual camera so that the virtual viewpoints coincide with the respective viewpoints of the persons.

[0128] The virtual viewpoint image generation unit 32 repeats a process of generating a plurality of virtual viewpoint images viewed from respective viewpoints of a person at the same time based on respective virtual cameras.

[0129] The motion image generation unit 34 repeats the following processing: selecting one of the virtual viewpoint images based on a plurality of virtual viewpoints, generating a frame based on the selected virtual viewpoint image, and adding the frame to the motion image. At this time, for example, the motion image generation unit 34 generates two or more consecutive frames based on the virtual viewpoint image viewed from a certain person's viewpoint, and then generates two or more consecutive frames based on the virtual viewpoint image viewed from another person's viewpoint.

[0130] Therefore, an image of a person viewed from the viewpoint of another person existing in the 3D space is reproduced. In addition, the viewpoint for reproducing the image can be switched instantaneously.

[0131] As described above, the representation range of a moving image generated by using a 3D model can be expanded.

[0132] <<2. Modification>>

[0133] Hereinafter, modifications of the above-described embodiments of the present technology will be described.

[0134] The number, arrangement, movement, and the like of the above-described virtual cameras are examples and can be freely changed.

[0135] Furthermore, the number, type, arrangement, movement, etc. of the above-mentioned 3D objects are also examples and can be freely changed.

[0136] <<3. Others>>

[0137] <Computer Configuration Example>

[0138] The above series of processing can be implemented by hardware, or can be implemented by software. In the case of executing a series of processing by software, the program included in the software is installed in a computer. Here, examples of computers include computers combined in dedicated hardware and general-purpose personal computers that can perform various functions by installing various programs, etc.

[0139] Fig.11 : is a block diagram showing a configuration example of hardware of a computer that executes the series of processes described above with the help of a program.

[0140] In a computer 1000 , a central processing unit (CPU) 1001 , a read only memory (ROM) 1002 , and a random access memory (RAM) 1003 are interconnected via a bus 1004 .

[0141] The bus 1004 is also connected to an input / output interface 1005. The input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a recording unit 1008, a communication unit 1009, and a drive 1010.

[0142] The input unit 1006 includes an input switch, a button, a microphone, an image pickup device, etc. The output unit 1007 includes a display device, a speaker, etc. The recording unit 1008 includes a hard disk, a nonvolatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives a removable medium 1011 (for example, a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory).

[0143] In the computer 1000 configured as described above, for example, the CPU 1001 loads a program recorded in the recording unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program, thereby performing the above-described series of processing.

[0144] For example, the program to be executed by the computer 1000 (CPU 1001) may be provided after being recorded on the removable medium 1011 as a package medium, etc. Furthermore, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0145] In the computer 1000, the program can be installed in the recording unit 1008 via the input / output interface 1005 by mounting the removable medium 1011 on the drive 1010. In addition, the program can be received by the communication unit 1009 via a wired or wireless transmission medium. In addition, the program can be pre-installed in the ROM 1002 or the recording unit 1008.

[0146] Note that the program to be executed by the computer may be a program that causes the processing to be performed on a time-series basis according to the order described in this specification. Alternatively, the program may be a program that causes the processing to be performed in parallel or at a necessary time (for example, at the time when a call is made).

[0147] In addition, in this specification, the term "system" refers to a group of multiple constituent elements (devices, modules (components), etc.), and it does not matter whether all the constituent elements are in the same housing. Therefore, multiple devices stored in separate housings and connected via a network, and a single device including multiple modules stored in a single housing are considered to be a system.

[0148] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

[0149] For example, in the present technology, a configuration of cloud computing may be adopted in which a plurality of devices share a single function and perform processing in cooperation with one another via a network.

[0150] Furthermore, each step described in the above flowchart may be performed by a single device, or may be performed by a plurality of devices sharing responsibilities.

[0151] Furthermore, in the case where a plurality of processes are included in a single step, the plurality of processes included in the single step may be executed by a single device, or may be executed by a plurality of devices sharing the responsibility.

[0152] <Configuration combination example>

[0153] The present technology can also adopt the following configurations.

[0154] (1) An image processing device comprising:

[0155] A virtual camera device setting unit, which sets the virtual camera device;

[0156] A virtual viewpoint image generating unit that generates a virtual viewpoint image, the virtual viewpoint image being an image when a 3D object is viewed from a viewpoint of the virtual camera device;

[0157] a synthesis unit that generates a synthesized image based on a plurality of virtual viewpoint images; and

[0158] a moving image generating unit that generates a moving image including a frame based on the virtual viewpoint image or the synthesized image,

[0159] After arranging the plurality of virtual camera devices, the virtual camera device setting unit moves at least one of the plurality of virtual camera devices to integrate the plurality of virtual camera devices.

[0160] (2) The image processing device according to (1) above, wherein:

[0161] The virtual viewpoint image generating unit generates a plurality of virtual viewpoint images before integration and a virtual viewpoint image after integration, wherein the plurality of virtual viewpoint images before integration are respectively based on virtual viewpoint images of the virtual camera devices before integration, and the virtual viewpoint image after integration is based on the virtual viewpoint image of the virtual camera device after integration,

[0162] The synthesis unit generates a pre-integration synthesized image, which is a synthesized image based on the plurality of pre-integration virtual viewpoint images, and

[0163] The moving image generation unit generates a moving image including a pre-integration frame and a post-integration frame, the pre-integration frame being a frame based on the pre-integration synthesized image and the post-integration frame being a frame based on the post-integration virtual viewpoint image.

[0164] (3) The image processing device according to (2) above, wherein:

[0165] The virtual viewpoint image generating unit generates the plurality of pre-integration virtual viewpoint images of a plurality of frames based on the respective virtual camera devices before the respective virtual camera devices are integrated.

[0166] The synthesis unit generates a plurality of pre-integration synthesized images of the frames based on the plurality of pre-integration virtual viewpoint images of each frame, and

[0167] The moving image generation unit generates a plurality of pre-integration frames based on the pre-integration synthesized image of each frame.

[0168] (4) The image processing device according to (2) or (3) above, wherein:

[0169] The moving image generation unit generates a moving image in which the pre-integration frame and the post-integration frame are continuous.

[0170] (5) The image processing device according to any one of (2) to (4) above, wherein:

[0171] Each of the pre-integration virtual viewpoint images includes a corresponding different 3D object, and

[0172] The integrated virtual viewpoint image includes all different 3D objects.

[0173] (6) The image processing device according to (2) above, wherein:

[0174] The synthesis unit generates a synthesized image by synthesizing images of the respective 3D objects in the pre-integration virtual viewpoint image.

[0175] (7) The image processing device according to any one of (2) to (4) above, wherein:

[0176] Each of the pre-integration virtual viewpoint images and the post-integration virtual viewpoint image include the same 3D object.

[0177] (8) The image processing device according to (2) above, wherein:

[0178] Each of the pre-integration virtual viewpoint images and the post-integration virtual viewpoint images include a plurality of identical 3D objects.

[0179] (9) The image processing device according to any one of (1) to (8) above, wherein:

[0180] The virtual camera setting unit arranges a plurality of virtual cameras so that the 3D object is surrounded by the plurality of virtual cameras.

[0181] The virtual viewpoint image generating unit generates a plurality of virtual viewpoint images based on respective virtual camera devices.

[0182] The synthesis unit generates a synthesized image in which images of 3D objects in respective virtual viewpoint images are arranged around a predetermined axis, and rotates the images of the 3D objects around the axis between frames, and

[0183] The moving image generation unit generates a moving image including a plurality of frames based on a synthesized image for each frame.

[0184] (10) The image processing device according to (2) above, wherein:

[0185] The virtual camera setting unit arranges the virtual cameras at equal angular intervals around the 3D object located at the center.

[0186] (11) The image processing device according to any one of (1) to (10) above, wherein:

[0187] The virtual camera device setting unit arranges different virtual cameras for the plurality of 3D objects so that the virtual cameras have the same positional relationship.

[0188] The virtual viewpoint image generation unit generates a plurality of virtual viewpoint images including respective 3D objects based on respective virtual camera devices, and

[0189] The moving image generation unit generates a frame based on a virtual viewpoint image selected from a plurality of virtual viewpoint images, and switches the virtual viewpoint image used for generating the frame in the middle of the moving image.

[0190] (12) The image processing device according to any one of (1) to (11) above, wherein:

[0191] The virtual camera setting unit sets a position and an orientation of the virtual camera in a 3D space where a 3D object is placed.

[0192] (13) The image processing device according to (2) above, wherein:

[0193] The virtual camera setting unit further sets a focal length of the virtual camera.

[0194] (14) An image processing method, comprising:

[0195] After arranging the plurality of virtual camera devices, causing the image processing device to move at least one of the plurality of virtual camera devices to integrate the plurality of virtual camera devices,

[0196] The image processing device is configured to: set the virtual camera device; generate a virtual viewpoint image, which is an image of a 3D object viewed from the viewpoint of the virtual camera device; generate a composite image based on multiple virtual viewpoint images; and generate a motion image including frames based on the virtual viewpoint image or the composite image.

[0197] (15) A program for causing a computer of an image processing device to execute the following processing:

[0198] After arranging the plurality of virtual camera devices, at least one of the plurality of virtual camera devices is moved to integrate the plurality of virtual camera devices,

[0199] The image processing device is configured to: set the virtual camera device; generate a virtual viewpoint image, which is an image of a 3D object viewed from the viewpoint of the virtual camera device; generate a composite image based on multiple virtual viewpoint images; and generate a motion image including frames based on the virtual viewpoint image or the composite image.

[0200] Note that the effects described in this specification are merely exemplary and not restrictive, and other effects may be achieved.

[0201] Reference numerals list

[0202] 11 Image processing device

[0203] 12 Image Processing Unit

[0204] 31 Virtual camera device setting unit

[0205] 32 Virtual viewpoint image generation unit

[0206] 33 Synthesis Unit

[0207] 34 Motion image generation unit

Claims

1. An image processing device, comprising: A virtual camera device setting unit, which sets the virtual camera device; A virtual viewpoint image generating unit that generates a virtual viewpoint image, the virtual viewpoint image being an image when a 3D object is viewed from a viewpoint of the virtual camera device; a synthesis unit that generates a synthesized image based on a plurality of virtual viewpoint images; and a moving image generating unit that generates a moving image including a frame based on the virtual viewpoint image or the synthesized image, After arranging a plurality of virtual cameras based on a setting instruction input by a user, the virtual camera setting unit moves at least one of the plurality of virtual cameras according to a visual effect to be produced to integrate the plurality of virtual cameras into a single virtual camera.

2. The image processing device according to claim 1, wherein: The virtual viewpoint image generating unit generates a plurality of virtual viewpoint images before integration and a virtual viewpoint image after integration, wherein the plurality of virtual viewpoint images before integration are respectively based on virtual viewpoint images of the virtual camera devices before integration, and the virtual viewpoint image after integration is based on the virtual viewpoint image of the virtual camera device after integration, The synthesis unit generates a pre-integration synthesized image, which is a synthesized image based on the plurality of pre-integration virtual viewpoint images, and The moving image generation unit generates a moving image including a pre-integration frame and a post-integration frame, the pre-integration frame being a frame based on the pre-integration synthesized image and the post-integration frame being a frame based on the post-integration virtual viewpoint image.

3. The image processing device according to claim 2, wherein: The virtual viewpoint image generating unit generates the plurality of pre-integration virtual viewpoint images of a plurality of frames based on the respective virtual camera devices before the respective virtual camera devices are integrated. The synthesis unit generates a plurality of pre-integration synthesized images of the frames based on the plurality of pre-integration virtual viewpoint images of each frame, and The moving image generation unit generates a plurality of pre-integration frames based on the pre-integration synthesized image of each frame.

4. The image processing device according to claim 2, wherein: The moving image generation unit generates a moving image in which the pre-integration frame and the post-integration frame are continuous.

5. The image processing device according to claim 2, wherein: Each of the pre-integration virtual viewpoint images includes a corresponding different 3D object, and The integrated virtual viewpoint image includes all different 3D objects.

6. The image processing device according to claim 5, wherein: The synthesis unit generates a synthesized image by synthesizing images of the respective 3D objects in the pre-integration virtual viewpoint image.

7. The image processing device according to claim 2, wherein: Each of the pre-integration virtual viewpoint images and the post-integration virtual viewpoint image include the same 3D object.

8. The image processing device according to claim 7, wherein: Each of the pre-integration virtual viewpoint images and the post-integration virtual viewpoint images include a plurality of identical 3D objects.

9. The image processing apparatus according to claim 1, wherein: The virtual camera setting unit arranges a plurality of virtual cameras so that the 3D object is surrounded by the plurality of virtual cameras. The virtual viewpoint image generating unit generates a plurality of virtual viewpoint images based on respective virtual camera devices. The synthesis unit generates a synthesized image in which images of 3D objects in respective virtual viewpoint images are arranged around a predetermined axis, and rotates the images of the 3D objects around the axis with transition from one frame to another, and The moving image generation unit generates a moving image including a plurality of frames based on a synthesized image for each frame.

10. The image processing device according to claim 9, wherein: The virtual camera setting unit arranges the virtual cameras at equal angular intervals around the 3D object located at the center.

11. The image processing apparatus according to claim 1, wherein: The virtual camera device setting unit arranges different virtual cameras for the plurality of 3D objects so that the virtual cameras have the same positional relationship. The virtual viewpoint image generation unit generates a plurality of virtual viewpoint images including respective 3D objects based on respective virtual camera devices, and The moving image generation unit generates a frame based on a virtual viewpoint image selected from a plurality of virtual viewpoint images, and switches the virtual viewpoint image used for generating the frame in the middle of the moving image.

12. The image processing apparatus according to claim 1, wherein: The virtual camera setting unit sets a position and an orientation of the virtual camera in a 3D space where a 3D object is placed.

13. The image processing apparatus according to claim 12, wherein: The virtual camera setting unit further sets a focal length of the virtual camera.

14. An image processing method, comprising: After arranging the plurality of virtual cameras based on a setting instruction input by a user, causing the image processing device to move at least one of the plurality of virtual cameras according to a visual effect to be produced to integrate the plurality of virtual cameras into a single virtual camera, The image processing device is configured to: set the virtual camera device; generate a virtual viewpoint image, which is an image of a 3D object viewed from the viewpoint of the virtual camera device; generate a composite image based on multiple virtual viewpoint images; and generate a motion image including frames based on the virtual viewpoint image or the composite image.

15. A program product comprising a program for causing a computer of an image processing apparatus to execute the following processing: After arranging the plurality of virtual cameras based on a setting instruction input by a user, moving at least one of the plurality of virtual cameras according to a visual effect to be produced to integrate the plurality of virtual cameras into a single virtual camera, The image processing device is configured to: set the virtual camera device; generate a virtual viewpoint image, which is an image of a 3D object viewed from the viewpoint of the virtual camera device; generate a composite image based on multiple virtual viewpoint images; and generate a motion image including frames based on the virtual viewpoint image or the composite image.

16. A computer-readable storage medium having stored thereon a program, the program being used, when executed, to cause a computer of an image processing apparatus to perform the following processing: After arranging the plurality of virtual cameras based on a setting instruction input by a user, moving at least one of the plurality of virtual cameras according to a visual effect to be produced to integrate the plurality of virtual cameras into a single virtual camera, The image processing device is configured to: set the virtual camera device; generate a virtual viewpoint image, which is an image of a 3D object viewed from the viewpoint of the virtual camera device; generate a composite image based on multiple virtual viewpoint images; and generate a motion image including frames based on the virtual viewpoint image or the composite image.

Images