Image processing apparatus, image processing method, and storage medium

Abstract

Provided are an image processing apparatus, an image processing method, and a storage medium. A natural image in which a foreground object included in a two-dimensional image and a background object included in a three-dimensional computer graphics space are coordinated with each other is obtained. A two-dimensional image including a foreground object corresponding to a specific viewpoint, parameters for specifying conditions at the time of obtaining the two-dimensional image, and position and shape data representing a three-dimensional position and shape of the foreground object included in the two-dimensional image are obtained. Then, an image including the foreground object and the background object is generated by arranging a screen in a computer graphics space including the background object based on the position and shape data and projecting an image of the foreground object included in the two-dimensional image.

Description

Technical Field

[0001] This disclosure relates to image processing techniques for synthesizing two-dimensional images and three-dimensional computer graphics. Background Technology

[0002] There exists a video game creation platform (i.e., a game engine) that performs real-time rendering of 3D computer graphics (3DCG). In recent years, the techniques used by this game engine to achieve high image quality have been improved, enabling the real-time generation of representations through realistic composite images (see Japanese Patent Application Laid-Open No. 2018-074294). Consequently, 3DCG production using game engines is also utilized in the field of image production. As one method of utilization, 3DCG production is used in a studio (often referred to as a "virtual studio"), where images are generated and distributed through real-time compositing of 2D motion images (2D images) captured against a green background in the studio.

[0003] In the compositing of 2D images and 3DCG, a screen is set at a predetermined position in 3DCG space, and an image is obtained by sequentially projecting images of the subject captured by the camera onto the screen and rendering them. Therefore, if the camera (viewpoint) moves significantly during recording or the subject moves significantly, positional shifts relative to the screen or inconsistencies in the depth relationship with the background created by CG can occur, sometimes resulting in unnatural images. Summary of the Invention

[0004] An image processing apparatus includes: an acquisition unit configured to acquire a two-dimensional image including a foreground object corresponding to a specific viewpoint; parameters for specifying conditions for acquiring the two-dimensional image; and position and shape data representing the three-dimensional position and shape of the foreground object included in the two-dimensional image; and a generation unit configured to generate an image including the foreground object and the background object by arranging a screen in a computer graphics space including a background object based on the position and shape data and projecting an image of the foreground object included in the two-dimensional image onto the screen based on the parameters.

[0005] Other features of the invention will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0006] Figure 1 This is a diagram illustrating an example configuration of an image processing system used to generate virtual viewpoint images;

[0007] Figure 2This is a diagram illustrating an example of the hardware configuration of an image processing device;

[0008] Figure 3 This is a diagram illustrating an example of the internal configuration of an image processing device according to a first embodiment;

[0009] Figure 4 This is a flowchart illustrating the processing flow in the synthesis unit;

[0010] Figure 5A This is a diagram showing an example of a 3DCG space with a background model arranged, and Figure 5B This is a diagram showing the state of forming screens and pseudo shadows in a 3DCG space where a background model is arranged;

[0011] Figure 6A It is a diagram illustrating chroma key processing, and Figure 6B This is a diagram showing the state of projecting the cut-out image onto a screen generated in 3DCG space;

[0012] Figure 7 This is a diagram showing an example of a synthesized image;

[0013] Figure 8A This is an example showing significant camera movement, and Figure 8B This is a diagram showing the state of projecting the cut-out image onto a screen generated in 3DCG space;

[0014] Figure 9A This is a diagram illustrating an example where the projection viewpoint and rendering viewpoint are inconsistent with the camera viewpoint, and Figure 9B This is a diagram illustrating an example of a case where the size of the foreground object has been increased;

[0015] Figure 10A This is an example diagram showing a mask image of a screen, and Figure 10B This is a diagram showing an example of a cut-out image obtained from a mask image;

[0016] Figure 11A This is an example diagram showing a rendered image based on a variant example, and Figure 11B This is a diagram showing an example of a synthesized image based on a modified example;

[0017] Figure 12 This is a diagram illustrating an example of the internal configuration of a virtual viewpoint image generation system according to a second embodiment; and

[0018] Figure 13 This is a diagram showing an example of a synthesized image. Detailed Implementation

[0019] The present disclosure will now be described in detail with reference to the accompanying drawings and preferred embodiments. The configurations shown in the following embodiments are merely exemplary, and the present disclosure is not limited to the configurations illustrated schematically.

[0020] [First Embodiment]

[0021] <System Configuration>

[0022] Figure 1 This diagram illustrates an example configuration of an image processing system for generating virtual viewpoint images according to this embodiment. The image processing system 100 includes a camera 110, a measuring device 120, an image processing apparatus 130, and a display device 140. In the image processing system 100, the image captured by the camera 110 is combined with three-dimensional computer graphics in the image processing apparatus 130 and displayed on the display device 140.

[0023] For example, camera 110 continuously captures images of objects such as performers in the foreground (hereinafter referred to as "foreground objects") in a studio with a green or blue background suitable for chroma keying, and outputs the captured images (moving images). Camera 110 is implemented, for example, by a digital camera including an image signal interface represented by a Serial Digital Interface (SDI). After appending time information represented by timecode to the image signal to be output, camera 110 sends the image signal to image processing device 130. At this time, imaging parameters specifying the camera's three-dimensional position (x, y, z), pose (pan, pitch, tilt), angle of view, lens distortion, focal length, and resolution are also sent. The imaging parameters are pre-calculated by performing known camera calibration and stored in memory or the like. For example, the three-dimensional position and pose of the camera can be obtained by installing markers in camera 110 and detecting these markers by multiple optical sensors arranged in the imaging space. Furthermore, when the camera 110 is attached to a crane or robotic arm, the position and orientation of the camera 110 can be determined based on the position and orientation of the crane or robotic arm.

[0024] The measuring device 120 measures the position and rough shape of a foreground object existing in a real camera space, such as a studio. In this embodiment, the measuring device 120 has a camera function and measures the position and rough shape of the foreground object by analyzing multiple images obtained from multiple directions. For example, by using two cameras, the position and pose of a person or similar object can be detected in real time without the use of markers. Thus, data representing the position and rough shape of the foreground object in the camera space (hereinafter referred to as "position and shape data") is generated and output. In this embodiment, it is assumed that position and shape data describing the three-dimensional positions (x, y, z) of each vertex of a cuboid surrounding the foreground object and the lengths of each side of the cuboid are generated and output. However, the configuration of the measuring device 120 is not limited to the above example, and a local positioning system using wireless LAN or Bluetooth can be accepted. Furthermore, the three-dimensional position can be detected using GPS by attaching sensors to a person or similar object. In this case, it is sufficient to generate the position and shape data by adding information about the rough size of the person or similar object that has been pre-surveyed. The object shape can be a simple shape, and a polyhedron surrounding the object can be accepted.

[0025] The image processing apparatus 130 generates a composite image that makes the foreground object appear to exist in 3DCG space by projecting the image of the foreground object obtained by the camera 110 onto 3DCG space and rendering the image from the same viewpoint as when the image was taken. Details of the function of the image processing apparatus 130 will be described later.

[0026] The display device 140 is, for example, a liquid crystal display, and acquires and displays image data (data for UI screens of graphical user interfaces and data for composite images) sent from the image processing device 130.

[0027] In this embodiment, a single camera 110 is assumed, but the configuration can, for example, use multiple cameras 110. For instance, an image processing device 130, the same number as the number of cameras 110, can be used, and a device such as a switcher can be used to switch the composite images obtained by each system to another composite image, and then output the composite image. Alternatively, the captured image used for compositing with 3DCG can be switched to another captured image from the individual images input from the multiple cameras 110 within the image processing device 130, and the captured image can be processed.

[0028] <About Hardware Configuration>

[0029] Figure 2This is a diagram illustrating an example of the hardware configuration of an image processing device 130. The image processing device 130, as an information processing device, includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, an operation unit 215, a communication I / F 216, and a bus 217.

[0030] CPU 211 controls the entire image processing device 130 by using computer programs and data stored in ROM 212 or RAM 213, thereby implementing the various functions of the image processing device 130. The image processing device 130 may also have one or more dedicated hardware components different from CPU 211, or a GPU (Graphics Processing Unit). The GPU or dedicated hardware can then perform at least a portion of the processing performed by CPU 211. Examples of dedicated hardware include ASIC (Application-Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and DSP (Digital Signal Processor).

[0031] ROM 212 stores programs that do not need to be changed. RAM 213 temporarily stores programs and data supplied from auxiliary storage device 214, as well as data supplied from the outside via communication I / F 216. Auxiliary storage device 214 includes, for example, a hard disk drive, and stores various types of data such as image data and sound data.

[0032] The operation unit 215 includes, for example, a keyboard and a mouse, and inputs various instructions to the CPU 211 when it receives user input. The CPU 211 operates as a display control unit configured to control the display device 104 and an operation control unit configured to control the operation unit 215. The communication I / F 216 is used for communication between the image processing device 130 and external devices. For example, when the image processing device 130 is connected to an external device via wiring, a communication cable is connected to the communication I / F 216. When the image processing device 130 has the function of wireless communication with an external device, the communication I / F 216 includes an antenna.

[0033] Bus 217 connects the various units of image processing device 130 and sends information. In this embodiment, camera device 110, measuring device 120 and display device 140 are configured as external devices, but they can also be configured as internal functional units of image processing device 130.

[0034] <Internal Configuration of Image Processing Equipment>

[0035] Figure 3This diagram illustrates an example of the internal configuration of an image processing apparatus 130 that composites a captured image of a foreground object obtained by camera 110 with 3DCG. The image processing apparatus 130 includes various functional units: a data acquisition unit 301, a synchronization unit 302, a compositing unit 303, and a data output unit 308. The compositing unit 303 further includes a background storage unit 304, a preprocessing unit 305, a projection unit 306, and a rendering unit 307. The image processing apparatus 130 may also be configured with multiple electronic devices. The individual functional units will be described below.

[0036] The data acquisition unit 301 receives and acquires captured image data of the foreground object and imaging parameters from the camera 110. Furthermore, the data acquisition unit 301 acquires position and shape data of the foreground object from the measuring device 120. The data acquired by the data acquisition unit 301 is input to the synchronization unit 302. It is desirable, but not mandatory, that the captured image data and position and shape data are acquired at a synchronized timing. In cases where the acquisition period (sampling period) differs between these two data sets, position and shape data synchronized with the captured image data can be generated separately based on the amount of change in the acquired data. The imaging parameters can be transmitted (or received) at the same timing as the captured image data, or they can be transmitted (or received) at different timings. Furthermore, the imaging parameters can be transmitted (or received) when changing. In this case, the amount of data transmitted (or received) can be reduced. The frequency, period, and frame rate of transmitting (or receiving) the imaging parameters can be less than the frequency, period, and frame rate of transmitting (or receiving) the captured image data.

[0037] Synchronization unit 210 adjusts the delay time between data caused by differences in the timing of the generation, transmission, and acquisition of each input data. Specifically, a variable-length buffer is set for each data, and the data synchronized via the buffer is output to synthesis unit 303. When establishing synchronization between data, the delay time is adjusted by using time information attached to each data and changing the buffer level of other data according to the latest input data, so that the output time is the same across the data.

[0038] The compositing unit 303 generates a composite image that appears to exist in 3DCG space by projecting the image of the foreground object onto 3DCG space, based on captured image data and the position and shape data of the foreground object. Details of the compositing unit 303 will be described later.

[0039] The data output unit 308 outputs the data of the composite image generated by the synthesis unit 303 to an external device such as the display device 140.

[0040] The above describes the functional configuration of the image processing device according to this embodiment. In this embodiment, a synchronization unit 302 is used to adjust the synchronization between three data (i.e., captured image data, camera parameters, and position and shape data), but this configuration is not limited to this configuration. For example, the configuration could be one in which each of the camera device 110 and the measuring device 120 has an internal buffer and sends data to the image processing device 130 at the same timing.

[0041] <Details of the Synthesis Unit>

[0042] like Figure 3 As shown, the compositing unit 303 includes a background storage unit 304, a preprocessing unit 305, a projection unit 306, and a rendering unit 307. (Reference) Figure 4 The flowchart illustrates the details of the function of the synthesis unit 303. The CPU 211 reads the control program stored in the ROM 212 or auxiliary storage device 214, loads the program into the RAM 213, and executes the program to achieve [the desired result]. Figure 4 The flowchart illustrates a series of processes. In the following description, the symbol "S" denotes a step.

[0043] When the aforementioned data is input from the synchronization unit 302 to the compositing unit 303, firstly, at S401, the rendering unit 307 reads the CG data of the background object from the background storage unit 304 and arranges the background object using the CG in 3DCG space. Here, the CG data stored in the background storage unit 304 represents the three-dimensional shape and texture (color and texture, etc.) of various objects, such as pillars and fixed objects, which serve as the background in addition to the foreground objects arranged in 3DCG space. The CG data may have three-dimensional position information in 3DCG space. Furthermore, the CG data may have three-dimensional position information in real space. Figure 5A This diagram illustrates an example of a 3DCG space with CG background objects arranged, where four background objects 501a to 501d, represented by solid-lined cuboids, are arranged in the 3DCG space. It is assumed that the position of each background object within the 3DCG space is predetermined. Subsequent processing steps S402 to S407 are performed on each frame of the input captured image data.

[0044] In the next step S402, the preprocessing unit 305, based on the position and shape data input from the synchronization unit 302, arranges the image of the foreground object in the target frame projected onto the screen in 3DCG space. Furthermore, the preprocessing unit 305 forms a false shadow of the foreground object in 3DCG space. Figure 5B As shown above Figure 5AThe 3DCG space shown contains three screens 502a to 502c, indicated by dashed lines, and also forms pseudo-shadows 503a to 503c, indicated by shadows. In this embodiment, a transparent bounding box corresponding to the cuboid of the circumscribed foreground object is arranged in the 3DCG space as a screen. The screen is represented by a cuboid because a cuboid is the simplest cube surrounding a three-dimensional foreground object. When the screen is arranged as a two-dimensional plane, projection from the front surface is possible without any problems, but when projecting from the side or top surface, a plane is not formed, and therefore, a proper projected image cannot be obtained. Thus, in this embodiment, the screen is arranged as a cuboid. Although a screen can be defined as "a surface that always faces the viewpoint when projecting," its area changes significantly depending on the viewing direction, for example, when the foreground object is a complex shape like a person, and therefore needs to be calculated each time. On the other hand, when the screen is a cuboid, by taking the maximum and minimum values ​​of the coordinates (x, y, z) representing the three-dimensional position of each point of the cuboid, the projected image of the foreground object is included without exception, and therefore, calculation is no longer required each time. Furthermore, the cuboid has the advantage of suppressing data volume, thus reducing the processing load such as transmission. The screen is made transparent to prevent the user from visually recognizing it in the composite image. For this reason, a transparent bounding box is used as the screen in this embodiment. Then, to make the projected image of the foreground object appear to be in contact with the ground, the representation of shadows is important. However, the position and shape data in this embodiment does not include information related to the detailed shape and pose of the foreground object, making it difficult to represent accurate shadows based on the light source. Therefore, a pseudo-shadow with a simplified shape is drawn to improve the feeling of contact with the ground. Specifically, a semi-transparent black or gray ellipse corresponding to the rough shape and size represented by the position and shape data is drawn near the bottom surface of the bounding box that serves as the screen. Therefore, by displaying false shadows in a way that links the shadows to the movement of the foreground object, a natural composite image is obtained in 3DCG space where the foreground object appears to be in contact with the ground. It is desirable to form false shadows (such as light-based false shadows) near the bottom surface of the bounding box that serves as the screen, thus preventing shadows from becoming unnatural, such as shadows extending towards the light source. Depending on the position of the virtual light source, the false shadows can be formed outside the bounding box. For example, this is the case where light is emitted from the background object 501d side towards the background object 501a side. Furthermore, false shadows can be formed for the background object.

[0045] In the next step S403, the projection unit 306 sets the conditions (projection parameters) for projecting the image of the foreground object onto the screen based on the camera parameters of the target frame. Specifically, the viewpoint, posture, and angle during projection are adjusted to match the conditions used when capturing the target frame, ensuring that these two conditions are consistent.

[0046] Then, at S404, the projection unit 306 projects the image of the target frame onto the screen generated at S402. At this time, a portion of the image (the cut-out image) obtained by cutting out only the image region corresponding to the foreground object is projected onto the screen. This image region is obtained by performing chroma keying on the image of the target frame. Figure 6A This is a diagram illustrating chroma key processing. Figure 6A In the example, three cutout images corresponding to characters 601 to 603 are extracted from the target frame 600. Figure 6B This illustrates how the corresponding cut-out images are projected onto individual screens generated in 3DCG space (see [link]). Figure 5B The state of the screen. As mentioned above, the screen has been made transparent, so it is unlikely that the user will visually recognize the margin area of ​​the screen after the projected cutout image.

[0047] In the next step, S405, the rendering unit 307 sets the conditions (rendering parameters) for rendering by using the 3DCG space projected onto the cut-out image as the target, based on the camera parameters of the target frame. In the above... Figure 6B In this context, viewpoint marker 610 indicates the rendering viewpoint set according to the camera viewpoint of camera 110. That is, the rendering viewpoint is set such that its position, pose, and angle are consistent with the position, pose, and angle of camera 110 when it captures the target frame.

[0048] Then, at S406, rendering unit 307 performs rendering processing targeting the 3DCG space based on the rendering parameters set at S405. This 3DCG space is in a state where the cut-out image of the foreground object has been projected onto a transparent screen. In the rendering process, based on the distance from the set rendering viewpoint to each object, processing is performed to adopt color values ​​(pixel values) closer to the rendering viewpoint for each pixel. For example, a depth image representing the distance from the rendering viewpoint to each object is first generated based on information about the three-dimensional positions of the foreground and background objects that has become clear, and the values ​​of the pixels constituting the composite image are determined sequentially by referring to this depth image. Thus, as shown... Figure 7The composite image 700 shown reflects occlusion relationships between objects, such as displaying a background object in front of a foreground object in 3DCG space. In this way, a natural composite image is obtained that makes foreground objects existing in real three-dimensional space (camera space) appear to exist in 3DCG space.

[0049] At S407, it is determined whether an instruction to stop image generation has been given. Until an instruction to stop image generation is given, the process returns to S402 and continues to process the input target frames one by one.

[0050] The above describes the processing performed in the compositing unit 303. Due to this functional configuration, the compositing unit 303 can composite the image of the foreground object captured by the camera 110 with 3DCG with high accuracy while following the movement of the foreground object and the camera 110. For example, assuming... Figure 8A As indicated by viewpoint marker 610', camera 110 is significantly moved in the direction obliquely behind the performer. In this case, as... Figure 8B As shown, the cut-out images 611', 612', and 613' of each foreground object are projected onto a screen in 3DCG space from the same viewpoint as the viewpoint of the moved camera 110. In this way, even when the camera 110 or the objects move significantly, a composite image that accurately represents the occlusion relationships between objects can be obtained.

[0051] <Variation Example>

[0052] As explained below, under the premise of chroma keying, the camera 110 captures images of the foreground object in an environment with a so-called green or blue background. However, capturing images in such environments is not mandatory. For example, it is also possible to distinguish the performer from other backgrounds in the captured images obtained by capturing images in a normal studio using object recognition technology, and to use the image obtained by performing image processing that only cuts out the image area corresponding to the performer for projection.

[0053] Furthermore, the case of using the bounding box of the circumscribed foreground object as the projection screen has been described, but the shape of the screen is not limited to a cuboid. For example, the pose of the foreground object can be estimated by taking images of the foreground object from multiple directions, a mesh model representing the estimated pose can be generated, and a screen with a 3D shape suitable for the pose of the foreground object can be used. Thus, a screen, such as a human model simulating the pose of a performer, can be arranged in 3DCG space.

[0054] Furthermore, in the above embodiments, the coordinate system (x, y, z) of the projection viewpoint and the screen is aligned with the coordinate system (x', y', z') of the 3DCG space where the background model is arranged, but this is not a mandatory requirement in this embodiment. It is necessary that the relative positional relationships between the foreground object and the camera viewpoint, as well as between the screen, the projection viewpoint, and the rendering viewpoint, are aligned. With these relative positional relationships aligned, the screen, the projection viewpoint, and the rendering viewpoint can be moved to any coordinate in the 3DCG space and their orientation changed. Additionally, the size relationship between the foreground object and the 3DCG space can be changed by simultaneously altering the screen's magnification and the relative distance magnification. Figure 9A This is an example where each screen is positioned in front of the background object, and the projection viewpoint and rendering viewpoint are set to be far from the camera viewpoint. Figure 9B This example demonstrates how to increase the size of a foreground object by further changing the magnification of each screen and the magnification of their relative distances. Figure 9B In this case, by changing the size of the coordinate system itself (for example, if 1 cm in the original image corresponds to 1 m in real space, the coordinate system is changed so that 1 cm in the image corresponds to 50 cm in real space), and thus, the absolute distance of the projection viewpoint increases. As described above, it is also possible to produce a representation that the foreground object has moved or that the foreground object has become smaller or larger.

[0055] Furthermore, the cropped image of the foreground object is projected onto the screen and rendered, but the processing order is not necessarily limited to this sequence. For example, it can also be obtained through the following process. Figure 11B The composite image 1101 shown: obtained, for example, as Figure 10A The screen mask image shown also takes into account occlusion relationships when viewed from the rendering viewpoint; then, by... Figure 10A The mask image in the middle is applied to Figure 6A The captured image generates a result obtained by cropping out a portion of the foreground object, such as... Figure 10B The cut-out images 1001 and 1002 are shown; then, they are generated by rendering the 3DCG space as the target. Figure 11A The arrangement shown only has Figure 5A The background object shown is rendered as CG image 1100, and will Figure 10B The cut-out images 1001 and 1002 shown are superimposed on the rendered image 1100. However, in this method, 3D distance information related to the foreground object is not used during rendering, making it difficult to accurately represent depth of field or fog in the background. Therefore, the method described in this embodiment yields a composite image with higher quality.

[0056] As described above, according to this embodiment, a natural composite image in which a 2D image of the foreground object captured by a camera and a background created by CG are in harmony with each other can be easily obtained.

[0057] [Second Embodiment]

[0058] Next, we will describe an aspect of using virtual viewpoint images as a 2D image synthesized with 3DCG as a second embodiment. Figure 12 This is a diagram illustrating an example configuration of a system for generating virtual viewpoint images input to an image processing device 130.

[0059] like Figure 12 As shown, in this embodiment, the virtual viewpoint parameters, virtual viewpoint image, and foreground model output from the virtual viewpoint image generation system 1200 are input to the image processing device 130. That is, the virtual viewpoint parameters and virtual viewpoint image of this embodiment correspond to the imaging parameters and captured image output from the camera 110 in the first embodiment, and the foreground model of this embodiment corresponds to the position and shape data output from the measuring device 120 in the first embodiment. This embodiment will be described below by focusing on the differences from the first embodiment.

[0060] The virtual viewpoint image generation system 1200 includes multiple camera devices 1201, an image processing server 1202, and a controller 1206. The image processing server 1202 then has functional units including a three-dimensional shape estimation unit 1203, a material data storage unit 1204, and a virtual viewpoint image generation unit 1205.

[0061] Multiple camera devices 1201 are arranged around a foreground object and capture images of the foreground object from multiple directions. Furthermore, the timing of the images captured by the multiple camera devices 1201 is synchronized based on a synchronization signal from a time server (not shown schematically). Data of images captured by the multiple camera devices 1201 from multiple different viewpoints (hereinafter referred to as "multi-viewpoint images"), along with camera parameters representing the position / pose and optical characteristics of each camera device, are sent to an image processing server 1202 and input to a three-dimensional shape estimation unit 1203.

[0062] The 3D shape estimation unit 1203 extracts the silhouette of the foreground object from each captured image constituting the input multi-view image, and then generates data representing the 3D shape of the foreground object (hereinafter referred to as the "foreground model") using methods such as the visual hull method. Information representing the 3D position at the time of shooting is attached to the foreground model. The generated foreground model, along with the multi-view image (including camera parameters) that served as the source of the generated foreground model, is sent to the material data storage unit 1204.

[0063] The material data storage unit 1204 stores / accumulates various data from the foreground model, multi-view image, and camera parameters input from the three-dimensional shape estimation unit 1203 as material for the virtual view image.

[0064] The controller 1206 includes a joystick for setting a virtual camera viewpoint at any location in the camera space and a keyboard for specifying values ​​such as focal length and angle of view. It generates virtual viewpoint parameters based on user input via these components. These virtual viewpoint parameters include information corresponding to external camera parameters representing the position and orientation of the virtual viewpoint, information corresponding to internal camera parameters such as focal length and angle of view, and time information specifying the frame from which the virtual viewpoint image is generated. The generated virtual viewpoint parameters are sent to the image processing server 1202 and input to the virtual viewpoint image generation unit 1205.

[0065] The virtual viewpoint image generation unit 1205 first obtains data of the foreground model and the captured image at a relevant time from the material data storage unit 1204 based on the time information included in the input virtual viewpoint parameters. Then, for the obtained foreground model, the virtual viewpoint image generation unit 1205 performs coloring corresponding to the specified virtual viewpoint based on the captured image, and generates a virtual viewpoint image representing the appearance from the virtual viewpoint. The generated virtual viewpoint image data, along with the virtual viewpoint parameters and foreground model used for generation, is output to the image processing device 130.

[0066] Operation of Image Processing Equipment

[0067] In this embodiment, the image processing device 130 composites the cut-out image of the virtual viewpoint image generated by the image processing server 1202 with 3DCG, replacing the cut-out image of the image captured by the camera 110 in the first embodiment. Therefore, the data acquisition unit 301 processes information related to the position / pose, focal length, and viewing angle of the virtual viewpoint included in the virtual viewpoint parameters, using it as the camera parameters of the first embodiment. That is, the data acquisition unit 301 treats the virtual viewpoint as the camera 110, treats the optical information related to the position / pose and viewing angle of the virtual viewpoint as the optical information related to the position / pose and viewing angle of the camera 110 in the previously described camera parameters, and outputs this information to the synchronization unit 302. Furthermore, based on the time information included in the virtual viewpoint parameters or embedded in the virtual viewpoint image, the data acquisition unit 301 in this embodiment treats the foreground model corresponding to each frame as the position and shape data of the first embodiment, and outputs the foreground model to the synchronization unit 302. At this time, the foreground model can also be output after processing by simplifying the shape of the foreground model to reduce the amount of data, making it easier for the compositing unit 303 to process it in subsequent stages. As part of shape simplification, this includes processes such as converting a foreground model into information related to a cuboid (bounding box) representing the 3D shape of the circumscribed foreground model, or gap culling of point clouds or meshes representing 3D shapes. Needless to say, processes such as format conversion are performed as needed to ensure consistency between data.

[0068] As in the first embodiment, the synchronization unit 302 in this embodiment establishes synchronization between the three data input from the data acquisition unit 301: the virtual viewpoint image, the virtual viewpoint parameters, and the foreground model, and outputs them to the synthesis unit 303.

[0069] In this embodiment, the compositing unit 303 generates a composite image, which is a composite of a virtual viewpoint image input from the synchronization unit 302 and 3DCG. Specifically, the preprocessing unit 305 generates a projection screen and false shadows in the 3DCG space where the background object is arranged, based on the foreground model (S401, S402). Next, the projection unit 306 sets the projection parameters according to the virtual viewpoint parameters (S403), generates a cutout image of the foreground object from the virtual viewpoint image, and projects the cutout image onto the screen (S404). Then, the rendering unit 307 sets the rendering parameters according to the virtual viewpoint parameters (S405), and uses the 3DCG space in the state of projecting the cutout image of the virtual viewpoint image as the target, and performs rendering processing a number of times corresponding to a predetermined number of frames (S406).

[0070] <Variation Example>

[0071] In this embodiment, as in the first embodiment, synchronization is established within the image processing device 130 between the three data sets: the virtual viewpoint image, the virtual viewpoint parameters, and the foreground model. However, the configuration is not limited to this. For example, the configuration could be as follows: synchronization between the data is established within the virtual viewpoint image generation system, and then the data is integrated into a single file by means of compression, for example, and this file is sent to the image processing device 130. In this configuration, the synchronization unit 302 within the image processing device 130 is no longer needed. Then, it is sufficient for the data acquisition unit 301 to expand the received data file and obtain the individual data sets in the virtual viewpoint image, virtual viewpoint parameters, and foreground model, and input them into the synthesis unit 303. Due to this configuration, synchronization is no longer needed on the image processing device side, and the complexity of data transmission can be resolved simultaneously.

[0072] Furthermore, in this embodiment, the data acquisition unit 301 obtains the corresponding foreground model based on temporal information included in the virtual viewpoint parameters or embedded in the virtual viewpoint image; however, the configuration is not necessarily limited to this. For example, the configuration could be as follows: in the image processing server 1202, the virtual viewpoint image generation unit 1205 associates the foreground model obtained from the material data storage unit 1204 with frames of the corresponding virtual viewpoint image and then sends it to the image processing device 130. In this case, the data acquisition unit 301 of the image processing device 130 can obtain the foreground model corresponding to each frame without using temporal information.

[0073] Furthermore, in this embodiment, the data acquisition unit 301 performs simplification processing to derive a rough shape based on the foreground model input from the image processing server 1202, but this is not limited to this. For example, the image processing server 1202 side may also perform processing up to the simplification process and input the simplified foreground model (i.e., a foreground model with reduced data volume) to the image processing device 130. For example, after generating a foreground model by performing normal shape estimation in the three-dimensional shape estimation unit 1203, a simplified foreground model is further generated by performing processing to derive a rough shape from the foreground model. Then, in addition to the normal foreground model, the simplified foreground model is stored and accumulated in the material data storage unit 1204, and the simplified foreground model is sent to the image processing device 130. By doing so, the shape simplification processing for the foreground model can be omitted on the image processing device 130 side, and the data transmission load can also be reduced.

[0074] Furthermore, this embodiment describes obtaining a simplified shape by gap-canceling point cloud data from the foreground model (which is the result of shape estimation) and replacing the foreground model with bounding boxes, but shape simplification is not limited to these methods. For example, the configuration could be as follows: generating a mesh model, depth image, and pose information based on the shape estimation result, and using these to obtain a simplified shape. Additionally, the shape estimation result can be divided at predetermined height intervals to generate bounding boxes for each division, and the set of these can be used as the simplified shape.

[0075] Furthermore, false shadows can be generated using the aforementioned pose information. For example, a shadow based on a simplified shape of the performer can be projected onto the bottom surface of the previously described humanoid screen, obtained based on pose information related to the performer. Therefore, false shadows can be represented based on the movement of the performer, etc. Additionally, the position and shape data of the portion of the three-dimensional shape represented by the foreground model (which is an estimation result) near the floor surface (e.g., an area 30 cm above the floor surface) can be obtained separately, and false shadows can be drawn using this data. Using this method, even based on simplified shape data such as bounding boxes that cannot capture the outline of actual foreground objects, for example, ... Figure 13 The synthesized image 1300 shown can also generate false shadows 1301 only near the performer's feet. Furthermore, in this method, it is unnecessary to detect the pose of the foreground object, thus offering the advantage of low processing load.

[0076] According to this embodiment, a natural composite image that harmonizes with a 2D image representing the appearance from a virtual camera viewpoint unrestricted by the actual camera position can be easily obtained, where the image and the CG-created background are mutually coordinated. Furthermore, material data generated during the virtual viewpoint image generation is also used for compositing, resulting in high efficiency. Additionally, synchronization between data can be established, and the accuracy of obtaining the 3D position of foreground objects can be more easily improved, thus enabling high-quality compositing.

[0077] Other embodiments

[0078] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0079] According to the technology disclosed herein, a natural composite image in which foreground objects included in a two-dimensional image and background objects included in a three-dimensional computer graphics space are coordinated with each other can be obtained.

[0080] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: The acquisition unit is configured to acquire a two-dimensional image including foreground objects corresponding to a specific viewpoint, parameters for specifying the conditions for acquiring the two-dimensional image, and position and shape data representing the three-dimensional position and shape of the foreground objects included in the two-dimensional image; as well as The generation unit is configured to generate an image including the foreground object and the background object by arranging a transparent bounding box corresponding to the cuboid surrounding the foreground object in a computer graphics space including the background object based on the position and shape data, and projecting an image of the foreground object included in the two-dimensional image onto the transparent bounding box based on the parameters.

2. The image processing apparatus according to claim 1, wherein, The position and shape data are data used to describe the three-dimensional positions of each vertex of the cuboid circumscribed in the foreground object and the lengths of each side of the cuboid.

3. The image processing apparatus according to claim 1, wherein, The generation unit generates an image including the foreground object and the background object by forming a false shadow of the foreground object in the three-dimensional space of the computer graphics based on the position and shape data.

4. The image processing apparatus according to claim 3, wherein, The pseudo-shadow is a semi-transparent black or gray ellipse.

5. The image processing apparatus according to claim 1, wherein, With the image of the foreground object included in the two-dimensional image already projected onto the transparent bounding box, the generation unit generates an image including the foreground object and the background object by rendering the computer graphics space from a rendering viewpoint corresponding to the specific viewpoint.

6. The image processing apparatus according to claim 1, wherein, The generation unit generates an image including the foreground object and the background object by rendering, wherein the rendering is used to determine the color of each pixel based on the distance from the viewpoint set based on the parameters to each foreground object.

7. The image processing apparatus according to claim 6, wherein, The generation unit: The conditions for performing the projection are set according to the conditions for obtaining the two-dimensional image specified by the parameters; and The rendering conditions are set according to the conditions specified by the parameters for obtaining the two-dimensional image.

8. The image processing apparatus according to claim 1, wherein, The two-dimensional image corresponding to the specific viewpoint is a frame constituting the moving image, and The image processing device further includes a synchronization unit configured to generate position and shape data synchronized with each frame of the motion image when the acquisition period of the position and shape data is different from the acquisition period of the frames constituting the motion image.

9. The image processing apparatus according to claim 8, wherein, The synchronization unit establishes synchronization between data by having buffers for each of the motion image and the position and shape data, and by changing the buffer level in each buffer.

10. The image processing apparatus according to claim 9, wherein, The synchronization unit adjusts the delay time by changing the buffer level of other data based on the latest input data, so that the output time is the same across the data.

11. The image processing apparatus according to claim 1, wherein, The two-dimensional image corresponding to the specific viewpoint is a frame constituting the moving image, and The obtaining unit obtains the position and shape data at a timing synchronized with the frames constituting the motion image.

12. The image processing apparatus according to claim 1, wherein, The two-dimensional image corresponding to the specific viewpoint is the captured image obtained by the camera device, and The parameters are camera parameters representing the conditions under which the foreground object is photographed by the camera device.

13. The image processing apparatus according to claim 12, wherein, The image of the foreground object is obtained by performing chroma keying on the captured image.

14. The image processing apparatus according to claim 1, wherein, The two-dimensional image corresponding to the specific viewpoint is a virtual viewpoint image, which is generated based on multiple captured images obtained by a camera device from multiple viewpoints of the foreground object, and represents the appearance from a virtual viewpoint different from the multiple viewpoints. The parameters are virtual viewpoint parameters that represent the conditions for generating the virtual viewpoint image.

15. The image processing apparatus according to claim 14, wherein, The obtaining unit obtains the position and shape data of the foreground object based on the data representing the three-dimensional shape of the foreground object generated by using the multiple captured images.

16. An image processing method, comprising: Obtain a two-dimensional image of foreground objects corresponding to a specific viewpoint, parameters for specifying the conditions for obtaining the two-dimensional image, and position and shape data representing the three-dimensional position and shape of the foreground objects included in the two-dimensional image; as well as An image including the foreground object and the background object is generated by arranging a transparent bounding box corresponding to the cuboid surrounding the foreground object in a computer graphics space including the background object based on the position and shape data, and by projecting the image of the foreground object included in the two-dimensional image onto the transparent bounding box based on the parameters.

17. A computer-readable storage medium storing a program for causing a computer to perform the image processing method according to claim 16.

18. A computer program product comprising a program that, when executed by a processor, implements the steps of the image processing method according to claim 16.

Images