Image processing apparatus, imaging apparatus, and image processing method

Abstract

The present invention provides an image processing apparatus, an imaging apparatus, and an image processing method that can easily obtain natural shading when compositing images of a photographed subject and background. [Solution] The image processing device (200) includes an input unit (250) that inputs first image data showing a subject image captured by an imaging device (100), and a control unit (210) that generates composite image data by combining the subject image and the background image based on the first image data input to the input unit and a second image data showing a predetermined background image. The control unit acquires light source information indicating a light source in the background image based on the second image data (S2), and generates composite image data (S4) in which the subject image is placed on the background image, and includes a shadow region (R1) indicating a shadow corresponding to the light source by the subject shown in the subject image.

Description

[Technical Field]

[0001] This disclosure relates to an image processing apparatus for synthesizing images, an imaging apparatus, and an image processing method. [Background technology]

[0002] Patent Document 1 discloses an image processing device that adds a shading effect using virtual illumination to an captured image. This image processing device extracts 3D shape data of the subject area from 3D shape data corresponding to the captured image, and if there is a defect in the 3D shape data of the subject area, it sets the movable range of the virtual illumination based on the defect in the 3D shape data. This prevents unnatural shading from being reproduced when generating an image that reproduces shading under virtual illumination conditions based on the 3D shape data of the subject.

[0003] Furthermore, in recent years, image compositing techniques have been developed to replace the background of photographs containing subjects such as people with a different image. However, in simple compositing, the way light falls on the subject, i.e., the lighting, becomes unnatural. Therefore, relighting techniques have been proposed that use image processing to reflect lighting appropriate to the new background onto the subject image (for example, Non-Patent Documents 1-2). [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-101978 [Non-patent literature]

[0005] [Non-Patent Document 1] Rohit Pandey, et al, "Total Relighting: Learning to Relight Portraits for Background Replacement", ACM Transactions on Graphics, Vol. 40, No. 4, 2021. [Non-Patent Document 2] Daichi Tajima, et al, "Relighting Humans in the Wild: Monocular Full-Body Human Relighting with Domain Adaptation," Computer Graphics Forum (Proc. of Pacific Graphics 2021), Vol. 40, No. 7, 2021. [Overview of the project] [Problems that the invention aims to solve]

[0006] This disclosure provides an image processing apparatus, an imaging apparatus, and an image processing method that can easily obtain natural shading in the image synthesis of a photographed subject and background. [Means for solving the problem]

[0007] In this disclosure, the image processing device includes an input unit that inputs first image data showing a subject image captured by an imaging device, and a control unit that generates composite image data by combining the subject image and the background image based on the first image data input to the input unit and a second image data showing a predetermined background image. Based on the second image data, the control unit acquires light source information indicating a light source in the background image, and based on the light source information of the background image, generates composite image data including a shadow region showing a shadow corresponding to the light source of the subject shown in the subject image in the composite image in which the subject image is placed on the background image.

[0008] In this disclosure, the imaging device comprises an imaging unit that captures an image of a subject and generates first image data, and an image processing device that generates composite image data based on the first image data generated by the imaging unit.

[0009] In the present disclosure, an image processing method is an image processing method executed by a computer, including: a step of inputting first image data indicating a subject image captured by an imaging device; a step of obtaining light source information indicating a light source in a background image based on second image data indicating a predetermined background image; and a step of generating composite image data in which the subject image and the background image are combined, including a shadow area indicating a shadow corresponding to the light source by the subject indicated by the subject image, based on the light source information of the background image, in a composite image in which the subject image is arranged on the background image.

Effect of the Invention

[0010] According to the image processing apparatus, imaging apparatus, and image processing method in the present disclosure, it is possible to easily obtain a natural shadow in the image synthesis of a captured subject and background.

Brief Description of the Drawings

[0011] [Figure 1] Diagram illustrating the configuration of an imaging system in Embodiment 1 of the present disclosure [Figure 2] Diagram showing the configuration of a digital camera in the imaging system [Figure 3] Diagram showing the configuration of an image editing terminal in the imaging system [Figure 4] Diagram for explaining the outline of the operation of the imaging system [Figure 5] Flowchart illustrating the operation of an image editing terminal in the imaging system of Embodiment 1 [Figure 6] Flowchart illustrating the background preparation process in the imaging system [Figure 7] Diagram for explaining the background preparation process in the imaging system [Figure 8] Diagram for explaining the background environment in the imaging system [Figure 9] Flowchart illustrating the image synthesis process in the imaging system [Figure 10] Diagram for explaining the image synthesis process in the imaging system [Figure 11]Flowchart exemplifying shadow correction processing in an imaging system [Figure 12] Diagram for explaining shadow correction processing in an imaging system [Figure 13] Flowchart exemplifying the operation of a modified imaging system

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, detailed descriptions of well-known matters and duplicate descriptions of substantially the same configurations may be omitted. Note that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims thereby.

[0013] (Embodiment 1) 1. Configuration The imaging system according to Embodiment 1 of the present disclosure will be described with reference to FIG. 1.

[0014] As shown in FIG. 1 for example, this system 10 includes a digital camera 100, an image editing terminal 200, and an image processing server 300. In this system 10, the digital camera 100 and the image editing terminal 200 are connected so as to be capable of data communication by, for example, wired communication or wireless communication. The image processing server 300 is communicatively connected to the digital camera 100 and the image editing terminal 200 via a communication network such as the Internet.

[0015] This system 10 is applicable, for example, to applications where a user shoots and edits a moving image or a still image with the digital camera 100, and can be applied to virtual production using image synthesis instead of large-scale studio equipment. In this system 10, for example, the live view image in the digital camera 100 can be confirmed on the image editing terminal 200.

[0016] The image editing terminal 200 is an information terminal for editing image data obtained from, for example, a digital camera 100. In this system 10, the image editing terminal 200 does not need to communicate with either or both of the digital camera 100 and the image processing server 300. For example, data from the digital camera 100 may be input to the image editing terminal 200 via a portable recording medium such as a memory card.

[0017] The image processing server 300 is a server device composed of various computers, such as a cloud server. For example, the image processing server 300 can appropriately perform various image processing tasks in the system 10. For example, the image processing server 300 is equipped with a processor such as a CPU or GPU, memory such as ROM or RAM, and various input / output interfaces. The system 10 does not necessarily have to include the image processing server 300.

[0018] 1.1. Digital Camera Configuration The configuration of the digital camera 100 in this embodiment will be explained with reference to Figure 2.

[0019] Figure 2 is a diagram illustrating the configuration of the digital camera 100 in this system 10. The digital camera 100 is an example of an imaging device in this embodiment. The digital camera 100 in this embodiment includes an image sensor 115, an image processing engine 120, a display monitor 130, and a control unit 135. Furthermore, the digital camera 100 includes a buffer memory 125, a card slot 140, a flash memory 145, an operation unit 150, a communication module 155, a microphone 160, and a speaker 170. The digital camera 100 also includes, for example, an optical system 110 and a lens drive unit 112.

[0020] The optical system 110 includes a focus lens, a zoom lens, an optical image stabilization (OIS) lens, an aperture, a shutter, etc. The focus lens is a lens for changing the focus state of the subject image formed on the image sensor 115. The zoom lens is a lens for changing the magnification of the subject image formed by the optical system. The focus lens, etc., are each composed of one or more lenses.

[0021] The lens drive unit 112 drives the focus lens and other components in the optical system 110. The lens drive unit 112 includes a motor and moves the focus lens along the optical axis of the optical system 110 based on the control of the control unit 135. The configuration for driving the focus lens in the lens drive unit 112 can be implemented using a DC motor, stepping motor, servo motor, or ultrasonic motor, etc.

[0022] The image sensor 115 captures an image of a subject formed through the optical system 110 and generates imaging data. The imaging data constitutes image data representing the image captured by the image sensor 115. The image sensor 115 generates image data of a new frame at a predetermined frame rate (e.g., 30 frames / second). The timing of image data generation and the operation of the electronic shutter in the image sensor 115 are controlled by the control unit 135. The image sensor 115 can use various image sensors, such as a CMOS image sensor, a CCD image sensor, or an NMOS image sensor.

[0023] The image sensor 115 performs operations such as capturing still images and capturing through images. Through images are mainly moving images and are displayed on the display monitor 130 for the user to determine the composition for capturing still images. Through images and still images are examples of captured images in this embodiment. The image sensor 115 is an example of the imaging unit in this embodiment.

[0024] The digital camera 100 of this embodiment may have an image plane phase-difference distance measuring function. For example, the image sensor 115 includes sensor pixels that constitute distance measuring points of the image plane phase-difference method. For example, the sensor pixels include a photoelectric conversion unit that is divided to form two types of optical images divided into pupils in the optical system 110. Such sensor pixels may be provided on the image sensor 115 as light-shielding pixels separate from the pixels for the RGB image, or they may be used in conjunction with the pixels for the RGB image.

[0025] The image processing engine 120 performs various processes on the imaging data output from the image sensor 115 to generate image data, and also performs various processes on the image data to generate an image for display on the display monitor 130. Examples of various processes include, but are not limited to, white balance correction, gamma correction, YC conversion, electronic zoom, compression, and decompression. The image processing engine 120 may be composed of hardwired electronic circuits, or it may be composed of a microcomputer or processor using a program.

[0026] In this embodiment, the image processing engine 120 includes a distance measuring unit 122 that implements, for example, an image plane phase-difference method distance measuring function. The distance measuring unit 122 performs image plane phase-difference method distance measurement based on sensor signals input from sensor pixels in the image sensor 115 and generates a depth map that shows the depth from the digital camera 100 to the subject in the captured image for each pixel. Image plane phase-difference method distance measurement can be performed, for example, by calculating a defocus amount, etc., corresponding to the difference between two types of optical images due to pupil division from the sensor signal for each distance measuring point by the sensor pixels.

[0027] The distance measuring unit 122 is not particularly limited to the image plane phase-detection method. In this case, the image sensor 115 of the digital camera 100 does not necessarily have image plane phase-detection sensor pixels. For example, the distance measuring unit 122 may generate a depth map by performing DFD (Depth From Defocus) calculation processing based on the difference in the amount of blur between frames. Alternatively, various distance measuring methods such as TOF (Time Of Flight), rangefinder, twin-lens stereo distance measuring, color-based distance measuring, or depth estimation using artificial intelligence such as machine learning may be applied to the distance measuring unit 122.

[0028] The display monitor 130 is an example of a display unit that displays various information. For example, the display monitor 130 displays an image (through image) represented by image data captured by the image sensor 115 and processed by the image processing engine 120. The display monitor 130 also displays a menu screen or the like for the user to make various settings for the digital camera 100. The display monitor 130 can be made of, for example, a liquid crystal display device or an organic EL device.

[0029] The operation unit 150 is a general term for the hard keys and user interface such as operation buttons and operation levers provided on the exterior of the digital camera 100, and accepts operations from the user. The operation unit 150 includes, for example, a shutter release button, a mode dial, and a touch panel. When the operation unit 150 accepts an operation from the user, it transmits an operation signal corresponding to the user operation to the control unit 135.

[0030] The control unit 135 provides overall control over the operation of the digital camera 100. The control unit 135 includes a CPU, and the CPU executes programs (software) to realize predetermined functions. Instead of a CPU, the control unit 135 may include a processor consisting of dedicated electronic circuits designed to realize predetermined functions. In other words, the control unit 135 can be realized with various processors such as a CPU, MPU, GPU, DSP, FPGA, and ASIC. The control unit 135 may consist of one or more processors. Alternatively, the control unit 135 may be configured on a single semiconductor chip together with the image processing engine 120, etc.

[0031] The buffer memory 125 is a recording medium that functions as work memory for the image processing engine 120 and the control unit 135. The buffer memory 125 is implemented using DRAM (Dynamic Random Access Memory) or the like. The flash memory 145 is a non-volatile recording medium. Although not shown in the diagram, the control unit 135 may also have various internal memories, such as built-in ROM. The ROM stores various programs that the control unit 135 executes. The control unit 135 may also have built-in RAM that functions as a work area for the CPU.

[0032] The card slot 140 is a means for inserting a removable memory card 142. The card slot 140 can electrically and mechanically connect to the memory card 142. The memory card 142 is an external memory equipped with recording elements such as flash memory. The memory card 142 can store data such as image data generated by the image processing engine 120.

[0033] The communication module 155 is a module (circuit) that connects to an external device according to a predetermined communication standard in wired or wireless communication. The predetermined communication standard includes, for example, USB, HDMI®, IEEE 802.11, Wi-Fi, Bluetooth, etc. The digital camera 100 can communicate with other devices via the communication module 155.

[0034] The microphone 160 includes, for example, one or more microphone elements built into the digital camera 100. The microphone 160 outputs an audio signal indicating the picked-up sound to the control unit 135. In the digital camera 100, an external microphone may be used. The digital camera 100 may be provided with a connection part such as a terminal for connecting to an external microphone, in place of or in addition to the built-in microphone 160.

[0035] The speaker 170 includes, for example, one or more speaker elements built into the digital camera 100, and outputs sound to the outside of the digital camera 100 under control from the control unit 135. In the digital camera 100, an external speaker or earphone may be used. The digital camera 100 may also be provided with a connection part for connecting to an external speaker or the like, in place of or in addition to the built-in speaker 170.

[0036] 1.2. Configuration of the image editing terminal The configuration of the image editing terminal 200 in this embodiment will be explained with reference to Figure 3.

[0037] Figure 3 is a diagram illustrating the configuration of an image editing terminal 200. The image editing terminal 200 is an example of an image processing device composed of, for example, a personal computer (PC), a tablet terminal, or a smartphone. The image editing terminal 200 illustrated in Figure 3 comprises a control unit 210, a storage unit 220, an operation unit 230, a display unit 240, a communication unit 250, a microphone 260, and a speaker 270.

[0038] The control unit 210 includes, for example, a CPU or MPU that works in cooperation with software to realize predetermined functions. The control unit 210 controls, for example, the overall operation of the image editing terminal 200. The control unit 210 reads data and programs stored in the storage unit 220, performs various calculations, and realizes various functions.

[0039] The control unit 210 executes a program that includes a set of instructions for realizing each of the above functions. This program may be provided via a communication network such as the Internet, or it may be stored on a portable recording medium. The control unit 210 may also be a dedicated electronic circuit or a hardware circuit such as a reconfigurable electronic circuit designed to realize each of the above functions. The control unit 210 may be composed of various semiconductor integrated circuits such as a CPU, MPU, GPU, GPGPU, TPU, microcontroller, DSP, FPGA, and ASIC.

[0040] The memory unit 220 is a storage medium that stores programs and data necessary to realize the functions of the image editing terminal 200. As shown in Figure 3, the memory unit 220 includes a storage unit 221 and a temporary storage unit 222.

[0041] The storage unit 221 stores parameters, data, and control programs, etc., for realizing predetermined functions. The storage unit 221 is composed of, for example, an HDD or an SSD. For example, the storage unit 221 stores the above-mentioned programs and various image data, etc.

[0042] The temporary storage unit 222 is composed of RAM such as DRAM or SRAM, and temporarily stores (i.e., holds) data. For example, the temporary storage unit 222 holds image data that is being edited. The temporary storage unit 222 may also function as a work area for the control unit 210, or it may be composed of a storage area in the internal memory of the control unit 210.

[0043] The operation unit 230 is a general term for the operating components that the user operates. The operation unit 230 is, for example, a touch panel superimposed on the display unit 240 for inputting various touch operations, and is an example of an input unit for the image editing terminal 200. The input unit may also be a connection software unit that communicates with various external input devices and receives operation signals. The operation unit 230 may be a physical button or switch provided on the image editing terminal 200, or a keyboard, mouse, or touchpad may be used. The operation unit 230 may also be various GUIs such as virtual buttons, icons, cursors, software keyboards, and objects displayed on the display unit 240.

[0044] The display unit 240 is composed of, for example, a liquid crystal display or an organic EL display. The display unit 240 may display various types of information, such as various GUIs for operating the operation unit 230 and information input from the operation unit 230.

[0045] The communication unit 250 is a module (circuit) that connects to an external device according to a predetermined communication standard in wired or wireless communication. The predetermined communication standard includes, for example, USB, HDMI, IEEE802.11, Wi-Fi, Bluetooth, etc. The communication unit 250 may also connect the image editing terminal 200 to a communication network such as the Internet. The communication unit 250 is an example of an acquisition unit that receives various information from an external device or a communication network.

[0046] The microphone 260 includes, for example, one or more microphone elements built into the image editing terminal 200. The microphone 260 outputs an audio signal indicating the picked-up sound to the control unit 210. The image editing terminal 200 may be provided with connection parts such as terminals for connecting to an external microphone, in place of or in addition to the built-in microphone 260.

[0047] The speaker 270 includes, for example, one or more speaker elements built into the digital camera 100, and outputs sound to the outside of the image editing terminal 200 under control from the control unit 210. The image editing terminal 200 may also be equipped with a connection part for connecting to an external speaker or earphones, etc., in place of or in addition to the built-in speaker 270.

[0048] The configuration of the image editing terminal 200 described above is just one example, and the configuration of the image editing terminal 200 is not limited to this. For example, the display unit 240 of the image editing terminal 200 may use various display devices such as a projector and a head-mounted display. Also, for example, when using an external display device, the display unit 240 of the image editing terminal 200 may be an output interface circuit for video signals such as those conforming to the HDMI standard.

[0049] 2.Operation The operation of the imaging system 10, configured as described above, will be explained below.

[0050] 2.1. Overview of Operation The operation of the imaging system 10 of this embodiment will be explained using Figure 4.

[0051] Figure 4(A) illustrates an example of an image 21 captured by the digital camera 100 of this system 10. The image exemplified in Figure 4(A) shows a subject 20, such as a person, in the shooting environment of the digital camera 100. In this embodiment, when a user takes a video or still image of a desired subject 20 with the digital camera 100, the imaging system 10 performs image synthesis using the image editing terminal 200 to change the background of the subject 20 in the captured image 21. Note that the subject 20 in this system 10 is not limited to a person, but can be any subject as appropriate.

[0052] Figure 4(B) illustrates a composite image 23 based on the captured image 21 in Figure 4(A). In the composite image 23 illustrated in Figure 4(B), the background has been replaced from Figure 4(A), and the lighting of the subject 20 has been changed by relighting according to the new background. This system 10 provides a solution to the problems of the prior art that the inventors have found regarding relighting in such image synthesis. First, these findings of the inventors will be explained.

[0053] Conventional relighting techniques (e.g., Non-Patent Documents 1-2) can reproduce the reflection of light on the surface of the subject 20 by formulating information including the surface irregularities of the subject 20. In this way, shading, where the brightness gradually decreases on the surface, can be reproduced, but conventional relighting techniques have difficulty reproducing shadows formed by light occlusion (e.g., self-shadows of the subject 20), and as a result, the inventors have found through diligent research that this leaves an unnatural appearance.

[0054] Therefore, the inventors diligently conducted research to resolve the problems of conventional rewriting technology, and as a result, created the imaging system 10 of this embodiment. The processing results by this system 10 are illustrated in Figure 4(C).

[0055] Figure 4(C) illustrates the composite image 23 resulting from the correction of Figure 4(B) in the system 10. The system 10 performs a correction on the relit composite image 22 (Figure 4(B)) by adding a shadow region R1 that indicates the shadow where light from the light source in the background is blocked. As a result, for example, in the examples of Figures 4(B) and (C), the shadow region R1 caused by the left arm of the subject 20 is not reproduced in the relit composite image 22 (Figure 4(B)), but is reproduced in the corrected composite image 23, as illustrated in Figure 4(C).

[0056] As described above, this system 10 can complement the expression of shadows that are difficult to reproduce with conventional relighting techniques, and can improve the naturalness of the image rendering in image synthesis compared to conventional techniques. The operation of this system 10 will be explained in detail below.

[0057] 2.2. Operation Details The overall operation of the imaging system 10 of this embodiment will be explained with reference to Figure 5.

[0058] Figure 5 is a flowchart illustrating the operation of the image editing terminal 200 in this system 10. Each process in the flowchart illustrated in Figure 5 is executed, for example, by the control unit 210 of the image editing terminal 200.

[0059] In this system 10, for example, the control unit 210 of the image editing terminal 200 receives image data indicating the result of capturing an image of the subject 20 by the digital camera 100 (S1). For example, through data communication between the digital camera 100 and the image editing terminal 200, the control unit 210 receives image data indicating the captured image 21 from the digital camera 100 via the communication unit 250 (S1).

[0060] In the digital camera 100, the control unit 135, for example, in response to user operation, causes the image sensor 115 to perform an imaging operation to capture an image of a subject, and causes the image processing engine 120 to generate image data of the shooting result based on the generated imaging data. For example, the image processing engine 120, acting as a distance measuring unit 122, generates a depth map that includes the depth of various positions in the image indicated by the imaging data, and includes it in the metadata of the image data of the shooting result. For example, in step S1, the control unit 135 transmits this image data of the shooting result to the image editing terminal 200 via the communication module 155. This image data of the captured image 21 is an example of the first image data in this embodiment. The depth map of the digital camera 100 may be managed in association with the image data of the shooting result in various ways, not limited to metadata.

[0061] Furthermore, the control unit 210 performs processing to prepare a new background image (i.e., background image) for image synthesis (S2). In the background preparation process (S2) of this embodiment, information regarding the light source in the new background environment is also acquired. Details of the processing in step S2 will be described later.

[0062] Next, the control unit 210 generates a composite image 22 based on the image data acquired in steps S1 and S2, respectively, as shown in Figure 4(B), for example (S3). In this embodiment, the image synthesis process (S3) can efficiently perform rewriting image processing in addition to replacing the background of the captured image 21 (Figure 4(A)). Details of the process in step S3 will be described later.

[0063] Next, the control unit 210 corrects the composite image 22 to reflect the shadows corresponding to the light sources in the new background environment (S4). In the shadow correction process (S4) of this embodiment, the shadow region R1 is drawn on the composite image 22 using information on the light sources of the new background environment and the depth map taken when the digital camera 100 captures the image, as illustrated in Figure 4(C). Details of the process in step S4 will be described later.

[0064] Next, the control unit 210 displays the shadow-corrected composite image 23 (Figure 4(C)) on the display unit 240, for example (S5). After outputting the shadow-corrected image, such as in step S5, the control unit 210 terminates the process shown in the flow chart of Figure 5.

[0065] Through the above processing, the system 10 can more easily achieve a natural appearance of the subject 20 in the composite image 23 by performing shadow correction processing (S4) that includes the shadow region R1, in addition to relighting in the image synthesis processing (S3). Furthermore, since the shadow correction processing (S4) for reproducing the shadow of the subject 10 is performed in the system 10, a relighting technique with reduced processing load can be used in the image synthesis processing (S4), for example.

[0066] The processing of this system 10 described above may be performed in real time, for example, when the digital camera 1 is shooting a video or still image upon receiving data from the digital camera 100 (S1), or it may be performed after the completion of such image capture.

[0067] Furthermore, the processing in step S1 is not limited to data communication between the digital camera 100 and the image editing terminal 200, but may also be performed via data communication with various external storage devices. For example, the user may input image data of the shooting result from the digital camera 100 to the image editing terminal 200 via a recording medium such as a memory card 142, or it may be done via an image processing server 300 or the like.

[0068] Furthermore, in addition to or instead of step S5, the system 10 may perform various outputs on the shadow-corrected image data from step S4. For example, the control unit 210 may store the corrected image data in the storage unit 220 or distribute it via the communication unit 250.

[0069] 2.2.1. Background preparation process The details of the background preparation process in step S2 of Figure 5 will be explained using Figures 6 and 7.

[0070] First, the control unit 210 acquires image data (S11) that shows the background environment corresponding to the area around the subject 20 in the new background after synthesis. The background environment data 30 acquired in step S11 will be explained using Figures 7(A) and 8.

[0071] Figure 7(A) illustrates background environment data 30 in the system 10. Figure 8 is a diagram illustrating the background environment in the system 10. Figure 8 illustrates a predetermined subject position P1 in the three-dimensional space of the background environment, a celestial sphere 35 that spherically surrounds the subject position P1 in all directions across the three-dimensional directions (X, Y, Z), and a cube 36 that surrounds the celestial sphere 35. The subject position P1 is set in advance, for example, in the background environment corresponding to the position of the composited subject 20.

[0072] The background environment data 30, as shown in Figures 8 and 7(A), for example, represents an omnidirectional image of the background environment viewed from the subject position P1 across the entire celestial sphere 35. This omnidirectional image of the background environment can be projected onto the inner surface of the celestial sphere 35, for example, to associate each position on the celestial sphere 35 with each pixel of the background environment data 30. For example, the background environment data 30 is composed in a cubemap format in which the omnidirectional image is projected onto the inner wall of a sufficiently large cube 36 surrounding the subject position P1. The background environment data 30 is an example of the second image data in this embodiment.

[0073] Furthermore, the dynamic range of the background environment data 30 may employ, for example, high dynamic range (HDR). For example, the background environment data 30 may be created by performing HDR photography using an omnidirectional camera or the like in a real background environment. Also, the background environment data 30 may represent a virtual background environment, not just a real background environment. The background environment data 30 may consist of computer graphics (CG) images created using computer graphics (CG) technology.

[0074] In this system 10, background environment data 30 can be stored in advance in a database managed by, for example, the image processing server 300. In step S11, the control unit 210 of the image editing terminal 200 accesses the image processing server 300, for example via a communication network, and receives the background environment data 30 from the image processing server 300 via the communication unit 250 (S11).

[0075] Next, the control unit 210 sets a background image representing the background in the composite image 22 (Figure 4(B)) based on the background environment data 30 (S12). The process in step S12 will be explained using Figures 7(B) and 8.

[0076] Figure 7(B) illustrates a background image 31 set from the background environment data 30 in Figure 7(A). In Figure 8, the field of view range and its reference position P2 corresponding to the background image 31 are illustrated on the celestial sphere 35 corresponding to the omnidirectional image of the background environment data 30. The reference position P2 corresponds, for example, to the center position of the background image 31.

[0077] For example, in step S12, the control unit 210 displays the omnidirectional image indicated by the background environment data 30 on the display unit 240 and accepts a user operation on the operation unit 230 to select a desired range from the omnidirectional image. Furthermore, the control unit 210 generates a background image 31 by, for example, extracting an image of the selected field of view range from the background environment data 30 and performing projection transformations as necessary (S12).

[0078] Furthermore, the control unit 210 extracts light source data of the background environment based on the background environment data 30 (S13). The process of step S13 will be explained with reference to Figures 7(C) and 8.

[0079] Figure 7(C) shows an example image of background environment light source data 32 extracted from background environment data 30 in Figure 7(A). For example, from the viewpoint of processing efficiency, this system 10 employs a low-resolution version of image-based lighting, in which each pixel of the omnidirectional image of the background environment data 30 is considered to be a light source placed on the celestial sphere 35 at infinity. For example, the light source data 32 obtained by extracting the low-frequency components from the background environment data 30 in Figure 7(A) as shown in Figure 7(C) can be compressed in terms of data size, etc., by encoding using the expansion of the background environment data 30 with basis functions such as spherical harmonics, as shown in equation (1) below.

number

[0080] In equation (1) above, F(θ,φ) on the left side represents the omnidirectional image of 30 background environment data points per RGB color. The spherical harmonic function Y on the right side represents the spherical harmonic function. l,m (θ,φ) is defined by the angular position (θ,φ) as an argument and the index (l,m) for identifying the basis function. The angular position (θ,φ) corresponds to a position on the celestial sphere 35 and is defined, for example, with respect to the reference position P2.

[0081] In equation (1) above, the SH coefficient F l,m This is the spherical harmonic function Y of the corresponding index (l,m).l,m It is the expansion coefficient of the omnidirectional image (color component F(θ, φ)) by (θ, φ). The right side of the above equation (1) is the spherical harmonic function Y for each such index (l, m). l,m (θ, φ) and the SH coefficient F l,m For the product with, the sum is taken over the range of integers l from "0" to "N" and integers m from "-l" to "l". "N" is a predetermined order and is appropriately set from the viewpoints such as the accuracy of the approximation by the above expansion (for example, N = 2).

[0082] In step S13, the control unit 210 calculates the orthogonal expansion of the background environment data 30 by the spherical harmonic function Y of each index (l, m) up to a predetermined order N, and calculates the corresponding SH coefficient F l,m respectively. For example, the control unit 210 calculates a set of 9 (= 1 + 3 + 5) SH coefficients F l,m for each of the RGB colors with N ≤ 2 as the light source data 32 (S13). According to such a set of SH coefficients F l,m of the light source data 32, the image of the light source as shown in FIG. 7(C) can be restored by the corresponding spherical harmonic function Y l,m (θ, φ). Note that such a cube map image has a weight in each pixel with a solid angle according to the projection relationship between the cube 36 and the celestial sphere 35 for image-based lighting. l,m After extracting the light source data 32 as described above, for example, the control unit 210 ends the background preparation process (S2) and proceeds to step S3 in FIG. 5.

[0083] According to the above background preparation process (S2), the system 10 can prepare the background image 31 to be synthesized and the light source data 32 of such a background environment based on the background environment data 30 (S12 to S13). The system 10 can prepare, for example, the light source data 32 of a background environment that can reduce the processing load of the image processing of lighting. <{

[0084]

[0085] ​In this system 10, the background environment data 30 is not limited to a cubemap format, but may also be in equirectangular format, hyperdome master format, or dome master format. Furthermore, the background environment data 30 is not limited to an all-around image, but may also represent, for example, a hemispherical or panoramic (i.e., all-around) image on a horizontal plane.

[0086] Furthermore, the acquisition of background environment data 30 (S11) is not limited to the image processing server 300, but may also be performed from outside the system 10 via data communication as appropriate. Alternatively, the background environment data 30 may be stored in advance in the storage unit 221 of the image editing terminal 200, and in step S11, the control unit 210 may read the background environment data 30 from the storage unit 221 to the temporary storage unit 222.

[0087] Furthermore, for example, in step S11, the subject position P1 (Figure 8) in the background environment data 30 may be set by user operation. For example, the control unit 210 may accept such user operation at the operation unit 230. Also, in step S12, the range in which the background image 31 can be set in the background environment data 30 may be limited. For example, the control unit 21 may refer to the depth map corresponding to the captured image 21 and perform such range limitations from the viewpoint of being able to add a natural shadow area R1.

[0088] 2.2.2. Image Synthesis Processing The details of the image synthesis process in step S3 of Figure 5 will be explained using Figures 9 and 10.

[0089] First, the control unit 210 extracts an image of the region to be combined from, for example, the image 21 (Figure 4(A)) captured by the digital camera 100 (S21). The process of step S21 will be explained using Figures 10(A) and 10(B).

[0090] Figure 10(A) illustrates a mask image 40 for the captured image 21 in Figure 4(A). Figure 10(B) illustrates a subject image 41 extracted from the captured image 21 in Figure 4(A) using the mask image 40 in Figure 10(A). The mask image 40 includes a subject region R2 in the captured image 21 that indicates the target of extraction in step S21, and a mask region R3 that indicates the area not to be extracted.

[0091] In step S21, the control unit 210 first inputs the captured image 21 (Figure 4(A)) to, for example, a pre-trained image segmentation model, and generates a mask image 40 as illustrated in Figure 10(A) from the output of the image segmentation model. The image segmentation model can be realized by known techniques such as semantic segmentation and alpha matting (e.g., Non-Patent Document 1). For example, the image segmentation model can be obtained by performing machine learning on a deep neural network (DNN) including a convolutional neural network (CNN) and a deconvolutional neural network (DCNN) to determine the region in the image where a set subject, such as a person, is present.

[0092] Furthermore, the control unit 210 multiplies the generated mask image 40 (Figure 10(A)) and the captured image 21 (Figure 4(A)) pixel by pixel to generate the cropped subject image 41, as illustrated in Figure 10(B) (S21). For example, the mask image 40 has a pixel value of "1" in the subject region R2 and a pixel value of "0" in the mask region R3. The mask image 40 may also have an alpha value to partially make the subject image 41 semi-transparent, for example, near the boundary of the subject region R2. The image data of the subject image 41 may be an example of the first image data.

[0093] Next, the control unit 210 analyzes various characteristics related to the lighting of the subject 20 based on, for example, the extracted subject image 41 (S22). The processing in step S22 is performed as a preprocessing step for relighting the subject image 41 (S23) in order to remove the influence of the original lighting, such as the light source of the shooting environment, from the subject image 41. The processing in step S22 will be explained using Figure 10(C).

[0094] Figure 10(C) illustrates various analysis data 42-44 showing the analysis results of the subject image 41 in Figure 10(B) in step S22. The analysis data 42-44 include, for example, light source data 42 of the subject image 41, an albedo map 43, and a light transfer map 44.

[0095] The light source data 42 indicates the light source in the environment in which the subject image 41 (and consequently the captured image 21) was taken. The light source data 42 of the shooting environment is calculated as a set of SH coefficients having indices (l,m) of a predetermined order N or less, similar to the light source data 32 of the background environment (Figure 7(C)).

[0096] The albedo map 43 has, for example, for each RGB color, the reflectance at each pixel position on the subject 20 in the subject image 41 as a pixel value. The albedo map 43 corresponds to an image showing the colors of the subject 20 after removing the effects of lighting under various environmental conditions, such as the light source of the shooting environment, from the subject image 41.

[0097] The light transfer map 44 displays, pixel by pixel, information about the light transfer function that indicates how the subject 20 transmits reflected light according to the incident light and reflectance at various positions on the subject 20 in the subject image 41. The light transfer function includes information such as the normal direction at various positions on the subject 20, and a visibility function that indicates whether incident light from a specific direction is allowed at that position. For example, the light transfer map 44 has sets of SH coefficients of an index (l,m) of a predetermined order N or less for each RGB, as pixel values ​​into which the light transfer function information is encoded.

[0098] According to the analysis data 42-44 described above, for example, the subject image 41 can be approximated by the following equation (2). This approximation is based on assumptions such as neglecting specular reflection, assuming that diffuse reflection is Lambertian reflection, and focusing only on low-frequency components in image-based lighting.

number

[0099] In equation (2) above, the image coordinates (x,y) indicate the pixel position, and G(x,y) on the left side indicates the color components of the subject image 41. On the right side of equation (2) above, the SH coefficient G of the light source data 42 at a common index (l,m) in a range of a predetermined order N or less (i.e., l=0 to N and m=-l to l) is expressed. l,m and the SH coefficient T of the optical transfer map 44 l,m The sum of the products with (x,y) is taken, and then the reflectance ρ(x,y) of the albedo map 43 is multiplied for each pixel.

[0100] In step S22, the control unit 210 inputs the subject image 41 (Figure 4(B)) to, for example, a pre-trained estimation model, and generates the above-mentioned analysis data 42 to 44 from the output of the estimation model. Such an estimation model can be realized by applying known techniques as appropriate (see, for example, Non-Patent Document 2).

[0101] For example, the estimation model in step S22 consists of a DNN that includes an encoder such as a CNN that takes the subject image 41 as input, a CNN that estimates the light source data 42, and a decoder such as a DCNN that estimates each map 43, 44. The machine learning of the estimation model is performed so that the various analysis data 42-44 generated from the input image of a specific subject such as a person, like the subject image 41, can reproduce the input image accurately according to equation (2) above.

[0102] Next, the control unit 210 performs relighting image processing to adjust the brightness of the subject image 41 based on the analysis results of the subject image 41 (S22) and the new background environment light source data 32 (Figure 7(C)) (S23). Step S23 will be explained using Figure 10(D).

[0103] Figure 10(D) illustrates a relit image 45 obtained by performing a relit process (S23) on the subject image 41 in Figure 10(B). In step S23, the control unit 210 performs the calculation in equation (2) above, for example, using the background environment light source data 32 instead of the shooting environment light source data 42. As a result, the control unit 210 can generate a relit image 45 with brightness adjusted to match the background environment light source data 32, as shown in Figure 10(D), for example (S23).

[0104] Next, the control unit 210 generates a composite image 22 (Figure 4(B)) based on the relighted image 45 of the subject 20 obtained in this way and the background image 31 (Figure 7(B)) obtained in the background preparation process (S2) (S24).

[0105] For example, in step S24, the control unit 210 superimposes a layer of the relighting image 45 on the background image 31 and appropriately aligns the subject image 41 on the background image 31. In this superposition, the control unit 210 combines the relighting image 45 and the background image 31 so that the background image 31 is used in the area corresponding to the area outside the subject region R2 of the mask image 40 (Figure 10(A)) (S24). In the image composition of step S24, if an alpha value is set for the mask image 40, alpha blending may be used to make the background image 31 partially visible through the subject image 41.

[0106] Thus, the control unit 210 completes the image synthesis process (S3) in Figure 9 by generating the synthesized image 22 (S23) and proceeds to step S4 in Figure 5.

[0107] According to the image synthesis process (S3) described above, when generating a composite image 22 of the subject image 41 and the background image 31 (S23), the system 10 can adjust the overall brightness of the subject image 41 to match the background image 31 by relighting (S23).

[0108] The above description illustrates an example of the process (S22) for analyzing the subject image 41 for relighting (S23), but the system 10 is not particularly limited to this. For example, in the estimation model described above, the estimation of light source data 42 may only be performed during machine learning, and may be omitted in step S22. Also, in the system 10, the estimation model for relighting (S23) is not particularly limited to the example above, and various models may be adopted.

[0109] For example, in step S22, the control unit 210 may estimate a depth map and a normal map from the subject image 41 instead of the various analysis data 42 to 44 described above. Also, in step S23, the control unit 210 may generate the rewriting image 45 using a trained model or the like instead of the calculation in equation (2) above.

[0110] Furthermore, the physical modeling of relighting may consider not only Lambertian reflection but also various diffuse or nondiffuse reflections, and specular reflection may be considered using Phong's reflection model, etc. (Non-patent documents 1, 2, etc.). The relighting process (S23) in the image synthesis process (S3) does not necessarily need to reduce the processing load, and various relighting techniques can be appropriately adopted depending on the required image quality accuracy.

[0111] 2.2.3. Shadow Correction Processing The details of the shadow correction process in step S4 of Figure 5 will be explained using Figures 11 and 12.

[0112] First, the control unit 210 acquires information such as a depth map relating to the shape of the subject 20 (S31). The depth map in step S31 is an example of subject information in this embodiment.

[0113] For example, in step S31, the control unit 210 reads out the depth map corresponding to the captured image 21 (Figure 4(A)) by referring to the metadata in the image data of the captured image 21 (Figure 4(A)) from the digital camera 100. Furthermore, the control unit 210 extracts the depth map for the subject area R2 from the depth map corresponding to the captured image 21 by cropping using the mask image 40, for example, in step S21 in Figure 9 (S31).

[0114] Furthermore, the control unit 210 determines the direction of the light source to be used to reflect the shadow, for example, based on the light source data 32 of the background environment (Figure 7(C)) (S32). The processing in step S32 is performed to narrow down the light sources to be considered for shadow correction, for example, from the viewpoint of sharpening the shadow formed by the subject 20 or reducing the processing load for reflecting the shadow.

[0115] For example, in step S32, the control unit 210 extracts pixels with a predetermined high light intensity from the image of the background environment light source data 32, and determines the direction of the light source for shadow reflection from the angular position (θ,φ) corresponding to the extracted pixels. The predetermined high light intensity may be a light intensity that indicates a reference for a light source that is bright enough to form a shadow, the highest light intensity in the background environment, or a locally peak light intensity. Such light intensity can be determined, for example, in the case of a cubemap, by appropriately weighting the solid angle with the brightness value of the pixels.

[0116] Next, the control unit 210 detects a shadow region R1 (Figure 4(C)) that is expected to be caused by light occlusion on the subject 20, for example, based on the depth map acquired in step S31 and the light source direction determined in step S32 (S33). The process in step S33 will be explained with reference to Figure 12.

[0117] Figure 12 is a diagram illustrating the shadow correction process (S4) of this embodiment. Figure 12 illustrates the shadow region R1 formed by the depth map 50 in step S31 and the light source direction D1 in step S32. The depth map 50 in step S31 includes, for example, the depth z of the subject 20 as seen from the digital camera 100 at the time of shooting as a pixel value.

[0118] In this system 10, the shadow region R1 is formed on the extension of the light ray 55 that is incident on the surface of the subject 20 from the light source direction D1 and is blocked, as shown in Figure 12, for example. In step S33, the control unit 210 can detect, for example, on the relit composite image 22 (Figure 4(B)) the set of pixels that are on the extension of the light ray 55 incident on the subject 20 from the light source direction D1 as the shadow region R1.

[0119] The detection of the shadow region R1 (S33) can be performed by applying various methods, such as the so-called ray tracing method, depth shadow method, or shadow volume method. The processing in step S33 may also use a depth map as seen from the light source direction D1, i.e., a shadow map. For example, the control unit 210 may calculate a shadow map from the depth map 50 at the time of shooting by performing a coordinate transformation according to triangulation. In such a shadow map, the shadow region R1 is superimposed on the part that is occluded on the subject 20, and for example, the depth of the occluded part is stored at the position of the shadow region R1. The control unit 210 may detect the shadow region R1 based on the depth shift in such a shadow map (S33).

[0120] Next, the control unit 210, based on the shadow region R1 detected in this way, corrects the composite image 22 produced by the image synthesis process (S3) to include the shadow region R1, as shown in Figures 4(B) and (C), and generates corrected image data (S34).

[0121] For example, in step S34, the control unit 210 generates a corrected composite image 23 as illustrated in Figure 4(C) by drawing the detected shadow region R1 in the composite image 22 of Figure 4(B). Drawing the shadow region R1 can be done, for example, by adjusting the pixel value to be darker for each pixel located in the shadow region R1 in the composite image 22. In this way, the control unit 210 adds the shadow region R1 to the composite image 22 to generate image data showing the shadow-corrected composite image 23 (S34). The image data in step S34 is an example of composite image data in this embodiment.

[0122] The control unit 210 terminates the shadow correction process (S4) in Figure 11 by generating image data of the corrected composite image 23 (S34), and proceeds to step S5 in Figure 5, for example.

[0123] According to the shadow correction process (S4) described above, the system 10 generates image data of the shadow correction result (S34) such that the shadow region R1 caused by the occlusion of light from the light source direction D1 of the background environment is included in the composite image 23, based on the light source data 32 of the background environment. As a result, the system 10 can further reflect the shadow region R1 caused by the light source of the background environment in the composite image 22 (Figure 4(B)) that has been relit according to the background environment, making it easier to obtain natural shading in the corrected composite image 23 (Figure 4(C)).

[0124] In this system 10, according to the depth map 50 as seen from the digital camera 100, a shadow region R1, i.e., a self-shadow, can be detected on the subject 20 depending on its shape (S33). The shadow region R1 in this system 10 is not limited to this, and may also be a cast shadow formed on the background image 31, for example.

[0125] For example, in step S31, the control unit 210 may acquire a depth map of the background environment in addition to the depth map 50 of the subject 20. For example, the system 10 may manage information indicating the depth of the background environment in a database of background environment data 30, and the processing in step S31 may be performed by referring to such a database. In step S33, the control unit 210 can further use the depth map of the background environment to detect shadow areas that the subject 20 casts on the background by blocking light to the background, and correct the composite image 22 to include these shadow areas (S34).

[0126] The acquisition of the depth map 50 of the subject 20 (S31) is not limited to the above example; for example, the control unit 210 may perform depth map estimation processing. This depth map estimation processing may be performed, for example, by image analysis of the subject image 41 using a trained model, or by using various information from when the image was captured from the digital camera 100. For example, the digital camera 100 may include various information such as image plane phase difference information or blur information in the metadata of the image data, and the control unit 210 may perform the same processing as the distance measuring unit 122 from this metadata.

[0127] Furthermore, in this system 10, the depth map estimation process described above is not limited to the depth map 50 of the subject 20 as seen from the digital camera 100. For example, the control unit 210 may generate a depth map as seen from the light source direction D1, or generate a depth map of the background environment, using a trained model of such estimation processing. Also, in this system 10, the depth map estimation process may use multiple frame images or image recognition of the subject 20. For example, the control unit 210 may interpolate the depth of parts of the subject 20 that are blind spots in one frame using estimation results from other frame images.

[0128] Furthermore, in the shadow correction process (S4) described above, the determination of the light source direction D1 (S32) may be performed using background environment data 30 instead of, or in addition to, the light source data 32 of the background environment. Also, the process in step S32 may be performed in the background preparation process (S2). The light source direction D1 for shadow reflection in step S32 is an example of light source information of the background environment. Such a light source direction D1 for shadow reflection may be one direction or multiple directions.

[0129] In the shadow correction process (S4) of this system 10, for example, in determining the light source direction D1 (S32), a higher-order SH coefficient F is used than during relighting (S23). l,m This can also be used (for example, N=5, 36 for each color). This reduces the processing load because the diffuse light of relighting can be sufficiently reproduced even in low dimensions, while improving the reproduction of hard shadows in shadowing. As described above, the accuracy of the light source information during relighting and shadowing may be changed considering the processing load and computational complexity of this system 10.

[0130] Furthermore, in the above description, a ray 55 parallel to the light source direction D1 was used for detecting the shadow region R1 (S33). This system 10 may also detect the shadow region R1 in a background environment where various light sources such as point light sources, spot light sources, or area light sources are arranged, not just parallel light sources (S33). The control unit 210 can generate corrected image data so that the shadow region R1 formed by these various light sources is included in the composite image 23, similar to the example above (S34).

[0131] Furthermore, in the detection of the shadow region R1 (S33), if the shape of the shadow region R1 is unknown due to blind spots in the depth map of the subject 20, the control unit 210 may interpolate the shadow region R1 as appropriate or estimate its shape and then draw the shadow region R1 (S34). Alternatively, the control unit 210 may perform drawing such as blurring part or all of the contour of the shadow region R1 (S34).

[0132] Furthermore, in this system 10, the detection and rendering of the shadow region R1 (S33, S34) do not necessarily have to be performed separately, but may be performed simultaneously. The control unit 210 may adjust the pixel value when rendering each pixel depending on whether or not the pixel falls within the shadow region R1. Various methods such as ambient occlusion or screen-based ambient occlusion may be applied to the shadow correction processing (S4) of this system 10.

[0133] 3. Summary As described above, in the imaging system 10 of this embodiment, an image editing terminal 200, which is an example of an image processing device, comprises a communication unit 250, which is an example of an input unit, and a control unit 210. The communication unit 250 inputs image data of a captured image 21 as an example of first image data showing an image of a subject taken by a digital camera 100, which is an example of an imaging device (S1). The control unit 210 generates composite image data by combining the subject image 41 and the background image 31 based on the first image data input to the communication unit 250 and background environment data 30, which is an example of second image data showing a predetermined background image 31. Based on the second image data, the control unit 210 acquires light source data 30, which is an example of light source information showing a light source in the background image 31 (S2). Based on the light source information of the background image 31, the control unit 210 generates composite image data in the composite image 23 in which the subject image 41 is placed on the background image 31, including a shadow region R1 showing a shadow corresponding to the light source by the subject 20 shown in the subject image 41 (S4).

[0134] According to the image editing terminal 200 described above, the shadow region R1 based on the light source information of the background image 31 is included in the composite image 23, making it easier to obtain natural shading when compositing the captured subject 20 and the background image.

[0135] In this system 10, the control unit 210 performs relighting processing to adjust the brightness of the subject image 41 according to the light source data 32 of the background image 31 (S23). Based on the light source direction D1 in the light source information of the background image 31, the control unit 210 corrects the composite image 22, in which the brightness of the subject image 41 has been adjusted, to include the shadow region R1, and generates composite image data (S4). As a result, in addition to relighting according to the background environment, this system 10 can easily obtain natural shading in the composite image 23 by including the shadow region R1.

[0136] In this system 10, the control unit 210 adjusts the overall brightness of the subject image 41 by relighting according to the light source information of the background environment (S23), and corrects the composite image 22 so that a shadow region R1 is included in a part of the subject image 41 (S4). As a result, local shadows of the subject 20 are corrected separately, which reduces the processing load of relighting that adjusts the entire image of the subject 20, and makes it easier to obtain natural shading in the composite image 23. For example, the relighting process (S23) may be performed at a lower resolution than the background image 31, and the shadow correction process (S4) may be performed at a higher resolution than the relighting process (S23).

[0137] In this system 10, the control unit 210 acquires a depth map 50, which is an example of subject information showing the shape of the subject 20 shown in the subject image 41 (S31). Based on the light source information and subject information of the background image 31, the control unit 210 generates composite image data in the composite image 22, including the shadow region R1 so as to reflect the shape of the subject 20 (S34). As a result, this system 10 can easily obtain natural shading in the composite image 23 using subject information such as the depth map 50.

[0138] In this system 10, the digital camera 100 generates a depth map 50 in association with the first image data when capturing a subject image 41. The control unit 210 acquires subject information from the digital camera 100, for example, via the communication unit 250. As a result, this system 10 can utilize the information captured by the digital camera 100 to easily obtain natural shading in the composite image 23.

[0139] In this system 10, the background image 31 includes at least one of an image of a real background and a computer graphics image. This system 10 makes it easier to obtain natural shading when compositing images onto such background image 31.

[0140] In this system 10, the communication unit 250 of the image editing terminal 200 receives first image data via data communication with an external storage device such as a digital camera 100, a memory card 142, or an image processing server 300 (S1). When the image editing terminal 200 performs image synthesis on the received image data, this system 10 makes it easier to obtain natural shading in the synthesized image 23.

[0141] This embodiment provides an image processing method that is executed by a computer, such as an image editing terminal 200. This method includes the steps of: inputting a first image data showing a subject image 41 taken by a digital camera 100 (S1); acquiring light source information showing a light source in the background image 31 based on a second image data showing a predetermined background image 31 (S13); and generating composite image data in which the subject image 41 and the background image 31 are combined, including a shadow region R1 showing a shadow corresponding to the light source of the subject shown in the subject image 41, in a composite image in which the subject image 41 is placed on the background image 31, based on the light source information of the background image 31 (S4). This method makes it easier to obtain natural shading in the image synthesis of the captured subject 20 and the background.

[0142] In this embodiment, a program is provided to be executed by a processor, such as the control unit 210 of an image editing terminal 200. This program includes the steps of: inputting image data showing a subject image 41 taken by a digital camera 100 (S1); setting a background image 31 according to a predetermined light source (S12); and acquiring composite image data in which the subject image 41 and the background image 31 are combined, including a shadow region R1 showing a shadow corresponding to the light source of the subject shown in the subject image 41, in a composite image in which the subject image 41 is placed on the background image 31 according to the input image data (S4). According to this program, it is possible to easily obtain natural shading in the image synthesis of the captured subject 20 and the background.

[0143] (Other embodiments) As described above, Embodiment 1 has been presented as an example of the technology disclosed in this application. However, the technology in this disclosure is not limited to this and can be applied to embodiments that are modified, substituted, added, or omitted as appropriate. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiment.

[0144] In Embodiment 1 described above, the image editing terminal 200 was described as an example of an imaging system 10 used as an image processing device, but this disclosure is not limited thereto. Such modifications will be explained with reference to Figure 13.

[0145] Figure 13 is a flowchart illustrating the operation of a modified imaging system 10. In this modified example, the image processing device may be integrated with the digital camera 100. Furthermore, the system 10 does not necessarily have to include an image editing terminal 200. In the digital camera 100 of this embodiment (Figure 2), for example, the image processing engine 120 and the control unit 135 may constitute the image processing device. The processing illustrated in Figure 13 is executed, for example, by the control unit 135 of the digital camera 100 controlling the image processing engine 120, etc.

[0146] In the imaging system 10 of Embodiment 1, image data from the digital camera 100 is input to the image editing terminal 200 (S1). In this embodiment, instead of step S1, for example as shown in Figure 13, the control unit 135 causes the image sensor 115 in the digital camera 100 to perform an imaging operation and executes the image capture operation of the subject 20 (S1A). The control unit 135 appropriately holds the image data of the capture result in the buffer memory 125 and inputs it to the image processing engine 120, and performs the processing of steps S2 to S5 in the same manner as in Embodiment 1. For example, the shadow-corrected composite image 22 is displayed on the display monitor 130 of the digital camera 100 (S5). For example, the user of the digital camera 100 can check the composite image 23 in real time when taking an image.

[0147] As described above, in this embodiment, the digital camera 100, an example of an imaging device, comprises an image sensor 115, an example of an imaging unit, and an image processing device consisting of an image processing engine 120 and a control unit 135, etc. The image processing engine 120 and the control unit 135 function as the input unit and control unit of the image processing device, respectively, and generate composite image data based on the image data generated by the image sensor 115 (S1A~S4). This also makes it easier to obtain natural shading in the image synthesis of the captured subject 20 and the background, similar to Embodiment 1.

[0148] Furthermore, in this embodiment, the processing in steps S2 to S4 of Figure 13 is not limited to within the digital camera 100, but may also be implemented, for example, through data communication with an image processing server 300. For example, the digital camera 100 may transmit image data of the shooting result to the image processing server 300 via the communication module 155 and receive processing result data from the image processing server 300 as a response. For example, the image processing server 300 may perform some or all of the background preparation processing (S2), image synthesis processing (S3), and shadow correction processing (S4), and generate image data of the shadow-corrected composite image 23.

[0149] Furthermore, in this embodiment, the image editing terminal 200 may also implement the processing in steps S2 to S4 of Figure 5 by data communication with the image processing server 300, similar to the above. That is, the digital camera 100 or the image editing terminal 200 may acquire composite image data via data communication with the image processing server 300 or internal processing in response to the input of image data resulting from shooting. This also makes it easier to obtain natural shading in the image synthesis of the captured subject 20 and the background, similar to the above embodiments.

[0150] As described above, the image processing device in this embodiment may be implemented as a network-type system constructed by data communication between an image editing terminal 200 or a digital camera 100 and an image processing server 300. In this embodiment, a program that runs on the terminal side, such as the image editing terminal 200 or the digital camera 100, may be provided in such a system. Alternatively, the image processing server may be an example of the image processing device in this embodiment.

[0151] In each of the embodiments described above, an imaging system 10 in which image synthesis processing (S3) and shadow correction processing (S4) are performed sequentially has been described, but the disclosure is not limited thereto. For example, in this embodiment, image synthesis processing (S3) and shadow correction processing (S4) may be performed integrally. In this embodiment, a rewritten and uncorrected composite image 22 is not necessarily generated, and the system 10 may omit such intermediate generation and generate a shadow-corrected composite image 23.

[0152] In each of the embodiments described above, an imaging system 10 that performs relighting processing (S23) in the image synthesis processing (S3) has been described. In this embodiment, relighting processing (S23) is not necessarily required. For example, if the desired naturalness can be obtained by simply adding a shadow region R1 for a light source corresponding to the light source information of the background environment to the synthesized image without performing any relighting, then the various relighting processes (S22 to S23) may be appropriately omitted.

[0153] In the embodiments described above, virtual production was given as an example of application of the imaging system 10, but this disclosure is not particularly limited thereto. For example, the system 10 may be applied to various video production applications other than virtual production, or to background compositing applications such as web conferencing. The subject 20 of the system 10 may be the user of the digital camera 100 themselves.

[0154] Furthermore, in each of the above embodiments, a display monitor 130 was exemplified as an example of the display unit of the digital camera 100. In the digital camera 100 of this embodiment, the display unit is not limited to the display monitor 130, but may also be, for example, an EVF (electronic viewfinder) or an output module that outputs a video signal using the HDMI standard or the like.

[0155] Furthermore, in each of the above embodiments, a digital camera 100 equipped with an optical system 110 was provided as an example. The imaging device in this embodiment does not necessarily have to be equipped with an optical system 110, and may be, for example, a camera with interchangeable lenses.

[0156] Furthermore, while a digital camera was described as an example of an imaging device in each of the embodiments described above, the invention is not limited to this. The imaging device of this disclosure may be any electronic device having an image capture function (e.g., a video camera, smartphone, tablet terminal, etc.). Also, the image processing device of this disclosure may be such an electronic device as described above, or it may be an electronic device that does not have an image capture function in particular.

[0157] (Example of a particular form) The following are examples of various aspects of this disclosure.

[0158] A first aspect of the present disclosure is an image processing apparatus comprising: an input unit that inputs first image data showing a subject image captured by an imaging device; and a control unit that generates composite image data by combining the subject image and a background image based on the first image data input to the input unit and second image data showing a predetermined background image. The control unit acquires light source information indicating a light source in the background image based on the second image data, and generates composite image data based on the light source information of the background image, including a shadow region showing a shadow corresponding to the light source of the subject shown in the subject image in the composite image in which the subject image is placed on the background image.

[0159] In the second embodiment, in the image processing apparatus described in the first embodiment, the control unit adjusts the brightness of the subject image according to the light source information of the background image, and corrects the composite image in which the brightness of the subject image has been adjusted based on the light source information of the background image to include shadow areas, thereby generating composite image data.

[0160] In a third embodiment, in the image processing apparatus described in the first or second embodiment, the control unit adjusts the overall brightness of the subject image according to the light source information and corrects the composite image so that a shadow area is included in a part of the subject image in the composite image.

[0161] In the fourth embodiment, in the image processing apparatus described in any of the first to third embodiments, the control unit acquires subject information indicating the shape of the subject shown in the subject image, and generates composite image data in the composite image, including shadow regions so as to reflect the shape of the subject, based on the light source information of the background image and the subject information.

[0162] In the fifth embodiment, in the image processing apparatus described in the fourth embodiment, the imaging device generates subject information in association with the first image data when capturing a subject image, and the control unit acquires the subject information from the imaging device.

[0163] In the sixth embodiment, in the image processing apparatus described in any of the first to fifth embodiments, the background image includes at least one of an image of a real background and a computer graphics image.

[0164] In the seventh aspect, in the image processing apparatus described in any of the first to sixth aspects, the input unit receives first image data via data communication with an imaging device or an external storage device.

[0165] The eighth embodiment is an imaging apparatus comprising an imaging unit that captures an image of a subject and generates first image data, and an image processing apparatus according to any one of the first to sixth embodiments that generates composite image data based on the first image data generated by the imaging unit.

[0166] The ninth aspect is an image processing method performed by a computer, comprising the steps of: inputting a first image data showing a subject image captured by an imaging device; acquiring light source information indicating a light source in a background image based on a second image data showing a predetermined background image; and generating composite image data in which a subject image and a background image are combined, based on the light source information of the background image, including a shadow region showing a shadow corresponding to the light source from the subject shown in the subject image in a composite image where the subject image is placed on the background image.

[0167] A tenth aspect is a program to be executed by a processor, comprising the steps of: inputting image data showing a subject image captured by an imaging device; setting a background image corresponding to a predetermined light source; and acquiring composite image data in which the subject image and the background image are combined, in a composite image in which the subject image is placed on the background image, and a shadow region showing a shadow corresponding to the light source cast by the subject shown in the subject image.

[0168] As described above, embodiments have been explained as examples of the technology in this disclosure. For this purpose, accompanying drawings and a detailed description have been provided. Therefore, among the components described in the accompanying drawings and detailed description, there may be not only components that are essential for solving the problem, but also components that are not essential for solving the problem, in order to illustrate the above technology. [Industrial applicability]

[0169] This disclosure is applicable to various uses of compositing background and subject images. [Explanation of symbols]

[0170] 10 Imaging System 100 Digital Cameras 115 Image Sensor 120 Image Processing Engines 135 Control Unit 200 Image Editing Terminals 210 Control Unit 220 Storage section 230 Operation section 240 Display section 250 Communications Department 300 Image Processing Servers

Claims

1. An input unit that receives first image data showing a subject image captured by an imaging device, The system includes a control unit that generates composite image data by combining the subject image and the background image based on the first image data input to the input unit and the second image data showing a predetermined background image, The control unit, Based on the second image data, light source information indicating the light source in the background image is obtained. Based on the light source information of the background image, the composite image is generated in a composite image in which the subject image is placed on the background image, including a shadow region showing the shadow cast by the subject shown in the subject image according to the light source. Image processing device.

2. The control unit, The brightness of the subject image is adjusted according to the light source information of the background image. Based on the light source information of the background image, the composite image, in which the brightness of the subject image has been adjusted, is corrected to include the shadow region, thereby generating the composite image data. The image processing apparatus according to claim 1.

3. The control unit, The overall brightness of the subject image is adjusted according to the aforementioned light source information. The composite image is corrected so that the shadow region is included in a part of the subject image in the composite image. The image processing apparatus according to claim 2.

4. The control unit, Obtain subject information indicating the shape of the subject shown in the aforementioned subject image, Based on the light source information of the background image and the subject information, the composite image data is generated in the composite image, including the shadow region so as to reflect the shape of the subject. The image processing apparatus according to claim 1.

5. The imaging device generates subject information in association with the first image data when capturing the subject image. The control unit acquires the subject information from the imaging device. The image processing apparatus according to claim 4.

6. The background image includes at least one of an image of a real background and a computer graphics image. The image processing apparatus according to claim 1.

7. The input unit receives the first image data via data communication with the imaging device or external storage device. The image processing apparatus according to claim 1.

8. An imaging unit that captures an image of a subject and generates the first image data, The image processing apparatus according to claim 1 comprises generating the composite image data based on the first image data generated by the imaging unit. Imaging device.

9. A computer-based image processing method, The steps include inputting a first image data showing a subject image captured by an imaging device, A step of acquiring light source information indicating a light source in a predetermined background image based on a second image data showing the background image, The step of generating composite image data by combining the subject image and the background image, in a composite image in which the subject image is placed on the background image based on the light source information of the background image, including a shadow region showing a shadow corresponding to the light source cast by the subject shown in the subject image. Image processing methods.

10. A program that is to be executed by the processor, The steps include inputting image data representing a subject image captured by an imaging device, The steps include setting a background image corresponding to a predetermined light source, The step of acquiring composite image data in which the subject image and the background image are combined, in a composite image in which the subject image is placed on the background image according to the input image data, and which includes a shadow region showing a shadow corresponding to the light source cast by the subject shown in the subject image. program.

Images