Image processing apparatus and method, image capturing apparatus and control method thereof, and storage medium

Abstract

To provide an image processing apparatus capable of easily segmenting an object area from a photographed image without using a monochromatic screen and compositing the segmented object area with a background image prepared in advance.SOLUTION: The image processing apparatus includes an acquisition unit configured to acquire a first image based on a captured image from an image capturing apparatus that performs replacement processing for replacing a background region that is a background of an object region with a specific color, a determination unit configured to determine whether the background region of the first image is replaced with the specific color, and a combining unit configured to combine a background image prepared in advance with the background region of the specific color of the first image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for combining an image different from a subject area with the subject area as a background area. [Background technology]

[0002] In the field of video production, a technique called chromakey compositing has traditionally been used, in which a subject area is cut out from a live-action image and composited with a previously prepared image as the background. Chromakey compositing involves placing the subject against a background screen with a specific color, such as green or blue, that is not contained in the subject, and capturing the live-action image with a camera to create the foreground image. A key image is generated from this foreground image using the green or blue color as a reference. The key image is information used to cut out the subject area contained in the foreground image, and a value that distinguishes the background from the subject area is set on a pixel-by-pixel basis. Based on this key image, the subject area contained in the foreground image is cut out and superimposed with previously prepared computer graphics or a live-action image as the background.

[0003] Patent Document 1 discloses the following automatic chromakey compositing device. An original image of a subject photographed against a single-color background and an arbitrary background image are input, and each pixel constituting the original image is classified into a trimap consisting of three regions: background region, foreground region, and unknown region (whether background or foreground). An α mask consisting of α values ​​that take into account the contribution of the hue components of the background and foreground regions for each pixel is then created, and the image is composited based on this α mask. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6119164 [Non-patent literature]

[0005] [Non-Patent Document 1] “Deep image matting”, N.Xu, B.Price, S.Cohen, and T.Huang, Proc. IEEE Conference on CVPR , 2017. Summary of the Invention [Problem to be solved by the invention]

[0006] However, with the above-mentioned conventional technology, it is necessary to photograph the subject against a background of a screen that is colored green or blue, etc., which poses challenges such as the need to find a place to set up the screen and the time it takes to set up the screen.

[0007] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device that can easily extract a subject area from a captured image and combine it with a pre-prepared background image without using a single-color screen. [Means for solving the problem]

[0008] The image processing device according to the present invention is characterized by comprising: an acquisition means for acquiring a first image based on a captured image from an imaging device that performs a replacement process for replacing a background area, which is the background of a subject area, with a specific color in the captured image; a determination means for determining whether the background area of ​​the first image has been replaced with the specific color; a combination means for combining a pre-prepared background image with the background area of ​​the specific color in the first image; and a control means for issuing a warning to the imaging device or transmitting a control signal to the imaging device to switch the shooting mode if the background area of ​​the first image has not been replaced with the specific color. [Effects of the Invention]

[0009] According to the present invention, it is possible to easily extract a subject area from a captured image and combine it with a background image prepared in advance, without using a single-color screen. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing an imaging system according to a first embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram showing the configuration of an imaging apparatus. [Figure 3] FIG. 1 is a block diagram showing the configuration of an imaging apparatus. [Figure 4A] 4 is a flowchart showing the operation of the imaging apparatus. [Figure 4B] 10 is a flowchart showing the operation of the image synthesis device. [Figure 5] 10A and 10B are diagrams showing examples of composite images with different subject resolutions. [Figure 6] FIG. 10 is a diagram showing an example in which there are two types of single colors in the subject edge region. [Figure 7A] 10 is a flowchart showing the operation of the imaging apparatus according to the second embodiment. [Figure 7B] 10 is a flowchart showing the operation of the image synthesis device according to the second embodiment. [Figure 8] FIG. 10 is a diagram showing a captured image with a single-color screen as the background. [Figure 9] 10 is a flowchart showing the operation of the imaging apparatus according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] (First embodiment) In the first embodiment, the time and effort required to install a screen or the like behind the subject is eliminated, and the subject area and background area contained in the captured image are detected, and the captured image is composited with a previously prepared background material image to output a composite image. In this case, if the background of the image input from the imaging device to the image composition device is not replaced with a specific single color, the image composition device issues a predetermined warning to the imaging device or issues an instruction to control the imaging mode of the imaging device.

[0013] FIG. 1 is a diagram showing the configuration of an imaging system according to a first embodiment of the present invention.

[0014] In FIG. 1, reference numeral 101 denotes the entire imaging system. Reference numeral 102 denotes an example of a shooting scene, in which a woman is in the center of the screen in a room without a green or blue screen set up behind the subject. The imaging device 103 detects the subject area and background area in the captured image, and outputs an image in which the background area has been replaced with a single color (specific color), for example, green, as shown in image 104, to the image synthesis device 106. Note that, while the above description refers to a single color, for example, green, this single color may have some range in hue or saturation, and may be any color whose range in hue or saturation is smaller than a predetermined value. Furthermore, the color is not limited to a single color, and may be a plurality of specific colors.

[0015] In the image capture device 103, the process of replacing the background area with a single color (replacement process) is configured to be switched ON and OFF depending on the shooting mode. The shooting mode in which this process is ON is called the "green backless mode" because it does not require the installation of a green screen in the background.

[0016] An image compositing device 106, which serves as an image processing device, receives an image output from the imaging device 103 in which the background has been replaced with a single color, and a previously prepared background material image 105, and performs chromakey compositing. In chromakey compositing, an image 104 input from the imaging device is used as a foreground image, and a key image is generated from this foreground image using green as a reference color. The key image is information for cutting out a subject area contained in the foreground image, and a value for distinguishing the background from the subject area is set on a pixel-by-pixel basis. The subject area and background area are identified based on this key image, and a previously prepared background material image 105, such as computer graphics or a live-action image, is superimposed on the background area to output a composite image 107.

[0017] Next, a description will be given of the imaging device 103. FIG.

[0018] The control unit 201 is, for example, a CPU, and controls the operation of each block of the imaging device 103 by reading out an operation program for each block of the imaging device 103 from a ROM 202 (described later), expanding the program into a RAM 203 (described later), and executing the program.

[0019] The ROM 202 is an electrically erasable / recordable non-volatile memory, and stores parameters and the like required for the operation of each block in addition to the operation programs of each block included in the image capture device 103. The RAM 203 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block included in the image capture device 103.

[0020] The optical system 204 is composed of a group of lenses including a zoom lens and a focus lens, and forms an image of a subject on the imaging unit 205 described below. The imaging unit 205 includes an imaging element 205a such as a CCD or CMOS sensor, and photoelectrically converts the optical image formed by the optical system 204, outputting the resulting analog image signal to the A / D conversion unit 206. In this embodiment, the imaging element 205a has color filters in a Bayer array of primary colors RGB.

[0021] The A / D converter 206 converts the input analog image signal into digital image data, and outputs the resulting digital image data to the RAM 203 .

[0022] The image processing unit 207 performs various image processing such as white balance adjustment, noise reduction, color interpolation (debayering), and gamma processing on the image data stored in the RAM 203. The image processing unit 207 may simultaneously generate a low-resolution (for example, VGA-sized) image (hereinafter referred to as a thumbnail image) for thumbnail display along with the original image. It also performs processing such as replacing only the background region with a single color based on the subject region and background region extracted by the region extraction unit 211 (described later).

[0023] The recording unit 208 is a detachable memory card or the like, and the image processed by the image processing unit 207 is recorded as a recording image via the RAM 203 .

[0024] The display unit 209 is a display device such as an LCD, and displays images stored in the RAM 203 and the recording unit 208, and a user interface for receiving instructions from the user.

[0025] The area extraction unit 211 detects a person area and a background area from the original image or thumbnail image processed by the image processing unit 207 using a known method such as machine learning.

[0026] The face detection unit 212 detects a human face area from the original image or thumbnail image processed by the image processing unit 207 using a known method such as machine learning.

[0027] The configuration and basic operation of the imaging device 103 have been described above.

[0028] Next, a description will be given of the image synthesis device 106. Fig. 3 is a block diagram showing the configuration of the image synthesis device 106.

[0029] The control unit 901 is, for example, a CPU, which reads out the operation programs of each block of the image synthesis device 106 from a ROM 902 (described later), expands them into a RAM 903 (described later), and executes them. This controls the operation of each block of the image synthesis device 106. The ROM 902 is an electrically erasable and recordable non-volatile memory, and stores the operation programs of each block of the image synthesis device 106 as well as parameters and the like required for the operation of each block. The RAM 903 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block of the image synthesis device 106.

[0030] The image synthesis unit 904 synthesizes two or more pieces of image data stored in the RAM 903. Based on information about the subject area and background area of ​​the input image, the image synthesis unit 904 performs synthesis processing such as replacing only the background area of ​​the input image with a single color, or replacing (substituting) only the background area of ​​the input image with an image previously recorded in the ROM 902.

[0031] The configuration and basic operation of the image synthesis device 106 have been described above.

[0032] Next, FIGS. 4A and 4B are flowcharts showing the operations of the image capturing device 103 and the image synthesizing device 106. In FIG.

[0033] First, the operation of the imaging device 103 will be described with reference to FIG. 4A.

[0034] In step S301, the control unit 201 of the image capturing device 103 causes the image capturing device 103 to perform image capturing.

[0035] In step S302, the control unit 201 uses the region extraction unit 211 to detect a subject region and a background region from the captured image. Specifically, the control unit 201 outputs a person region as a likelihood map using a technique such as machine learning for the original image or thumbnail image processed by the image processing unit 207. For example, the person region and background region are detected based on a segmentation result that can output a matching output that can perform person detection with high accuracy, as used in Non-Patent Document 1. This makes it possible to extract complex scene regions where the foreground and background blend together, such as a person's hair or animal fur.

[0036] In step S303, the control unit 201 uses the image processing unit 207 to replace the background region detected by the region extraction unit 211 in step S302 with a specific single color, for example, green.

[0037] In step S304, the control unit 201 outputs the image in which the background region has been replaced with a single color in step S303 to the image synthesis device .

[0038] In step S305, the control unit 201 determines whether or not it has received information warning that the background region is not a single color from the image synthesis device 106. If it has, it proceeds to step S306, and if it has not, it proceeds to step S307.

[0039] In step S306, the control unit 201 displays on the display unit 209 a warning that the background area is not a single color.

[0040] In step S307, the control unit 201 determines whether or not a control signal for setting the shooting mode to the greenbackless mode has been acquired from the image composition device 106. If so, the process proceeds to step S308; if not, the operation of this flow ends.

[0041] In step S308, the control unit 201 switches the shooting mode of the imaging device 103 to the greenbackless mode.

[0042] Next, the operation of the image synthesis device 106 will be described with reference to FIG. 4B.

[0043] In step S309, the control unit 901 of the image synthesis device 106 acquires from the image capture device 103 an image in which the background region has been replaced with a single color.

[0044] In step S310, the control unit 901 determines the color of the background region of the image acquired in step S309.

[0045] In step S311, the control unit 901 determines whether the color determined in step S310 is a single color. If the range of hue falls within an arbitrary fixed range, it is determined to be a single color, and the process proceeds to step S312; if not, the process proceeds to step S314.

[0046] In step S312, the control unit 901 inputs the image in which the background region obtained in step S309 has been replaced with a single color and the background material image 105 prepared in advance to the image synthesis unit 904, and performs chromakey synthesis.

[0047] The chromakey compositing method uses the image input in step S309, in which the background area has been replaced with green, as the foreground image, and generates a key image from the foreground image using the green color as a reference. The key image is information used to cut out the subject area contained in the foreground image, and a value that distinguishes the background from the subject area is set on a pixel-by-pixel basis. The subject area and background area are identified based on this key image, and chromakey compositing is performed.

[0048] In step S313, the control unit 901 outputs the image that has been subjected to chromakey compositing in step S312.

[0049] In step S314, since it was determined in step S311 that the background area is not a single color, the control unit 901 outputs a warning to the image capture device 103 notifying that the background area is not a single color, or outputs a control signal to switch the shooting mode of the image capture device 103 to a greenbackless mode.

[0050] The operations of the image capturing device 103 and the image synthesizing device 106 have been described above.

[0051] As described above, in the first embodiment of the present invention, when the shooting mode of the image capturing device 103 is not set to the greenbackless mode, which is a shooting mode in which the background area is replaced with a single color, the image synthesis device 106 outputs a warning to the image capturing device 103. Alternatively, it outputs a control signal to switch the shooting mode to the greenbackless mode. These controls enable the image capturing device 103 to output an image in which the background area has been replaced with a single color.

[0052] Furthermore, in the present embodiment, an example has been described in which the region extraction unit 211 detects the subject region and the background region using a technique such as machine learning, but the region extraction method of the present invention is not limited to this. For example, instead of performing region extraction within the imaging device, the imaging device 103 may be connected to a network (not shown), and region extraction may be performed on the cloud via this network. Also, for example, the imaging device 103 may acquire distance information of the shooting scene, and the subject region and background region may be detected from the acquired distance information.

[0053] Furthermore, in this embodiment, an example has been described in which a still image is captured as the shooting scene, but the shooting scene is not limited to a still image and may be a moving image.

[0054] In addition, in this embodiment, the shooting scene has been described using an example of a shooting scene with one subject, but this is not limited to this, and the subject may be multiple people, or the subject may not be a person but a pet, vehicle, building, etc.

[0055] Furthermore, in this embodiment, the operation of controlling the image composition device 106 to switch the shooting mode of the image capture device 103 to the greenbackless mode has been described, but the control signal sent to the camera is not limited to this. For example, the image composition device 106 may be configured to control the resolution or power-saving shooting mode of the image capture device 103 according to the size of the subject area of ​​the composite image 107 output by performing chromakey composition.

[0056] FIG. 5 shows an example of a composite image 107. FIG. 5(a) shows an example in which the size of the subject area is large, and FIG. 4(b) shows an example in which the size of the subject area is small. When combining a small size subject area as in FIG. 5(b), the imaging resolution of the imaging device 103 may be low, and imaging can be performed even in power-saving mode. For this reason, a control signal may be sent from the image composition device 106 to the imaging device 103 to control the imaging mode in relation to resolution and power saving.

[0057] (Second embodiment) A second embodiment of the present invention will now be described.

[0058] In this second embodiment, a case will be described in which a single-color screen is placed behind the subject as in the conventional case, and the image capturing device 103 is captured in a state where the image capturing device 103 is set to a green backless mode. In this case, there is a possibility that two types of colors may exist in the edge area of ​​the subject in the image output from the image capturing device 103 to the image synthesis device 106: the color of the single-color screen that was actually placed, and the color of the background replaced with a single color by the image capturing device 103.

[0059] Figure 6 shows a conceptual diagram of a case where two types of colors exist in the edge region of a subject area. If chromakey compositing were performed in this state, the composite image would be degraded. To avoid this, the image compositing device 106 determines the color of the subject's edge region in the image input, and outputs a warning to the imaging device 103 or sends a control signal to switch the shooting mode.

[0060] The block diagram showing the configuration of the imaging system 101 in Fig. 1, the block diagram showing the configuration of the imaging device 103 in Fig. 2, and the block diagram showing the configuration of the image composition device 106 in Fig. 3 are the same as those explained in the first embodiment, so detailed explanations will be omitted. Since the operations of the imaging device 103 and the image composition device 106 differ from those in Figs. 4A and 4B showing the first embodiment, they will be explained using the flowcharts in Figs. 7A and 7B. The same processing steps as in the first embodiment are assigned the same reference numerals, and detailed explanations will be omitted.

[0061] First, the operation of the imaging device 103 will be described with reference to FIG. 7A.

[0062] In step S301, the control unit 201 of the image capturing device 103 causes the image capturing device 103 to perform image capturing.

[0063] In step S302, the control unit 201 uses the area extraction unit 211 to detect a subject area and a background area from the captured image.

[0064] In step S303, the control unit 201 uses the image processing unit 207 to replace the background region detected by the region extraction unit 211 in step S302 with a specific single color, for example, green.

[0065] In step S304, the control unit 201 outputs the image in which the background region has been replaced with a single color in step S303 to the image synthesis device .

[0066] In step S601, the control unit 201 determines whether or not warning information indicating that the subject edge area has two types of single colors has been acquired from the image synthesis device 106. If so, the process proceeds to step S602; if not, the process proceeds to step S603.

[0067] In step S602, the control unit 201 displays on the display unit 209 a warning that the subject edge area is made up of two types of single color.

[0068] In step S603, the control unit 201 determines whether or not a control signal for switching the shooting mode to a shooting mode other than the green backless mode has been received from the image synthesis device 106. If a control signal has been received, the process proceeds to step S604; if not, the operation of this flow ends.

[0069] In step S604, the control unit 201 controls the imaging device 103 to a shooting mode other than the green backless mode.

[0070] Next, the operation of the image synthesis device 106 will be described with reference to FIG. 7B.

[0071] In step S309, the control unit 901 of the image synthesis device 106 acquires an image from the imaging device 103.

[0072] In step S606, the control unit 901 determines the color of the subject edge area in the background area of ​​the image acquired in step S309.

[0073] In step S607, the control unit 901 determines whether the colors determined in step S606 are two types of single colors. If the range of hues is within a predetermined range, it is determined to be one single color and the process proceeds to step S312. If there are two types of hue ranges, the process proceeds to step S608.

[0074] In step S312, the control unit 901 executes chromakey composition using the image in which the background region obtained in step S309 has been replaced with a single color and the background material image 105 prepared in advance.

[0075] In step S313, the control unit 901 outputs the image that has been subjected to chromakey compositing in step S312.

[0076] In step S608, since it was determined in step S607 that the subject edge region in the background region is composed of two types of single colors, the control unit 901 outputs warning information to the image capture device 103 that the subject edge region is composed of two types of single colors, or outputs a control signal to switch the image capture mode of the image capture device 103 to an image capture mode other than the greenbackless mode.

[0077] The operations of the image capturing device 103 and the image synthesizing device 106 have been described above.

[0078] As described above, in the second embodiment of the present invention, a single-color screen is placed behind the subject as in the conventional case, and the image capture device 103 is set to the green backless mode, for example. In this case, the user can easily notice that the green backless mode has been set incorrectly by receiving warning information from the image composition device 106 or a control signal for the capture mode.

[0079] In addition, in this embodiment, the image synthesis device 106 is described as determining whether the subject edge region has two types of single colors, but the present invention is not limited to this. For example, there are cases where it is difficult to determine whether the subject edge region has two types of single colors. In such cases, the determination can be made more easily by controlling the image processing of the image capture device 103 so that the image synthesis device 106 changes the color of the background replaced by the image capture device 103 to a different color.

[0080] (Third embodiment) A third embodiment of the present invention will now be described.

[0081] In this third embodiment, a case will be described in which a single-color screen is placed behind the subject as in the conventional case, and the image capturing device 103 is set to the greenbackless mode when capturing an image. In this case, the image capturing device 103 alone, rather than the image synthesis device 106, is controlled to switch to a capturing mode different from the greenbackless mode.

[0082] 8 is a conceptual diagram showing a case where a scene is captured against a single-color background such as a screen. In order to avoid deterioration in the quality of the composite image, in the second embodiment, the image composition device 106 controls the image capture device 103 to switch to a capture mode other than the green backless mode. In contrast, in the third embodiment, a case will be described in which the image capture device 103 alone displays a predetermined warning and controls the capture mode.

[0083] The block diagram showing the configuration of the imaging system 101 in Fig. 1, the block diagram showing the configuration of the imaging device 103 in Fig. 2, and the block diagram showing the configuration of the image synthesis device 106 in Fig. 3 are the same as those explained in the first embodiment, so detailed explanations will be omitted. Since the operations of the imaging device 103 and the image synthesis device 106 differ from those in Figs. 4A and 4B showing the first embodiment, they will be explained using the flowchart in Fig. 9. The same processing steps as in the first embodiment are assigned the same reference numerals, and detailed explanations will be omitted.

[0084] The operation of the imaging device 103 will be described with reference to FIG.

[0085] In step S301, the control unit 201 of the image capturing device 103 causes the image capturing device 103 to perform image capturing.

[0086] In step S302, the control unit 201 uses the area extraction unit 211 to detect a subject area and a background area from the captured image.

[0087] In step S801, it is determined whether the background area of ​​the captured image detected in step S302 is a specific single color. If it is a specific single color, the process proceeds to step S802, and if it is not a specific single color, the process proceeds to step S303.

[0088] In step S303, the control unit 201 uses the image processing unit 207 to replace the background region detected by the region extraction unit 211 in step S302 with a specific single color, for example, green.

[0089] In step S304, the control unit 201 outputs the image in which the background region has been replaced with a single color in step S303 to the image synthesis device .

[0090] In step S802, since it was determined in step S801 that the background region is a single color, the control unit 201 displays a warning indicating that the background region is a single color on the display unit 209. Alternatively, the control unit 201 controls the shooting mode of the imaging device 103 to a shooting mode other than the green backless mode.

[0091] The operation of the imaging device 103 has been described above.

[0092] As described above, in the third embodiment of the present invention, even if a single-color screen is placed behind the subject as in the conventional case and an image is captured while the imaging device 103 is set to the greenbackless mode, the imaging device 103 detects this. This makes it possible to display a warning or switch to a shooting mode other than the greenbackless mode.

[0093] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions.

[0094] The disclosure of this specification includes the following image processing device and method, imaging device and control method therefor, program, and storage medium.

[0095] (Item 1) an acquisition means for acquiring a first image based on a captured image from an imaging device that performs a replacement process for replacing a background area that is a background of a subject area with a specific color; a determining means for determining whether a background area of ​​the first image has been replaced with the specific color; a synthesis means for synthesizing a background image prepared in advance with the background area of ​​the specific color of the first image; a control means for issuing a warning to the imaging device or transmitting a control signal to the imaging device for switching a photographing mode when the background area of ​​the first image is not replaced with the specific color; An image processing device comprising:

[0096] (Item 2) Item 1. The image processing device according to item 1, wherein the combining means combines a prepared background image with the background area of ​​the first image when the background area of ​​the first image is replaced with the specific color.

[0097] (Item 3) 3. The image processing device according to item 1 or 2, characterized in that, when the background area of ​​the first image has not been replaced with the specific color, the control means transmits to the imaging device a control signal for switching to a shooting mode in which the background area of ​​the first image is replaced with the specific color.

[0098] (Item 4) 4. The image processing device according to any one of items 1 to 3, further comprising a second determination means for determining whether the color of the background area replaced with the specific color is the only color of the background area, and when the color of the background area is not the only color of the specific color, the control means notifies the imaging device of a warning or transmits a control signal to the imaging device to switch a shooting mode.

[0099] (Item 5) 5. The image processing device according to item 4, wherein the second determination means determines whether the color of the background area at the edge of the subject area is only the specific color.

[0100] (Item 6) 6. The image processing device according to item 4 or 5, wherein the control means transmits a control signal to the imaging device to switch the shooting mode to a shooting mode that does not replace the background region with the specific color when the color of the background region is not the only specific color.

[0101] (Item 7) 7. The image processing device according to any one of items 4 to 6, characterized in that, when the color of the background area is not only the specific color, the control means transmits to the imaging device a control signal to switch the specific color that replaces the background area to another color.

[0102] (Item 8) The image processing device described in any one of items 1 to 7, characterized in that the control means transmits a control signal for controlling the image resolution or a control signal related to power saving to the imaging device based on the resolution of the subject area in the composite image synthesized by the synthesis means.

[0103] (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the specific color is a single color whose hue range and saturation range are smaller than predetermined values.

[0104] (Item 10) An imaging device including an imaging means for imaging a subject, a detection means for detecting a background region from the image captured by the imaging means; a determining means for determining whether the subject is an image taken against a background of a first specific color; a replacement means for replacing the background region detected by the detection means with a second specific color; a control means for issuing a warning or controlling switching of a photographing mode of the imaging device when the subject is photographed against a background of a first specific color; An imaging device comprising:

[0105] (Item 11) Item 11. The imaging device according to item 10, characterized in that, when the subject is an image taken against a background of a first specific color, the control means switches the imaging mode of the imaging device to a shooting mode that does not replace the background area with the second specific color.

[0106] (Item 12) an acquisition step of acquiring a first image based on the captured image from an imaging device that performs a replacement process on the captured image to replace a background area that is a background of a subject area with a specific color; a determining step of determining whether a background area of ​​the first image has been replaced with the specific color; a combining step of combining a background image prepared in advance with the background area of ​​the specific color of the first image; a control step of issuing a warning to the imaging device or transmitting a control signal to switch an imaging mode to the imaging device when the background area of ​​the first image is not replaced with the specific color; An image processing method comprising:

[0107] (Item 13) A control method for an imaging device including an imaging unit that captures an image of a subject, comprising: a detection step of detecting a background region from the image captured by the imaging means; a determining step of determining whether the subject is an image taken against a background of a first specific color; a replacement step of replacing the background region detected in the detection step with a second specific color; a control step of issuing a warning or controlling the imaging device to switch an imaging mode when the subject is captured against a background of a first specific color; 10. A method for controlling an imaging device, comprising:

[0108] (Item 14) Item 13. A program for causing a computer to execute each step of the image processing method according to Item 12.

[0109] (Item 15) Item 13. A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to Item 12.

[0110] (Item 16) Item 14. A program for causing a computer to execute each step of the method for controlling an imaging device according to Item 13.

[0111] (Item 17) A computer-readable storage medium storing a program for causing a computer to execute each step of the method for controlling an imaging device according to item 13.

[0112] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0113] 101: imaging system, 102: photographed scene, 103: imaging device, 104: image in which background area is replaced with a single color, 105: background material image, 106: image synthesis device, 107: synthesized image

Claims

1. an acquisition means for acquiring a first image based on a captured image from an imaging device that performs a replacement process for replacing a background area, which is a background of a subject area, with a specific color; a determining means for determining whether a background area of ​​the first image has been replaced with the specific color; a combining means for combining a background image prepared in advance with the background area of ​​the specific color of the first image; a control unit that issues a warning to the imaging device or transmits a control signal to the imaging device to switch a photographing mode when the background area of ​​the first image is not replaced with the specific color; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the combining means combines a prepared background image with the background area of ​​the first image when the background area of ​​the first image has been replaced with the specific color.

3. 2. The image processing device according to claim 1, wherein the control means, when the background area of ​​the first image has not been replaced with the specific color, transmits to the imaging device a control signal for switching to a shooting mode in which the background area of ​​the first image is replaced with the specific color.

4. 2. The image processing device according to claim 1, further comprising a second determination means for determining whether the color of the background area replaced with the specific color is the only color of the background area, and if the color of the background area is not the only color of the specific color, the control means notifies the imaging device of a warning or transmits a control signal to the imaging device to switch a shooting mode.

5. 5. The image processing apparatus according to claim 4, wherein the second determining means determines whether the color of the background area at the edge of the subject area is only the specific color.

6. The image processing device according to claim 4, characterized in that, when the color of the background area is not only the specific color, the control means sends to the imaging device a control signal to switch the shooting mode to a shooting mode that does not replace the background area with the specific color.

7. 5. The image processing device according to claim 4, wherein, when the color of the background area is not only the specific color, the control means transmits to the imaging device a control signal for switching the specific color that replaces the background area to another color.

8. 2. The image processing device according to claim 1, wherein the control means transmits to the imaging device a control signal for controlling the resolution of the image or a control signal related to power saving, based on the resolution of the subject area in the composite image synthesized by the synthesis means.

9. 2. The image processing apparatus according to claim 1, wherein the specific color is a single color whose hue range and saturation range are smaller than predetermined values.

10. An imaging device including an imaging means for imaging a subject, a detection means for detecting a background region from the image captured by the imaging means; a determining means for determining whether the subject is an image taken against a background of a first specific color; a replacement means for replacing the background region detected by the detection means with a second specific color; a control means for issuing a warning or controlling switching of a photographing mode of the imaging device when the subject is photographed against a background of a first specific color; An imaging device comprising:

11. The imaging device according to claim 10, characterized in that the control means switches the imaging mode of the imaging device to a shooting mode that does not replace the background area with the second specific color when the subject is an image photographed against a background of a first specific color.

12. an acquisition step of acquiring a first image based on the captured image from an imaging device that performs a replacement process on the captured image to replace a background area that is a background of a subject area with a specific color; a determining step of determining whether a background area of ​​the first image has been replaced with the specific color; a combining step of combining a background image prepared in advance with the background area of ​​the specific color of the first image; a control step of notifying the imaging device of a warning or transmitting a control signal to switch a photographing mode to the imaging device when the background area of ​​the first image is not replaced with the specific color; An image processing method comprising:

13. A control method for an imaging device including an imaging unit that captures an image of a subject, comprising: a detection step of detecting a background region from the image captured by the imaging means; a determining step of determining whether the subject is an image taken against a background of a first specific color; a replacement step of replacing the background region detected in the detection step with a second specific color; a control step of issuing a warning or performing control to switch a shooting mode of the imaging device when the subject is photographed against a background of a first specific color; 10. A method for controlling an imaging device, comprising:

14. A program for causing a computer to execute each step of the image processing method according to claim 12.

15. A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 12.

16. A program for causing a computer to execute each step of the method for controlling an imaging apparatus according to claim 13.

17. 14. A computer-readable storage medium storing a program for causing a computer to execute each step of the method for controlling an imaging apparatus according to claim 13.

Images