Image processing device, image processing method, and program

Abstract

To provide an image processing apparatus, an image processing method, and a program for creating a highly accurate Trimap by using distance information acquired through photography using an image surface phase difference sensor.SOLUTION: An image processing apparatus comprises: acquisition means that acquires a picked-up image and a plurality of parallax images created through photography using an image pickup device in which a plurality of photoelectric conversion units receiving light beams passing through different pupil partial areas of an imaging optical system are arranged; creation means that creates an image for background separation classifying an area of the picked-up image into a foreground area, a background area, and an unknown area based on distance distribution information obtained from the plurality of parallax images; and output means that outputs the picked-up image and the image for background separation.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In a wide range of fields, there is a demand to extract any subject area from an image. One technique for extracting subject areas is to create AlphaMatte and use it to extract the subject. AlphaMatte is an image that is separated into a foreground area (subject) and a background area.

[0003] A method that uses intermediate data called a trimap is often used to create a highly accurate AlphaMatte. A trimap is an image that is divided into three parts: the foreground region, the background region, and the unknown region.

[0004] Known techniques for creating a trimap include, for example, the technique described in Patent Document 1. Patent Document 1 discloses a technique for creating a binary image of the foreground and background from an input image using an object extraction technique, generating a ternary image by setting an undetermined region of a predetermined width at the boundary between the foreground and background, and creating an AlphaMatte. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2010-066802 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, since the technique disclosed in Patent Document 1 does not use distance information, the accuracy of the trimap deteriorates when, for example, the subject and the background have the same color.

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a technology for generating a highly accurate trimap by using distance information acquired through shooting using an image plane phase difference sensor. [Means for solving the problem]

[0008] In order to solve the above problem, the present invention provides an imaging system including: an acquisition unit that acquires a captured image and a plurality of parallax images generated by photographing using an imaging element in which a plurality of photoelectric conversion units are arranged, each of which receives a light beam passing through a different pupil partial region of an imaging optical system; a generation unit that generates a background separation image that classifies a region of the captured image into a foreground region, a background region, and an unknown region based on distance distribution information obtained from the plurality of parallax images; and an output unit that outputs the captured image and the background separation image. a third display control means for displaying a histogram of distances indicated by the distance distribution information on a display means; wherein the generating means generates the background separation image such that a region in which the distance in the distance distribution information is within a first range is classified as the foreground region, a region in which the distance in the distance distribution information is outside a second range wider than the first range is classified as the background region, and a region in which the distance in the distance distribution information is outside the first range and within the second range is classified as the unknown region. The third display control means displays the histogram in a manner that allows the first range and the second range to be distinguished. The present invention provides an image processing device comprising: Effect of the Invention

[0009] According to the present invention, it is possible to generate a highly accurate trimap.

[0010] Other features and advantages of the present invention will become apparent from the accompanying drawings and the following detailed description of the preferred embodiments of the present invention. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing the internal configuration of an image processing device 100 used in each embodiment. [Diagram 2] FIG. 2 is a diagram showing a part of a light receiving surface of an imaging unit 107 serving as an image sensor. [Diagram 3]23 is a flowchart of a trimap generation process according to the tenth embodiment. [Figure 4] 23A to 23C are diagrams showing examples of images displayed in the shooting standby process (S1001 in FIG. 3) according to the tenth embodiment. [Diagram 5] FIG. 23 is a diagram showing an example of a display of a setting menu for a reference value of a foreground threshold used when generating a trimap in the tenth embodiment. [Figure 6] FIG. 23 is a diagram showing an example of a display of a setting menu for a reference value of a background threshold used when generating a trimap in the tenth embodiment. [Figure 7] 13 is a diagram showing an example of distance information calculated by the CPU 102 when the image capturing unit 107 captures the image shown in FIG. 4 in the tenth embodiment. [Figure 8] FIG. 23 is a diagram showing an example of the relationship between a reference value of a threshold set by a user and a range of values ​​corresponding to the reference value in the tenth embodiment. [Figure 9] FIG. 12 is a diagram showing an example of a trimap generated based on the distance information of FIG. 7 in the tenth embodiment. [Figure 10] 13 is a flowchart of a process for displaying the boundary lines of each region of the trimap by superimposing them on a captured image in the twentieth embodiment. [Figure 11] 13A to 13C are diagrams showing examples of display of setting menus for setting the boundary lines between the foreground region and the unknown region of the Trimap and the boundary lines between the unknown region and the background region of the Trimap when the boundary lines are displayed superimposed on a captured image in the twentieth embodiment. [Figure 12] FIG. 22 is a diagram showing an example of a screen in which a boundary line 2201 between a foreground region and an unknown region, and a boundary line 2202 between an unknown region and a background region are superimposed on the image shown in FIG. 4 in the twentieth embodiment. [Figure 13] 13 is a flowchart of a process for superimposing a trimap on an image in the 30th and 31st embodiments. [Figure 14] FIG. 23 is an explanatory diagram of a Trimap transparency setting menu screen in the 30th and 31st embodiments. [Figure 15] FIG. 13 is an explanatory diagram of a Trimap transparency setting menu screen in the 30th embodiment. [Figure 16] FIG. 13 is a diagram showing an example of a trimap superimposed image in the 30th embodiment. [Figure 17] FIG. 13 is a diagram showing an example of a trimap superimposed image in the 30th embodiment. [Figure 18] FIG. 13 is a diagram showing an example of a trimap superimposed image in the 30th embodiment. [Figure 19] FIG. 13 is a diagram showing an example of a trimap superimposed image in the 30th embodiment. [Figure 20] FIG. 13 is a diagram showing an example of a trimap superimposed image in the 30th embodiment. [Figure 21] FIG. 23 is an explanatory diagram of a Trimap transparency setting menu screen in the thirty-first embodiment. [Figure 22] 13 is a flowchart showing a process for changing the transparency in the thirty-second embodiment. [Figure 23] 13 is a flowchart of a process for generating a distance distribution display histogram and displaying it on the display unit 114 in the fortieth embodiment. [Figure 24] FIG. 13 is an explanatory diagram of the relationship between the overall scene and a distance distribution display histogram in the fortieth embodiment. [Diagram 25] FIG. 13 is a diagram showing an example of a distance distribution display histogram in the fortieth embodiment. [Figure 26] FIG. 13 is an explanatory diagram of the relationship between the overall scene and a distance distribution display histogram in the 41st embodiment. [Figure 27] 13 is a flowchart of an overall process in the 41st embodiment. [Figure 28A] 13 is a flowchart showing details of the process of S4405 in the forty-first embodiment. [Figure 28B] 13 is a flowchart showing details of the process of S4405 in the forty-first embodiment. [Figure 29A] 13 is a flowchart showing details of the process of S4406 in the forty-first embodiment. [Figure 29B] 13 is a flowchart showing details of the process of S4406 in the forty-first embodiment. [Diagram 30]13A to 13C are diagrams showing display examples of a distance distribution display histogram and an emphasized image in the forty-first embodiment. [Figure 31A] 13 is a flowchart of a process for generating a distance distribution display histogram and displaying it on the display unit 114 in the forty-second embodiment. [Figure 31B] 13 is a flowchart of a process for generating a distance distribution display histogram and displaying it on the display unit 114 in the forty-second embodiment. [Diagram 32] 13A to 13C are views showing display examples of a distance distribution display histogram and a colored image in the 42nd embodiment. [Diagram 33] 13 is a flowchart of a process for generating an overhead view image and displaying it on the display unit 114 in the fiftieth embodiment. [Diagram 34] FIG. 13 is an explanatory diagram of the relationship between the distance of an acquired image and an image subjected to superimposition processing in the 50th embodiment. [Diagram 35] FIG. 50 is an explanatory diagram of a display screen according to the 50th embodiment. [Diagram 36] FIG. 51 is an explanatory diagram of a display screen in the 51st embodiment. [Figure 37] FIG. 52 is an explanatory diagram of a display screen in the 52nd embodiment. [Figure 38] 13 is an explanatory diagram of a disparity information range, pixels, and a trimap in the sixtieth embodiment. [Figure 39A] 13 is a flowchart of a second trimap generation process in the sixtieth embodiment. [Figure 39B] 13 is a flowchart of a second trimap generation process in the sixtieth embodiment. [Diagram 40] 13A to 13C are diagrams for explaining edge detection results and trimaps in the sixtieth embodiment. [Diagram 41] 13 is a flowchart of a second trimap generation process in the 70th embodiment. [Diagram 42] FIG. 13 is a diagram for explaining details of the process of S7004 in the 70th embodiment. [Diagram 43] FIG. 13 is a diagram for explaining details of the process of S7005 in the 70th embodiment. [Diagram 44]13 is a flowchart of a second trimap generation process in the 71st embodiment. [Diagram 45] FIG. 13 is a diagram for explaining details of the process of S7106 in the seventieth embodiment. [Diagram 46] 13 is a flowchart of a process for changing a threshold value in response to a change in the F-number in the seventieth embodiment. [Figure 47] FIG. 13 is an explanatory diagram of a frame image according to the 80th embodiment. [Figure 48] FIG. 13 is an explanatory diagram of an image separation method according to the 80th embodiment. [Figure 49] FIG. 13 is an explanatory diagram of a focus region in the 90th embodiment. [Figure 50] FIG. 19 is an explanatory diagram of a defocus amount in the 90th embodiment. [Figure 51] FIG. 13 is an explanatory diagram of a focus area boundary in the 90th embodiment. [Figure 52] 13 is a flowchart of a trimap generation process in the 90th embodiment. [Diagram 53] FIG. 13 is an explanatory diagram of a focus area boundary in the 91st embodiment. [Figure 54] FIG. 13 is an explanatory diagram of the setting resolution of a focus area boundary in the 91st embodiment. [Figure 55] FIG. 13 is a side view illustrating the setting resolution of a focus area boundary in the 91st embodiment. [Figure 56] 13 is a flowchart of a process for setting an adjustment resolution and an association threshold value of a focus area boundary in the 91st embodiment. [Figure 57A] 13 is a flowchart of a trimap generation process in the embodiment A0. [Figure 57B] 13 is a flowchart of a trimap generation process in the embodiment A0. [Figure 58A] 13 is a flowchart of a trimap generation process in the A1 embodiment. [Figure 58B] 13 is a flowchart of a trimap generation process in the A1 embodiment. [Figure 59A] 13 is a flowchart of a trimap generation process according to the A2 embodiment. [Figure 59B] 13 is a flowchart of a trimap generation process according to the A2 embodiment. [Figure 60] 11 is a flowchart showing details of a process of SA203 in the A2 embodiment. [Figure 61] 11A to 11C are diagrams showing examples of captured images and trimaps in the B0th to B2nd embodiments. [Figure 62] 13 is a flowchart of a trimap generation process according to the B0 embodiment. [Figure 63] 13 is a flowchart of a trimap generation process in the B1 embodiment. [Figure 64] 13 is a flowchart of a trimap generation process according to the B2 embodiment. [Figure 65] FIG. 4 is a diagram for explaining the data structure of SDI in the C0 embodiment. [Figure 66] 13 is a flowchart of a stream generation process in the C0 embodiment. [Figure 67A] 11 is a flowchart showing details of processing in SC002 in the C0 embodiment. [Figure 67B] 11 is a flowchart showing details of processing in SC002 in the C0 embodiment. [Figure 68A] 11 is a flowchart showing details of the processes in SC003 and SC004 in the C0 embodiment. [Figure 68B] 11 is a flowchart showing details of the processes in SC003 and SC004 in the C0 embodiment. [Figure 69] 11 is a flowchart showing details of the process of SC005 in the C0 embodiment. [Figure 70] FIG. 13 is a diagram showing a data packing structure in the C0 embodiment. [Figure 71] FIG. 13 is a diagram showing the structure of an ancillary packet in the C0 embodiment. [Figure 72A] 11 is a flowchart showing details of processing in SC002 in the C1 embodiment. [Fig. 72B]13 is a flowchart of a data packing process in the C1 embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] <First embodiment> First, the internal configuration of an image processing device 100 used in each embodiment will be described with reference to Fig. 1. In Fig. 1, the image processing device 100 is capable of executing processes from image input to output and recording.

[0014] 1, a CPU 102, a ROM 103, a RAM 104, an image processing unit 105, a lens unit 106, an imaging unit 107, a network terminal 108, an image terminal 109, and a recording medium I / F 110 are connected to an internal bus 101. In addition, a frame memory 111, an operation unit 113, a display unit 114, an object detection unit 115, a power supply unit 116, and an oscillation unit 117 are connected to the internal bus 101. A recording medium 112 is connected to the recording medium I / F 110. The units connected to the internal bus 101 are configured to be able to exchange data with each other via the internal bus 101.

[0015] The lens unit 106 (imaging optical system) includes a group of lenses including a zoom lens and a focus lens, an aperture mechanism, and a drive motor. The optical image that passes through the lens unit 106 is received by the imaging unit 107. The imaging unit 107 uses a CCD or CMOS sensor, and plays a role in converting optical signals into electrical signals. Since the electrical signals obtained here are analog values, the imaging unit 107 also has a function of converting analog values ​​into digital values. The imaging unit 107 is an image plane phase difference sensor, and details thereof will be described later.

[0016] The CPU 102 controls each unit of the image processing device 100 using the RAM 104 as a work memory in accordance with a program stored in the ROM 103. This control includes display control corresponding to the display unit 114 and recording control for the recording medium 112. The ROM 103 is a non-volatile recording element, and stores programs for operating the CPU 102, various adjustment parameters, and the like. The RAM 104 is a volatile memory that uses a semiconductor element, and generally has a lower speed and a smaller capacity than the frame memory 111.

[0017] The frame memory 111 is an element that temporarily stores image signals and can read them out when necessary. Image signals are huge amounts of data, so high bandwidth and large capacity are required. In recent years, DDR4-SDRAM (Dual Data Rate 4 - Synchronous Dynamic RAM) and the like are often used. By using this frame memory 111, it becomes possible to perform processes such as synthesizing images that differ over time and cutting out only the necessary areas from an image.

[0018] The image processing unit 105 performs various image processing on data from the imaging unit 107 or image data stored in the frame memory 11 or the recording medium 112 under the control of the CPU 102. The image processing performed by the image processing unit 105 includes pixel interpolation, encoding, compression, decoding, enlargement / reduction (resizing), noise reduction, color conversion, and the like of the image data. The image processing unit 105 also performs processing such as correction of performance variations of the pixels of the imaging unit 107, correction of defective pixels, white balance correction, brightness correction, and correction of distortion and peripheral light loss caused by lens characteristics. The image processing unit 105 may be configured with a dedicated circuit block for performing a specific image processing. Depending on the type of image processing, the CPU 102 may also perform image processing according to a program without using the image processing unit 105.

[0019] Based on the calculation results obtained by the image processing unit 105, the CPU 102 controls the lens unit 106, and can adjust the optical image enlargement, the focal length, the aperture to adjust the amount of light, etc. Also, it is possible to perform camera shake correction by moving a part of the lens group on a plane perpendicular to the optical axis.

[0020] The operation unit 113 is one of the interfaces with the outside of the device, and receives operations from the user. The operation unit 113 uses elements such as mechanical buttons and switches, and includes a power switch and a mode change switch.

[0021] The display unit 114 has a function of displaying an image. The display unit 114 is a display device that can be visually recognized by the user, and can display, for example, an image processed by the image processing unit 105, a setting menu, and the like. The user can check the operating status of the image processing device 100 by looking at the display unit 114. In recent years, a small, low-power device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) is used as the display device for the display unit 114. Furthermore, the display unit 114 may also be equipped with a resistive film type or capacitive film type element called a touch panel, and may be used as a substitute for the operation unit 113.

[0022] The CPU 102 generates character strings for informing the user of the setting status of the image processing device 100 and a menu for setting the image processing device 100, and displays them on the display unit 114, superimposed on the image processed by the image processing unit 105. In addition to character information, shooting assist displays such as a histogram, vector scope, waveform monitor, zebra, peaking, false color, etc. can also be superimposed.

[0023] Another interface is an image terminal 109. Representative interfaces include SDI (Serial Digital Interface), HDMI (registered trademark) (High Definition Multimedia Interface), DisplayPort (registered trademark), and various other interfaces. By using the image terminal 109, it becomes possible to display a real-time image on an external monitor or the like.

[0024] The image processing device 100 also includes a network terminal 108 that can transmit not only images but also control signals. The network terminal 108 is an interface for inputting and outputting image signals and audio signals. The network terminal 108 can also communicate with external devices via the Internet or the like, and transmit and receive various types of data such as files and commands.

[0025] The image processing device 100 has a function of not only outputting an image to the outside, but also recording an image internally. The recording medium 112 is capable of recording image data and various setting data, and a large-capacity storage element is used. For example, the recording medium 112 may be an HDD (Hard Disc Drive) or an SSD (Solid State Drive). The recording medium 112 is attached to the recording medium I / F 110.

[0026] The object detection unit 115 is a block for detecting an object using artificial intelligence, such as deep learning using a neural network. In the case of object detection using deep learning, the CPU 102 transmits to the object detection unit 115 a program for processing stored in the ROM 103, as well as a network structure such as SSD (Single Shot Multibox Detector) or YOLO (You Only Look Once), weight parameters, and the like. The object detection unit 115 performs processing for detecting an object from an image signal based on various parameters obtained from the CPU 102, and loads the processing result in the RAM 104.

[0027] Finally, in order to drive these systems, the image processing device 100 is also provided with a power supply unit 116, an oscillator unit 117, and the like. The power supply unit 116 supplies power to each of the above-mentioned blocks, and has the function of converting an external power source such as a commercial power source or a battery into an arbitrary voltage and distributing it. The oscillator unit 117 is an oscillator element called a crystal. The CPU 102 and the like generate desired timing signals based on the periodic signal input from this oscillator element, and proceed with the program sequence.

[0028] The above is an example of the entire system of the image processing device 100.

[0029] 2 shows a part of the light receiving surface of the imaging unit 107 as an image sensor. In order to enable image plane phase difference autofocus, the imaging unit 107 has pixel units, each of which has two photoelectric conversion units (photodiodes that are light receiving units) for one microlens, arranged in an array. This makes it possible for each pixel unit to receive a light beam that has been split by the exit pupil of the lens unit 106.

[0030] Fig. 2(A) is a schematic diagram of a part of the image sensor surface in an example of a Bayer array of red (R), blue (B), and green (Gb, Gr). Fig. 2(B) is an example of a pixel section that holds two photodiodes as photoelectric conversion sections for one microlens, corresponding to the color filter array in Fig. 2(A).

[0031] The image sensor having the configuration of Fig. 2(B) is capable of outputting two signals for phase difference detection (hereinafter also referred to as image signal A and image signal B) from each pixel unit. The image sensor having the configuration of Fig. 2(B) is also capable of outputting an imaging signal (image signal A+image signal B) obtained by adding together the signals from two photodiodes. This added signal is equivalent to the output of the image sensor of the Bayer array example outlined in Fig. 2(A).

[0032] The imaging unit 107 can output a phase difference detection signal for each pixel unit, but can also output a value obtained by averaging the phase difference detection signals of multiple adjacent pixel units. Outputting the average value can shorten the time required to read out a signal from the imaging unit 107 and reduce the bandwidth of the internal bus 101.

[0033] Using the output signal from the imaging unit 107 as such an image sensor, the CPU 102 performs correlation calculation of the two image signals to calculate information such as the defocus amount, parallax information, and various types of reliability. The defocus amount on the image plane is calculated based on the deviation between the A image signal and the B image signal. The defocus amount has a positive or negative value, and whether the focus is front or back can be determined depending on whether the defocus amount is a positive or negative value. Furthermore, the absolute value of the defocus amount indicates the degree of focus, and if the defocus amount is 0, the focus is achieved. That is, the CPU 102 calculates information on whether the focus is front or back based on the positive or negative value of the defocus amount. Furthermore, the CPU 102 calculates focus degree information, which is the degree of focus (degree of focus deviation), based on the absolute value of the defocus amount. The CPU 102 outputs information on whether the focus is front or back when the defocus amount exceeds a predetermined value, and outputs information that the focus is achieved when the absolute value of the defocus amount is within a predetermined value. The CPU 102 controls the lens unit 106 according to the defocus amount to perform focus adjustment.

[0034] Furthermore, the CPU 102 calculates the distance to the subject using the principle of triangulation based on the parallax information and the lens information of the lens unit 106. Furthermore, the CPU 102 generates a trimap by taking into account the distance to the subject, the lens information of the lens unit 106, and the setting status of the image processing device 100. A method for generating the trimap will be described in detail later.

[0035] Note that here, two signals, (A image signal+B image signal) for imaging and an A image signal for phase difference detection, are output from the imaging unit 107 for each pixel. In this case, the A image signal can be subtracted from (A image signal+B image signal) after output to calculate the B image signal for phase difference detection. The method is not limited to this, and output from the imaging unit 107 may be in the form of the A image signal and the B image signal, in which case the (A image signal+B image signal) for imaging can be calculated by adding the A image signal and the B image signal.

[0036] Also, in FIG. 2, an example is shown in which pixel units each having two photodiodes as photoelectric conversion units are arranged in an array for one microlens. In this regard, pixel units each having three or more photodiodes as photoelectric conversion units for one microlens may be arranged in an array. Also, a plurality of pixel units each having a different opening position of a light receiving unit for a microlens may be provided. In other words, it is sufficient as long as two signals for phase difference detection capable of detecting a phase difference, such as an A image signal and a B image signal, can be obtained as a result.

[0037] Since the image processing device 100 has the above-mentioned configuration, it is possible to acquire an imaged image and a plurality of parallax images generated by photographing using an image sensor having an array of a plurality of photoelectric conversion units, each of which receives a light beam passing through a different pupil sub-region of the imaging optical system.

[0038] In the following embodiments, unless otherwise specified, the image processing device 100 described above is used. Furthermore, the configurations of the following embodiments can be appropriately combined.

[0039] <Tenth embodiment> In the tenth embodiment, an example of a process for generating a trimap (image for background separation) will be described.

[0040] 3 is a flowchart of a trimap generation process in the tenth embodiment. Each process in this flowchart is realized by the CPU 102 loading a program stored in the ROM 103 into the RAM 104 and executing it.

[0041] When the user operates the operation unit 113 to turn on the power supply unit 116, the CPU 102 performs shooting standby processing in S1001. In the shooting standby processing, the CPU 102 displays, on the display unit 114, an image captured by the imaging unit 107 and processed by the image processing unit 105 as shown in FIG. 4, and a menu for setting the image processing device 100.

[0042] In S1002, the user operates the operation unit 113 while looking at the display unit 114. The CPU 102 performs settings and processing on each processing unit of the image processing device 100 according to the above operation.

[0043] 5 is a diagram showing a display example of a setting menu for a reference value of a foreground threshold used when generating a trimap. A specific example of the reference value of the foreground threshold will be described later. First, the user operates the operation unit 113, whereby the CPU 102 displays a foreground threshold setting menu screen 1200 on the display unit 114 and accepts the setting of the reference value of the foreground threshold. The user operates the operation unit 113 to move a cursor 1201 displayed on the foreground threshold setting menu screen 1200, thereby setting the reference value of the foreground threshold.

[0044] 6 is a diagram showing an example of a display menu for setting a reference value of a background threshold used when generating a trimap. A specific example of the reference value of the background threshold will be described later. When a user operates the operation unit 113, the CPU 102 displays a background threshold setting menu screen 1300 on the display unit 114 and accepts the setting of a reference value of the background threshold. The user operates the operation unit 113 to move a cursor 1301 displayed on the background threshold setting menu screen 1300, and sets the reference value of the background threshold.

[0045] Here, CPU 102 displays background threshold setting menu screen 1300 so that the user cannot set a value smaller than the value set as the reference value of the foreground threshold. For example, when 2 is set as the reference value of the foreground threshold, CPU 102 performs control so that a gray display like 1302 in Fig. 6 cannot be set as the background threshold.

[0046] Furthermore, the CPU 102 determines the foreground threshold and the background threshold in accordance with the reference value of the foreground threshold and the reference value of the background threshold set in S1002.

[0047] In S1003, the CPU 102 calculates distance information to the subject for each pixel based on the parallax information and lens information of the lens unit 106 (that is, distance distribution information is obtained).

[0048] Fig. 7 is a diagram showing an example of distance information calculated by the CPU 102 when the image capturing unit 107 captures the image shown in Fig. 4. In Fig. 7, pixels at positions where the defocus amount is 0 are shown in white, and pixels are shown in darker black as the defocus amount becomes larger or smaller than 0.

[0049] In S1004, the CPU 102 determines for each pixel whether the distance information to the subject is within the range of the foreground threshold determined in S1002. If the distance information is within the range of the foreground threshold, the process proceeds to S1006, and if the distance information is outside the range of the foreground threshold, the process proceeds to S1005.

[0050] In S1005, the CPU 102 determines for each pixel whether the distance information to the subject is outside the range of the background threshold determined in S1002. If it is outside the range of the background threshold, the process proceeds to S1007, and if it is within the range of the background threshold, the process proceeds to S1008.

[0051] In S1006, the CPU 102 classifies the pixel region, whose distance information has been determined to be within the range of the foreground threshold in S1004, as a foreground region, and performs processing to replace the pixel values ​​of that region with white data.

[0052] In S1007, the CPU 102 classifies the pixel region, for which it has been determined in S1005 that the distance information is outside the range of the background threshold, as a background region, and performs processing to replace the pixel values ​​of that region with black data.

[0053] In S1008, the CPU 102 classifies the pixel region, whose distance information has been determined to be within the range of the background threshold in S1005, as an unknown region, and performs processing to replace the pixel values ​​of that region with gray data.

[0054] Specifically, for example, it is assumed that the distance information calculated by CPU 102 in S1003 has a value in the range of -128 to +127, and the value of the distance information at a position where the defocus amount is 0 is 0. It is also assumed that the reference value of the threshold set by the user in S1002 and the value range according to the reference value have the relationship shown in FIG. 8. If the reference value of the foreground threshold set in S1002 is 2 and the reference value of the background threshold is 4, CPU 102 classifies the area of ​​distance information -50 to +50 as the foreground area, the areas of -128 to -101 and +101 to +127 as the background area, and the areas of -100 to -51 and +51 to +100 as the unknown area. CPU 102 then performs a process of replacing pixel values ​​of the foreground area with white data, pixel values ​​of the background area with black data, and pixel values ​​of the unknown area with gray data.

[0055] Through the above process, the CPU 102 generates a trimap divided into three regions: a foreground region, a background region, and an unknown region. Fig. 9 is a diagram showing an example of a trimap generated based on the distance information in Fig. 7.

[0056] In S1009, the CPU 102 performs processing to output the Trimap to the display unit 114, the image terminal 109, or the network terminal 108.

[0057] As described above, in this embodiment, a trimap is generated using distance information calculated from data from an image plane phase difference sensor, so that the trimap can be easily generated without performing calibration.

[0058] In this embodiment, the trimap is displayed or output, but it may be recorded on the recording medium 112 via the recording medium I / F 110. The trimap may be displayed, output, or recorded as a single still image, or multiple consecutive trimaps may be displayed, output, or recorded as a video.

[0059] In addition, in this embodiment, the imaging unit 107 is configured to output a phase difference detection signal for each pixel unit, but a configuration may be adopted in which a value obtained by averaging phase difference detection signals of a plurality of adjacent pixel units of the imaging unit 107 is output and a reduced trimap is generated by using the value. The reduced trimap may be displayed, output, or recorded at the original image size, or a resizing process may be performed by the image processing unit 105, and a trimap of a different image size may be displayed, output, or recorded.

[0060] In addition, in this embodiment, a trimap is displayed in which the foreground region is white data, the background region is black data, and the unknown region is gray data, but the color data of each region may be replaced with color data different from the example described above.

[0061] <20th embodiment> In the tenth embodiment, it is difficult for the user to grasp the positional relationship between the captured image and the boundaries of each area of ​​the trimap. Therefore, in the twentieth embodiment, an example of a process for displaying the boundaries of each area of ​​the trimap by superimposing them on the captured image will be described.

[0062] 10 is a flowchart of a process for displaying the boundaries of each region of the trimap superimposed on a captured image in the twentieth embodiment. Each process in this flowchart is realized by the CPU 102 expanding a program stored in the ROM 103 into the RAM 104 and executing it. In this embodiment, the same or similar configurations and steps as those in the tenth embodiment are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0063] 10, the user operates the operation unit 113 while looking at the display unit 114. The CPU 102 performs settings and processing on each processing unit of the image processing device 100 according to the above operation.

[0064] 11 is a diagram showing an example of a display of a setting menu for setting each of the boundary lines when the boundary line between the foreground region and the unknown region and the boundary line between the unknown region and the background region of the trimap are displayed superimposed on the captured image. When the user operates the operation unit 113, the CPU 102 displays a boundary line setting menu screen 2100 on the display unit 114 and accepts various settings related to the boundary line between the foreground region and the unknown region and the boundary line between the unknown region and the background region. Then, the user operates the operation unit 113 to move a cursor 2101 displayed on the boundary line setting menu screen 2100 and select each setting item, thereby performing various settings related to the boundary line between the foreground region and the unknown region and the boundary line between the unknown region and the background region. Each setting item will be described later.

[0065] In addition, in S2001, similarly to S1002, the user also sets the reference value of the foreground threshold and the reference value of the background threshold.

[0066] In S2002, the CPU 102 generates a trimap by performing the same processes as those in S1003 to S1008 described in the tenth embodiment.

[0067] In S2003, the CPU 102 extracts the boundaries of each region of the trimap. Specifically, for example, the foreground region, background region, and unknown region are respectively made up of white data, black data, and gray data, and the boundaries of each region can be extracted by applying a high-pass filter with a predetermined cutoff frequency to the luminance values ​​of the trimap to extract high-frequency components. The cutoff frequency is determined by the CPU 102 according to the frequency value set by the user via the operation unit 113 in S2001.

[0068] Furthermore, the CPU 102 can also determine whether the boundary is between white data and gray data, between gray data and black data, or between white data and black data, based on the positive / negative and magnitude of the value extracted by the high-pass filter. For example, since the difference in luminance between white data and gray data is smaller than the difference in luminance between white data and black data, it is possible to determine whether a pixel in the white data area is on the boundary between gray data and black data based on the magnitude of the value extracted by the high-pass filter. Furthermore, since the difference in luminance between gray data and white data and the difference in luminance between gray data and black data are opposite in positive / negative when gray data is used as a reference, it is possible to determine whether a pixel in the gray data area is on the boundary between white data and black data based on the positive / negative of the value extracted by the high-pass filter.

[0069] In this manner, it is possible to determine whether the boundary is between white data and gray data, between gray data and black data, or between white data and black data, i.e., between a foreground region and an unknown region, between an unknown region and a background region, or between a foreground region and a background region.

[0070] In S2004, the CPU 102 determines for each pixel whether the boundary extracted in S2003 is a boundary between a foreground region and an unknown region. If it is a boundary between a foreground region and an unknown region, the process proceeds to S2005. If it is not, that is, if it is a boundary between an unknown region and a background region or a boundary between a foreground region and a background region, the process proceeds to S2006.

[0071] In S2005, CPU 102 superimposes color data according to the setting of the boundary line between the foreground region and the unknown region set in S2001 onto the output image signal of image processing unit 105 at the same position as the pixel determined to be the boundary between the foreground region and the unknown region in S2004. Specifically, the larger the gain value set on boundary line setting menu screen 2100, the darker the color set as the color appears, which is superimposed onto the output image signal of image processing unit 105.

[0072] In S2006, CPU 102 superimposes color data corresponding to the setting of the boundary line between the unknown region and the background region set in S2001 onto the output image signal of image processing unit 105 at the position of a pixel determined in S2004 to be a boundary that is not between the foreground region and the unknown region, i.e., a boundary between the unknown region and the background region or a boundary between the foreground region and the background region. Specifically, the larger the gain value set on boundary line setting menu screen 2100, the darker the color set as the color appears, which is superimposed on the output image signal of image processing unit 105.

[0073] In S2007, the CPU 102 performs processing to output the image signal on which the boundary line has been superimposed in S2005 or S2006 to the display unit 114, the image terminal 109, or the network terminal 108. Fig. 12 is a diagram showing an example of a screen on which a boundary line 2201 between the foreground region and the unknown region, and a boundary line 2202 between the unknown region and the background region are superimposed on the image shown in Fig. 4. As shown in Fig. 12, the captured image is displayed in a manner in which the foreground region, the background region, and the unknown region can be distinguished.

[0074] As described above, in this embodiment, the boundaries between the regions of the trimap are displayed superimposed on the captured image, making it easier for the user to grasp the relationship between the captured image and the boundaries between the regions of the trimap.

[0075] Furthermore, by setting the boundary line between the foreground and background regions in the same way as the boundary line between the unknown and background regions, it is possible to make it easier for the user to recognize that the subject is in the unknown region.

[0076] <Thirtieth embodiment> When an image and a trimap are displayed separately, it is difficult to confirm whether the foreground region and unknown region of the trimap cover the subject of the image. In this embodiment, a configuration that solves this problem will be described.

[0077] In this embodiment, the image processing unit 105 shown in Fig. 1 sets transparency α for each of the foreground region, unknown region, and background region of the trimap in the image, and performs processing to superimpose the trimap with the set transparency on the image. Then, the CPU 102 displays the image with the trimap superimposed on the display unit 114. Here, the transparency α represents an opaque state when its value is 0, a transparent state when its value is 1, and a semi-transparent state when its value is between 0 and 1. Then, α=1 may be set for all of the foreground region, unknown region, and background region of the trimap so that only the image is displayed, or α=0 may be set for all of them so that only the trimap is displayed.

[0078] An example in which the user selects a trimap transparency setting from presets will be described with reference to Fig. 13. First, in S3001, the CPU 102 acquires an image processed by the image processing unit 105. In S3002, the CPU 102 generates a trimap by performing the same processes as those in S1003 to S1008 described in the tenth embodiment.

[0079] In S3003, the user operates the operation unit 113, whereby the CPU 102 displays a trimap transparency setting menu screen 3100 shown in Fig. 14 on the display unit 114. Here, Fig. 14 shows an example in which the trimap transparency setting menu screen 3100 and a cursor 3101 are displayed on the display unit 114 in S3003.

[0080] In S3004, the user operates the operation unit 113 to move the cursor 3101 displayed on the trimap transparency setting menu screen 3100, and selects a preset setting as the trimap transparency setting. In response to the user's operation, the CPU 102 displays a list of presets on the trimap transparency setting menu screen 3100. In this case, the process proceeds from S3004 to S3005. Here, the list of presets may be displayed when the trimap transparency setting menu screen 3100 is displayed in S3003. Note that the case where the user setting is selected (the case where the process proceeds from S3004 to S3007) will be described in the thirty-first embodiment.

[0081] In S3005, the user operates the operation unit 113 to move the cursor 3201 displayed on the trimap transparency setting menu screen 3100, and select any preset of the trimap transparency setting. Here, Fig. 15 is an example in which the trimap transparency setting menu screen 3100 and the cursor 3201 are displayed on the display unit 114 in S3005. Also, the trimap transparency setting preset represents a combination of the transparency of each of the foreground region, unknown region, and background region of the trimap that is defined. For example, ROM 103 holds trimap transparency settings as presets, such as (a) image (foreground region: α=0, unknown region: α=0, background region: α=0), (b) trimap (foreground region: α=1, unknown region: α=1, background region: α=1), (c) image + trimap (foreground region: α=0.3, unknown region: α=0.5, background region: α=0.7), and (d) simple cutout (foreground region: α=0, unknown region: α=0, background region: α=1). In S3006, CPU 102 reads out the transparency of the preset selected in S3005 from ROM 103.

[0082] In S3008, the CPU 102 performs transparency processing on the trimap based on the transparency read out in S3006. Here, the transparency processing may be performed on the entire trimap in one process with different transparency levels for each region based on the region information of the trimap. Alternatively, the transparency processing may be performed on each region of the trimap in order, and the intermediate data may be temporarily stored in the frame memory 111 and read out when the transparency processing is performed on the next region.

[0083] In S3009, the CPU 102 superimposes the trimap that has been subjected to the transparency processing in S3008 on the image acquired in S3001. In S3010, the CPU 102 loads the trimap superimposed image in the frame memory 111 and displays it on the display unit 114. The trimap superimposed image may be displayed in a picture-in-picture format, may be output from the image terminal 109, or may be recorded on the recording medium 112. The CPU 102 may record the trimap superimposed image and the trimap area information, and may change the transparency during playback, or may display the recorded trimap superimposed image on the display unit 114 only during REC review. Here, FIG. 16, FIG. 17, FIG. 18, and FIG. 19 are examples in which the trimap superimposed image is displayed on the display unit 114 in S3010. In addition, in the example of the transparency setting in S3005, "(a) Image" corresponds to FIG. 16, "(b) Trimap" corresponds to FIG. 17, "(c) Image + Trimap" corresponds to FIG. 18, and "(d) Simple Cutout" corresponds to FIG. 19. Note that in this embodiment, a trimap in which the foreground region is white data, the unknown region is gray data, and the background region is black data is superimposed, but it is also possible to superimpose and display images in which each region is represented by horizontal lines, vertical lines, and diagonal lines. An example of such a display is shown in FIG. 20.

[0084] As described above, according to the thirtieth embodiment, it is possible to easily check the image and the trimap at the same time.

[0085] <Thirty-first embodiment> In the thirtieth embodiment, an example has been described in which the user selects the trimap transparency setting from presets. However, as another embodiment, an example in which the user manually sets the trimap transparency setting may be considered.

[0086] In the 31st embodiment, an example in which a user manually sets the transparency setting of Trimap will be described with reference to the flowchart of Fig. 13. In the following, differences from the 30th embodiment will be mainly described, and the description of the same configuration and processing as the 30th embodiment will be omitted.

[0087] First, steps S3001 to S3003 are the same as those in the 30th embodiment, and therefore will be omitted. Next, in S3004, the user operates the menu as in the 30th embodiment, and selects user setting as the trimap transparency setting. In accordance with the user's operation, the CPU 102 displays a trimap transparency setting screen 3800 on the display unit 114. In this case, the processing steps proceed from S3004 to S3007. Here, FIG. 21 shows an example in which the trimap transparency setting screen 3800, scroll bar 3801, scroll bar 3802, and scroll bar 3803 are displayed on the display unit 114 in S3004.

[0088] In S3007, the user operates the operation unit 113 to move scroll bars 3801, 3802, and 3803 displayed on the trimap transparency setting screen 3700. In accordance with the user's operation, the CPU 102 sets the transparency α of each of the foreground region, the unknown region, and the background region of the trimap. Here, the transparency setting of the trimap is not limited to an operation using a GUI (Graphical User Interface) such as a scroll bar, but may be realized by an operation using a physical interface such as a volume knob that can arbitrarily change the setting value. Next, S3008 to S3010 are omitted because they are the same as those in the 30th embodiment.

[0089] As described above, according to the thirty-first embodiment, it is possible to easily check the image and the trimap at the same time.

[0090] <Thirty-second embodiment> In the 30th and 31st embodiments, when a state that affects the image or trimap area occurs, or when an operation that affects the image or trimap area is performed, there is a problem that it is difficult to check the image or trimap. In this embodiment, a configuration that solves this problem will be described.

[0091] In the 32nd embodiment, an example of automatically setting the transparency of a trimap will be described with reference to the flowchart of Fig. 22. In the following, differences from the 30th and 31st embodiments will be mainly described, and descriptions of configurations and processes similar to those of the 30th and 31st embodiments will be omitted.

[0092] First, S3901 and S3902 are similar to S3001 and S3002 in Fig. 13, and therefore the description will be omitted. In S3903, the same processing as S3003 to S3007 in Fig. 13 is performed.

[0093] Next, in S3904, CPU 102 determines whether or not a transparency change condition for the trimap held in ROM 103 is satisfied. Here, the transparency change condition refers to the presence or absence of detection of a state or operation that affects the image or trimap area, such as when a subject enters from outside the angle of view and an additional foreground area is detected, or when a lens operation is detected. If the transparency change condition is satisfied, processing proceeds to S3905, and if the transparency change condition is not satisfied, processing proceeds to S3906.

[0094] In order to prevent continuous changes in transparency and improve visibility, a configuration may be adopted in which the process proceeds to step S3905 and the transparency is changed even if the transparency change condition is not satisfied, within a predetermined number of frames after the transparency change condition is satisfied. Also, the transparency change condition may be determined not only based on the presence or absence of detection, but also based on other conditions.

[0095] In S3905, the CPU 102 reads out the transparency corresponding to the transparency and the transparency change conditions set in S3903 from the ROM 103, and changes the transparency. For example, since the user wants to give priority to checking the image during the lens operation, the CPU 102 reads out the set value of α=1 for all of the foreground region, the unknown region, and the background region as the transparency of the trimap when the lens operation is detected, and changes the transparency. In this case, during the lens operation, only the image is displayed on the display unit 114, and after the lens operation is completed, the image is displayed on the display unit 114 after the transparency processing reflecting the transparency set in S3903 is performed. Here, the transparency corresponding to the transparency change conditions may be set arbitrarily by the user. In addition, when using a transparency change condition other than a state that affects the image or the trimap region or the presence or absence of the detection of the operation, a configuration may be adopted in which the transparency corresponding to each condition is stored in the ROM 103, and the set value of the transparency corresponding to the condition is read out and the transparency is changed.

[0096] Next, a case will be described where the transparency change condition is not satisfied in S3904 and the process proceeds to S3906. In S3906, CPU 102 maintains the transparency set in S3903 as it is without changing it.

[0097] S3907, S3908, and S3909 following the processing of S3905 or S3906 are similar to S3008, S3009, and S3010 in FIG. 13, and therefore description thereof will be omitted.

[0098] As described above, according to the 32nd embodiment, it is possible to easily check the image and trimap at the same time, and to easily check the image or trimap when a condition or operation occurs that affects the image or trimap area.

[0099] <Fortieth embodiment> A configuration will be described that makes it easy for a user to recognize the relationship between a threshold value used in generating a trimap output by the image processing device 100 and distance information of a photographed subject. In this embodiment, an example will be described in which a distance distribution display histogram is generated from the distribution of distance information and output.

[0100] 23 is a flowchart of a process for generating a distance distribution display histogram from the distribution of distance information and displaying it on display unit 114. The process of this flowchart is executed when a user operates operation unit 113 to select a histogram generation mode. Each process in this flowchart is realized by CPU 102 expanding a program stored in ROM 103 into RAM 104 and executing it.

[0101] In S4001, the CPU 102 acquires the foreground threshold and the background threshold set in S1002 of the tenth embodiment, and stores them in the RAM 104. S4004 is similar to S1003 in Fig. 3, and therefore a description thereof will be omitted.

[0102] In S4005, CPU 102 determines whether the display setting of the distance distribution display histogram is ON or OFF. The display setting of the distance distribution display histogram is set by the user operating a menu using operation unit 113. If the display setting is ON, the process proceeds to step S4006, and if the display setting is OFF, the process proceeds to step S4014.

[0103] In S4006, the CPU 102 generates a distance distribution display histogram based on the distance information obtained in S4004. In this embodiment, the CPU 102 obtains distance information of corresponding pixels in the image obtained from the frame memory 111 in S4004, and generates a distance distribution display histogram that expresses the distribution of the distance information.

[0104] The distance distribution display histogram has distance on the horizontal axis, and the center value is where distance information is 0. Distance has a ± range, with the direction away from the image processing device being the positive direction. For example, the actual distance (meters) is normalized to a real value between -128 and 127, and the in-focus position is expressed as 0. Furthermore, the number of pixels in the image having each distance value is expressed as a frequency on the vertical axis.

[0105] Fig. 24 is an example showing the relationship between a captured overall landscape and a distance distribution display histogram. Fig. 24(a) shows a scene in which a subject 4102 to be cut out, an object 4103 not to be cut out, and a background 4104 are arranged in front of the image processing device 100. Consider a case in which the image processing device 100 captures this scene with the focus on the subject 4102 and attempts to cut out only the subject 4102. When the image processing device 100 captures this scene, the CPU 102 generates a distance distribution display histogram 4109 as shown in Fig. 24(b) from a distribution corresponding to the distance at which the subject 4102, the object 4103, and the background 4104 are arranged.

[0106] In S4007, the CPU 102 reads out the foreground threshold and the background threshold stored in the RAM 104. The foreground threshold is composed of a first foreground threshold having a negative value and a second foreground threshold having a positive value. The background threshold is composed of a first background threshold having a negative value and a second background threshold having a positive value.

[0107] In S4008, the CPU 102 superimposes the foreground threshold and the background threshold read in S4007 on the distance distribution display histogram generated in S4006. Specifically, as shown in FIG. 24B, the CPU 102 superimposes a vertical dotted line 4106 at a position corresponding to the first foreground threshold and a vertical dotted line 4107 at a position corresponding to the second foreground threshold on the horizontal axis of the distance distribution display histogram 4109. Next, the CPU 102 superimposes a vertical dotted line 4105 at a position corresponding to the first background threshold and a vertical dotted line 4108 at a position corresponding to the second background threshold. This makes it possible to show the positional relationship between the subject to be cut out and the thresholds. Note that the method of superimposing the foreground threshold and the background threshold on the distance distribution display histogram is not limited to this, and other superimposing methods may be used as long as the positions of the foreground threshold and the background threshold can be recognized and the foreground region, the background region, and the unknown region can be distinguished from each other. For example, the background of the distance distribution display histogram may be color-coded into foreground, background, and unknown regions.

[0108] 24(b), the CPU 102 may color the foreground region 4112 in white, the background region 4110 and the background region 4114 in black, and the unknown region 4111 and the unknown region 4113 in gray on the horizontal axis of the distance distribution display histogram. This allows the distance distribution display histogram to be displayed so that it is easy to recognize whether each distribution belongs to the foreground region, the background region, or the unknown region. Note that the method of showing the foreground region, the background region, and the unknown region in the distance distribution display histogram is not limited to this, and other methods may be used as long as they realize a display in which the foreground region, the background region, and the unknown region can be easily recognized.

[0109] In S4009, the CPU 102 acquires an image from the frame memory 111. In S4010, the CPU 102 superimposes the distance distribution display histogram generated in S4008 on the image acquired in S4009.

[0110] 25 is a diagram showing an example in which a distance distribution display histogram 4205 is superimposed on the bottom of an image 4206 acquired in S4009. This allows the user to check the image and the distance distribution display histogram at the same time. Note that when the image and the distance distribution display histogram are superimposed, they are not limited to being arranged above and below each other, and any other superimposing method may be used as long as it allows the image and the distance distribution display histogram to be checked at the same time. For example, the image and the distance distribution display histogram may be displayed side by side, or the distance distribution display histogram may be made transparent and superimposed on a part of the image.

[0111] In S4011, the CPU 102 outputs the image as shown in FIG. 25, which is synthesized in S4010, to the display unit 114, and displays it on the display unit 114. In S4012, the CPU 102 determines whether at least one of the foreground threshold and the background threshold, which are set by operating the menu using the operation unit 113 as shown in FIG. 5 and FIG. 6 of the tenth embodiment, has been changed. The CPU 102 determines whether or not there has been a change by comparing the foreground threshold and the background threshold stored in the RAM 104 with the foreground threshold and the background threshold set by operating the menu using the operation unit 113. If the threshold has been updated (if at least one of the foreground threshold and the background threshold has been changed), the process proceeds to S4013, and if the threshold has not been updated, the process returns to S4004. The process of S4013 is the same as S4001, and therefore the description will be omitted. This allows the user to adjust each threshold while checking the distance distribution display histogram and the image.

[0112] Next, a case where the process proceeds from S4005 to S4014 will be described. The process of S4014 is similar to that of S4009, and therefore a description thereof will be omitted. In S4015, CPU 102 outputs the image acquired in S4014 to display unit 114, and causes display unit 114 to display the image. In this way, when the distance distribution display histogram is set to not be displayed, only the captured image can be displayed on display unit 114.

[0113] As described above, according to this embodiment, the distribution of distance information of an image is represented by a distance distribution display histogram, and the user can easily recognize the relationship between the threshold value used to generate a trimap and the distance information of the subject being photographed. In addition, the user can adjust the threshold value range while visually checking it.

[0114] <Forty-first embodiment> In the fortieth embodiment, an example was described in which a distance distribution display histogram is generated from the distribution of distance information, and is displayed so that the positional relationship between the subject and the foreground and background thresholds can be easily recognized. Also, an example was described in which the foreground and background thresholds are displayed, allowing the user to adjust the threshold range while visually checking it. However, in the above embodiment, when the subject moves or moves, the user may not notice that the subject is outside the range of the background threshold, and the trimap intended by the user may not be generated, and the subject may not be cut out in the intended form.

[0115] In contrast, in the 41st embodiment, a configuration will be described in which a distance distribution display histogram and an image are emphasized in order to reduce the possibility of a captured subject going outside the range of the background threshold and resulting in a failed extraction.

[0116] FIG. 26(a) shows a state where a part (part 4301) of the subject 4102 protrudes beyond the vertical dotted line 4105 (first background threshold) in a scene similar to that shown in FIG. 24(a) of the 40th embodiment. If an image is captured in this state, a trimap in which the part 4301 is the background region is output from the image processing device 100, and re-capture is required. For example, if an external PC performs a cropping process using a trimap in which the part 4301 is the background region, the image will lose the part 4301 of the subject 4102 (i.e., cropping fails). In this embodiment, the part protruding outside the range of the background threshold, such as the part 4301, is highlighted and shown to the user before and during capture, prompting the user to adjust the position of the subject and the background threshold, thereby preventing re-capture due to a failure to generate a trimap.

[0117] FIG. 26(b) shows a distance distribution display histogram 4302 with a foreground threshold, a background threshold, and a display threshold superimposed thereon. The display threshold defines the range of the distance distribution display histogram to be displayed on the display unit 114. When the distance distribution display histogram is displayed for all the captured scenery as in FIG. 24(b) of the 40th embodiment, the histogram of the background 4104 is also displayed at the same time. However, the histogram of the background 4104 is not necessary for adjusting the foreground threshold and the background threshold, and it is easier to recognize the relationship between the subject and the threshold by hiding it. Therefore, in this embodiment, the display threshold is set so that unnecessary histograms can be hidden. The display threshold is calculated from the background threshold and the display range offset value, and is composed of a first display threshold having a negative value and a second display threshold having a positive value. The image processing device 100 displays only the distance distribution display histograms that belong to the range from the first display threshold to the second display threshold, and hides histograms outside the range.

[0118] 27, 28A, 28B, 29A, and 29B are flowcharts for generating a distance distribution display histogram from the distribution of distance information, and outputting an image in which the protrusion of a subject into a background region is emphasized to display unit 114. These flowcharts are executed when a user operates operation unit 113 to select a mode for performing histogram generation and image emphasis expression. Each process in these flowcharts is realized by CPU 102 expanding a program stored in ROM 103 into RAM 104 and executing it.

[0119] 27, the processes of S4401 and S4404 are similar to those of S4001 and S4004 in the fortieth embodiment, and therefore description thereof will be omitted. In S4405, the CPU 102 generates a distance distribution display histogram based on the distance information obtained in S4404.

[0120] 28A and 28B are flowcharts showing details of the processing of S4405. In S4501, CPU 102 determines whether the display setting of the distance distribution display histogram is ON or OFF. The display setting of the distance distribution display histogram is set by the user operating a menu using operation unit 113. If it is ON, the processing step proceeds to S4502, and if it is OFF, the processing step proceeds to S4520.

[0121] The processing of S4502 and S4503 is similar to S4006 and S4007 in the 40th embodiment, and therefore description thereof will be omitted. In S4504, the CPU 102 acquires a display range offset value stored in advance in the ROM 103. Note that the storage location of the display range offset value is not limited to the ROM 103, and the recording medium 112 may be used. Also, the display range offset value may be changed arbitrarily by the user. For example, the user operates a menu using the operation unit 113 to select a display range offset value, and the CPU 102 acquires the display range offset value from the operation unit 113.

[0122] In S4505, the CPU 102 calculates the display threshold based on the background threshold read in S4503 and the display range offset value acquired in S4504. A specific calculation method of the display threshold will be described with reference to FIG. 26(b). First, the CPU 102 subtracts the display range offset value 4308 from the vertical dotted line 4105 (first background threshold) to obtain the first display threshold (vertical dotted line 4303). Next, the CPU 102 adds the display range offset value 4309 to the vertical dotted line 4108 (second background threshold) to obtain the second display threshold (vertical dotted line 4304). In this way, two display thresholds are determined. Note that the calculation of the display threshold is not limited to the addition and subtraction of the display range offset value, and other calculation methods may be used as long as the relationship in which the second display threshold is greater than the first display threshold is maintained within the range of the distance information. Regarding the display range offset value, the offset value used to calculate the first display threshold and the offset value used to calculate the second display threshold may be the same value or different values.

[0123] In S4506, the CPU 102 superimposes the foreground threshold and background threshold read in S4503 and the display threshold calculated in S4505 on the distance distribution display histogram generated in S4502. The method of superimposing the foreground threshold and background threshold on the distance distribution display histogram is the same as S4008 in the 40th embodiment, and therefore the description will be omitted. With reference to FIG. 26(b), the method of superimposing the display threshold on the distance distribution display histogram will be described. The CPU 102 superimposes a vertical dotted line 4303 at a position that coincides with the first display threshold on the horizontal axis of the distance distribution display histogram 4302, and a vertical dotted line 4304 at a position that coincides with the second display threshold. Note that the method of superimposing the display threshold on the distance distribution display histogram is not limited to this, and any other superimposing method may be used as long as the position of the display threshold can be recognized. For example, the background of the distance distribution display histogram that belongs to the range of the display threshold may be colored, or a single pattern such as a striped pattern or a checkered pattern may be superimposed.

[0124] In S4507, the CPU 102 acquires color setting information stored in advance in the ROM 103. The color setting information is information on colors designated for each area in order to distinguish the areas to which the distance distribution display histogram and image belong. In this embodiment, if the area belongs to the foreground area or the unknown area, the area is colored with a first color. If the distance information is negative, the background area is colored with a second color, and if the distance information is positive, the background area is colored with a third color. The storage location of the color setting information is not limited to the ROM 103, and the recording medium 112 or the like may be used. In addition, the color setting information may be changed by the user as desired. For example, the user operates a menu using the operation unit 113 to designate the first color, the second color, and the third color, and the CPU 102 acquires the color setting information from the operation unit 113.

[0125] In S4508, the CPU 102 acquires the number of classes of the distance distribution display histogram. The acquired number of classes is stored as a variable Nmax in the RAM 104. For example, if the number of classes of the distance distribution display histogram is 256, the variable Nmax is 256.

[0126] In S4509, the CPU 102 focuses on the class of the distance distribution display histogram with the shortest distance information. Specifically, the class of the distance distribution display histogram to be focused on is set as a variable n, n is set to 1, and stored in the RAM 104. The larger the variable n, the greater the histogram corresponds to a class of distances farther from the image processing device.

[0127] In S4510, CPU 102 determines whether variable n is within the range from the first display threshold to the second display threshold, If it is within the range of the display threshold, processing proceeds to S4511, If ​​it is not within the range, processing proceeds to S4516.

[0128] At S4511, CPU 102 determines whether variable n is within a range from the first background threshold to the second background threshold, and if it is within the range from the first background threshold to the second background threshold, the process proceeds to S4512, and if it is not within the range from the first background threshold to the second background threshold, the process proceeds to S4513.

[0129] In S4512, the CPU 102 sets the histogram of the class of the variable n to be colored in the first color.

[0130] At S4513, CPU 102 determines whether variable n is within the range from the first display threshold to the first background threshold. If it is within the range from the first display threshold to the first background threshold, processing proceeds to S4514, and if it is not within the range from the first display threshold to the first background threshold, processing proceeds to S4515.

[0131] In S4514, the CPU 102 sets the histogram of the class of the variable n to be colored in the second color.

[0132] In S4515, the CPU 102 sets the histogram for the class of the variable n to be colored in the third color.

[0133] In S4516, the CPU 102 sets the histogram for the class of the variable n to be hidden.

[0134] In S4517, CPU 102 determines whether variable n is equal to the number of classes Nmax of the histogram. If equal, processing proceeds to S4517, and if not equal, processing proceeds to S4518.

[0135] In S4518, the CPU 102 assigns n+1 to the variable n, and stores the result in the RAM 104. As a result, the histogram that the CPU 102 focuses on is shifted to the next higher class.

[0136] In S4519, the CPU 102 stores the distance distribution display histogram to which the color setting has been applied in the RAM 104.

[0137] The processes of S4520 and S4521 are similar to those of S4012 and S4013 in the 40th embodiment, and therefore the description thereof will be omitted. If the determination in S4520 is "NO", the process proceeds to S4406 in FIG.

[0138] As described above, by the CPU 102 executing the processes in the flowcharts of FIGS. 28A and 28B, it is possible to generate a distance distribution display histogram in which the distribution existing outside the range of the background threshold is emphasized.

[0139] Please refer to Fig. 27 again. In S4406, the CPU 102 generates an image by applying emphasis to the image obtained by the image processing unit 105, based on the distance information obtained in S4404.

[0140] 29A and 29B are flowcharts showing details of the process of S4406. In S4601, the CPU 102 acquires information on the image and image size from the image processing unit 105. The CPU 102 stores the horizontal size of the image size as Xmax and the vertical size as Ymax in the RAM 104.

[0141] In S4602, the CPU 102 focuses on the distance information corresponding to the pixel (x, y) among the distance information calculated in S4404. Note that the variable x represents the coordinate on the horizontal axis of the image, and the variable y represents the coordinate on the vertical axis of the image.

[0142] In S4603, CPU 102 determines whether the distance information of the pixel (x, y) focused on in S4602 is within the range from the first display threshold to the second display threshold. If it is within the range of the display thresholds, processing proceeds to S4604, and if it is not within the range, processing proceeds to S4608.

[0143] In S4604, CPU 102 determines whether the distance information of the pixel (x, y) of interest in S4602 is within the range from the first background threshold to the second background threshold. If it is within the range of the background thresholds, processing proceeds to S4608, and if it is not within the range, processing proceeds to S4605.

[0144] In S4605, CPU 102 determines whether the distance information of pixel (x, y) focused on in S4602 is within the range from the first display threshold to the first background threshold. If it is within the range from the first display threshold to the first background threshold, processing proceeds to S4606, and if it is not within the range, processing proceeds to S4607.

[0145] In S4606, the CPU 102 performs setting so as to superimpose the second color acquired in S4507 on the pixel (x, y) of the image acquired in S4601.

[0146] In S4607, CPU 102 performs setting so as to superimpose the third color acquired in S4507 on pixel (x, y) of the image acquired in S4601.

[0147] In S4608, CPU 102 determines whether variable x is equal to the horizontal size Xmax of the image. If equal, processing proceeds to S4610, and if not equal, processing proceeds to S4609.

[0148] In S4609, the CPU 102 assigns x+1 to the variable x and stores the result in the RAM 104. As a result, the CPU 102 focuses on the pixel immediately to the right on the same line.

[0149] In S4610, CPU 102 determines whether variable y is equal to Ymax, the vertical size of the image. If equal, processing proceeds to S4612, and if not equal, processing proceeds to S4611.

[0150] In S4611, 0 is substituted for the variable x, and y+1 is substituted for the variable y, and the result is stored in the RAM 104. As a result, the CPU 102 focuses on the top pixel one line below.

[0151] In S4612, the CPU 102 stores in the RAM 104 the image that has been subjected to the processes shown in S4603 to S4611.

[0152] As described above, by the CPU 102 executing the processes in the flowcharts of FIGS. 29A and 29B, an image in which subjects existing outside the range of the background threshold are emphasized can be generated.

[0153] Referring again to Fig. 27, in S4407, the CPU 102 superimposes the distance distribution display histogram generated in S4405 on the highlighted image generated in S4406.

[0154] FIG. 30 shows an example in which a distance distribution display histogram 4302 is superimposed on the bottom of an image 4703 processed by the image processing unit 105. A distribution 4305 of the distance distribution display histogram that is within the range from the first background threshold to the second background threshold is colored in a first color. An area 4701 of the image that is within the range from the first display threshold to the first background threshold and a distribution 4306 of the distance distribution display histogram are colored and highlighted in a second color. An area 4702 of the image that is within the range from the second background threshold to the second display threshold and a distribution 4307 of the distance distribution display histogram are colored and highlighted in a third color. This allows the user to simultaneously check the image and the distance distribution display histogram that are outside the range of the background thresholds among the subjects being photographed.

[0155] Furthermore, if the subject moves during shooting and a part of the subject protrudes beyond the background threshold, the CPU 102 performs emphasis similar to that performed on the image regions 4701 and 4702, and on the distance distribution display histogram distributions 4306 and 4307. This makes it possible to notify the user in real time that a part of the subject has protruded, thereby preventing the need to reshoot.

[0156] Note that the image and the distance distribution display histogram are not limited to being arranged above and below each other when being superimposed, and other superimposing methods may be used as long as they allow the image and the distance distribution display histogram to be simultaneously confirmed. For example, the image and the distance distribution display histogram may be displayed side by side, or the distance distribution display histogram may be made transparent and superimposed on a part of the image.

[0157] In S4408, CPU 102 outputs the image generated in S4407 to display unit 114 to display it.

[0158] As described above, according to this embodiment, when the subject being photographed goes outside the range of the background threshold, the distance distribution display histogram and the image are colored to notify the user, thereby making it possible to prevent the need to re-shoot due to a failed extraction.

[0159] <Forty-second embodiment> In the 40th embodiment, an example was described in which a distance distribution display histogram is generated from the distribution of distance information and displayed so that the positional relationship between the subject and the foreground and background thresholds can be easily recognized. Also, an example was described in which the foreground and background thresholds are displayed, allowing the user to adjust the threshold range while visually checking it. Also, in the 41st embodiment, an example was described in which a distance distribution display histogram and an image are emphasized and presented to the user in order to prevent the subject to be photographed from jumping out of the background threshold range and the subject to be photographed from failing to be cropped.

[0160] However, it is unclear to the user which parts of the image have distance information of 0, and the user cannot fully grasp the relationship between the subject of the image and the distribution of the distance distribution display histogram.

[0161] Therefore, in the forty-second embodiment, an example will be described in which pixels in an image whose distance information is 0 are colored and presented to the user together with a distance distribution display histogram.

[0162] According to this embodiment, it is possible to clearly indicate pixels whose distance information is 0, making it easier for the user to identify which part of the subject being photographed corresponds to the distance distribution display histogram.

[0163] 31A and 31B are flowcharts for generating a distance distribution display histogram from the distribution of distance information and displaying it on display unit 114. This flowchart is executed when a user operates operation unit 113 to select a mode for generating a histogram. Each process in this flowchart is realized by CPU 102 expanding a program stored in ROM 103 into RAM 104 and executing it.

[0164] The processes of S4801 and S4804 are similar to those of S4001 and S4004 in the fortieth embodiment, and therefore the description thereof will be omitted.

[0165] In S4805, CPU 102 acquires coloring setting information stored in advance in ROM 103. The coloring setting information has information on a fourth color to be applied to pixels with distance information of 0. The coloring setting information may be stored in ROM 103, and may be stored in recording medium 112 or the like. The coloring setting information may be changed by the user as desired. For example, the user operates a menu using operation unit 113 to specify the fourth color, and CPU 102 acquires coloring setting information from operation unit 113.

[0166] The processing from S4806 to S4809 is similar to that from S4005 to S4008 in the fortieth embodiment, and therefore the description thereof will be omitted.

[0167] In S4810, the CPU 102 acquires an image from the frame memory 111. In S4811, the CPU 102 sets a flag to 1 for pixels whose distance information is 0 and sets a flag to 0 for other pixels, with respect to the distance information acquired in S4804, and stores the set flags in the frame memory 111.

[0168] In S4812, CPU 102 refers to the flag stored in frame memory 111 in S4811. For pixels whose flag is 1, CPU 102 colors the corresponding pixel in the image acquired in S4810 with the fourth color acquired in S4805. For pixels whose flag is 0, CPU 102 uses the pixel in the image acquired in S4810 as is. As a result, an image onto which the fourth color is partially superimposed is generated.

[0169] In S4813, the CPU 102 superimposes the distance distribution display histogram generated in S4809 on the image generated in S4812.

[0170] 32 is a diagram showing an example in which a distance distribution display histogram 4205 is superimposed on the bottom of an image 4902 processed in S4812. In the image 4902, a pixel corresponding to a portion 4901 of the subject has distance information corresponding to that pixel that is 0, and therefore is colored in the fourth color by the processing of S4812. This allows the user to confirm that, among the subject being photographed, the distance information of the portion 4901 is 0.

[0171] Note that the image and the distance distribution display histogram are not limited to being arranged above and below each other when being superimposed, and other superimposing methods may be used as long as they allow the image and the distance distribution display histogram to be simultaneously confirmed. For example, the image and the distance distribution display histogram may be displayed side by side, or the distance distribution display histogram may be made transparent and superimposed on a part of the image.

[0172] In S4814, the CPU 102 outputs the image generated in S4813 to the display unit 114 to display it.

[0173] The processes of S4815 and S4816 are similar to those of S4012 and S4013 in the fortieth embodiment, and therefore the description thereof will be omitted.

[0174] The processes of S4817 and S4818 are similar to those of S4014 and S4015 in the 40th embodiment, and therefore the description thereof will be omitted. As a result, when the distance distribution display histogram is set to not be displayed, only the captured image can be displayed on the display unit 114.

[0175] As described above, according to this embodiment, it is possible to clearly indicate areas of a subject in an image of the subject where distance information is 0, making it easier to identify which part of the subject being photographed the distance distribution display histogram corresponds to.

[0176] <Fiftieth embodiment> As one embodiment, a trimap can be generated using parallax information and a defocus amount that can be calculated by the CPU 102 based on information obtained from an image plane phase difference sensor. In actual shooting, there is a problem that it is not possible to check in real time whether the captured image and the foreground area in the trimap match. In this embodiment, a configuration is described that solves this problem by generating and outputting an overhead view image from distance information, thereby clearly indicating the image that is the foreground area in real time.

[0177] The overhead view image will be described with reference to Fig. 34 and Fig. 35. Fig. 35(a) is an image acquired by the image processing device 100. In Fig. 35(a), it is assumed that the image processing device 100 focuses on a subject 5201. The image processing device 100 calculates distance information by the method described above.

[0178] Fig. 35(b) is a diagram overlooking the distribution of distance information for each pixel in an image including a background 5202, with the distance information of a subject 5201 on which the image processing device 100 is focused in Fig. 35(a) being set to 0. Fig. 35(b) is a graph with the vertical axis representing the distance information acquired by the image processing device 100 and the horizontal axis representing the horizontal coordinate of the image (horizontal coordinate), and is drawn by distributing the distance information in the image by dots or regions. Fig. 35(b) is what is displayed on the display unit 114.

[0179] FIG. 34 is a diagram showing the relationship between the assumed distance between the subject and the background in the image, assuming an overhead view from above, for the image in FIG. 35(a). The region 5101 is a range that the image processing device 100 recognizes as a foreground region, and is determined by the upper and lower limits of the distance information including the subject (foreground threshold range). The region 5101 is displayed on the display unit 114, and is drawn by a straight line 5102 in the horizontal axis direction that represents the upper and lower limits of the distance information. In addition, this region may be drawn by a method that explicitly indicates that it is within the region 5101, such as by displaying the color of a dot or region that is the distribution of distance information within the range of the region 5101 in a different color from the background, instead of drawing by the straight line 5102. In addition, although not shown, the range of the background threshold is also displayed in FIG. 34.

[0180] 33 is a flowchart of a process for generating an overhead view image from the distribution of distance information and displaying it on the display unit 114. Each process in this flowchart is realized by the CPU 102 expanding a program stored in the ROM 103 into the RAM 104 and executing it.

[0181] The processes of S5001 and S5004 are similar to those of S4001 and S4004 in the fortieth embodiment, and therefore the description thereof will be omitted.

[0182] In S5005, CPU 102 determines whether the display setting of the overhead view image is ON or OFF. The display setting of the overhead view image is set by the user operating a menu using operation unit 113. If it is ON, the process proceeds to step S5006, and if it is OFF, the process proceeds to step S5014.

[0183] In S5006, the CPU 102 generates an overhead view image as shown in FIG. 35(b) based on the distance information obtained in S5004.

[0184] The process of S5007 is similar to S4007 in the fortieth embodiment, and therefore a description thereof will be omitted.

[0185] In S5008, the CPU 102 superimposes a foreground threshold and a background threshold on the overhead view image.

[0186] The process of S5009 is similar to S4009 in the fortieth embodiment, and therefore a description thereof will be omitted.

[0187] In S5010, the CPU 102 combines the overhead view image generated in S5008 and the image acquired in S5009 into a parallel or overlaid image. In S5011, the CPU 102 outputs the image generated in S5010 to the display unit 114.

[0188] The processes of S5012 and S5013 are similar to those of S4012 and S4013 in the fortieth embodiment, and therefore the description thereof will be omitted.

[0189] The processes of S5014 and S5015 are similar to those of S4014 and S4015 in the fortieth embodiment, and therefore the description thereof will be omitted.

[0190] As described above, according to this embodiment, by generating and outputting an overhead view image from distance information, it is possible to clearly display an image that is to be a foreground region in real time.

[0191] <Fifty-first embodiment> As described in the 50th embodiment, by generating and outputting an overhead view image from distance information, it is possible to clearly display an image that is to be the foreground region in real time.

[0192] On the other hand, the method described in the 50th embodiment has a problem that when the subject itself requires a deep depth of field, it is difficult to check in real time whether it is outside the image separation area. This embodiment describes a method that expects the effect of making it easier to understand the part that is outside the image separation area.

[0193] The present embodiment is configured to carry out the process described in the 50th embodiment, which is expected to have the effect of making the captured image and the overhead view image easier to understand.

[0194] Fig. 36(a) is an image acquired by the image processing device 100, and Fig. 36(b) is an overhead view image generated by the processing of Fig. 33 described in the 50th embodiment. An object 5301 in Fig. 36(a) is contained in the same image as a background 5302. The background 5302 has a different relative distance from the object 5301, whose relative distance is zero, and is assumed to be at a distance that is desired to be recognized as a background region when generating a trimap.

[0195] Region 5306 in Fig. 36(b) is a range between the thresholds of distance information to be recognized as a foreground region when the trimap is generated, and is determined based on the foreground threshold. Region 5308 in Fig. 36(b) is a range between the thresholds of distance information to be recognized as a background region when the trimap is generated, and is determined based on the background threshold. Region 5307 in Fig. 36(b) is a range between the thresholds of distance information to be recognized as an unknown region when the trimap is generated, and is determined based on the foreground threshold and background threshold.

[0196] A subject 5301 in Fig. 36(a) is holding a rod-shaped tool 5303. In this state, it is assumed that the image processing device 100 has acquired an image. It is also assumed that an area 5304 at the tip of the tool 5303 is distant from the subject 5301 in focus, and the distance information of the area 5304 is within a range recognized as a background area in Fig. 36(b).

[0197] In this embodiment, the CPU 102 performs a process of coloring, in a predetermined color, a portion (region 5304) of the captured image and the overhead view image where the tool 5303 overlaps with the region 5308. In this embodiment, the CPU 102 also performs a process of coloring, in a predetermined color, a portion (region 5305) of the captured image and the overhead view image where the region 5308 overlaps with the background 5302.

[0198] As described above, according to this embodiment, it is possible to expect the effect of making it easier to understand the portion that is outside the region of the image separation.

[0199] <Fifty-second embodiment> As described in the 50th and 51st embodiments, it is possible to clearly display an image that is a foreground area in real time by generating and outputting an overhead view image from distance information. On the other hand, the method described in the 50th and 51st embodiments has a problem that it is difficult to check in real time whether the subject itself is in focus. This embodiment describes a method for easily checking whether the in-focus area is equal to the subject itself.

[0200] This embodiment is configured to perform processing on a captured image and an overhead view image with the expectation of making it easier to see where an in-focus portion is located.

[0201] In this embodiment, the CPU 102 performs processing for coloring the corresponding pixels in the image shown in FIG. 37(a) in a predetermined color for pixels corresponding to an area 5402 in which the relative distance is recognized as 0 as shown in FIG. 37(b).

[0202] By visually checking both the area 5401 and the subject in the image on FIG. 37(a), the user can check whether the subject itself is in focus in the image acquired by the image processing device 100.

[0203] As described above, according to this embodiment, it is possible to easily check whether the in-focus area is the same as the subject itself.

[0204] <60th embodiment> In order to reduce the bandwidth of the internal bus 101, the imaging unit 107 of the image processing device 100 can collectively transmit disparity information of a range of multiple pixels of an image signal as shown in Fig. 38. Fig. 38 is a diagram showing a part of the output of the imaging unit 107 and a part of a trimap generated from the disparity information output from the imaging unit 107. In this embodiment, the imaging unit 107 will be described as collectively transmitting disparity information of a range of 12 pixels of an image signal.

[0205] In the parallax information range A shown in Fig. 38, all 12 pixels within the range capture the background, so all 12 pixels are background regions. In the parallax information range C, all 12 pixels within the range capture the subject, so a trimap is generated with all 12 pixels as the foreground region. In the parallax information range B, the background, the subject, and the boundary between the background and the subject are captured in the 12 pixels within the range, but since the parallax information is consolidated, a trimap is generated with all 12 pixels as unknown regions. Therefore, the area occupied by the unknown regions in the generated trimap becomes large.

[0206] In the 60th embodiment, an example is described in which the edge detection results of the image signal are used to reclassify pixels of an unknown region into a foreground region, a background region, and an unknown region in units more detailed than the disparity information range, and a second trimap is generated in which the area of ​​the unknown region is reduced.

[0207] 39A and 39B are a flowchart of a second trimap generation process in the sixtieth embodiment. Each process in this flowchart is realized by the CPU 102 loading a program recorded in the ROM 103 into the RAM 104 and executing it.

[0208] In S6001, the CPU 102 generates a first trimap by performing processes similar to those in S1003 to S1008 described in the tenth embodiment. The CPU 102 records the first trimap in the frame memory 111.

[0209] In S6002, the CPU 102 performs edge detection by having the image processing unit 105 process the image signal read from the frame memory 111. The edge detection performed by the image processing unit 105 detects positions where the luminance change or color change of the image signal is discontinuous, for example. Specifically, the edge detection is realized by a gradient method or a Laplacian method. The CPU 102 records the edge detection result processed by the image processing unit 105 in the frame memory 111. The image processing unit 105 outputs the edge detection result as a flag for each pixel of the image signal indicating whether or not it corresponds to an edge.

[0210] In S6003, the CPU 102 reads out an area of ​​the first trimap corresponding to the disparity information range to be processed from the frame memory 111, and determines whether the area is classified as an unknown area. If the disparity information range to be processed is classified as an unknown area, the process proceeds to S6004. If the disparity information range to be processed is not classified as an unknown area, the process proceeds to S6016.

[0211] In S6004, the CPU 102 reads out an area of ​​the edge detection result corresponding to the disparity information range to be processed from the frame memory 111, and determines whether or not there is a pixel corresponding to an edge within that area. If the disparity information range to be processed includes a pixel corresponding to an edge, the process proceeds to S6005. If the disparity information range to be processed does not include a pixel corresponding to an edge, the process proceeds to S6016.

[0212] In S6005, the CPU 102 maintains pixels corresponding to edges in the area of ​​the first trimap corresponding to the disparity information range to be processed as unknown areas.

[0213] In S6006, the CPU 102 reads out from the frame memory 111 an area of ​​the first trimap that corresponds to the adjacent disparity information range to the left of the disparity information range to be processed, and determines whether the area is classified as a foreground area. If the disparity information range to the left is classified as a foreground area, the process proceeds to S6007. If the disparity information range to the left is not classified as a foreground area, the process proceeds to S6008.

[0214] In S6007, the CPU 102 changes the pixels located to the left of the edge pixels in the first trimap area corresponding to the disparity information range to the foreground area. The CPU 102 records the changed trimap in the frame memory 111.

[0215] In S6008, the CPU 102 reads out from the frame memory 111 an area of ​​the first trimap corresponding to the adjacent disparity information range to the left of the disparity information range to be processed, and determines whether the area is classified as a background area. If the adjacent disparity information range to the left is classified as a background area, the process proceeds to S6009. If the adjacent disparity information range to the left is not classified as a background area, the process proceeds to S6010.

[0216] In S6009, the CPU 102 changes the pixels located to the left of the pixels corresponding to the edge in the first trimap area corresponding to the disparity information range to the background area. The CPU 102 records the changed trimap in the frame memory 111.

[0217] In S6010, the CPU 102 maintains, in the area of ​​the first trimap corresponding to the disparity information range to be processed, pixels disposed to the left of the pixels corresponding to the edge as unknown areas.

[0218] In S6011, the CPU 102 reads out from the frame memory 111 an area of ​​the first trimap that corresponds to a disparity information range adjacent to the right of the disparity information range to be processed, and determines whether the area is classified as a foreground area. If the disparity information range to the right is classified as a foreground area, the process proceeds to S6012. If the disparity information range to the right is not classified as a foreground area, the process proceeds to S6013.

[0219] In S6012, the CPU 102 changes the pixels located to the right of the edge pixels in the first trimap area corresponding to the disparity information range to the foreground area. The CPU 102 records the changed trimap in the frame memory 111.

[0220] In S6013, the CPU 102 reads out from the frame memory 111 an area of ​​the first trimap corresponding to the adjacent disparity information range on the right side of the disparity information range to be processed, and determines whether the area is classified as a background area. If the adjacent disparity information range on the right is classified as a background area, the process proceeds to S6014. If the adjacent disparity information range on the right is not classified as a background area, the process proceeds to S6015.

[0221] In S6014, the CPU 102 changes the pixels located to the right of the edge pixels in the first trimap area corresponding to the disparity information range to the background area. The CPU 102 records the changed trimap in the frame memory 111.

[0222] In S6015, the CPU 102 maintains, in the area of ​​the first trimap corresponding to the disparity information range to be processed, pixels disposed to the right of the pixels corresponding to the edge as unknown areas.

[0223] In S6016, the CPU 102 determines whether or not all disparity information ranges of the image signal recorded in the frame memory 111 have been processed. If all disparity information ranges have been processed, the process proceeds to S6018. If all disparity information ranges have not been processed, the process proceeds to S6017.

[0224] In S6017, the CPU 102 selects an unprocessed disparity information range as a next processing target. For example, the processing target disparity information range is selected in order from the top left in the raster direction. After that, the processing step returns to S6003.

[0225] In S6018, the CPU 102 outputs the trimap recorded in the frame memory 111 as a second trimap to the outside from the image terminal 109 or the network terminal 108. The CPU 102 may record the second trimap in the recording medium 112.

[0226] FIG. 40 is a diagram showing a part of the output of the imaging unit 107, a part of the first trimap, a part of the edge detection result described in S6002, and a part of the second trimap obtained by the processing from S6003 to S6015. In FIG. 40, the output of the imaging unit 107 and the first trimap are the same as the output of the imaging unit 107 and the trimap in FIG. 38, and therefore the description will be omitted. The pixels at the boundary between the background and the subject are determined to be edges as indicated by diagonal lines in the edge detection result in FIG. 40 by the edge detection in S6002. The second trimap is generated by the processing from S6003 to S6015. In FIG. 40, the pixels at the edge of the disparity information range B are classified as an unknown region, the pixels sandwiched between the pixels at the edge of the disparity information range B and the disparity information range A are classified as a background region, and the pixels sandwiched between the pixels at the edge of the disparity information range B and the disparity information range C are classified as a foreground region.

[0227] As described above, according to the sixtieth embodiment, by using the edge detection result of the image signal, it is possible to reclassify the pixels of the unknown region into a foreground region, a background region, and an unknown region in units finer than the parallax information range, and generate a second trimap in which the area of ​​the unknown region is reduced. By reducing the area of ​​the unknown region of the trimap, it is possible to improve the detection accuracy of the neural network that uses the trimap to cut out the foreground and background.

[0228] <70th embodiment> When photographing a subject such as a human body down to the feet, the floor surface near where the feet are in contact with the ground is at approximately the same distance as the subject's feet, so if a trimap is generated from distance information, the floor surface will be mistakenly determined to be in the foreground area.

[0229] In the 70th embodiment, an example is described in which a second trimap is generated by detecting the subject's feet and reclassifying the floor surface that has been erroneously determined to be a foreground area at the same relative distance as the subject's feet into an unknown area or background area.

[0230] 41 is a flowchart of a second trimap generation process in the seventieth embodiment. Each process in this flowchart is realized by the CPU 102 loading a program stored in the ROM 103 into the RAM 104 and executing it.

[0231] In S7001, the CPU 102 generates a first trimap by performing processes similar to those in S1003 to S1008 described in the tenth embodiment. The CPU 102 records the first trimap in the frame memory 111.

[0232] In S7002, CPU 102 loads parameters for detecting the feet of a human body recorded in ROM 103 into object detection unit 115, and detects the feet of a human body by having object detection unit 115 process an image read from frame memory 111. Object detection unit 115 records, as part detection information in RAM 104, two coordinates indicating diagonal vertices of a rectangle containing the area of ​​the feet detected in the image, with the horizontal direction of the image as the x-axis, the vertical direction as the y-axis, and the lower left corner of the image as coordinates (0,0).

[0233] In this embodiment, the object detection unit 115 is described as a neural network that outputs the coordinates of a detected area, but may be a neural network that detects other human skeletons.

[0234] In S7003, CPU 102 determines whether part detection information is recorded in RAM 104. If part detection information is recorded in RAM 104, CPU 102 determines that a human foot has been detected in the image, and proceeds to processing step S7004. If part detection information is not recorded in RAM 104, CPU 102 determines that a human foot has not been detected in the image, and ends the processing of this flowchart.

[0235] In S7004, the CPU 102 reads out the first trimap recorded in the frame memory 111 and the part detection information recorded in the RAM 104, and changes the inside of the rectangular area on the trimap indicated by the part detection information to an unknown area. Details of the process in S7004 will be described later with reference to FIG.

[0236] In S7005, the CPU 102 changes the area classified as a foreground area or an unknown area in the trimap, among the areas of the y coordinate within the same range as the y coordinate of the rectangle on the trimap indicated by the part detection information, and not including the x coordinate within the same range as the x coordinate of the rectangle, to a background area. The CPU 102 records the trimap changed in S7004 and S7005 in the frame memory 111. Details of the process in S7005 will be described later with reference to FIG.

[0237] In S7006, CPU 102 determines whether other body part detection information is recorded in RAM 104. If other body part detection information is recorded in RAM 104, CPU 102 determines that another body's foot has been detected in the image, and proceeds to processing step S7004 again. If no body part detection information is recorded in RAM 104, CPU 102 determines that another body's foot has not been detected in the image, and proceeds to processing step S7007.

[0238] In S7007, the CPU 102 outputs the trimap recorded in the frame memory 111 as the second trimap to the outside from the image terminal 109 or the network terminal 108. The process proceeds to an end step. The CPU 102 may record the second trimap in the recording medium 112.

[0239] The process of S7004 will be described in detail with reference to Fig. 42. Fig. 42 is a diagram showing two coordinates obtained from the part detection information output by the object detection unit 115 and a rectangle including the detected foot area indicated by the part detection information, displayed on an image recorded in the frame memory 111. The two coordinates obtained from the part detection information are (X1, Y1) and (X2, Y2). The rectangular area indicated by the four points (X1, Y1), (X2, Y1), (X1, Y2), and (X2, Y2) with these two coordinates as diagonal vertices is set as an unknown area in S7004.

[0240] The details of the process of S7005 will be described with reference to FIG. 43. FIG. 43 is a diagram showing rectangular areas set as background areas in S7005 on an image recorded in the frame memory 111. Two rectangular areas are set as background areas, which are areas from Y1 to Y2 within the same range as the y coordinate of the rectangular area (FIG. 42) corresponding to the peripheral area of ​​the foot, and do not include areas from X1 to X2 within the same range as the x coordinate of the rectangular area (FIG. 42) corresponding to the peripheral area of ​​the foot. That is, the insides of two rectangular areas, a rectangle defined by four points (X0, Y1), (X1, Y1), (X0, Y2), and (X1, Y2) and a rectangle defined by four points (X2, Y1), (X3, Y1), (X2, Y2), and (X3, Y2), are set as background areas in S7005. Note that the x coordinate X0 is the leftmost end of the image, and the x coordinate X3 is the rightmost end of the image.

[0241] As described above, according to the 70th embodiment, a second trimap can be generated in which the floor surface that is erroneously determined as a foreground area despite being at the same relative distance as the subject's feet is reclassified as an unknown area or a background area.

[0242] In this embodiment, an example has been described in which a neural network that detects the feet of a human body is used to reclassify the floor surface that is in contact with the feet of the human body into an unknown area or a background area. For example, when a car or a motorcycle is used as a subject, this embodiment can be applied by using a neural network that detects the tires that are in contact with the floor surface. Similarly, this embodiment can be applied to other subjects by using a neural network that detects the parts of other subjects that are in contact with the floor surface.

[0243] <71st embodiment> In the seventieth embodiment, an example has been described in which a second trimap is generated by reclassifying a floor surface that has been erroneously determined as a foreground region into an unknown region or a background region. However, the range of the floor surface that is erroneously determined as a foreground region at the same distance as the subject is wider if the image processing device 100 is tilted forward, and narrower if it is tilted backward.

[0244] In the 71st embodiment, an example is described in which, when generating a second trimap in which a floor surface that has been erroneously determined as a foreground area is reclassified as an unknown area or a background area, the range to be reclassified is changed by referring to the inclination of the image processing device 100 using information from an acceleration sensor for image stabilization built into the lens unit 106.

[0245] 44 is a flowchart of a second trimap generation process in the 71st embodiment. Each process in this flowchart is realized by the CPU 102 loading a program recorded in the ROM 103 into the RAM 104 and executing it.

[0246] The processes from S7101 to S7104 are similar to those from S7001 to S7004 described in the seventieth embodiment, and therefore the description thereof will be omitted.

[0247] In S7105, the CPU 102 reads out tilt information from the acceleration sensor of the lens unit 106. The tilt information is a numerical value indicating whether the image processing device 100 is tilted forward or backward. The CPU 102 determines a background region adjustment value t based on the tilt information. The background region adjustment value t is set to 0 if the image processing device 100 is horizontal to the floor surface, increases if the image processing device 100 is tilted forward, and decreases if the image processing device 100 is tilted backward.

[0248] In S7106, the CPU 102 changes to background regions areas classified as foreground regions or unknown regions in the trimap, which are within the same range as the y coordinates obtained by extending the background region adjustment value t in the y coordinate direction from the top and bottom of the rectangle on the trimap indicated by the part detection information, and which do not include an x ​​coordinate within the same range as the x coordinate of the rectangle. The CPU 102 records the trimaps changed in S7104 and S7106 in the frame memory 111. Details of the processing in S7106 will be described later with reference to FIG. 45.

[0249] The processes in S7107 and S7108 are similar to those in S7006 and S7007 described in the seventieth embodiment, and therefore a description thereof will be omitted.

[0250] The process of S7106 will be described in detail with reference to Fig. 45. Fig. 45 is a diagram showing rectangular areas set as the background area in S7106 displayed on an image recorded in frame memory 111. Two rectangular areas are set as the background area, which are areas from (Y1+t) to (Y2-t) within the same range as the y coordinate obtained by extending the background area adjustment value t in the y coordinate direction from the top and bottom of the rectangular area (Fig. 42) corresponding to the peripheral area of ​​the foot, and do not include the area from X1 to X2 within the same range as the x coordinate of the rectangular area (Fig. 42) corresponding to the peripheral area of ​​the foot. That is, the two regions, a rectangle defined by four points (X0, Y1+t), (X1, Y1+t), (X0, Y2-t), and (X1, Y2-t) and a rectangle defined by four points (X2, Y1+t), (X3, Y1+t), (X2, Y2-t), and (X3, Y2-t), are set as background regions in S7106. Note that x-coordinate X0 is the leftmost edge of the image, and x-coordinate X3 is the rightmost edge of the image.

[0251] As described above, according to the 71st embodiment, when generating a second trimap in which the floor surface that was erroneously determined as a foreground area is reclassified as a background area, the range to be reclassified as a background area can be changed by referring to the inclination of the image processing device 100 using information from the acceleration sensor for image stabilization built into the lens unit 106.

[0252] <80th embodiment> As one embodiment, a trimap can be generated using parallax information and a defocus amount that can be calculated by the CPU 102 based on information obtained from an image plane phase difference sensor. When the lens aperture is changed during shooting, the parallax information for each frame at the boundary between the foreground and background regions also changes, which causes a problem that the boundary of the unknown region changes. In this embodiment, a configuration that solves this problem will be described.

[0253] A function of the image processing device 100 generating a trimap based on disparity information will be described with reference to Fig. 46. Fig. 46 shows a process for determining a threshold value for the defocus amount for separating each boundary between the foreground region, the background region, and the unknown region when the image processing device 100 generates a trimap for each frame. The process of Fig. 46 is repeatedly executed in the image processing device 100 every time trimap generation is performed on a frame-by-frame basis.

[0254] The processes of S8001 and S8002 are similar to those of S4001 and S4004 in the fortieth embodiment, and therefore the description thereof will be omitted.

[0255] In S8003, the image processing apparatus 100 (CPU 102) generates a trimap by performing processes similar to those in S1003 to S1008 described in the tenth embodiment.

[0256] In S8004, the image processing device 100 judges whether the depth of field has been changed based on the amount of change in the F-number. The F-number used in the judgment in S8004 may be replaced with a variable capable of calculating the focal length and the amount of light incident on the lens unit 106. For example, the image processing device 100 may compare the amount of change in the T-number or H-number, which is an index calculated from the transmittance of the optical system, for each frame. If the F-number has been changed, the processing step proceeds to S8006, and if the F-number has not been changed, the processing step proceeds to S8008.

[0257] In S8006, the image processing device 100 refers to a table that defines the relationship between the F value and the threshold value. This table is assumed to be stored in the image processing device 100 (for example, in the ROM 103).

[0258] In S8007, the image processing apparatus 100 sets new thresholds (foreground threshold and background threshold) in the RAM 104 based on the table referred to in S8006 and the current (changed) F-number.

[0259] In S8008, the image processing apparatus 100 stores the thresholds (foreground threshold and background threshold) in association with the next frame.

[0260] The image processing device repeats the processes from S8001 to S8008 every time a frame is acquired, thereby achieving optimal image separation for each frame.

[0261] Note that a configuration may be adopted in which the process of S8008 is performed only when, for example, the depth of field is changed, rather than for all consecutive frame images that make up a moving image. Also, a method may be adopted in which the processes of S8004 to S8008 are performed for every certain number of frames, rather than for all consecutive frame images that make up a moving image.

[0262] In the eightieth embodiment, when the F-number is changed, optimal image separation is realized for each frame. This example will be described with reference to Figs. 47 and 48.

[0263] Fig. 47 shows frame images captured using the configuration of this embodiment with the focus on a subject 811. Fig. 47(a) shows a frame image captured in an arbitrary state.

[0264] Fig. 47(b) is a frame image captured with a shallower depth of field, i.e., a smaller F-number, compared to Fig. 47(a). The background 812 other than the subject 811 in the frame image of Fig. 47(b) appears blurred due to the increased defocus amount. In Fig. 47(b), the boundary between the subject 811 and the background 812 is likely to have a large difference in defocus amount, so when the images are separated, it becomes easier to separate the subject 811 into the foreground region and the boundary of the background 812 into the background region.

[0265] Fig. 47(c) is a frame image acquired with a deeper depth of field, i.e., a larger F-number, compared to Fig. 47(a). The background 812 other than the subject 811 in the frame image of Fig. 47(c) is visually clearer due to the smaller defocus amount. In Fig. 47(c), the boundary between the subject 811 and the background 812 is prone to have a small difference in defocus amount, so there is a disadvantage in that the part of the background 812 outside the subject 811 is also classified as a foreground area when the images are separated.

[0266] Fig. 48 is a diagram showing a method of separating all pixels in a frame into three regions: foreground region, background region, and unknown region, depending on the defocus amount. Fig. 48(a) shows the divisions when image separation is performed corresponding to the frame image of Fig. 47(a) acquired in an arbitrary state. Region 821 is a range where the defocus amount is small and is classified as a foreground region. Region 822 is a range where the defocus amount is large and is classified as a background region. Region 823 is a range where it cannot be determined as either a foreground region or a background region depending on the defocus amount and is classified as an unknown region.

[0267] Fig. 48(b) shows the range of divisions when performing image separation when the depth of field is made shallower, that is, the F-number is made smaller, compared to Fig. 48(a). In the boundary portion between the subject 811 and the background 812, the difference in the defocus amount is likely to be large in the state of Fig. 47(b). For this reason, as shown in Fig. 48(b), the table of S8006 is set so that the range of the defocus amount for the region 823 is narrower compared to Fig. 48(a).

[0268] Fig. 48(c) shows the range of divisions when performing image separation when the depth of field is deepened, that is, the F-number is increased, compared to Fig. 48(a). In the boundary portion between the subject 811 and the background 812, the difference in the defocus amount is likely to be small in the state of Fig. 47(c). For this reason, as shown in Fig. 48(c), the table of S8006 is set so that the range of the defocus amount for the region 823 is wider compared to Fig. 48(a).

[0269] Furthermore, in the configuration of this embodiment, under conditions where the entire subject 811 in Fig. 47 appears blurred, the table of S8006 may be set so that when the F-number is reduced, the boundary between the subject 811 and the background 812 becomes wider. Similarly, under conditions where the entire subject 811 in Fig. 47 appears blurred, the table of S8006 may be set so that when the F-number is increased, the boundary between the subject 811 and the background 812 becomes narrower.

[0270] As described above, according to the 80th embodiment, it is expected that the boundaries between the foreground region, background region, and unknown region can be appropriately identified even when the F-number is changed due to the lens aperture.

[0271] <90th embodiment> As one embodiment, a trimap can be generated using parallax information and a defocus amount that can be calculated by the CPU 102 based on information obtained from an image surface phase difference sensor.

[0272] First, the acquisition of parallax information will be described with reference to FIG. 49. FIG. 49 shows the optical path from a subject to an image sensor when a point of interest of a certain subject is photographed. FIG. 49(a) is a diagram of a focused state (a state in which the subject is at a focal position). An image is collected by a focus lens and formed on an image pickup surface. At this time, the A image signal and the B image signal in the same pixel output the same information. FIG. 49(b) is a diagram of a pre-focused state. An image is collected by a focus lens, but since the image is formed in front of the image pickup surface, the image enters the image pickup surface after the optical paths cross. At this time, the positional relationship between the A image signal and the B image signal becomes more distant as shown in the figure compared to the focused state. By detecting the degree of this separation, it is possible to determine that the image is in a pre-focused state. FIG. 49(c) is a diagram of a post-focused state. An image is collected by a focus lens, but since the image is formed behind the image pickup surface, the image enters the image pickup surface without the optical paths crossing. At this time, compared to when the image is in focus, the positional relationship between the A and B image signals is farther apart as shown in the figure, which means that the positions of the A and B image signals are reversed compared to when the image was in front of the camera. Detecting this indicates that the image is in back focus.

[0273] As shown in Figure 50, the separation of the detected pixels is the defocus amount, and the greater the separation of the detected pixels, the greater the defocus amount, which means that the image will be out of focus. If we can control the pixel separation to be smaller, we can take a focused image.

[0274] In this embodiment, a trimap is generated by detecting the positional deviation of the detection pixels of the A image signal and the B image signal. Based on the ideas of Figs. 49 and 50, a boundary (threshold) between the region in focus (in-focus region) and the front or rear focus region is set as shown in Fig. 51(a). By providing this boundary, it is possible to simply determine the in-focus region as the foreground region and the front or rear focus region as the background region, thereby binarizing the image. Alternatively, it is also possible to determine the in-focus region and the front or rear focus region as the foreground region and the rear focus region as the background region. Furthermore, it is also possible to set an intermediate region at the boundary between the in-focus region and the front or rear focus region as shown in Fig. 51(b). By determining this intermediate region as an unknown region, it is possible to set three values, that is, the foreground region, the background region, and the unknown region, and a trimap image can be generated.

[0275] The above process will be described with reference to the flowchart in Fig. 52. This is mainly executed by the CPU 102 of the image processing device 100, and in this example, the in-focus area and the front focus area are set as the foreground area, the back focus area is set as the background area, and the boundary area is set as the unknown area.

[0276] First, in S9001, the user photographs a desired subject using the image processing device 100. The image of the subject is received by the imaging unit 107. In S9002, the CPU 102 acquires information on the image plane phase difference from the imaging unit 107 and detects the degree of positional deviation of the information coming in the A image signal or the B image signal. The CPU 102 generates focus information from the information. In S9003, if the CPU 102 determines that the positional deviation of the A image signal and the B image signal of a pixel of interest is small and that the pixel is in a focused area, the process proceeds to S9004, and the pixel is determined to be in a foreground area. On the other hand, in S9005, if the CPU 102 determines that the positional deviation is large and that the pixel is in a pre-focus state, the process proceeds to S9006, and the pixel is determined to be in a foreground area. This is because anything in front of the focused area is often a subject desired by the user, and is therefore determined to be in the foreground area. Furthermore, in S9007, if the CPU 102 determines that the positional deviation between the A and B image signals of a pixel of interest is large and that this is a back-focus state, the process proceeds to S9008, where the pixel is determined to be a background region. Furthermore, if the pixel is not in a focused region, a pre-focus region, or a back-focus region, the CPU 102 proceeds to S9009, where the pixel is determined to be an unknown region. In this example, the focused region and the pre-focus region are considered to be foreground regions, so there is no need to create an unknown region between them.

[0277] In S9010, the CPU 102 temporarily stores the result of this processing in the frame memory 111. In S9011, the CPU 102 determines whether the processing is completed for all pixels of the imaging unit 107, and if completed, advances the processing step to S9012, reads the image from the frame memory 111, generates a trimap image, and outputs it to the display unit 114 or the like.

[0278] As described above, a trimap image can be generated using focus information and the defocus amount that can be detected from the degree of deviation between the A and B image signals.

[0279] <91st embodiment> In the 90th embodiment, a trimap image is generated using the defocus amount, which is focus information. In the 91st embodiment, a method of generating a trimap image with further improved accuracy will be described. FIG. 53 is a diagram in which the focus area is separated in the same way as in FIG. 51. At this time, the boundary portion may be different between the front focus area and the rear focus area. For example, in the case of FIG. 53(a), the boundary (threshold) is set so that the in-focus area becomes wider in the front focus area. On the other hand, in the case of FIG. 53(b), the boundary (threshold) is set so that the in-focus area becomes narrower in the rear focus area. If the thresholds of the boundaries can be set individually in the front focus area and the rear focus area in this way, it becomes possible to fine-tune according to the movement of the subject. For example, when the subject is a human, it becomes possible to generate a trimap image according to the actual situation that the movement of the human face or hand often enters the front focus area.

[0280] Furthermore, as an adjustment function, it is possible to freely change the boundary threshold setting, and to provide different adjustment resolutions for the front focus area and the rear focus area. This is shown in Figure 54. Figure 54(a) shows the adjustment resolution in the front focus area, and Figure 54(b) shows the adjustment resolution in the rear focus area. Here, the resolution in the front focus area is set coarse, and the resolution in the rear focus area is set fine. Figure 55 shows a diagram that shows the relationship between resolution and distance. By setting it in this way, it becomes possible to finely adjust according to the movement of the subject, and it becomes possible to generate a trimap image with high accuracy while adapting to the actual shooting conditions.

[0281] The above processing will be described with reference to the flowchart in Fig. 56. This is mainly processed by the CPU 102 of the image processing device 100, and in this example, it is related to setting the adjustment resolution and setting the area threshold using it. First, in S9101, the image processing device 100 performs lens information acquisition processing. This is a task in which the CPU 102 acquires information on the lens unit 106 attached to the image processing device 100. The lens units 106 have various functions and performances, such as high or low resolution, high or low transmittance, the number of aperture blades, and image stabilizer functions. The CPU 102 performs the task of setting initial values ​​based on this information.

[0282] In S9102, the CPU 102 sets a zero point that is the center of the focus region, which is the midpoint between the front and back focus regions, and the boundary separation is processed starting from this zero point.

[0283] In S9103, the CPU 102 sets the adjustment resolution of the front focus area. In S9104, the CPU 102 sets the adjustment resolution of the rear focus area. These adjustment resolutions are set based on the lens information of the attached lens unit 106 described above, and are set independently for each area.

[0284] In S9105, when the user wishes to change the boundary threshold and starts an operation using the operation unit 113, the CPU 102 displays on the display unit 114 a screen regarding which area to set.

[0285] If the user selects a front focus region in S9106, the process proceeds to S9107, where the user can change the boundary threshold of the front focus region, whereas if the user selects a rear focus region, the process proceeds to S9108, where the user can change the boundary threshold of the rear focus region.

[0286] In S9109, the CPU 102 reflects the set boundary threshold. In S9110, the CPU 102 displays the set boundary threshold on the display unit 114 or the like, and notifies the user that the setting has been completed. In S9111, when the user completes the setting operation, the processing of this flowchart ends.

[0287] As described above, the user can set any boundary threshold value in the front focus area and the rear focus area, and can select the adjustment resolution to generate an optimal trimap image that matches the shooting conditions.

[0288] The above-mentioned adjustment resolution may be stored in ROM 103 in advance as a table or the like, and the CPU 102 may call it up in RAM 104 or the like for use, in addition to the lens type information. Alternatively, the user may set an arbitrary adjustment resolution. It is also possible to flexibly change the adjustment resolution according to the state of the lens, for example, the opening and closing state of the iris and the operation speed of the focus lens. Also, the above description has been given with a focus on the front focus region and the rear focus region, but it is also possible to add an intermediate region (unknown region).

[0289] <Embodiment of A0> When photographing multiple subjects, there are cases where it is desired to recognize the multiple subjects as the foreground region of the trimap. However, in the above embodiment, when the subjects are far apart in the depth direction, some of the subjects may be recognized as background regions. In view of the above problem, this embodiment describes a process for generating a trimap with all subjects as the foreground region even when there are multiple subjects.

[0290] In this embodiment, face detection is performed by the image processing device 100 in Fig. 1. The face detection function will be described. The CPU 102 sends image data for face detection to the object detection unit 115. Under the control of the CPU 102, the object detection unit 115 applies a horizontal band-pass filter to the image data. Also, under the control of the CPU 102, the object detection unit 115 applies a vertical band-pass filter to the processed image data. These horizontal and vertical band-pass filters detect edge components from the image data.

[0291] After that, CPU 102 performs pattern matching on the detected edge components to extract candidates for eyes, noses, mouths, and ears. Then, CPU 102 determines that those that satisfy a preset condition (e.g., distance between two eyes, inclination, etc.) from the extracted eye candidates are eye pairs, and narrows down the eye candidates to only those with eye pairs. CPU 102 then associates the narrowed-down eye candidates with other features (nose, mouth, ears) that form a face, and detects a face by passing the candidates through a preset non-face condition filter. CPU 102 outputs face information according to the face detection result, and ends the process. At this time, CPU 102 stores feature amounts such as the number of faces in RAM 104.

[0292] Next, the trimap generation process in the embodiment A0 will be described with reference to the flowcharts of Figures 57A and 57B. First, in SA001, the CPU 102 acquires the number of face areas detected by the image processing unit 105 from the image processing unit 105. In SA002, the CPU 102 determines whether or not a face area is present based on the number of face areas acquired in SA001. That is, if the number of face areas is 0, there is no face area, and otherwise it is determined that there is a face area. If it is determined that there is a face area, the process proceeds to SA003, and if not, the process proceeds to SA016.

[0293] In SA003, CPU 102 sets internal variable N to 1, and sets internal variable M to 1. In SA004, CPU 102 acquires the coordinates of the Nth face region from image processing unit 105. In SA005, CPU 102 calculates the average defocus amount in the face region specified by the coordinates acquired in SA004. In SA006, CPU 102 determines whether the average defocus amount calculated in SA005 is equal to or less than a threshold. That is, CPU 102 determines whether the average defocus amount in the face region is equal to or less than a threshold and is not a blurred image. If it is determined that the average defocus amount is equal to or less than the threshold, the process proceeds to SA007; if not, the process proceeds to SA013.

[0294] In SA007, the CPU 102 sets a threshold parameter for generating a trimap according to the average defocus amount. The threshold here is a threshold for determining the foreground region, background region, and unknown region. In SA008, the CPU 102 calculates the average relative distance within the face region specified by the coordinates acquired in SA004.

[0295] In SA009, CPU 102 generates a new DepthMap by subtracting the average of the relative distances calculated in SA008 from the relative distances of each pixel in the DepthMap (for example, distance information acquired by the process of S1003 in FIG. 3). In SA010, CPU 102 generates an M-th Trimap based on the new DepthMap generated in SA009.

[0296] On the other hand, if it is determined in SA006 that the average defocus amount is greater than the threshold value, the CPU 102 decrements the value of the internal variable M by 1 in SA013.

[0297] Following the processing of SA010 or SA013, in SA011, CPU102 determines whether or not an unprocessed face region exists. That is, CPU102 determines that an unprocessed face region does not exist when the number of face regions acquired in SA001 matches the internal variable N. If an unprocessed face region exists, the process proceeds to SA012. In SA012, CPU102 increments the value of the internal variable N by 1, increments the value of the internal variable M by 1, and returns the process to SA004.

[0298] On the other hand, if it is determined in SA011 that there is no unprocessed face region, in SA014, the CPU 102 determines whether the internal variable M is 0. If M=0, it means that there is no face region determined in SA006 to have an average defocus amount greater than the threshold. This means that there is no need to generate a new DepthMap. If it is determined in SA014 that the internal variable M is not 0, the process proceeds to SA015.

[0299] In SA015, the CPU 102 synthesizes the M trimaps generated in SA010. The synthesis here is a process of generating one trimap by taking the logical sum of the foreground region and the region determined to be an unknown region.

[0300] On the other hand, if it is determined in SA014 that the internal variable M is 0, or if it is determined in SA002 that no face region exists, the CPU 102 generates a Trimap based on the DepthMap in SA016.

[0301] As described above, according to the A0th embodiment, when there are multiple subjects in an image, it is possible to generate a trimap with each of the subjects as a foreground region.

[0302] <Embodiment of A1> In the embodiment A0, there is a problem that it takes a long time to generate a trimap for each detected object. In view of the above problem, in this embodiment, a process of generating a trimap with all objects as the foreground region without creating multiple trimaps even when there are multiple objects will be described.

[0303] Trimap generation processing in the embodiment A1 will be described with reference to the flowcharts of Figures 58A and 58B. In the flowcharts of Figures 58A and 58B, steps performing the same processing as in Figures 57A and 57B are assigned the same reference numerals as in Figures 57A and 57B, and descriptions thereof will be omitted.

[0304] First, the process from SA001 to SA008 is the same as that in Figures 58A and 58B, so the description thereof will be omitted. However, if SA007 does not exist and the determination in SA006 is "YES," the process proceeds to SA008. After that, the process proceeds to SA101.

[0305] In SA101, CPU 102 stores the average calculated in SA008 as the average of the Mth relative distance in RAM 104. The subsequent processes from SA011 to SA014 are similar to those in Figures 58A and 58B, and therefore will not be described here.

[0306] Next, in SA102, CPU 102 calculates an average D of the averages of the M relative distances stored in RAM 104. In SA103, CPU 102 generates a new DepthMap by subtracting the average D calculated in SA102 from the relative distance of each pixel. In SA104, CPU 102 sets a threshold parameter for the unknown region determination process according to the average of the M relative distances stored in RAM 104 and the average D calculated in SA102. In SA105, CPU 102 generates a Trimap based on the new DepthMap.

[0307] As described above, according to the A1 embodiment, when there are multiple subjects in an image, it is possible to generate a trimap with each of the subjects as a foreground region.

[0308] <Embodiment of A2> In the embodiment A1, there is a problem that when there is an object between the subjects, what should be the background region is recognized as the foreground region. In view of the above problem, in this embodiment, a process of generating a trimap is described in which, even when there are multiple subjects, when there is an object between the subjects, the part that should be the background region is recognized as the background region.

[0309] Trimap generation processing in the A2 embodiment will be described with reference to the flowcharts of Figures 59A and 59B. In the flowcharts of Figures 59A and 59B, steps performing the same processing as in Figures 57A and 57B are assigned the same reference numerals as in Figures 57A and 57B, and descriptions thereof will be omitted.

[0310] First, the order of the flow from SA001 to SA008 is the same as in Fig. 57, and therefore the description thereof will be omitted. Following the processing of SA008, in SA201, CPU 102 stores the threshold parameter of the unknown region determination processing set in SA007 and the average relative distance calculated in SA008 as the M-th threshold and average relative distance in RAM 104. The subsequent processing from SA011 to SA014 is the same as in Figs. 57A and 57B, and therefore the description thereof will be omitted.

[0311] Next, in SA202, the CPU 102 sets the M thresholds and the average of the relative distances stored in the RAM 104 as threshold parameters. In SA203, the CPU 102 generates a Trimap using the DepthMap and the parameters set in SA202. Details of the processing in SA203 will be described later with reference to FIG.

[0312] Next, the details of the process of SA203 will be described with reference to the flowchart of Fig. 60. First, in SA301, CPU 102 sets the value of internal variable I, which determines which threshold parameter is to be set, to 1. In SA302, CPU 102 determines whether or not an unused parameter exists. That is, CPU 102 determines whether or not the value of internal variable I exceeds internal variable M. If it is determined that an unused parameter exists, the process proceeds to SA303.

[0313] Next, in SA303, the CPU 102 sets a parameter for the I-th threshold. In SA304, the CPU 102 determines whether the trimap data being generated is classified as data that is classified as a foreground region. If it is determined that the data is not classified as a foreground region, the process proceeds to SA305.

[0314] In SA305, the CPU 102 determines whether the distance information to the subject is within the range of the foreground threshold determined in SA303. If it is determined to be within the range of the foreground threshold, the process proceeds to SA306. In SA306, the CPU 102 classifies the area for which the distance information is determined to be within the range of the foreground threshold in SA304 as a foreground area, and performs a process of replacing the trimap data of that area with the data of the foreground threshold.

[0315] On the other hand, if it is determined in SA305 that the foreground threshold is outside the range, the process proceeds to SA307. In SA307, the CPU 102 determines whether the trimap data being generated is classified as an unknown area. If it is determined that the data is not classified as an unknown area, the process proceeds to SA308.

[0316] In SA308, CPU 102 determines whether the distance information to the subject is outside the range of the background threshold determined in SA303. If it is determined that it is outside the range of the background threshold, the process proceeds to SA309. In SA309, CPU 102 classifies the area where the distance information is determined to be outside the range of the background threshold in SA308 as a background area, and performs a process of replacing the trimap data of that area with the data of the background threshold.

[0317] On the other hand, if it is determined in SA308 that the distance information is within the range of the background threshold, the process proceeds to SA310. In SA310, the CPU 102 classifies the region for which the distance information is determined to be within the range of the background threshold in SA308 as an unknown region, and performs a process of replacing the trimap data of that region with data of the unknown region.

[0318] On the other hand, if it is determined in SA307 that the data is classified as an unknown region, the process proceeds to step SA311. Also, if it is determined in SA304 that the data of the trimap is classified as a foreground region, the process proceeds to step SA311.

[0319] In SA311, the CPU 102 increments the value of the internal variable I by 1, and returns the processing step to SA302.

[0320] On the other hand, if it is determined in SA302 that there are no unprocessed parameters, the process of this flowchart ends.

[0321] As described above, according to embodiment A2, when there are multiple subjects in an image, if there are objects between the subjects, they are treated as background regions, and a trimap can be generated in which only the subjects are treated as foreground regions.

[0322] <Embodiment of B0> In this embodiment, when shooting multiple subjects located at the same distance, an example will be described in which a trimap is generated in which only a specific subject is displayed by changing distance information outside a selected area. The specific subject represents a subject that the user wants to display as a trimap, and is called a target subject.

[0323] 62 is a flowchart of a process for detecting a subject and displaying only the subject of interest as a trimap by adding an offset value to distance information outside the area of ​​the subject of interest. Each process in this flowchart is realized by the CPU 102 loading a program stored in the ROM 103 into the RAM 104 and executing it.

[0324] In the SB101, the CPU 102 controls the object detection unit 115 to detect an object from an image processed by the image processing unit 105. In this embodiment, the object detection unit 115 performs a process of detecting an object, which outputs coordinate data as a processing result, and is, for example, deep learning using a neural network called SSD (Single Shot Multibox Detector) or YOLO (You Only Look Once). The CPU 102 displays a detection area indicating the area of ​​the detected object on the display unit 114, superimposed on the image processed by the image processing unit 105, based on the coordinate data obtained from the object detection unit 115.

[0325] FIG. 61(a) is a diagram showing an example in which a first detection area B003 and a second detection area B004 are displayed on the display unit 114 for a first subject B001 and a second subject B002 detected in the SB101.

[0326] In SB102, the user selects a detection area. Various selection methods can be used. For example, the user may select a detection area using a cross key of operation unit 113. If display unit 114 is a touch panel, the user may select the detection area by directly touching the displayed detection area. The number of selections is not limited to one. Based on the result of the user's selection, CPU102 displays a selection area indicating the detection area of ​​the target subject on display unit 114 by superimposing it on the image processed by SB101. The displayed selection area is displayed, for example, with a thicker frame than the detection area.

[0327] FIG. 61(b) is a diagram showing an example in which a corresponding selection area B005 is displayed on the display unit 114 when a first object B001 is set as an object of interest in SB102.

[0328] In SB104, CPU 102 determines whether each pixel in the image is a selected region. Specifically, CPU 102 determines the coordinate position of the selected region based on the coordinate data obtained from object detection unit 115, and if the coordinate position of each pixel is within the range of the coordinate positions of the selected region, it determines that the pixel is a selected region. If it is a selected region, processing proceeds to SB103; if not, processing proceeds to SB105.

[0329] In SB105, CPU 102 determines whether each pixel in the image is a background region. The classification of foreground, background, and unknown regions is the same as that described in the tenth embodiment, so the description will be omitted. If it is a background region, the process proceeds to SB103, and if not, the process proceeds to SB106.

[0330] In SB106, CPU102 adds a predetermined offset value to distance information (relative distance) corresponding to pixels outside the selected region. The offset value is a value at which the pixel is determined to be in the background region after the addition. Specifically, for example, if the range of distance information is 0 to 255 and the range of 127 to 255 is determined to be in the background region, then if 255 is given as the offset value, all pixels outside the selected region will be determined to be in the background region. Note that when an offset value is added to distance information, a limit of 255 is set to prevent overflow.

[0331] In the SB103, the CPU 102 generates a trimap by performing the same processes as those from S1003 to S1008 described in the tenth embodiment. The CPU 102 loads the generated trimap in the frame memory 111 and outputs it to the display unit 114, the image terminal 109, or the network terminal 108. The CPU 102 may record the trimap in the recording medium 112.

[0332] FIG. 61(c) is a diagram showing an example of the trimap finally generated in this embodiment.

[0333] As described above, according to this embodiment, when photographing multiple subjects located at the same distance, it is possible to generate a trimap that displays only the subject of interest, with subjects other than the subject of interest not included in the foreground area.

[0334] <Embodiment of B1> Using Figure 62, we have explained an example of generating a trimap in which only the subject of interest is displayed by changing the distance information outside the selected area. However, as another embodiment, an example of changing the color data of the trimap outside the selected area can also be considered.

[0335] In this embodiment, an example will be described in which, when photographing multiple subjects located at the same distance, a trimap in which only the target subject is displayed is generated by changing the color data of the trimap outside the selected area.

[0336] Fig. 63 is a flowchart of a process for detecting a subject and displaying only the subject of interest as a trimap by filling in color data of the trimap outside the area of ​​the subject of interest with a color corresponding to the background area. Each process in this flowchart is realized by CPU 102 expanding a program stored in ROM 103 into RAM 104 and executing it. Note that the processes of SB201 to SB203 in Fig. 63 are similar to SB101 to SB103 in Fig. 62 described in the embodiment of B0, and therefore description thereof will be omitted.

[0337] In SB204, the CPU 102 judges whether each pixel in the trimap is a selected region or not. The judgment process is the same as that in SB104 in FIG. 62 described in the embodiment of B0, so the explanation is omitted. If it is a selected region, the CPU 102 ends the process of this flowchart, and if not, the CPU 102 advances the process step to SB205.

[0338] In SB205, the CPU 102 judges whether each pixel in the trimap is a background region or not. The classification of the foreground region, background region, and unknown region is the same as that described in the tenth embodiment, so the description will be omitted. If it is a background region, the CPU 102 ends the processing of this flowchart, and if not, the CPU 102 advances the processing step to SB206.

[0339] In SB206, CPU 102 paints the color data of each pixel outside the selected region with a predetermined color corresponding to the background region. Specifically, for example, if the color corresponding to the background region is black, CPU 102 paints the color data of the pixels outside the selected region with black.

[0340] The CPU 102 loads the processed trimap in the frame memory 111 and outputs it to the display unit 114, the image terminal 109, or the network terminal 108. The CPU 102 may record the trimap in the recording medium 112. Fig. 61(c) shows an example of the trimap finally generated in this embodiment.

[0341] As described above, according to this embodiment, it is possible to generate a trimap that displays only the target subject without changing the distance information.

[0342] <Embodiment of B2> Using Figure 63, we have explained an example of generating a trimap in which only the subject of interest is displayed by changing the color data of the trimap outside the selected area. However, as another embodiment, an example in which the color data of the trimap within the selected area is changed can also be considered.

[0343] In this embodiment, an example will be described in which, when photographing multiple subjects located at the same distance, a trimap in which only the target subjects are displayed is generated by changing the color data of the trimap within a selected area.

[0344] Fig. 64 is a flowchart of a process for detecting a subject and displaying only the subject of interest as a trimap by filling in color data of the trimap in the area of ​​the subject other than the subject of interest with a color corresponding to the background area. Each process in this flowchart is realized by CPU 102 expanding a program stored in ROM 103 into RAM 104 and executing it. Note that the processes of SB301 to SB303 in Fig. 64 are similar to SB101 to SB103 in Fig. 62 described in the embodiment of B0, and therefore description thereof will be omitted. However, in this embodiment, the selection area represents a detection area other than the subject of interest. Therefore, in SB302, unlike SB102, the user selects a subject other than the subject of interest.

[0345] FIG. 61(d) is an example in which a selection region B006 is displayed on the display unit 114 when a first subject B001 is set as a target subject in SB302.

[0346] In SB304, the CPU 102 judges whether each pixel in the trimap is a selected region. The judgment method is the same as the process in SB104 in FIG. 62 described in the embodiment of B0, so the explanation is omitted. If it is a selected region, the process proceeds to SB305, and if not, the process of this flowchart ends.

[0347] In SB305, the CPU 102 judges whether each pixel in the trimap is a background region or not. The classification of the foreground region, background region, and unknown region is the same process as that described in the tenth embodiment, so the description will be omitted. If it is a background region, the CPU 102 ends the processing of this flowchart, and if not, the CPU 102 advances the processing step to SB306.

[0348] In SB306, the CPU 102 fills the color data of each pixel in the selected region with a predetermined color corresponding to the background region. Note that the content of the processing here is similar to that of SB206 in FIG. 63 described in the B1 embodiment, and therefore a description thereof will be omitted.

[0349] The CPU 102 loads the processed trimap in the frame memory 111 and outputs it to the display unit 114, the image terminal 109, or the network terminal 108. The CPU 102 may record the trimap in the recording medium 112. Fig. 61(c) shows an example of the trimap that is finally generated in this embodiment.

[0350] As described above, according to this embodiment, it is possible to generate a trimap that displays only the target subject without displaying anything outside the selected region.

[0351] <Embodiment of C0> One method of outputting the generated trimap to the outside is to output it via SDI (Serial Digital Interface). A method of superimposing trimap data on SDI is to convert the data into ancillary packets and superimpose them on an auxiliary data area. If an attempt is made to efficiently pack trimap data to generate data, it may become a prohibited code. In view of the above problem, this embodiment will explain a process of mapping data so that it does not become a prohibited code.

[0352] The structure of an HD-SDI data stream when the frame rate is 29.97 fps will be described with reference to FIG. 65. In this embodiment, the image processing device 100 transmits moving image data in accordance with the SDI standard. More specifically, the image processing device 100 distributes each pixel data in accordance with SMPTE ST 292-1. FIG. 65 shows a data stream on which one line of Y data is superimposed, and a data stream on which C data is superimposed. In one frame, there are 1125 lines of this data stream. The Y data and C data are composed of 2200 words, with 10 bits per word. The number of bits per word may be N bits (N≧10). An identifier EAV for recognizing the delimiter position of the image signal is superimposed on the data from the 1920th word, followed by LN (Line Number) and data CRCC (Cycle Redundancy Check Code) for checking transmission errors. Then, a data area in which ancillary data may be superimposed continues for 268 words, and an identifier SAV for recognizing the delimiter position of the image signal like EAV is superimposed. Then, 1920 words of image data are superimposed and transmitted. If the frame rate changes, the number of words per line changes, and so does the number of words in the data area in which ancillary data may be superimposed.

[0353] Next, the stream generation process in the embodiment C0 will be described with reference to the flowcharts of FIG. 66, FIG. 67A, FIG. 67B, FIG. 68A, FIG. 68B, and FIG. 69. In the flowchart of FIG. 66, in SC001, the CPU 102 judges whether it is the line where the valid image data starts. For example, in the case of a progressive image, the 42nd line is the start line of the valid image, and the valid image continues to the 1121st line. In the case of an interlaced image, the valid image data of the first field is from the 21st line to the 560th line, and the valid image data of the second field is from the 584th line to the 1123rd line. If it is judged to be the line where the valid image data starts, the process step proceeds to SC002. On the other hand, if the valid image data has not started, the CPU 102 waits until the valid image data starts.

[0354] At SC002, CPU 102 packs the Trimap data into data of 10 bits per word. Details of the packing process will be described later. At SC003, CPU 102 generates Y ancillary packets to be superimposed on the Y data stream. At SC004, CPU 102 generates C ancillary packets to be superimposed on the C data stream. Details of the process of generating Y ancillary packets and C ancillary packets will be described later. At SC005, CPU 102 superimposes the Y ancillary packets and C ancillary packets on the data stream. Details of the ancillary packet superimposition process will be described later. The process of the flowchart in Figure 66 corresponds to the processing of one frame or one field, and this process is repeated for each frame or field.

[0355] Next, the process of packing Trimap data into data of 10 bits per word will be described with reference to the flowcharts in Figures 67A and 67B. At SC101, the CPU 102 sets an internal variable L to 1. At SC102, the CPU 102 sets an internal variable P to 0. At SC103, the CPU 102 sets an internal variable I to 0. At SC104, the CPU 102 sets an internal variable W to 0.

[0356] In SC105, the CPU 102 determines whether the trimap data of the Pth pixel is white data. That is, the CPU 102 determines whether the trimap data is 0x00. If the trimap data is determined to be white data in SC105, the process proceeds to SC106; if not, the process proceeds to SC109.

[0357] In SC106, the CPU 102 determines whether the value of the internal variable P is an even number. If it is determined that the value is an even number, the process proceeds to SC107. In SC107, the CPU 102 sets the white data to 0x00.

[0358] On the other hand, if it is determined in SC106 that the internal variable P is not an even number, the process proceeds to SC 108. In SC108, the CPU 102 sets the white data to 0x11.

[0359] In SC109, the CPU 102 allocates the data of the Trimap to the I bit and the I+1 bit of the Wth word.

[0360] In SC110, the CPU 102 determines whether the internal variable I is 8. If it is determined that the internal variable I is 8, the process proceeds to SC111. In SC111, the CPU 102 sets the internal variable I to 0. In SC112, the CPU 102 increments the internal variable W by 1.

[0361] On the other hand, if it is determined in SC110 that the internal variable I is not 8, the process proceeds to SC113. In SC113, the CPU 102 increments the internal variable I by two.

[0362] Next, in SC114, CPU 102 determines whether the current pixel (P-th pixel) is the final pixel. That is, since the number of pixels in the valid image is 1920, CPU 102 determines whether the internal variable P is 1919. If it is determined in SC114 that the pixel is not the final pixel, the process proceeds to SC115. In SC115, CPU 102 increments the internal variable P by 1 and returns the process to SC105.

[0363] On the other hand, if it is determined in SC114 that it is the last pixel, the process proceeds to SC116. In SC116, CPU 102 stores one line of word data obtained by packing the Trimap data in RAM 104. In SC117, CPU 102 determines whether the current line (Lth line) is the last line. For example, in the case of a progressive image, the number of effective image lines is 1080, so CPU 102 determines whether the internal variable L is 1080. If it is determined that it is not the last line, the process proceeds to SC118. In SC118, CPU 102 increments internal variable L by 1, and returns the process to SC102.

[0364] On the other hand, if it is determined in SC117 that it is the last line, the process of this flowchart ends.

[0365] FIG. 70 shows a data structure generated by the process of the flowcharts of FIG. 67A and FIG. 67B. The data structure of FIG. 70 is a data structure when trimap data is packed into 10 bits per word. As shown in FIG. 70(a), trimap data for 5 pixels is packed into 1 word. Specifically, trimap data for the 1st pixel is assigned to the 0th and 1st bits, the 2nd pixel to the 2nd and 3rd bits, the 3rd pixel to the 4th and 5th bits, the 4th pixel to the 6th and 7th bits, and the 5th pixel to the 8th and 9th bits. In the flowcharts of FIG. 67A and FIG. 67B, the process of packing 5 pixels into 1 word has been described, but as shown in FIG. 70(b), the process of packing 4 pixels into 1 word may be used. In that case, Even Parity and Not Even Parity are assigned to the 8th and 9th bits. The bit assignment here is an example, and the assignment method may be other bit structures. Furthermore, EvenParity is just one example, and other information may be assigned.

[0366] Next, the generation process of an ancillary packet will be described with reference to the flowcharts of Figures 68A and 68B. An example of the ancillary packet generated here is shown in Figure 71(a).

[0367] In FIG. 71(a), ADF (Ancillary Data Flag) indicates the start of an ancillary data packet. DID (Data ID) is an ID that indicates the type of ancillary. SDID (Secondary Data ID) is also an ID that indicates the type of ancillary, like DID. DC (Data Count) indicates the number of data. LN (Line Number) indicates the number of lines.

[0368] The details of the bit allocation of LN are shown in Figure 71 (b). Bits 0 and 1 of LN0 are reserved data, and bits 0 to 6 of the line number are allocated to bits 2 to 8. The inverted data of the 8th bit is allocated to the 9th bit. Bits 0 and 1, and bits 6 to 8 of LN1 are reserved data. Bits 7 to 11 of the line number are allocated to bits 2 to 5. The inverted data of the 8th bit is allocated to the 9th bit. Next, Status is information that indicates the status of the Trimap data.

[0369] Details of Status are shown in FIG. 71(c). Bits 0 and 1 of Status0 indicate what data represents white data. Bits 2 and 3 indicate what data represents black data. Bits 4 and 5 indicate what data represents gray data. Bit 6 is a flag indicating whether 0x00 data is inverted. Bit 7 indicates polarity (whether 0x00 or 0x11 data is assigned to even pixel data). Bit 8 is Even Parity, and Bit 9 is Not Even Parity. Bits 0 to 2 of Status1 indicate how many pixels of data are packed into one word. Bits 3 to 7 are reserved data. Bit 8 is Even Parity, and Bit 9 is Not Even Parity.

[0370] In Fig. 71(a), from TrimapData0, the Trimap data is superimposed by the number of words packed. CS (Check Sum) is a checksum. This is an example of an ancillary packet, and bit allocation may be other than this example.

[0371] First, in SC201, the CPU 102 sets an internal variable L to 1. In SC202, the CPU 102 sets an internal variable W to 0. In SC203, the CPU 102 overlays an ADF (Ancillary Data Flag). In SC204, the CPU 102 overlays a DID (Data ID). In SC205, the CPU 102 overlays a SDID (Secondary Data ID). In SC206, the CPU 102 overlays a DC (Data Count). In SC207, the CPU 102 overlays a LN (Line Number). In SC208, the CPU 102 overlays a Status.

[0372] In SC209, the CPU 102 determines whether the word into which the Trimap data is packed is the final word. For example, when 5 pixels are packed into one word, the number of words is 384. That is, the CPU 102 determines whether the internal variable W is 384. If it is determined in SC209 that it is not the final word, the process proceeds to SC210. ​​In SC210, the CPU 102 determines whether to generate a Y ancillary. If it is determined that a Y ancillary is to be generated, the process proceeds to SC211. In SC211, the CPU 102 reads the data of the Wth word of the Lth line from the RAM 104 and superimposes it.

[0373] On the other hand, if it is determined in SC210 that the Y ancillary is not to be generated (that is, the C ancillary is to be generated), the process proceeds to SC212. In SC212, the CPU 102 superimposes the data of the (W+1)th word of the Lth line.

[0374] At SB213, the CPU 102 increments the internal variable W by 2 and returns the processing step to SC209.

[0375] On the other hand, if it is determined in SC209 that it is the final word, the process proceeds to SC214. In SC214, the CPU 102 superimposes a CS. In SC215, the CPU 102 stores the generated ancillary packet in the RAM 104.

[0376] In SC216, CPU 102 determines whether the current line (i.e., the Lth line) is the last line. For example, in the case of a progressive image, the number of effective image lines is 1080, so CPU 102 determines whether the internal variable L is 1080. If it is determined that it is not the last line, the process proceeds to SC217. In SC217, CPU 102 increments the internal variable L by 1 and returns the process to SC202.

[0377] On the other hand, if it is determined in SC216 that it is the last line, the process of this flowchart ends.

[0378] Next, the process of superimposing ancillary packets will be described with reference to the flowchart in Fig. 69. In step SC301, the CPU 102 sets an internal variable L to 1. In step SC302, the CPU 102 sets an internal variable P to 0.

[0379] In SC303, CPU 102 judges whether the Pth pixel is the position where the ancillary packet is superimposed. For example, the ancillary can be superimposed from the 1928th pixel in FIG. 65. When trimap data is packed with 5 pixels per word, the ancillary packet will have 203 words, so the superimposition position is from 1928 pixels to 2130 pixels. In other words, CPU 102 judges whether internal variable P is within the range from 1928 to 2130. If it is judged to be the position where the ancillary packet is superimposed, the process proceeds to SC304, and if not, the process proceeds to SC306.

[0380] In SC304, the CPU 102 reads data to be superimposed on the Pth pixel in the Y ancillary packet on the Lth line from the RAM 104 and superimposes it. In SC305, the CPU 102 reads data to be superimposed on the Pth pixel in the C ancillary packet on the Lth line from the RAM 104 and superimposes it.

[0381] Next, in SC306, CPU 102 determines whether the current pixel (P pixel) is the final pixel. That is, since the number of pixels in one line is 2200, CPU 102 determines whether the internal variable P is 2099. If it is determined in SC306 that it is not the final pixel, the process proceeds to SC307. In SC307, CPU 102 increments the internal variable P by 1 and returns the process to SC303.

[0382] On the other hand, if it is determined in SC306 that it is the last pixel, the process proceeds to SC308. In SC308, CPU 102 determines whether the current line (Lth line) is the last line. For example, in the case of a progressive image, the number of effective image lines is 1080, so CPU 102 determines whether internal variable L is 1080. If it is determined that it is not the last line, the process proceeds to SC309. In SC309, CPU 102 increments internal variable L by 1, and returns the process to SC302.

[0383] On the other hand, if it is determined in SC308 that it is the last line, the processing of this flowchart ends.

[0384] As described above, according to the C0 embodiment, trimap data is packed to generate SDI ancillary packets and then superimposed, making it possible to output trimap data from SDI.

[0385] <Embodiment of C1> In the embodiment C0, when multiple trimap data are output, the auxiliary area is insufficient and the data cannot be transmitted. In view of the above problem, the present embodiment describes a process for mapping multiple trimap data so that prohibited codes are not generated.

[0386] The structure of a 3G-SDI data stream when the frame rate is 29.97 fps will be described. In this embodiment, the image processing device 100 transmits video data in accordance with the SDI standard. In detail, the image processing device 100 conforms to SMPTE ST 425-1 and distributes each pixel data by applying the R'G'B'+A 10-bit multiplexing structure of SMPTE ST 372. Since any data may be superimposed on the A channel, in this embodiment, the image processing device 100 superimposes and transmits data of multiple trimaps.

[0387] Next, the process of the embodiment C1 will be described with reference to the flowcharts of Figures 72A and 72B. The flowcharts of Figures 72A and 72B show the process of packing data of a plurality of Trimaps in the A channel.

[0388] At SC701, the CPU 102 sets an internal variable L for counting lines to 1. At SC702, the CPU 102 sets an internal variable P for counting pixels to 0. At SC703, the CPU 102 sets an internal variable N for counting trimaps to 1. At SC704, the CPU 102 obtains the maximum number Nmax of trimaps.

[0389] In SC705, the CPU 102 determines whether the trimap data of the Pth pixel in the Nth frame is white data. If it is determined that the trimap data is white data, the process proceeds to SC706, and if not, the process proceeds to SC709. In SC706, the CPU 102 determines whether the internal variable N is an odd number. If it is determined that it is an odd number, the process proceeds to SC707. In SC707, the CPU 102 sets the white data to 0x00.

[0390] On the other hand, if it is determined in SC706 that the internal variable N is an even number, the process proceeds to SC708. In SC708, the CPU 102 sets the white data to 0x11.

[0391] Next, in SC709, the CPU 102 allocates data to the (N*2) bits and (N*2)+1 bits of the A channel of the Pth pixel. In SC710, the CPU 102 determines whether the internal variable N is equal to Nmax. If it is determined that N is not equal to Nmax, the process proceeds to SC711. In SC711, the CPU 102 increments the value of the internal variable N by 1, and returns the process to SC705.

[0392] On the other hand, if it is determined in SC710 that N is equal to Nmax, the process proceeds to SC712. In SC712, CPU 102 determines whether the current pixel (P pixel) is the final pixel. That is, since the number of pixels in the valid image is 1920, CPU 102 determines whether the internal variable P is 1919. If it is determined in SC712 that it is not the final pixel, the process proceeds to SC713. In SC713, CPU 102 increments the internal variable P by 1 and returns the process to SC703.

[0393] On the other hand, if it is determined in SC712 that it is the last pixel, the process proceeds to SC714. In SC714, CPU 102 stores the A channel. In SC715, CPU 102 determines whether the current line (the Lth line) is the last line. For example, in the case of a progressive image, the number of effective image lines is 1080, so CPU 102 determines whether the internal variable L is 1080. If it is determined that it is not the last line, the process proceeds to SC716. In SC716, CPU 102 increments the internal variable L by 1, and returns the process to SC702.

[0394] On the other hand, if it is determined in SC715 that it is the last line, the processing of this flowchart ends.

[0395] In this embodiment, the CPU 102 may generate an ancillary packet as shown in the embodiment C0. In this embodiment, the CPU 102 superimposes the data obtained by packing the Trimap data on the A channel, so there is no need to include the TrimapData in the ancillary packet. In addition, the CPU 102 only needs to superimpose one ancillary packet at any position in the area where the ancillary can be superimposed.

[0396] In this embodiment, the case where there is one transmission path has been described, but the present invention is not limited to this, and a configuration in which multiple transmission paths are prepared and trimap data is output using a transmission path separate from the image may be adopted. In addition, the transmission technology is not limited to SDI, and may be a transmission technology capable of transmitting images, such as HDMI (registered trademark), DisplayPort (registered trademark), USB, or LAN, and multiple transmission paths may be prepared by combining these.

[0397] When a reduced trimap is generated, the CPU 102 may output the reduced data, or may duplicate the same data multiple times to the SDI format size and output it.

[0398] As described above, according to the C1 embodiment, by packing a plurality of trimap data and superimposing the data on the A channel of SDI, it becomes possible to output a plurality of trimap data from SDI.

[0399] The above-mentioned embodiments merely show various specific examples, and each embodiment can be appropriately combined. For example, the first embodiment to the C1 embodiment can be partially combined. In addition, the image processing device 100 may be configured to have the user select a function from a menu display to execute control.

[0400] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0401] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0402] 100: image processing device, 102: CPU, 103: ROM, 104: RAM, 105: image processing unit, 106: lens unit, 107: imaging unit, 111: frame memory, 112: recording medium, 113: operation unit, 114: display unit, 115: object detection unit

Claims

1. an acquisition means for acquiring a captured image and a plurality of parallax images generated by photographing using an image sensor in which a plurality of photoelectric conversion units are arranged, each of which receives a light beam passing through a different pupil partial region of an imaging optical system; a generation means for generating a background separation image for classifying an area of ​​the captured image into a foreground area, a background area, and an unknown area based on distance distribution information obtained from the plurality of parallax images; an output means for outputting the captured image and the background separation image; a third display control means for displaying a histogram of distances indicated by the distance distribution information on a display means; Equipped with The generating means includes: A region in which the distance in the distance distribution information is within a first range is classified as the foreground region; A region in which the distance in the distance distribution information is outside a second range that is wider than the first range is classified as the background region; A region in which the distance in the distance distribution information is outside the first range and within the second range is classified as the unknown region. The background separation image is generated as follows: The third display control means displays the histogram in a manner in which the first range and the second range can be distinguished.

13. An image processing device comprising:

2. a first display control means for displaying the background separation image on the display means; a recording control means for recording the background separation image on a recording medium; The image processing device according to claim 1 , further comprising:

3. An input means for accepting user input; a first setting means for setting at least one of the first range and the second range based on an input received by the input means; 3. The image processing device according to claim 1, further comprising:

4. a second display control means for displaying the captured image on the display means; The second display control means displays the captured image in a manner in which the foreground region, the background region, and the unknown region can be distinguished based on the background separation image.

4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

5. The second display control means displays a boundary line between the foreground region and the unknown region and a boundary line between the unknown region and the background region on the captured image, based on the background separation image.

5. The image processing device according to claim 4.

6. The second display control means detects the boundary between the foreground region and the unknown region and the boundary between the unknown region and the background region by extracting high-frequency components in the background separation image using a high-pass filter with a predetermined cutoff frequency.

6. The image processing device according to claim 5,

7. a second setting unit for setting a transparency of the foreground region, the background region, and the unknown region in the background separation image; The second display control means displays the background separation image by superimposing it on the captured image at the set transparency.

7. The image processing device according to claim 4, wherein the first and second inputs are input to the image processing apparatus.

8. The first range is a range between a threshold value Th2 and a threshold value Th3, The second range is a range between thresholds Th1 and Th4, where Th1<Th2<Th3<Th4.

8. The image processing device according to claim 1,

9. An acquisition means for acquiring an image and a plurality of parallax images generated by photographing using an image sensor having an array of a plurality of photoelectric conversion units each receiving a light beam passing through a different pupil partial region of an imaging optical system; a generation means for generating a background separation image for classifying an area of ​​the captured image into a foreground area, a background area, and an unknown area based on distance distribution information obtained from the plurality of parallax images; an output means for outputting the captured image and the background separation image; a fourth display control means for displaying, on a display means, an overhead view showing a relationship between a horizontal coordinate of the captured image and a distance based on the distance distribution information; Equipped with The generating means includes: A region in which the distance in the distance distribution information is within a first range is classified as the foreground region; A region in which the distance in the distance distribution information is outside a second range that is wider than the first range is classified as the background region; A region in which the distance in the distance distribution information is outside the first range and within the second range is classified as the unknown region. The background separation image is generated as follows: The fourth display control means displays the overhead view in a manner in which the first range and the second range can be distinguished.

13. An image processing device comprising:

10. The generating means detects edges from the captured image and generates the background separation image based on the detected edges.

10. The image processing device according to claim 1,

11. The generating means detects an object from the captured image, and generates the background separation image based on an area in which the detected object exists.

11. The image processing device according to claim 1,

12. The generating means determines at least one of the first range and the second range such that a distance range corresponding to the unknown area changes depending on an aperture value at the time of shooting the captured image.

12. The image processing device according to claim 1,

13. The generating means generates the background separation image by determining the foreground region, the background region, and the unknown region according to information on a focal position at the time of shooting the captured image.

13. The image processing device according to claim 1,

14. Further comprising an object detection means for detecting a plurality of objects, The generating means generates the background separation image for each of the plurality of objects detected by the object detecting means.

14. The image processing device according to claim 1,

15. The generating means further generates one background separation image by synthesizing a plurality of the background separation images.

15. The image processing device according to claim 14.

16. A selection unit is further provided for selecting at least one of the plurality of objects detected by the object detection unit, The generating means generates at least one of the images for background separation based on the selection of the object by the selecting means.

15. The image processing device according to claim 14.

17. The output means adds the background separation image to a data stream composed of N-bit (N≧10) units and outputs the data stream together with the captured image.

17. The image processing device according to claim 1,

18. The output means adds the background separation image to the data stream so as to invert the data for each pixel.

18. The image processing device according to claim 17,

19. The output means outputs the data stream to a transmission means that performs transmission according to SDI.

19. The image processing device according to claim 17 or 18.

20. The image processing device according to claim 1 , further comprising the image sensor.

21. An image processing method executed by an image processing device, comprising: an acquisition step of acquiring a captured image and a plurality of parallax images generated by photographing using an image sensor in which a plurality of photoelectric conversion units are arranged, each of which receives a light beam passing through a different pupil partial region of an imaging optical system; a generation step of generating a background separation image for classifying regions of the captured image into a foreground region, a background region, and an unknown region based on distance distribution information obtained from the plurality of parallax images; an output step of outputting the captured image and the background separation image; a third display control step of displaying a histogram of distances indicated by the distance distribution information on a display means; Equipped with In the generating step, A region in which the distance in the distance distribution information is within a first range is classified as the foreground region; A region in which the distance in the distance distribution information is outside a second range that is wider than the first range is classified as the background region; A region in which the distance in the distance distribution information is outside the first range and within the second range is classified as the unknown region. The background separation image is generated as follows: In the third display control step, the histogram is displayed in a manner in which the first range and the second range can be distinguished.

13. An image processing method comprising:

22. An image processing method executed by an image processing device, comprising: an acquisition step of acquiring a captured image and a plurality of parallax images generated by photographing using an image sensor in which a plurality of photoelectric conversion units are arranged, each of which receives a light beam passing through a different pupil partial region of an imaging optical system; a generation step of generating a background separation image for classifying regions of the captured image into a foreground region, a background region, and an unknown region based on distance distribution information obtained from the plurality of parallax images; an output step of outputting the captured image and the background separation image; a fourth display control step of displaying, on a display means, an overhead view showing a relationship between a horizontal coordinate of the captured image and a distance based on the distance distribution information; Equipped with In the generating step, A region in which the distance in the distance distribution information is within a first range is classified as the foreground region; A region in which the distance in the distance distribution information is outside a second range that is wider than the first range is classified as the background region; A region in which the distance in the distance distribution information is outside the first range and within the second range is classified as the unknown region. The background separation image is generated as follows: In the fourth display control step, the overhead view is displayed in a manner in which the first range and the second range can be distinguished.

13. An image processing method comprising:

23. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 20.

Images