Image processing device, image processing method, and computer program

Abstract

To provide an image processing device that can easily perform setting regarding layer information based on distance information.SOLUTION: The image processing device has image acquisition means acquiring via a lens unit an image including a subject, distance information acquisition means acquiring distance information indicating a distance to the subject, layer information generation means generating layer information on a layer for each distance on the basis of the distance information, and setting means setting a reference for generating the layer information and switching a display of a setting value that can be set according to lens information of the lens unit.SELECTED DRAWING: Figure 20

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a computer program, and the like. [Background technology]

[0002] Currently, a common image compositing technique used in movies and television is chromakey compositing, which makes certain colored parts of an image transparent and then composites a different image onto them. When shooting images for chromakey compositing, a specific color background such as a green or blue screen must be prepared, and work such as smoothing out wrinkles and adjusting the lighting must be done to prevent color unevenness on the background. Also, if light reflected from the green screen is reflected onto the subject, it may be difficult to separate the background, so work such as removing the reflected light may be required in post-production after shooting.

[0003] On the other hand, a video compositing technique that has been used in recent years is a method of using information on the distance to the subject to separate the subject from the background and compositing the subject with another image. Using this method, there is no need to prepare a background such as a green screen, and the complicated work required for chromakey compositing mentioned above is also unnecessary.

[0004] In addition, there is also a video synthesis technique that uses information about the distance to the subject to insert computer graphics (hereafter referred to as CG) into the space between the subject and the background, or between the subject and the camera.

[0005] As a method for acquiring distance information to a subject, for example, there is a method for measuring the distance to a subject using a device equipped with a distance sensor, separate from the image processing device for image capture. In this case, since the angle of view and resolution differ for each device, a calibration work needs to be performed before shooting in order to perform image synthesis in post-processing.

[0006] Meanwhile, as another method for acquiring distance information to a subject, for example, the technology of Patent Document 1 is known. Patent Document 1 discloses a technology for generating distance information indicating the distance from an image processing device to a subject by using an image sensor having a distance measuring function based on an image plane phase difference method. With this technology, image shooting and distance information generation can be performed simultaneously with a single image sensor, making the above-mentioned calibration work unnecessary.

[0007] Furthermore, as a technique for classifying images according to distance information to a subject, for example, the techniques of Patent Document 2 and Patent Document 3 are known. Patent Document 2 discloses a technique for generating a histogram representing the distribution frequency of distances to a subject corresponding to image data, and classifying images according to distance information according to the frequency of occurrence of the distance information. Patent Document 3 discloses a technique for classifying images according to distance information included in a subject region by using a means for recognizing a subject region in image data. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent Publication No. 2021-48560 [Patent Document 2] JP 2016-143919 A [Patent Document 3] JP 2018-29270 A Summary of the Invention [Problem to be solved by the invention]

[0009] However, with conventional devices, distance information is output as a number indicating the distance to the subject, or displayed as a heat map according to the distance information, making it difficult for the photographer to intuitively understand whether they have captured footage that is easy to insert CG into.

[0010] In addition, Patent Documents 2 and 3 disclose techniques for classifying images by distance information, but they only classify images according to distance information to the subject, and there is an issue that they are unable to classify distance information in which there are few subjects in the image, which makes it easier to insert CG in a post-process.Furthermore, there is an issue that it is difficult to intuitively understand what kind of distance layer MAP can be generated depending on the conditions of the optical system.

[0011] SUMMARY OF THE PRESENTLY PREFERRED EMBODIMENT An object of the present invention is to provide an image processing device that can easily set layer information based on distance information. [Means for solving the problem]

[0012] In the image processing device, an image capture means for capturing an image including a subject through a lens unit; a distance information acquisition means for acquiring distance information indicating a distance to the subject; a layer information generating means for generating layer information relating to layers for each distance based on the distance information; The present invention is characterized by comprising a setting means for setting a standard for generating the layer information and for switching a display of set values ​​that can be set according to the lens information of the lens unit. Effect of the Invention

[0013] According to the present invention, it is possible to provide an image processing device that can easily set layer information based on distance information. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram of an image processing apparatus according to a first embodiment. [Diagram 2] 2(A) is a diagram showing an example of color filters arranged on the light receiving surface of an image sensor, and (B) is a diagram showing an example in which two photoelectric conversion units (photodiodes) are arranged in each pixel in correspondence with the color filter arrangement in FIG. 2(A). [Diagram 3]11 is a flowchart illustrating a process of generating distance information according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing the calculation result of equation (1) when correlation between a pair of image signal strings is high in an ideal state in which no noise is present. [Diagram 5] FIG. 11 is a diagram showing the calculation result when formula (1) is applied to a small block containing noise. [Figure 6] 11 is a flowchart illustrating an example of converting distance information into a distance layer MAP using a histogram according to the first embodiment. [Figure 7] FIG. 13 is a diagram showing an example of a histogram generated in step S602. [Figure 8] 11 is a flowchart for explaining a process of converting distance information into a distance layer MAP using an image recognition means. [Figure 9] 2 is a block diagram showing an example of an internal configuration of the mobile terminal according to the first embodiment; FIG. [Figure 10] 13 is a flowchart showing an example of presenting to the photographer which layer of the distance layer MAP to insert CG into. [Figure 11] 11 is a diagram showing an example of an image displayed on the display unit 114 when the flowchart of FIG. 10 is executed. FIG. [Figure 12] 12 is a diagram showing the positional relationship in the front-rear direction between the subject displayed in FIG. 11 and image processing device 100. FIG. [Figure 13] 13 is a flowchart for explaining a process in the second embodiment. [Figure 14] 14 is a flowchart showing the detailed process of step S1301 in FIG. 13. [Figure 15] FIG. 11 is a table showing examples of acquired lens information according to the second embodiment. [Figure 16] 13A and 13B are diagrams showing examples of tables acquired in step S1403. [Figure 17] 11 is a graph showing the resolution of an image plane in the second embodiment. [Figure 18]18(A) is a diagram showing an example of a selection screen for a layer parameter setting mode in Example 2, and FIG. 18(B) is a diagram showing a menu screen transitioning from the state shown in FIG. 18(A) to a state in which both the number of layers and layer width are selected. [Figure 19] 14 is a detailed flowchart illustrating the layer width setting process in step S1305 of FIG. 13. [Figure 20] 13(A) to 13(F) are diagrams showing display examples in a layer width setting mode. [Figure 21] 14 is a detailed flowchart illustrating the number of layers setting process in step S1306 in FIG. 13. [Figure 22] 13(A) to 13(F) are diagrams showing display examples in a layer number setting mode. [Figure 23] 14 is a detailed flowchart illustrating the layer number / layer width setting process in step S1307 in FIG. 13. [Figure 24] 13(A) to 13(E) are diagrams showing display examples in a layer number / layer width setting mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0016] In the embodiment, an example in which the image processing device is applied to a digital still camera will be described. However, the image processing device also includes electronic devices having an imaging function, such as digital movie cameras, network cameras, smartphones with cameras, tablet computers with cameras, in-vehicle cameras, drone cameras, and cameras mounted on robots. In the embodiment, the images include not only still images but also moving images and the like.

[0017] <Example 1> 1 is a block diagram of an image processing apparatus according to embodiment 1. In the figure, an image processing apparatus 100 is an apparatus capable of inputting, outputting, and recording an image.

[0018] 1, a CPU 102 as a computer, a ROM 103, a RAM 104, an image processing unit 105, a lens unit 106, an imaging unit 107, a network module 108, and an image output unit 109 are connected to an internal bus 101. A recording medium I / F (interface) 110, an object detection unit 115, etc. are also connected to the internal bus 101. The blocks connected to the internal bus 101 are configured to be able to exchange data with each other via the internal bus 101.

[0019] Some of the blocks shown in Fig. 1 are realized by causing a CPU, which is a computer included in the image processing device, to execute a computer program stored in a memory such as a ROM, which is a storage medium. However, some or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP), etc., may be used. Moreover, each block shown in Fig. 1 does not have to be built into the same housing, and may be configured by separate devices connected to each other via signal paths.

[0020] The lens unit 106 is a unit consisting of a group of lenses including a zoom lens and a focus lens, an aperture mechanism, a drive motor, etc. An optical image that passes through the lens unit 106 is formed on the light receiving surface of an imaging section 107. The imaging section 107 functions as an image acquisition means for acquiring an image including a subject, and includes an imaging element such as a CCD image sensor or a CMOS image sensor. The optical image formed on the light receiving surface of the imaging element is converted into an imaging signal, which is further converted into a digital signal and output. The imaging element in this embodiment is a sensor having an image plane phase difference detection function, and details of which will be described later.

[0021] The CPU 102 as a computer controls each unit of the image processing device 100 according to a computer program stored in the ROM 103, using the RAM 104 as a work memory. Also, according to the computer program stored in the ROM 103, the CPU 102 executes the processes shown in the flowcharts of Figures 3, 6, 8, 10, and 13 to 24 described below. The ROM 103 is a non-volatile semiconductor memory, and stores therein the computer program for operating the CPU 102, various adjustment parameters, and the like.

[0022] The RAM 104 is a volatile semiconductor memory, and generally has a lower speed and a lower capacity than the frame memory 111. The frame memory 111 is a semiconductor element that temporarily stores image signals and can read them out when necessary. Since image signals are a huge amount of data, a high-bandwidth and large-capacity memory is required. Here, a DDR4-SDRAM (Dual Data Rate4-Synchronous Dynamic RAM) or the like is used. By using this frame memory 111, it becomes possible to perform processes such as synthesizing images that differ over time and cutting out only the required area.

[0023] Under the control of the CPU 102, the image processing unit 105 performs various types of image processing on data from the imaging unit 107 or image data stored in the frame memory 111 or the recording medium 112. The image processing performed by the image processing unit 105 includes pixel interpolation, encoding, compression, decoding, enlargement / reduction (resizing), noise reduction, color conversion, and the like of the image data.

[0024] Furthermore, the image processing unit 105 performs correction processes such as correcting variations in pixel characteristics of the imaging unit 107, correcting defective pixels, correcting white balance, correcting brightness, and correcting distortion and peripheral light loss caused by lens characteristics. The image processing unit 105 also generates a distance map, the details of which will be described later. The image processing unit 105 may be configured with a dedicated circuit block for performing specific image processing. Depending on the type of image processing, the CPU 102 may perform image processing according to a program without using the image processing unit 105.

[0025] Based on the image processing results in the image processing unit 105, the CPU 102 controls the lens unit 106 and adjusts the aperture that optically adjusts the image enlargement, focal length, and light amount, etc. Camera shake may also be corrected by moving a part of the lens group in a plane perpendicular to the optical axis. Reference numeral 113 denotes an operation unit that receives user operations as an interface with the outside of the device, and is composed of elements such as mechanical buttons and switches, and includes a power switch, a mode change switch, etc.

[0026] Reference numeral 114 denotes a display unit for displaying an image, and for example, an image processed by the image processing unit 105, a setting menu, and the operating status of the image processing device 100 can be confirmed on the display unit. The display unit 114 uses a small, low-power device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) as a display device. Furthermore, the display unit 114 may have a touch panel structure using a resistive or capacitive thin-film element and be used as a part of the operation unit 113.

[0027] The CPU 102 generates character strings for informing the user of the setting status of the image processing device 100 and a menu for setting the image processing device 100, and displays these on the display unit 114, superimposed on the image processed by the image processing unit 105. In addition to character information, shooting assist displays such as a histogram, vector scope, waveform monitor, zebra, peaking, false color, etc. can also be superimposed.

[0028] An image output unit 109 employs an SDI (Serial Digital Interface) or an HDMI (High Definition Multimedia Interface) (registered trademark) as an interface. Alternatively, an interface such as Display Port (registered trademark) may be used. A real-time image can be displayed on an external monitor or the like via the image output unit 109.

[0029] Also provided is a network module 108 capable of transmitting not only images but also control signals. The network module 108 is an interface for inputting and outputting image signals and audio signals. The network module 108 can also communicate with external devices via the Internet or the like, and transmit and receive various types of data such as files and commands. The network module 108 may be wireless or wired.

[0030] The image processing device 100 has a function of not only outputting images to the outside, but also recording images within the main body. The recording medium 112 is a large-capacity storage element, such as a hard disc drive (HDD) or a solid state drive (SSD), capable of recording image data and various setting data, and can be attached to the recording medium I / F 110.

[0031] The object detection unit 115 is a block for detecting objects, and performs object detection using artificial intelligence, such as deep learning using a neural network. When performing object detection using deep learning, the CPU 102 transmits to the object detection unit 115 a program for processing stored in the ROM 103, a network structure such as SSD or YOLO, weight parameters, and the like.

[0032] In addition, SSD is an abbreviation for Single Shot Multibox Detector, and YOLO is an abbreviation for You Only Look Once. The object detection unit 115 performs processing for detecting an object from an image signal based on various parameters obtained from the CPU 102, and loads the processing result in the RAM 104.

[0033] Fig. 2(A) is a diagram showing an example of color filters arranged on the light receiving surface of an image sensor. Fig. 2(A) shows an example of a Bayer array of red (R), blue (B), and green (Gb, Gr), in which multiple pixels are arranged two-dimensionally on the image sensor, and a color filter of R, B, Gb, or Gr is arranged in front of each pixel as shown in Fig. 2(A). Although Fig. 2(A) shows only two rows of a color filter array, this color filter array is repeatedly arranged in two rows in the vertical scanning direction.

[0034] In addition, a microlens is arranged in front of the color filter arranged in front of each pixel of the image sensor, and each pixel has two photoelectric conversion units (photodiode A, photodiode B) arranged side by side in the horizontal scanning direction.

[0035] Fig. 2(B) is a diagram showing an example in which two photoelectric conversion units (photodiodes) are arranged in each pixel in correspondence with the color filter arrangement in Fig. 2(A). In Fig. 2(B), each pixel is composed of a pair of photodiode A and photodiode B, and color filters of the same color are arranged for the two photodiodes in the pair. Note that photodiode A and photodiode B each receive light beams from different exit pupils of the optical system via a microlens.

[0036] In the image sensor of this embodiment, an A image signal can be obtained from a plurality of photodiodes A of pixels arranged in the row direction. Similarly, a B image signal can be obtained from a plurality of photodiodes B of pixels arranged in the row direction. These A and B image signals are processed as signals for phase difference detection.

[0037] That is, for example, CPU 102 or image processing unit 105 performs a correlation calculation between the A and B image signals, detects the phase difference between the A and B image signals, and calculates the subject distance based on the phase difference. Here, CPU 102 and image processing unit 105 function as distance information acquisition means for acquiring distance information indicating the distance to the subject.

[0038] It is also possible to obtain an image signal (image signal A+image signal B) by adding together the signals from the two photodiodes A and B of each pixel, and this added image signal is processed by image processing unit 105 as a color image signal according to the Bayer array shown in FIG. 2(A).

[0039] In the imaging unit 107, it is possible to output a phase difference detection signal (A image signal, B image signal) for each pixel, but it is also possible to average the A image signals of multiple adjacent pixels and output an average value of the B image signals of multiple adjacent pixels. Outputting the average value makes it possible to shorten the time required to read out signals from the imaging unit 107 and reduce the bandwidth of the internal bus 101.

[0040] Using the output signal from the imaging unit 107 having such an image sensor, the CPU 102 and image processing unit 105 perform correlation calculations of the two image signals and calculate information such as the defocus amount, parallax information, and various types of reliability based on the phase difference between the two image signals. The defocus amount on the light receiving surface is calculated based on the deviation (phase difference) between the A and B image signals. The defocus amount has positive and negative values, and whether the focus is front or back can be determined depending on whether the defocus amount is positive or negative.

[0041] The absolute value of the defocus amount indicates the degree of focus, and if the defocus amount is 0, the focus is achieved. That is, the CPU 102 calculates information on whether the focus is front or back based on the positive or negative defocus amount, and calculates focus degree information, which is the degree of focus (amount of focus deviation), based on the absolute value of the defocus amount. The information on whether the focus is front or back is output when the defocus amount exceeds a predetermined value, and outputs information indicating that the focus is achieved when the absolute value of the defocus amount is within the predetermined value.

[0042] The CPU 102 performs focus adjustment by controlling the lens unit 106 in accordance with the defocus amount. The CPU 102 also calculates the distance to the subject from the phase difference information and lens information of the lens unit 106 using the principle of triangulation.

[0043] 2, an example has been described in which pixels, each having two photodiodes as photoelectric conversion means arranged for one microlens, are arranged in an array. However, each pixel may have three or more photodiodes as photoelectric conversion means arranged for one microlens. Also, all pixels do not have to have the above configuration. For example, pixels for distance detection may be periodically and discretely arranged among a plurality of pixels for image detection arranged two-dimensionally.

[0044] In this case, the distance detection pixel may have a structure having two photodiodes as described above, or each distance detection pixel may have a structure having only one of photodiode A and photodiode B. When only one of photodiode A and photodiode B is included, photodiode A and photodiode B are arranged so that images of different pupil regions (exit pupils) of the lens unit are incident on them.

[0045] Alternatively, one of the light beams is blocked. In this way, this embodiment is not limited to the pixel structure described above as long as it is configured to obtain two image signals, such as an A image signal and a B image signal, that are capable of detecting a phase difference. Also, the imaging unit 107 may be a so-called stereo camera made up of two imaging elements having parallax.

[0046] Next, the distance information generation process will be described with reference to Fig. 3 to Fig. 5. Fig. 3 is a flowchart for explaining the distance information generation process according to the embodiment 1. The operation of each step in the flowchart in Fig. 3 is performed by the CPU 102 as a computer executing a computer program stored in the ROM 103 as a storage medium.

[0047] 3, first, in step S300, an image is acquired by reading out two signals, (A image signal+B image signal) for imaging and A image signal for phase difference detection, from the imaging unit 107. Here, step S300 functions as an image acquisition step for acquiring an image including a subject via the lens unit. Next, in step S301, the image processing unit 105 calculates the B image signal for phase difference detection by finding the difference between (A image signal+B image signal) and the A image signal.

[0048] In the above steps S300 and S301, an example has been described in which the B signal is calculated by reading out (A image signal+B image signal) and the A image signal and calculating the difference. However, the A image signal and the B image signal may be read out from the imaging unit 107, respectively. In addition, in a case where two image sensors are provided, such as a stereo camera, the image signals output from the respective image sensors may also be processed as the A image signal and the B image signal.

[0049] In step S302, optical shading correction is performed on each of the A image signal for phase difference detection and the B image signal for phase difference detection. In step S303, filter processing is performed on each of the A image signal for phase difference detection and the B image signal for phase difference detection. For example, low frequencies are cut using a high-pass filter configured with FIR. Note that the signals may also be passed through a band-pass filter or low-pass filter with different filter coefficients.

[0050] Next, in step S304, the A image signal for phase difference detection and the B image signal for phase difference detection that have been subjected to the filter processing in step S303 are divided into minute blocks, and correlation calculations are performed. Note that there is no restriction on the size or shape of the minute blocks, and adjacent blocks may have overlapping areas.

[0051] The correlation calculation for the A and B image signals, which are a pair of images, will be explained below. The signal sequence of the A image signal at a pixel position of interest will be denoted as E(1) to E(m), and the signal sequence of the B image signal at a pixel position of interest will be denoted as F(1) to F(m). While shifting the signal sequence of the B image signal F(1) to F(m) relative to the signal sequence of the A image signal E(1) to E(m), the correlation amount C(k) at shift amount k between the two signal sequences is calculated using the following formula (1).

[0052]

number

[0053] Fig. 4 is a diagram showing the calculation result of equation (1) when the correlation between a pair of image signal sequences is high in an ideal state where there is no noise. As shown in Fig. 4, at a shift amount (k = kj = 0) at which the correlation between a pair of image signal sequences is high, the correlation amount C(k), which is the difference, is minimum. Hereinafter, k at which the discrete correlation amount C(k) is minimum is represented as kj, and x that gives the minimum value C(x) for the continuous correlation amount is calculated by the three-point interpolation process shown in equations (2) to (4). Note that the pixel shift amount x is a real value and its unit is pixel.

[0054]

number

number

number

[0055] The SLOP in equation (4) represents the minimum and local minimum correlation amount and the slope of the change in the adjacent correlation amount. C(kj) = C(0) = 1000 C(kj-1)=C(-1)=1700 C(kj+1)=C(1)=1830 Let us assume that.

[0056] In this example, kj = 0. From equations (2) to (4), SLOP=830 x=-0.078pixel In addition, in the case of a focused state, the pixel shift amount x for the signal sequence of the image A and the signal sequence of the image B is ideally 0.00.

[0057] On the other hand, Fig. 5 is a diagram showing the calculation results when formula (1) is applied to a small block containing noise. As shown in Fig. 5, the correlation between the signal sequence of image A and the signal sequence of image B decreases due to the influence of randomly distributed noise. The minimum value of the correlation amount C(k) is larger than the minimum value shown in Fig. 4, and the correlation amount curve has a generally flat shape (a shape in which the absolute difference between the maximum and minimum values ​​is small).

[0058] In FIG. 5, as a specific example, C(kj) = C(0) = 1300 C(kj-1)=C(-1)=1480 C(kj+1)=C(1)=1800 Let us assume that.

[0059] In this example, kj = 0. From equations (2) to (4), SLOP=500 x=-0.32pixel That is, compared to the calculation result in the absence of noise shown in FIG.

[0060] When the correlation between a pair of image signal series is low, the change in the correlation amount C(k) is small and the correlation amount curve is generally flat, so the SLOP value is small. Similarly, when the subject image has low contrast, the correlation between a pair of image signal series is low and the correlation amount curve is generally flat.

[0061] Based on this property, the reliability of the calculated pixel shift amount x can be judged by the SLOP value. That is, when the SLOP value is large, it can be judged that the correlation between the pair of image signal series is high, and when the SLOP value is small, it can be judged that no significant correlation has been obtained between the pair of image signal series.

[0062] In this embodiment, since the formula (1) is used for the correlation calculation, the correlation amount C(k) is minimum and minimal at the amount of deviation where the correlation between the pair of image signal sequences is highest. However, a correlation calculation method may be used where the correlation amount C(k) is maximum and maximal at the amount of deviation where the correlation between the pair of image signal sequences is highest.

[0063] Next, in step S305, the reliability is calculated. As described above, the reliability can be calculated based on C(kj), which indicates the degree of match between the two images calculated in step S304, and the value of SLOP. Next, in step S306, an interpolation process is performed. Even if the correlation calculation was performed in step S304, there are cases in which the reliability calculated in step S305 is low and cannot be used as the pixel shift amount.

[0064] In that case, the pixel shift amount is interpolated from the pixel shift amount calculated for the surrounding pixels. The interpolation method may be a median filter, or a calculation may be performed to reduce the pixel shift amount data and then enlarge it again. Also, color data may be extracted from the (A image signal + B image signal) used for imaging, and the pixel shift amount may be interpolated using the color data.

[0065] Next, in step S307, the defocus amount is calculated by referring to the amount x calculated in step S304. Specifically, the defocus amount (denoted as DEF) can be obtained by the following formula (5).

number

[0066] Next, in step S308, the distance is calculated from the defocus amount calculated in step S307. When the distance to the subject is Da, the focal position is Db, and the focal length is F, the following formula (6) approximately holds. Note that step S308 functions as a distance information acquisition step for acquiring distance information indicating the distance to the subject.

[0067]

number

number

[0068] Therefore, if Db when DEF=0 is set to Db0, equation (7) becomes equation (8) below, and the absolute distance to the subject can be obtained.

number

[0069] On the other hand, the relative distance is Da-Da', and can be calculated from equations (7) and (8) using the following equation (9).

number

[0070] As described above, by performing correlation calculations according to the flowchart of Fig. 3, it is possible to calculate the pixel shift amount, defocus amount, and distance information from the A image signal for phase difference detection and the B image signal for phase difference detection. That is, it is possible to acquire distance information based on the phase difference between the outputs of a plurality of photoelectric conversion units. Note that the distance information in this embodiment may be the distance data itself, or may be the shift amount or defocus amount, and the distance information includes these.

[0071] Next, a process for generating a distance layer MAP in the first embodiment will be described with reference to Fig. 6 to Fig. 8. Fig. 6 is a flowchart showing an example of converting distance information into a distance layer MAP using a histogram according to the first embodiment. Note that the operation of each step in the flowchart in Fig. 6 is performed by CPU 102 as a computer executing a computer program stored in ROM 103 or the like as a storage medium.

[0072] 6, CPU 102 initializes the values ​​of internal processing variables N, X, and T to 1. Here, processing variable N is a variable that counts the number of processing operations, processing variable X is a temporary variable for calculation, and processing variable T is a variable that indicates the layer number.

[0073] Next, in step S601, the CPU 102 acquires lens information and focal position and lens aperture information as distance information from the lens unit 106 and the imaging unit 107, and calculates the measurable range L and the minimum resolution width M of the subject distance.

[0074] Furthermore, in step S602, the CPU 102 generates a histogram showing the distribution of distances at which subjects exist in the depth direction based on the distance information. Here, Fig. 7 is a diagram showing an example of a histogram generated in step S602. The horizontal axis represents the distance in the depth direction, and the vertical axis represents the number of occurrences of distance information. The histogram is generated by accumulating distance information that appears in units of minimum resolution width M within the range L where distance measurement is possible. The CPU 102 tags the data of the generated histogram with numbers starting from 1 in units of minimum resolution width M in order of distance closest to the image processing device 100, and stores the tagged data in RAM 104.

[0075] Next, in step S603, the operation unit 113 accepts a setting input of the frequency threshold S from the user. The user sends information on the frequency threshold S to the CPU 102 via the operation unit 113. Note that instead of accepting a setting by the user via the operation unit 113, the frequency threshold S may be set wirelessly from an external device via the network module 108.

[0076] Next, in step S604, CPU 102 reads out histogram data for the Nth processing variable and histogram data for the N+1th processing variable from RAM 104. Then, it is determined whether the change in the number of occurrences of distance information for the Nth processing variable and the N+1th processing variable crosses the frequency threshold value S. If it does cross, the process proceeds to step S605, and if it does not cross, the process proceeds to step S607.

[0077] In step S605, CPU 102 classifies the distance information from minimum resolution width M*processing variable X to minimum resolution width M*processing variable N as the Tth layer of distance layer MAP. Next, in step S606, CPU 102 assigns the value of processing variable N+1 to processing variable X, and increments the value of processing variable T by 1.

[0078] Next, in step S607, the CPU 102 determines whether all the histogram data stored in the RAM 104 has been read. If all the data has been read, the process proceeds to the end step, and if not, the process proceeds to step S608. Next, in step S608, the CPU 102 increments the value of the processing variable N by 1 and returns to step S604.

[0079] In the above-described embodiment 1, a histogram is generated based on distance information by the flowchart in Fig. 6, and a distance layer MAP can be classified (generated) into layers with many objects in the depth direction and layers with few objects. Also, each layer can be numbered in order of proximity to the image processing device 100.

[0080] 6, the number of layers (number of layers) and layer width (width of the layer in the distance direction) that are the criteria for generating layer information according to the image are changed. Here, CPU 102 executes steps S600 to S608 to function as a layer information generating step (layer information generating means) that generates a histogram based on distance information and generates layer information for each distance based on the histogram.

[0081] In addition, in Figures 6 and 7, a method of classifying (generating) the distance layer MAP using a histogram has been explained, but it is also possible to recognize the subject using an image recognition means and classify (generate) the distance layer MAP according to the recognized subject.

[0082] Fig. 8 is a flowchart for explaining a process for converting distance information into a distance layer MAP using an image recognition means. Note that the operation of each step in the flowchart in Fig. 8 is performed by a CPU 102 as a computer executing a computer program stored in a storage medium such as a ROM 103.

[0083] In step S800, the CPU 102 initializes the values ​​of processing variables P and Q to 1. Here, the processing variable P is a variable for counting the number of recognized subjects, and the processing variable Q is a temporary variable for executing processing.

[0084] Next, in step S801, object detection unit 115 detects a subject in the image data. Object detection unit 115 stores in RAM 104 coordinate data indicating which area in the image data the subject is in. The coordinate data here is data representing the contour of the subject.

[0085] Next, in step S802, the CPU 102 determines whether or not all of the objects in the image data have been detected. If all of the objects have been detected, the process proceeds to step S803. If all of the objects have not been detected, the process proceeds to step S804.

[0086] In step S803, CPU 102 sorts the coordinate data of the detected subjects stored in RAM 104 in order of proximity to image processing device 100 based on distance information about the subject region, numbering the data in order from 1, and then proceeds to step S805. On the other hand, in step S804, CPU 102 increments the value of processing variable P by 1, and then returns to step S801.

[0087] In step S805, CPU 102 determines whether the same distance information is included within the subject area indicated by the coordinate data of the subject in the Qth processing variable stored in RAM 104 and the subject area indicated by the coordinate data of the subject in the Q+1th processing variable. If included, the process proceeds to step S806; if not included, the process proceeds to step S807.

[0088] In step S806, the object area indicated by the coordinate data of the object of the Qth processing variable stored in the RAM 104 and the object area indicated by the coordinate data of the object of the Q+1th processing variable stored in the RAM 104 are merged as the object area indicated by the coordinate data of the object of the Q+1th processing variable, and stored in the RAM 104.

[0089] Next, in step S807, CPU 102 increments processing variable Q by 1. Furthermore, in step S808, CPU 102 determines whether the value of processing variable Q is equal to or greater than the value of processing variable P. If the value of processing variable Q is equal to or greater than the value of processing variable P, the process proceeds to step S809; otherwise, the process returns to step S805.

[0090] In step S809, the layer number setting of the distance layer MAP is allocated alternately to the distance information for each object region indicated by the coordinate data of the object stored in the RAM 104 and the distance information not included in any object region, starting from the distance closest to the image processing device 100. In this way, layers where objects exist and layers where objects do not exist are generated.

[0091] As described above, according to the flowchart in Fig. 8, the image recognition means can be used to classify distance information into a distance layer MAP consisting of layers with many subjects in the depth direction and layers with few subjects. Also, each layer can be numbered in order of proximity to the image processing device 100. In this way, in the flowchart in Fig. 8, the number of layers (number of layers) and layer width (width of the layer in the distance direction) that are the basis for generating layer information according to the image are changed.

[0092] The object detected in step S801 may be selectable from a variety of types, such as a human body, a face, a car, etc. That is, the width of the layer in the distance direction for each distance may be changed according to the type of object recognized. Also, the number of objects may be selectable as one or more. Also, the processes of steps S800 to S804 may be performed for each frame, and the accuracy of image recognition may be improved using the results for multiple frames before performing steps S805 and after. That is, layer information may be generated based on images of multiple frames.

[0093] Here, CPU 102 executes steps S800 to S809 as a layer information generating step, thereby functioning as layer information generating means that recognizes a subject by image recognition means and generates layer information for each distance according to the recognized subject.

[0094] In addition, it is also possible to combine a method of recognizing a subject using an image recognition means and classifying (generating) a distance layer MAP according to the recognized subject with a method of generating a histogram based on distance information and classifying (generating) a distance layer MAP based on the histogram.

[0095] Next, a method of presenting to a user (photographer, etc.) which layer of the distance layer MAP classified in the above-mentioned embodiment 1 into which CG should be inserted will be described with reference to Fig. 9 to Fig. 12. Here, a case will be described in which layer information and coordinate information into which CG should be inserted are sent from a mobile terminal to the image processing device 100.

[0096] Fig. 9 is a block diagram showing an example of the internal configuration of a mobile terminal according to the embodiment 1. In the figure, a mobile terminal 900 allows a user to set layer information and coordinate information, and can transmit the set values ​​wirelessly.

[0097] 9, a network module 908, an operation unit 913, a display unit 914, a CPU 902, a ROM 903, and a RAM 904 are connected to an internal bus 901. The units connected to the internal bus 901 are configured to be able to exchange data with each other via the internal bus 901.

[0098] The CPU 902 uses the RAM 904 as a working memory in accordance with a program stored in the ROM 903 to control each unit of the mobile terminal 900. The ROM 903 is a non-volatile storage element, and stores programs for operating the CPU 902, various adjustment parameters, and the like.

[0099] The display unit 914 is a display for displaying various setting states, data (including digital image data and analog image signals) received from the network module 908, and the like, based on the control of the CPU 902. The operation unit 913 is an operation unit that receives operations from the user, such as a power switch for supplying power to the mobile terminal 900, and setting layer information and coordinate information.

[0100] If the operation unit 913 includes a touch panel, the CPU 902 can detect when the touch panel is touched with a finger or pen (hereinafter referred to as touch down), or when the touch panel is in a state where the finger or pen is touching the touch panel (hereinafter referred to as touch on).The CPU 902 can also detect when the finger or pen is moving while still touching the touch panel (hereinafter referred to as move), when the finger or pen that was touching the touch panel is released (hereinafter referred to as touch up), or when nothing is touching the touch panel (hereinafter referred to as touch off).

[0101] These operations and the position coordinates of the touch panel where the finger or pen is touching are notified to the CPU 902, which determines what operation was performed on the touch panel based on the notified information. Regarding moves, the direction of movement of the finger or pen moving on the touch panel can also be determined for each vertical and horizontal component on the touch panel based on changes in the position coordinates.

[0102] Also, a stroke is considered to be drawn when a user touches down on the touch panel, moves a certain distance, and then touches up. The operation of quickly drawing a stroke is called a flick. A flick is an operation in which a user touches the touch panel with a finger, moves the finger quickly a certain distance, and then releases the finger. In other words, it is an operation in which a user quickly traces the touch panel with a finger as if flicking it.

[0103] When a move is detected over a predetermined distance or more at a predetermined speed or more, and a touch-up is detected as it is, it can be determined that a flick has been performed. When a move is detected over a predetermined distance or more at less than a predetermined speed, it can be determined that a drag has been performed. The touch panel may be of any of various types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, or an optical sensor type.

[0104] The network module 908 transmits and receives data to and from external devices such as an external camera or a personal computer via wireless communication under the control of the CPU 902. The network module 908 receives setting information and operation information of the image processing device 100 and transmits commands for operating the image processing device 100 and additional information to be recorded together with image data. Data that can be transmitted and received includes digital image data and analog image signals.

[0105] Fig. 10 is a flowchart showing an example for presenting to the photographer which layer of the distance layer MAP to insert CG into. Note that the operation of each step in the flowchart in Fig. 10 is performed by the CPU 102 of the image processing device 100 and the CPU 902 of the mobile terminal 900 executing computer programs stored in the storage media such as the ROM 103 and the ROM 903, respectively.

[0106] In step S1000, the CPU 902 of the mobile terminal 900 accepts settings of layer information and coordinate information from the user via the operation unit 913. The layer information here is a value that specifies which layer of the distance layer MAP the CG is to be inserted into, or an image may be transmitted from the image processing device 100 to the mobile terminal 900 to allow the user to select a subject, and the layers before and after the selected subject may be selected. The coordinate information is coordinate information that specifies the position on the screen where the CG is to be inserted.

[0107] Next, in step S1001, the CPU 902 of the mobile terminal 900 transmits the layer information and coordinate information set in step S1000 to the network module 108 within the image processing device 100 via the network module 908.

[0108] The layer information and coordinate information may be transmitted to the image processing device 100 at the same time that the mobile terminal 900 transmits to the image processing device 100 commands for operating the image processing device 100, additional information to be added to image data and recorded, etc. Here, steps S1000 and S1001 function as steps for setting, for the image, a composite image insertion area for inserting the composite image, and insertion layer information which is the layer into which the composite image is inserted.

[0109] Next, in step S1002, the CPU 102 in the image processing device 100 receives the layer information and coordinate information via the network module 108. Next, in step S1003, the CPU 102 calculates a CG insertion layer from the layer information, and a CG insertion area in the image data from the coordinate information.

[0110] Next, in step S1004, the CPU 102 combines a CG insertion color with the pixel corresponding to the position of the CG insertion area of ​​the image data. Here, the CG insertion color is a color that represents the position where the CG is to be inserted in post-processing. The CG insertion color may be a color that is separately set by the user.

[0111] Next, in step S1005, CPU 102 determines whether the layer (layer information) of the distance layer MAP of the target pixel (object pixel) that corresponds to the position of the CG insertion area of ​​the image data is the same as the CG insertion layer (layer information). If they are the same, proceed to step S1006, and if they are different, proceed to step S1007.

[0112] Next, in step S1006, CPU 102 controls image processing unit 105 to change the pixel data corresponding to the position of the CG insertion area in the image data of the subject to a predetermined warning color (warning color). Here, the warning color is a color that indicates that the subject is present in the same position as the CG insertion area where the CG is scheduled to be inserted in post-processing. Note that the warning color may be a color separately set by the user.

[0113] In addition to the warning color, the overlapping area may be displayed in a predetermined pattern (such as a dot pattern or stripe pattern) different from other areas. In this manner, in this embodiment, when the layer information of the subject image and the insertion layer information into which the composite image is inserted are the same, the overlapping area is displayed in a predetermined color (warning color) or pattern different from other areas.

[0114] On the other hand, in step S1007, it is determined whether the layer of the distance layer MAP of the pixel corresponding to the position of the CG insertion area of ​​the image data of the subject is behind the CG insertion layer. If it is behind, proceed to step S1008, and if it is before, proceed to step S1009.

[0115] In step S1008, CPU 102 controls image processing unit 105 to combine pixel data of the subject corresponding to the position of the CG insertion area with the background at a transmittance according to the pixel distance layer MAP and the distance of the CG insertion layer.

[0116] For example, if the distance layer MAP of the subject's pixels is classified as immediately behind the CG insertion layer, the transparency of the image in front is lowered so that the image of the subject is displayed more faintly. On the other hand, if it is classified as a back layer far away from the CG insertion layer, the transparency of the image in front is increased so that the subject in the background is displayed darker. This makes it easier for the photographer to grasp the sense of distance between the inserted CG and the subject.

[0117] Regardless of which side is behind, the transmittance of the overlapping area of ​​the front image can be changed in the same way. That is, the transmittance of the front image in the overlapping area can be changed as the distance between the layer information of the subject image and the insertion layer information into which the composite image is inserted increases. In this embodiment, the transmittance of the front image in the overlapping area is increased as the distance between the layer information of the subject image and the insertion layer information into which the composite image is inserted increases. However, conversely, the transmittance can also be decreased.

[0118] On the other hand, in step S1009, the CPU 102 determines whether all the pixels of the object corresponding to the position of the CG insertion area have been processed. If all have been processed, the process proceeds to step S1010, and if not, the process proceeds to S1005.

[0119] Next, in step S1010, CPU 102 displays image data combined with the warning color generated in the processing of steps S1005 to S1009 and the color of the subject that serves as the background on display unit 114. By executing steps S1004 to S1010 as display control steps, CPU 102 functions as a display control means for displaying an overlapping area where the composite image and the subject image overlap in a predetermined color or pattern according to the layer information of the subject.

[0120] In this embodiment, the image is displayed on the display unit 114, but it may be output from the image output unit 109 and recorded on the recording medium 112 via the recording medium I / F 110, or transmitted to an external device via the network module 108. In addition, all pixels of the subject are processed in the processing of steps S1005 to S1009 before being displayed on the display unit 114, but it may also be processed and displayed one pixel at a time in the raster direction.

[0121] According to the above-described flowchart of Fig. 10, it is possible to perform a display such as that shown in Fig. 11. Fig. 11 is a diagram showing an example of an image displayed on the display unit 114 when the flowchart of Fig. 10 is executed. Fig. 12 is a diagram showing the positional relationship in the front-rear direction between the subject displayed in Fig. 11 and the image processing device 100.

[0122] In the example of the display image shown in Fig. 11, subjects 1100 to 1102 and a CG insertion area 1103 calculated from coordinate information are displayed. As shown in Fig. 12, the subjects are arranged in the order of subject 1100, subject 1101, and subject 1102 in increasing distance from image processing device 100. Therefore, the positions of the respective subjects are classified as follows: subject 1100 is in the first layer of the distance layer MAP, subject 1101 is in the second layer of the distance layer MAP, and subject 1102 is in the third layer of the distance layer MAP.

[0123] 11, the display is for when the CG insertion layer is specified as the first layer of the distance layer MAP, so the pixels in part 1104 where subject 1100 placed on the first layer of the distance layer MAP overlaps with CG insertion area 1103 are displayed in a warning color. In the pixels in part 1105 where subject 1101 placed on the second layer of the distance layer MAP overlaps with CG insertion area 1103, the background subject is displayed faintly because subject 1101 and the CG insertion layer are close to each other.

[0124] On the other hand, in the pixel area 1106 where the subject 1102 arranged on the third layer of the distance layer MAP overlaps with the CG insertion area 1103, the subject in the background is displayed darker because the subject 1102 and the CG insertion layer are far away. The rest of the CG insertion area 1103 is displayed in the CG insertion color.

[0125] As described above with reference to Figures 9 to 12, this embodiment makes it possible to clearly show the user of the mobile terminal which layer of the classified distance layer MAP CG will be inserted into. In addition, since a distance layer MAP that is easy to insert CG into is generated, the distance layer MAP can be displayed in a way that allows the photographer to easily understand it.

[0126] A program for realizing one or more functions of the present embodiment may be supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device may read and execute the program. Also, one or more functions may be realized by a circuit (e.g., ASIC) that realizes the functions.

[0127] <Example 2> Next, the process and procedure for generating and displaying a distance layer MAP in the second embodiment will be described with reference to Figures 13 to 24. For simplicity of explanation, differences from the first embodiment will be mainly described, and explanations of common parts will be omitted.

[0128] In the second embodiment, the lens unit 106 is described as having an aperture value of F5.6, a focal length of 50 mm, and an ISO sensitivity value at the time of capturing an image of 400. It is also assumed that the subject is located at a distance of 3 m and is approximately in focus when the user starts an operation to set the distance layer MAP.

[0129] Here, it is assumed that the user has entered a setting mode for the number of layers and the layer width, similarly to the first embodiment. Fig. 13 is a flowchart for explaining the processing in the second embodiment, and Fig. 14 is a flowchart showing the detailed flow of step S1301 in Fig. 13. The operation of each step in the flowcharts in Fig. 13 and Fig. 14 is performed by the CPU 102 as a computer executing a computer program stored in the ROM 103 as a storage medium.

[0130] 13, when a mode for setting the number of layers and layer width is entered and processing starts, in step S1301, a sub-processing is entered for calculating the minimum resolution width M and measurable range L in the distance layer MAP. Here, this sub-processing will be described with reference to the flowchart in FIG.

[0131] 14, various types of lens information related to the current lens state of the attached lens unit 106 are acquired as shown in FIG 15. FIG 15 is a diagram showing a table summarizing examples of acquired lens information according to the second embodiment.

[0132] 15, the first piece of information about the lens state (lens information 1) is the lens aperture value (hereafter, F-number), the second piece (lens information 2) is the lens focal length in mm, and the third piece (lens information 3) is position information (focus position) on the image plane of the focus lens position, and the difference from the reference position is expressed in mm.

[0133] Here, the position of the focus lens relative to an object at infinity is set as the reference position (0 mm), and the distance from the reference position to the current focus lens position is shown. In this embodiment, for example, when the focus lens is located 1 mm away from the reference position, it is assumed that the lens is positioned to focus on an object with a distance of 3 m. However, this also depends on the focal length of the lens and changes depending on the zoom position.

[0134] The fourth piece of information about the lens state (lens information 3) is the distance of the subject focused at the focal position of the focus lens, and the subject distance is acquired in meters. This can be calculated based on the optical design information of the lens, and is stored as a design value in the image processing device 100. As described above, in this embodiment, the value is, for example, 3 m.

[0135] The fifth (lens information 5) is information on the relative amount of image plane movement from the current focus lens position to the infinity end or close end of the focus lens. For example, when the subject distance is 3 m, the signed data is in mm units, such as -1 mm to the infinity end and +9 mm to the close end. Hereafter, this information will be referred to as infinity defocus information and close defocus information, respectively. In this lens state, the movement range of the image plane focus from the infinity end of the lens to the close end is 10 mm.

[0136] After acquiring these pieces of lens information, in step S1402, measurable limit information and noise information are acquired. The measurable limit information is an index of the maximum image shift amount that can be detected as the detection limit performance of the image plane phase difference technology. In the image plane phase difference technology, the shift amount during correlation calculation may be changed depending on the system conditions, and if the shift amount of the correlation calculation process is limited to 20 shifts, image shifts of 10 shifts or more cannot be detected, so such an index is generated and stored.

[0137] Moreover, the noise information is information on parameters that have a large effect on the noise level, and here is, for example, the ISO setting value that is the sensitivity setting of the sensor.In step S1403, various table information such as that shown in FIG.

[0138] 16A and 16B are diagrams showing examples of tables acquired in step S1403. In Fig. 16A, the left column shows F-values, and the right column shows the resolution on the image plane of the focus lens corresponding to the F-values ​​in the left column in mm. In Fig. 16B, the left column shows ISO values, and the right column shows the degree of influence of the ISO values ​​in the left column on the resolution on the image plane of the focus lens in mm.

[0139] Returning to Fig. 14, in the next step S1404, the minimum resolution range M of the subject distance in the distance layer MAP is calculated using the lens information in Fig. 15 and the table information in Fig. 16. Here, the explanation will be given assuming an F value of 5.6 and an ISO value of 400.

[0140] First, by looking up the value for F5.6 in the table in Fig. 16(A), the image plane resolution is 0.025 mm, and by looking up the value for ISO400 in the table in Fig. 16(B), the noise component is 0.002 mm. Using these, the minimum resolution that can be calculated from the focal point of the focus lens on the image plane can be calculated as 0.025 + 0.002 = 0.027 mm.

[0141] Furthermore, by converting this image plane information into subject distance, the minimum resolution width M converted into subject distance is calculated. 0.027 mm on the image plane is roughly equivalent to about 30 cm when converted into distance from a subject position of 3 m and a lens state with a focal length of 50 mm. In other words, under these lens conditions, a focus position shift of 0.027 mm on the image plane is set as the minimum resolution, and the minimum resolution width M converted into subject distance is positioned as 30 cm that can be detected.

[0142] The minimum resolution width M of the subject distance calculated here is an approximate value, and may be in units of, for example, 10 cm. Furthermore, the graph in FIG. 17 will be used to supplement the explanation of the calculation of this minimum resolution width M. FIG. 17 is a graph showing the image plane resolution in the second embodiment, with the horizontal axis representing the lens aperture value (the right side is the lower value, the open aperture side) and the vertical axis representing the image plane resolution. 281 is a curve plotting the data in the table in FIG. 16. For example, the resolution at intersection A with the auxiliary line for F5.6 is 0.025 as shown in the table.

[0143] Also, when noise correction is performed based on the noise information corresponding to the ISO setting in Figure 16(B), the result is point B, and the curve connecting this in the F-number direction is 282, which is the curve of the image plane resolution after noise correction at ISO 400. In other words, the resolution at point B is 0.027 mm.

[0144] Returning to the flowchart of Fig. 14, in the next step S1405, the measurable range L0 is calculated from the lens end information and table information. That is, the defocus information for each of the infinity end and close end at the current focus lens position is that the image plane is 1 mm at the infinity end and 9 mm at the close end, so the image plane positions at each end are converted into subject distances, and the resulting range is calculated as the measurable range L0. This conversion from image plane position to distance is calculated by CPU 102 as described above.

[0145] In the next step S1406, the measurable range L1 of the distance layer MAP is calculated from the measurable limit information and table information. Here, the measurable limit information is acquired as a correlation calculation shift amount of 20, and using a coefficient for conversion to defocus after correlation calculation (the coefficient is set to 0.025 under the conditions of this embodiment), the measurable range L1 can be calculated as 20×0.025=0.5 mm.

[0146] On the other hand, when the defocus state is large, the signal quality on the image plane deteriorates, resulting in a detection limit, and the limit value due to the defocus value must be further corrected in calculating L1 as second distance measurement limit information.

[0147] Furthermore, when the subject distance becomes long, even a subject movement of 50 cm results in little fluctuation on the image plane, making it difficult to distinguish whether the fluctuation is due to noise or an actual change in subject distance; therefore, it is also necessary to take noise information into account.

[0148] As a result of these factors, while the image plane has a limit of 1 mm on the infinity side and 9 mm on the close-up side based on the lens end information, it is actually possible to detect defocus in a range of ±0.5 mm. In this way, the resolution changes depending on the constraints of the lens mechanism end and noise constraints.

[0149] If the range of ±0.5 mm on this image plane is converted to distance, the measurable range L1 is roughly between 1.7 m and 8 m. The following explanation will be continued using this range L1. The above-mentioned range L0 is the measurable range due to lens restrictions, and L1 is the measurable range due to correlation calculation performance restrictions, and the smaller of either is the measurable range L for the image processing device 100.

[0150] Therefore, in the next step S1407, a comparison process is performed and the narrower of L0 and L1 is selected to determine the measurable range L. Note that calculations are performed for the far and near sides for distance conversion at infinity and close range. Here, the range of L1 is clearly narrower, and since this range applies to both infinity and close range, the measurable range L is also determined to be a 6.3m range between 1.7m and 8m.

[0151] 14, a sub-process (step S1301) is performed to calculate the minimum resolution width M converted into the subject distance and the measurable range L, and layer information for each distance is generated based on the lens information of the lens unit and the distance information. Here, the lens information includes at least one of the focal length, the focus position, and the aperture information.

[0152] 13, in the next step S1302, a setting mode selection screen is displayed for the user on, for example, the display unit 114. Next, in step S1303, the operator (user) of the image processing device 100 is prompted to select whether to set the number of layers of the distance map, to set the layer width, or to set both.

[0153] Here, steps S1302 and S1303 function as a setting step (setting means) for setting layer information. Note that steps S1302 and S1303 as a setting means may be sufficient if they can set at least one of the number of layers and the width of the layer in the subject distance direction as layer information for each distance.

[0154] An example of the display at that time is shown in Fig. 18. Fig. 18(A) is a diagram showing an example of a selection screen for a layer parameter setting mode in the second embodiment. In Fig. 18(A), 1801 indicates the entire menu screen, and 1802 to 1804 show the respective options. Fig. 18(B) is a diagram showing the menu screen which has transitioned from the state of Fig. 18(A) to a state in which both the number of layers and the layer width are selected. A menu screen 1805 shows a state in which both the number of layers and the layer width are selected, and the option 1804 is displayed in inverted black and white as shown in 1806.

[0155] Returning to FIG. 13, in step S1303, as described above, a setting mode selection screen is displayed to the user. When the user selects a setting mode on the menu screen, in the next step S1304, the set setting mode is identified.

[0156] Then, when a layer width setting mode in which only the layer width is set is selected, when a layer number setting mode in which only the number of layers is set is selected, or when a layer number and layer width setting mode in which both are set is selected, the process proceeds to steps S1305 to S1307, respectively. Then, a process is performed to enable the corresponding layer number change flag or layer width change flag, and the process transitions to the respective setting process mode.

[0157] Each setting processing mode will be described in further detail with reference to the flowcharts of Figures 19 to 23. First, when a layer width setting mode in which only the layer width is set is selected, the process proceeds to step S1305 in Figure 13.

[0158] Fig. 19 is a flowchart detailing the layer width setting process in step S1305 in Fig. 13. Fig. 20(A) to (F) are diagrams showing display examples in the layer width setting mode. In Fig. 19, when the layer width setting mode is entered, in step S1901, the above-mentioned measurable range L and the minimum resolution width M of the subject distance are acquired.

[0159] In the next step S1902, it is determined whether the layer width change flag is enabled (On), and if it is not enabled, the flow in Figure 19 ends. When the layer width setting mode is entered in step S1305, the layer width change flag should be enabled (On), so if the flag is not enabled when this layer width change process is entered, it is an error process. Under normal conditions, the layer width change flag is enabled (On), so proceed to the next step S1903.

[0160] In step S1903, the farthest distance value and the nearest distance value when displaying the menu are determined based on the measurable range L. In the next step S1904, only layer widths that are equal to or greater than the minimum resolution width M of the subject distance and that can be set based on the measurable range L are displayed in the menu.

[0161] That is, the minimum resolution width M of the subject distance is also used to calculate the layer width for displaying the menu, and the menu is displayed based on the calculation result. If the measurable range is L and the minimum resolution width is M, the calculation L / M is performed, and the width when the layer width is set to the minimum resolution width M becomes the layer width for displaying the menu. Also, layer widths larger than the minimum resolution width M and smaller than the measurable range L become display candidates.

[0162] In this way, step S1904 sets the layer width as a reference for generating layer information, and also functions as a step for switching the display of set values ​​(layer width display) that can be set according to the lens information of the lens unit.

[0163] However, there is a display limit on the menu screen, so the maximum number of candidates is 20. Also, by setting the minimum unit for dividing the layer width to 10 cm here, for example, if the minimum resolution width M is 30 cm and the step is 10 cm, the layer width candidates can be calculated as 30 cm, 40 cm, 50 cm, ... 310 cm.

[0164] Note that 310cm is the maximum because it is half the maximum value of the measurable range L, and the number of layers is assumed to be 2. However, the maximum number in the menu is 20, and if it exceeds that, the display will be limited to the smallest layer width. In other words, it will be 20 in 10cm increments starting from 30cm. In this way, it is possible to calculate candidates for layer width when creating a distance layer MAP.

[0165] The maximum layer width is when M = L, but since separation becomes meaningless, the minimum number of layers is calculated as 2. It is also possible to set the minimum number of layers to 3 and the layer width to the measurable range L / 3. This is because composite images are inserted before and after the focal position.

[0166] The display state in this case is shown in Fig. 20(A). In Fig. 20(A), the layer width setting menu is displayed on the left side of the screen, and options for layer width setting are displayed. All options can be selected by scrolling. A person is displayed near the center of the screen as the subject of the image.

[0167] Fig. 20(D) shows an example of the display when the display is switched immediately after the layer width setting menu is displayed, and shows a schematic diagram of the depth direction at this time. As the photographable range, the nearest distance of 170 cm and the farthest distance of 800 cm are displayed, and the minimum resolution of 30 cm and the subject at the focus position are simply displayed.

[0168] The display of Fig. 20(A) and Fig. 20(D) can be switched by switching the operation unit 113. With such a menu display, a process of switching the display is performed using the layer width selected by the user. In the next step S1905, the layer width set by moving the cursor on the menu display is detected. Then, the layer width that can be divided according to the set layer width is displayed.

[0169] For example, when a width of 120 cm is set as shown in Fig. 20(B), Fig. 20(E) displays where the main subject is in the layer, and also clearly shows how the layers are separated for the nearest distance of 170 cm and the farthest distance of 800 cm. Fig. 20(C) shows the state when the layer width setting is selected as 60 cm. Fig. 20(F) displays the layer width divided according to the set layer width of 60 cm.

[0170] In step S1906, it is determined whether or not a layer width menu selection has been made in step S1905 and a layer width determination operation has been performed. If the result is No, the process proceeds to step S1908, where it is checked whether or not the user has moved the cursor in the layer width setting menu.

[0171] If the cursor is moved and the layer width is changed, the process returns to step S1905, and the selected layer width is displayed again as described above, and if the cursor is not moved, the process waits for input of a layer width determination operation in step S1906. If it is determined in step S1906 that a layer width determination operation has been performed, the process proceeds to step S1907, where the layer width and number of layers are determined, and the flow in FIG. 19 ends.

[0172] Next, if a layer number setting mode in which only the number of layers is selected in step S1304 in FIG. 13, the process proceeds to step S1306 in FIG. 13, where a layer number setting process is performed.

[0173] FIG. 21 is a flowchart detailing the number-of-layers setting process in step S1306 in FIG. 13, and FIGS. 22(A) to (F) are diagrams showing display examples in the number-of-layers setting process.

[0174] When the number of layers setting processing mode is entered, the flow in Fig. 21 starts, and in step S2101, the above-mentioned measurable range L and minimum resolution width M of the subject distance are acquired. In the next step S2102, it is determined whether the number of layers change flag is enabled (On), and if No, the flow in Fig. 21 ends.

[0175] When the layer number setting mode is entered in step S1306, the layer number change flag should be enabled (On), so if the flag is not enabled during this layer number change process, it is treated as an error. In the normal state, the layer number change flag is enabled (On), so the process proceeds to step S2103, where the farthest and nearest distance values ​​for displaying the menu based on the measurable range L are determined.

[0176] In the next step S2104, the number of layers for displaying the menu is calculated based on the measurable range L and the minimum resolution width M of the subject distance, and only the number of layers that can be specified is displayed on the menu based on the calculation result. Regarding the calculation of the number of layers, if the measurable range is L and the minimum resolution width is M, the maximum number of layers can be calculated by calculating L / M. In this case, it is 670 cm / 30 cm, which is a maximum of 22 layers.

[0177] The minimum value can be set to 2 layers. Note that there is a display limit on the menu screen, with a maximum of 20 candidates, and due to this restriction, the display targets are limited in order of the smallest number of layers, i.e. from 2nd layer to 3rd layer, 4th layer, and so on up to 21st layer.

[0178] In this way, candidates for the number of layers when creating a distance layer MAP are calculated. The display state in this case is shown in Fig. 22(A). In Fig. 22(A), a menu for setting the number of layers is displayed on the left side of the screen, and options for setting the number of layers are displayed. Note that all candidates can be selected by scrolling the display. The rest is the same as Fig. 20, so a description will be omitted.

[0179] In this way, step S2104 sets the number of layers as a reference for generating layer information, and also functions as a step for switching the display of set values ​​(display of the number of layers) that can be set according to the lens information of the lens unit.

[0180] Fig. 22(D) shows the display immediately after the layer number setting menu is displayed, and is a schematic diagram of the depth direction at this time. The shooting range is displayed with a nearest distance of 170 cm and a furthest distance of 800 cm, and the minimum resolution of 30 cm and the subject at the in-focus position are also displayed in a simplified manner.

[0181] The display can be switched between Fig. 22(A) and Fig. 22(D) by operating the operation unit 113. The display is switched using the number of layers selected by the user on this menu display. In the next step S2105, the number of layers set in association with cursor operation on the menu display is detected. Then, the number of layers divided according to the set number of layers is displayed.

[0182] For example, when five layers are set as shown in Fig. 22(B), the location of the main subject in the layer is displayed as shown in Fig. 22(E), and the layer separation for the nearest distance of 170 cm and the farthest distance of 800 cm is clearly displayed. Fig. 22(C) and Fig. 22(F) show the state when the number of layers is set to three.

[0183] In the next step S2106, it is determined whether or not a menu selection for the number of layers was made in step S2105 and an operation to determine the number of layers was performed. If No, the process proceeds to step S2108, where a check is made to see whether or not the user has moved the cursor in the number of layers setting menu. If the cursor has been moved and the number of layers has been changed, the process returns to step S2105, and the selected number of layers is displayed again, as described above. If the cursor has not been moved, the process waits for an operation input to determine the number of layers in step S2106.

[0184] If it is determined in step S2106 that a layer number determination operation has been performed, the process proceeds to the next step S2107, where the layer width is calculated from the determined number of layers and the distance measurement range L, and the flow in Fig. 21 ends. Finally, a case where both the layer width and the number of layers have been selected will be described. In the case of the layer number and layer width setting processing mode, the process proceeds to step S1307 in Fig. 13, where this layer number and layer width setting processing is performed.

[0185] Fig. 23 is a flowchart detailing the layer number / layer width setting process in step S1307 in Fig. 13, and Fig. 24(A)-(E) are diagrams showing display examples in the layer number / layer width setting mode. When the layer number / layer width setting process mode is entered, the flow in Fig. 23 starts, and in step S2301, the above-mentioned measurable range L and minimum resolution width M are obtained. In the next step S2302, first, the layer width setting menu is enabled, and the setting cursor is also moved to the layer width setting menu to select a layer width.

[0186] In the next step S2303, the farthest and nearest distance values ​​are determined when displaying a menu from the measurable range L. Furthermore, the number of layers for displaying the menu is calculated using the measurable range L and the minimum resolution width M, and only layer widths that are equal to or greater than the minimum resolution and that can be set within the measurable range are displayed in the menu.

[0187] Regarding the calculation of the number of layers, if the measurable range is L and the minimum resolution width is M, the maximum number of layers can be calculated by calculating L / M. In this case, it is 670 cm / 30 cm, which is a maximum of 22 layers. The minimum value can be set to 2 layers. Note that there is a display limit on the menu screen, with a maximum of 20 candidates, and due to this restriction, the displayed targets are limited in order of the smallest number of layers. That is, from 2nd layer to 3rd layer, 4th layer, up to 21st layer.

[0188] In this way, candidates for the number of layers when creating the distance layer MAP are calculated. Furthermore, to calculate the layer width, the measurable range L and the minimum resolution width M are used to calculate L / M, and the layer width when the minimum resolution width is M is calculated. The candidates are widths that are larger than the minimum resolution width M and smaller than the measurable range L.

[0189] However, there is a display limit on the menu screen, so the maximum number of candidates is 20. Also, by setting the minimum unit for the layer width to 10 cm here, for example, if the minimum resolution width M is 30 cm and the step is 10 cm, the layer width candidates can be calculated as 30 cm, 40 cm, 50 cm, ... 310 cm.

[0190] The maximum value of 310 cm is half the maximum value of the measurable range L, and the number of layers is assumed to be 2. However, the maximum number in the menu is 20, and if it exceeds that, the display will be limited to the smallest layer width. In other words, it will be 20 layers starting from 30 cm, in 10 cm increments.

[0191] In this way, it is possible to calculate candidates for the layer width when creating a distance layer MAP. The maximum layer width is when M = L, but since this would mean no separation, the minimum number of layers is calculated as 2. The menu display state at this time is shown in Figure 24 (A). On the left side of the screen, menus for setting the number of layers and the layer width are displayed side by side, and the options for each are displayed. All candidates can be selected by scrolling each one.

[0192] Fig. 24(D) shows the display immediately after the menu is displayed, and is a diagram that shows a schematic of the depth direction at this time. As the photographable range, a nearest distance of 170 cm and a furthest distance of 800 cm are displayed, along with a minimum resolution of 30 cm and a simplified display of the subject at the in-focus position. The display can be switched between Fig. 24(A) and Fig. 24(D) by operating the operation unit 113. The display is switched using the layer width and number of layers selected by the user in this menu display.

[0193] In the next step S2304, the layer width selected here is used to display only the number of layers that can be divided at that layer width in the layer number setting menu. Here, valid layer numbers are displayed in black and bold, while invalid layer numbers are not displayed or are displayed in gray.

[0194] Fig. 24(B) shows the state where only the layer width is selected. If 120cm is selected, the menu for the number of layers of 6 or more will not be displayed. Or, it will be grayed out. In the next step S2305, the menu for the layer width is selected and it is determined whether or not a layer width determination operation has been performed.

[0195] If the answer is No, proceed to step S2306 to check whether the user has moved the cursor in the layer width setting menu. If the answer is Yes, it is determined that the layer width has been changed, and the process returns to step S2304 to redisplay the layer number menu with the selected layer width as described above.

[0196] If there is no change in the cursor movement or layer width, step S2305 waits for input of a layer width determination operation. If it is determined in step S2305 that a layer width determination operation has been performed, the process proceeds to step S2307, where the setting cursor is moved to the layer number setting menu, enabling selection of the number of layers. In the next step S2308, when the number of layers is selected, only layer widths that can be divided by the selected number of layers are displayed, and other layer width options are hidden or grayed out.

[0197] In this way, steps S2304 and S2308 function as steps for switching the display of set values ​​(display of layer width and number of layers) that can be set according to the lens information of the lens unit.

[0198] In the next step S2309, it is determined whether or not a layer number determination operation has been performed. If No, the process proceeds to step S2311, where it is determined whether or not the cursor has been moved on the layer number setting menu. If Yes, it is determined that the number of layers has been changed and the process returns to step S2308, and if No, it returns to step S2309. If it is determined that the number of layers has been determined, the process proceeds to step S2310, where it is determined whether or not the set number of layers and layer width can be set.

[0199] This is because the cursor can be moved to non-displayed or grayed-out setting values. If the answer is No, that is, if the combination is not available for setting, the process returns to step S2302. If the answer is Yes, the flow in FIG. 23 ends.

[0200] Once both the number of layers and the layer width are set, the display will change according to the number of layers and the layer width. For example, if five layers are set as in Figure 24(C), the distance layer map image will be displayed as in Figure 24(E).

[0201] As described above, the detectable distance conditions change depending on the focus position, zoom state, and aperture state of the lens. Therefore, in each of the setting modes for the number of layers and layer width described in this embodiment, if these lens conditions are changed, resetting is immediately started for each of them.

[0202] Alternatively, in each of the setting modes for the number of layers and the layer width, the lens operations may be configured not to be accepted or ignored. When there are multiple areas into which a composite image is inserted and multiple pieces of insertion layer information into which a composite image is inserted, the colors of the areas of the composite image to be changed that overlap with the subject may be different.

[0203] It is desirable to generate layer information for each distance when the lens information changes by a predetermined amount or more. For example, when the imaging unit moves significantly, when the focal length changes by a predetermined value or more, when the brightness of the subject changes and the aperture value changes, it is desirable to detect the occurrence of such changes and recalculate the number of layers (number of layers) and layer width (width in the distance direction).

[0204] Specifically, for example, an acceleration sensor is provided in the imaging section, and when it detects that the acceleration is equal to or greater than a predetermined value, the number of layers and the layer width (width in the distance direction) are recalculated. Also, when the lens unit attached to the imaging section is replaced or the zoom state of the lens unit is changed, the number of layers and the layer width (width in the distance direction) are recalculated.

[0205] In addition, it is desirable to configure the system to detect when a recognized subject moves by a predetermined amount or when a background subject that is farther than a predetermined distance away changes by a predetermined value or more, and to recalculate the number of layers (number of layers) and layer width (width in the distance direction). Conversely, once generation of layer information for each distance has started, it may be possible to not accept operations that change lens information by a predetermined amount or more, such as focal length, aperture value, or focus adjustment operation.

[0206] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and are not excluded from the scope of the present invention. In addition, parts of the above embodiments may be appropriately combined, including the following combinations.

[0207] (Configuration 1) An image processing device comprising: an image acquisition means for acquiring an image including a subject through a lens unit; a distance information acquisition means for acquiring distance information indicating the distance to the subject; a layer information generation means for generating layer information regarding layers for each distance based on the distance information; and a setting means for setting a standard for generating the layer information and for switching a display of setting values ​​that can be set according to the lens information of the lens unit.

[0208] (Configuration 2) The image processing device described in Configuration 1, characterized in that the image acquisition means includes an imaging element in which a plurality of photoelectric conversion units, each of which receives a light beam passing through a different pupil region of the lens unit, are arranged at each pixel, and the distance information can be acquired based on a phase difference between the outputs of the plurality of photoelectric conversion units.

[0209] (Configuration 3) The image processing device according to configuration 2, wherein the image is acquired by adding up outputs from the plurality of photoelectric conversion units.

[0210] (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the layer information generating means converts distance information of a distance where the subject is not present into the layer information.

[0211] (Configuration 5) The image processing device according to any one of configurations 1 to 4, wherein the lens information includes at least one of focal length, focus position, and aperture information.

[0212] (Configuration 6) The image processing device according to any one of configurations 1 to 5, wherein the layer information for each distance is generated when the lens information has changed by a predetermined amount or more.

[0213] (Configuration 7) The image processing device according to any one of configurations 1 to 6, wherein the layer information generating means changes the criterion for generating the layer information in accordance with the image.

[0214] (Configuration 8) The image processing device according to any one of configurations 1 to 7, wherein the criteria include the number of the layers and a width of the layers in the distance direction.

[0215] (Configuration 9) The image processing device according to any one of configurations 1 to 8, wherein the layer information generating means generates a histogram based on the distance information, and generates the layer information for each distance based on the histogram.

[0216] (Configuration 10) The image processing device described in any one of configurations 1 to 9, characterized in that the layer information generation means recognizes the subject using an image recognition means, and generates the layer information for each distance according to the recognized subject.

[0217] (Configuration 11) The image processing device according to configuration 10, wherein the layer information generating means changes a width in the distance direction of the layer for each distance according to the type of the recognized subject.

[0218] (Configuration 12) The image processing device according to configuration 10 or 11, wherein the layer information generating means is capable of selecting the type of subject that can be recognized.

[0219] (Configuration 13) The image processing device according to any one of configurations 1 to 12, wherein the layer information generating means generates the layer information based on a plurality of frames of the image.

[0220] (Configuration 14) The image processing device according to any one of configurations 1 to 13, wherein the layer information generating means generates the layer information for each distance based on the lens information of the lens unit and the distance information.

[0221] (Configuration 15) The image processing device according to any one of configurations 1 to 14, wherein the layer information generating means calculates a minimum resolution in the layer information for each distance based on the lens information of the lens unit.

[0222] (Configuration 16) The image processing device according to any one of configurations 1 to 15, wherein the setting means is capable of setting, for the image, a composite image insertion area for inserting a composite image, and insertion layer information which is layer information into which the composite image is inserted.

[0223] (Configuration 17) The image processing device described in Configuration 16, characterized in that it has a display control means for displaying, when the composite image is displayed in the composite image insertion area, an overlap area where the composite image and the image of the subject overlap in a predetermined color or pattern corresponding to the layer information of the subject.

[0224] (Configuration 18) The image processing device described in Configuration 17, characterized in that when the layer information of the image of the subject and the insertion layer information into which the composite image is inserted are the same, the overlapping area is displayed in a predetermined color or pattern different from other areas.

[0225] (Configuration 19) The image processing device according to any one of configurations 15 to 18, wherein the setting means is capable of selecting at least one of the number of layers and the width of the layers in the distance direction as the layer information.

[0226] (Configuration 20) The image processing device described in any one of configurations 1 to 19, characterized in that the layer information generating means generates the layer information based on the distance information when the acceleration of the image acquiring means becomes equal to or greater than a predetermined value.

[0227] (Configuration 21) The image processing device described in any one of configurations 1 to 20, characterized in that the layer information generation means generates the layer information based on the distance information when a subject in the image that is more than a predetermined distance away changes by more than a predetermined value.

[0228] (Configuration 22) An image processing device comprising: an image acquisition means for acquiring an image including a subject; a distance information acquisition means for acquiring distance information indicating the distance to the subject; a layer information generation means for generating layer information for each distance based on the distance information; a setting means for setting a composite image insertion area for inserting a composite image into the image and insertion layer information which is layer information into which the composite image is inserted; and a display control means for displaying an overlap area where the composite image and the image of the subject overlap when the composite image is displayed in the composite image insertion area in a predetermined color or pattern corresponding to the layer information of the subject.

[0229] (Configuration 23) An image processing device comprising: an image acquisition means for acquiring an image including a subject; a distance information acquisition means for acquiring distance information indicating the distance to the subject; and a layer information generation means for generating layer information regarding layers for each distance based on the distance information and changing a criterion for generating the layer information according to the image.

[0230] (Method 1) An image processing method comprising: an image acquisition step of acquiring an image including a subject through a lens unit; a distance information acquisition step of acquiring distance information indicating the distance to the subject; a layer information generation step of generating layer information regarding layers for each distance based on the distance information; and a setting step of setting standards for generating the layer information and switching a display of settable setting values ​​according to the lens information of the lens unit.

[0231] (Method 2) An image processing method comprising: an image acquisition step of acquiring an image including a subject; a distance information acquisition step of acquiring distance information indicating the distance to the subject; a layer information generation step of generating layer information for each distance based on the distance information; a setting step of setting, in the image, a composite image insertion area for inserting a composite image and insertion layer information which is layer information into which the composite image is inserted; and a display control step of displaying, when the composite image is displayed in the composite image insertion area, an overlap area where the composite image and the image of the subject overlap in a predetermined color or pattern corresponding to the layer information of the subject.

[0232] (Method 3) An image processing method comprising: an image acquisition step of acquiring an image including a subject; a distance information acquisition step of acquiring distance information indicating the distance to the subject; and a layer information generation step of generating layer information relating to layers for each distance based on the distance information, and changing the criteria for generating the layer information depending on the image.

[0233] (Program) A computer program for controlling each means of the image processing device according to any one of configurations 1 to 23 by a computer.

[0234] A computer program for implementing all or part of the control in this embodiment may be supplied to an image processing device or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0235] 100: Image processing device 101, 1301: Internal bus 102, 1302: CPU 103, 1303: ROM 104:RAM 105: Image processing unit 106: Lens unit 107: Imaging unit 108, 1308: Network module 109: Image output unit 110: Recording medium I / F 111: Frame memory 112: Recording media 113, 1313: Operation section 114, 1314: Display section 115: Object detection unit 900: Mobile terminal 1100, 1101, 1102: Subject

Claims

1. an image capturing means for capturing an image including a subject through a lens unit; distance information acquisition means for acquiring distance information indicating the distance to the subject; a layer information generating means for generating layer information relating to layers for each distance based on the distance information; a setting unit that sets a standard for generating the layer information and switches a display of set values ​​that can be set according to the lens information of the lens unit, The image processing device is characterized in that the layer information generating means generates the layer information based on a plurality of frames of the image.

2. the image acquisition means includes an image sensor in which a plurality of photoelectric conversion units are arranged for each pixel, the photoelectric conversion units receiving light beams passing through different pupil regions of the lens unit, respectively; The image processing device according to claim 1 , wherein the distance information can be obtained based on a phase difference between outputs of the plurality of photoelectric conversion units.

3. 3. The image processing device according to claim 2, wherein the image is acquired by adding outputs from the plurality of photoelectric conversion units.

4. 2. The image processing apparatus according to claim 1, wherein the layer information generating means converts distance information of a distance where the subject does not exist into the layer information.

5. 2. The image processing device according to claim 1, wherein the lens information includes at least one of focal length, focus position, and aperture information.

6. The image processing device according to claim 1 , wherein the layer information for each distance is generated when the lens information has changed by a predetermined amount or more.

7. 2. The image processing apparatus according to claim 1, wherein the layer information generating means changes the criteria for generating the layer information depending on the image.

8. The image processing device according to claim 1 , wherein the criteria include the number of layers and the width of the layers in the distance direction.

9. 2. The image processing apparatus according to claim 1, wherein the layer information generating means generates a histogram based on the distance information, and generates the layer information for each distance based on the histogram.

10. 2. The image processing apparatus according to claim 1, wherein the layer information generating means recognizes the subject by an image recognition means, and generates the layer information for each distance according to the recognized subject.

11. 11. The image processing apparatus according to claim 10, wherein the layer information generating means changes the width of the layer in the distance direction for each distance according to the type of the recognized subject.

12. 11. The image processing apparatus according to claim 10, wherein the layer information generating means is capable of selecting the type of subject that can be recognized.

13. 2. The image processing apparatus according to claim 1, wherein the layer information generating means generates the layer information for each distance based on the lens information of the lens unit and the distance information.

14. 2. The image processing device according to claim 1, wherein the layer information generating means calculates a minimum resolution in the layer information for each distance based on the lens information of the lens unit.

15. 2. The image processing device according to claim 1, wherein the setting means is capable of setting, for the image, a composite image insertion area for inserting a composite image and insertion layer information which is layer information into which the composite image is inserted.

16. The image processing device described in claim 15, characterized in that it has a display control means for displaying, when the composite image is displayed in the composite image insertion area, an overlapping area where the composite image and the image of the subject overlap in a predetermined color or pattern corresponding to the layer information of the subject.

17. The image processing device described in claim 16, characterized in that when the layer information of the image of the subject and the insertion layer information into which the composite image is inserted are the same, the overlapping area is displayed in a predetermined color or pattern different from other areas.

18. 15. The image processing apparatus according to claim 14, wherein the setting means is capable of selecting at least one of the number of layers and the width of the layers in the distance direction as the layer information.

19. 2. The image processing apparatus according to claim 1, wherein the layer information generating means generates the layer information based on the distance information when the acceleration of the image acquiring means reaches or exceeds a predetermined value.

20. 2. The image processing device according to claim 1, wherein the layer information generating means generates the layer information based on the distance information when a subject in the image that is farther away than a predetermined distance changes by more than a predetermined value.

21. image acquisition means for acquiring an image including a subject; distance information acquisition means for acquiring distance information indicating the distance to the subject; a layer information generating means for generating layer information for each distance based on the distance information; a setting means for setting a composite image insertion area for inserting a composite image into the image and insertion layer information, which is layer information into which the composite image is inserted; a display control means for displaying an overlapping area where the composite image and the image of the subject overlap in a predetermined color or pattern according to layer information of the subject when the composite image is displayed in the composite image insertion area, The image processing device is characterized in that the layer information generating means generates the layer information based on a plurality of frames of the image.

22. image acquisition means for acquiring an image including a subject; distance information acquisition means for acquiring distance information indicating the distance to the subject; a layer information generating means for generating layer information relating to layers for each distance based on the distance information and changing a criterion for generating the layer information according to the image; and The image processing device is characterized in that the layer information generating means generates the layer information based on a plurality of frames of the image.

23. an image acquisition step of acquiring an image including a subject through a lens unit; a distance information acquisition step of acquiring distance information indicating a distance to the subject; a layer information generating step of generating layer information relating to layers for each distance based on the distance information; a setting step of setting a standard for generating the layer information and switching a display of settable setting values ​​according to the lens information of the lens unit, The image processing method, wherein the layer information generating step generates the layer information based on a plurality of frames of the image.

24. an image acquisition step of acquiring an image including a subject; a distance information acquisition step of acquiring distance information indicating a distance to the subject; a layer information generating step of generating layer information for each distance based on the distance information; a setting step of setting a composite image insertion area for inserting a composite image into the image and insertion layer information, which is layer information into which the composite image is inserted; a display control step of displaying an overlapping area where the composite image and the image of the subject overlap in a predetermined color or pattern according to layer information of the subject when the composite image is displayed in the composite image insertion area, The image processing method, wherein the layer information generating step generates the layer information based on a plurality of frames of the image.

25. an image acquisition step of acquiring an image including a subject; a distance information acquisition step of acquiring distance information indicating a distance to the subject; a layer information generating step of generating layer information relating to layers for each distance based on the distance information and changing a criterion for generating the layer information according to the image, The image processing method, wherein the layer information generating step generates the layer information based on a plurality of frames of the image.

26. A computer program for causing a computer to function as each means of an image processing device described in any one of claims 1 to 22.

Images