Image processing device, image processing method, camera device and storage medium

Abstract

An image processing device is provided, which displays an area capable of measuring distances corresponding to a plurality of measurement points specified in an object area in an image based on measurement accuracy. An image processing unit (13) of a digital camera performs control to measure and display the size of an object corresponding to the plurality of measurement points on a captured image. An image generation unit (130) generates data related to the captured image, and a depth generation unit (131) generates depth information related to the distance distribution in the depth direction corresponding to the captured image. A display control unit (132) displays the captured image on a display unit, and performs control to display an object area corresponding to an object within a predetermined distance range relative to a reference distance in the depth direction in a recognizable manner.

Description

Technical Field

[0001] The present invention relates to a technology for measuring distances between a plurality of measurement points designated in an image and highlighting a specific area. Background Art

[0002] There is a technology for measuring the size between measurement points in an image specified by a user. For example, in the technology for measuring the size of a subject area in an image, when a plurality of size measurement points are specified on the subject area, a process of measuring the size between the size measurement points is performed and the measurement result is displayed.

[0003] Patent document 1 discloses a method, apparatus, and terminal device for specifying spatial parameters from an image. A method has been proposed in which edge detection is performed on an image to specify whether the specified size measurement position is an edge corresponding to both ends of a subject, so as to provide a notification of whether to adjust the measurement position.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent No. 6392991 Summary of the invention

[0007] Technical issues

[0008] In the technology disclosed in Patent Document 1, the case where the distances in the depth direction from the camera to both ends of the subject are different is not considered. In the case of acquiring an image by a conventional camera, a central projection is performed, and the camera magnification is different depending on the distance in the depth direction of the image. Therefore, the size of the subject in the captured image is different between the front side (camera side) position and the depth side position. If the size between the edges of the two ends of the subject is measured under such camera conditions, there is a problem that the correct size of the subject cannot be obtained.

[0009] An object of the present invention is to provide an image processing device that displays an area where distances corresponding to a plurality of measurement points designated in a subject area of ​​an image can be measured based on measurement accuracy.

[0010] Technical means of solving problems

[0011] According to an embodiment of the present invention, there is provided an image processing device, comprising: an acquisition unit for acquiring an image and information related to the distance distribution in a depth direction corresponding to the image; and a control unit for controlling the display of the image, wherein the control unit performs control so as to display an object area in the image corresponding to a object within a distance range relative to a reference distance in the depth direction in a recognizable manner, wherein in the distance range, the distance can be measured with a measurement accuracy higher than the reference.

[0012] Advantageous Effects of the Invention

[0013] According to the image processing apparatus of the present invention, it is possible to provide an image processing apparatus that displays an area where distances corresponding to a plurality of measurement points designated in an object area of ​​an image can be measured based on measurement accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 : is a block diagram showing the functional configuration of an image pickup apparatus according to the embodiment and its modifications.

[0015] Figure 2 It is a diagram showing an imaging element according to the embodiment and a modification.

[0016] Figure 3 2 is a diagram showing the distance measurement principle based on the imaging plane phase difference method.

[0017] Figure 4 is a flowchart showing the processing performed in the first embodiment.

[0018] Figure 5 It is a diagram showing operations performed when the size derivation process is executed.

[0019] Figure 6 2 is a diagram showing a method of highlighting a subject area.

[0020] Figure 7 is a flowchart showing the processing performed in the second embodiment.

[0021] Figure 8 2 is a diagram showing operations performed when a size derivation process is executed according to the second embodiment.

[0022] Fig. 9 is a flowchart showing processing performed in the modification example.

[0023] Fig.10 2 is a diagram illustrating a method of highlighting a subject region in the third embodiment. DETAILED DESCRIPTION

[0024] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the embodiment, an application example of a digital camera capable of acquiring depth information related to the distance distribution of a subject will be described as an image processing device. The present invention is applicable to any device capable of calculating the size of a subject based on a captured image and depth information corresponding to the captured image. Depth information is information in the depth direction of a captured image, and a depth image or a depth map is information indicating the distribution of depth information. As depth information, an image offset map calculated based on a plurality of viewpoint images having different viewpoints, a defocus map calculated by multiplying an image offset by a predetermined conversion coefficient, and a distance map obtained by converting a defocus amount into distance information of a subject can be used.

[0025] [First embodiment]

[0026] Will refer to Figure 1 The structure of the digital camera 100 is described. Figure 1 1 is a block diagram showing the functional structure of the digital camera 100. The imaging optical system 10 is composed of a lens unit included in the digital camera 100 or a lens device attachable to a camera body, and forms an optical image of a subject on the imaging element 11. The imaging optical system 10 is composed of a plurality of lenses (not shown) arranged on an optical axis 102, and has an exit pupil 101 at a position separated from the imaging element 11 by a predetermined distance. In the present specification, a direction parallel to the optical axis 102 is set as a z direction (depth direction). That is, the depth direction is a direction in which a subject exists relative to the position of the digital camera 100. A direction orthogonal to the optical axis 102 and parallel to the horizontal direction of the imaging element 11 is set as an x ​​direction, and a direction orthogonal to the optical axis 102 and parallel to the vertical direction of the imaging element 11 is set as a y direction.

[0027] The imaging element 11 is, for example, a charge coupled device (CCD) type image sensor or a complementary metal oxide semiconductor (CMOS) type image sensor. The imaging element 11 performs photoelectric conversion on the subject image formed on the imaging plane via the imaging optical system 10, and outputs an image signal related to the subject image. As described later, the imaging element 11 of the present embodiment has a distance measurement function based on the imaging plane phase difference method, and in addition to capturing an image, the imaging element 11 is also capable of generating and outputting distance information indicating the distance from the imaging device to the subject (subject distance).

[0028] The control unit 12 includes a central processing unit (CPU) and a microprocessor, etc., and controls the operation of each component of the digital camera 100. For example, the control unit 12 performs automatic focusing (AF: automatic focus adjustment) when shooting, change of the focus (focus) position, change of the F value (aperture value), shooting of an image, etc. The control unit 12 controls the image processing unit 13, the storage unit 14, the input unit 15, the display unit 16, and the communication unit 17.

[0029] The image processing unit 13 performs various types of image processing in the digital camera 100. The image processing unit 13 includes an image generating unit 130, a depth generating unit 131, a display control unit 132, a position specifying unit 133, and a size setting unit 134. A geometric transformation unit 136 will be described in an embodiment described later. The image processing unit 13 has a memory 135 used as a work area for image processing. In addition to the structure using a logic circuit, the image processing unit 13 may be configured to include a CPU and a memory storing an arithmetic processing program.

[0030] The image generation unit 130 performs various types of signal processing such as noise removal, demosaicing, brightness signal conversion, aberration correction, white balance adjustment, and color correction on the image signal output from the image pickup element 11. The captured image data output from the image generation unit 130 is temporarily stored in the memory 135. The display control unit 132 reads the captured image data from the memory 135 to generate display data, and the image is displayed on the screen of the display unit 16.

[0031] The depth generation unit 131 generates a depth image indicating the distribution of depth information based on signals obtained by distance measuring pixels of the imaging element 11. The depth image information is two-dimensional information in which a value indicated by each pixel represents distance information of an object existing in a region of the captured image corresponding to the pixel.

[0032] The display control unit 132 controls so that the subject area in the captured image that satisfies the predetermined size measurement accuracy is displayed in a recognizable manner based on the distance information of the subject. The distance information is based on at least the depth image. Figure 1 The distance information or distance range information in the z direction in the . Three-dimensional distance information including distance information in the x direction and the y direction may be used.

[0033] The position specifying unit 133 performs processing for the user to specify a position when measuring the size of the object. When the user specifies at least two desired measurement positions in the image, coordinate information can be acquired.

[0034] The size setting unit 134 measures the size between at least two points specified by the user in the position specifying unit 133. The size may be any of a size in a pixel portion on an image, a size on an image plane converted from a pixel size, and a size on the object side converted with respect to an imaging magnification.

[0035] The storage unit 14, the input unit 15, the display unit 16, and the communication unit 17 are all connected to the bus. The storage unit 14 includes a nonvolatile storage medium. For example, captured image data, intermediate data generated in the processing of each component, and parameters referred to in the operation of the image processing unit 13 and the digital camera 100 are stored in the storage unit 14. The storage unit 14 may be any storage unit as long as the processing performance allowed by the realization of the processing is guaranteed. It is preferable that a storage medium capable of high-speed reading and writing and having a large capacity is used, and for example, a flash memory or the like is used.

[0036] The input unit 15 is a user interface unit including a device for detecting an operation input by an operator. For example, the input unit 15 detects an operation of inputting information to the digital camera 100 or changing settings through a dial, button, switch, or touch panel, and outputs a signal corresponding to the operation input to the control unit 12.

[0037] The display unit 16 includes, for example, a display device such as a liquid crystal display or an organic electroluminescent (EL) display. The display unit 16 is used to check the composition during image capture based on the through display of the captured image, display various setting screens, and report message information. In the case of an embodiment using a touch panel, both a display function and an input function can be provided. The user can perform an operation of specifying a measurement position in dimension measurement while referring to the display screen.

[0038] The communication unit 17 is a communication interface unit that transmits and receives information between the digital camera 100 and an external device. The communication unit 17 can transmit acquired captured image data, depth information, size, coordinate information, size measurement accuracy, and the like to the external device.

[0039] Next, we will refer to Figure 2 The structure of the image pickup element 11 is described. Figure 2 (A) is a schematic diagram showing an array of pixel groups, wherein the Figure 2 The direction of the paper plane of (A) is set as the z direction, and two directions orthogonal to each other in the paper plane are set as the x direction and the y direction. Figure 2 (B) is a schematic diagram showing a pixel structure, in which the direction perpendicular to the paper surface is set as the y direction, Figure 2 Two directions orthogonal to each other in the paper plane of (B) are set as the x direction and the z direction.

[0040] like Figure 2 As shown in (A), the imaging element 11 is composed of a pixel group in which a plurality of pixel units 110 are arranged. One pixel unit 110 has a two-row and two-column structure to which different color filters are applied. As shown in the enlarged view, color filters of red (R), green (G), and blue (B) are provided, and an image signal corresponding to color information of any one of R, G, and B is output from the photoelectric conversion element constituting the pixel unit 110. Figure 2 (A) shows an example in which R is arranged at the upper left, B is arranged at the lower right, G is arranged at the lower left, and G is arranged at the upper right, but the array of color filters is not limited thereto.

[0041] The imaging element 11 has a distance measurement function based on an imaging plane phase difference method. Figure 2 (B) is along Figure 2 (A) is a cross-sectional view of a unit pixel taken along line II* in FIG. The unit pixel is constructed with a light guide layer 113 including a microlens 111 and a color filter 112, and a light receiving layer 114 including a first photoelectric conversion unit 115 and a second photoelectric conversion unit 116. In the light guide layer 113, the microlens 111 efficiently guides incident light to the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116. The color filter 112 allows light of a predetermined wavelength band to pass. The color filter 112 allows only light of a wavelength bandwidth corresponding to any one of R, G, and B to pass through, and guides the light to the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 of the subsequent stage.

[0042] The light receiving layer 114 is provided with a first photoelectric conversion unit 115 and a second photoelectric conversion unit 116 that perform photoelectric conversion on the received light to output analog image signals respectively. Two signals output from the two photoelectric conversion units are used for distance measurement. That is, in the two photoelectric conversion units arranged along a predetermined direction (horizontal direction) in the imaging element 11, an image signal formed by a signal output from the first photoelectric conversion unit 115 will be referred to as an A image signal, and an image signal formed by a signal output from the second photoelectric conversion unit 116 will be referred to as a B image signal. Depth information or distance information can be acquired based on the phase difference between the A image signal and the B image signal. That is, the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 each partially receive a light beam incident through the microlens 111. Therefore, the A image signal and the B image signal are pupil-divided image signals related to light beams that have passed through different pupil partial areas of the exit pupil 101 of the imaging optical system 10. The image signal (so-called addition image signal) photoelectrically converted by the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 in each pixel unit is used as a captured image. That is, a signal obtained by combining the A image signal and the B image signal is equivalent to an image signal for viewing output from the photoelectric converter in a structure in which each unit pixel has only one photoelectric converter.

[0043] The imaging element 11 of the present embodiment can output an image signal for viewing, and an A image signal and a B image signal (pupil division image) for distance measurement. All pixel portions constituting the imaging element 11 are provided with a plurality of photoelectric conversion portions, and high-density depth information can be acquired. Although a structure in which two photoelectric conversion portions are arranged in one pixel has been described, the present invention is not limited thereto, and a structure in which three or more photoelectric conversion portions are arranged in one pixel may be used.

[0044] Will refer to Figure 3 The distance measurement principle based on the imaging plane phase difference method is described. The object distance can be derived based on the outputs of the first photoelectric conversion section 115 and the second photoelectric conversion section 116 (pupil division image group). Figure 3 (A) is a schematic diagram showing light beams received by the exit pupil 101 of the imaging optical system 10 and the first photoelectric conversion portion 115 . Figure 3 (B) is a schematic diagram showing light beams received by the exit pupil 101 of the imaging optical system 10 and the second photoelectric conversion section 116. Figure 3 (A) and Figure 3 In (B), the direction perpendicular to the paper plane is set as the y direction, and two directions orthogonal to each other in the paper plane are set as the x direction and the z direction.

[0045] Figure 3 (A) and Figure 3 The microlens 111 shown in (B) is arranged so that the exit pupil 101 and the light receiving layer 114 are optically conjugate. The light beam that has passed through the exit pupil 101 of the imaging optical system 10 is focused by the microlens 111 and guided to the first photoelectric conversion unit 115 or the second photoelectric conversion unit 116. In this case, as shown in FIG. Figure 3 (A) and Figure 3 As shown in (B), the first photoelectric conversion portion 115 and the second photoelectric conversion portion 116 respectively mainly receive light that has passed through different pupil partial areas 310 and 320. The first photoelectric conversion portion 115 receives light that has passed through the first pupil partial area 310, and the second photoelectric conversion portion 116 receives light that has passed through the second pupil partial area 320.

[0046] The plurality of first photoelectric conversion units 115 included in the imaging element 11 output first image signals corresponding to the A image signals. The plurality of second photoelectric conversion units 116 included in the imaging element 11 output second image signals corresponding to the B image signals. The intensity distribution of the image formed on the imaging element 11 by the light that has passed through the first pupil partial area 310 can be acquired from the first image signal. The intensity distribution of the image formed on the imaging element 11 by the light that has passed through the second pupil partial area 320 can be acquired from the second image signal. The relative position deviation amount (so-called parallax amount) between the first image signal and the second image signal is a value corresponding to the defocus amount. The reference image 300 will be referred to as the image signal. Figure 3 (C) Figure 3 (D) and Figure 3 (E) Describes the relationship between the amount of parallax and the amount of defocus.

[0047] Figure 3 (C) Figure 3 (D) and Figure 3 (E) shows a first light beam 311 passing through the first pupil partial area 310 and a second light beam 321 passing through the second pupil partial area 320 . Figure 3 (C) shows a state when focusing, and the first light beam 311 and the second light beam 321 are converged on the light receiving surface of the image pickup element 11. In this case, the amount of parallax between the first image signal and the second image signal is zero.

[0048] Figure 3 (D) shows a defocused state in which the focus is located in the negative direction (left direction) of the z-axis (optical axis) on the image side. The parallax amount between the first image signal and the second image signal is a negative value. Figure 3 (E) shows a defocused state in which a focus is formed in the positive direction of the z-axis on the image side. The parallax amount between the first image signal and the second image signal is a positive value. Figure 3 (D) with Figure 3 As can be seen from the comparison between (E), the direction of the positional deviation switches depending on the positive and negative of the defocus amount. It can be seen that, depending on the defocus amount, a positional deviation occurs according to the imaging relationship (geometric optical relationship) of the imaging optical system. The parallax amount corresponding to the positional deviation between the first image signal and the second image signal can be detected by using an area-based matching method.

[0049] Will refer to Figure 4 A size derivation process for deriving the size of an imaged object is described. Figure 4(A) is a flowchart showing a specific processing example. The processing described below can be realized by the control unit 12 reading a corresponding processing program stored in, for example, the storage unit 14, loading the program into a volatile memory (not shown), and executing the program. For example, when the camera is set to the size measurement mode and performs imaging, the size derivation processing is started.

[0050] In S401, the control unit 12 performs an image pickup process according to the image pickup settings such as the set focus position, aperture, and exposure time. More specifically, the control unit 12 performs an image pickup operation using the image pickup element 11, and controls to send the acquired captured image signals to the image processing unit 13. The image-processed data is stored in the memory 135. Here, the captured image signals include an image signal S1 formed by a signal output only from the first photoelectric conversion unit 115 of the image pickup element 11, and an image signal S2 formed by a signal output only from the second photoelectric conversion unit 116.

[0051] In S402, the image processing unit 13 generates a viewing image based on the acquired image signal. More specifically, in the image processing unit 13, first, the image generation unit 130 generates a Bayer array image by adding the pixel values ​​of each pixel in the image signal S1 and the image signal S2. The image generation unit 130 performs a demosaicing process related to the images of each color of R, G and B on the generated Bayer array image, and generates a viewing image. The demosaicing process is a process performed according to the color filter set on the imaging element, and various methods are used as a demosaicing method. In addition to this, the image generation unit 130 performs processes such as noise removal or reduction, brightness signal conversion, aberration correction, white balance adjustment, and color correction. Data related to the generated viewing image is stored in the memory 135.

[0052] In S403, the image processing unit 13 generates a depth image based on the acquired image signal. The depth generation unit 131 performs generation-related processing on the depth image. Figure 4 (B) is a flowchart describing the depth image generation process. In the next S404, the display control unit 132 performs a display process of the subject that satisfies the measurement accuracy in the size measurement. Figure 4 The flowchart of (C) describes the display processing of the object related to the measurement target.

[0053] In S405, the position specifying unit 133 performs measurement point specifying processing. Since the display unit 16 displays the viewing image so that the subject within the distance range that satisfies the size measurement accuracy can be identified, the user can specify a desired measurement point. The position specifying unit 133 calculates the coordinate value of the pixel position specified by the operation instruction and sends the coordinate value to the size setting unit 134.

[0054] In S406, the size setting unit 134 calculates the distance on the image between the measurement points specified in S405, that is, the distance in pixel units. The distance in pixel units is, for example, the Euclidean distance on the image, and is not necessarily an integer value. Subsequently, the size setting unit 134 calculates the imaging magnification based on the distance to the subject at the focus position (subject distance) and the focal length of the imaging optical system 10 during imaging. According to the pixel size and imaging magnification of the imaging element 11, the size of the space occupied by one pixel at the focusing distance on the object side can be calculated. The size setting unit 134 calculates the distance between the measurement points on the object side, that is, the distance in three-dimensional space, based on the distance in pixel units between the measurement points and the size of one pixel on the object side. Data related to the calculated distance is stored in the memory 135.

[0055] In S407, the display control unit 132 reads the distance data calculated in S406 from the memory 135, and controls to display the size on the screen of the display unit 16. In S408, the size setting unit 134 determines whether the user is to change the measurement point. The change of the measurement point is determined based on the operation instruction from the user. In the case where it is determined that the measurement point is to be changed, the flow returns to S405 and the processing from S405 to S407 is continued. In the case where it is determined that the measurement point is not to be changed, the flow proceeds to S409.

[0056] In S409, the size setting unit 134 adds information about the designated measurement point (coordinates on the image) and the calculated size information to the viewing image data as additional information, and stores the information in the storage unit 14. Information about measurement accuracy may be added.

[0057] Will refer to Figure 4 (B) Describe the depth image generation process. In S4031, the depth generation unit 131 performs light amount correction processing on the image signal S1 and the image signal S2. At the peripheral viewing angle of the imaging optical system 10, due to the difference in shape between the first pupil partial area 310 and the second pupil partial area 320 caused by vignetting, the light amount between the image signal S1 and the image signal S2 loses balance. Therefore, in this step, the depth generation unit 131 corrects the light amount of the image signal S1 and the image signal S2 by using, for example, a light amount correction value pre-stored in the memory 135.

[0058] In S4032, the depth generation unit 131 performs processing for reducing the noise generated during the conversion in the imaging element 11. Specifically, the depth generation unit 131 performs filtering processing on the image signal S1 and the image signal S2 to achieve noise reduction. Generally, the higher the spatial frequency, the lower the signal-to-noise (S / N) ratio, and thus the noise component increases relatively. Therefore, the depth generation unit 131 performs processing by using a low-pass filter whose pass rate decreases as the spatial frequency increases. Depending on the manufacturing error of the imaging optical system 10, etc., the light amount correction shown in S4031 may not be a suitable result. Therefore, the depth generation unit 131 adopts a bandpass filter that cuts off the DC component and has a low pass rate for high-frequency components.

[0059] In S4033, the depth generation unit 131 derives the amount of parallax between the A image and the B image based on the image signal S1 and the image signal S2. Specifically, the depth generation unit 131 sets a focus point corresponding to the representative pixel information and a control area centered on the focus point in the A image corresponding to the image signal S1. The control area is, for example, a rectangular area, such as a square area with a predetermined length on one side centered on the focus point. Next, the depth generation unit 131 sets a reference point in the B image corresponding to the image signal S2, and sets a reference area centered on the reference point. The reference area has the same size and shape as the above-mentioned control area. While moving the reference point in sequence, the depth generation unit 131 calculates the correlation between the image included in the control area related to the image signal S1 and the image included in the reference area related to the image signal S2. The reference point with the highest correlation is designated as the corresponding point corresponding to the focus point in the image signal S2. The relative position deviation amount between the designated corresponding point and the focus point corresponds to the amount of parallax at the focus point.

[0060] The depth generation unit 131 calculates the amount of disparity while sequentially changing the focus point according to the representative pixel information, thereby deriving the amount of disparity at a plurality of pixel positions determined by the representative pixel information. In the present embodiment, for simplicity, the number of pixel positions (pixel groups included in the representative pixel information) used to calculate the amount of disparity is set to be the same as that in the viewing image so that the depth information is obtained at the same resolution as the viewing image. As a method for deriving the correlation, methods such as NCC, SSD, and SAD are used. NCC stands for "normalized cross correlation". SSD stands for "sum of square differences", and SAD stands for "sum of absolute differences".

[0061] The calculated parallax amount (indicated by d) is converted into a defocus amount by using a predetermined conversion coefficient. The defocus amount corresponds to the distance from the imaging element 11 to the focal point of the imaging optical system 10. Here, the predetermined conversion coefficient is represented as K, and the defocus amount is represented as ΔL. The parallax amount d is converted into the defocus amount ΔL by using the following equation (1).

[0062] ΔL=K×d (1)

[0063] The defocus amount ΔL can be converted into the object distance by using a lens formula in terms of geometrical optics expressed in the following equation (2).

[0064] 1 / A+1 / B=1 / F (2)

[0065] In equation (2), A represents the distance from the object plane to the principal point of the imaging optical system 10 (subject distance), B represents the distance from the principal point of the imaging optical system 10 to the image plane, and F represents the focal length of the imaging optical system 10. In equation (2), since the value of B can be derived from the defocus amount ΔL, the distance A can be derived based on the setting of the focal length F at the time of imaging. The depth generation unit 131 generates two-dimensional information having the derived subject distance as a pixel value, and stores depth image data based on the two-dimensional information in the memory 135.

[0066] Reference Figure 4 (C) The processing performed by the display control unit 132 in S404, namely, the processing related to the display of the subject satisfying the measurement accuracy in the size measurement will be described. First, the size measurement accuracy will be described. The subject corresponding to the area existing in the image captured by the central projection optical system has different imaging magnifications depending on the distance from the imaging device (the distance in the z direction). As a result, the subject far from the imaging device is imaged as a smaller area in the image compared to the subject located near the imaging device. Even for the same subject, such a difference in imaging magnification occurs due to the distance difference within the subject, and affects the size measurement accuracy.

[0067] As can be seen from the above description, when the distance to the measurement point for dimension measurement in the z direction is different, the imaging magnification is different, so the dimension measurement accuracy deteriorates. Therefore, in order to suppress the degradation of the dimension measurement accuracy caused by the imaging magnification and measure within the desired dimension measurement error, it is necessary to keep the measurement point within a predetermined distance range in the z direction.

[0068] Here, the real size of the object to be measured is represented by h, and the error caused by the change in the imaging magnification is represented by Δh [%]. The position of the subject (i.e., the distance from the imaging device to the subject) is represented by Z, and the distance error is represented by ΔZ. In this case, the error Δh is represented by the following equation (3).

[0069] Δh[%]=(-ΔZ×100) / (Z+ΔZ) (3)

[0070] In order to maintain the size measurement accuracy within ±Δh [%], the error ΔZ with respect to the object distance Z as a size measurement reference needs to be within the value expressed by the following equation (4).

[0071] ΔZ=Δh×Z / (100+Δh) (4)

[0072] The distance range for satisfying the specified dimension measurement accuracy ±Δh is the range of Z+ΔZ to Z-ΔZ. These values ​​are shown in the following equations (5) and (6).

[0073] Z+ΔZ=(100+2×Δh)Z / (100+Δh) (5)

[0074] Z-ΔZ=(100-2×Δh)Z / (100-Δh) (6)

[0075] The measurement accuracy Δh is acquired by reading a value pre-stored in the storage unit 14 . Alternatively, the user may input a desired measurement accuracy by using the input unit 15 .

[0076] exist Figure 4 In S4041 of (C), a distance range calculation process is performed. The distance Z from the camera to the subject is set to the distance to the focused subject during the image capture. If the desired size measurement accuracy is given, a process of specifying a distance range that satisfies the desired size measurement accuracy is performed by using equations (5) and (6). The subject area in the image within the distance range is specified by using a depth image.

[0077] Subsequently, in S4042, enhancement processing for the viewing image is performed. Data related to the viewing image generated in S402 and stored in the memory 135 is read. Image processing is performed on the read viewing image data so that the subject area within the distance range that satisfies the desired size measurement accuracy specified in S4041 can be identified. As identifiable display methods, there are a method of replacing the subject area within the distance range that satisfies the size measurement accuracy with a subject area of ​​a specific color and displaying the subject area, and a method of displaying the subject area with a specific color sense.

[0078] The display process is performed in S4043. The display unit 16 displays the viewing image processed by S4042 on the screen. The user can recognize the subject area within the distance range that satisfies the desired size measurement accuracy. Figure 5 1 is an explanatory diagram showing an operation of reproducing and displaying an image on the screen of the display unit 16 .

[0079] exist Figure 5 In the viewing image 501 shown in (A), the subject region within the distance range that satisfies the desired size measurement accuracy is displayed in a recognizable manner. Of the subject regions 504 and 505, the subject region 504 indicates a region that can be recognized by changing color.

[0080] exist Figure 4 In S405 of (A), the user specifies a measurement point in the viewing image 502 displayed on the display unit 16. For example, Figure 5 As shown in (B), the user specifies the starting point position on the object whose size is desired to be measured on the touch panel provided on the display unit 16. Point 506 indicates the starting point specified by the user. Figure 5 As shown in (C), the user specifies the end point 507 of the measurement on the viewing image 503. The position specifying unit 133 calculates the coordinate value of the pixel position specified by the user and sends the coordinate value to the size setting unit 134.

[0081] The measurement position can be selected by touching two points on the image with the user's fingers, but the measurement position can also be specified by drawing a trajectory on the image. In the case of the specification method by drawing a trajectory on the image, the trajectory drawn by the user can be acquired and the size of the curve along the trajectory can be measured.

[0082] like Figure 5 As shown in (C), a straight line connecting the start point and end point of the specified dimension measurement or a curve drawn by the user is superimposed on the viewing image and presented to the user. In particular, when the user specifies the measurement position by drawing a trajectory, a process of displaying the trajectory starting from the start point in real time is performed. Figure 5 An arrow 508 in (C) indicates an example in which a start point 506 and an end point 507 of a designated dimension measurement are connected by a straight line and displayed.

[0083] Display control is performed so that the subject area within the distance range that satisfies the desired size measurement accuracy from the reference distance in the depth direction can be identified, so the user can visually recognize the area where the measurement accuracy is guaranteed. The distance range that satisfies the desired size measurement accuracy is a distance range in which the size setting unit 134 can measure the distance with a measurement accuracy equal to or higher than the reference. When specifying a measurement point, the position specifying unit 133 can ensure the measurement accuracy by making it possible to select only pixels in the subject area within the distance range that satisfies the size measurement accuracy displayed in a recognizable manner. In this case, in the case where the position of the selected pixel is in an area outside the distance range that satisfies the size measurement accuracy, the display control unit 132 performs display control to provide a notification that it cannot be selected on the display unit 16.

[0084] exist Figure 4 In S406 of (A), the size setting unit 134 calculates the distance on the image between the measurement points specified in S405, that is, the distance in pixel units. The calculated distance information is temporarily stored in the memory 135, and in S407, the display control unit 132 reads the distance information from the memory 135 and converts the distance information into the distance on the object side (the distance in the three-dimensional space), and the converted distance is calculated by Figure 5 The numerical expression 509 in (C) is displayed on the screen of the display unit 16. The conversion of the distance to the object side can be performed by the size setting unit 134.

[0085] In the description so far, as a display method that enables identification of a subject area within a distance range that satisfies the size measurement accuracy, a method of replacing a subject area with a subject area of ​​a specific color and a method of displaying a subject area with a specific color sense have been described. The display method is not limited to these examples. Figure 6 Other display methods are shown.

[0086] Figure 6 Examples of viewing images 601 to 604 with different display methods are shown. A subject region 605 is a subject region within a distance range that satisfies the size measurement accuracy. A subject region 606 is a subject region other than the subject within the distance range that satisfies the size measurement accuracy.

[0087] Figure 6 (A) shows an example in which a method of displaying the subject area 605 by changing the brightness instead of the color is applied. Figure 6 (B) shows an example of applying a method of changing and displaying the color or brightness of an image area other than the subject area 605 .

[0088] In size measurement, usually, edge-to-edge measurement is performed to measure the size from one end to the other end of the subject area. It is not always necessary to display the entire subject existing within the range that satisfies the desired size measurement accuracy in a recognizable manner. Edges in an image may be detected, and only edges existing in the subject area within the distance range that satisfies the size measurement accuracy may be displayed in a recognizable manner. Figure 6 (C) shows an example of applying a method of highlighting the edge in the subject area 605 in a recognizable manner.

[0089] Specific symbols can be displayed in a superimposed manner. Figure 6 (D) shows an example of applying a method of superimposing and displaying dots on an edge in the subject region 605. As for the display method, any method other than the method described here may be used as long as the user can be notified of the subject region within the distance range satisfying the specified size measurement accuracy.

[0090] According to the present embodiment, an image area corresponding to a subject within a distance range that can be measured with the set size measurement accuracy can be highlighted. More specifically, based on the size measurement accuracy set with the focus distance as a reference, a subject area existing within a specific distance range is extracted by using a depth image and displayed in a recognizable manner. As a result, the user can be assisted in selecting a measurement point in size measurement.

[0091] In the present embodiment, an example has been described in which the imaging element 11 has a photoelectric conversion unit based on the imaging surface phase difference method, and a viewing image and a depth image are acquired. When the present invention is performed, the method of acquiring depth information is not limited thereto. For example, depth information can be acquired based on a plurality of captured images obtained from a plurality of imaging optical systems or imaging devices. Alternatively, depth information can be acquired by using a stereo ranging method using a light irradiation unit and an imaging device, a method using a combination of a time of flight (TOF) method and an imaging device, and the like.

[0092] In this embodiment, an example of processing from executing position designation to executing size setting has been described, but when the present invention is implemented, the position designation unit and the size setting unit are not necessary. If a reference distance and a desired size measurement accuracy are given, the display can be controlled so that a subject within a range that satisfies the desired size measurement accuracy can be identified.

[0093] [Second embodiment]

[0094] Next, a second embodiment of the present invention will be described. In the first embodiment, the following aspect has been described, in which a distance range that satisfies a predetermined size measurement accuracy is calculated based on the focus distance, and the subject within the distance range is displayed in a recognizable manner to assist in specifying a measurement point. The reference distance does not necessarily have to be a distance based on the focus position. Therefore, in this embodiment, the distance corresponding to the measurement point specified by the user is set as the reference distance. For example, the user specifies the measurement point on the image displayed on the display unit 16. A process is performed to obtain the distance corresponding to the specified point from the depth image, and the acquired distance is set as the reference distance. The differences from the first embodiment will be described below, and the detailed description thereof will be omitted by using the used figure marks or symbols for the same matters as the first embodiment. The method of omitting such a description is the same in the embodiments described later.

[0095] Will refer to Figure 7 The flowchart of FIG. 1 describes the size derivation process according to the present embodiment. Figure 4 The processing in (A) is different from the processing in S701 to S704. After S401 to S403, the flow proceeds to S701.

[0096] In S701 , the display control unit 132 controls to read data related to the viewing image generated in S402 from the memory 135 , and displays the viewing image on the screen of the display unit 16 . Figure 8 (A) shows an example of a displayed viewing image 801. Subject areas 803 and 804 correspond to subjects at different distances from the digital camera 100, respectively.

[0097] In S702, the position specifying unit 133 performs processing related to specifying the first measuring point. The processing of prompting the user to specify the first measuring point as a reference for performing dimension measurement is performed. Figure 8 As shown in (A), the user specifies a first measurement point 805 as a desired measurement point in the viewing image 801 displayed on the display unit 16. The following process is performed: the distance on the depth image corresponding to the coordinates of the specified measurement point 805 on the viewing image is acquired. The acquired distance is sent to the process of S703 as the reference distance.

[0098] In S703, in addition to the method of setting the distance range in the first embodiment, the range is also limited by the distance on the image plane (xy plane). Due to the distortion aberration of the imaging optical system 10, the captured image may be distorted. The distortion depends on the viewing angle and therefore differs according to the image height on the image plane. The distortion rate data is pre-stored in the storage unit 14 as a function depending on the image height. By using this data, a process for setting an image height range (distance range in the xy plane) corresponding to the measurement error Δhxy[%] caused by the distortion is performed. When the error caused by the imaging magnification is expressed as Δhz, the area in which the measurement error ultimately falls within Δh[%] satisfies the following expression (7).

[0099] Δh≤Δhz+Δhxy (7)

[0100] According to Figure 4 The same method as the display method described in S404 in (A) is used to perform processing to display the subject area satisfying the size measurement accuracy defined by expression (7) in a recognizable manner. Figure 8 As shown in (B), the image 802 is displayed on the screen of the display unit 16. Based on the distance of the first measurement point 805 specified by the user, the subject area ( Figure 8 (B) The flower region on the right side.) As a result of considering the influence of the imaging magnification and distortion in the object region 804, it is shown that the stem region 806 does not satisfy the specified size measurement accuracy.

[0101] In S704, the position specifying unit 133 performs processing related to the specification of the second measuring point. The processing of prompting the user to specify the second measuring point as the end point is performed. When the user specifies the second measuring point, the processing of acquiring the coordinate information of the specified second measuring point is performed, and the flow enters the processing of S406, and the processing in S406, S407 and S409 is performed.

[0102] In this embodiment, when the user specifies the first measurement point used as a measurement reference, the distance range that satisfies the dimension measurement accuracy is calculated based on the subject distance (the distance from the camera to the subject) at the specified measurement point. Therefore, the dimension can be measured with a higher degree of freedom. When calculating the distance range that satisfies the dimension measurement accuracy, the distance range in each direction (x, y, and z directions) is set by considering not only the influence of the camera magnification but also the error caused by distortion. By extending the distance range that satisfies the dimension measurement accuracy in multiple directions, the distance range that satisfies the dimension measurement accuracy can be limited to a higher accuracy.

[0103] Will refer to Fig. 9 The flowchart of FIG. 1 describes a variation of the second embodiment. Fig. 9 In the second embodiment, the object area within the distance range satisfying the size measurement accuracy is determined with the position of the designated first measurement point as the reference position, but in this variation, the reference point designation process S901 is added.

[0104] In S901 after S701, a reference point is specified by the user. Based on the position of the specified reference point, in S703, the subject area within the distance range that satisfies the size measurement accuracy is displayed in a recognizable manner. Subsequently, in S702 and S704, the processing of specifying the first measurement point and the second measurement point is performed respectively. Thereafter, the size measurement is performed (S406).

[0105] [Third embodiment]

[0106] Will refer to Fig.10 A third embodiment of the present invention is described. In the above-mentioned embodiments and modifications, a case has been described where only the subject area within the distance range satisfying a specific size measurement accuracy is displayed in a recognizable manner. It is not necessary to limit the subject area displayed in a recognizable manner to a single subject area. For example, as a size measurement error, a subject area corresponding to the following multiple non-overlapping ranges is assumed.

[0107] The first subject area with size measurement error less than 1%

[0108] · The second subject area with a size measurement error of 1% or more and less than 3%

[0109] ・A third subject area with a size measurement error of 3% or more and less than 5%

[0110] The display control unit 132 of this embodiment controls to display each of the multiple object regions that meet the same degree of size measurement accuracy in a recognizable manner. In this case, each object region is displayed on the screen, for example, with different colors or brightness, so that the user can recognize each region separately.

[0111] Fig.10An example is shown in which subject regions 1002 and 1003 corresponding to a plurality of subjects are present in a viewing image 1001. In the subject region 1002, the endpoints of the flower petals are set as the measurement reference point 1004. In this case, the size measurement error of the flower region within the subject region 1002 from the measurement reference point 1004 is less than 1%, which corresponds to the first subject region. The first subject region is displayed so as to be identifiable with a first brightness, for example, by changing the brightness. The stem region 1002a is at a different distance from the measurement reference point 1004, and the size measurement error is 1% or more. Assuming that this region is, for example, the second subject region, this region is displayed so as to be identifiable with a second brightness different from the subject region 1002 (flower region).

[0112] Fig.10 The subject region 1003 shown in FIG. 1 has a flower region and a stem region 1003a thereof which are different from the subject region 1002. These regions are greatly different in distance from the measurement reference point 1004, and therefore are displayed separately by using different brightness. Fig.10 In the example of , the display control unit 132 performs display so that the larger the size measurement error, the lower the brightness.

[0113] Since the display is performed by using different brightness, color, symbol, etc. for the areas with different size measurement errors, the user can easily identify each area. The display control unit 132 controls the display of the subject area corresponding to multiple size measurement errors. In this case, the user can switch at any time whether to display the area corresponding to a single size measurement accuracy or the area corresponding to multiple size measurement accuracies. The user can use the input unit 15 to set how many size measurement accuracy ranges to be separated (split) and displayed, and set the input settings of the number and range of separations. Regarding the display of each subject area corresponding to multiple size measurement accuracies, the focus distance can be used as a reference, or the position of the measurement point specified by the user can be used as a reference as in the modified example.

[0114] In this embodiment, multiple object areas corresponding to multiple size measurement precisions can be distinguished and displayed, and the user can identify the object area in more detail. For example, in the case of distinguishing the object area that meets the size measurement precision range, especially in the case where the width of the depth distribution of the measurement object is large, this embodiment is effective.

[0115] [Fourth embodiment]

[0116] Next, a fourth embodiment of the present invention will be described. In the above embodiment, in the case of acquiring distance information, the distance resolution at the time of distance measurement is different depending on the distance. The farther the distance from the camera to the subject, the lower the distance resolution. Since the uncertainty of the distance resolution occurs in the distance measurement, the uncertainty of the camera magnification occurs, and the size measurement accuracy at the calculated distance can be obtained. If the distance measurement error is considered, in addition to the uncertainty of the distance resolution, uncertainty corresponding to the distance measurement error will also occur. Taking this into account, the size measurement accuracy for each calculated distance can be calculated.

[0117] By using the size measurement accuracy calculated for each distance, a distance range having a distance resolution that satisfies the size measurement accuracy specified by the user can be specified. Distance ranges having different size measurement accuracies can also be specified according to the distance resolution.

[0118] In this embodiment, the depth generation unit 131 can display the predetermined subject area in a recognizable manner by using the size measurement accuracy calculated according to the distance resolution in the depth image. The predetermined subject area is a distance range that satisfies the size measurement accuracy, or a subject area corresponding to a plurality of divided distance ranges according to the size measurement accuracy.

[0119] The display control unit 132 acquires the size measurement accuracy calculated from the distance resolution and the distance measurement error in the distance measurement, calculates the subject area satisfying the predetermined size measurement accuracy, and controls the highlighted display. This embodiment is effective when the size measurement is performed by taking the size measurement accuracy into consideration using the distance measurement method.

[0120] [Fifth embodiment]

[0121] Next, a fifth embodiment of the present invention will be described. In this embodiment, effective processing in a case where the object corresponding to the area including the first measurement point specified by the position specifying unit 133 is an object having a substantially planar shape will be described.

[0122] When a subject having a substantially planar shape is imaged facing the camera, the entire surface of the subject is approximately equidistant from the camera. In the above embodiment, the entire surface area of ​​the subject is displayed as the subject area satisfying the predetermined size measurement accuracy.

[0123] On the other hand, when a subject having a substantially planar shape is photographed in a manner not directly facing the camera, only the subject area within the distance range that satisfies the size measurement accuracy is displayed as identifiable for the distance at the designated first measurement point. That is, the size measurement that satisfies the predetermined accuracy can be performed on only a portion of the subject, while the size measurement accuracy can be reduced for other subject areas in the same subject.

[0124] In the case where the subject is a plane that is photographed in a non-directly facing manner and a depth image for the plane can be acquired, a geometric transformation can be performed. That is, the inclined plane can be transformed into a plane facing forward by a geometric transformation. Through geometric transformation, images of planar subjects that are photographed in different sizes according to the distance from the camera are converted into images of planar subjects of the same size at the same distance. For example, in the case where a square subject is photographed in a non-directly facing manner, the subject is not a square (trapezoidal, etc.) on the image, but can be converted into a square by calculating the slope based on the distance information.

[0125] The image processing apparatus of this embodiment includes a geometric transformation unit 136 (refer to Figure 1 ), and determines whether the subject corresponding to the area including the specified first measurement point has a substantially planar shape. In the case where it is determined that the subject has a substantially planar shape, the geometric transformation unit 136 calculates an area that satisfies the size measurement accuracy when the tilt is corrected by geometric transformation. The display control unit 132 performs control for displaying the calculated area in a recognizable manner. In this case, the area corresponding to the entire surface of the planar subject is an area that satisfies the size measurement accuracy, and is displayed to be equally recognizable on the entire surface of the planar subject.

[0126] When it is detected that the subject has a planar shape, the image processing device performs straight line detection and vanishing point detection. By using both the detected plane and the plane based on the depth image, the plane is corrected and a final plane (subject region) is acquired.

[0127] In this embodiment, by geometrically transforming an image in which the subject has a substantially planar shape and is photographed in a manner not directly facing the camera, the image of the subject can be converted into an image facing the face. By performing the geometric transformation, the magnification change caused by the tilt is corrected, and the entire subject region having a substantially planar shape in the captured image satisfies the size measurement accuracy. The subject region is displayed in a recognizable manner.

[0128] According to the above-described embodiment, when measuring the size of a subject in an image, the user can be notified of the subject corresponding to the size measurement accuracy, so that measurement with known size measurement accuracy can be performed. By displaying the subject area corresponding to the size measurement accuracy in a recognizable manner, more stable and high-precision size measurement can be achieved, and user convenience can be improved.

[0129] [Other embodiments]

[0130] The present invention can also be implemented by providing a program that implements one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be implemented by a circuit (e.g., ASIC) that implements one or more functions.

[0131] [reference numerals list]

[0132] 10 Camera Optical System

[0133] 11. Camera Components

[0134] 12 Control Unit

[0135] 13 Image processing device

[0136] 130 Image generation unit

[0137] 131 Depth Generation Unit

[0138] 132 Display control unit

[0139] 133 Position specifying unit

[0140] 134 Dimension setting unit

[0141] 136 Geometric Transformation Unit

Claims

1. An image processing device, comprising: an acquisition unit for acquiring an image captured by the imaging element and information related to a distance distribution in a depth direction corresponding to the captured image; as well as a control unit, configured to control the display of the captured image, wherein the control unit performs control so that, based on a reference distance in the depth direction, a first subject area in the captured image corresponding to a subject within a distance range and a second subject area in the captured image corresponding to a subject not within the distance range are displayed in a manner that can be distinguished from each other, wherein in the distance range, the distance can be measured with a measurement accuracy equal to or higher than a reference, and the reference distance is set to the distance of the subject at a focus position during imaging, The distance to be measured is a distance corresponding to a plurality of measurement points specified in the captured image.

2. An image processing device, comprising: an acquisition unit for acquiring an image captured by the imaging element and information related to a distance distribution in a depth direction corresponding to the captured image; as well as a control unit, configured to control the display of the captured image, The control unit performs control so that, based on a reference distance in the depth direction, a first subject area in the captured image corresponding to a subject within a distance range and a second subject area in the captured image corresponding to a subject not within the distance range are displayed in a manner that can be distinguished from each other, wherein in the distance range, the distance can be measured with a measurement accuracy equal to or higher than a reference, and the reference distance corresponds to a point specified by a user in the captured image and is set as a distance in the information related to the distance distribution.

3. The image processing device according to claim 1 or 2, in, The distance range is determined based on a measurement accuracy associated with a measurement point in the distance measurement or a measurement accuracy associated with distances corresponding to a plurality of measurement points in the distance measurement.

4. The image processing device according to claim 3, in, The measurement accuracy is a measurement accuracy due to a difference in imaging magnification, a measurement accuracy due to distortion aberration of an imaging optical system, or a measurement accuracy designated by using an operation section.

5. The image processing device according to claim 1 or 2, in, Distance measurement is performed by using a plurality of measurement points specified in the captured image and the information on the distance distribution.

6. The image processing device according to claim 1 or 2, in, The distance that is the object of distance measurement is a distance in a three-dimensional space corresponding to a plurality of measurement points specified in the captured image.

7. The image processing device according to claim 1 or 2, in, The control unit performs control to display distances between a plurality of measurement points in the first object area.

8. The image processing device according to claim 1 or 2, in, The distance ranges are a single distance range or a plurality of non-overlapping distance ranges.

9. The image processing device according to claim 1 or 2, in, The control unit performs control to change one or more of color, brightness, and symbol for the first subject area or the second subject area.

10. The image processing device according to claim 1 or 2, in, The control unit performs control to change one or more of color, brightness, and symbol for a plurality of object areas respectively corresponding to the plurality of distance ranges.

11. The image processing device according to claim 1 or 2, in, The control section performs control to detect an edge of the captured image and change one or more of color, brightness, and sign for the edge of the first subject area corresponding to the distance range.

12. The image processing device according to claim 1 or 2, in, The control unit performs control so that a straight line or a curved line connecting a plurality of measurement points designated in the captured image is displayed superimposed on the captured image.

13. The image processing device according to claim 1 or 2, in, As the measurement point in the distance measurement, a measurement point in the first object area can be specified.

14. The image processing device according to claim 1 or 2, in, The control unit performs control to obtain measurement accuracy calculated based on distance resolution and distance measurement error in distance measurement, and to display the first object area in a recognizable manner.

15. The image processing apparatus according to claim 1 or 2, further comprising: a geometric transformation unit, configured to correct the captured image by geometric transformation, When it is determined that the object corresponding to the area including the measurement point specified in the captured image has a planar shape, the geometric transformation unit calculates an area that satisfies the measurement accuracy, and the control unit performs control to display the calculated area in a recognizable manner.

16. A camera device, comprising: The image processing device according to claim 1 or 2; as well as Camera element.

17. The imaging device according to claim 16, in, The imaging element has a plurality of microlenses and a plurality of photoelectric converters respectively corresponding to the microlenses, and acquires information on the distance distribution from a plurality of image signals having different viewpoints outputted from the plurality of photoelectric converters.

18. The imaging device according to claim 16, in, The control unit calculates distances between a plurality of measurement points specified in the captured image, calculates an imaging magnification based on a distance from the imaging device to a subject and a focal length of an imaging optical system, and converts the distances between the plurality of measurement points into distances on the object side by using a pixel size of the imaging device and the imaging magnification.

19. The camera device according to claim 18, further comprising: The display unit is used to display the distances between the plurality of measurement points specified in the captured image.

20. An image processing method, which is performed by an image processing device, the image processing method comprising: An acquisition step for acquiring an image captured by the imaging element and information related to a distance distribution in a depth direction corresponding to the captured image; as well as a control step of controlling the display of the captured image, Wherein, in the control step, control is performed so that, based on a reference distance in the depth direction, a first subject area in the captured image corresponding to a subject within a distance range and a second subject area in the captured image corresponding to a subject not within the distance range are displayed in a manner that can be distinguished from each other, wherein in the distance range, the distance can be measured with a measurement accuracy equal to or higher than a reference, and the reference distance is set to the distance of the subject at the focus position during imaging.

21. An image processing method, which is performed by an image processing device, the image processing method comprising: An acquisition step for acquiring an image captured by the imaging element and information related to a distance distribution in a depth direction corresponding to the captured image; as well as a control step of controlling the display of the captured image, In the control step, control is performed so that, based on a reference distance in the depth direction, a first subject area in the captured image corresponding to a subject within a distance range and a second subject area in the captured image corresponding to a subject not within the distance range are displayed in a manner that can be distinguished from each other, wherein in the distance range, the distance can be measured with a measurement accuracy equal to or higher than a reference, and the reference distance corresponds to a point specified by a user in the captured image and is set to a distance in the information related to the distance distribution.

22. A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 20 or 21.

23. A computer program product, comprising a program for causing a computer to execute the steps of the image processing method according to claim 20 or 21.

Images