Image processing device and method for controlling the image processing device

Abstract

To generate a highly accurate full size image of a subject which exists at a desired position in an image.SOLUTION: An image processing apparatus comprises: acquisition means which acquires an image photographed by imaging means and photographic information at the photography of the image; and calculation means which calculates an object side pixel dimension of a target subject in the image on the basis of the photographic information and the pixel dimension of the imaging means. The acquisition means acquires focus information indicating the focusing state of the subject in the image as the photographic information. The calculation means calculates the object side pixel dimension on the basis of the focus information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method for performing processing related to the dimensions of a subject in an image based on information at the time the image was captured and information on an imaging device used to capture the image. [Background technology]

[0002] There is a technology that uses information from when the image was captured to display the subject in the image at actual size.

[0003] Patent Document 1 discloses a display system and method for displaying an image in actual size on a display device screen. The system proposes a method for calculating and displaying the actual size by acquiring specification information of the display screen on which the image is displayed and information at the time of shooting from an image saved in the Exif (Exchangeable image file format) format. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-78677 A Summary of the Invention [Problem to be solved by the invention]

[0005] The technology described in Patent Document 1 uses information stored in an Exif file to calculate the actual size, but since the focus state is not taken into consideration, there is a problem that it is difficult to calculate the actual size with high accuracy depending on the focus state of the image. In addition, the technology only discloses the use of the information in the form of displaying the dimensions of a single subject in an image in actual size.

[0006] In view of the above problems, the present invention aims to provide an image processing device and an image processing method that solve at least one of the above problems. That is, the present invention aims to provide an image processing device and an image processing method that generate an output image in which the dimensions of a subject present at a desired position in an image can be reproduced with higher accuracy. Another aim is to provide an image processing device and an image processing method that further utilizes the dimensions of the subject. [Means for solving the problem]

[0007] In order to achieve the above object, the image processing device of the present invention includes an acquisition means for acquiring an image captured by an image capture means and image capture information at the time of capturing the image, a calculation means for calculating an object side pixel dimension of a target subject in the image based on the image capture information and a pixel dimension of the image capture means, a control means for calculating at least one number of pixels per unit length from the object side pixel dimension calculated by the calculation means, and for recording the number of pixels per unit length in association with a corresponding image; the acquisition means acquires focusing information indicating a focusing state of a subject in an image as the shooting information, and the calculation means calculates the object-side pixel dimension based on the focusing information. The image processing method of the present invention includes an acquisition step of acquiring an image captured by an imaging means and imaging information at the time of capturing the image, and a calculation step of calculating an object side pixel dimension of a subject in the image based on the imaging information and a pixel dimension of the imaging means. a control step of calculating at least one number of pixels per unit length from the object side pixel dimension calculated in the calculation step, and recording the number of pixels per unit length in association with a corresponding image; In the acquiring step, focusing information indicating a focusing state of a subject in an image is acquired as the shooting information, and in the calculating step, the object side pixel dimension is calculated based on the focusing information.

[0008] In addition, the image processing method of the present invention includes an acquisition step of acquiring an image captured by an imaging means and shooting information at the time of capturing the image, and a calculation step of calculating object-side pixel dimensions of a subject in the image based on the shooting information and pixel dimensions of the imaging means, wherein the acquisition step acquires focusing information indicating a focus state of the subject in the image as the shooting information, and the calculation step calculates the object-side pixel dimensions based on the focusing information. Effect of the Invention

[0009] According to the present invention, it is possible to generate an image in which the dimensions of an object present at a desired position in the image are clear with a high degree of accuracy. In addition, the dimensions of an object present at a desired position in the image are utilized to enable a comparison of dimensions between objects. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a functional configuration of an image processing device according to a first embodiment. [Diagram 2] 4 is a flowchart illustrating a process executed in the first embodiment. [Diagram 3] FIG. 11 is a block diagram showing the functional configuration of an image processing device according to a second embodiment. [Figure 4] 10 is a flowchart illustrating a process executed in the second embodiment. [Diagram 5] FIG. 11 is a diagram illustrating a process executed in the second embodiment. [Figure 6] FIG. 11 is a block diagram showing the functional configuration of an imaging device according to a third embodiment. [Figure 7] 13 is a diagram illustrating an imaging element according to a third embodiment. FIG. [Figure 8] 1A and 1B are diagrams illustrating the principle of distance measurement using an imaging surface phase difference method. [Figure 9] 13 is a flowchart illustrating a process executed in a third embodiment. [Figure 10] FIG. 11 is a block diagram showing the functional configurations of a display device and a printing device according to a third embodiment. [Figure 11] 13 is a flowchart illustrating a process executed in the fourth embodiment. [Figure 12] FIG. 13 is a diagram illustrating a process executed in the fourth embodiment. [Figure 13] FIG. 13 is a block diagram showing the functional configuration of an imaging device according to a fifth embodiment. [Figure 14] 13 is a flowchart illustrating a process executed in the fifth embodiment. [Figure 15] FIG. 13 is a diagram illustrating a reporting method in the fifth embodiment. [Figure 16] 13 is a flowchart illustrating processing executed in the sixth embodiment. [Figure 17] FIG. 23 is a diagram illustrating a display screen associated with the processing executed in the sixth embodiment. [Figure 18] FIG. 23 is a diagram illustrating a display screen associated with the processing executed in the sixth embodiment. [Figure 19] FIG. 23 is a diagram illustrating a case where grid lines are displayed on a display screen in accordance with a process executed in a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [First embodiment] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] In the embodiment described below, an example of an image processing device incorporating the image processing device of the present invention will be described as an example of an image processing device. This image processing device acquires a captured image from an imaging device such as a digital camera, and outputs a life-size image of a subject to an output device such as a display device or a printer. The present invention is applicable to any device that can calculate a life-size image based on the captured image, the capture information corresponding to the captured image, and output device information.

[0013] Configuration of image processing device 100 The configuration of the image processing device 100 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of the image processing device 100.

[0014] The input unit 10 is an interface (I / F) that acquires image information (image data) captured by an imaging device such as the digital camera 1 and shooting information (shooting conditions, image processing parameters, etc.) from the digital camera 1 or another external device.

[0015] The image processing unit 12 performs various image processing such as luminance and color conversion processing, correction processing for defective pixels, shading, noise components, etc., filter processing, image synthesis processing, and various calculations for image processing on image data acquired from the input unit 11, the storage unit 13, or the communication unit 14. The image processing unit 12 also includes a conversion information calculation unit 121 and an image generation unit 122. The conversion information calculation unit 121 executes a calculation to convert, for example, the size of a pixel (pixel dimension) in the image sensor of the digital camera 1 that acquired the image data from the shooting information of the image of the acquired image data to a size in a space at a desired subject distance. The image generation unit 122 generates an output image using the pixel dimension in the subject space calculated by the conversion information calculation unit 121 and device information (output destination information) of a device that displays or prints the subject at actual size. The output image is an image that has been subjected to image conversion so that the final output (display on a display device, print from a printing device) at the output destination device is actual size. Here, the image processing unit 12 can be configured using a logic circuit. As another embodiment, the computer may be configured with a central processing unit (CPU) and a memory that stores a processing program.

[0016] The storage unit 13 is composed of a recording medium such as a memory that stores various information such as image data and parameters input via the input unit 11 or the communication unit 14. The storage unit 13 also stores an output image generated by the image generation unit 122 that has been converted so that the subject appears at actual size when output to the output device, and device information of the digital camera 1 required by the conversion information calculation unit 121.

[0017] The communication unit 14 is a communication I / F that transmits and receives data to and from an external device. In this embodiment, the communication unit 14 communicates with the digital camera 1, the display 2, or the printer 3, and obtains device information of the digital camera 1, the display 2, or the printer 3 required for the conversion information calculation unit 121 and the image generation unit 122.

[0018] The output unit 15 is an I / F that outputs actual size information including the output image generated by the image processing unit 12 to the display 2 or printer 3 as the output destination.

[0019] The control unit 16 has a built-in CPU as a computer, and controls each part of the entire image processing device 100 via a bus based on a computer program stored in a non-volatile memory.

[0020] <<Output image generation process>> 2(a) is used to explain the specific process of output image generation processing, which is executed in the image processing device 100 of this embodiment and which processes an input image to generate an output image for outputting a photographed subject at a life-size size. The process corresponding to the flowchart is realized by the control unit 16 reading out a corresponding processing program stored in a non-volatile memory in the control unit 16, for example, and expanding and executing the program in a volatile memory in the control unit 16, thereby operating the control unit 16 or each unit.

[0021] In S121, the image processing unit 12 acquires image data of a captured image, to which information about the time of shooting is attached, from the digital camera 1 via the input unit 11. The input unit 11 may acquire image data to which similar information about the time of shooting is attached from another external device.

[0022] In S122, the conversion information calculation unit 121 calculates conversion information for converting the dimensions of one pixel in the captured image into dimensions that occupy a space at a desired subject distance. The conversion information calculation process will be described with reference to the flowchart in Fig. 2(b). The process corresponding to the flowchart is realized by the control unit 16 reading out a corresponding processing program stored in a non-volatile memory in the control unit 16, for example, and expanding and executing the program in a volatile memory in the control unit 16, thereby causing the control unit 16 or each unit to operate.

[0023] In S1221, the shooting information when the captured image was shot is acquired from information attached to the captured image, for example, tag data (metadata) saved in Exif format. The shooting information acquired here is, for example, the name of the camera (camera ID), the name of the lens (lens ID), the focal length, the F value, and the subject distance used to shoot the acquired image. Generally, shooting information such as the focal length, F value, and subject distance saved in Eixf format is a value rounded to an integer value, or a discrete value with low resolution. For example, the focal length of the lens is stored with the decimal point rounded, and it is not possible to obtain a value after the decimal point. Also, in an inner focus lens, the focal length varies depending on the focus distance, but the same focal length is stored in the image file even if the focus distance is different. In this way, since the information saved in the tag of Exif format is not an accurate value, when the magnification conversion information is calculated using the acquired value, an error in the actual size of the subject in the final life-size image becomes large. As a countermeasure against such errors, the information on the focus state at the time of shooting in the imaging device is written in the maker note area of ​​Exif, etc., and saved as shooting information. Here, the information on the focusing state is, for example, information on the steps and positions of a motor driven for focusing, the steps and positions of a motor that drives a zoom lens, etc. In S1221, in addition to the above-mentioned information, the information on the focusing state (focus information) is also acquired, and is used to correct the shooting information in the next S1222.

[0024] Furthermore, lens aberration affects the captured image, and there is a possibility that the aberration makes it impossible to obtain a correct actual-size output when generating an output image for actual-size output. If distortion aberration remains, the image is distorted, and the influence of the distortion makes it impossible to obtain a correct actual-size output. Therefore, it is preferable to obtain information on distortion aberration as shooting information. The distortion aberration information is obtained by recording a value according to the state of the lens at the time of shooting in the maker note area of ​​Exif and reading it out in S1221. Alternatively, the distortion aberration information of multiple cameras and lenses can be stored in advance in the storage unit 13 as correction information, and can be obtained by reading it out from the storage unit 13 based on the camera name, lens name, and focal length information at the time of shooting.

[0025] As another type of aberration information, axial chromatic aberration information is also necessary for generating an output image for high-precision life-size output. When axial chromatic aberration is large, the lens focus state differs depending on the subject color even for subjects at the same distance, and as a result, different subject distances are output. Axial chromatic aberration information is required to correct this subject distance error. Axial chromatic aberration information may be recorded and read out in the Exif maker note area of ​​the captured image, like distortion aberration information, or it may be stored in advance in the storage unit 13 as correction information and the corresponding information may be read out based on the camera and lens information.

[0026] In S1222, the control unit 16 executes correction of the acquired shooting information using the shooting information including the acquired focus state information. In S1222, based on the acquired camera name and lens name, correction information for correcting the shooting information of the corresponding model stored in advance in the storage unit 13 is acquired. Correction information of a model not stored in the storage unit 13 may be acquired from the outside via the communication unit 14.

[0027] One of the correction information here is table information of focal states and corresponding focal length values ​​to obtain the amount of variation of focal length corresponding to the focusing state, and the focal length is corrected using the table information. When the focusing state is controlled by dividing it into, for example, 100 states from close to infinity, focal length information corresponding to the 100 different focusing states is stored. Alternatively, the relationship between the focusing state and the corresponding focal length can be stored as a function, and when the focusing state is input, the corresponding focal length can be calculated and obtained. It is also necessary to correct the difference between the focal length and the design value caused by manufacturing. It is possible to reduce the average manufacturing error by using the distribution information of the focal length due to the manufacturing error. The manufacturing error information may be obtained from the image file by measuring the focal length for each individual during manufacturing, storing the information in a storage medium attached to the camera body or lens, and reading it during shooting and recording it in the maker note area of ​​the captured image.

[0028] Another correction information is information for correcting the subject distance. The subject distance at the time of shooting attached to the captured image is generally the distance from the focused subject to the front of the lens, or the distance from the focused subject to the image sensor. Therefore, the definition of the subject distance for each camera is stored in advance in the storage unit 13 as one of the correction information, and the definition of the acquired subject distance is determined based on the acquired camera name. Furthermore, information for converting the acquired subject distance into the distance from the subject to the front principal point or the front focal point is used as the correction information. These correction information can be generated based on the design information of the lens and the design information of the lens barrel. Since the front principal point (principal point position of the optical system) or the front focal point position changes depending on the focusing state, the difference position information of the front principal point or the front focal point corresponding to each focusing state from the reference position (the front of the lens or the image sensor surface) is stored as the correction information in the form of a table or function. Finally, the focal length is corrected based on the acquired subject distance at the time of shooting, the definition of the subject distance of the camera used for shooting, and the difference position information from the reference position to the front principal point or the front focal point position based on the definition.

[0029] Another information for correcting the object distance is the aforementioned axial chromatic aberration information. Due to axial chromatic aberration, the focal position on the image side differs depending on the color even for objects at the same distance, and as a result, the output object distance differs. In order to correct this effect, the amount of axial chromatic aberration at multiple visible light wavelengths (for example, the central wavelength of the color filter of the image) is calculated and stored in advance using lens design information. Furthermore, color information of the focal area on the object is obtained (luminance information of each color component of the corresponding image area). The deviation from the reference image side focal position is calculated from the mixture ratio of each color component and the value of axial chromatic aberration, and the object distance is corrected. The reference image side focal position here is the focal position of the color of the reference object used when calculating the object distance. For example, when a black and white chart is used as a reference, the reference image side focal position can be calculated from the luminance information of the color components and the axial chromatic aberration information.

[0030] In S1223, the magnification at the time of shooting is calculated based on the corrected focal length and subject distance. If the distance from the subject to the front focal point is the subject distance x and the focal length is f, the magnification m can be calculated by the following formula (1). m=x / f formula (1)

[0031] The magnification here is set as an enlargement ratio for converting the subject on the imaging plane into the actual size on the object side.

[0032] If the subject distance is defined as the distance from the subject to the front principal point (subject distance S), the distance S' from the rear principal point to the image plane can be calculated using the subject distance S and the focal length x, and the magnification m can be found using the following equation (2). m=S′ / S Equation (2)

[0033] Furthermore, in S1223, the control unit 16 acquires the pixel dimension p (pixel pitch, interval between pixels) of the imaging element used for shooting. By multiplying the pixel dimension p by the magnification m, it is possible to calculate the size of one pixel of the imaging element at the subject distance (object side pixel dimension. How long and how large an object covering one pixel will be in real space), and it is possible to obtain the actual size of the object from the number of pixels covered by the object.

[0034] Here, the case where the magnification is calculated from the corrected focal length and subject distance has been described, but it is also possible to calculate and store highly accurate magnification information in advance from the lens design information. In that case, a table corresponding to information such as the step and position of the motor driven for focusing and the step and position of the motor driven for the zoom lens is stored, and the magnification according to the shooting situation is calculated from these values. The table of magnification information does not need to be stored corresponding to all values ​​that the motor can take, and discrete values ​​can be stored and values ​​between the discrete values ​​can be calculated by interpolation. Also, function fitting may be performed on the magnification information and the motor value, and the coefficients may be stored.

[0035] Returning to the flowchart of Fig. 2(a), in S1213, the control unit 16 causes the image generation unit 122 to generate an output image for actual size output in accordance with the device to which the image is to be output. The output image generation process will be described with reference to the flowchart of Fig. 2(c).

[0036] In S1231, information on the output destination of the output image is obtained. For example, if the output image is to be output as a printout by the printer 3, information on the size of the paper to be output is obtained. For example, if the output image is to be displayed on the display 2, the display screen size and display screen resolution of the display 2 are obtained. Here, the display screen resolution indicates the number of pixels displayed on the display screen.

[0037] In S1232, the image generating unit 122 generates an output image according to the output destination using the acquired output destination information and the object side pixel dimensions. In the case where the output is a printed matter output from the printer 3 or the like, it is possible to calculate the number of pixels in the vertical and horizontal directions of the captured image relative to the paper dimensions by dividing the paper dimensions by the object side pixel dimensions. This number of pixels is set as the number of pixels of the output image. If the number of pixels of the captured image is greater than the calculated number of pixels of the output image, the output image may be generated by selecting and extracting an area of ​​the required number of pixels of the output image from the captured image. On the other hand, if the number of pixels of the captured image is less than the number of pixels set as the calculated number of pixels of the output image, the number of pixels of the output image is matched by adding margin pixels to the captured image, and the output image is generated. When extracting an area in an image, it is optimal to select an area that fits the focused subject, and moreover, it is preferable to select an area that does not cause margin addition as much as possible.

[0038] On the other hand, if the output image is to be output to a display device such as the display 2, the display screen dimensions of the display 2 and the object side pixel dimensions of the imaging element are used to calculate the number of vertical and horizontal pixels in the captured image required for display, just as in printing. Next, the required area is extracted from the captured image according to the calculated number of pixels. This process matches the display screen dimensions with the image dimensions of the extracted captured image at the subject distance. Next, a conversion magnification is calculated from the calculated number of pixels and the display screen resolution, and a scaling process is performed on the extracted captured image so that the number of pixels of the extracted captured image matches the display screen resolution, thereby generating an output image for actual size output on the display 2.

[0039] In S124, image processing unit 12 outputs the generated output image to the destination device via output unit 15, and ends the output image generation process. At this time, the generated output image may be recorded in storage unit 13 together with destination device information such as print paper size, display screen size, and display screen resolution.

[0040] As described above, according to this embodiment, by using shooting information attached to the captured image and correction information that enhances the accuracy of the shooting information, it is possible to generate an output image in which the dimensions of a subject located at a desired position in the image can be reproduced with higher accuracy.

[0041] [Second embodiment] In the first embodiment, the information for converting an image to output a subject in a captured image at actual size is calculated with high accuracy, and an output image for outputting at actual size is generated by acquiring output device information. When the information for outputting a subject at actual size is calculated, it is possible to output the subject so that the actual size of the subject can be recognized by comparison with other objects without necessarily generating an output image for outputting at actual size. Therefore, in this embodiment, an image processing device is described that inputs a plurality of captured images, calculates actual size conversion information for each image, and outputs an image that allows a relative dimensional comparison taking actual size into consideration. Below, differences from the first embodiment are described, and details of matters similar to those in the first embodiment are omitted by using already used symbols and symbols. Such a method of omitting explanations is similar to the embodiments described later.

[0042] The configuration of the image processing device 100 in this embodiment will be described with reference to Fig. 3. This embodiment is characterized in that the image processing unit 22 has a relative image generating unit 222. The configuration other than the image processing unit 22 is similar to that of the first embodiment. Also, the relative image generating unit 222 may be provided in addition to the output image generating unit 122 of the first embodiment, so that the aspects of the first embodiment can also be implemented.

[0043] <<Relative comparison image generation process>> A specific relative image generation process for generating a relative image, which is executed in the image processing device 100 of this embodiment, will be described with reference to the flowchart in Fig. 4(a). The process corresponding to the flowchart is realized by the control unit 16 reading out a corresponding processing program stored in a non-volatile memory in the control unit 16, for example, and expanding and executing the program in a volatile memory in the control unit 16, thereby operating the control unit 16 or each unit.

[0044] In S221, the image processing unit 22 acquires, via the input unit 11, a plurality of captured images to which information about the time of capture is attached, from the imaging device 1 or an external storage device.

[0045] In S222, the conversion information calculation unit 121 calculates conversion information for each captured image to calculate the size of one pixel in the captured image in a space at a desired subject distance, as in the first embodiment.

[0046] In S223, the relative image generating unit 222 generates relative images that can be relatively compared for a plurality of subjects in the captured image, taking into account their actual sizes. The relative image generating process will be described with reference to the flowchart of FIG.

[0047] In S2231, the control unit 16 converts each of the input multiple captured images so that the pixel dimensions (object side pixel dimensions) at the object distance of the multiple captured images are equal. First, a captured image (reference image) that serves as a reference is selected from the multiple captured images input as shown in Figs. 5(a) to (c). The reference image may be any image, or may be selected by the user when inputting the image. If the relative image that is the final output is to have a high resolution, the captured image with the smallest object side pixel dimensions calculated in S222 is selected. Conversely, if a low-resolution image is required, the captured image with the largest calculated object side pixel dimensions is selected. After the reference image is determined, the enlargement / reduction ratio Ri of the comparison image for making the pixel dimensions of the comparison image the same as those of the reference image is calculated from the object side pixel dimensions Ls of the reference image and the object side pixel dimensions Lr of the other captured images (comparison images) using the following formula (3). Ri=Lr / Ls Equation (3)

[0048] A scaling ratio Ri is calculated for each comparison image, and a scaling process is performed for each comparison image according to the scaling ratio Ri. As a result, the object pixel dimensions of all the scaled input photographed images (the processed images are called scaled photographed images) become the same as the object pixel dimensions of the reference image, as shown in Figures 5(d) to (f).

[0049] In S2232, the control unit 16 recognizes the subject in each of the enlarged / reduced photographed images, and executes extraction of the subject area of ​​the specific subject. As an example of subject recognition and area specification, person recognition in a person image as shown in FIG. 5 will be described. Person recognition utilizes machine learning, and person detection and area specification are performed using the learning result. When multiple people are detected in one enlarged / reduced photographed image as shown in FIG. 5(d), the same person appearing in other enlarged / reduced photographed images is identified using face recognition technology, and area extraction of the identified person is executed. This process makes it possible to compare dimensional changes of the same person over time. Also, not limited to this embodiment, when multiple input images are acquired in advance in S221, an image in which a desired subject is recognized (an image in which the area of ​​the subject is further extracted, etc.) may be acquired.

[0050] It is preferable to extract the area along the outer edge of the subject, but depending on the application, it may be desirable to simultaneously extract background information from the time of shooting. In that case, it is preferable to extract a rectangular or elliptical area that approximately circumscribes the selected subject.

[0051] In addition, although the present embodiment has been described with reference to a person as an example of the subject, this can be performed for any subject, and it is also possible to perform relative comparison of the dimensions of different subjects. Furthermore, the selection and extraction of subjects from the enlarged / reduced captured image does not necessarily have to be automatic, and the user may be able to select and extract the subjects.

[0052] Here, the order of the processes of S2231 and S2232 can be reversed.

[0053] In S2233, the subjects extracted from each of the enlarged / reduced photographed images are composited into one image to generate a relative image as shown in FIG. 5(g). It is preferable to adjust the size of the largest subject so that the entire image of the largest subject fits within the composite image, and composite the other subjects with correct relative dimensions based on the largest subject. It is also possible to set a reference subject, adjust the size so that the entire image of the reference subject fits within the composite image, and composite the other subjects with correct relative dimensions. It is preferable to arrange the subjects in the image in order of size in the left-right or up-down direction, or in order of the photographed date and time. Note that photographed date and time information can be used by acquiring it when acquiring photographed information in S1221.

[0054] Here, we have explained the case where multiple subjects are selected, area extraction is performed, and the subjects are arranged to match their relative size based on the reference image, but it is also possible to select a background image and set that image as the reference image. By enlarging or reducing the selected subjects so that their relative size matches the actual size of the background image, it becomes possible to compare the difference in size between the background and the subjects.

[0055] In addition, it is not necessary to arrange the subjects automatically, and the user may arrange them as in S2232. Furthermore, when generating a composite image that allows relative comparison interactively with the user, it is preferable to generate and display the composite image while detecting the largest subject at any time each time the user selects a subject and performing a scaling process so that the largest subject does not protrude from the image.

[0056] If the background image has different object pixel dimensions or distance information depending on the position on the screen, the relative size of the subject may be changed depending on the position on the background image where the user places the subject. For example, if the background image has depth, the subject will be displayed relatively large if placed in the foreground according to the distance, and relatively small if placed in the distance.

[0057] In S224, the image processing unit 22 outputs the generated relative image to an output destination device via the output unit 15 and ends the relative image generation process, or stores the generated relative image in the storage unit 13 and ends the relative image generation process.

[0058] According to this embodiment, conversion information to actual size is calculated from multiple captured images, and an image is generated in which the subjects in the multiple captured images are enlarged or reduced based on the conversion information, thereby generating an image that allows for relative dimensional comparison taking into account the actual size.

[0059] [Third embodiment] In the first and second embodiments, the conversion information calculation process and the output image generation process or the relative image generation process for actual size output are executed by the same image processing device. The conversion information calculation process can be incorporated and executed as part of an imaging device, and the actual size image generation process and the relative image generation process can be incorporated and executed in an output device such as a printing device or a display device. Therefore, in this embodiment, an image processing device in which the conversion information calculation process, the actual size image generation process, and the relative image generation process are separated and incorporated into the imaging device and the output device will be described.

[0060] <Imaging device> The configuration of the digital camera 300 will be described with reference to Fig. 6. As shown in Fig. 6, the conversion information calculation processing section is incorporated in the image processing section 33 in this embodiment.

[0061] The imaging optical system 30 is composed of a lens unit included in the digital camera 300 or a lens device that can be attached to the camera body, and forms an optical image of a subject on the imaging element 31. The imaging optical system 30 is composed of a plurality of lenses arranged in the direction of the optical axis 30a, and has an exit pupil 30b at a position a predetermined distance away from the imaging element 31. In this specification, the direction parallel to the optical axis 30a is defined as the z direction (depth direction). In other words, the depth direction is the direction in which a subject exists in real space when the position of the digital camera 300 is used as a reference. In addition, the direction perpendicular to the optical axis 30a and parallel to the horizontal direction of the imaging element 31 is defined as the x direction, and the direction perpendicular to the optical axis 30a and parallel to the vertical direction of the imaging element 31 is defined as the y direction.

[0062] The imaging element 31 is, for example, a CCD (charge-coupled device) type image sensor or a CMOS (complementary metal-oxide semiconductor) type image sensor. The imaging element 31 performs photoelectric conversion on the subject image formed on the imaging surface via the imaging optical system 30, and outputs an image signal related to the subject image. The imaging element 31 of this embodiment has a function of outputting a signal capable of distance measurement by an imaging surface phase difference method as described later, and outputs a parallax signal for generating distance information indicating the distance from the imaging device to the subject (subject distance) in addition to the captured image.

[0063] The control unit 32 includes a CPU (Central Processing Unit), a microprocessor, etc., and controls the operation of each component included in the digital camera 100. For example, the control unit 32 performs autofocus (AF: automatic focus adjustment) during imaging, changes the focus position, changes the F-number (aperture value), captures images, etc. The control unit 32 also controls the image processing unit 33, the storage unit 34, the input unit 35, the display unit 36, and the communication unit 37.

[0064] The image processing unit 33 executes various image processing operations of the digital camera 100. The image processing unit 33 has an image generation unit 330, a depth generation unit 331, and a conversion information calculation unit 332. The image processing unit 33 has a memory used as a working area for image processing. The image processing unit 33 can be configured using a logic circuit, or can be configured from a CPU and a memory that stores a calculation processing program.

[0065] The image generation unit 330 performs various signal processing such as noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, and color correction of the image signal output from the imaging element 31. The image data (captured image) output from the image generation unit 330 is accumulated in a memory or storage unit 34, and is used by the control unit 32 to display the image on the display unit 36 ​​or to output to an external device via the communication unit 37.

[0066] The depth image generating unit 331 generates a depth image (depth distribution information) that represents the distribution of depth information, based on a signal obtained in relation to a distance measurement image signal obtained from the imaging element 31, which will be described later. Here, the depth image is two-dimensional information in which the value stored in each pixel is the subject distance of a subject present in the area of ​​the captured image corresponding to the pixel. The conversion information calculation unit 332 uses the control information at the time of shooting, the information of the image sensor 31, and the correction information generated from the design information of the imaging optical system 30 to calculate conversion information for calculating the dimensions of one pixel in the captured image in space at the desired subject distance.

[0067] The memory unit 34 is a non-volatile recording medium that records the captured image data, correction information required by the conversion information calculation unit 332, intermediate data generated during the operation of each block, parameters referenced in the operation of the image processing unit 33 and the digital camera 300, etc.

[0068] The memory unit 34 may be any type of recording medium that can read and write data at high speed and has a large capacity, as long as it can guarantee the processing performance acceptable for realizing the processing, and a flash memory, for example, is preferable.

[0069] The input unit 35 is a user interface that detects operation inputs such as dials, buttons, switches, and touch panels that input information and change settings to the digital camera 300. When the input unit 35 detects an operation input, it outputs a corresponding control signal to the control unit 32.

[0070] The display unit 36 ​​is, for example, a display device such as a liquid crystal display or an organic EL display. The display unit 36 ​​is used to check the composition during shooting by displaying the captured image through the display unit 32, and to notify various setting screens and message information under the control of the control unit 32. In addition, by providing a touch panel as the input unit 35 integrally with the display surface of the display unit 26, it is possible to provide both a display function and an input function.

[0071] The communication unit 37 is a communication interface provided in the digital camera 300 and realizes sending and receiving information to and from the outside. The communication unit 37 may be configured to be capable of sending the obtained captured image, depth information, dimensions, coordinate information, dimensional measurement accuracy, and the like to another device.

[0072] Configuration of the image sensor Next, an example of the configuration of the above-mentioned imaging element 31 will be described with reference to FIG.

[0073] As shown in Fig. 7(a), the image sensor 31 is configured by connecting and arranging a plurality of pixel groups 310 in 2 rows x 2 columns to which different color filters are applied. As shown in the enlarged view, the pixel groups 310 are arranged with red (R), green (G), and blue (B) color filters, and each pixel (photoelectric conversion element) outputs an image signal indicating color information of R, G, or B. Note that, as an example, in this embodiment, the color filters are described as being distributed as shown in the figure, but it will be easily understood that the implementation of the present invention is not limited to this.

[0074] In order to realize a ranging function of an imaging surface phase difference ranging method, the imaging element 31 of this embodiment has one pixel (photoelectric conversion element) configured with a plurality of photoelectric conversion units lined up in the II' cross section of Fig. 7(a) in the horizontal direction of the imaging element 31. More specifically, as shown in Fig. 7(b), each pixel is configured with a light guide layer 313 including a microlens 311 and a color filter 312, and a light receiving layer 314 including a first photoelectric conversion unit 315 and a second photoelectric conversion unit 316.

[0075] In the light guide layer 313, the microlenses 311 are configured to efficiently guide the light flux incident on the pixels to the first photoelectric conversion section 315 and the second photoelectric conversion section 316. The color filters 312 transmit light of a predetermined wavelength band, and only transmit light of any one of the above-mentioned R, G, and B wavelength bands and guide it to the first photoelectric conversion section 315 and the second photoelectric conversion section 316 in the subsequent stage.

[0076] The light receiving layer 314 is provided with two photoelectric conversion units (a first photoelectric conversion unit 315 and a second photoelectric conversion unit 316) that convert the received light into an analog image signal, and two types of signals output from these two photoelectric conversion units are used for distance measurement. That is, each pixel of the image sensor 31 has two photoelectric conversion units similarly arranged in the horizontal direction, and an image signal composed of a signal output from the first photoelectric conversion unit 315 of all pixels and an image signal composed of a signal output from the second photoelectric conversion unit 316 are used. In other words, the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 each partially receive a light beam that enters the pixel through the microlens 311. Therefore, the two types of image signals finally obtained are a group of pupil division images related to light beams that have passed through different regions of the exit pupil of the image pickup optical system 30. Here, the composite image signal photoelectrically converted by the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 in each pixel is equivalent to the image signal (for viewing) output from the single photoelectric conversion unit in an embodiment in which only one photoelectric conversion unit is provided in the pixel.

[0077] With such a structure, the image sensor 31 of this embodiment is capable of outputting an image signal for viewing and an image signal for distance measurement (two types of pupil division images). Note that, in this embodiment, it is described that all pixels of the image sensor 31 are provided with two photoelectric conversion units and are configured to be able to output high-density depth information, but the present invention is not limited to this, and a configuration in which only some pixels are provided with multiple photoelectric conversion units may be used. Also, in an aspect in which only one photoelectric conversion unit is provided in a pixel, a configuration in which photoelectric conversion units that receive light beams that have passed through different regions of the exit pupil are provided in a part of the image sensor 31 and in different pixels may be used.

[0078] <Distance measurement principle of imaging plane phase difference distance measurement method> Here, the principle of calculating the subject distance based on the group of pupil division images output from the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316, which is performed in the digital camera 300 of this embodiment, will be described with reference to FIG. 8.

[0079] Fig. 8(a) is a schematic diagram showing the exit pupil 30b of the imaging optical system 30 and a light beam received by a first photoelectric conversion unit 315 of a pixel in the imaging element 31. Fig. 8(b) is a schematic diagram showing a light beam received by a second photoelectric conversion unit 316 in a similar manner.

[0080] The microlens 311 shown in Figures 8(a) and (b) is disposed so that the exit pupil 30b and the light receiving layer 314 are in an optically conjugate relationship. A light beam passing through the exit pupil 30b of the imaging optical system 30 is collected by the microlens 311 and guided to the first photoelectric conversion unit 315 or the second photoelectric conversion unit 316. At this time, the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 mainly receive light beams that have passed through different pupil regions, as shown in Figures 8(a) and (b), respectively. The first photoelectric conversion unit 315 receives a light beam that has passed through the first pupil region 330, and the second photoelectric conversion unit 316 receives a light beam that has passed through the second pupil region 330.

[0081] The multiple first photoelectric conversion units 315 included in the image sensor 31 mainly receive the light beam that has passed through the first pupil region 320, and output a first image signal. At the same time, the multiple second photoelectric conversion units 316 included in the image sensor 31 mainly receive the light beam that has passed through the second pupil region 330, and output a second image signal. From the first image signal, it is possible to obtain the intensity distribution of the image formed on the image sensor 31 by the light beam that has passed through the first pupil region 320. From the second image signal, it is possible to obtain the intensity distribution of the image formed on the image sensor 31 by the light beam that has passed through the second pupil region 330.

[0082] The relative positional shift amount (so-called parallax amount) between the first image signal and the second image signal is a value according to the defocus amount. The relationship between the parallax amount and the defocus amount will be described with reference to Figs. 8(c), (d), and (e). Figs. 8(c), (d), and (e) are schematic diagrams illustrating the image sensor 31 and the image pickup optical system 30 of this embodiment. Reference numeral 321 in the diagram indicates a first light flux passing through the first pupil region 320, and reference numeral 331 indicates a second light flux passing through the second pupil region 330.

[0083] FIG. 8(c) shows a state at the time of focusing, in which the first light beam 321 and the second light beam 331 are converged on the image sensor 31. At this time, the amount of parallax between the first image signal formed by the first light beam 321 and the second image signal formed by the second light beam 331 is 0. FIG. 8(d) shows a state in which the image side is defocused in the negative direction of the z axis. At this time, the amount of parallax between the first image signal formed by the first light beam and the second image signal formed by the second light beam is not 0, but has a negative value. FIG. 8(e) shows a state in which the image side is defocused in the positive direction of the z axis. At this time, the amount of parallax between the first image signal formed by the first light beam and the second image signal formed by the second light beam has a positive value. From a comparison between FIG. 8(d) and FIG. 8(e), it can be seen that the direction of positional deviation is reversed depending on whether the amount of defocus is positive or negative. It can also be seen that positional deviation occurs according to the imaging relationship (geometric relationship) of the imaging optical system depending on the amount of defocus. The amount of parallax, which is the positional shift between the first image signal and the second image signal, can be detected by a region-based matching method, which will be described later.

[0084] <Image generation and conversion information calculation process> The image generation and conversion information calculation process for a photographed subject, which is executed in the digital camera 300 of this embodiment having such a configuration, will be described in detail with reference to the flowchart in FIG. 9(a).

[0085] In S331, the control unit 32 performs processing to perform shooting with shooting settings such as the set focus position, aperture, exposure time, etc. More specifically, the control unit 32 controls the image sensor 31 to shoot and transmit the obtained shot image to the image processing unit 33 and store it in the memory. Here, the shot image is assumed to be of two types: an image signal S1 composed of a signal output from the first photoelectric conversion unit 315 of the image sensor 31, and an image signal S2 composed of a signal output from the second photoelectric conversion unit 316. In addition, two types of signals may be output from the image sensor 31: a signal obtained by mixing (adding) the signals output from the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 by, for example, floating diffusion, and a signal (S1 or S2) output from either of the two photoelectric conversion units.

[0086] In S332, the image processing unit 33 generates an image for viewing from the obtained captured image. More specifically, the image generating unit 330 of the image processing unit 33 first generates one Bayer array image by adding the pixel values ​​of each pixel of the image signal S1 and the image signal S2. The image generating unit 330 performs demosaicing processing of the images of each color of R, G, and B for the Bayer array image to generate an image for viewing. Note that the demosaicing processing is performed according to the color filters arranged on the imaging element, and any method may be used as the demosaicing method. In addition, the image generating unit 330 performs processing such as noise removal, luminance signal conversion, aberration correction, white balance adjustment, and color correction, and generates a final image for viewing and stores it in memory.

[0087] In S333, the image processing unit 33 generates a depth image (depth distribution information) from the obtained photographed image. The depth image is generated by the depth generation unit 331. The process of generating the depth image will be described below with reference to the flowchart in FIG. 9(b).

[0088] In S3331, the depth generation unit 331 performs light amount correction processing on the image signal S1 and the image signal S2. At the peripheral angle of view of the imaging optical system 30, the light amount balance is lost between the image signal S1 and the image signal S2 due to the difference in shape between the first pupil region 320 and the second pupil region 330 caused by vignetting. Therefore, in this step, the depth generation unit 331 performs light amount correction on the image signal S1 and the image signal S2 using, for example, a light amount correction value stored in advance in a memory.

[0089] In S3332, the depth generation unit 331 performs a process to reduce noise generated during conversion in the image sensor 31. Specifically, the depth generation unit 331 achieves noise reduction by applying a filter process to the image signal S1 and the image signal S2. In general, the higher the spatial frequency is, the lower the S / N ratio is, and the more noise components there are. Therefore, the depth generation unit 331 performs a process to apply a low-pass filter whose pass rate is reduced as the spatial frequency is higher. Note that the light quantity correction in S3331 may not produce a favorable result due to manufacturing errors in the imaging optical system 30, so it is preferable that the depth generation unit 331 applies a band-pass filter that blocks direct current components and has a low pass rate for high-frequency components.

[0090] In S3333, the depth generation unit 331 calculates the amount of parallax between the image signals S1 and S2 (both signals together are a phase difference signal). Specifically, the depth generation unit 331 sets, in the image signal S1, a point of interest corresponding to the representative pixel information and a matching area centered on the point of interest. The matching area may be, for example, a rectangular area such as a square area with one side having a predetermined length centered on the point of interest. Next, the depth generation unit 331 sets a reference point in the image signal S2 and sets a reference area centered on the reference point. The reference area has the same size and shape as the above-mentioned matching area. The depth generation unit 331 calculates the degree of correlation between the image included in the matching area of ​​the image signal S1 and the image included in the reference area of ​​the image signal S2 while sequentially moving the reference point, and identifies the reference point with the highest degree of correlation as the corresponding point in the image signal S2 corresponding to the point of interest. The relative positional shift amount between the corresponding point identified in this way and the point of interest becomes the amount of parallax at the point of interest.

[0091] The depth generation unit 331 calculates the amount of parallax at a plurality of pixel positions determined by the representative pixel information by sequentially changing the attention point in this manner. For simplicity, in this embodiment, in order to obtain depth information at the same resolution as the image for viewing, it is assumed that the pixel positions (pixel groups included in the representative pixel information) at which the amount of parallax is calculated are set to be the same in number as the image for viewing. Note that, as a method for calculating the degree of correlation, methods such as NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), and SAD (Sum of Absolute Difference) may be used.

[0092] In addition, the calculated parallax amount can be converted into a defocus amount, which is the distance from the image sensor 31 to the focal point of the imaging optical system 30, by using a predetermined conversion coefficient. Here, if the predetermined conversion coefficient K and the defocus amount ΔL are used, the parallax amount d is expressed as follows: ΔL=K×d Equation (4) Here, it is assumed that the conversion coefficient K is set for each region based on information including the aperture value, the exit pupil distance, the image height at the image sensor 31, and the like.

[0093] The depth generation unit 331 forms two-dimensional information with the calculated defocus amount as pixel values, and stores the information in memory as a depth image. The depth generation unit 331 may also store the distribution of the parallax amount in the calculated parallax state as a depth image in memory. In this case, when the depth image is used for dimension calculation later, it may be converted into a defocus amount. That is, the depth distribution information may be any one of distribution information of the parallax amount indicated by the phase difference signal corresponding to the image up to the subject, distribution information of the defocus amount of the subject in the image, and distribution information of the subject distance up to the subject in the image.

[0094] In step S334, the conversion information calculation unit 332 performs a process of calculating conversion information for converting the dimensions of one pixel in the captured image into dimensions that occupy a space at a desired subject distance. Here, the process related to the measurement subject display will be described with reference to the flowchart in FIG. 9(c).

[0095] In S3341, information such as the lens name, focal length, subject distance, and focusing state information is obtained from the control unit 32. The focusing state information is, for example, information such as the number of pulses of the motor that drives the focusing lens for focusing. In addition, distortion aberration information and axial chromatic aberration information corresponding to the lens name are obtained from the storage unit 34. If the imaging optical system 30 is provided with a separate storage unit and the aberration information is stored in the storage unit, the aberration information may be obtained from the storage unit of the imaging optical system 32 via the control unit 32.

[0096] In S3342, the focal length and subject distance are corrected as described in the first embodiment using the acquired shooting information, focus state information, and various aberration information.

[0097] In S3343, the defocus amount ΔL is calculated based on the corrected focal length and the defocus amount calculated by the depth image generation process using the lens formula in geometric optics, 1 / A+1 / B=1 / F formula (5) Here, A is the distance from the object surface to the principal point of the imaging optical system 30 (object distance), B is the distance from the principal point of the imaging optical system 30 to the image surface, and F is the focal length of the imaging optical system 30. That is, in the lens formula, the value of B can be calculated from the defocus amount ΔL, so that the distance A from the object surface to the object surface can be calculated based on the focal length setting at the time of shooting.

[0098] Furthermore, the magnification m at the time of shooting at each pixel can be calculated from the distance A and the distance B at each pixel position of the captured image.

[0099] In S3343, by taking the product of the pixel dimension p of the imaging element used for shooting and the magnification m at any pixel position, the object side pixel dimension at any pixel position can be calculated, and the actual size can be obtained. By calculating the magnification at all pixel positions, it is possible to obtain the object side pixel dimension at all pixel positions (object side pixel dimension image). The object side pixel dimension image is stored as conversion information together with the captured image in the storage unit 34, and the process ends. Instead of the object side pixel dimension image, magnification information at all pixel positions (magnification information image) can also be stored as conversion information together with the captured image in the storage unit 34.

[0100] Here, when the digital camera 300 is used, the conversion information can be calculated in a plurality of regions, and as described above, it is possible to obtain a conversion information image in which the conversion information is calculated for all pixels and stored as pixel values. However, if it is desired to reduce the capacity of the captured image, it is not necessary to store the conversion information image, and a conversion information image having fewer pixels than the captured image may be obtained by reducing or thinning it out. It is also possible to store only the conversion information for one or a plurality of focus regions. Furthermore, it is also possible to perform object detection such as face detection and store a representative value (for example, an average value) of the conversion information in the detected object region.

[0101] In this embodiment, the image sensor 31 has a photoelectric conversion element of an image plane phase difference distance measuring method, and is described as being capable of acquiring an image for viewing and a depth image, but the acquisition of distance information is not limited to this in the practice of the present invention. The distance information may be acquired, for example, by a stereo distance measuring method based on a plurality of captured images obtained from a binocular image sensor or a plurality of different image sensors. Alternatively, the distance information may be acquired, for example, by a stereo distance measuring method using a light irradiation unit and an image sensor, or a method that combines a TOF (Time of Flight) method and an image sensor.

[0102] <Output device> Next, a configuration in which the output image generation process and the relative image generation process are incorporated into an output device such as a printing device or a display device will be described with reference to Fig. 10. Fig. 10(a) is a block diagram showing the functional configuration of a display device 301 in which the output image generation process and the relative image generation process are incorporated. Fig. 10(b) is a block diagram showing the functional configuration of a printing device 302 in which the output image generation process and the relative image generation process are incorporated.

[0103] The display device 301 or the printing device 302 acquires the captured image and object side pixel dimension information from the digital camera 300 via the input unit 11, and sends the captured image and the object side pixel dimension information to the image processing unit 33'. The image processing unit 33' includes an image generation unit 333 and a relative image generation unit 334, and one of the processes is selected and executed. The processes executed by the image generation unit 333 and the relative image generation unit 334 are the same as the output image generation process and the relative image generation process described in the first and second embodiments. The generated output image or relative image is sent to the display unit 36 ​​and displayed in the case of the display device 301, and is sent to the printing unit 39 and displayed in the case of the printing device 302. The generated output image or relative image can be stored in the storage unit 34, and can also be transmitted to the outside via the communication unit 37.

[0104] As described above, according to the image processing device of this embodiment, it is possible to calculate highly accurate object-side pixel dimensions within the imaging device, and there is no need to output information such as the focus state of the imaging device to the outside. On the other hand, there is no need for the display device and the printing device to hold correction information for calculating highly accurate object-side pixel dimensions from the shooting information.

[0105] Furthermore, by incorporating it into an imaging device capable of imaging surface phase difference ranging, it is possible to generate an output image of any subject other than the focused subject that has been image converted so that it appears at actual size when output to the destination device.

[0106] [Modification of the third embodiment] In this embodiment, the imaging element 31 has a photoelectric conversion element so as to output a phase difference signal and perform imaging surface phase difference ranging, and a depth image (depth distribution information) can be acquired in multiple regions of a captured image. However, the present invention is not limited to this, and the conversion information calculation process can be incorporated even in a digital camera that calculates the subject distance of one in-focus region or a digital camera that calculates the subject distance to multiple in-focus regions. In this way, when the subject distance that can be calculated is only one point, it is preferable to obtain the number of pixels per unit length from the calculated object side pixel dimension information and write it in the Exif format information. The flow of this process will be described using the flowchart in FIG. 9(d).

[0107] Steps S3341 to S3343 are similar to steps S1221 to S1223 in the first embodiment, but differ in that they are executed within the digital camera.

[0108] In S3344, the number of pixels per unit length is calculated using the calculated object pixel dimensions and recorded as Exif format resolution information. Note that the unit length may be 1 mm, but when printing is taken into consideration, 1 inch may be used and recorded as PPI (Pixels Per Inch) information.

[0109] Furthermore, as described in the third embodiment, it is also possible to calculate and record the number of pixels per unit length not only for the in-focus area but also for each area detected by object detection processing such as face detection.

[0110] By recording the number of pixels per unit length as auxiliary information for the captured image in association with the corresponding image, there is an advantage that there is no need to provide an output image generation process in the printing device, because normal printing devices are equipped with a function to read PPI information and execute printing according to that information.

[0111] As described above, according to the image processing device of the modified example of this embodiment, it is possible to obtain highly accurate object side pixel dimension information from a captured image without changing the capacity of the shooting information in the conventional Exif format.

[0112] [Fourth embodiment] Next, a fourth embodiment of the present invention will be described. In the first and second embodiments, the position on the captured image where the object pixel size can be calculated is limited to the in-focus area. If the in-focus area is different from the desired object position and the desired object is at a different distance from the in-focus area, the actual size of the desired object cannot be obtained correctly. Therefore, in this embodiment, when the in-focus position and the desired object distance are different, the object pixel size is calculated and corrected to generate a full-size image.

[0113] The output image generation process in this embodiment, which generates an output image converted so that the subject appears at actual size when output to an output device, will be described with reference to the flowchart in Fig. 11 and Fig. 12. Fig. 12(a) shows a captured image input to generate an output image. In this example, the focus position is the left eye of person 410, and a case will be described in which an output image of person 420 is generated. As with the other flows, the process corresponding to this flowchart is realized by the control unit 16 reading out a corresponding processing program stored in a non-volatile memory in the control unit 16, for example, and expanding and executing the program in a volatile memory in the control unit 16, thereby causing the control unit 16 or each unit to operate.

[0114] In S401, the in-focus area in the captured image is obtained from the captured image. The captured image may be analyzed, for example, by extracting edges to estimate the in-focus area. In addition, the in-focus area frame (or the coordinate information indicating the in-focus area frame) set at the time of shooting may be recorded in Exif format and read out. In FIG. 12(b), the obtained in-focus area is shown as a rectangle 430.

[0115] In S402, the image generating unit executes subject recognition of the captured image. As an example, face detection will be shown. By the face detection process, a face frame 411 of a person 410 and a face frame 421 of a person 420 are detected, as shown in FIG. 12(c).

[0116] In S403, feature quantities such as age and gender are calculated from the image of the detected face area.

[0117] In S404, the calculated feature amounts are compared according to a predetermined rule to determine the main subject among the detected faces. For example, a rule may be determined such that a subject with a young estimated age is determined as the main subject. In this case, person 420 is determined as the main subject.

[0118] In S405, the shift in subject distance between the in-focus area 430 and the main subject area 421 is calculated. In the case of a digital camera capable of acquiring depth information as described in the third embodiment, it is possible to easily calculate the subject shift. On the other hand, in the cases of the first and second embodiments, since depth information cannot be acquired, the difference in distance is estimated from the contrast and blur of the images of the in-focus area 430 and the main subject area 421.

[0119] In S406, the object pixel size is corrected using the calculated distance difference and the object distance and focal length of the focused area calculated in the conversion information calculation process, and the object pixel size at the object distance of the main object 421 is calculated.

[0120] In S407, output destination information is acquired, and in S408, an output image is generated that has been image converted so that the output of the device that is the output destination of the main subject 421 is actual size.

[0121] Although the case where a subject at a position different from the in-focus position is automatically found has been described above as an example, the user may directly specify the subject at S404 without executing S402 and S403.

[0122] As described above, according to the image processing device of this embodiment, it is possible to generate an output image that has been converted so that the image appears at actual size when output to a destination device, even for subjects in areas other than the focused area.

[0123] [Modification of the fourth embodiment] In this embodiment, the case where the determination of the subject to be image-converted so as to be actual size when output to the output device (S401 to S406) is performed in the output image generation process has been described. However, the determination of the subject to be image-converted so as to be actual size when output to the output device (S401 to S406) can also be performed inside the imaging device described in the third embodiment. The object side pixel dimension or the number of pixels per unit length of the main subject obtained as a result of the execution is associated with the corresponding image as auxiliary information of the captured image and recorded. This makes it possible to generate an output image that has been image-converted so as to be actual size when output to the output device by the display device and printing device described in the third embodiment, or a relative image that allows for relative comparison based on the actual size.

[0124] 12, when there are multiple subjects, the object side pixel dimensions or the number of pixels per unit length in each subject region may be added as auxiliary information to the captured image. In this case, the object side pixel dimensions or the number of pixels per unit length in each subject region may be assigned as auxiliary information to the captured image in order of priority according to the rules described in S404.

[0125] [Fifth embodiment] Next, a fifth embodiment of the present invention will be described. In the above embodiment, a case where the object pixel dimensions calculated with high accuracy are applied to generation of a life-size image and generation of a relative image is described. In this embodiment, notification of dimension measurement as a different application destination of the object pixel dimensions is described.

[0126] The configuration of the digital camera 500 will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the functional configuration of the digital camera 500 incorporating the notification information calculation process. The configuration other than the image processing unit 53 shown in Fig. 13 is similar to that of the digital camera 300 described in the third embodiment, and therefore description thereof will be omitted.

[0127] The notification information calculation process for calculating notification information that notifies dimensional information related to the subject dimensions at the time of shooting, which is executed in the digital camera 500 of this embodiment having such a configuration, will be described in detail with reference to the flowchart in Fig. 14(a). This flow is executed by the control unit 32 or each unit in response to an instruction from the control unit 32.

[0128] The process starts when shooting starts, and in S531, a display image is generated to display the subject photographed by the digital camera on the display unit. The generated display image is an image reduced to match the resolution of the display unit.

[0129] In S532, the control unit 32 displays the generated display image on the display unit. In S533, it is confirmed whether the shutter button has been pressed down to the first position to focus (SW1). If SW1 has not been pressed, the process returns to S531 and continues the same process.

[0130] When SW1 is pressed, the depth image generation of S333 is executed as described in the third embodiment, and the conversion information calculation is further executed in S344. Here, the number of pixels of the depth image does not need to be the same as the number of pixels of the imaging element, and may be the same as the number of pixels of the display image. It is preferable to reduce the number of pixels of the depth image to be generated to reduce the calculation load of the depth image generation process.

[0131] In S534, the control unit 32 executes a calculation of notification information for notifying the user that the image capture mode allows dimension measurement by displaying the notification information on the display image. The notification information calculation process will be described with reference to the flowchart in FIG. 14(b).

[0132] In S5331, the control unit 32 acquires the number of pixels of the area where the focusing process has been performed. Here, the case where the focusing area is rectangular is described as an example, but the focusing area is not limited to a rectangle. Any shape may be used as long as it is a GUI (Graphical User Interface) for showing an area in an image. In order to acquire the number of pixels of the focusing area, first, information on the rectangular area used for focusing is acquired from the control unit, and the length of at least one of the long side and the short side is converted into the number of pixels in the displayed image. The number of pixels of the acquired side is multiplied by the object side pixel dimension, which is the conversion information calculated in S334, to calculate the dimension of the side of the focusing frame at the focused object distance. The focusing rectangular frame and the dimension of its side are used as notification information.

[0133] In S5332, the control unit 32 superimposes the calculated notification information on the display image to generate a superimposed image for notification. As shown in Fig. 15(a), a frame 551 of the in-focus area is displayed in a predefined color, and the calculated object side dimension 552 of the rectangle is displayed in a superimposed manner near the sides of the rectangle. It is preferable that the object side dimension is displayed in a superimposed manner only near one of the two horizontal sides in the horizontal and vertical sides as shown in Fig. 15(a).

[0134] Here, the processing of the depth image generating unit 331 is executed because the explanation is based on a digital camera equipped with an image sensor 31 capable of the image plane phase difference ranging explained in the third embodiment, but the present invention is also applicable to digital cameras that do not have an image plane phase difference ranging method. In the case of a digital camera that does not have an image plane phase difference ranging method, it is possible to calculate and display the object side dimension of the focusing frame using the subject distance information of the focusing area.

[0135] On the other hand, when a sensor capable of measuring distances in multiple regions is used, such as the digital camera 500 equipped with the image plane phase difference ranging method described in this embodiment, the object side dimensions of the region other than the frame of the focused region can also be calculated. For example, as shown in FIG. 15(b), a case will be described in which multiple people are present in the screen as subjects and their subject distances are different. In this case, multiple face detection frames can be obtained by face detection, and by using depth information, it is possible to calculate the object side pixel dimensions separately even if the subject distances are different. By accumulating the number of pixels on the sides of the calculated face frame and the corresponding object side pixel dimensions, the object side dimensions of each face frame can be calculated, and it is possible to superimpose and display notification information as shown in FIG. 15(b).

[0136] Furthermore, when multiple subjects are present on the screen as shown in Fig. 15(c), it is also effective to display the rectangular frames in different colors depending on the subject distance. Fig. 15(c) shows that the display color of face frame 553 is a different color from face frames 554 and 555.

[0137] As described above, according to the image processing device of this embodiment, it becomes possible to calculate the object side pixel dimensions through the conversion information calculation process, thereby making it possible to inform the photographer of the subject dimensions at the time of shooting, or to inform the photographer that shooting is being performed in which dimensional information can be obtained.

[0138] [Sixth embodiment] Next, a sixth embodiment of the present invention will be described. In the first embodiment, a full-size image is displayed and printed, and in the second embodiment, a case where a comparison image of the relative sizes of multiple images and subjects in the images is generated and output is described. In this embodiment, an application of image synthesis using object pixel dimension information will be described. As an example, a case where face detection is performed from a single image or multiple images, and face images of apparently different sizes are extracted taking into account their relative size by using object pixel dimension information, and image synthesis will be performed will be described. Note that in this embodiment, a face is described as the target, but the same applies to general objects.

[0139] Specific processing of the image synthesis application executed in this embodiment will be described with reference to the flowchart in Fig. 16. This synthesis application can be operated by the control unit 16 (Figs. 1 and 3), control unit 32 (Figs. 6 and 13), control unit 41 (Fig. 10(a)), and control unit 42 (Fig. 10(b)) in each of the above-mentioned embodiments. The processing executed by the control unit 16 will be described below as an example.

[0140] As with the other flows, the processing corresponding to this flowchart is realized by the control unit 16 reading out a corresponding processing program stored, for example, in a non-volatile memory within the control unit 16, expanding it into a volatile memory within the control unit 16, and executing it, thereby causing the control unit 16 or each unit to operate.

[0141] In S601, the control unit 16 accepts a selection of an image including a subject to be composited, based on an operation input from the user via the operation input unit 17. At this time, candidates for images including a subject to be composited may be displayed on the display 2 in a selectable manner, such as by list display or sequential display.

[0142] In S602, the control unit 16 or the image processing unit 22 executes the same process as S222 for the selected image to calculate the object side pixel dimensions. If an image that does not have information for calculating the object side pixel dimensions is selected, the process is interrupted. Alternatively, the information that the object side pixel dimension information could not be calculated is retained and the process proceeds to the next step.

[0143] In S603, face detection processing is performed on the selected image, and coordinate information of the face area is output. The coordinate information is information that specifies the rectangular area in which a face is detected, such as the coordinates of the upper left vertex and the lower right vertex of the rectangle, or the coordinates of the upper left vertex and the width and height of the rectangle. If multiple faces are detected in one image, similar information is output for the multiple faces. Furthermore, the coordinates of the head center are estimated using the output coordinate information of the face area.

[0144] Here, the head center coordinate estimation is performed by further detecting facial organs to detect positions such as the position between the eyebrows. The position to be determined as the head center depends on the application.

[0145] Next, a circular area of ​​radius R [mm] with the head center position calculated based on the object side pixel dimensions calculated in S602 as the origin or a rectangular area circumscribing the circle is cut out from the image selected as the synthesis target. The user can specify the desired cut-out range here, but in order to save the effort of setting, it is preferable to set the cut-out range in advance for each detected object according to the detected object. The cut-out shape is not limited to a circle or a rectangle, and may be any other shape. For example, cut-out may be performed along a region determined according to the contour of the detected subject.

[0146] For an image in which the metadata required for calculating the object side pixel dimension information is not included in the image file in which the selected image is recorded and the object side pixel dimension information cannot be calculated, the physical size of the rectangular area in which the face is detected is estimated by assuming the size of a typical human face. The object side pixel dimension is estimated from the estimated physical size of the rectangular area and the number of pixels on one side of the rectangle. In this case, the difference in size of each face is not taken into account, and the size is unified to the size assumed as the size of a typical face. In order to alleviate this problem, it is preferable to estimate the age and gender during face detection and set a typical face size according to the estimated age and gender. It is also possible to perform face organ detection in the detected face area and estimate the face size from position information of the face organs, such as the distance between the centers of the eyeballs. The face direction can also be estimated by face organ detection, and by performing face size estimation taking into account the change in the distance between the centers of the eyeballs due to the face direction, it is possible to estimate the face size with higher accuracy in the absence of object side pixel dimension information.

[0147] In this case, the error of the cropping range with respect to the specified size depends on the detection result of the rectangular region size of the face detection, and the region of accurate size cannot be cropped. If the object side pixel dimension information cannot be calculated, it is desirable to add information to the metadata of the image file that contains the cropped image so that it is possible to identify that the cropping region is inaccurate.

[0148] Next, a case will be described in which only one point of subject distance information at the focus position is held when calculating the object pixel size, and multiple people are captured in the image. If all the people are at the same distance as the focus position, no problem occurs, but as shown in Fig. 17(a), if the people are at different distances from the imaging device, even people of the same size will have different sizes. As described above, since the object pixel size can be calculated only at the focus position, the respective distances are not taken into consideration, and the faces of people in the foreground are cropped large, and the faces of people in the distance are cropped small.

[0149] To avoid this problem, the size of the rectangular area of ​​the face detection result of the person appearing at the focus position is used as a reference, and the ratio of the rectangular area size of the face detection result of other people is calculated. The object side pixel dimensions according to the distance of each person are estimated based on this ratio, and the cut-out range is calculated based on the estimated object side pixel dimensions. The result is as shown in Figure 17(b). In this case, the error of the cut-out range with respect to the specified size depends on the detection result of the rectangular area size of the face detection.

[0150] Also, when a face is captured in the captured image but is not at the focus position, it is possible to suppress a large difference in the cut-out range of the face region by assuming the size of a typical human face, as in the case of an image for which the object side pixel dimension information could not be calculated. Whether the face and the focus position match is determined by saving the coordinates of the focused area as metadata and determining whether the face detection position and the focus area match on the screen (on the XY plane). It is also possible to check how much the rectangular area detected by face detection changes during cut-out, and if the change is large, to determine that the face was not in focus at the time of shooting and a focus shift has occurred. Focus shift information is added, and object side pixel dimension information estimated from the size of a typical face as described above is also added.

[0151] It is also desirable to provide information that allows a user to distinguish between a subject for which cropping has been performed by calculating the object pixel dimensions and a subject for which cropping has been performed without using the object pixel dimensions. It is also desirable to allow a user to distinguish whether a cropped image has been cropped by calculating the object pixel dimensions or not when performing image synthesis, which will be described later. This information is stored in association with the cropped image.

[0152] If the resolution of the detected face image is low, cropping out an area of ​​the same size requires enlarging the image, resulting in a decrease in image quality. To avoid this problem, it is necessary to set a minimum resolution in the face detection process. The minimum resolution depends on the size to be printed, so whether or not the required resolution is ultimately met is determined by the print settings, which will be described later.

[0153] In S604, the user selects a desired face area image from the cropped face area images and places it at a desired position in a separately prepared working image area. It is possible to select and place multiple faces from the same image, and if it is desired to place a face from another image, the processes from S601 to S604 are repeated. Fig. 18(a) shows an example in which the faces of three children are selected and placed, as displayed on the display 2.

[0154] Here, if the control unit 16 selects an image whose resolution is significantly lower than that of other face area images when selecting a face area image, the control unit 16 may notify the user by issuing a warning, etc. For example, as shown in Fig. 17(a), a detected face 173 is out of focus and appears small. If the face 173' is cropped to the same size as the face 173, as shown in Fig. 17(b), the image will be degraded because the face 173' has been enlarged.

[0155] In S605, the control unit 16 accepts an operation by the user via the operation input unit 17 to place composite objects other than faces in the work area. The composite objects are used to decorate the composite image, which is the final product, and may be images, icons, illustrations, or characters. The user may operate to place each composite object in the work area in advance before face synthesis, or may place them during face synthesis. FIG. 18(b) shows an example in which the faces of the three children placed in FIG. 18(a) are decorated with flower illustrations around them.

[0156] In S606, the control unit 16 accepts an operation to change the size of the composite object placed by the user via the operation input unit 17. The user can select each composite object individually and change the size. Since the physical size of the cutout area of ​​the composite faces is the same, it is preferable that the composite faces can be changed at the same time. Therefore, in this embodiment, in response to accepting an operation to change the size of one selected face, the control unit 16 instructs the image processing unit 22 to change the size of the other composite faces to the same extent in conjunction with the selected face. The image processing unit 22 performs the same enlargement or reduction process on multiple images that have been set in advance to be linked, and the control unit 16 displays the multiple images as the processing results on the display 2. FIG. 18(c) shows a case where the size is changed based on the central face 182. When the size of the face 182 is reduced as in FIG. 18(c), the size of the other faces 181 and 183 is also changed in conjunction with the face as in FIG. 18(d). This is an example of a case where the size of only the faces other than the flower, which is a decoration, is changed in conjunction with the face. If the decorated image also has the object side pixel dimensions, the size can be changed while maintaining the relative size to the face. Even if it is not a face, composite objects that maintain the object side pixel dimensions can be handled by changing the size in conjunction. Also, by separating the memory area that stores the composite image to be sized from the memory area that stores the decorative images, it is possible to identify which area the image was selected from. If it is desired to change the size of the decorative objects in conjunction, the user can set it to be able to change the size in conjunction.

[0157] In S607, the control unit 16 receives a print setting instruction from the user via the operation input unit 17. In the print setting, the paper size to be printed and the desired print size (desired output size) are set. The print paper size is fixed to common A4 or A3 size, and the print setting can be easily implemented by setting only the desired print size. In addition, actual size printing can be specified, and the print size and divided printing are automatically calculated using the object side pixel dimensions so that the composited main subject is actual size.

[0158] When the size to be printed differs from the print paper size, including the above-mentioned actual size printing, and split printing is performed, for example, grid lines are superimposed on the working image area so that the split position can be identified. By displaying the grid lines, composite images that will be split when printed can be identified. FIG. 19(a) shows an example in which grid lines 191 are displayed at the split position when A4 size print paper is used to obtain A3 size print output. It is easy to recognize that a face 192 exists on the split line and will be split and printed. This allows the user to easily move an important composite image such as a face image to a position where it will not be split when printed. For example, by reducing the entire image arranged as shown in FIG. 19(b) and moving it to the right, the face 192 can be rearranged so that it will not be split when printed. In addition, it is also possible to control so that a face image is not automatically arranged on a grid line.

[0159] When the printing paper size and the size to be enlarged for printing are known, it is preferable to superimpose dividing grid lines on the working image area where the compositing work will be performed.

[0160] In addition, in the print setting, when the desired print size is set, the resolution of the arranged composite image is confirmed, and a composite image with a large image enlargement ratio and a low resolution when printed is extracted and notified so as to be identifiable.

[0161] As described above, the image synthesis application of this embodiment makes it possible to synthesize images using object pixel dimension information, and also to estimate dimension information and synthesize images that do not have object pixel dimension information. Also, by making it possible to distinguish whether or not there is object pixel dimension information, the user can synthesize images while understanding the dimensional accuracy of the subject.

[0162] [Modification of the sixth embodiment] A modified example of the sixth embodiment will be described. As shown in the sixth embodiment, in an image in which only distance information of the focus position is held, there may be cases where multiple faces exist in the image other than at the focus position. In such a case, a process for enabling faces appearing at distances other than the focus position to be printed at actual size will be described.

[0163] The control unit 16 or image processing unit 22 estimates the object pixel dimensions by assuming that faces at distances other than the focus position are of a general face size. The estimated object pixel dimensions or PPI information is stored in addition to the metadata of the image together with the detected face area information. By selecting the face to be printed at actual size when printing, it is possible to print at approximately actual size.

[0164] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0165] 1. Digital camera 2. Display 3. Printer 100 Image processing device 11 Input section 12 Image processing section 121 Conversion information calculation unit 122 Image Generation Unit 13 Storage section 14 Communications Department 15 Output section

Claims

1. An acquisition means for acquiring an image captured by the imaging means and imaging information at the time of capturing the image; a calculation means for calculating an object side pixel size of a target subject in the image based on the shooting information and a pixel size of the imaging means; a control means for calculating at least one number of pixels per unit length from the object side pixel dimension calculated by the calculation means, and for recording the number of pixels per unit length in association with a corresponding image; having the acquiring means acquires, as the photographing information, focusing information indicating a focusing state of a subject in an image; The image processing device according to claim 1, wherein the calculation means calculates the object side pixel dimension based on the focus information.

2. 2. The image processing apparatus according to claim 1, further comprising image processing means for processing the image captured by said imaging means on the basis of said object side pixel size.

3. the acquiring means acquires output destination information relating to an output destination device to which the image is to be output; 3. The image processing device according to claim 2, wherein the image processing means converts the image based on the output destination information to generate an output image so that the subject of the image appears at actual size when output from the output destination device, based on the object pixel dimensions calculated by the calculation means.

4. The image processing device described in claim 2 or 3, characterized in that the image processing means estimates the object side pixel dimensions of the object region where the object side pixel dimensions could not be calculated by using the size of the object region detected by the object detection means in the object region where the object side pixel dimensions could not be calculated and the size of the object region detected by the object detection difference means in the object region where the object side pixel dimensions could be calculated, and calculates the estimated object side pixel dimensions.

5. 5. The image processing apparatus according to claim 4, wherein said image processing means records information in the image data so that said object side pixel dimension and said estimated object side pixel dimension can be distinguished.

6. 6. The image processing device according to claim 2, wherein, when combining a plurality of subjects having the object pixel dimension into a single image, the image processing means selects one subject from the plurality of subjects and changes the size of the selected subject, thereby changing the size of subjects having the object pixel dimension other than the selected subject while maintaining their relative sizes.

7. 7. The image processing device according to claim 2, wherein the image processing means calculates a size of the subject using the number of pixels of the subject detected by subject detection and the object side pixel dimensions, and when the calculated dimensions of the subject are larger or smaller than the pre-stored dimensions of the subject by a threshold value or more, cuts out the subject based on the pre-stored dimensions of the subject.

8. 8. The image processing device according to claim 2, wherein when actual size printing is set, the image processing means calculates the size of the print using the object side pixel dimensions of the main subject, and generates divided images for printing based on the set paper size.

9. 9. The image processing device according to claim 2, wherein the image processing means converts the images of the plurality of subjects in the plurality of images according to their actual sizes by using the object-side pixel dimensions of each of the subjects in the plurality of images, thereby generating images in which the dimensions of the plurality of subjects can be compared relatively.

10. 10. The image processing device according to claim 1, wherein the calculation means calculates the object-side pixel dimension based on at least one of a focal length variation amount corresponding to the focus information, a subject distance, and a principal point position of an optical system.

11. 10. The image processing apparatus according to claim 1, wherein the calculation means calculates the object-side pixel size at a focused object distance by using the object distance to the focused object as the photographing information.

12. 10. The image processing device according to claim 1, wherein the calculation means calculates the object side pixel size based on a shooting magnification corresponding to the focus information.

13. the acquiring means acquires depth distribution information corresponding to the image; 10. The image processing device according to claim 1, wherein the calculation means calculates the object side pixel dimension at each pixel position of the depth distribution information by using the depth distribution information.

14. The image processing device according to claim 13, characterized in that the calculation means obtains a subject distance to at least one subject present at any position in the image using the depth distribution information, and calculates the object-side pixel dimension at the subject distance using the subject distance.

15. The image processing device according to claim 13 or 14, characterized in that the depth distribution information is any one of distribution information of the amount of parallax indicated by a phase difference signal corresponding to the image, distribution information of the amount of defocus of a subject present in the image, and distribution information of a subject distance to a subject present in the image.

16. 4. The image processing apparatus according to claim 3, wherein the output destination information is at least one of information on a print paper size, a display screen size, and a display screen resolution.

17. 17. The image processing apparatus according to claim 1, wherein the control means records PPI (Pixel Per Inch) information as the number of pixels per unit length in an image file in Exif format together with image data of the corresponding image.

18. the image processing means executes a face detection process on the image, and executes the calculation means using a subject distance of the detected face area to calculate an object pixel dimension or the number of pixels per unit length of at least one area of ​​the image; 3. The image processing apparatus according to claim 2, wherein said control means records at least one of said object side pixel dimension and said number of pixels per unit length in association with said corresponding image.

19. 19. The image processing device according to claim 1, wherein the control means calculates the dimensions of the area indicated by the GUI from the object pixel dimensions based on the number of pixels of the area indicated by the GUI to be superimposed on the image, and displays the calculated dimensions on a display means.

20. an imaging element capable of acquiring phase difference signals from a plurality of photoelectric conversion units; An image processing device according to any one of claims 1 to 19, An imaging device comprising a conversion means.

21. An image processing device according to any one of claims 1 to 19, and a display unit for displaying the image.

22. An image processing device according to any one of claims 1 to 19, and a printing means for printing the image.

23. an acquisition step of acquiring an image captured by an imaging means and imaging information at the time of capturing the image; a calculation step of calculating an object side pixel dimension of the subject in the image based on the photographing information and a pixel dimension of the imaging means; and a control step of calculating at least one number of pixels per unit length from the object side pixel dimension calculated in the calculation step, and recording the number of pixels per unit length in association with a corresponding image. having In the acquiring step, focusing information indicating a focusing state of a subject in an image is acquired as the photographing information, The image processing method according to claim 1, wherein the calculation step calculates the object side pixel dimension based on the focus information.

24. 20. A program for causing a computer to execute procedures executed by each unit of the image processing apparatus according to claim 1.

25. A storage medium storing the program according to claim 24.

Images