Image processing device and method for controlling image processing device

Abstract

An image processing device includes: an acquisition unit configured to acquire an image captured by a camera unit and camera information when the image is captured; and a calculation unit configured to calculate the object-side pixel size of a subject in the image based on the camera information and the pixel size of the camera unit, wherein the acquisition unit acquires focus information representing a focus state of the subject in the image as the camera information, and wherein the calculation unit calculates the object-side pixel size based on the focus information.

Description

Technical Field

[0001] The present invention relates to an image processing device and an image processing method for performing processing related to the size of a subject in an image based on information at the time of image capture and information related to an image capture device used in the image capture. Background Art

[0002] There is a technology that uses information at the time of image capture to display a subject in an image at full size (full scale).

[0003] Patent Document 1 discusses a display system and method for displaying an image at actual size on a display device screen. This method proposes a method that obtains information from the image capture process based on the specifications of the display screen on which the image is displayed and the image stored in the Exchangeable Image File Format (Exif), calculates the actual size, and displays the image.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2010-78677 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] The technology discussed in Patent Document 1 calculates the actual size using information stored in Exif files, but does not include processing that takes focus into account. This makes it difficult to accurately calculate the full size based on the image's focus state. Furthermore, the technology discussed only applies to configurations that display the actual size of a single subject in an image.

[0009] In view of the above-mentioned problems, the present invention is directed to providing an image processing apparatus and an image processing method that solve at least one of the above-mentioned problems. More specifically, the present invention is directed to providing an image processing apparatus and an image processing method that generate an output image that more accurately reproduces the size of a subject located at a desired position in the image. Furthermore, the present invention is directed to providing an image processing apparatus and an image processing method that further utilize the size of the subject.

[0010] Solutions for solving problems

[0011] In order to achieve the above-mentioned purpose, an image processing device according to the present invention includes: an acquisition unit, which is configured to acquire an image captured by a camera unit and camera information when the image is captured; and a calculation unit, which is configured to calculate the object-side pixel size of the object subject in the image based on the camera information and the pixel size of the camera unit, wherein the acquisition unit acquires focus information representing the focus state of the subject in the image as the camera information, and wherein the calculation unit calculates the object-side pixel size based on the focus information.

[0012] In addition, according to an image processing method of the present invention, the method includes: an acquisition step for acquiring an image captured by a camera unit and camera information when the image is captured; and a calculation step for calculating the object-side pixel size of the subject in the image based on the camera information and the pixel size of the camera unit, wherein, in the acquisition step, focus information representing the focus state of the subject in the image is acquired as the camera information, and wherein, in the calculation step, the object-side pixel size is calculated based on the focus information.

[0013] Effects of the Invention

[0014] According to the present invention, an image can be generated in which the size of a subject at a desired position in the image is more accurate and clearer. The size of the subject at the desired position in the image can also be used to compare the sizes of the subjects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to a first exemplary embodiment.

[0016] Figure 2A is a flowchart illustrating processing to be executed in the first exemplary embodiment.

[0017] Figure 2B is a flowchart illustrating processing to be executed in the first exemplary embodiment.

[0018] Figure 2C is a flowchart illustrating processing to be executed in the first exemplary embodiment.

[0019] Figure 3 is a block diagram illustrating a functional configuration of an image processing apparatus according to a second exemplary embodiment.

[0020] Figure 4A is a flowchart illustrating processing to be executed in the second exemplary embodiment.

[0021] Figure 4B is a flowchart illustrating processing to be executed in the second exemplary embodiment.

[0022] Figure 5A is a diagram illustrating processing to be executed in the second exemplary embodiment.

[0023] Figure 5B is a diagram illustrating processing to be executed in the second exemplary embodiment.

[0024] Figure 5C is a diagram illustrating processing to be executed in the second exemplary embodiment.

[0025] Figure 5D is a diagram illustrating processing to be executed in the second exemplary embodiment.

[0026] Figure 5E is a diagram illustrating processing to be executed in the second exemplary embodiment.

[0027] Figure 5F is a diagram illustrating processing to be executed in the second exemplary embodiment.

[0028] Figure 5G is a diagram illustrating processing to be executed in the second exemplary embodiment.

[0029] Figure 6 is a block diagram illustrating a functional configuration of an imaging apparatus according to a third exemplary embodiment.

[0030] Figure 7A is a diagram illustrating an image sensor according to a third exemplary embodiment.

[0031] Figure 7B is a diagram illustrating an image sensor according to a third exemplary embodiment.

[0032] Figure 8A A diagram illustrating the distance measurement principle of the imaging plane phase difference method.

[0033] Figure 8B A diagram illustrating the distance measurement principle of the imaging plane phase difference method.

[0034] Figure 8C A diagram illustrating the distance measurement principle of the imaging plane phase difference method.

[0035] Figure 8D A diagram illustrating the distance measurement principle of the imaging plane phase difference method.

[0036] Figure 8E A diagram illustrating the distance measurement principle of the imaging plane phase difference method.

[0037] Figure 9A is a flowchart illustrating processing to be executed in the third exemplary embodiment.

[0038] Figure 9Bis a flowchart illustrating processing to be executed in the third exemplary embodiment.

[0039] Figure 9C is a flowchart illustrating processing to be executed in the third exemplary embodiment.

[0040] Figure 9D is a flowchart illustrating processing to be executed in the third exemplary embodiment.

[0041] Figure 10A is a block diagram illustrating a functional configuration of a display device according to a third exemplary embodiment.

[0042] Figure 10B is a block diagram illustrating a functional configuration of a printing apparatus according to a third exemplary embodiment.

[0043] Figure 11 is a flowchart illustrating processing to be executed in the fourth exemplary embodiment.

[0044] Figure 12A is a diagram illustrating processing to be executed in the fourth exemplary embodiment.

[0045] Figure 12B is a diagram illustrating processing to be executed in the fourth exemplary embodiment.

[0046] Figure 12C is a diagram illustrating processing to be executed in the fourth exemplary embodiment.

[0047] Figure 13 is a block diagram illustrating a functional configuration of an imaging apparatus according to a fifth exemplary embodiment.

[0048] Figure 14A is a flowchart illustrating processing to be executed in the fifth exemplary embodiment.

[0049] Figure 14B is a flowchart illustrating processing to be executed in the fifth exemplary embodiment.

[0050] Figure 15A is a diagram illustrating a notification method according to the fifth exemplary embodiment.

[0051] Figure 15B is a diagram illustrating a notification method according to the fifth exemplary embodiment.

[0052] Figure 15C is a diagram illustrating a notification method according to the fifth exemplary embodiment.

[0053] Figure 16 is a flowchart illustrating processing to be executed in the sixth exemplary embodiment.

[0054] Figure 17A: is a diagram illustrating a display screen to be displayed in processing to be executed in the sixth exemplary embodiment.

[0055] Figure 17B : is a diagram illustrating a display screen to be displayed in processing to be executed in the sixth exemplary embodiment.

[0056] Figure 18A : is a diagram illustrating a display screen to be displayed in processing to be executed in the sixth exemplary embodiment.

[0057] Figure 18B : is a diagram illustrating a display screen to be displayed in processing to be executed in the sixth exemplary embodiment.

[0058] Figure 18C : is a diagram illustrating a display screen to be displayed in processing to be executed in the sixth exemplary embodiment.

[0059] Figure 18D : is a diagram illustrating a display screen to be displayed in processing to be executed in the sixth exemplary embodiment.

[0060] Figure 19A : is a diagram illustrating a case where grid lines are displayed on a display screen to be displayed in processing to be executed in the sixth exemplary embodiment.

[0061] Figure 19B : is a diagram illustrating a case where grid lines are displayed on a display screen to be displayed in processing to be executed in the sixth exemplary embodiment. DETAILED DESCRIPTION

[0062] [First exemplary embodiment]

[0063] Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. The following exemplary embodiments are not intended to limit the invention set forth in the appended claims. Various features are described in the exemplary embodiments, but not all of these features are essential to the present invention, and any combination of these features may be used. Furthermore, in the accompanying drawings, identical or similar structures are assigned identical reference numerals, and redundant descriptions will be omitted.

[0064] In the exemplary embodiments described below, an example of an image processing device incorporating the image processing device of the present invention will be described. This image processing device captures a captured image from an imaging device such as a digital camera and outputs a full-size image of the subject to an output device such as a display device or a printer. The present invention is applicable to any device that can calculate a full-size image based on a captured image, imaging information corresponding to the captured image, and output device information.

[0065] <Structure of Image Processing Apparatus 100>

[0066] Will refer to Figure 1 The structure of the image processing apparatus 100 will be described. Figure 1 is a block diagram showing the functional structure of the image processing apparatus 100 .

[0067] The input unit 11 is an interface (I / F) that acquires image information (image data) captured by an imaging device such as the digital camera 1 and imaging information (imaging conditions, image processing parameters, etc.) from the digital camera 1 or other external devices.

[0068] The image processing unit 12 performs various types of image processing (such as brightness or color conversion, correction for defective pixels, shading, and noise components, filtering, and image synthesis) on image data acquired from the input unit 11, storage unit 13, or communication unit 14, as well as various types of calculations used for image processing. The image processing unit 12 includes a conversion information calculation unit 121 and an output image generation unit 122. The conversion information calculation unit 121 performs calculations to convert the size of pixels (pixel size) in the image sensor of the digital camera 1 that acquired the image data, for example, to a size in space at a desired subject distance, based on imaging information of the image data to be acquired. The output image generation unit 122 uses the pixel size in the subject space calculated by the conversion information calculation unit 121 and device information (output destination information) related to a device that displays or prints the subject at full size to generate an output image. The output image is an image converted so that the final output on the output destination device (display on a display device or printout from a printing device) is at full size. The image processing unit 12 may include logic circuitry. Furthermore, as another configuration, the image processing unit 12 may include a central processing unit (CPU) and a memory that stores a calculation processing program.

[0069] The storage unit 13 includes a recording medium such as a memory, and stores various types of information such as image data and parameters input via the input unit 11 or the communication unit 14. In addition, the storage unit 13 stores an output image generated by the output image generation unit 122 and converted so that the subject takes full size in output on the output destination device, and device information related to the digital camera 1 required by the conversion information calculation unit 121.

[0070] The communication unit 14 is a communication interface for transmitting and receiving data with external devices. In the present exemplary embodiment, the communication unit 14 communicates with the digital camera 1, the display 2, or the printer 3, and acquires device information related to the digital camera 1, the display 2, or the printer 3 required by the conversion information calculation unit 121 and the output image generation unit 122.

[0071] The output unit 15 is an I / F that outputs full-size information including the output image generated by the image processing unit 12 to the display 2 or the printer 3 as an output destination.

[0072] The control unit 16 includes a CPU serving as a computer, and controls the components of the entire image processing apparatus 100 via a bus based on a computer program stored in a nonvolatile memory.

[0073] <Output Image Generation Processing>

[0074] Regarding output image generation processing (which is executed by the image processing apparatus 100 of the present exemplary embodiment and in which an output image is generated by performing image processing on an input image for full-size output of an imaged subject), reference will be made to the following. Figure 2A The specific processing is described with reference to a flowchart. The processing corresponding to the flowchart is implemented, for example, by the control unit 16 or each component operating as follows: the control unit 16 reads out a corresponding processing program stored in the non-volatile memory within the control unit 16, loads the processing program into the volatile memory within the control unit 16, and executes the processing program.

[0075] In step S121, the image processing unit 12 acquires image data of a captured image to which information at the time of image capture is attached from the digital camera 1 via the input unit 11. The input unit 11 may acquire similar image data to which information at the time of image capture is attached from another external device.

[0076] In step S122, the conversion information calculation unit 121 calculates conversion information for converting the size of one pixel in the captured image into a size occupied in a space at a desired subject distance. Figure 2B The conversion information calculation process is described with reference to a flowchart. The process corresponding to the flowchart is implemented, for example, by the control unit 16 or each component operating as follows: the control unit 16 reads out a corresponding processing program stored in the non-volatile memory within the control unit 16, loads the processing program into the volatile memory within the control unit 16, and executes the processing program.

[0077] In step S1221, the imaging information used when capturing the captured image is acquired from information attached to the captured image (such as data (metadata) stored in an Exchangeable Image File Format (Exif) tag). The acquired imaging information includes, for example, the camera name (camera ID), lens name (lens ID), focal length, F-number, and subject distance of the camera used to capture the captured image. Typically, imaging information such as focal length, F-number, and subject distance stored in Exif format is rounded to integers or stored as discrete values ​​with low resolution. For example, the focal length of a lens is rounded to the nearest integer and stored, and values ​​after the decimal point cannot be acquired. Furthermore, with an internal focus lens, the focal length varies depending on the focus distance, but the same focal length is stored in the image file even if the focus distance varies. Because the information stored in the Exif tag is not precise, using the acquired values ​​to calculate magnification conversion information can result in a larger error in the final full-size image relative to the actual size of the subject. As a countermeasure against such errors, the imaging device records information related to the focus state at the time of imaging in the manufacturer's comment area in EXIF ​​format and stores this information as imaging information. This information includes, for example, the step length or position of the motor driven for focus adjustment, or the step length or position of the motor driving the zoom lens. In step S1221, in addition to the aforementioned information, this information (focus information) is also acquired and used to correct the imaging information in the subsequent step S1222.

[0078] Furthermore, lens aberration affects the captured image, and due to this aberration, it may be impossible to correctly obtain full-size output when generating an output image for full-size output. If distortion aberration still exists, distortion occurs in the image, and due to the influence of distortion, it is impossible to obtain correct full-size output. Therefore, it is desirable to also obtain information related to distortion aberration as imaging information. During imaging, a value corresponding to the lens status at the time of imaging is recorded in the manufacturer's comment area in EXIF ​​format, and the distortion aberration information is obtained by reading out the recorded value in step S1221. Alternatively, distortion aberration information for multiple cameras and lenses can be pre-stored in storage unit 13 as information for correction, and the distortion aberration information can also be obtained by reading out the distortion aberration information from storage unit 13 based on the camera name and lens name at the time of imaging, as well as the focal length information.

[0079] Axial chromatic aberration information, another type of aberration information, is also essential for generating highly accurate output images for full-size output. When axial chromatic aberration is large, the lens focus state varies depending on the subject's color, even if the subject is at the same distance, resulting in different output subject distances. Axial chromatic aberration information is required to correct errors in subject distance. Similar to distortion information, axial chromatic aberration information can be recorded in the manufacturer's notes area of ​​the captured image in Exif format and read out, or it can be pre-stored in storage unit 13 as information for correction purposes and read out based on camera and lens information.

[0080] In step S1222, using the acquired imaging information including the focus state information, the control unit 16 performs correction of the acquired imaging information. In step S1222, correction information for correcting the imaging information of the corresponding model pre-stored in the storage unit 13 is acquired based on the acquired camera name and lens name. Correction information for models not stored in the storage unit 13 can be acquired externally via the communication unit 14.

[0081] One of the correction information used here is table information showing the focus state and the focal length values ​​corresponding to each focus state, used to obtain the amount of change in focal length relative to the focus state. This table information is then used to correct the focus. For example, if the focus state is controlled based on 100 split states at distances from the closest distance to infinity, focal length information corresponding to 100 different focus states is stored. Alternatively, the relationship between the focus state and the focal length corresponding to each focus state can be stored as a function, and the focal length corresponding to the input focus state can be calculated and obtained. This information is also necessary to correct for manufacturing-related differences from the design focal length value. Focal length distribution information based on manufacturing errors can be used to reduce average manufacturing errors. Information related to manufacturing errors can be obtained from image files by measuring the focal length for each individual device during manufacturing, storing this information on a storage medium attached to the camera body or lens, and reading this stored information during capture and recording it in the manufacturer's note area of ​​the captured image.

[0082] Other correction information includes information for correcting the subject distance. As the subject distance at the time of recording, attached to the captured image, the distance from the focused subject to the frontmost surface of the lens or the distance from the focused subject to the image sensor is typically stored. Therefore, the definition of the subject distance for each camera is pre-stored in storage unit 13 as one of the correction information, and the definition of the acquired subject distance is determined based on the acquired camera name. The correction information also includes information for converting the acquired subject distance into the distance from the subject to the front principal point or front focus. This correction information can be generated based on design information related to the lens and design information related to the lens barrel. Since the front principal point (the position of the principal point of the optical system) or the front focus position changes depending on the focus state, the difference position information relative to the reference position (the frontmost surface of the lens or the image sensor plane) of the front principal point or front focus corresponding to each focus state is stored as a table or function as correction information. Finally, the focal length is corrected based on the acquired subject distance during imaging, the definition of the subject distance of the camera used in imaging, and the difference position information from the reference position to the front principal point or front focus position based on the definition.

[0083] Other information used to correct the subject distance includes the aforementioned axial chromatic aberration information. Due to axial chromatic aberration, the image-side focus position varies depending on the color between subjects at the same distance, causing the output subject distance to vary. To correct this effect, the amount of axial chromatic aberration at multiple visible wavelengths (e.g., the center wavelength of the image filter) is precalculated and stored using lens design information. Furthermore, color information related to the focused area on the focused subject (luminance information related to each color component of the corresponding image area) is acquired. Based on the color component mixing ratio and the axial chromatic aberration value, the offset from the image-side focus position, serving as a reference, is calculated, and subject distance correction is performed. The image-side focus position, serving as the reference, is the focus position of the reference subject's color used when calculating the subject distance. For example, when using a black and white chart as a reference, the image-side focus position, serving as the reference, can be calculated based on the luminance information related to the color components and the axial chromatic aberration information.

[0084] In step S1223, the magnification at the time of image capture is now calculated based on the corrected focal length and the subject distance. When the distance from the subject to the front focus is expressed as the subject distance x and the focal length is expressed as f, the magnification m is obtained by the following formula (1):

[0085] m=x / f Formula (1).

[0086] The magnification here is set as a magnification for converting the subject on the imaging plane into an actual size on the object side.

[0087] In the case where the subject distance is set as the distance from the subject to the front principal point (the subject distance is represented by S), the distance S′ from the rear principal point to the image plane can be calculated using the subject distance S and the focal length x, and the magnification m can be obtained by the following formula (2):

[0088] m=S′ / S Formula (2).

[0089] Furthermore, in step S1223, the control unit 16 obtains the pixel size p (pixel pitch, the interval between pixels) of the image sensor used for imaging. By calculating the product of the pixel size p and the magnification m, the size of one pixel of the image sensor at the subject distance (the object-side pixel size representing the length or size of the subject existing on one pixel in real space) can be calculated, and the actual size of the subject can be obtained from the number of pixels that the subject extends over.

[0090] While the magnification is obtained based on the corrected focal length and subject distance, it is possible to precalculate and store highly accurate magnification information based on lens design information. In this case, a table corresponding to information such as the step size or position of the motor driven for focusing, or the step size or position of the motor driving the zoom lens, is stored, and a magnification suitable for the imaging situation is calculated based on these values. The table of magnification information does not need to be stored in a manner corresponding to all possible values ​​that the motor can take. Discrete values ​​can be stored, and values ​​between discrete values ​​can also be calculated through interpolation. In addition, a function can be fitted to the motor magnification information and values, and its coefficients can be stored.

[0091] Return Reference Figure 2A In step S123, the control unit 16 and the output image generation unit 122 generate an output image suitable for full-size output of the output destination device of the image. Figure 2C The output image generation process is described with reference to a flowchart.

[0092] In step S1231, information regarding the output destination of the output image is acquired. For example, if the output image is output as a printed product by printer 3, information regarding the size of the sheet to be output is acquired. For example, if the output image is displayed on display 2, the display screen size and display screen resolution of display 2 are acquired. Here, display screen resolution refers to the number of display pixels on the display screen.

[0093] In step S1232, the output image generation unit 122 uses the acquired output destination information and the object-side pixel size to generate an output image appropriate for the output destination. If the output is a printed product from a printer 3 or the like, the number of pixels in the captured image in the longitudinal and transverse directions relative to the sheet size is calculated by dividing the sheet size by the object-side pixel size. This number of pixels is considered the number of pixels in the output image. If the number of pixels in the captured image is greater than the calculated number of pixels in the output image, the output image is generated by selecting and extracting a desired area from the captured image that corresponds to the number of pixels in the output image. On the other hand, if the number of pixels in the captured image is less than the number of pixels set as the calculated number of pixels in the output image, blank pixels are added to the captured image to bring the number of pixels in the captured image into line with that of the output image, and the output image is generated. When extracting an area in the image, it is most appropriate to select the area so that the focused subject is included in the area. Furthermore, it is desirable to select the area so that blank pixels are minimized.

[0094] On the other hand, if the output destination of the output image is a display device such as display 2, the number of pixels in the vertical and horizontal directions of the captured image required for display is calculated using the display screen size of display 2 and the object-side pixel size of the image sensor, similar to the case of printed output images. Subsequently, the desired area is extracted from the captured image based on the calculated number of pixels. This process aligns the display screen size with the image size at the subject distance of the extracted captured image. Next, a conversion factor is calculated based on the calculated number of pixels and the display screen resolution, and the extracted captured image is enlarged / reduced so that its pixel count matches the display screen resolution, generating an output image for full-size output on display 2.

[0095] In step S124, the image processing unit 12 outputs the generated output image to the output destination device via the output unit 15, and the output image generation process ends. At this time, the generated output image may be recorded in the storage unit 13 together with output destination device information such as print sheet size, display screen size, and display screen resolution.

[0096] As described above, according to the present exemplary embodiment, by using imaging information added to a captured image and correction information for improving the accuracy of the imaging information, an output image capable of more accurately reproducing the size of a subject present at a desired position in an image can be generated.

[0097] [Second exemplary embodiment]

[0098] In the first exemplary embodiment, a configuration is described that highly accurately calculates information for converting an image so that a subject in a captured image is output at full size, acquires output device information, and generates an output image for full-size output. By calculating information for outputting a subject at full size, it is possible to output the subject in a manner that allows the subject's actual size to be identified by comparison with other objects, without having to generate an output image for full-size output. In view of the foregoing, in the present exemplary embodiment, an image processing device is described that outputs images so that relative size comparison can be performed taking into account actual size by using multiple captured images as input and calculating full-size conversion information for each of the multiple captured images. Below, differences from the first exemplary embodiment are described, with reference numerals and symbols assigned to matters similar to those in the first exemplary embodiment, and detailed description of these matters will be omitted. This omission of description applies similarly to the exemplary embodiments described below.

[0099] Will refer to Figure 3 The configuration of the image processing apparatus 200 according to the present exemplary embodiment will be described below. In the present exemplary embodiment, the image processing unit 22 is characterized by including a comparison image generation unit 222. The configuration other than the image processing unit 22 is similar to that of the first exemplary embodiment. Alternatively, in addition to the output image generation unit 122 included in the first exemplary embodiment, the comparison image generation unit 222 may be included, and the configuration of the first exemplary embodiment may also be implemented.

[0100] <Relative Comparison Image Generation Processing>

[0101] Regarding the relative image generation processing for generating a relative image to be executed by the image processing apparatus 200 of the present exemplary embodiment, reference will be made to FIG. Figure 4A The specific processing is described with reference to a flowchart. The processing corresponding to the flowchart is implemented, for example, by the control unit 16 or each component operating as follows: the control unit 16 reads out a corresponding processing program stored in the non-volatile memory within the control unit 16, loads the processing program into the volatile memory within the control unit 16, and executes the processing program.

[0102] In step S221 , the image processing unit 22 acquires a plurality of captured images to which information at the time of image capture is added from the digital camera 1 or the external storage device via the input unit 11 .

[0103] In step S222 , similar to the first exemplary embodiment, the conversion information calculation unit 121 calculates conversion information for each captured image, the conversion information being used to calculate a size in a space at a desired subject distance converted from the size of one pixel in the captured image.

[0104] In step S223, the comparison image generation unit 222 generates a relative image by which a plurality of subjects in the captured image can be relatively compared in consideration of actual size. Figure 4B Flowchart and Figures 5A to 5G The related image generation process will be described.

[0105] In step S2231, the control unit 16 converts the image so that the pixel sizes at the subject distances of the input plurality of captured images (object side pixel sizes) become equal to each other. Figures 5A to 5C As shown, a reference captured image (reference image) is selected from the multiple captured images input. Any image can be selected as the reference image, or the user can select the reference image when the image is input. In addition, in the case where the resolution of the relative image as the final output is to have a high resolution, the captured image with the smallest object side pixel size calculated in step S222 is selected. On the contrary, in the case where an image with a low resolution is required, the captured image with the largest object side pixel size calculated is selected. After determining the reference image, the following formula (3) is used to calculate the enlargement / reduction ratio Ri of the comparison image for making the pixel size of the comparison image consistent with the pixel size of the reference image based on the object side pixel size Ls of the reference image and the object side pixel size Lr of another captured image (comparison image).

[0106] Ri=Lr / Ls Formula (3)

[0107] The enlargement / reduction ratio Ri is calculated for each comparison image, and the enlargement / reduction process is performed on each comparison image according to the enlargement / reduction ratio Ri. Figures 5D to 5F As shown, the object-side pixel size of all enlarged / reduced input captured images (the processed images will be referred to as enlarged / reduced captured images) becomes the same size as the object-side pixel size of the reference image.

[0108] In step S2232, the control unit 16 recognizes the subject in each magnified / reduced captured image and extracts the subject region of the specific subject. As an example of subject recognition and region discrimination, the following will be described. Figures 5A to 5G Figure 2 shows the person recognition in the person image shown in Figure 2. Person recognition uses machine learning and uses the learning results to perform person detection and region discrimination. Figure 5DAs shown, when multiple people are detected in one zoomed / reduced captured image, facial recognition technology is used to identify the same person appearing in another zoomed / reduced captured image, and the area of ​​the identified person is extracted. This processing allows comparison of changes in the size of the same person over time. Furthermore, without limitation to this exemplary embodiment, when multiple input images are pre-acquired in step S221, an image from which the desired subject is identified (or an image from which the subject's area is extracted, etc.) can be acquired.

[0109] Furthermore, although it is desirable to perform region extraction to extract the subject along its outer edge, depending on the application, it is desirable to simultaneously extract background information while capturing the image. In the case of simultaneously extracting background information, it is desirable to extract a rectangular or elliptical region substantially circumscribing the selected subject.

[0110] In addition, in this exemplary embodiment, the description is given for an example case where the subject is a person, but the processing can be performed for any subject, and relative comparison of the dimensions of different subjects can also be performed. The selection and extraction of the subject in the magnified / reduced captured image does not always need to be performed automatically, and the user can select and extract the subject.

[0111] The processing of step S2231 and step S2232 can also be performed in the reverse order.

[0112] In step S2233, the subject extracted from the magnified / reduced captured image is synthesized into one image, and a Figure 5G The relative image shown. It is desirable to adjust the size in such a manner that the entire image of the relatively largest subject falls within the composite image, and to composite another subject with the correct relative size based on the largest subject. Alternatively, a reference subject may be set, and the size may be adjusted in such a manner that the entire image of the reference subject falls within the composite image, and another subject may be composited to have the correct relative size. Regarding the arrangement of the subjects in the image, it is desirable to arrange the subjects in the horizontal direction or the vertical direction in order of size or in order of the shooting date and time. If information related to the shooting date and time is acquired when the shooting information is acquired in step S1221, this information becomes available.

[0113] This description describes the case where multiple subjects are selected, region extraction is performed, and the selected subjects are arranged at relative sizes adjusted based on a reference image. However, a background image can be selected and set as the reference image. By enlarging or reducing the selected subjects so that their relative sizes are consistent based on the actual size of the background image, and then arranging them, the size difference between the background and the subject can be compared.

[0114] Furthermore, it is not always necessary to automatically arrange the subjects. The user can arrange the subjects similarly to the process in step S2232. Furthermore, when interactively generating a composite image that enables relative comparison with the user, it is desirable to detect the largest subject each time the user selects a subject, and to generate and display the composite image while performing zooming in / out processing so that the largest subject does not fall outside the image.

[0115] If the background image has different object-side pixel sizes or distance information depending on the position in the image, the relative size of the subject can be changed depending on the position of the subject placed by the user on the background image. For example, if the background image has depth, the subject may be displayed at a relatively large size when placed in front, and at a relatively small size when placed far away, depending on the distance.

[0116] In step S224, the image processing unit 22 outputs the generated relative image to the output destination device via the output unit 15 and ends the relative image generation process. Alternatively, the generated related image is stored in the storage unit 13 and ends the related image generation process.

[0117] According to this typical embodiment, by calculating conversion information for converting into actual size based on multiple captured images and generating an image in which the subjects in the multiple captured images are arranged after being enlarged / reduced based on the conversion information, an image capable of relative size comparison in consideration of the actual size can be generated.

[0118] [Third exemplary embodiment]

[0119] In the first and second exemplary embodiments, a configuration is described in which the conversion information calculation processing and the output image generation processing for full-size output or the relative image generation processing are performed by the same image processing device. The conversion information calculation processing can be embedded in the image capture device as part of the image capture device and performed by the image capture device, and the actual size image generation processing and the relative image generation processing can be embedded in an output device (such as a printer or display device) and performed by the output device. In view of the foregoing, in the present exemplary embodiment, an image processing device is described in which the conversion information calculation processing, the actual size image generation processing, and the relative image generation processing are embedded in the image capture device and the output device, respectively.

[0120] <Camera Equipment>

[0121] Will refer to Figure 6 The structure of the digital camera 300 is described below. Figure 6As shown, in the present exemplary embodiment, the conversion information calculation processing unit is included in the image processing unit 33 .

[0122] The imaging optical system 30 includes a lens unit included in the digital camera 300 or a lens device that can be attached to a camera body unit, and forms an optical image of the subject on the image sensor 31. The imaging optical system 30 includes a plurality of lenses arranged along the direction of the optical axis 30a, and includes an exit pupil 30b at a position at a predetermined distance from the image sensor 31. In this specification, the direction parallel to the optical axis 30a is defined as the z-direction (depth direction). In other words, the depth direction is the direction in which the subject exists in real space with the position of the digital camera 300 as a reference. In addition, the direction perpendicular to the optical axis 30a and parallel to the horizontal direction of the image sensor 31 is defined as the x-direction, and the direction perpendicular to the optical axis 30a and parallel to the vertical direction of the image sensor 31 is defined as the y-direction.

[0123] The image sensor 31 is, for example, a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. The image sensor 31 performs photoelectric conversion on the subject image formed on the imaging plane via the imaging optical system 30 and outputs an image signal related to the subject image. As described below, the image sensor 31 of this exemplary embodiment has a function of outputting a signal that enables distance measurement using the imaging plane phase difference method. In addition to outputting a captured image, the image sensor 31 also outputs a parallax signal used to generate distance information indicating the distance from the imaging device to the subject (subject distance).

[0124] The control unit 32 includes a central processing unit (CPU) or a microprocessor and controls the operation of the components included in the digital camera 300. For example, the control unit 32 performs automatic focusing (automatic focus adjustment: AF) when shooting, changes the focus (focus) position, changes the F value (aperture value), and imports images. The control unit 32 also controls the image processing unit 33, the storage unit 34, the operation input unit 35, the display unit 36, and the communication unit 37.

[0125] The image processing unit 33 performs various types of image processing included in the digital camera 300 .

[0126] The image processing unit 33 includes an image generating unit 330, a depth image generating unit 331, and a conversion information calculating unit 332. The image processing unit 33 includes a memory used as a work area for image processing.

[0127] The image processing unit 33 may be constituted by a CPU and a memory storing a calculation processing program, in addition to a configuration using a logic circuit.

[0128] The image generation unit 330 performs various types of signal processing on the image signal output from the image sensor 31, such as noise removal, demosaicing, brightness signal conversion, aberration correction, white balance adjustment, and color correction. The image data (captured image) output from the image generation unit 330 is stored in the memory or storage unit 34 and is used by the control unit 32 for image display on the display unit 36 ​​or output to an external device via the communication unit 37.

[0129] The depth image generation unit 331 generates a depth image (depth distribution information) indicating the distribution of depth information based on an acquired signal related to a distance measurement image signal acquired from the image sensor 31, which will be described below. Here, the depth image is two-dimensional information in which the value stored in each pixel is the subject distance of the subject existing in the area corresponding to the pixel in the captured image.

[0130] In addition, the conversion information calculation unit 332 uses the control information used during imaging, information related to the image sensor 31, and correction information generated based on the design information of the imaging optical system 30 to calculate conversion information, which is used to calculate the size in space at the desired subject distance obtained by converting the size of a pixel in the captured image.

[0131] The storage unit 34 is a non-volatile recording medium on which captured image data, correction information required by the conversion information calculation unit 332, intermediate data generated during the operation process of each block, and parameters to be referenced in the operation of the image processing unit 33 or the digital camera 300 are recorded.

[0132] The storage unit 34 may be any recording medium as long as the processing capability permitted in the execution of the process is ensured, reading and writing can be performed at high speed, and the recording medium has a large capacity. For example, a flash memory is desirable.

[0133] The operation input unit 35 is a user interface such as a dial, button, switch, or touch panel that detects information input or setting change operation input performed on the digital camera 300. Upon detecting the performed operation input, the operation input unit 35 outputs a corresponding control signal to the control unit 32.

[0134] The display unit 36 ​​is a display device such as a liquid crystal display or an organic electroluminescent (EL) display. The display unit 36 ​​is used to check the composition during image capture by displaying a live view image of the captured image under the control of the control unit 32, and is also used to present various setting screens and message information. Furthermore, by integrally forming the touch panel serving as the operation input unit 35 and the display surface of the display unit 36, a display function and an input function can be provided in combination.

[0135] The communication unit 37 is a communication interface included in the digital camera 300 to enable information transmission and reception with an external device. The communication unit 37 may be configured to transmit the obtained captured image, depth information, size, coordinate information, and size measurement accuracy to another device.

[0136] <Structure of Image Sensor>

[0137] Next, refer to Figure 7A and Figure 7B An example of the structure of the above-mentioned image sensor 31 will be described.

[0138] like Figure 7A As shown, the image sensor 31 includes a plurality of pixel groups 310 arranged in a linked manner, and each pixel group 310 includes 2×2 pixels to which different color filters are applied. As shown in the enlarged view, a red filter (R), a green filter (G), and a blue filter (B) are arranged in the pixel group 310. An image signal representing color information of any one of R, G, and B is output from each pixel (photoelectric conversion element). In this exemplary embodiment, a description will be given of the color filters having the following characteristics: Figure 7A The illustrated description of the distributed configuration is provided as an example, but it should be readily understood that implementation of the present invention is not limited thereto.

[0139] In order to realize the distance measurement function of the imaging plane phase difference distance measurement method, in the image sensor 31 of the present exemplary embodiment, one pixel (photoelectric conversion element) includes Figure 7A More specifically, as shown in FIG. Figure 7B As shown, each pixel includes a light guiding layer 313 and a light receiving layer 314 . The light guiding layer 313 includes a micro lens 311 and a color filter 312 . The light receiving layer 314 includes a first photoelectric conversion unit 315 and a second photoelectric conversion unit 316 .

[0140] In the light guide layer 313, the microlenses 311 are configured to efficiently guide the light beam incident on the pixel to the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316. In addition, the color filter 312 for transmitting light within a predetermined wavelength band transmits only light within the wavelength band of any one of R, G, and B, and guides the light to the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 at the subsequent stage.

[0141] The light-receiving layer 314 is equipped with two photoelectric conversion units (a first photoelectric conversion unit 315 and a second photoelectric conversion unit 316) that convert received light into analog image signals. Two types of signals output from these two photoelectric conversion units are used for distance measurement. In other words, each pixel of the image sensor 31 similarly includes two photoelectric conversion units arranged horizontally, and uses an image signal composed of signals output from the first photoelectric conversion unit 315 and an image signal composed of signals output from the second photoelectric conversion unit 316 across all pixels. In other words, the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 each partially receive the light beam incident on the pixel via the microlens 311. As a result, the two types of image signals ultimately obtained become a pupil-divided image group associated with light beams that have passed through different regions of the exit pupil of the imaging optical system 30. The signal obtained by synthesizing the image signals photoelectrically converted by the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 in each pixel is equivalent to the image signal (for viewing) output from a single photoelectric conversion unit in a configuration where each pixel includes only one photoelectric conversion unit.

[0142] With this structure, the image sensor 31 of this exemplary embodiment can output both a viewing image signal and a distance measurement image signal (two types of pupil-segmented images). In this exemplary embodiment, the description will be given assuming that all pixels of the image sensor 31 each include two photoelectric conversion units and are configured to output high-density depth information. However, implementation of the present invention is not limited to this, and only a portion of the pixels may include multiple photoelectric conversion units. Alternatively, in a configuration in which only one photoelectric conversion unit is provided in each pixel, the photoelectric conversion units that receive light beams that have passed through different regions of the exit pupil may be arranged in a portion of the image sensor 31 and may be arranged in different pixels.

[0143] <Distance Measurement Principle of the Imaging Surface Phase Difference Distance Measurement Method>

[0144] Will refer to Figure 8A and Figure 8B The principle of calculating the subject distance based on the pupil-divided image group output from the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 , performed in the digital camera 300 of the present exemplary embodiment, will be described.

[0145] Figure 8A 3 is a schematic diagram illustrating the exit pupil 30 b of the imaging optical system 30 and light beams to be received by the first photoelectric conversion units 315 of the pixels in the image sensor 31 . Figure 8B is a schematic diagram illustrating a light beam to be similarly received by the second photoelectric conversion unit 316 .

[0146] Figure 8A and Figure 8B The microlens 311 shown is arranged so that the exit pupil 30B and the light receiving layer 314 are in an optical conjugate relationship. The light beam that has passed through the exit pupil 30b of the imaging optical system 30 is condensed by the microlens 311 and guided to the first photoelectric conversion unit 315 or the second photoelectric conversion unit 316. At this time, the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 are respectively as shown in FIG. Figure 8A and Figure 8B The first photoelectric conversion unit 315 receives the light beam that has passed through the first pupil area 320 , and the second photoelectric conversion unit 316 receives the light beam that has passed through the second pupil area 340 .

[0147] The plurality of first photoelectric conversion units 315 included in the image sensor 31 primarily receive the light beams that have passed through the first pupil region 320 and output first image signals. Simultaneously, the plurality of second photoelectric conversion units 316 included in the image sensor 31 primarily receive the light beams that have passed through the second pupil region 340 and output second image signals. Based on the first image signals, the intensity distribution of the image formed on the image sensor 31 by the light beams that have passed through the first pupil region 320 can be obtained. Furthermore, based on the second image signals, the intensity distribution of the image formed on the image sensor 31 by the light beams that have passed through the second pupil region 340 can be obtained.

[0148] The relative position shift amount between the first image signal and the second image signal (so-called parallax amount) becomes a value corresponding to the defocus amount. Figure 8C 、 Figure 8D and Figure 8E To illustrate the relationship between the amount of parallax and the amount of defocus. Figure 8C 、 Figure 8D and Figure 8E 3 and 4 are schematic diagrams illustrating the image sensor 31 and the imaging optical system 30 of the present exemplary embodiment. In these figures, a first light beam 321 passes through a first pupil area 320 , and a second light beam 341 passes through a second pupil area 340 .

[0149] Figure 8C A focused state is shown, and the first light beam 321 and the second light beam 341 converge on the image sensor 31. At this time, the amount of parallax between the first image signal formed by the first light beam 321 and the second image signal formed by the second light beam 341 becomes 0. Figure 8D The state in which defocusing occurs in the negative direction of the z-axis on the image side is shown. At this time, the parallax amount between the first image signal formed by the first light beam and the second image signal formed by the second light beam does not become 0 and has a negative value. Figure 8E1 shows a state where defocusing occurs in the positive direction of the z-axis on the image side. At this time, the parallax amount between the first image signal formed by the first light beam and the second image signal formed by the second light beam has a positive value. Figure 8D and Figure 8E Comparison between the two reveals that the direction of positional offset switches depending on whether the defocus amount is positive or negative. Furthermore, it can be seen that positional offset occurs according to the imaging relationship (geometric relationship) of the imaging optical system depending on the defocus amount. The parallax amount, which is the positional offset between the first and second image signals, can be detected using the region-based matching method described below.

[0150] <Image Generation and Conversion Information Calculation Processing>

[0151] Regarding image generation and conversion information calculation processing of an imaged object to be executed by the digital camera 300 of the present exemplary embodiment having the above-described configuration, reference will be made to FIG. Figure 9A The specific processing is explained with reference to the flowchart of FIG.

[0152] In step S331, the control unit 32 performs processing in a manner of capturing an image using the set image capture settings (such as focus position, aperture, and exposure time). More specifically, the control unit 32 controls the image sensor 31 to capture an image and transmit the obtained captured image to the image processing unit 33, and controls the captured image to be stored in the memory. The captured image includes two types of signals, namely, an image signal S1 composed of a signal output from the first photoelectric conversion unit 315 included in the image sensor 31 and an image signal S2 composed of a signal output from the second photoelectric conversion unit 316. Alternatively, two types of signals may be output from the image sensor 31, namely, a signal obtained by mixing (adding) the signals output from the first photoelectric conversion unit 315 and the second photoelectric conversion unit 316 using, for example, floating diffusion, and a signal (S1 or S2) output from either photoelectric conversion unit.

[0153] In step S332, the image processing unit 33 generates a review image based on the captured image. More specifically, the image generation unit 330 of the image processing unit 33 first generates a Bayer array image by adding the pixel values ​​of the pixels of the image signals S1 and S2. The image generation unit 330 then generates a review image by demosaicing the Bayer array image as an RGB color image. Demosaicing is performed based on the color filters arranged on the image sensor, and any demosaicing method can be used. Furthermore, the image generation unit 330 performs processing such as noise removal, brightness signal conversion, aberration correction, white balance adjustment, and color correction to generate a final review image, and stores the generated final review image in memory.

[0154] In step S333, the image processing unit 33 generates a depth image (depth distribution information) based on the acquired captured image. The depth image generation unit 331 performs processing related to the generation of the depth image. Figure 9B The flowchart of FIG. 1 is used to illustrate the processing related to depth image generation.

[0155] In step S3331, the depth image generation unit 331 performs light intensity correction processing on the image signals S1 and S2. At the peripheral field angles of the imaging optical system 30, the light intensity balance between the image signals S1 and S2 is disrupted due to vignetting caused by the different shapes of the first pupil area 320 and the second pupil area 340. Therefore, in this step, the depth image generation unit 331 uses, for example, a light intensity correction value pre-stored in memory to perform light intensity correction on the image signals S1 and S2.

[0156] In step S3332, the depth image generation unit 331 performs a process of reducing the noise generated during the conversion in the image sensor 31. Specifically, the depth image generation unit 331 achieves noise reduction by applying a filtering process to the image signal S1 and the image signal S2. Generally, in a high-frequency region where the spatial frequency is high, the signal-to-noise ratio (S / N) is low and the noise component increases relatively. Therefore, the depth image generation unit 331 performs a process of applying a low-pass filter, wherein with this low-pass filter, the higher the spatial frequency, the lower the pass rate becomes. Because the desired result is sometimes not obtained in the light amount correction in step S3331 due to errors in the manufacture of the imaging optical system 30, the depth image generation unit 331 desirably applies a band-pass filter that blocks the DC component and has a low pass rate for high-frequency components.

[0157] In step S3333, based on image signal S1 and image signal S2 (collectively referred to as phase difference signals), the depth image generation unit 331 calculates the amount of parallax between these images. Specifically, the depth image generation unit 331 sets a target point corresponding to the representative pixel information in image signal S1 and a reference area centered on the target point. The reference area can be a rectangular area, such as a square area centered on the target point and having a predetermined length on each side. Next, the depth image generation unit 331 sets a reference point in image signal S2 and a reference area centered on the reference point. The reference area has the same size and shape as the reference area described above. While sequentially moving the reference point, the depth image generation unit 331 calculates the correlation between the image included in the reference area of ​​image signal S1 and the image included in the reference area of ​​image signal S2, and identifies the reference point with the highest correlation as the corresponding point in image signal S2 corresponding to the target point. The relative positional offset between the corresponding point identified in this manner and the target point is the amount of parallax at the target point.

[0158] By calculating the disparity amount while sequentially changing the target point based on the representative pixel information in this manner, the depth image generation unit 331 calculates the disparity amount at a plurality of pixel positions defined by the representative pixel information. In this exemplary embodiment, for simplicity, the number of pixel positions used for disparity amount calculation (the pixel groups included in the representative pixel information) is set to the same number as the number of view images in order to obtain depth information at the same resolution as the view image. As a correlation calculation method, methods such as normalized cross correlation (NCC), sum of squared differences (SSD), or sum of absolute differences (SAD) can be used.

[0159] Furthermore, by using a predetermined conversion coefficient, the calculated parallax amount can be converted into a defocus amount corresponding to the distance from the image sensor 31 to the focal point of the imaging optical system 30. When the predetermined conversion coefficient is represented by K and the defocus amount is represented by ΔL, the parallax amount d can be converted into the defocus amount by the following formula:

[0160] ΔL=K×d Formula (4).

[0161] The conversion coefficient K is set for each area based on information including the aperture value, the exit pupil distance, and the image height in the image sensor 31 .

[0162] The depth image generation unit 331 constructs two-dimensional information including the defocus amount calculated in this manner as pixel values, and stores the two-dimensional information in a memory as a depth image. In addition, the depth image generation unit 331 can store the distribution of the parallax amount in the calculated parallax state in the memory as a depth image. In this case, when the depth image is used for size calculation in a subsequent step, the depth image is converted into the defocus amount. In other words, the depth distribution information can be any of the distribution information of the parallax amount represented by the phase difference signal corresponding to the image up to the subject, the distribution information of the defocus amount of the subject in the image, and the distribution information of the subject distance up to the subject in the image.

[0163] In step S334, the conversion information calculation unit 332 performs a process of calculating conversion information for converting the size of one pixel in the captured image into the size occupied in the space at the desired subject distance. Figure 9C The processing related to the display of the measurement object is described with reference to the flowchart of FIG.

[0164] In step S3341, information such as the lens name, focal length, subject distance, and information related to the focus state is acquired from the control unit 32. Information related to the focus state includes, for example, the number of pulses of the motor that drives the focus lens for focusing. Furthermore, distortion aberration information and axial chromatic aberration information corresponding to the lens name are acquired from the storage unit 34. If the imaging optical system 30 is embedded in a separate storage unit and aberration information is stored in that storage unit, the aberration information can be acquired from the storage unit of the imaging optical system 30 via the control unit 32.

[0165] In step S3342 , correction of the focal length and the subject distance is performed as described in the first exemplary embodiment using the acquired imaging information, focus state information, and various types of aberration information.

[0166] In step S3343, based on the corrected focal length and the defocus amount calculated in the depth image generation process, the defocus amount ΔL can be converted into the subject distance by using the following lens formula in geometric optics:

[0167] 1 / A+1 / B=1 / F Formula (5).

[0168] In this formula, A represents the distance from the object plane to the principal point of the imaging optical system 30 (subject distance), B represents the distance from the principal point of the imaging optical system 30 to the image plane, and F represents the focal length of the imaging optical system 30. In other words, since the value of the distance B can be calculated from the defocus amount ΔL in the lens formula, the distance A from the subject to the imaging optical system 30 can be calculated based on the focal length setting during imaging.

[0169] Furthermore, the magnification m at each pixel used in imaging can be calculated based on the distance A and the distance B at each pixel position of the captured image.

[0170] In step S3343, by calculating the product of the pixel size p of the image sensor used during imaging and the magnification m at any pixel position, the object-side pixel size at any pixel position is calculated, and the actual size is obtained. By calculating the magnification at all pixel positions, the object-side pixel size at all pixel positions (object-side pixel size image) is obtained. This object-side pixel size image is stored as conversion information along with the captured image in storage unit 34, and the process ends. Instead of the object-side pixel size image, magnification information at all pixel positions (magnification information image) may be stored as conversion information along with the captured image in storage unit 34.

[0171] When using the digital camera 300, conversion information can be calculated for multiple areas, and a conversion information image can be obtained as described above, in which conversion information is calculated for all pixels and stored as pixel values. However, if it is desired to conserve the capacity of captured images, it is not necessary to always store the conversion information image, and a conversion information image with a smaller number of pixels than the captured image through reduction or thinning can be stored. Alternatively, conversion information can be stored only for a single point or multiple focus areas. In addition, object detection, such as face detection, can be performed, and a representative value (e.g., average value) of the conversion information in the detected object area can also be stored.

[0172] In this exemplary embodiment, the image sensor 31 is described as including a photoelectric conversion element for an imaging plane phase difference distance measurement method and capable of acquiring a view image and a depth image. However, in the implementation of the present invention, the acquisition of distance information is not limited to this. For example, distance information may be acquired using a stereo distance measurement method based on multiple captured images obtained from a binocular imaging device or multiple different imaging devices. Alternatively, for example, distance information may be acquired using a stereo distance measurement method using a light-emitting unit and an imaging device, or a method combining a time-of-flight (TOF) method and an imaging device.

[0173] <Output device>

[0174] Then, reference will be made to Figure 10A and Figure 10B A structure in which output image generation processing and related image generation processing are embedded in an output device such as a printing apparatus or a display device will be described. Figure 10A 3 is a block diagram showing the functional structure of the display device 301 in which the output image generation process and the related image generation process are embedded. Figure 10B 3 is a block diagram showing the functional structure of the printing apparatus 302 in which output image generation processing and related image generation processing are embedded.

[0175] The display device 301 or the printing apparatus 302 acquires the captured image and object-side pixel size information from the digital camera 300 via the input unit 11, and transmits the captured image and object-side pixel size information to the image processing unit 33'. The image processing unit 33' includes an output image generation unit 333 and a related image generation unit 334, and selects and executes either process. The processes to be executed by the output image generation unit 333 and the related image generation unit 334 are similar to the output image generation process and the related image generation process described in the first and second exemplary embodiments. In the case of the display device 301, the generated output image or related image is transmitted to the display unit 36 ​​and displayed thereon, and in the case of the printing apparatus 302, the generated output image or related image is transmitted to the printing unit 39 and printed. The generated output image or related image can be stored in the storage unit 34 and can also be transmitted to the outside via the communication unit 37.

[0176] As described above, according to the image processing apparatus of this exemplary embodiment, a highly accurate object-side pixel size can be calculated within the imaging apparatus, without having to transmit information related to the focus state of the imaging apparatus, etc., to the outside. Furthermore, the display device and the printing apparatus do not need to store correction information for calculating a highly accurate object-side pixel size based on the imaging information.

[0177] Furthermore, by embedding this processing in an imaging apparatus capable of performing imaging plane phase difference distance measurement, it is possible to generate an output image of any subject other than the focused subject converted so as to adopt full size in output on an output destination device.

[0178] [Modification of the Third Exemplary Embodiment]

[0179] In this typical embodiment, it is assumed that the following description is given: the image sensor 31 outputs a phase difference signal, includes a photoelectric conversion element so that the phase difference distance measurement of the imaging surface can be performed, and a depth image (depth distribution information) can be acquired in a plurality of areas of the captured image. However, the configuration is not limited to this, and the conversion information calculation process can also be embedded in a digital camera that calculates the subject distance of one focusing area or a digital camera that calculates the subject distance of a plurality of focusing areas. In the case where the subject distance that can be calculated in this manner is used for only one point, it is expected that the number of pixels per unit length is obtained from the calculated object side pixel size information, and the obtained number of pixels is written in the information in the Exif format. Reference will be made to Figure 9D The flow of this processing is described with reference to the flowchart of FIG.

[0180] The processing of step S3341 to step S3343 is similar to the processing of step S1221 to step S1223 in the first exemplary embodiment, but differs in that these processes are executed in a digital camera.

[0181] In step S3344, the number of pixels per unit length is calculated using the calculated object-side pixel size and recorded as resolution information in Exif format. 1 mm may be used as the unit length, but 1 inch may be used in consideration of printing, and the calculated number of pixels per unit length may be recorded as pixel per inch (PPI) information.

[0182] Furthermore, as described in the third exemplary embodiment, in addition to the focused area, the number of pixels per unit length may be calculated for each area detected in subject detection processing such as face detection and recorded.

[0183] By recording the number of pixels per unit length as attribute information of the captured image in association with the corresponding image, there is an advantage in not having to provide output image generation processing in the printing device. This is because ordinary printing devices have the function of reading PPI information and performing printing based on it.

[0184] As described above, according to the image processing apparatus of the modification example of the present exemplary embodiment, it is possible to acquire highly accurate object-side pixel size information from a captured image without changing the capacity of image pickup information in the conventional Exif format.

[0185] [Fourth exemplary embodiment]

[0186] Next, a fourth exemplary embodiment of the present invention will be described. In the first and second exemplary embodiments, the position on the captured image where the object-side pixel size can be calculated is limited to the focus area. If the focus area differs from the desired subject position, or if the desired subject is at a different distance from the focus area, the actual size of the desired subject cannot be accurately acquired. In view of the foregoing, in this exemplary embodiment, the object-side pixel size calculated when the focus position and the desired subject distance differ is corrected to generate a full-size image.

[0187] Will refer to Figure 11 Flowchart and 12A to 12C Output image generation processing for generating an output image converted in such a manner that a subject is output in full size on an output destination device according to the present exemplary embodiment will be described. Figure 12A The captured image input for generating an output image is shown. In this example, a description will be given of a case where the focus position is on the left eye of person 410 and an output image of person 420 is generated. Similar to other processes, the processing corresponding to this flowchart is implemented by, for example, control unit 16 or each component performing the following operations: control unit 16 reads out a corresponding processing program stored in non-volatile memory within control unit 16, loads the processing program into volatile memory within control unit 16, and executes the processing program.

[0188] In step S401, a focus area in the captured image is obtained from the captured image. For example, the focus area can be estimated by analyzing the captured image and performing edge extraction. Alternatively, the focus area frame (coordinate information indicating the focus area frame) set during image capture can be recorded and read out in Exif format. Figure 12B The acquired focus area is shown as focus area 430 .

[0189] In step S402, the image generation unit performs subject recognition on the captured image. As an example, the case of face detection will be described. By face detection processing, as shown in FIG. Figure 12C As shown, a face frame 411 of a person 410 and a main subject region 421 of a person 420 are detected.

[0190] In step S403 , feature quantities such as age and gender are calculated from the image of the detected face region.

[0191] In step S404, the calculated feature values ​​are compared according to a prepared rule to determine the main subject from among the detected faces. For example, a rule is defined to determine the subject with the estimated younger age as the main subject. In this case, person 420 is determined to be the main subject.

[0192] In step S405, the subject distance difference between the focused area 430 and the main subject area 421 is calculated. In the case of a digital camera that can acquire depth information, as described in the third exemplary embodiment, the subject distance difference can be easily calculated. On the other hand, since depth information cannot be acquired in the first and second exemplary embodiments, the distance difference is estimated based on the contrast or blur amount of the images of the focused area 430 and the main subject area 421.

[0193] In step S406 , the object side pixel size is corrected using the calculated distance difference, the subject distance of the focus area calculated in the conversion information calculation process, and the focal length, and the object side pixel size at the subject distance of the main subject area 421 is calculated.

[0194] In step S407 , output destination information is acquired, and then in step S408 , an output image converted in such a manner that the main subject region 421 is in full size in output on the output destination device of the main subject region 421 is generated.

[0195] Here, a description is given of an example case in which a subject existing at a position different from the focus position is automatically obtained, but the user may directly designate the main subject in step S404 without executing steps S402 and S403 .

[0196] As described above, according to the image processing apparatus of the present exemplary embodiment, it is possible to generate an output image converted in such a manner that a subject existing in an area other than a focused area is output in full size on an output destination device.

[0197] [Modification of the Fourth Exemplary Embodiment]

[0198] In this exemplary embodiment, a description is given of a case where, during the output image generation process, determination of a subject is performed so that the subject is converted into an image in full size when output on the output destination device (steps S401 to S406). However, determination of a subject so that the subject is converted into an image in full size when output on the output destination device (steps S401 to S406) may also be performed within the imaging apparatus described in the third exemplary embodiment. The object-side pixel size or the number of pixels per unit length of the main subject obtained as a result of this execution is recorded as attribute information of the captured image in association with the corresponding image. With this configuration, in the display apparatus and printing apparatus described in the third exemplary embodiment, an output image in which the main subject is converted into an image in full size when output on the output destination device, or a relative image that can be relatively compared based on actual size, can be generated.

[0199] In addition, in 12A to 12C If there are multiple subjects, the object-side pixel size or the number of pixels per unit length in each subject area may be added as attribute information of the captured image. In this case, the priority order may be set according to the rule described in step S404, and the object-side pixel size or the number of pixels per unit length in each subject area may be added as attribute information of the captured image.

[0200] [Fifth Exemplary Embodiment]

[0201] Next, a fifth exemplary embodiment of the present invention will be described. In the above exemplary embodiments, a case where the highly accurately calculated object-side pixel size is used for full-size image generation and relative image generation is described. In this exemplary embodiment, notification of size measurement will be described as a different application of the object-side pixel size.

[0202] Will refer to Figure 13 The structure of the digital camera 500 will be described. Figure 13 This is a block diagram showing the functional structure of a digital camera 500 in which notification information calculation processing is embedded. Figure 13 The configuration other than the illustrated image processing unit 53 is similar to that of the digital camera 300 described in the third exemplary embodiment, and thus description will be omitted.

[0203] Regarding notification information calculation processing for calculating notification information executed by the digital camera 500 of the present exemplary embodiment having such a configuration, reference will be made to FIG. Figure 14A The specific processing is described in the flowchart of FIG. , which is used to notify the size information related to the size of the subject when shooting. The process is executed by the control unit 32, or by each component under the instruction of the control unit 32.

[0204] This process starts when imaging starts, and in step S531, a display image for displaying a subject imaged by the digital camera on a display unit is generated. The display image to be generated is an image reduced in size according to the resolution of the display unit.

[0205] In step S532, the control unit 32 displays the generated display image on the display unit. In step S533, it is checked whether the shutter button is pressed to the first level for focusing (SW1). If SW1 is not pressed, the process returns to step S531 and similar processing is continued.

[0206] When SW1 is pressed, depth image generation is performed in step S333 as described in the third exemplary embodiment, and conversion information calculation is also performed in step S344. The number of pixels in the depth image does not need to be equal to the number of pixels of the image sensor, and may be equal to the number of pixels of the display image. It is desirable to reduce the computational load of the depth image generation process by reducing the number of pixels in the depth image to be generated.

[0207] In step S534, the control unit 32 performs calculation of notification information for notifying the image capturing mode in which the size measurement can be performed by displaying it on the display image. Figure 14B The notification information calculation process is described with reference to the flowchart of FIG.

[0208] In step S5331, the control unit 32 obtains the pixel count of the area where the focusing process was performed. Here, an example case in which the focusing area is rectangular will be described, but the focusing area is not limited to rectangles. The focusing area can have any shape as long as it is a graphical user interface (GUI) used to represent an area in an image. To obtain the pixel count of the focusing area, the control unit first obtains information regarding the rectangular area used for focusing and converts the length of at least one of the long and short sides into the number of pixels in the displayed image. By adding the object-side pixel size, which is the conversion information calculated in step S334, to the obtained number of pixels for each side, the size of the side of the focus frame at the focused subject distance is calculated. The size of the rectangular focus frame and its sides serves as notification information.

[0209] In step S5332, the control unit 32 superimposes the calculated notification information on the display image and generates a superimposed image for notification. Figure 15A As shown, a frame 551 of the focus area is displayed in a predefined color in an overlay manner, and the calculated object side size 552 of the rectangle is displayed in an overlay manner near the edge of the rectangle. Figure 15A As shown, it is desired to display the size in an overlapping manner near only one of the two sides in the horizontal direction, among the sides in the horizontal and vertical directions.

[0210] In this example, the processing of the depth image generation unit 331 is performed based on a digital camera equipped with an image sensor 31 capable of performing imaging plane phase difference distance measurement, as described in the third exemplary embodiment. However, the exemplary embodiment can also be applied to digital cameras that do not support the imaging plane phase difference distance measurement method. In the case of a digital camera that does not support the imaging plane phase difference distance measurement method, the object side size of the focus frame can be calculated and displayed using the subject distance information of the focus area.

[0211] On the other hand, when a sensor capable of measuring distances in multiple areas is used, as in the digital camera 500 supporting the imaging plane phase difference distance measurement method described in this exemplary embodiment, the object side size of an area other than the frame of the focused area can also be calculated. For example, Figure 15B As shown in the figure, we will explain the case where multiple people are present as subjects in the picture and their subject distances are different. In this case, multiple face detection frames can be acquired through face detection, and by using depth information, the object side pixel size can be calculated separately even if their subject distances are different. By adding the calculated number of pixels of the side of the face frame to the corresponding object side pixel size, the object side size of each face frame can be calculated, and it can be Figure 15B The notification information is displayed as shown.

[0212] In addition, in Figure 15C As shown, when there are multiple subjects in the screen, it is effective to display the rectangular frames in different colors according to the distance of the subjects. Figure 15C It is shown that the display color of the face frame 553 is different from the display colors of the face frame 554 and the face frame 555 .

[0213] As described above, according to the image processing apparatus of this exemplary embodiment, since the object side pixel size can be calculated by conversion information calculation processing, the photographer can be notified of the subject size during imaging or that imaging capable of acquiring size information is being performed.

[0214] [Sixth Exemplary Embodiment]

[0215] Next, the sixth exemplary embodiment of the present invention will be described. In the first exemplary embodiment, a case of displaying and printing full-size images was described, and in the second exemplary embodiment, a case of generating and outputting multiple images and images comparing the relative sizes of subjects in these images was described. In this exemplary embodiment, an application of image synthesis using object-side pixel size information will be described. As an example, a case will be described in which face detection is performed on a single image or multiple images, facial images that appear to have different sizes are extracted using object-side pixel size information, and image synthesis is performed. Furthermore, in this exemplary embodiment, the description will be given assuming a face as the object, but the same applies to general objects.

[0216] Regarding the image synthesis application to be executed in the present exemplary embodiment, reference will be made to Figure 16 The specific processing is explained in the flowchart of FIG. Figure 1 and Figure 3 ), control unit 32 ( Figure 6 and Figure 13), control unit 41 ( Figure 10A ) or control unit 42( Figure 10B Hereinafter, as an example, the process will be described below as a process executed by the control unit 16.

[0217] Similar to other processes, the processing corresponding to this flowchart is implemented, for example, by the control unit 16 or each component through the following operations: the control unit 16 reads out the corresponding processing program stored in the non-volatile memory within the control unit 16, loads the processing program into the volatile memory within the control unit 16, and executes the processing program.

[0218] In step S601, the control unit 16 receives a selection of an image including a subject to be synthesized based on an operation input by the user via the operation input unit 17. At this time, candidates for images including a subject to be synthesized can be displayed on the display 2 in a selectable manner using a method such as list display or sequential display.

[0219] In step S602, the control unit 16 or image processing unit 22 performs processing similar to that in step S222 on the selected image and calculates the object-side pixel size. If an image that does not include information for calculating the object-side pixel size is selected, the processing is terminated. Alternatively, information indicating that the object-side pixel size information was not calculated is stored, and the process proceeds to the next step.

[0220] In step S603, face detection processing is performed on the selected image, and coordinate information related to the facial region is output. This coordinate information specifies the rectangular region in which the face was detected, and includes information such as the coordinates of the upper left and lower right vertices of the rectangle, or the coordinates of the upper left vertex, width, and height of the rectangle. If multiple faces are detected in a single image, similar information is output for each of these multiple faces. Furthermore, the coordinates of the head center are estimated using the output coordinate information of the facial region.

[0221] In the estimation of the coordinates of the head center, facial part detection is further performed, and a position such as a position between eyebrows is detected. The position to be detected as the head center depends on each application.

[0222] Next, a circular area with a radius R [mm] specified based on the center position of the head calculated based on the object-side pixel size calculated in step S602, or a rectangular area circumscribed around the circle, is cropped from the image selected as the synthesis object. The user can specify a desired range as the range to be cropped here, but in order to save the user's labor for setting, it is desirable to preset a cropping range suitable for each detected object. The cropping shape is not limited to a circle or a rectangle and can be another arbitrary shape. For example, cropping can be performed along an area determined based on the outline of the detected subject.

[0223] If the image file recording the selected image does not include the metadata required to calculate the object-side pixel size information, for images for which calculation of the object-side pixel size information failed, the physical size of the rectangular area in which the face was detected is estimated by assuming the size of a typical person's face. The object-side pixel size is estimated based on the estimated physical size of the rectangular area and the number of pixels on each side of the rectangle. In this case, size differences between individual faces are not taken into account, and facial sizes are unified to the size assumed as a typical face size. To alleviate this problem, it is desirable to estimate age and gender during face detection and set a typical face size based on the estimated age and gender. Alternatively, facial part detection can be performed within the detected facial area, and the face size can be estimated based on positional information related to the facial parts, such as the distance between the eye centers. Facial part detection can also estimate the orientation of the face, and by estimating the face size based on the variation in the distance between the eye centers depending on the orientation of the face, the face size can be estimated more accurately than when object-side pixel size information is not available.

[0224] In this case, the error in the cropping range relative to the specified size depends on the detection result of the rectangular area size in face detection, and it is impossible to perform area cropping with the correct size. In the event that calculation of the object-side pixel size information fails, it is desirable to add information to the metadata of the image file including the cropped image in such a manner that it can be recognized that the cropping area is inaccurate.

[0225] Next, we will explain the case where the subject distance information is stored only for one point of the focus position in the calculation of the object side pixel size and multiple people appear in the image. There is no problem when all people are at the same distance from the focus position, but in the case of Figure 17A As shown, when a person is at different distances from the camera, the size of the person varies even if the person is the same size. Since the object-side pixel size can only be calculated at the focus position as described above, individual distances are not taken into account, and the face of a person appearing in front is cropped at a larger size, while the face of a person appearing in the distance is cropped at a smaller size.

[0226] To avoid this problem, the ratio of the rectangular area size in the face detection result for the person appearing at the focus position is calculated based on the rectangular area size in the face detection result for another person. Based on this ratio, the object side pixel size suitable for the distance of each person is estimated, and the cropping range is calculated based on the estimated object side pixel size. Figure 17B In this case, the error in the cropping range relative to the specified size depends on the detection result of the rectangular area size in face detection.

[0227] In cases where a face appears in a captured image but is not in focus, similar to images where calculation of object-side pixel size information fails, significant changes in the cropping range of the facial area can be prevented by assuming the typical face size of a person. This is done by storing the coordinates of the area being focused on as metadata and determining whether the face detection position and the focused area match on the screen (XY plane). Furthermore, the amount of change in the rectangular area detected during face detection during cropping is examined. If the change is significant, it can be determined that the face was out of focus during image capture and focus shift occurred. Information related to focus shift is added, and as described above, object-side pixel size information estimated based on the typical face size is further added.

[0228] It is desirable to add information in a manner that allows for differentiation between subjects cropped by calculating the object-side pixel size and subjects cropped without using the object-side pixel size. Furthermore, when performing image synthesis, as described below, it is desirable to enable the user to identify whether a cropped image is an image cropped by calculating the object-side pixel size. This information is stored in association with the cropped image.

[0229] If the resolution of a detected facial image is too low, cropping an area of ​​the same size will require image enlargement, resulting in a decrease in image quality. To avoid this problem, a minimum resolution must be set during face detection. Since the minimum resolution depends on the print size, the print settings described below will determine whether the required resolution is ultimately met.

[0230] In step S604, the user selects a desired facial region image from the cropped facial region images and arranges the selected facial region image at a desired position in the separately prepared working image area. Multiple faces can be selected and arranged from the same image, and if it is desired to arrange a face from another image, the processing in steps S601 to S604 is repeated. Figure 18A An example is shown in which faces of three children are selected, arranged, and displayed on the display 2 .

[0231] In the case where an image having a significantly lower resolution than other face region images is selected in the face region image selection, the control unit 16 may notify the user by issuing a warning. Figure 17A As shown, the detected face 173 is at a distance away from the focus position and appears small. Figure 17B In the case where the face 173 is cropped in such a manner as to have the same size as shown, the image is degraded because the face 173 ′ is enlarged.

[0232] In step S605, the control unit 16 receives an operation from the user via the operation input unit 17 to arrange composite objects other than faces in the work area. Composite objects are objects that decorate the composite image as the final product and can be images, icons, illustrations, or characters. The user can perform the operation to arrange each composite object in the work area in advance before face composition, or can arrange it during face composition. Figure 18B Shown in Figure 18A This is an example of a floral illustration placed around the faces of three children arranged in a decorative pattern.

[0233] In step S606, the control unit 16 receives an operation from the user via the operation input unit 17 to change the size of the arranged composite faces. The user can select each composite face individually and change its size. Since the cropped areas of the composite faces have the same physical size, it is desirable to make the sizes of the composite faces simultaneously changeable. Therefore, in this exemplary embodiment, upon receiving an operation to change the size of one selected face, the control unit 16 issues an instruction to the image processing unit 22 to change the sizes of the other composite faces by the same amount in a chained manner. The image processing unit 22 performs the same enlargement or reduction processing on the multiple images preset as a chain, and the control unit 16 displays the multiple images resulting from the processing on the display 2. Figure 18C The following shows how the size of the central face 182 is changed. Figure 18C As shown, the size of the face 182 is reduced, then as Figure 18D As shown, the sizes of the other faces 181 and 183 are changed in a chain-like manner. This is an example case where the sizes of only the faces other than the flower used as a decoration are changed in a chain-like manner. In the case where the decorative image also maintains the object-side pixel size, the size can be changed while maintaining the relative size with the face. The size of the composite object other than the face that maintains the object-side pixel size can be changed in a chain-like manner. In addition, by separating the storage area for storing the composite image to be resized from the storage area for storing the decorative image, it is also possible to determine the area from which the image is selected. In the case where it is also desired that the size of the decorative object be changed in a chain-like manner, the size can be changed in a chain-like manner by the user performing a chain setting.

[0234] In step S607, the control unit 16 receives a print setting instruction from the user via the operation input unit 17. In the print setting, the sheet size to be printed and the desired print size (desired output size) are set. By fixing the print sheet size to a typical A4 or A3 size and setting only the size at which printing is desired, print setting can be easily performed. Furthermore, full-size printing can be specified, and the print size and split printing are automatically calculated using the object-side pixel size so that the composited main subject becomes actual size.

[0235] When the desired print size differs from the print sheet size and split printing is performed (including the full-size printing described above), grid lines are displayed in a superimposed manner in the job image area, for example, so that the split positions can be discerned. By displaying the grid lines, the composite images to be split during printing can be discerned. Figure 19A An example is shown in which a grid line 191 is shown at a division position in the case where an A3-sized printout is to be obtained using an A4-sized print sheet. It is easily recognized that a face 192 exists on the division line and will be printed in a divided manner. Therefore, the user can easily move an important composite image such as a face image to a position where the image will not be divided when printed. For example, by Figure 19B As shown, by reducing the entire arrangement image and moving the reduced image to the right, the face 192 can be rearranged in a manner not to be divided when printed. In addition, control can be automatically performed in a manner that the face image is not arranged on the grid lines.

[0236] When the print sheet size and the desired size for enlarged printing are known, it is desirable to display the dividing grid lines in a superimposed manner in the job image area where the job is to be combined.

[0237] Furthermore, in print setting, when setting a desired print size, the resolution of the arranged composite images is checked, and composite images having a large image magnification ratio that causes a decrease in resolution when printing are extracted and notified in a discriminable manner.

[0238] As described above, the image composition application of this exemplary embodiment allows, in addition to composition using object-side pixel size information, also the ability to composite images without object-side pixel size information by estimating size information. By making the presence or absence of object-side pixel size information discriminable, the user can perform image composition while checking the accuracy of the subject's size.

[0239] [Modification of the Sixth Exemplary Embodiment]

[0240] A modification of the sixth exemplary embodiment will be described. As described in the sixth exemplary embodiment, in an image in which only distance information of the focus position is stored, multiple faces may exist at positions other than the focus position in the image. In this case, a process for enabling full-size printing of faces appearing at distances other than the focus position will be described.

[0241] The control unit 16 or image processing unit 22 estimates the object-side pixel size by assuming the typical face size of faces appearing at distances other than the focus position. The estimated object-side pixel size or PPI information is additionally stored in the image metadata along with the detected facial region information. By selecting the face you wish to print at full size when printing, you can perform near-full-size printing.

[0242] [Other Typical Embodiments]

[0243] The present invention can also be implemented by the following process: a program for implementing one or more functions of the above-described exemplary embodiments is supplied to a system or device via a network or storage medium, and one or more processors in a computer of the system or device reads and executes the program. In addition, the present invention can also be implemented by a circuit (e.g., an application-specific integrated circuit) for implementing the one or more functions.

[0244] The present invention is not limited to the above exemplary embodiments, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are added to set forth the scope of the present invention.

[0245] This application claims the benefit of Japanese Patent Application No. 2020-031079, filed February 26, 2020, and Japanese Patent Application No. 2020-183896, filed November 2, 2020, which are hereby incorporated by reference herein in their entireties.

Claims

1. An image processing device, comprising: an acquisition unit configured to acquire an image captured by the camera unit and camera information of the image when the image is captured; a calculation unit configured to calculate an object-side pixel size of a subject in an image based on the imaging information and a pixel size of the imaging unit; as well as an image processing unit configured to process the image captured by the camera unit based on the object-side pixel size, The acquiring unit acquires focus information indicating a focus state of a subject in an image as the imaging information. wherein the calculation unit calculates the object side pixel size based on the focus information, and In which, in the subject area where the calculation of the object side pixel size fails, the image processing unit estimates the object side pixel size of the subject area where the calculation of the object side pixel size fails by using the size of the subject area detected by the subject detection unit and the size of the subject area detected by the subject detection unit in the subject area where the object side pixel size is calculated, and calculates the estimated object side pixel size.

2. The image processing apparatus according to claim 1, in, The acquisition unit acquires output destination information about an output destination device for outputting an image, and The image processing unit generates an output image by converting the image based on the output destination information so that the subject of the image is output at full size on the output destination device, based on the object side pixel size calculated by the calculation unit.

3. The image processing apparatus according to claim 1, wherein The image processing unit records information in image data in a manner capable of distinguishing the object-side pixel size and the estimated object-side pixel size.

4. The image processing apparatus according to claim 1, wherein When a plurality of subjects having the object-side pixel size are synthesized into one image, the image processing unit changes the sizes of subjects having the object-side pixel size other than the subject while maintaining the relative sizes by selecting one subject from the plurality of subjects and changing the size of the subject. The image processing apparatus according to claim 1 , wherein: The image processing unit calculates the size of the subject by using the object side pixel size and the number of pixels of the subject detected by the subject detection, and when the calculated size of the subject is larger or smaller than the pre-stored size of the subject by a threshold value or more, the image processing unit performs subject cropping based on the pre-stored size of the subject. The image processing apparatus according to claim 1 , wherein: When full-size printing is set, the image processing unit calculates the size of a print using the object-side pixel size of the main subject, and generates divided images for printing based on the set sheet size.

7. The image processing apparatus according to claim 1, wherein The image processing unit converts images of a plurality of subjects in an image according to actual sizes by using the object-side pixel size of each of the plurality of subjects, and generates an image for relative comparison of sizes of the plurality of subjects.

8. The image processing apparatus according to any one of claims 1 to 7, wherein: The calculation unit calculates the object-side pixel size based on at least any one of an amount of change in focal length corresponding to the focus information, a subject distance, and a principal point position of an optical system.

9. The image processing apparatus according to any one of claims 1 to 7, wherein: The calculation unit calculates the object-side pixel size at the subject distance by using the subject distance to the focused subject as the imaging information.

10. The image processing apparatus according to any one of claims 1 to 7, wherein: The calculation unit calculates the object-side pixel size based on an imaging magnification corresponding to the focus information.

11. The image processing apparatus according to any one of claims 1 to 7, in, The acquiring unit acquires depth distribution information corresponding to the image, and The calculation unit uses the depth distribution information to calculate the object-side pixel size at each pixel position of the depth distribution information.

12. The image processing apparatus according to claim 11, wherein The calculation unit acquires a subject distance to at least one subject existing at an arbitrary position in an image using the depth distribution information, and calculates the object-side pixel size at the subject distance using the subject distance.

13. The image processing apparatus according to claim 11, wherein The depth distribution information is at least any one of distribution information of a parallax amount represented by a phase difference signal corresponding to an image, distribution information of a defocus amount of a subject existing in the image, and distribution information of a subject distance to the subject existing in the image.

14. The image processing apparatus according to claim 2, wherein: The output destination information is information regarding at least any one of a print sheet size, a display screen size, and a display screen resolution.

15. The image processing device according to any one of claims 1 to 7, further comprising a control unit configured to calculate at least one number of pixels per unit length based on the object-side pixel size calculated by the calculation unit, and record the number of pixels per unit length in association with the corresponding image.

16. The image processing apparatus according to claim 15, wherein The control unit records the pixel per inch information (PPI) as the number of pixels per unit length together with the image data of the corresponding image into an image file in the Exchangeable Image File Format (Exif) format.

17. The image processing apparatus according to claim 15, in, The image processing unit calculates the object-side pixel size of at least one area of ​​the image or the number of pixels per unit length by performing face detection processing on the image and using the subject distance of the detected face area, and The control unit records at least one of the object-side pixel size and the number of pixels per unit length in association with the corresponding image.

18. The image processing apparatus according to claim 15, wherein The control unit calculates the size of the area designated by a graphical user interface (GUI) to be superimposed on an image from the object-side pixel size based on the number of pixels of the area designated by the GUI, and displays the size on the display unit.

19. A camera device comprising: an image sensor configured to acquire phase difference signals from a plurality of photoelectric conversion units; The image processing device according to any one of claims 1 to 18; as well as Conversion unit.

20. A display device comprising: The image processing device according to any one of claims 1 to 18; as well as A display unit is configured to display an image.

21. A printing device comprising: The image processing device according to any one of claims 1 to 18; as well as A printing unit is configured to print an image.

22. An image processing method, comprising: An acquisition step for acquiring an image captured by the camera unit and camera information when the image is captured; a calculating step for calculating the object-side pixel size of the subject in the image based on the imaging information and the pixel size of the imaging unit; as well as an image processing step for processing the image captured by the camera unit based on the object side pixel size, wherein, in the acquisition step, focus information indicating a focus state of a subject in an image is acquired as the imaging information; wherein, in the calculation step, the object side pixel size is calculated based on the focus information, and Wherein, in the subject area where the calculation of the object side pixel size fails, in the image processing step, the object side pixel size of the subject area where the calculation of the object side pixel size fails is estimated by using the size of the subject area detected by the subject detection unit and the size of the subject area detected by the subject detection unit in the subject area where the object side pixel size is calculated, and the estimated object side pixel size is calculated.

23. A computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 22.

24. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the steps of the image processing method according to claim 22 are implemented.

Images