Image processing device, image processing method, program, and storage medium

Abstract

An image processing apparatus according to the present invention includes: an input component configured to input a distance information distribution calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging component; an estimation component configured to estimate a depth direction in the image according to an imaging condition of the imaging component; and a determination component configured to determine an evaluation value representing a degree of deviation of the optical system and the imaging element from a design position according to a relationship between the distance information distribution and the estimated depth direction.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, and particularly, to information related to long-term changes in an optical system and an imaging element, and information related to the posture of the image processing apparatus. Background Art

[0002] Conventionally, there has been known a technique for diagnosing changes in the relative positional relationship between a pair of stereo cameras due to long-term changes or the like by referring to distance information obtained from the pair of stereo cameras, and for supporting calibration of the stereo cameras. For example, Patent Document 1 discloses the following method. A subject is imaged with a stereo camera mounted on the head of a robot in a predetermined positional relationship on a substantially flat surface provided with a texture for diagnosis, and flatness is obtained by calculation of distance information from the obtained parallax image. Then, the obtained flatness and a predetermined reference amount are compared with each other to determine whether calibration is required.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2004-306249 Summary of the Invention

[0006] Solution to the Problem

[0007] In addition, in a digital camera that is not mounted on a robot and is used by a typical user, a lens as an optical system or an imaging element using a CMOS may change from its attached position (designed position) at the time of manufacture due to long-term changes or the like. When the lens or the imaging element is tilted, the relationship between the actual distance and the depth of field deviates, and an image that is not desired by the user is obtained. Therefore, also for a digital camera used by a typical user, a method for determining whether calibration of the lens or the imaging element is required and a solution therefor are desired.

[0008] Further, in imaging using a digital camera, when the digital camera faces a subject to be imaged and images the subject, even if the lens or the imaging element is calibrated, a tilted imaging device or an inappropriate imaging distance prevents a good captured image from being obtained. In particular, tilting or distance error in the depth direction causes the captured image (of the subject object) to become blurred.

[0009] Therefore, an object of the present invention is to provide an image processing apparatus that enables notification of at least one of information related to tilting of a lens or an imaging element and information related to the position or posture of an imaging device based on a distance information distribution corresponding to the distance to a subject.

[0010] To solve the above problems, an image processing apparatus according to the present invention includes: an input component configured to input a distance information distribution calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging component; an estimation component configured to estimate a depth direction in the image based on imaging conditions of the imaging component; and a determination component configured to determine an evaluation value indicating a degree of deviation of the optical system and the imaging element from a design position based on a relationship between the distance information distribution and the estimated depth direction.

[0011] In addition, an image processing apparatus according to the present invention includes: an input component configured to input a distance information distribution calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging component; an estimation component configured to estimate a depth direction in the image based on imaging conditions of the imaging component; and a determination component configured to determine an evaluation value indicating a degree of deviation in a depth direction of a subject in the image based on a relationship between the distance information distribution and the estimated depth direction.

[0012] In addition, an image processing apparatus according to the present invention includes: a first acquisition component configured to acquire imaging conditions related to an image captured by an imaging component, the imaging conditions including at least an F-number and a conversion coefficient for converting an image offset amount into a defocus amount; a second acquisition component configured to acquire a distance information distribution that is a distribution of distance information corresponding to respective regions of the image captured by the imaging component; and an image processing component configured to normalize the distance information distribution based on the F-number and the conversion coefficient.

[0013] In addition, an image processing method according to the present invention includes: an input step of inputting a distance information distribution calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging component; an estimation step of estimating a depth direction in the image based on imaging conditions of the imaging component; and a determination step of determining an evaluation value indicating a degree of deviation of the optical system and the imaging element from a design position based on a relationship between the distance information distribution and the estimated depth direction.

[0014] In addition, an image processing method according to the present invention includes: an input step for inputting a distance information distribution, which is calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging component; an estimation step for estimating a depth direction in the image according to the imaging conditions of the imaging component; and a determination step for determining an evaluation value indicating a deviation degree in the depth direction of a subject in the image according to the relationship between the distance information distribution and the estimated depth direction.

[0015] In addition, an image processing method according to the present invention includes: a first acquisition step for acquiring imaging conditions related to an image captured by an imaging component, where the imaging conditions at least include an F-number and a conversion coefficient for converting an image offset amount into a defocus amount; a second acquisition step for acquiring a distance information distribution, which is a distribution of distance information corresponding to each region of the image captured by the imaging component; and an image processing step for normalizing the distance information distribution based on the F-number and the conversion coefficient.

[0016] According to the present invention, an image processing device capable of confirming a deviation degree of an optical system and an imaging element with respect to a design position can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram showing an example of a functional structure of an image processing device according to an embodiment of the present invention.

[0018] Figure 2 is a block diagram showing an example of a functional structure of a digital camera according to an embodiment of the present invention.

[0019] Figure 3 is a block diagram showing an example of a functional structure of a computer according to an embodiment of the present invention.

[0020] Figure 4A shows an example of a structure of an imaging unit according to an embodiment of the present invention.

[0021] Figure 4B shows an example of a structure of an imaging unit according to an embodiment of the present invention.

[0022] Figure 5A is a flowchart showing an operation of an image processing device according to an embodiment of the present invention.

[0023] Figure 5B is a flowchart showing an operation of an image processing device according to an embodiment of the present invention.

[0024] Figure 5C is a flowchart showing an operation of an image processing device according to an embodiment of the present invention.

[0025] Figure 6 Shows an image for recording a still image according to an embodiment of the present invention.

[0026] Figure 7 Shows a defocus map according to an embodiment of the present invention.

[0027] Figure 8 Is a block diagram showing an example of the functional structure of the image processing unit 306 according to an embodiment of the present invention.

[0028] Figure 9 Shows a plane where the defocus amount becomes zero according to an embodiment of the present invention.

[0029] Figure 10 Shows a defocus map when the focus plane is normal according to an embodiment of the present invention.

[0030] Figure 11A Shows a phenomenon that occurs when the optical system and the imaging element are displaced from the designed positions according to an embodiment of the present invention.

[0031] Figure 11B Shows a phenomenon that occurs when the optical system and the imaging element are displaced from the designed positions according to an embodiment of the present invention.

[0032] Figure 12 Shows a defocus map when the focus plane is tilted according to an embodiment of the present invention.

[0033] Figure 13 Shows an evaluation value representing the degree of deviation according to an embodiment of the present invention.

[0034] Figure 14 Shows a notification to the user according to an embodiment of the present invention.

[0035] Figure 15 Shows the estimated results of the vanishing point and the depth direction according to an embodiment of the present invention.

[0036] Figure 16 Shows a histogram of the defocus map according to an embodiment of the present invention.

[0037] Figure 17A Shows an image for recording a still image, a defocus map, and a histogram of the defocus map in a portrait scene according to an embodiment of the present invention.

[0038] Figure 17B Shows an image for recording a still image, a defocus map, and a histogram of the defocus map in a portrait scene according to an embodiment of the present invention.

[0039] Figure 17CShows an image for recording a still image in a portrait scene according to an embodiment of the present invention, a defocus map, and a histogram of the defocus map.

[0040] Figure 18 Shows the optical vignetting characteristics of an optical system according to a first embodiment of the present invention.

[0041] Figure 19A Is a block diagram showing an example of the hardware structure of a camera device 1900 and a lens device 1913 according to a second embodiment of the present invention.

[0042] Figure 19B Is a block diagram showing an example of the functional structure of a camera device 1900 according to a second embodiment of the present invention.

[0043] Figure 20 Is a block diagram showing an example of the hardware structure of a gimbal device 2000 according to a second embodiment of the present invention.

[0044] Figure 21A Shows an imaging method for imaging social infrastructure according to a second embodiment of the present invention.

[0045] Figure 21B Shows an imaging method for imaging social infrastructure according to a second embodiment of the present invention.

[0046] Figure 22 Is a flowchart of the operation of an imaging system according to a second embodiment of the present invention.

[0047] Figure 23 Shows a switch 2007 according to a second embodiment of the present invention.

[0048] Figure 24A Is a diagram related to the rotation control of a camera device 1900 according to a second embodiment of the present invention.

[0049] Figure 24B Is a diagram related to the rotation control of a camera device 1900 according to a second embodiment of the present invention.

[0050] Figure 24C Is a diagram related to the rotation control of a camera device 1900 according to a second embodiment of the present invention.

[0051] Figure 25 Shows an example of the structure of Table 2515 according to a second embodiment of the present invention. Detailed Description of the Invention

[0052] [First Embodiment]

[0053] Hereinafter, an image processing apparatus, an image processing method, and an image processing program according to a first embodiment of the present invention will be described in detail with reference to some drawings. AsFigure 1 As shown, an example of applying the present invention to an image processing apparatus 100 will be described. A digital camera 101 as an example of a imaging apparatus and a computer 102 as an example of an image processing apparatus are communicably connected to each other via a communication circuit 103. However, in the following description, the processing performed by the computer 102 may also be performed by the digital camera 101. In addition, the digital camera 101 may be any given electronic device having a imaging function, and the computer 102 may be any given electronic device or a computer in a server device capable of performing the processing described below. The computer 102 may also be a mobile computer or a desktop computer.

[0054] Figure 2 FIG. 5 is a block diagram showing an example of the functional structure of the digital camera 101 according to an embodiment of the present invention. The system control unit 201 is, for example, a CPU, reads operation programs of blocks included in the digital camera 101 from a ROM 202, loads these operation programs into a RAM 203, and executes these operation programs to control the operations of the blocks included in the digital camera 101. The ROM 202 is a rewritable non-volatile memory, and stores not only operation programs of the blocks included in the digital camera 101 but also parameters required for the operations of the blocks. The RAM 203 is a rewritable volatile memory, and serves as a temporary storage area for data output during the operations of the blocks included in the digital camera 101.

[0055] The optical system 204 forms a field image on the imaging unit 205. The imaging unit 205 is, for example, an imaging element such as a CCD or a CMOS sensor, photoelectrically converts an optical image formed on the imaging element of the imaging unit 205 by the optical system 204, and outputs the obtained analog image signal to an A / D conversion unit 206. In addition, an IS mechanism for reducing the influence of camera shake is provided in each of the optical system 204 and the imaging unit 205. The A / D conversion unit 206 applies A / D conversion processing to the input analog image signal, and outputs the obtained digital image data to the RAM 203 for storage.

[0056] The image processing unit 207 applies various types of image processing such as white balance adjustment, color interpolation, reduction / enlargement, and filtering to the image data stored in the RAM 203.

[0057] The recording medium 208 is a removable memory card or the like, on which images processed by the image processing unit 207 stored in the RAM 203 and images A / D-converted by the A / D conversion unit 206 are recorded as recorded images.

[0058] The communication unit 209 transmits image data files and the like recorded on the recording medium 208 to an external device in a wired or wireless manner.

[0059] The display unit 210 displays the image data obtained by imaging or the image data read from the recording medium 208, etc., or displays various menu screens. The display unit 210 also serves as an electronic viewfinder by displaying a live view image.

[0060] The operation unit 211 is a group of input devices for a user to input various instructions or settings, etc. to the digital camera 101, and includes keys and buttons such as a shutter button, a menu button, arrow keys, and a decision key, etc. typically provided in a digital camera. Additionally, when the display unit 210 is a touch display, the display unit 210 also serves as the operation unit 211. Note that the operation unit 211 can be configured to omit physical operations such as a combination of a microphone and a voice command recognition unit.

[0061] The detection unit 212 includes a gyro sensor or sensors, and acquires angular velocity information or posture information, etc. of the digital camera 101. Note that the posture information includes information related to the tilt, etc. of the digital camera 101 with respect to the horizontal direction.

[0062] Figure 3 is a block diagram showing an example of the functional structure of the computer 102 according to the present embodiment. The system control unit 301 is, for example, a CPU, reads programs from the ROM 302, loads these programs into the RAM 303, and executes these programs to control the operations of the blocks included in the computer 102. The ROM 302 is a rewritable non-volatile memory, and in addition to storing the programs executed by the system control unit 301, also stores parameters, etc. required for controlling the blocks. The RAM 303 is a rewritable volatile memory, and uses the respective blocks included in the computer 102 as a temporary storage area for the output data.

[0063] The communication unit 304 communicates with an external device such as the digital camera 101 through wired or wireless communication. The recording device 305 is, for example, a hard disk, and stores the image data, etc. received by the communication unit 304 from the digital camera 101.

[0064] The image processing unit 306, for example, calculates the defocus amount (to be described later) of the image data loaded from the recording device 305 into the RAM 303, estimates the depth direction based on the image, or calculates information related to the degree of deviation of the optical system and the imaging element from the design position.

[0065] The display unit 307 is configured to display a GUI or various types of data provided by an OS or an application operating in the computer 102. The display unit 307 may be included in the computer 102 or may be connected as an external device.

[0066] The operation unit 308 is a group of input devices for a user to input various instructions, settings, etc. to the computer 102, and generally includes a keyboard, a mouse, a touchpad, etc. Additionally, when the display unit 307 is a touch display, the display unit 307 also serves as the operation unit 308. Note that the operation unit 308 may be configured to omit physical operations such as a combination of a microphone and a voice command recognition unit.

[0067] Figure 4A Shows Figure 2 the arrangement structure of pixels in the imaging unit 205. As Figure 4A shown, a plurality of pixels 400 are arranged two-dimensionally and regularly in the imaging unit 205. Specifically, the plurality of pixels 400 are arranged, for example, in the form of a two-dimensional lattice. Note that the arrangement structure of the pixels 400 is not limited to the lattice form, and other arrangement structures may also be adopted.

[0068] Figure 4B Shows in an enlarged manner Figure 4A the pixel 400 shown. As Figure 4B shown, each pixel 400 includes a microlens 401 and a pair of photoelectric conversion units 402A and 403B (hereinafter referred to as pupil-divided pixels 402A and 403B, respectively). Both of the pupil-divided pixels 402A and 403B have the same planar shape, and the planar shape has a rectangular shape with the long side direction being the y-axis direction. In each pixel 400, the pupil-divided pixels 402A and 403B are arranged to be axially symmetric with respect to the vertical bisector along the y-axis direction of the microlens 401. Note that the planar shape of the pupil-divided pixels 402A and 403B is not limited to this, and other planar shapes may also be adopted. Additionally, the arrangement manner of the pupil-divided pixels 402A and 403B is not limited to this, and other arrangement manners may also be adopted.

[0069] In the present embodiment, an A image and a B image are respectively output as parallax images from the pupil-divided pixels 402A and 403B arranged two-dimensionally and regularly. Additionally, an A + B image obtained by adding the A image and the B image is recorded as a still image on the recording medium 208. By as Figure 4A and Figure 4BThe configuration shown forms the imaging unit 205, which images a pair of light beams passing through different regions of the pupil of the optical system 204 into a pair of optical images, and can output these images as an A image and a B image. Note that the method for obtaining the A image and the B image is not limited to the above method, and various methods can be adopted. For example, a parallax image obtained by an imaging device such as a plurality of cameras installed at a spatial interval can be used as the A image and the B image. Additionally, a parallax image obtained by an imaging device such as a single camera including a plurality of optical systems and imaging units can also be used as the A image and the B image.

[0070] The operation of the image processing device 100 will be described below. In response to an imaging instruction such as a full press of a shutter button input through the operation unit 211 of the digital camera 101, the image processing device 100 performs Figure 5A , Figure 5B and Figure 5C the processing shown. Note that Figure 5A and Figure 5C the processing in is performed by the digital camera 101, and Figure 5B the processing in is performed by the computer 102.

[0071] First, in step S500, the system control unit 201 detects the state of the camera when the shutter button is pressed from the detection unit 212. Here, as the state of the camera, the tilt of the digital camera 101 with respect to the horizontal direction and the orientation in the up and down directions are detected.

[0072] In the subsequent step S501, the system control unit 201 performs imaging processing according to the exposure conditions determined in the imaging preparation state, and obtains an A image and a B image as a pair of parallax images from the imaging unit 205. Note that the A image and the B image pre-recorded on the recording medium 208 can also be read and obtained. Additionally, the A image and the B image can be added together and recorded on the recording medium 208 as an image for recording a still image. In Figure 6 the image for recording a still image in this embodiment is shown. Figure 6 is an image obtained by adding the captured A image and B image. Additionally, 600 is an autofocus frame.

[0073] In a subsequent step S502, the system control unit 201 controls the image processing unit 207 and outputs data representing the spatial (two-dimensional) defocus amount distribution in the imaging range based on the parallax image obtained in step S501. In the following description, the data representing the spatial defocus amount distribution will be referred to as a defocus map. The defocus amount is the amount of focus shift from the distance at which the optical system 204 is focused, and thus is a type of distance information. Regarding the method for obtaining the defocus amount, for example, a method for calculating the phase difference between parallax images as disclosed in Japanese Unexamined Patent Application Publication No. 2008-15754 can be used. Specifically, the relationship between the offset amount of the parallax image and the defocus amount is expressed by the following expression.

[0074] DEF = KX · PY · x ·…· (1)

[0075] In expression (1), DEF is the defocus amount, PY is the detection pitch (the pitch for arranging pixels of the same type), KX is a conversion coefficient determined by the degree of the opening angle of the centroid of a pair of light beams passing through the pupil, and x is the offset amount of the parallax image.

[0076] In addition, the present invention is not limited thereto, and a distribution of the offset amount, which is the offset amount of the parallax image, can also be obtained as the distance information distribution.

[0077] In addition, the distance information distribution can also be information represented in units of a length (such as micrometers) obtained by multiplying the offset amount of the parallax image by the detection pitch PY.

[0078] In addition, the present invention is not limited thereto, and the distance information distribution can be converted from the defocus amount to the distribution of the actual distance by further referring to the position of the focusing lens.

[0079] In addition, the present invention is not limited thereto, and a distribution of values obtained by normalizing the defocus amount with Fδ (where F is the f-number and δ is the diameter of the circle of confusion) can also be obtained as the distance information distribution. This distribution represents the amount of blur with respect to δ. Here, although the f-number used for imaging can be applied as the f-number F to the entire distribution, in order to obtain a more accurate distribution of the amount of blur, it is preferable to apply the effective f-number (effective f-value) that takes into account the vignetting characteristics of the optical system 204 under the imaging conditions. Figure 18 is a graph showing the vignetting characteristic V(h), where the horizontal axis represents the distance from the optical center (image height), and the vertical axis represents the amount of light normalized by setting the amount of light at the center of the image height to 1 at each image height. Vignetting occurs according to the lens frame or aperture frame, and Figure 18In this case, as the image height increases (towards the end in the imaging range), the amount of light decreases. The optical vignetting characteristic has unique characteristics depending on the lens. Here, by referring to the optical vignetting characteristic, the effective f-number F' at the image height h is expressed by the following expression.

[0080]

[0081] In Figure 7 is shown Figure 6 the defocus map of the image in. The defocus map 700 is represented by a grayscale of continuous values that become whiter (higher pixel values) as the distance gets closer. Additionally, 701 is the autofocus frame, and the in-focus area (where the defocus amount is zero) is shown in gray. Additionally, 702 is a straight line connecting the in-focus areas.

[0082] Subsequently, in step S503, the system control unit 201 sends the following information mainly including image data to the computer 102 through the communication unit 209.

[0083] - The image for recording a still image

[0084] - The defocus map

[0085] - Camera state detection information (information related to detecting the tilt of the camera)

[0086] - Autofocus frame position information

[0087] - The identification number (ID) of the camera body and the identification number (ID) of the attached lens

[0088] - Imaging information such as the f-number and ISO sensitivity

[0089] Record or transmit the above-mentioned information in relation to each other. For example, the above-mentioned information can be recorded in Exif information when using the JPEG format, or can be recorded as image attachment information in a single file when using the RAW data format. Optionally, the necessary information can be recorded or transmitted together with the image as a container file, in which multiple associated data can be jointly stored. Optionally, these information can be recorded or transmitted as different files without being collected together. For example, processing such as setting the same file name, storing them in the same folder, or sequentially transmitting these data in order (the receiver can recognize that these information are related to each other based on the order or type of the data, etc.) needs to be performed so that these data files can be grasped as being related to each other. Since the transmission control such as the file structure or transmission protocol related to recording or transmission is not directly related to the present invention and known methods can be used, details will be omitted from the description. Note that the above-mentioned information can be recorded on the recording medium 208 and then can be transmitted to the computer 102 through the communication unit 209, or the recording medium 208 can be removed from the digital camera 101 to read the image data in the computer 102. Additionally, instead of generating a defocus map (recording information distribution), the camera can record a pair of parallax images together with the above-mentioned associated information, and the computer 102 can generate a defocus map.

[0090] In this embodiment, the processing from step S504 to step S508 is performed by the computer 102. Since the processing from step S504 to step S508 is performed by a device different from the digital camera 101, the user can know the information related to the degree of deviation of the optical system and the imaging element from the designed position without performing special camera operations. Additionally, although a defocus map is generated in the digital camera 101 in this example, the parallax images can be transmitted to the computer 102, and the computer 102 can generate a defocus map. When the computational load on the digital camera 101 is heavy during continuous imaging, by dispersing the calculation for generating the defocus map to the computer 102, the time for calculating the information related to the degree of deviation of the optical system and the imaging element from the designed position can be reduced. Additionally, by further transmitting the information related to the transformation coefficient KX and the effective f-number F' uniquely determined by the lens in use and the imaging conditions, the computer 102 can generate various distance information distributions. Additionally, the information related to the transformation coefficient KX and the effective f-number F' can be pre-stored in the computer 102 and can be read from the stored information based on the received lens identification (ID) number and imaging information.

[0091] Figure 8is a block diagram schematically showing an example of the functional structure of the image processing unit 306 included in the computer 102 according to the present embodiment. Now, the operation of the image processing unit 306 will be described below by further referring to Figure 5B Note that the operation of the image processing unit 306 is implemented according to the control using the system control unit 301.

[0092] Initially, the system control unit 301 receives the information transmitted in step S503 and loads the read data into the RAM 303 (step S504).

[0093] Subsequently, in step S505, the depth direction estimation unit 800 estimates the depth direction in the image based on the camera state detection information 803 (imaging condition) when acquiring the disparity image recorded on the acquisition RAM 303. In this example, the depth direction is estimated with reference to the plane where the defocus amount becomes zero. Here, the plane where the defocus amount becomes zero will be described with reference to Figure 9 FIG.

[0094] Figure 9 FIG. shows a state in which a planar subject 901 on the ground is imaged by a digital camera 101 in which the overlooking optical system and the imaging element are not displaced from the designed positions, and the inclination of the digital camera 101 with respect to the horizontal direction ( Figure 9 the x-axis direction in ) is zero. In addition, when the autofocus frame 903 is at the point where the optical axis 900 and the subject 901 intersect each other, the plane connecting the focus areas (hereinafter referred to as the focus plane) is a plane 902 parallel to the imaging unit 205 and perpendicular to the optical axis 900. In addition, the focus plane 902 in the captured image of the subject 901 can also be represented by a straight line 904 passing through the autofocus frame 903. Figure 10 FIG. shows a defocus map 1000 of the captured image of the subject 901, an autofocus frame 1001 therein, and a focus plane 1002 therein. According to Figure 10 , in the case of imaging in a state where a camera in which the optical system and the imaging element are not displaced from the designed positions is overlooking and its inclination with respect to the horizontal direction is zero, the focus plane 1002 is a horizontal straight line with respect to the image. In addition, since the orientation of the camera in the up-down direction during imaging is known, the part above the focus plane 1002 is far away, and the part below the focus plane 1002 is close. That is, it can be estimated that the depth direction changes from the lower part to the upper part of the captured image. The depth direction estimation unit 800 outputs an expression of the straight line representing the focus plane 1002 as the depth estimation information 804.

[0095] Based on the defocus map 802 and the depth estimation information 804 (the expression of the straight line representing the focal plane) obtained by the depth direction estimation unit 800, the deviation degree calculation unit 801 calculates information 805 related to the deviation degree of the optical system and the imaging element for shooting the parallax image with respect to the design position (step S506).

[0096] Here, reference Figure 11A and Figure 11B will be used to describe the phenomena that occur when the optical system and the imaging element are offset from the design position. Figure 11A Shows the state of the optical system 204 and the imaging unit 205 at the design position. When shooting in this state, the focal plane 1100 is parallel to the imaging unit 205. On the other hand, Figure 11B Shows the state where the optical system 204 is offset from the design position and eccentricity occurs. In this case, based on the Scheimpflug's law, the focal plane 1101 is tilted according to the angle θ formed by the optical system 204 and the imaging unit 205. In Figure 11B 's state, when the camera is overlooking and the tilt of the camera with respect to the horizontal direction is zero, in Figure 12 , 1200 represents the defocus map of the captured image of the planar object on the ground, 1201 represents the autofocus frame therein, and 1202 represents the focal plane therein. According to Figure 12 , it should be understood that the depth changes from the lower right to the upper left of the screen. Therefore, there is a deviation from the depth change direction ( Figure 10 ) in the state where the optical system and the imaging element are not offset from the design position. Therefore, the relationship between the user's depth perception and the focal plane deviates, resulting in a captured image that the user does not expect.

[0097] The deviation degree calculation unit 801 calculates the angle θ_diff formed by the expression of the straight line representing the focal plane 1202 in the defocus map 802 and the straight line 1002 representing the focal plane estimated by the depth direction estimation unit 800, and causes the RAM 303 to store this angle θ_diff as the evaluation value 805 representing the deviation degree. Figure 13 Shows θ_diff. The larger θ_diff is, the greater the deviation of the optical system and the imaging element with respect to the design position (calibration is required). Note that when the camera is tilted with respect to the horizontal direction during shooting, this tilt angle can be subtracted from θ_diff for correction, and the effects of the present invention can also be obtained even for images captured in a state where the camera is tilted with respect to the horizontal direction.

[0098] In subsequent step S507, the system control unit 301 compares the calculated θ_diff with a threshold value stored in advance, and if θ_diff is greater than the threshold value, step S508 is performed, and if θ_diff is less than or equal to the threshold value, the process ends.

[0099] In step S508, the system control unit 301 sends the following information to the digital camera 101 via the communication unit 304 to notify the optical system and the imaging element of the deviation from the design position in the camera used by the user.

[0100] - Identification number of the camera body where the deviation is detected

[0101] - Identification number of the lens where the deviation is detected

[0102] In step S509, the system control unit 201 in the digital camera 101 determines whether it has received the information sent from the computer 102. If these information are received, step S510 is performed; if these information are not received, the process ends.

[0103] In step S510, the system control unit 301 outputs a display as Figure 14 such to the display unit 210, and recommends the user to repair the camera and the lens at the customer center. The customer center receives the ID information of the camera and the lens, etc. and the image data from the user (digital camera 101), which is useful for determining or counting repair / failure information, etc.

[0104] In the above manner, according to the present embodiment, information related to the degree of deviation of the optical system and the imaging element from the design position can be calculated without hindering the convenience of the user, and this information can be notified to the user.

[0105] In addition, in the present embodiment, although text notifying the user of the deviation of the optical system and the imaging element from the design position is displayed, in order to enable the user to more easily recognize the occurrence of the deviation, an image can be displayed on the display unit 210. Specifically, the display unit 210 can be configured to display the Figure 12 defocus map using grayscale in, or the defocus map whose color values have been converted through look-up table conversion, etc. generated in step S502.

[0106] In addition, in the present embodiment, although information is displayed to the user in the case of deviation of the optical system or the imaging element from the design position, information can also be displayed in the case where the occurrence of the deviation is not detected or in both cases. According to this configuration, when the user immediately needs to know the judgment result, the user can know whether calibration is required.

[0107] In addition, in the present embodiment, although the case of generating a defocus map by calculating a disparity amount using paired disparity images is described as an example, the present invention is not limited thereto. As a method for generating a defocus map, for example, a DFD (Depth From DeFocus) method can be adopted, in which a defocus map is obtained based on the correlation between two images with different focusing positions or f-numbers. Since information related to the degree of deviation of the optical system and the imaging element from the design position can be calculated by using the images obtained in the aperture bracket imaging mode, the opportunity for detecting the deviation increases, and information can be provided to the user at an appropriate timing.

[0108] In addition, in the present embodiment, although the depth direction in the image is estimated based on the camera state detection information, the present invention is not limited thereto. For example, the depth direction can also be estimated by using information related to the vanishing point. With this configuration, even if the camera does not include a gyroscope or a sensor for detecting the state of the camera, the effects of the present invention can be obtained, and the convenience of the user is increased. Now, a method for estimating the depth direction by using vanishing point detection and for calculating information related to the degree of deviation of the optical system and the imaging element from the design position will be described below.

[0109] The vanishing point is the point at which the straight lines on the picture plane corresponding to parallel lines in a three-dimensional space converge when projected onto the image plane through a projective transformation. That is, the vanishing point is an "infinitely distant point" on the planar image onto which the space actually having depth is projected, and is recognized as the point at which the extension lines of the lines parallel to the depth direction intersect each other or the extension of the plane extending in the depth direction converges to an infinitely distant point. Therefore, a plurality of straight lines in the image are detected by a known method such as the Hough transform, and the point at which the maximum number of the detected straight lines converge can be detected as the vanishing point. The result of detecting the vanishing point is shown in Figure 15 in Figure 6 The result of detecting the vanishing point is shown in

[0110] In Figure 15 1500 is the vanishing point. In addition, the depth direction can be estimated as the direction 1501 from the autofocus frame 600 toward the vanishing point. The depth direction estimation unit 800 outputs the direction 1501 toward the vanishing point as the depth direction estimation information 804.

[0111] The deviation degree calculation unit 801 calculates the gradient (change direction) of the defocus amount near the autofocus frame in the defocus map 802 by using a known technique. Subsequently, based on the difference from the depth direction estimation information 804, the deviation degree calculation unit 801 calculates an evaluation value representing the deviation degree of the optical system and the imaging element relative to the design position. Specifically, the direction toward the vanishing point and the gradient direction of the defocus amount are each regarded as a vector, and the difference between the vectors is the evaluation value. The greater the deviation of the optical system and the imaging element from the design point, the greater the evaluation value.

[0112] In addition, the method for estimating the depth direction in the image with reference to the features extracted from the image can use not only the above-described vanishing point detection but also the information related to the change in the density of the texture. A method for detecting the depth in the image with reference to the change in the density of the texture. As a method, for example, the method described in “Texture Structure Classification and Depth Estimation using Multi-Scale Local Autocorrelation Features”, KANG Y, HASEGAWA O, NAGAHASHI H (Tokyo Inst. Technol.), JST-PRESTO (non-patent literature) can be adopted.

[0113] Specifically, in the case of referring to an image for recording a still image where there is a uniform texture in the image (for example, Figure 6 a road), the depth direction estimation unit 800 uses the fact that the density of the texture decreases as the distance increases. That is, if a region where the density of the same texture gradually decreases is detected in the image, the depth direction estimation unit 800 determines that the plane covered by the predetermined texture is moving away from the imaging position. The direction from the front side to the opposite side is output as the depth direction estimation information 804. In particular, in the focused area, it is possible to detect fine textures, and by making the above determination near the autofocus frame, the depth direction can be estimated with high accuracy. In addition, by using known typical object detection, it is possible to pre-detect regions where there is likely to be a uniform texture (such as the ground like a road, a water surface, or a hedge as a structure built in the vertical direction on the ground or water surface), and the target area can be restricted. Therefore, the processing time for estimating the distribution of positions in the depth direction can be reduced.

[0114] In addition, similar to the case of using the vanishing point, the deviation degree calculation unit 801 sets the difference between the vector of the gradient (change direction) of the defocus amount near the autofocus frame in the defocus map 802 and the vector of the depth direction estimated based on the change in the density of the texture as the evaluation value.

[0115] In addition, in the present embodiment, an evaluation value representing the degree of deviation of the optical system and the imaging element from the designed position is calculated based on a single captured image, and it is determined whether an offset has occurred. However, the present invention is not limited thereto. The determination can be made when the number of images captured by the user reaches a certain reference number. By making the determination based on the evaluation values in a plurality of captured images, the reliability of the determination regarding whether an offset has occurred can be increased. In addition, regarding the condition related to whether to notify the user of the occurrence of the offset of the optical system and the imaging element from the designed position, by checking whether the number of images in which the offset is detected reaches a certain reference number, the reliability of the determination can be further improved.

[0116] In addition, instead of determining the degree of deviation of the optical system and the imaging element from the designed position for all captured images, it is preferable to perform the deviation degree determination process after pre-evaluating whether the images in a large number of images are suitable for the deviation degree determination. Specifically, by performing the above-described typical object detection, an evaluation related to whether there is a uniform texture such as a road and whether the image is suitable for estimating the depth direction is performed. With this configuration, the processing time for determining the deviation degree can be reduced. In addition, a subject having a texture such as a road is also suitable for detecting the phase difference between parallax images in step S502, and thus a more accurate deviation degree evaluation value can be expected. In addition, by recording known GPS information as image attachment information, it is possible to determine whether a subject having a texture such as a road is likely to be included in the captured image. By pre-selecting images that are expected to include textures from a large number of images captured by the user, the time for calculating a highly reliable evaluation result can be reduced. In addition, in the case of a camera having a detachable optical system, statistics related to whether an offset of the optical system and the imaging element from the designed position can be detected for each attached lens are collected. This makes it possible to determine whether an offset has occurred in the optical system or the imaging element and to provide the user with a more detailed determination result.

[0117] In addition, in the deviation degree calculation unit 801, by considering the following details to obtain the gradient of the defocus map 802 with high precision, a highly reliable evaluation result can be obtained. Specifically, a histogram (statistical information) of the calculated defocus map is acquired, and based on the shape of the histogram, it is determined whether the gradient of the defocus amount can be obtained with high precision. Although it will be described below, it is preferable to select an image in which the histogram has a large width and a smooth change. Figure 16 shows Figure 7 the histogram of the defocus map in. According to Figure 16 , the defocus amount is widely distributed from the front side to the opposite side, and in addition, the change in the defocus amount is smooth. Therefore, Figure 7 the image in is suitable for evaluating the change direction of the defocus amount in the entire image. On the other hand,Figure 17B An out-of-focus image of a bust of a person in a portrait photograph as shown in Figure 17A is shown, and Figure 17C its histogram is shown. Since the amount of defocus in the image is concentrated on the person in the bust, it should be understood that this image is not suitable for evaluating the direction of change in the amount of defocus in the entire image. In the deviation degree calculation unit 801, the histogram of the defocus image is confirmed before comparison with the depth direction estimation information 804, and if the image is not suitable for evaluating the deviation degree, the determination process is interrupted, thereby reducing the calculation time.

[0118] In addition, in order to obtain the phase difference between the parallax images with high precision in step S502, an image with a high S / N can be selected. Therefore, an image captured at the lowest possible sensitivity is preferentially selected from a large number of captured images, thereby improving the reliability of the evaluation value.

[0119] In addition, since the phase difference between the parallax images is affected by the aberration in the optical system 204, it is preferable to correct the aberration known as the design information of the optical system before comparison with the estimation result using the depth direction estimation unit 800. In addition, the area to be compared can be limited to an area where the influence of the aberration is small. In this way, a more reliable evaluation result can be calculated.

[0120] In addition, in step S510, when detection information related to the occurrence of the deviation of the optical system and the imaging element from the design position is received, a display encouraging the user to repair the camera and lens at the customer center is output to the display unit of the digital camera 101. However, other solutions are also possible. Specifically, in step S508, the evaluation value 805 indicating the deviation degree is further sent to the digital camera 101. According to the evaluation value 805 indicating the deviation degree, simple calibration can be performed by driving the IS mechanism mounted on the optical system and the imaging unit to approach the state where no deviation of the optical system and the imaging element from the design position has occurred. Alternatively, referring to the evaluation value 805 indicating the deviation degree, the image can be processed to approach the image obtained in the state where no deviation of the optical system and the imaging element from the design position has occurred by performing image processing (sharpening or blurring) on the area where the focal plane is tilted. Specifically, compared with the state where no deviation has occurred, the area where the amount of defocus is close to the focused state is blurred. Conversely, the area close to the background or the front side compared to the original amount of defocus is sharpened. With the above configuration, even in a situation where the user cannot repair the camera and lens, an image with the depth of field desired by the user can be obtained, and user convenience can be improved.

[0121] In addition to the user's camera, detection information related to the occurrence of an offset of the optical system and the imaging element from the designed position can also be sent to the customer center. The customer center manages customer information registered by the user himself / herself and information related to all the owned devices, and records the number of occurrences of the offset of the optical system and the imaging element from the designed position and the number of maintenance times. Thereby, user convenience can be further improved, such as reducing the repair time, etc.

[0122] In addition, in the case where the camera and the lens used by the user for imaging are integrated, the display unit 201 can be configured to present to the user whether to execute an operation mode for distinguishing the cause of occurrence between the optical system and the imaging element. In response to the user selecting the operation mode for distinguishing the cause, the display unit 201 is instructed to encourage the user to capture an image suitable for distinguishing the cause. As specific details of this instruction, the user pastes graph paper on the wall directly in front of the user, and captures an image while changing the imaging conditions (focal length, focusing lens position, aperture) of the optical system. As a result of the analysis, if the judgment result changes by changing the imaging conditions of the optical system, the cause is the lens; if the occurrence of the offset is detected regardless of the imaging conditions, the cause is the imaging element. With this configuration, the cause of the offset of the optical system and the imaging element from the designed position can be found, and more appropriate notification or repair can be performed.

[0123] [Second Embodiment]

[0124] Hereinafter, an image processing apparatus, an image processing method, and an image processing program according to a second embodiment of the present invention will be described in detail with reference to some drawings. Note that components that are substantially the same as those in the image processing apparatus according to the above-described first embodiment are denoted by the same reference numerals, and these components will be omitted from the description or briefly described.

[0125] The first embodiment illustrates an embodiment in which information related to the degree of deviation of the optical system and the imaging element from the designed position, which is an internal parameter of the camera device 100, is calculated and notified to the user. The second embodiment of the present invention will illustrate an embodiment in which calibration of the position or orientation of the image processing apparatus, which is an external parameter, is performed.

[0126] First, the imaging system according to the present embodiment will be described. The imaging system according to the present embodiment images an inspection object surface of a structure that is an object of social infrastructure inspection, and in particular, can easily face the inspection object surface and image it or evaluate the captured image. The imaging system according to the present embodiment includes a camera device that periodically / irregularly captures moving images or still images, a lens device to be attached to the camera device, and a pan-tilt device for rotating the camera device.

[0127] First, with reference to Figure 19A the block diagram, an example of the hardware configuration of the camera device 1900 and the lens device 1913 according to this embodiment will be described. Note that the imaging device in this embodiment may also be configured as the digital camera 101 as in the first embodiment. Figure 19A The state where the lens device 1913 is attached to the camera device 1900 is shown.

[0128] First, an example of the hardware configuration of the camera device 1900 will be described. The camera device 1900 according to this embodiment acquires a distance information distribution at a plurality of positions in the imaging range of the camera device 1900, obtains information related to an instruction for rotating or translating the camera device 1900 based on the difference between the acquired distance information, and outputs the acquired information. Here, as in the first embodiment, the distance information and the distance information distribution may be any one of the image offset amount and the image offset amount distribution between a pair of parallax images, the defocus amount and the defocus map acquired by any means, or the subject distance information and the subject distance map.

[0129] The CPU (Central Processing Unit) 1901 performs various processes by using computer programs or data stored in the ROM (Read Only Memory) 1902 or the RAM (Random Access Memory) 1903. Therefore, the CPU 1901 controls the operation of the entire camera device 1900, and also performs or controls the processes described later as the processes performed by the camera device 1900.

[0130] The ROM 1902 stores the setting data of the camera device 1900, computer programs or data related to the startup of the camera device 1900, and computer programs or data related to the basic operations of the camera device 1900, etc.

[0131] The RAM 1903 has areas for storing computer programs or data read from the ROM 1902 or computer programs or data read from the memory card 1909 via the recording medium I / F 1908. The RAM 1903 also has areas for storing captured images output from the imaging element 1904, computer programs or data received from an external device via the external I / F 1910, or data received from the lens device 1913 through the camera communication unit 1907. The RAM 1903 also has a work area used when the CPU 1901 performs various processes. In this way, the RAM 1903 can appropriately provide various areas.

[0132] The pixel arrangement of the imaging element 1904 has the same as Figure 2It has the same arrangement structure as the imaging unit 205, and generates and outputs a captured image corresponding to the light that enters through the lens device 1913. The display unit 1905 is a liquid crystal display (LCD), an organic EL display (OLED), or the like, and is a device that displays images or text on a display screen or a viewfinder screen. Note that the display unit 1905 may not be included in the camera device 1900, and may be, for example, an external device that can communicate with the camera device 1900 wiredly and / or wirelessly.

[0133] The operation unit 1906 is a user interface such as buttons, dials, touch panels, or joysticks, and can input various instructions to the CPU 1901 through user operations.

[0134] The camera communication unit 1907 performs data communication between the camera device 1900 and the lens device 1913. The recording medium I / F 1908 is an interface for attaching the memory card 1909 to the camera device 1900, and the CPU 1901 reads data from and writes data to the memory card 1909 via the recording medium I / F 1908.

[0135] As the memory card 1909, for example, card-type recording media such as SD, CF, CFexpress, XQD, or CFast are known. In addition, the memory card 1909 can also record data on an external device via a wireless network.

[0136] The external I / F 1910 is a communication interface for data communication with an external device, and the CPU 1901 performs data communication with the external device via the external I / F 1910. The power supply unit 1911 supplies and manages the power in the camera device 1900.

[0137] The CPU 1901, ROM 1902, RAM 1903, imaging element 1904, display unit 1905, operation unit 1906, camera communication unit 1907, recording medium I / F 1908, external I / F 1910, and power supply unit 1911 are all connected to the system bus 1912.

[0138] Next, an example of the hardware structure of the lens device 1913 will be described. The CPU 1914 performs various processes by using computer programs or data stored in the ROM 1915 or RAM 1916. Therefore, the CPU 1914 controls the operation of the entire lens device 1913, and also performs or controls the processes described later as the processes performed by the lens device 1913.

[0139] The ROM 1915 stores setting data of the lens device 1913, computer programs or data related to the startup of the lens device 1913, and computer programs or data related to the basic operations of the lens device 1913, etc.

[0140] The RAM 1916 has areas for storing computer programs or data read from the ROM 1915 or data received from the camera device 1900 via the lens communication unit 1919. The RAM 1916 also has a working area used when the CPU 1914 performs various processes. In this way, the RAM 1916 can appropriately provide various areas.

[0141] The lens communication unit 1919 performs data communication between the camera device 1900 and the lens device 1913. For example, the lens communication unit 1919 receives control information from the camera device 1900 to the lens device 1913, sends the operation state of the lens device 1913, etc. to the camera device 1900, or receives power supply from the camera device 1900.

[0142] The display unit 1917 is a liquid crystal display (LCD) or an organic EL display (OLED), etc., and is a device for displaying the operation state of the lens device 1913, etc. Note that the display unit 1917 may not be included in the lens device 1913 and may be, for example, an external device that can communicate with the lens device 1913 wired and / or wirelessly.

[0143] The operation unit 1918 is a user interface such as buttons, dials, touch panels, or joysticks, etc., and can input various instructions to the CPU 1914 through user operations. In addition, the instructions input by operating the operation unit 1918 through user operations can be sent to the camera device 1900 via the lens communication unit 1919.

[0144] The lens drive unit 1920 controls the optical lenses included in the lens device 1913 based on instructions from the CPU 1901 or the CPU 1914, and thereby controls the aperture, focusing, zoom focus, and camera shake correction, etc. The light that enters via the optical lens after being controlled by the lens drive unit 1920 for the aperture, focusing, zoom focus, and camera shake correction, etc. is received by the above-mentioned imaging element 1904, and the imaging element 1904 generates and outputs a captured image corresponding to the received light.

[0145] The CPU 1914, ROM 1915, RAM 1916, lens communication unit 1919, display unit 1917, operation unit 1918, and lens drive unit 1920 are all connected to the system bus 1921.

[0146] Next, reference will be made toFigure 20 The block diagram is used to illustrate an example of the hardware structure of the pan-tilt device 2000 according to this embodiment.

[0147] The CPU 2001 performs various processes by using computer programs or data stored in the ROM 2002 or the RAM 2003. Therefore, the CPU 2001 controls the operation of the entire pan-tilt device 2000 and also performs or controls the processes described later as the processes performed by the pan-tilt device 2000.

[0148] The ROM 2002 stores the setting data of the pan-tilt device 2000, computer programs or data related to the startup of the pan-tilt device 2000, and computer programs or data related to the basic operations of the pan-tilt device 2000, etc.

[0149] The RAM 2003 has an area for storing computer programs or data read from the ROM 2002. The RAM 2003 also has a working area used when the CPU 2001 performs various processes. In this way, the RAM 2003 can appropriately provide various areas.

[0150] The external I / F 2004 is a communication interface for obtaining various instructions from the remote control device 2010 through wireless or wired communication. The remote control device 2010 is a device for inputting various instructions to the pan-tilt device 2000 and can input, for example, a change instruction for changing the pan angle or the tilt angle of the camera device 1900 mounted on the pan-tilt device 2000. The external I / F 2004 can also communicate with the camera device 1900 mounted on the pan-tilt device 2000.

[0151] The power supply unit 2005 supplies and manages the power in the pan-tilt device 2000. The display unit 2006 is a liquid crystal display (LCD) or an organic EL display (OLED), etc., and is a device for displaying the operation state of the pan-tilt device 2000, etc. Note that the display unit 2006 may not be included in the pan-tilt device 2000 and can be, for example, an external device that is communicable with the pan-tilt device 2000 in a wired and / or wireless manner.

[0152] The operation unit 2007 is a user interface such as buttons, dials, touch panels, or joysticks, etc., and can input various instructions to the CPU 2001 through user operations.

[0153] The drive unit 2008 includes a base (fixing member) for fixing the camera device 1900, and a drive mechanism for panning the base, tilting the base, or translating the base in the XYZ directions. By controlling the drive mechanism, the drive unit 2008 controls the panning angle, tilting angle, and position in the XYZ directions of the camera device 1900 based on an instruction received via the external I / F 2004 from the remote control device 2010 or the like. Further, in the present embodiment, the panning, tilting, and imaging position of the camera device 1900 are controlled by mounting the camera device 1900 on the above-described pan-tilt device 2000. However, the present invention is not limited thereto, and can be applied, for example, to devices such as drones, in which at least one of the panning, tilting, and imaging position of the camera device 1900 is controlled by the movement of the device itself.

[0154] The CPU 2001, ROM 2002, RAM 2003, external I / F 2004, power supply unit 2005, display unit 2006, operation unit 2007, and drive unit 2008 are all connected to the system bus 2009.

[0155] Next, a block diagram of Figure 19B will be used to describe an example of the functional structure of the camera device 1900. Although each functional unit shown in the following description Figure 19B mainly performs processing, in reality, the CPU 1901 executes a computer program corresponding to the functional unit, thereby performing the operation of the functional unit. Further, Figure 19B at least a part of the functional units shown can be implemented by hardware.

[0156] The subject recognition unit 1928 identifies whether the inspection target surface of the structure, which is the object of social infrastructure inspection, is included in the captured image by using known typical object detection. Specifically, the subject recognition unit 1928 pre-stores the feature amounts related to the structure that is the object of infrastructure inspection, and compares the feature amounts of the image obtained by imaging with the stored image. The result is also used as information for estimating the depth direction in the image. As a result of the recognition, if the inspection target surface of the structure that is the object of social infrastructure inspection is imaged, a calibration process is performed so that the camera device 1900 is in a facing relationship with the inspection target surface of the structure. Here, if the inspection target surface is flat, since the camera and the structure are in a facing relationship, it is assumed that the defocus amount in the imaging range is almost uniform. As the calibration target, as in the method for determining the deviation degree of the optical system and the imaging element from the design position described in the first embodiment, the position and posture of the camera device 1900 can be corrected so that the values in the distance information distribution in the imaging range become uniform (fall within a predetermined range). However, the present embodiment shows a method for simply setting the control in the pan or tilt direction based on the defocus amount in a partial area within the imaging range hereinafter.

[0157] The determination unit 1922 acquires setting information indicating the setting information of "the rotation direction and translation direction for operating the camera device 1900 so that the camera device 1900 faces the inspection target surface of the structure that is the object of social infrastructure inspection". This setting information is determined, for example, by the user operating the operation unit 1906. If the driving of the camera indicated by the setting information is the rotation direction and the horizontal direction (pan direction), the determination unit 1922 sets each of the two regions arranged in the left-right direction within the imaging range of the camera device 1900 as a "region for acquiring the defocus amount". (For example, positions near the left end and near the right end within the imaging range). Here, the minimum unit of each region is 1 pixel.

[0158] On the other hand, if the driving of the camera indicated by the setting information is the rotation direction and the vertical direction, the determination unit 1922 sets each of the two regions arranged in the up-down direction within the imaging range of the camera device 1900 as a "region for acquiring the defocus amount". (For example, positions near the upper end and near the lower end within the imaging range). Here, the minimum unit of each region is also 1 pixel.

[0159] In addition, if the driving of the camera indicated by the setting information is translation, the determination unit 1922 sets each of the four regions arranged in the up-down direction and the left-right direction within the imaging range of the camera device 1900 as a "region for acquiring the defocus amount". Here, the minimum unit of each region is also 1 pixel.

[0160] In addition, in the present embodiment, instead of using the setting information set by the user (or regardless of the setting information), as in the first embodiment, distance information can be obtained in a plurality of regions, that is, a distance information distribution can be obtained, and the driving of the pan-tilt device 2000 (the position and posture of the camera) can be controlled based on the analysis result of the distribution information. At this time, the "region for obtaining the defocus amount" is, for example, the entire region where the defocus amount can be obtained. By obtaining the distance information distribution, for example, a two-dimensional or three-dimensional inclination of the distance information can be obtained through plane detection, and the position and posture of the camera can be controlled so that the inclination is facing and becomes close to zero in each direction.

[0161] The control unit 1924 obtains the defocus amount from the "region for obtaining the defocus amount" determined by the determination unit 1922 within the imaging range of the camera device 1900. The acquisition unit 1923 acquires the defocus amount acquired by the control unit 1924. The difference calculation unit 1925 calculates the difference between the defocus amount acquired by the acquisition unit 1923 and another defocus amount.

[0162] The determination unit 1926 determines notification information for notifying "the degree of rotation or translation (including direction) for driving the camera device 1900" based on the difference calculated by the difference calculation unit 1925. The output unit 1927 outputs the notification information determined by the determination unit 1926 to the pan-tilt device 2000 via the external I / F 1910. The pan-tilt device 2000 acquires the notification information via the external I / F 2004, and controls the drive unit 2008 based on the notification information to set the camera device 1900 in a desired position and posture.

[0163] In the present embodiment, such an imaging system is used to image social infrastructure as an inspection object, and the social infrastructure is inspected based on the captured image obtained by imaging. Reference will be made to Figure 21A and Figure 21B to describe an imaging method for imaging social infrastructure by using the imaging system according to the present embodiment.

[0164] Figure 21A An example of an inspection object surface of social infrastructure as an inspection object is shown. Figure 21AThe social infrastructure 2100 shown is a wall-like structure having a side surface 2101 and being long in the lateral direction. Reference numeral 2102 denotes a joint portion that occurs when the social infrastructure 2100 is divided based on a design drawing and constructed with construction joints. The portion 2102 is also referred to as a construction joint portion, but is referred to here as a joint for ease of understanding. The joint portion 2102 can be visually observed, and thus the joint portion 2102 also serves as a unit for inspection operations. Reference numeral 2103 denotes an area (inspection target area) that is the target of a single inspection, and the imaging system images an imaging area 2104 including the inspection target area 2103. In the captured image obtained by imaging the imaging area 2104, "a partial image corresponding to the peripheral area of the inspection target area 2103 in the imaging area 2104" is information for grasping the positional relationship with an adjacent inspection target area. Therefore, this partial image is used for alignment when combining the images into a single image including the entire social infrastructure 2100. In addition, the partial image corresponding to the peripheral area is also used for inspection of deformations in a wide range not limited to a single inspection target area.

[0165] Figure 21B Shows a state in which the imaging area 2104 is imaged by using the imaging system according to the present embodiment. In Figure 21B this, the camera device 1900 is attached to the pan-tilt device 2000 having the tripod 2108, and the lens device 1913 is attached to the camera device 1900. The width of the imaging range 2109 on the inspection target surface imaged by the combination of the camera device 1900 and the lens device 1913 (the size in the lateral direction in this figure) corresponds to the width of the imaging area 2104 (the size in the lateral direction in this figure).

[0166] When the imaging of the inspection target area 2103 is completed, an inspection target area adjacent to the inspection target area 2103 and not yet imaged is imaged. The imaging system according to the present embodiment is moved to the position denoted by reference numeral 2110, and the inspection target area in the imaging range 2112 is imaged in substantially the same manner. When the imaging at the position denoted by reference numeral 2110 is completed, in order to image an inspection target area adjacent to the inspection target area and not yet imaged, the imaging system according to the present embodiment is moved to the position denoted by reference numeral 2111, and the inspection target area in the imaging range 2113 is imaged in substantially the same manner. In the case where the camera device 1900 is mounted on a moving object such as a drone, the user manually or automatically moves the moving object to each imaging position and sequentially performs imaging.

[0167] Here, in the present embodiment, the camera device 1900 needs to face the inspection object area. In the present embodiment, it is determined whether the camera device 1900 faces the inspection object area. If the camera device 1900 does not face the inspection object area, a notification for rotating or translating the camera device 1900 to face the inspection object area is issued.

[0168] To issue this notification, as described above, the control unit 1924 obtains the defocus amount at the position determined by the determination unit 1922. The method for obtaining the defocus amount is the same as the method in step S502 of the first embodiment, and thus is omitted from the description here. Here, the obtained defocus amount has a continuous value, and the defocus amount corresponding to the focusing degree can be determined to be "-11" for the front focus, "0" for the in-focus state, and "+7" for the rear focus. Additionally, as in the first embodiment, data representing the spatial (two-dimensional) defocus amount distribution in the imaging range can be created, and the control unit 1924 can be configured to obtain the defocus amount at the position determined by the determination unit 1922 in the defocus amount distribution (distance information distribution).

[0169] Next, reference will be made to Figure 22 the flowchart to describe the operation of the imaging system according to the present embodiment. As described above, the user installs the imaging system to face the inspection object surface to image the inspection object surface by using the imaging system according to the present embodiment. At this time, the user can install the camera device 1900 in a direction assumed to be substantially facing the inspection object area. However, if there is no reference point for the structure or the installation position and there is no accurate measurement information of the surrounding environment, the camera device 1900 cannot be accurately installed to face the inspection object area. In response to the power-on of the camera device 1900 after the camera device 1900 is installed, the captured image captured by the imaging element 1904 is displayed as a live view image on the display screen on the back of the camera device 1900 by the display unit 1905. Subsequently, the processing according to Figure 22 the flowchart starts.

[0170] In step S2200, the subject recognition unit 1928 performs typical object detection processing on the captured image. In the present embodiment, since the inspection object surface of the structure to be imaged is included in the objects to be detected as typical objects, information related to the feature amount representing the inspection object surface is stored in the ROM 1902 in advance.

[0171] In step S2216, the subject recognition unit 1928 determines whether the object detected in step S2200 is the inspection target surface of the structure to be imaged when the camera device 1900 is in a facing relationship. If it is determined that the object is the inspection target surface, the process continues and proceeds to step S2201. On the other hand, if it is determined that the object is not the inspection target surface, the processing ends according to the Figure 22 flowchart.

[0172] In step S2201, the determination unit 1922 acquires setting information that represents "the drive of the camera device 1900 for making the camera device 1900 face the inspection subject surface".

[0173] For example, as Figure 23 shown, the operation unit 1906 controls "the drive of the camera device 1900 for making the camera device 1900 face the inspection subject surface" (the facing detection direction). That is, the operation unit 1906 has a switch corresponding to the operation unit 2007, and this switch is used to set the facing detection direction to at least any one of "vertical direction", "horizontal direction", and "translation". By operating this switch, the user can set the facing detection direction to either the vertical direction (rotation axis = pitch axis) or the horizontal direction (rotation axis = pan axis). The determination unit 1922 acquires the facing detection direction set by using the switch as the setting information. As Figure 21A and Figure 21B shown, if the user images a horizontally long structure while moving horizontally, the facing detection direction in the horizontal (rotation) direction is selected.

[0174] In addition, as described above, in the case of performing control including translation in the XYZ directions to obtain a multi-region focused image from the front (for example, through mode setting), the setting of the facing detection direction in step S2201 is not performed. The control unit 1924 estimates the position and posture of the plane of the subject to be focused based on the distance information distribution obtained in multiple regions in the captured image, and controls the position and posture of the pan-tilt device 2000 (camera device 1900) as in the first embodiment.

[0175] As an example of setting the facing detection direction to either the vertical direction (rotation axis = pitch axis) or the horizontal direction (rotation axis = pan axis), the case of setting the facing detection direction to the horizontal direction will be described below.

[0176] Subsequently, in step S2202, since the facing detection direction is the horizontal direction, the determination unit 1922 sets each of the two regions arranged in the left-right direction within the imaging range of the camera device 1900 as "the region for acquiring the defocus amount". For example, as Figure 24AAs shown, the determination unit 1922 sets the area 2400 near the left end and the area 2401 near the right end of the imaging area 2104 of the social infrastructure 2100 that falls within the imaging range 2402 of the imaging device 1900 as the "area for obtaining the defocus amount". Additionally, this embodiment is not limited thereto, and in the case of setting the front facing the detection direction as well, similar to the first embodiment, the defocus amount of the entire screen (entire image) can be obtained.

[0177] In step S2203, the control unit 1924 obtains the defocus amount at the position set in step S2202 (areas 2400 and 2401 in the Figure 24A case). At this time, the camera device 1900 does not have to be focused on the inspection object surface, and the defocus amount in the area set in step S2202 is obtained.

[0178] In step S2204, the acquisition unit 1923 acquires the "defocus amount in the left area" and the "defocus amount in the right area" obtained in step S2203. Subsequently, the difference calculation unit 1925 calculates the difference by subtracting the "defocus amount in the right area" from the "defocus amount in the left area".

[0179] In step S2206, the determination unit 1926 acquires the "information indicating the rotation direction and rotation degree of the camera device 1900" corresponding to the difference between the defocus amounts calculated in step S2204 as the rotation instruction information (notification information).

[0180] Here, as Figure 25 shown, the table 2515 is registered in the ROM 1902. In the table 2515, the rotation instruction information corresponding to the difference between the defocus amounts is registered. In the column 2516, the range of the difference between the defocus amounts is registered. For example, in the row 2519 of the column 2516, the range of the difference between the defocus amounts "+11 or greater" is registered, and in the row 2524 of the column 2516, the range of the difference between the defocus amounts "-5 to -10" is registered.

[0181] In the column 2517, the icons corresponding to the rotation amount in the case of rotating the camera device 1900 counterclockwise are registered. The icon registered in the row 2519 of the column 2517 indicates a larger rotation amount than the icon registered in the row 2520 of the column 2517. The icon registered in the row 2520 of the column 2517 indicates a larger rotation amount than the icon registered in the row 2521 of the column 2517. The icons registered in the rows 2522 to 2525 of the column 2517 indicate that no counterclockwise rotation is required.

[0182] In column 2518, icons corresponding to the amount of rotation in the case of rotating the camera device 1900 clockwise are registered. The icon registered in row 2525 of column 2518 represents a larger amount of rotation than the icon represented by the icon registered in row 2524 of column 2518. The icon registered in row 2524 of column 2518 represents a larger amount of rotation than the icon represented by the icon registered in row 2523 of column 2518. The icons registered in rows 2519 to 2522 of column 2518 represent that no clockwise rotation is required.

[0183] Therefore, for example, if the difference between the defocus amounts calculated in step S2204 is "+7", the determination unit 1926 acquires the two icons registered in row 2520 corresponding to the range "+10 to +5" including the difference "+7" as rotation instruction information.

[0184] In addition, for example, if the difference between the defocus amounts calculated in step S2204 is "-12", the determination unit 1926 acquires the two icons registered in row 2525 corresponding to the range "-11 or less" including the difference "-12" as rotation instruction information.

[0185] That is to say, in Figure 25 's table, rotation instruction information for notifying the rotation direction corresponding to the sign of the difference between the defocus amounts and the rotation degree corresponding to the absolute value of the difference between the defocus amounts is registered.

[0186] In step S2214, the output unit 1927 outputs the rotation instruction information acquired in step S2206 as "notification information for notifying the user of the rotation direction and rotation degree of the camera device 1900" to the display unit 1905. The display unit 1905 displays this notification information on the display screen on the back of the camera device 1900. For example, as Figure 24A shown, the icon 2405 acquired from column 2517 is displayed at the lower left of the live view image 2404 displayed on the display screen on the back of the camera device 1900. In addition, the icon 2406 acquired from column 2518 is displayed at the lower right of the live view image 2404. Note that the display positions of the icon 2405 and the icon 2406 are not limited to specific display positions, and for example, the icon 2405 and the icon 2406 can be displayed so as to be superimposed on the live view image 2404. In addition, in Figure 24A the icons 2400a and 2401a are displayed in a superimposed manner at positions corresponding to the positions 2400 and 2401 respectively on the live view image 2404.

[0187] A user who sees the displayed icons 2405 and 2406 recognizes the notification for counterclockwise rotation of the camera device 1900 and rotates the camera device 1900 counterclockwise. Figure 24B Shows the state after the camera device 1900 rotates counterclockwise from Figure 24A the state.

[0188] Similarly, in Figure 24B the state, since the icons 2409 and 2410 are still displayed, the user similarly recognizes the notification for counterclockwise rotation of the camera device 1900 and rotates the camera device 1900 counterclockwise. Here, both the icon 2406 and the icon 2410 indicate that clockwise rotation is not required. On the other hand, both the icon 2405 and the icon 2409 indicate that counterclockwise rotation is required, but the icon 2409 indicates a rotation with a smaller rotation amount compared to the icon 2405. Figure 24C Shows the state after the camera device 1900 further rotates counterclockwise from Figure 24B the state.

[0189] In Figure 24C it, the icon 2413 indicating that counterclockwise rotation is not required and the icon 2414 indicating that clockwise rotation is not required are displayed. A user who sees the displayed icons 2413 and 2414 recognizes the notification indicating that the camera device 1900 does not need to be rotated clockwise or counterclockwise and does not rotate the camera device 1900.

[0190] Figure 23 Shows the state where the camera device 1900 is assembled on the pan-tilt device 2000, and the remote control device 2010 for the pan / tilt operation of the pan-tilt device 2000 and the imaging operation of the camera device 1900 is connected. At this time, by connecting to the camera device 1900 via the external I / F 1910 of the camera device 1900, the remote control device 2010 can perform imaging by using the camera device 1900.

[0191] Return to reference Figure 22 , in step S2215, the CPU 1901 determines whether the condition for ending the processing according to Figure 22 the flowchart is satisfied. For example, when the user inputs an instruction for ending the processing by operating the operation unit 1906 or disconnects the power supply of the camera device 1900, the CPU 1901 determines that the condition for ending the processing according to Figure 22 the flowchart is satisfied.

[0192] As a result of this determination, if the condition for ending the processing according to Figure 22 the flowchart is satisfied, according to Figure 22The processing of the flowchart ends. If the end condition is not satisfied, the processing proceeds to step S2203.

[0193] In the above manner, by installing the pan-tilt device 2000 equipped with the camera device 1900 as described above Figure 23 towards the inspection object surface, it is possible to notify the user of the rotation / translation instruction information for aligning the camera device 1900 with the inspection object surface. In addition, since the user who receives the notification operates the pan-tilt device 2000 or the like based on this notification, the camera device 1900 can be accurately aligned with the inspection object surface, and accurate inspection of deformation becomes possible. At the same time, by accurately aligning the camera device 1900 with the inspection object surface, when imaging an area adjacent to the inspection object surface, by translating the camera device 1900, it is possible to continuously image the inspection object surface under unified conditions. In addition, even when the imaging element 1904 or the lens device 1913 of the camera device 1900 deviates from the designed position due to long-term changes, since the camera device 1900 is accurately aligned with the inspection object surface, accurate inspection of deformation is also possible.

[0194] In the present embodiment, although the rotation direction for aligning the camera device 1900 with the inspection object surface is the horizontal (rotation) direction and the pan axis of the pan-tilt device 2000 is operated, a rotation instruction for aligning the camera device 1900 with the inspection object surface in the vertical (rotation) direction can be issued by switching the alignment detection direction, and the tilt axis can be operated. In addition, detection in the horizontal (rotation) direction and the vertical (rotation) direction can be performed simultaneously, and the rotation instruction information in these two directions can be presented.

[0195] In addition, in the present embodiment, although an example of the value of the defocus amount is presented and the rotation instruction information is defined as three types, since the value of the defocus amount varies depending on the type of the image plane phase difference sensor to be used, a coefficient or the like can be appropriately multiplied for use, and the types are not limited to these.

[0196] In addition, in the present embodiment, although an icon indicating both the rotation direction and the rotation degree is displayed, an icon indicating the rotation direction and an icon indicating the rotation degree can be displayed separately, or only one of the two can be displayed. In addition, the information indicating the rotation direction or the rotation degree is not limited to an icon, and for example, it can be text information. In addition, the method for notifying the rotation direction or the rotation degree is not limited to a specific notification method.

[0197] In addition, in the present embodiment, although an icon is displayed for the direction that does not require rotation, it is not necessary to display an icon for the direction that does not require rotation. In addition, for the direction that requires rotation, in addition to displaying an icon, other information such as text information can also be displayed.

[0198] In addition, in this embodiment, although the camera device 1900 is mounted on the pan-tilt device 200, as described above, the camera device 1900 can also be mounted on a UAV (unmanned aerial vehicle) such as a drone device. With this configuration, it is possible to directly face the inspection object surface of the object structure in an environment where a pan-tilt cannot be installed and capture an image thereof.

[0199] In addition, in this embodiment, although the rotation and / or translation instruction information is notified to the user, the rotation and / or translation instruction information can also be output to the pan-tilt device 2000. The pan-tilt device 2000 can be configured to control the rotation of the camera device 1900 according to the rotation and / or translation instruction information, and can automatically direct the camera device 1900 to face the inspection object surface. With this configuration, the operation load on the user is reduced, thereby improving convenience.

[0200] In addition, in this embodiment, although the camera device 1900 calculates the defocus amount (distance information distribution), as in the first embodiment, a computer communicably connected via a communication circuit can be configured to calculate the defocus amount.

[0201] Furthermore, in this embodiment, although the distance information distribution is calculated to control the position and posture of the camera device 1900 by operating the pan-tilt device 2000, the use of the calculated distance information distribution is not limited to this.

[0202] For example, the CPU 1901 records the data of a pair of parallax images captured by the imaging element 1904 and the imaging conditions including at least the F-number and the KX value in association with the image data on a memory card 1909 or the like. Based on the data of the pair of recorded images and the imaging conditions, the CPU 1901 or the CPU of an external device to which each data is output generates and acquires a distance information distribution. Here, the distance information distribution to be acquired is a defocus amount distribution, and a blur map is generated by converting each defocus amount based on the F-number (or effective F-number) as the imaging condition and the conversion coefficient KX. The blur map can be used for quality evaluation related to blur in a captured image. In particular, in imaging for social infrastructure inspection, when inspecting deformation or the like on an inspection object surface, crack detection and crack width measurement cannot be correctly performed unless evaluation is made using an image in which the inspection object surface is not blurred. Therefore, by referring to the defocus amount distribution (or blur map), for example, by performing measurement only on a non-blurred area (imaging range), more accurate inspection can be performed. Additionally, for example, if it is determined that blur having a blur amount greater than or equal to a reference occurs in a captured image at a predetermined ratio or greater, the CPU 1901 can notify the user that the captured image is unusable (deformation detection cannot be performed). As a notification method, an image or an icon can be displayed on the display unit 1905, or notification can be made using light, sound, vibration, or the like from another device. Additionally, the CPU 1901 can generate the above-described blur map, can generate an image in which each blur amount is simply visualized, and can display the image on the display unit. For example, by referring to the blur map, the user can manually or automatically re-capture an image or move the camera device 1900.

[0203] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, the appended claims are added to disclose the scope of the present invention.

[0204] [Other Embodiments]

[0205] The object of the present invention can also be achieved as follows. More specifically, a storage medium storing program code of software that describes a process for implementing the functions of the above-described embodiments is provided to a system or a device. Then, a computer (or a CPU, an MPU, etc.) of the system or the device reads and executes the program code stored in the storage medium.

[0206] In this case, the program code read from the storage medium itself implements the novel functions of the present invention. The storage medium storing the program code and the program constitute the present invention.

[0207] As a storage medium for providing program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, etc. can be given. Additionally, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, non-volatile memory card, or ROM, etc. can be used.

[0208] Furthermore, the functions of the above-described embodiments are implemented by making the program code read by the computer executable. In addition, there is also the following case: an OS (operating system) or the like operating on the computer performs part or all of the actual processing according to the instructions of the program code, and the functions of the above-described embodiments are implemented through these processes.

[0209] In addition, there is also the following case. First, the program code read from the storage medium is written into the memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, a CPU or the like included in the function expansion board or the function expansion unit performs part or all of the actual processing according to the instructions of the program code.

[0210] This application claims the benefit of Japanese Patent Application No. 2019-140818 filed on July 31, 2019 and Japanese Patent Application No. 2020-124031 filed on July 20, 2020, the entire contents of both of which are incorporated herein by reference.

Claims

1. An image processing apparatus, comprising: an input component configured to input a distance information distribution, the distance information distribution being calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging component; an estimation component configured to estimate a depth direction in the image according to an imaging condition of the imaging component; and a determination component configured to determine an evaluation value indicating a deviation degree of the optical system and the imaging element from a design position according to a relationship between the distance information distribution and the estimated depth direction, wherein the imaging condition is at least one of the following: posture information of the apparatus when capturing the image, a vanishing point in the image, a change in density of texture in the image, and a determination result related to whether a structure having a known shape is included in the image.

2. The image processing apparatus according to claim 1, wherein, The distance information distribution is information related to a distribution obtained by normalizing a defocus amount distribution of a subject by an F-number and a circle of confusion.

3. The image processing apparatus according to claim 1, wherein, The distance information distribution is any one of the following information: information related to a distribution of a parallax amount of a subject, information related to a distribution of a defocus amount of a subject, information related to a distribution obtained by normalizing a defocus amount distribution of a subject by an F-number and a circle of confusion, and information related to a distribution of an actual distance from an imaging position to a subject.

4. The image processing apparatus according to claim 3, wherein The information related to the distribution of the parallax amount of the subject is obtained from a pair of images having parallax.

5. The image processing apparatus according to any one of claims 1 to 4, wherein, The relationship of the depth direction is an angle formed by a straight line where the defocus amount is zero in the distance information distribution and a straight line representing a focused area calculated by the component for estimating the depth direction.

6. The image processing apparatus according to any one of claims 1 to 4, wherein, The relationship of the depth direction is a difference between the following vectors: a vector of a gradient of the defocus amount in the distance information distribution; and a vector of a direction toward a vanishing point in the image or a vector of a change direction of density of texture in the image calculated by the component for estimating the depth direction.

7. The image processing apparatus according to any one of claims 1 to 4, further comprising a notification component configured to notify the evaluation value.

8. The image processing apparatus according to any one of claims 1 to 4, wherein, In a case where the input image is determined to include a typical object, the determination component determines an evaluation value indicating the deviation degree, the typical object including a ground, a water surface, and a structure built in a vertical direction of the ground or the water surface.

9. The image processing apparatus according to claim 8, wherein, The statistics of the distance information distribution is a histogram of the distance information distribution.

10. The image processing apparatus according to any one of claims 1 to 4, wherein, Correction is performed by controlling an IS mechanism, performing image processing on the image, or rotating the image processing apparatus according to the evaluation value indicating the deviation degree, so as to reduce the deviation degree.

11. The image processing apparatus according to any one of claims 1 to 4, wherein, By associating the evaluation value indicating the deviation degree with information related to the imaging element and the optical system of the image for which the evaluation value is calculated, the information is output to an external device.

12. An image processing apparatus, comprising: an input component configured to input a distance information distribution, the distance information distribution being calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging component; An estimation component for estimating the depth direction in the image according to the imaging conditions of the imaging component; and A determination component for determining an evaluation value representing the degree of deviation in the depth direction of the subject in the image according to the relationship between the distance information distribution and the estimated depth direction, wherein the imaging conditions are at least one of the following: the posture information of the device when the image is captured, the vanishing point in the image, the change in the density of the texture in the image, and the judgment result related to whether a structure with a known shape is included in the image.

13. The image processing apparatus according to claim 12, wherein, The distance information distribution is information related to the distribution obtained by normalizing the defocus amount distribution of the subject using the F-number and the circle of confusion.

14. The image processing apparatus according to claim 12, wherein, The distance information distribution is any one of the following information: information related to the distribution of the parallax amount of the subject, information related to the distribution of the defocus amount of the subject, information related to the distribution obtained by normalizing the defocus amount distribution of the subject using the F-number and the circle of confusion, and information related to the distribution of the actual distance from the imaging position to the subject.

15. The image processing apparatus according to claim 14, wherein, The information related to the distribution of the parallax amount of the subject is obtained from a pair of images with parallax.

16. The image processing apparatus according to any one of claims 12 to 15, wherein, The relationship of the depth direction is the angle formed by the straight line with zero defocus amount in the distance information distribution and the straight line representing the focused area calculated by the component for estimating the depth direction.

17. The image processing apparatus according to any one of claims 12 to 15, wherein, The relationship of the depth direction is the difference between the following vectors: the vector of the gradient of the defocus amount in the distance information distribution; and the vector towards the vanishing point in the image or the vector of the change direction of the density of the texture in the image calculated by the component for estimating the depth direction.

18. The image processing apparatus according to claim 12, wherein, The degree of deviation in the depth direction of the subject in the image is the difference between the defocus amounts at multiple positions in the distance information distribution.

19. The image processing device according to any one of claims 12 to 15, further comprising a notification component for notifying the evaluation value.

20. The image processing apparatus according to any one of claims 12 to 15, wherein, In the case where the input image is determined to include typical objects, the determination component determines an evaluation value representing the degree of deviation, and the typical objects include the ground, the water surface, and structures built in the vertical direction of the ground or the water surface.

21. The image processing apparatus according to claim 20, wherein, The statistics of the distance information distribution is the histogram of the distance information distribution.

22. The image processing apparatus according to any one of claims 12 to 15, wherein, Correction is performed by controlling the IS mechanism, performing image processing on the image, or rotating the image processing device according to the evaluation value representing the degree of deviation, so as to reduce the degree of deviation.

23. The image processing apparatus according to any one of claims 12 to 15, wherein, By associating the evaluation value representing the degree of deviation with the information related to the imaging element and the optical system of the image from which the evaluation value is calculated, the information is output to an external device.

24. An image processing method, comprising: An input step for inputting a distance information distribution, which is calculated from an image captured by using an optical system for forming a field image on the imaging element of an imaging component; An estimation step for estimating the depth direction in the image according to the imaging conditions of the imaging component; and A determination step for determining an evaluation value representing the degree of deviation of the optical system and the imaging element from the design position based on the relationship between the distance information distribution and the estimated depth direction. Wherein the imaging condition is at least one of the following: the posture information of the device when the image is captured, the vanishing point in the image, the change in the density of the texture in the image, and the judgment result related to whether a structure with a known shape is included in the image.

25. An image processing method, comprising: An input step for inputting a distance information distribution calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging component. An estimation step for estimating the depth direction in the image according to the imaging condition of the imaging component. And A determination step for determining an evaluation value representing the degree of deviation in the depth direction of the subject in the image based on the relationship between the distance information distribution and the estimated depth direction. Wherein the imaging condition is at least one of the following: the posture information of the device when the image is captured, the vanishing point in the image, the change in the density of the texture in the image, and the judgment result related to whether a structure with a known shape is included in the image.

26. A computer-executable program product comprising computer-executable program instructions for causing a computer to execute the functions of the components of the image processing device according to any one of claims 1 to 23.

27. A computer-readable storage medium storing a program for causing a computer to execute the functions of the components of the image processing device according to any one of claims 1 to 23.

Images