Image processing device, image processing method and program

Abstract

To make it possible to report at least one piece of information about the tilt of a lens or an imaging element, or the position or attitude of an imaging apparatus.SOLUTION: An image processing device has input means for inputting a distance information distribution calculated from an image captured using an optical system that forms an image of a field on an imaging element of imaging means, estimation means for estimating the depth direction in the image from photographing conditions of the imaging means, and determination means for determining an evaluation value that indicates the degree of deviation in the depth direction of a subject in the image from the relationship between the distance information distribution and the estimated depth direction.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device, and more particularly to information relating to aging of an optical system and an image sensor, and information on the posture of the image processing device. [Background technology]

[0002] Conventionally, there is known a technique for assisting the calibration of a stereo camera by referring to distance information acquired from a pair of stereo cameras to diagnose changes in the relative positional relationship of the stereo cameras due to aging or the like. For example, Patent Document 1 discloses a method in which a stereo camera mounted on the head of a robot captures an image of a subject on a substantially flat surface provided with a diagnostic texture at a predetermined positional relationship, calculates distance information from the obtained parallax image, and determines flatness. The method then discloses a method for determining whether calibration is necessary by comparing the determined flatness with a predetermined reference amount. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-306249 Summary of the Invention [Problem to be solved by the invention]

[0004] Even in digital cameras used by general users and not mounted on robots, the optical system, such as lenses and image sensors like CMOS, can shift from their original mounting positions (design positions) due to aging and other factors. If the lens or image sensor is tilted, the relationship between the actual distance and depth of field will be distorted, resulting in images that the user did not expect. Therefore, even in digital cameras used by general users, there is a need for a method to determine whether lens and image sensor calibration is necessary and how to deal with this situation.

[0005] Furthermore, when photographing a subject with a digital camera, even if the lens and image sensor are calibrated, if the imaging device is tilted or the image is not taken at an appropriate distance, a good image cannot be obtained. In particular, tilt in the depth direction and distance errors can lead to blurring of the target subject in the photographed image.

[0006] Therefore, an object of the present invention is to provide an image processing device that can notify at least one piece of information, such as the tilt of the lens or the image sensor, or the position or attitude of the image capturing device, based on the distance information distribution corresponding to the distance to the subject. [Means for solving the problem]

[0007] In order to solve the above problem, the image processing device of the present invention uses a distance information distribution calculated from an image captured using an optical system that forms an image of an object scene on an image sensor of an image capturing means. acquisition do acquisition means and 、 an estimation means for estimating a depth direction in the image from the photographing conditions of the imaging means; 、 From the relationship between the distance information distribution and the estimated depth direction, 、 determining means for determining an evaluation value indicating the degree of deviation of the subject in the image in the depth direction; The distance information distribution is information related to the distribution of the defocus amount of the subject normalized by the F-number and the allowable circle of confusion diameter. It is characterized by:

[0008] The image processing method of the present invention also provides a method for processing a distance information distribution calculated from an image captured using an optical system that forms an image of a scene on an image sensor of an image capturing means. acquisition do acquisition Steps and 、 an estimation step of estimating a depth direction in the image from the photographing conditions of the imaging means; 、 From the relationship between the distance information distribution and the estimated depth direction, 、 a determining step of determining an evaluation value indicating a degree of deviation in the depth direction of the subject in the image. The distance information distribution is information related to the distribution of the defocus amount of the subject normalized by the F-number and the allowable circle of confusion diameter. It is characterized by: [Effects of the Invention]

[0010] According to the present invention, it is possible to notify at least one piece of information about the tilt of the lens or the imaging element, or the position or orientation of the imaging device. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of the functional arrangement of a digital camera according to a first embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram showing an example of the functional arrangement of a computer according to a first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating an example of the arrangement of an imaging unit according to the first embodiment of the present invention. [Figure 5] 5 is a flowchart illustrating the operation of the image processing device according to the first embodiment of the present invention. [Figure 6] 1A to 1C are diagrams illustrating images for recording still images according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating a defocus map according to the first embodiment of the present invention. [Figure 8] FIG. 3 is a block diagram showing an example of the functional arrangement of an image processing unit 306 according to the first embodiment of the present invention. [Figure 9] FIG. 2 is a diagram for explaining a surface where the defocus amount is zero according to the first embodiment of the present invention. [Figure 10] FIG. 3 is a diagram illustrating a defocus map when the focus plane is normal according to the first embodiment of the present invention. [Figure 11] 3A and 3B are diagrams for explaining a phenomenon that occurs when the optical system and the image sensor according to the first embodiment of the present invention are displaced from their designed positions. [Figure 12] FIG. 3 is a diagram illustrating a defocus map when the focal plane is tilted according to the first embodiment of the present invention. [Figure 13] FIG. 4 is a diagram for explaining an evaluation value indicating the degree of deviation according to the first embodiment of the present invention. [Figure 14] FIG. 2 is a diagram for explaining a notification to a user according to the first embodiment of the present invention. [Figure 15] 5A to 5C are diagrams for explaining the estimation results of the vanishing point and the depth direction according to the first embodiment of the present invention. [Figure 16] FIG. 3 is a diagram illustrating a histogram for a defocus map according to the first embodiment of the present invention. [Figure 17] 3A to 3C are diagrams illustrating an image for recording a still image in a portrait scene, a defocus map, and a histogram for the defocus map according to the first embodiment of the present invention. [Figure 18] FIG. 2 is a diagram illustrating the peripheral light reduction characteristics of the optical system according to the first embodiment of the present invention. [Figure 19A] FIG. 19 is a block diagram showing an example of the hardware arrangement of a camera device 1900 and a lens device 1913 according to a second embodiment of the present invention. [Figure 19B] FIG. 19 is a block diagram showing an example of the functional arrangement of a camera device 1900 according to a second embodiment of the present invention. [Figure 20] FIG. 10 is a block diagram showing an example of the hardware configuration of a pan head device 2000 according to a second embodiment of the present invention. [Figure 21] 10A and 10B are diagrams illustrating an imaging method for imaging a social infrastructure structure according to a second embodiment of the present invention. [Figure 22] 10 is a flowchart showing the operation of the imaging system according to the second embodiment of the present invention. [Figure 23] FIG. 2 is a diagram for explaining a switch 2007 according to a second embodiment of the present invention. [Figure 24] 19A and 19B are diagrams relating to rotation control of a camera device 1900 according to a second embodiment of the present invention. [Figure 25] FIG. 10 is a diagram showing an example of the configuration of a table 2515 according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] [First embodiment] An image processing device, an image processing method, and an image processing program according to a first embodiment of the present invention will be described below in detail with reference to the drawings. As shown in FIG. 1, an example in which the present invention is applied to an image processing device 100 will be described, in which a digital camera 101 as an example of an imaging device and a computer 102 as an example of an image processing device are communicatively connected via a communication circuit 103. However, the processing performed by the computer 102 below may also be performed by the digital camera 101. Furthermore, the digital camera 101 may be any electronic device with a photographing function, and the computer 102 may be any electronic device capable of performing the processing described below, or a computer in a server device. Furthermore, the computer may be a mobile computer or a desktop computer.

[0013] 2 is a block diagram showing an example of the functional configuration of a digital camera 101 according to an embodiment of the present invention. A system control unit 201, which is, for example, a CPU, reads out an operation program for each block of the digital camera 101 from a ROM 202, expands it into a RAM 203, and executes it to control the operation of each block of the digital camera 101. The ROM 202 is a rewritable non-volatile memory that stores the operation program for each block of the digital camera 101 as well as parameters and the like required for the operation of each block. The RAM 203 is a rewritable volatile memory that is used as a temporary storage area for data output in the operation of each block of the digital camera 101.

[0014] The optical system 204 forms an image of the object scene on the imaging unit 205. The imaging unit 205 is an imaging element such as a CCD or CMOS sensor, and photoelectrically converts the optical image formed on the imaging element of the imaging unit 205 by the optical system 204, and outputs the obtained analog image signal to the A / D conversion unit 206. The optical system 204 and the imaging unit 205 are also equipped with an IS mechanism that reduces the effects of camera shake. The A / D conversion unit 206 applies A / D conversion processing to the input analog image signal, and outputs the obtained digital image data to the RAM 203 for storage.

[0015] An image processing unit 207 applies various image processing to the image data stored in the RAM 203, such as white balance adjustment, color interpolation, reduction / enlargement, and filtering.

[0016] The recording medium 208 is a removable memory card or the like, and images processed by the image processing unit 207 stored in the RAM 203 and images A / D converted by the A / D conversion unit 206 are recorded as recorded images.

[0017] A communication unit 209 transmits image data files and the like recorded on the recording medium 208 to an external device via wired or wireless communication.

[0018] The display unit 210 displays image data obtained by shooting, image data read from the recording medium 208, and various menu screens. It can also display a live view image and function as an electronic viewfinder.

[0019] The operation unit 211 is a group of input devices that allow the user to input various instructions and settings to the digital camera 101, and includes keys and buttons that are typically found on digital cameras, such as a shutter button, menu button, directional keys, and an enter key. If the display unit 210 is a touch display, it also serves as the operation unit 211. The operation unit 211 may be configured in a way that does not require physical operation, such as by combining a microphone and a voice command recognition unit.

[0020] The detection unit 212 includes a gyro and a sensor, and acquires angular velocity information, attitude information, etc. of the digital camera 101. The attitude information includes information such as the tilt of the digital camera 101 relative to the horizontal direction.

[0021] 3 is a block diagram showing an example of the functional configuration of the computer 102 according to this embodiment. The system control unit 301 is, for example, a CPU, and controls the operation of each block of the computer 102 by reading a program from a ROM 302, expanding it into a RAM 303, and executing it. The ROM 302 is a rewritable non-volatile memory, and stores parameters and the like necessary for controlling each block in addition to the program executed by the system control unit 301. The RAM 303 is a rewritable volatile memory, and is used as a temporary storage area for data output by each block of the computer 102.

[0022] The communication unit 304 communicates with external devices such as the digital camera 101 via wired or wireless communication. The recording device 305 is, for example, a hard disk, and stores image data received by the communication unit 304 from the digital camera 101.

[0023] The image processing unit 306 performs, for example, calculation of the defocus amount (to be described later) for image data expanded from the recording device 305 to the RAM 303, estimation of the depth direction from the image, and calculation of information related to the degree of deviation from the design positions of the optical system and the image sensor.

[0024] The display unit 307 is used to display a GUI and various data provided by the OS and applications running on the computer 102. The display unit 307 may be provided in the computer 102 or may be connected as an external device.

[0025] The operation unit 308 is a group of input devices that allow the user to input various instructions and settings to the computer 102, and generally includes a keyboard, a mouse, a trackpad, etc. If the display unit 307 is a touch display, it also serves as the operation unit 308. The operation unit 308 may be configured in a way that does not require physical operation, such as by combining a microphone and a voice command recognition unit.

[0026] Fig. 4(a) shows the pixel arrangement of the imaging unit 205 of Fig. 2. As shown in Fig. 4(a), in the imaging unit 205, a plurality of pixels 400 are arranged two-dimensionally and regularly. Specifically, the plurality of pixels 400 are arranged, for example, in a two-dimensional lattice pattern. Note that the arrangement of the pixels 400 is not limited to a lattice-like arrangement, and other arrangements may also be adopted.

[0027] FIG. 4(b) is an enlarged view of the pixel 400 shown in FIG. 4(a). As shown in FIG. 4(b), each pixel 400 includes a microlens 401 and a pair of photoelectric conversion units 402A and 403B (hereinafter referred to as pupil-divided pixels 402A and 403B, respectively). The pupil-divided pixels 402A and 403B have the same planar shape, each a rectangular shape with its longitudinal direction in the y-axis direction. In each pixel 400, the pupil-divided pixels 402A and 403B are arranged symmetrically with respect to the perpendicular bisector of the microlens 401 along the y-axis direction as the axis of symmetry. Note that the planar shapes of the pupil-divided pixels 402A and 403B are not limited to this, and other planar shapes can be used. Furthermore, the arrangement of the pupil-divided pixels 402A and 403B is not limited to this, and other arrangements can be used.

[0028] In this embodiment, images A and B are output as parallax images from pupil division pixels 402A and 403B, which are regularly arranged two-dimensionally. Furthermore, an image A+B, which is the sum of images A and B, is recorded on the recording medium 208 as a recorded still image. By configuring the imaging unit 205 as shown in FIGS. 4(a) and 2(b), a pair of light beams passing through different regions of the pupil of the optical system 204 can be formed as a pair of optical images, which can be output as images A and B. Note that the method for acquiring images A and B is not limited to the above, and various methods can be used. For example, images A and B may be images with mutual parallax acquired by imaging devices such as multiple cameras installed at spatial intervals. Furthermore, images A and B may be parallax images acquired by an imaging device such as a single camera having multiple optical systems and imaging units.

[0029] The operation of the image processing device 100 will be described below. When a shooting instruction, such as fully pressing the shutter button, is input via the operation unit 211 of the digital camera 101, the image processing device 100 executes the processes shown in Figures 5(a), 5(b), and 5(c). It is assumed that the processes in Figures 5(a) and 5(c) are executed by the digital camera 101, and the process in Figure 5(b) is executed by the computer 102.

[0030] First, in step S500, the system control unit 201 detects the state of the camera when the shutter button is pressed via the detection unit 212. Here, the tilt of the digital camera 101 relative to the horizontal direction and the orientation in the up-down direction are detected as the state of the camera.

[0031] In the next step S501, the system control unit 201 performs shooting processing according to the exposure conditions determined in the shooting preparation state, and acquires a pair of parallax images, image A and image B, from the imaging unit 205. Note that a configuration may be adopted in which image A and image B are read out and acquired from images previously recorded on the recording medium 208. Alternatively, image A and image B may be added together and recorded on the recording medium 208 as an image for recording a still image. An image for recording a still image in this embodiment is shown in FIG. 6. FIG. 6 shows an image obtained by adding together the captured image A and image B. Also, reference numeral 600 denotes an autofocus frame.

[0032] In the following step S502, the system control unit 201 controls the image processing unit 207 to output data representing the spatial (two-dimensional) distribution of defocus amounts in the shooting range from the parallax images acquired in step S501. In the following description, the data representing the spatial distribution of defocus amounts will be referred to as a defocus map. The defocus amount is a type of distance information, since it is the amount of deviation from the focus distance of the optical system 204. A method of calculating the phase difference between parallax images, such as the method disclosed in Japanese Patent Application Laid-Open No. 2008-15754, may be used to acquire the defocus amount. Specifically, the relationship between the deviation amount of the parallax images and the defocus amount is expressed by the following equation: DEF=KX PY x (1)

[0033] In equation (1), DEF is the defocus amount, PY is the detection pitch (the arrangement pitch of pixels of the same type), KX is a conversion coefficient determined by the magnitude of the opening angle of the center of gravity of the light beam passing through a pair of pupils, and x is the amount of shift of the parallax images.

[0034] Furthermore, the present invention is not limited to this, and a distribution of shift amounts, which are the amounts of deviation of parallax images, may be acquired as the distance information distribution.

[0035] The distance information distribution may also be information expressed in units of length such as micrometers by multiplying the displacement amount of the parallax image by the detection pitch PY.

[0036] The present invention is not limited to this, and the distance information distribution may be converted from the defocus amount to a distribution of actual distances with reference to the focus lens position.

[0037] The present invention is not limited to this, and a distribution of values ​​obtained by normalizing the defocus amount by Fδ (F is the aperture value, and δ is the diameter of the permissible circle of confusion) may be obtained as the distance information distribution. This distribution represents the amount of blur relative to δ. Here, the aperture value F may be applied to the entire distribution. However, to obtain a more accurate distribution of the amount of blur, it is preferable to apply an effective aperture value (effective aperture value) that takes into account the vignetting characteristics of the optical system 204 under the shooting conditions. FIG. 18 is a graph showing an example of vignetting characteristics V(h). The horizontal axis represents the distance from the optical center (image height), and the vertical axis represents the amount of light at each image height, normalized with the amount of light at the center of the image height set to 1. Vignetting occurs due to the lens frame and aperture frame, and in FIG. 18, the amount of light decreases as the image height increases (approaching the edge of the shooting range). This vignetting characteristic has unique properties for each lens. Here, the effective aperture value F' at image height h is expressed by the following equation, referring to the vignetting characteristics: F´=F / √V(h) (2)

[0038] A defocus map for the image in Figure 6 is shown in Figure 7. The defocus map 700 is expressed as a continuous grayscale value that becomes whiter (higher pixel values) as the distance becomes closer. 701 is an autofocus frame, and in-focus areas (where the defocus amount is zero) are expressed in gray. 702 is a line connecting the in-focus areas.

[0039] Next, in step S503, the system control unit 201 transmits the image data and the following information to the computer 102 via the communication unit 209. -Images for recording still images Defocus Map Camera status detection information (detected camera tilt information) Autofocus frame position information · Camera body identification number (ID) · Lens identification number (ID) Shooting information such as F-stop and ISO sensitivity

[0040] The above information is recorded or transmitted in association with each other. For example, if the format is JPEG, it may be recorded as Exif information, or if the format is RAW data, it may be recorded as image supplementary information in a single file. Alternatively, the necessary information may be recorded or transmitted together with the image as a container file that can store multiple related data together. Alternatively, the necessary information may be recorded or transmitted as separate files without being combined. For example, the data must be processed so that they can be identified as related data files, such as by using the same file name, including them in the same folder, or transmitting each data sequentially in order (so that the receiving party can recognize the related information from the order and type of the data). The file structure and transmission control according to the transmission protocol involved in recording and transmission are not directly related to the present invention, and well-known methods can be used, so detailed explanations are omitted. Note that the above information may be recorded on the recording medium 208 and then transmitted to the computer 102 via the communication unit 209, or the recording medium 208 may be removed from the digital camera 101 and the image data read by the computer 102. Furthermore, the defocus map (recorded information distribution) may not be generated on the camera side, but may be generated by the computer 102 by recording the corresponding parallax image together with the related information.

[0041] In this embodiment, the processing from step S504 to step S508 is executed by the computer 102. By executing the processing from step S504 to step S508 on a device other than the digital camera 101, the user can know information related to the degree of deviation of the optical system and the image sensor from their designed positions without performing any special camera operations. Furthermore, although the defocus map is generated in the digital camera 101 in this embodiment, a configuration is also possible in which a parallax image is transmitted to the computer 102, and the defocus map is generated in the computer 102. When the computational load on the digital camera 101 is high during continuous shooting, distributing the computation for generating the defocus map to the computer 102 can reduce the time required to calculate information related to the degree of deviation of the optical system and the image sensor from their designed positions. Furthermore, by transmitting information on the conversion coefficient KX and the effective aperture value F', which are uniquely determined by the lens being used and the shooting conditions, various distance information distributions can also be generated on the computer 102 side. Alternatively, the information on the conversion coefficient KX and the effective aperture value F' may be stored in advance in the computer 102, and read from the stored information based on the received lens identification (ID) number and shooting information.

[0042] 8 is a block diagram showing a schematic example of the functional configuration of the image processing unit 306 of the computer 102 according to this embodiment. The operation of the image processing unit 306 will be described below with further reference to FIG. 5(b). The operation of the image processing unit 306 is realized under the control of the system control unit 301.

[0043] First, the system control unit 301 receives the information transmitted in step S503 and loads the read data into the RAM 303 (step S504).

[0044] Next, in step S505, the depth direction estimation unit 800 estimates the depth direction of the image from the camera state detection information 803 (shooting conditions) at the time of acquiring the parallax image, which is recorded in the RAM 303. In this embodiment, the depth direction is estimated with reference to a plane where the defocus amount is zero. Here, the plane where the defocus amount is zero will be described with reference to FIG. 9.

[0045] 9 shows a state in which a planar subject 901 on the ground is photographed from a bird's-eye view of a digital camera 101 with the optical system and image sensor not displaced from their designed positions and with the digital camera 101 tilted zero relative to the horizontal direction (the x-axis direction in FIG. 9). If an autofocus frame 903 is located at the point where an optical axis 900 intersects with the subject 901, the plane connecting the in-focus areas (hereinafter referred to as the focal plane) is a plane 902 that is parallel to the imaging unit 205 and perpendicular to the optical axis 900. Furthermore, the focal plane 902 in the image of the subject 901 can be represented by a straight line 904 that passes through the autofocus frame 903. In FIG. 10, 1000 indicates a defocus map for the image of the subject 901, 1001 indicates the autofocus frame, and 1002 indicates the focal plane. 10, when a camera with the optical system and image sensor not displaced from their designed positions captures an image from above with zero tilt relative to the horizontal direction, the focal plane 1002 is a horizontal straight line relative to the image. Furthermore, since the vertical orientation of the camera at the time of capture is known, the area above the focal plane 1002 is farther away and the area below is closer. In other words, it can be estimated that the depth direction changes from the bottom to the top of the captured image. The depth direction estimation unit 800 outputs the equation for the straight line representing the focal plane 1002 as depth estimation information 804.

[0046] The deviation calculation unit 801 calculates information 805 related to the degree of deviation from the designed position of the optical system and the image sensor that captured the parallax image, based on the defocus map 802 and the depth estimation information 804 (the equation of the line representing the focal plane) obtained by the depth direction estimation unit 800 (step S506).

[0047] Here, we will use FIG. 11 to explain the phenomenon that occurs when the optical system and image sensor are displaced from their designed positions. FIG. 11(a) shows the state in which the optical system 204 and image sensor 205 are positioned at their designed positions. When an image is captured in this state, the focal plane 1100 is parallel to the image sensor 205. On the other hand, FIG. 11(b) shows the state in which the optical system 204 is displaced from its designed position, causing decentering. In this case, according to Scheimpflug's law, the focal plane 1101 also tilts according to the angle θ between the optical system 204 and image sensor 205. In the state shown in FIG. 11(b), the camera is facing downwards, and its tilt relative to the horizontal direction is zero. The defocus map for an image captured of a planar subject on the ground is shown in FIG. 12 as 1200, the autofocus frame as 1201, and the focal plane as 1202. From FIG. 12, we can see that the depth changes from the bottom right to the top left of the screen. Therefore, a deviation occurs from the direction of depth change (FIG. 10) when the optical system and image sensor are not displaced from their designed positions. As a result, the relationship between the user's sense of distance and the focal plane becomes misaligned, resulting in photographic results that the user did not expect.

[0048] The deviation calculation unit 801 calculates the angle θ_diff formed between the equation of the line representing the focal plane 1202 in the defocus map 802 and the line 1002 representing the focal plane estimated by the depth direction estimation unit 800, and stores this angle in the RAM 303 as an evaluation value 805 indicating the degree of deviation. FIG. 13 shows θ_diff. The larger θ_diff is, the greater the deviation of the optical system and the image sensor from their designed positions (calibration is necessary). Note that if the camera is tilted relative to the horizontal direction when capturing an image, the angle of tilt can be subtracted from θ_diff for correction, and the effects of the present invention can be obtained even when the image is captured with the camera tilted relative to the horizontal direction.

[0049] In the following step S507, the system control unit 301 compares the calculated θ_diff with a threshold value stored in advance, and if θ_diff is greater than the threshold value, executes step S508, and if θ_diff is equal to or less than the threshold value, ends the process.

[0050] In step S508, the system control unit 301 transmits the following information to the digital camera 101 via the communication unit 304 to notify the user that the optical system and image sensor in the camera they are using are misaligned from their designed positions: -Identification number of the camera that detected the misalignment -Identification number of the lens that detected the misalignment In step S509, the system control unit 201 in the digital camera 101 checks whether the information transmitted from the computer 102 has been received. If the information has been received, step S510 is executed; if the information has not been received, the process ends.

[0051] 14 on the display unit 210, recommending that the user send the camera and lens to the customer center for repair. The customer center receives the image data from the user (digital camera 101) along with camera and lens ID information, which is useful for identifying repair and malfunction information, statistics, etc.

[0052] As described above, according to this embodiment, it is possible to calculate information relating to the degree of deviation from the designed positions of the optical system and the image sensor and notify the user of this information without impairing user convenience.

[0053] Furthermore, in this embodiment, a message is displayed informing the user that the optical system and the image sensor are misaligned from their designed positions, but to make it easier for the user to recognize the occurrence of the misalignment, an image may be displayed on the display unit 210. Specifically, the grayscale defocus map of Fig. 12 generated in step S502, or a defocus map converted into color values ​​by lookup table conversion or the like, may be displayed on the display unit 210.

[0054] In addition, in the present embodiment, information is displayed to the user when deviation from the design position occurs in the optical system or the image sensor, but the information may be displayed when deviation cannot be detected, or in either case. By configuring in this way, the user can know whether calibration is necessary when he or she needs the judgment result immediately.

[0055] Furthermore, in the present embodiment, an example has been described in which a defocus map is generated from the calculation of the amount of parallax using a pair of parallax images, but the present invention is not limited to this. A defocus map may be generated, for example, by using the Depth From Defocus (DFD) method, which acquires a defocus map from the correlation between two images with different focus positions or aperture values. Images acquired in aperture bracket shooting mode can be used to calculate information related to the degree of deviation of the optical system and image sensor from their designed positions, increasing the opportunities to detect deviations and providing information to the user at the appropriate time.

[0056] Furthermore, in this embodiment, the depth direction of an image is estimated from camera state detection information, but the present invention is not limited to this. For example, the depth direction can be estimated using information about the vanishing point. By configuring in this manner, the effects of the present invention can be obtained even with a camera that does not have a gyro or sensor for detecting the camera state, thereby improving user convenience. Below, we will explain how to estimate the depth direction using vanishing point detection and how to calculate information related to the degree of deviation from the designed positions of the optical system and image sensor.

[0057] A vanishing point is a point where the straight lines on the screen plane that correspond to parallel lines in a three-dimensional space converge when those parallel lines are projected onto an image plane using perspective transformation. In other words, a vanishing point is an "infinitely distant point" on a flat image onto which a space that actually has depth is projected, and is recognized as a point where extensions of lines parallel to the depth direction intersect, or a point where extensions of a surface extending in the depth direction converge to infinity. Therefore, straight lines in an image can be detected using a known method such as Hough transform, and the point where the detected multiple straight lines converge most frequently can be detected as the vanishing point. The results of detecting vanishing points in Figure 6 are shown in Figure 15.

[0058] 15 is the vanishing point. The depth direction can be estimated as the direction 1501 from the autofocus frame 600 toward the vanishing point. The depth direction estimation unit 800 outputs the direction 1501 toward the vanishing point as depth direction estimation information 804.

[0059] Deviation calculation unit 801 calculates the gradient (direction of change) of the defocus amount near the autofocus frame in defocus map 802 using known technology. Then, an evaluation value indicating the degree of deviation from the designed positions of the optical system and image sensor is calculated from the difference with depth direction estimation information 804. Specifically, the direction toward the vanishing point and the gradient direction of the defocus amount are each treated as vectors, and the difference between these vectors is used as the evaluation value. The greater the deviation of the optical system and image sensor from their designed positions, the greater the evaluation value.

[0060] Furthermore, the method of estimating the depth direction of an image by referring to features extracted from the image is not limited to the above-mentioned vanishing point detection; information on changes in texture density can also be used. The depth direction of an image can be detected by referring to changes in texture density. For example, the method described in "Texture Structure Classification and Depth Estimation using Multi-Scale Local Autocorrelation Features," by KANG Y, HASEGAWA O, and NAGAHASHI H (Tokyo Inst. Technol.), JST-PRESTO (non-patent document) can be used as a method.

[0061] Specifically, the depth direction estimation unit 800 references an image for still image recording and utilizes the fact that when a uniform texture is present in the image (e.g., a road in FIG. 6), the texture density decreases with distance. In other words, when the depth direction estimation unit 800 detects an area in the image where the density of a uniform texture gradually decreases, it determines that a plane covered by the specified texture is moving away from the shooting position. The direction of movement away from the foreground is output as depth direction estimation information 804. Since fine textures are particularly easy to detect in focused areas, performing the above-described determination on the vicinity of the autofocus frame allows for accurate estimation of the depth direction. Furthermore, known general object detection may be used to pre-detect areas likely to contain uniform textures, such as roads, water surfaces, or hedges, which are structures laid perpendicular to the ground or water surface, to limit the target area. This reduces the processing time required for estimating the depth distribution.

[0062] Then, the deviation calculation unit 801 uses the difference between the vector of the gradient (direction of change) of the defocus amount near the autofocus frame in the defocus map 802 and the depth direction vector estimated from the change in texture density as an evaluation value, just as when using a vanishing point.

[0063] In this embodiment, an evaluation value indicating the degree of deviation of the optical system and the image sensor from their designed positions is calculated from a single captured image, and a determination as to whether a deviation has occurred is made. However, the present invention is not limited to this, and a configuration in which a determination is made when the number of images captured by the user reaches a certain reference number is also possible. By making a determination based on evaluation values ​​for multiple captured images, the accuracy of determining whether a deviation has occurred can be improved. Furthermore, the condition for whether to notify the user that the optical system and the image sensor have detected a deviation from their designed positions can be further improved by checking whether the number of images in which a deviation has been detected has reached a certain reference number.

[0064] Furthermore, rather than determining the degree of deviation from the design positions of the optical system and image sensor for all captured images, it is preferable to first evaluate a large number of images to determine whether they are suitable for determining the degree of deviation and then perform the deviation determination process. Specifically, the above-mentioned general object detection is performed to evaluate whether the image has a uniform texture, such as a road, and is suitable for estimating the depth direction. This configuration reduces the processing time required to determine the degree of deviation. Furthermore, since objects with texture, such as roads, are also suitable for detecting the phase difference of parallax images in step S502, a more accurate deviation evaluation value can be expected. Furthermore, by recording publicly known GPS information as additional information for images, it is possible to determine whether a captured image is likely to contain an object with texture, such as a road. By selecting images that are expected to contain texture from the large number of images captured by the user in advance, the time required to calculate a highly accurate evaluation result can be shortened. Furthermore, for cameras with detachable optical systems, statistics are collected for each attached lens to determine whether the optical system and image sensor can detect deviation from the design positions. This makes it possible to determine whether the misalignment is occurring in the optical system or the image sensor, and provides more detailed judgment results to the user.

[0065] Furthermore, the deviation calculation unit 801 can obtain highly accurate evaluation results by considering the following factors to accurately obtain the gradient of the defocus map 802. Specifically, a histogram (statistical information) of the calculated defocus map is acquired, and whether the gradient of the defocus amount can be obtained accurately is determined from the shape of the histogram. As will be explained below, it is best to select an image with a wide histogram width and smooth change. FIG. 16 shows a histogram of the defocus map of FIG. 7. As shown in FIG. 16, FIG. 7 has a wide distribution of defocus amounts from the foreground to the background, and the defocus amount changes smoothly. Therefore, it can be said that this image is suitable for evaluating the direction of change in defocus amount throughout the entire image. On the other hand, FIG. 17(b) shows a defocus map and FIG. 17(c) shows a histogram for an image of a bust-up person captured in portrait photography, as shown in FIG. 17(a). It can be seen that the defocus amount in the image is concentrated around the bust-up person, making it unsuitable for evaluating the direction of change in defocus amount throughout the entire image. The deviation calculation unit 801 checks the histogram of the defocus map before comparing it with the depth direction estimation information 804, and if the image is not suitable for evaluating the deviation degree, the judgment process is interrupted, thereby reducing the calculation time.

[0066] Furthermore, in step S502, to accurately obtain the phase difference of the parallax images, it is sufficient to select images with a high S / N ratio. Therefore, by prioritizing the selection of images taken with as low a sensitivity as possible from a large number of images taken, the accuracy of the evaluation value can be improved.

[0067] Furthermore, since the phase difference of the parallax image is affected by aberration in the optical system 204, it is preferable to correct the aberration, which is known as design information of the optical system, before comparing the result with the result estimated by the depth direction estimation unit 800. Alternatively, the area to be compared may be limited to an area that is less affected by aberration. By taking the above measures, it is possible to calculate a more accurate evaluation result.

[0068] Furthermore, in step S510, upon receiving detection information indicating that the optical system and the image sensor have deviated from their designed positions, a message is output on the display unit of the digital camera 101 urging the user to have the camera and lens repaired at a customer center. However, other measures can also be taken. Specifically, in step S508, an evaluation value 805 indicating the degree of deviation is also sent to the digital camera 101. Then, according to the evaluation value 805 indicating the degree of deviation, a simple calibration can be performed by driving the IS mechanism installed in the optical system and the image sensor so that the optical system and the image sensor approach a state where there is no deviation from their designed positions. Alternatively, by referring to the evaluation value 805 indicating the degree of deviation, image processing (sharpness or blurring) can be performed on the area where the focal plane is tilted, so that the image approaches an image obtained when there is no deviation from the designed positions of the optical system and the image sensor. Specifically, blurring is performed on an area where the defocus amount is closer to the in-focus state compared to when there is no deviation. Conversely, sharpness processing is applied to areas that are closer to the background or foreground than the original defocus amount.By configuring as described above, even if the user is unable to send the camera and lens for repair, it is possible to obtain an image with the depth of field that the user expects, thereby improving user convenience.

[0069] Furthermore, the information on the detection of deviations of the optical system and image sensor from their designed positions can be sent not only to the user's camera but also to a customer center. By having the customer information registered by the user and information on all the equipment owned managed by the customer center, it becomes possible to record the number of deviations of the optical system and image sensor from their designed positions and to record maintenance, thereby further improving user convenience, such as by shortening the time required for repairs.

[0070] Furthermore, if the user uses a single camera and lens for photography, the display unit 201 may be configured to prompt the user to decide whether to execute an operation mode to determine whether the cause of the problem lies in the optical system or the image sensor. If the user selects the operation mode to determine the cause, instructions are given on the display unit 201 to prompt the user to take images suitable for determining the cause. Specific instructions include having the user attach graph paper to a wall facing the camera and take images while changing the optical system's photography conditions (focal length, focus lens position, aperture). If the analysis results change when the optical system's photography conditions are changed, the lens is the cause; if deviation is detected regardless of the photography conditions, the image sensor is the cause. This configuration makes it possible to identify the cause of deviations between the optical system and the image sensor from their designed positions, and to provide more accurate notification and repairs.

[0071] [Second embodiment] An image processing device, an image processing method, and an image processing program according to a second embodiment of the present invention will be described in detail below with reference to the drawings. Note that components similar to those of the image processing device according to the first embodiment are designated by the same reference numerals, and descriptions thereof will be omitted or simplified.

[0072] In the first embodiment, a configuration has been described in which information relating to the degree of deviation from the design positions of the optical system and the image sensor, which are internal parameters of the camera device 100, is calculated and notified to the user. In the second embodiment of the present invention, a configuration in which the position or orientation, which is an external parameter of the image processing device, is calibrated, will be described.

[0073] First, an imaging system according to this embodiment will be described. The imaging system according to this embodiment is for capturing images of the inspection target surface of a structure that is the subject of a social infrastructure inspection, and in particular, is for easily achieving frontal imaging of the inspection target surface or evaluation of the captured images. The imaging system according to this embodiment includes a camera device as an imaging device that captures moving images or captures still images periodically / irregularly, a lens device attached to the camera device, and a pan head device that rotates the camera device.

[0074] First, an example of the hardware configuration of a camera device 1900 and a lens device 1913 according to this embodiment will be described using the block diagram in Fig. 19A. However, in this embodiment as well, the imaging device may be configured with the same digital camera 101 as in the first embodiment. Fig. 19A shows a state in which a lens device 1913 is attached to the camera device 1900.

[0075] First, an example of the hardware configuration of the camera device 1900 will be described. The camera device 1900 according to this embodiment acquires distance information distributions at multiple positions within the imaging range of the camera device 1900, acquires information instructing the rotation or translation of the camera device 1900 based on the difference between the acquired distance information, and outputs the acquired information. Here, the distance information and distance information distribution may be in the form of an image shift amount and image shift amount distribution based on a pair of parallax images, a defocus amount and defocus map acquired by any means, or subject distance information and subject distance map, as in the first embodiment.

[0076] A CPU (Central Processing Unit) 1901 executes various processes using computer programs and data stored in a ROM (Read-Only Memory) 102 and a RAM (Random Access Memory) 1903. As a result, the CPU 1901 controls the overall operation of the camera device 1900, and also executes or controls each process to be performed by the camera device 1900, which will be described later.

[0077] The ROM 1902 stores setting data for the camera device 1900, computer programs and data related to the startup of the camera device 1900, computer programs and data related to the basic operation of the camera device 1900, and the like.

[0078] The RAM 1903 has an area for storing computer programs and data read from the ROM 1902, and computer programs and data read from a memory card 1909 via a recording medium I / F 1908. The RAM 1903 also has an area for storing captured images output from the image sensor 1904, computer programs and data received from an external device via an external I / F 1910, and data received from a lens device 1913 via the camera communication unit 107. The RAM 1903 also has a work area used by the CPU 1901 when executing various processes. In this way, the RAM 1903 can provide various areas as needed.

[0079] The pixel arrangement of the image sensor 1904 is the same as that of the image capturing unit 205 in Fig. 2, and generates and outputs a captured image according to light incident through the lens device 1913. The display unit 1905 is a device such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) that displays images and text on a display screen or finder screen. Note that the display unit 1905 does not have to be included in the camera device 1900, and may be, for example, an external device capable of wireless and / or radio communication with the camera device 1900.

[0080] The operation unit 1906 is a user interface such as a button, a dial, a touch panel, a joystick, etc., and allows the user to input various instructions to the CPU 1901 by operating it.

[0081] A camera communication unit 1907 performs data communication between the camera device 1900 and a lens device 1913. A recording medium I / F 1908 is an interface for attaching a memory card 1909 to the camera device 1900, and the CPU 1901 reads and writes data from and to the memory card 1909 via the recording medium I / F 1908.

[0082] Known examples of the memory card 1909 include card-type recording media such as SD, CF, CFexpress, XQD, and CFast. The memory card 109 may also be one that records data in an external device via a wireless network.

[0083] The external I / F 1910 is a communication interface for performing data communication with an external device, and the CPU 1901 performs data communication with the external device via the external I / F 1910. The power supply unit 1910 supplies power to and manages the power supply in the camera device 1900.

[0084] The CPU 1901 , ROM 1902 , RAM 1903 , image sensor 1904 , display unit 1905 , operation unit 1906 , camera communication unit 1907 , recording medium I / F 1908 , external I / F 1910 , and power supply unit 1911 are all connected to a system bus 1912 .

[0085] Next, we will explain an example of the hardware configuration of the lens device 1913. The CPU 1914 executes various processes using computer programs and data stored in the ROM 1915 and RAM 1916. As a result, the CPU 1914 controls the overall operation of the lens device 1913, and also executes or controls each process to be performed by the lens device 1913, which will be described later.

[0086] The ROM 1915 stores setting data for the lens device 1913, computer programs and data related to the startup of the lens device 1913, computer programs and data related to the basic operation of the lens device 1913, and the like.

[0087] The RAM 1916 has an area for storing computer programs and data read from the ROM 1915, and data received from the camera device 1900 via the lens communication unit 1919. The RAM 1916 also has a work area used when the CPU 1914 executes various processes. In this way, the RAM 1916 can provide various areas as needed.

[0088] The lens communication unit 1919 performs data communication between the camera device 1900 and the lens device 1913. For example, the lens communication unit 1919 receives control information from the camera device 1900 to the lens device 1913, communicates the operating state of the lens device 1913 to the camera device 1900, and receives power supply from the camera device 1900.

[0089] The display unit 1917 is a liquid crystal display (LCD), an organic light emitting display (OLED), or the like, and is a device that displays the operating state of the lens device 1913. Note that the display unit 1917 does not have to be included in the lens device 1913, and may be, for example, an external device that can communicate with the lens device 1913 wirelessly and / or wirelessly.

[0090] The operation unit 1918 is a user interface such as a button, a dial, a touch panel, or a joystick, and can be operated by a user to input various instructions to the CPU 114. In addition, instructions input by the user operating the operation unit 1918 can also be transmitted to the camera device 1900 by a lens communication unit 1919.

[0091] The lens driving unit 1920 controls the optical lens of the lens device 1913 based on instructions from the CPU 1901 or CPU 114, thereby controlling the aperture, focus, zoom focus, image stabilization, etc. After the aperture, focus, zoom focus, image stabilization, etc. have been controlled by the lens driving unit 1920, the light that enters through the optical lens is received by the image sensor 1904, and the image sensor 1904 generates and outputs a captured image according to the received light.

[0092] The CPU 1914 , ROM 1915 , RAM 1916 , lens communication unit 1919 , display unit 1917 , operation unit 1918 , and lens driving unit 1920 are all connected to a system bus 1921 .

[0093] Next, an example of the hardware configuration of the camera platform device 2000 according to this embodiment will be described with reference to the block diagram of FIG.

[0094] The CPU 2001 executes various processes using computer programs and data stored in the ROM 2002 and RAM 2003. As a result, the CPU 2001 controls the overall operation of the pan head device 2000, and also executes or controls each process to be performed by the pan head device 2000, which will be described later.

[0095] The ROM 2002 stores setting data for the pan head device 2000, computer programs and data related to the startup of the pan head device 2000, computer programs and data related to the basic operation of the pan head device 2000, and the like.

[0096] The RAM 2003 has an area for storing computer programs and data read from the ROM 2002. The RAM 2003 also has a work area used when the CPU 2001 executes various processes. In this way, the RAM 2003 can provide various areas as needed.

[0097] The external I / F 2004 is a communication interface for receiving various instructions from the remote control device 2010 via wireless or wired communication. The remote control device 2010 is a device for inputting various instructions to the camera platform device 2000, and can input, for example, instructions for changing the pan angle or tilt angle of the camera device 1900 placed on the camera platform device 2000. The external I / F 2004 can also communicate with the camera device 1900 placed on the camera platform device 2000.

[0098] The power supply unit 2005 supplies and manages power to the pan / tilt head device 2000. The display unit 206 is a device such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) that displays the operating status of the pan / tilt head device 2000. Note that the display unit 2006 does not have to be provided in the pan / tilt head device 2000, and may be, for example, an external device capable of wireless and / or radio communication with the pan / tilt head device 2000.

[0099] The operation unit 2007 is a user interface such as a button, a dial, a touch panel, a joystick, etc., and allows the user to input various instructions to the CPU 2001 by operating it.

[0100] The driving unit 2008 includes a base (fixing member) to which the camera device 1900 is fixed, and a driving mechanism that drives the base to pan, tilt, and translate in the X, Y, and Z directions. The driving unit 2008 controls the driving mechanism to control the pan angle, tilt angle, and position in the X, Y, and Z directions of the camera device 1900 based on instructions received from the remote control device 210 via the external I / F 2004. In this embodiment, the camera device 1900 is mounted on the above-described camera platform device 2000 to control the pan, tilt, and shooting position of the camera device 1900. However, the present invention is not limited to this, and can also be applied to a device, such as a drone, in which at least one of the pan, tilt, and shooting position of the camera device 1900 is controlled by the movement of the device itself.

[0101] The CPU 2001 , ROM 2002 , RAM 2003 , external I / F 2004 , power supply unit 2005 , display unit 2006 , operation unit 2007 , and drive unit 2008 are all connected to a system bus 2009 .

[0102] Next, an example of the functional configuration of the camera device 1900 will be described using the block diagram in Fig. 19B. In the following, each functional unit shown in Fig. 19B will be described as the subject of processing, but in reality, the operation of each functional unit is realized by the CPU 1901 executing a computer program corresponding to that functional unit. Also, at least some of the functional units shown in Fig. 19B may be implemented in hardware.

[0103] The object recognition unit 1928 uses a known general object detection technique to recognize whether the captured image captures the surface of a structure targeted for social infrastructure inspection. Specifically, the object recognition unit 1928 stores feature values ​​related to the structure targeted for infrastructure inspection in advance and compares the captured image with the feature values ​​of the stored image. This result is also used as information for estimating the depth direction in the image. If the recognition result indicates that the surface of a structure targeted for social infrastructure inspection is being captured, a calibration process is performed to position the camera device 1900 directly opposite the surface of the structure. If the surface to be inspected is flat, the camera and the structure are directly opposite each other, so the amount of defocus within the imaging range should be approximately uniform. As with the method for determining the degree of deviation of the optical system and image sensor from their designed positions described in the first embodiment, the calibration goal is to correct the position and orientation of the camera device 1900 so that each value of the distance information distribution within the imaging range is uniform (within a predetermined range). However, in the following embodiment, as an example, a method of simply setting control of the pan or tilt direction from the defocus amount of a partial area within the shooting range will be shown.

[0104] The determination unit 1922 acquires setting information indicating "the rotational direction and translational movement direction in which the camera device 1900 is operated so that the camera device 1900 faces the inspection target surface of a structure that is the subject of social infrastructure inspection." The setting information is determined, for example, by the user operating the operation unit 1906. If the setting information indicates that the camera is to be driven in the rotational direction and horizontal direction (pan direction), the determination unit 1922 sets two areas aligned left and right within the imaging range of the camera device 1900 as "areas for obtaining the defocus amount" (for example, positions near the left end and positions near the right end within the imaging range). Here, the minimum unit of each area is one pixel.

[0105] On the other hand, if the camera drive indicated by the setting information is in the rotational and vertical directions, the determination unit 1922 sets two areas aligned vertically within the imaging range of the camera device 1900 as "areas for obtaining the defocus amount" (for example, positions near the top and bottom of the imaging range). Here, too, the minimum unit of each area is one pixel.

[0106] Furthermore, if the setting information indicates that the camera is to be driven in a translational manner, the determination unit 1922 sets four areas aligned vertically and horizontally within the imaging range of the camera device 1900 as "areas for obtaining the defocus amount." Here, too, the minimum unit of each area is one pixel.

[0107] Furthermore, this embodiment does not use (or is not dependent on) user setting information, but rather acquires distance information in multiple regions, i.e., acquires a distance information distribution, as in the first embodiment, and controls the driving of the camera platform device 2000 (position and orientation of the camera) based on the analysis results of the distribution information. In this case, the "region for acquiring the defocus amount" is, for example, the entire region in which the defocus amount can be acquired. By acquiring the distance information distribution, the two-dimensional or three-dimensional tilt of the distance information can be determined, for example, by plane detection, and the position and orientation of the camera can be controlled so that the tilt approaches zero in each direction when facing forward.

[0108] The control unit 1924 acquires the defocus amount from the "area for acquiring the defocus amount" determined by the determination unit 1922 within the imaging range of the camera device 1900. The acquisition unit 1923 acquires the defocus amount acquired by the control unit 1924. The difference calculation unit 1925 calculates the difference between one defocus amount acquired by the acquisition unit 1923 and the other defocus amount.

[0109] The specifying unit 1926 specifies notification information for notifying "the degree of rotation and degree of translation (including the direction) by which the camera device 1900 is driven" based on the difference calculated by the difference calculation unit 1925. The output unit 1927 outputs the notification information specified by the specifying unit 1926 to the camera platform device 2000 via the external I / F 1910. The camera platform device 2000 acquires the notification information via the external I / F 2004, and controls the camera device 1900 by the driver 2008 based on the notification information so that the camera device 1900 is positioned and oriented as desired.

[0110] In this embodiment, such an imaging system is used to capture an image of a social infrastructure structure to be inspected, and the social infrastructure structure is inspected based on the captured image. An imaging method for capturing an image of a social infrastructure structure using the imaging system according to this embodiment will be described with reference to FIG.

[0111] FIG. 21(a) shows an example of an inspection target surface of a social infrastructure structure to be inspected. The social infrastructure structure 2100 shown in FIG. 21(a) is a horizontally long wall-like structure having a side surface 2101. Reference numeral 2102 indicates a joint portion that was generated when the social infrastructure structure 2100 was divided based on blueprints and constructed by pouring joints. The portion 2102 is also called a pouring joint, but for ease of understanding, it is referred to as a joint here. Because the joint portion 2102 can be visually inspected, it is also used as a unit of inspection work. Reference numeral 2103 indicates the area to be inspected in one inspection (inspection target area), and the imaging system captures an imaging area 2104 that includes the inspection target area 2103. In the captured image of the imaging area 2104, a "partial image corresponding to the periphery of the inspection target area 2103 within the imaging area 2104" provides information for understanding the positional relationship with adjacent inspection target areas. Therefore, this partial image is also used for alignment when combining into a single image that includes the entire social infrastructure structure 2100. In addition, this partial image corresponding to the peripheral area is also used for inspecting a wide range of abnormalities that are not limited to a single inspection target area.

[0112] Fig. 21(b) shows how such an imaging area 2104 is imaged using the imaging system according to this embodiment. In Fig. 21(b), a camera device 1900 is attached to a pan head device 2000 having a tripod 2108, and a lens device 1913 is attached to the camera device 1900. The width (the horizontal size in the figure) of an imaging range 2109 on the inspection surface imaged by the combination of the camera device 1900 and the lens device 1913 corresponds to the width (the horizontal size in the figure) of the imaging area 2104.

[0113] Then, when imaging of the inspection target area 2103 is completed, imaging is performed on an unimaged inspection target area adjacent to the inspection target area 2103. The imaging system according to this embodiment is moved to the position indicated by reference number 2110, and imaging of the inspection target area within the imaging range 2112 is performed in a similar manner. When imaging at the position indicated by reference number 2110 is completed, the imaging system according to this embodiment is moved to the position indicated by reference number 2111 to image an unimaged inspection target area adjacent to the inspection target area, and imaging of the inspection target area within the imaging range 2113 is performed in a similar manner. Furthermore, when the camera device 1900 is mounted on a mobile object such as a drone, the user manually or automatically moves the mobile object to each imaging position and sequentially images the images.

[0114] In this embodiment, the camera device 1900 needs to be directly facing the inspection target area. In this embodiment, it is determined whether the camera device 1900 is directly facing the inspection target area, and if it is not directly facing the inspection target area, a notification is issued to rotate or translate the camera device 1900 so that it faces the inspection target area.

[0115] To perform this notification, the control unit 1924 acquires the defocus amount at the position determined by the determination unit 1922, as described above. The method for acquiring the defocus amount is the same as step S502 in the first embodiment, and therefore description thereof will be omitted. Here, the acquired defocus amount has a continuous value, and the defocus amount corresponding to the degree of focus can be set to "-11" for a front-focused state, "0" for an in-focus state, and "+7" for a back-focused state. Alternatively, as in the first embodiment, data representing a spatial (two-dimensional) distribution of defocus amounts in the shooting range may be created, and the control unit 1924 may acquire the defocus amount at the position determined by the determination unit 1922 in the defocus amount distribution (distance information distribution).

[0116] Next, the operation of the imaging system according to this embodiment will be described with reference to the flowchart in FIG. 22 . As described above, a user installs the imaging system according to this embodiment, pointing the imaging system at the surface to be inspected in order to capture an image of the surface to be inspected. At this time, the user can install the camera device 1900 in an orientation that is considered to be roughly facing the area to be inspected. However, without precise measurement information on the reference points of the structure and installation location and the surrounding area, it is not possible to install the camera device 1900 in an orientation that accurately faces the area. After installing the camera device 1900, when the camera device 1900 is turned on, the image captured by the image sensor 1904 is displayed as a live view image on the display screen on the back of the camera device 1900 by the display unit 1905. Then, processing according to the flowchart in FIG. 22 begins.

[0117] In step S2200, the object recognition unit 1928 performs a general object detection process on the captured image. In this embodiment, the general objects to be detected include the inspection target surface of the structure to be imaged, and therefore, information on the feature amount indicating the inspection target surface is stored in advance in the ROM 1902.

[0118] In step S2216, the object recognition unit 1928 determines whether the object detected in step S2200 is the inspection target surface of a structure that should be imaged in a directly facing relationship with the camera device 1900. If it is determined that it is the inspection target surface, the process proceeds to step S2201 and continues. On the other hand, if it is determined that it is not the inspection target surface, the process according to the flowchart in FIG. 22 ends.

[0119] In step S2201, the determination unit 1922 acquires setting information indicating "driving of the camera device 1900 so that the camera device 1900 faces the surface to be inspected."

[0120] For example, as shown in FIG. 23, the operation unit 1906 controls "driving the camera device 1900 so that the camera device 1900 faces the surface to be inspected" (the orientation detection direction). That is, the operation unit 2007 has a switch for setting at least one of "vertical direction," "horizontal direction," and "translational direction." By operating this switch, the user can set the orientation detection direction to either the vertical direction (rotation axis = tilt axis) or the horizontal direction (rotation axis = pan axis). The determination unit 1922 acquires the orientation detection direction set by the switch as setting information. As shown in FIGS. 21(a) and 21(b), when an image of a horizontally elongated structure is captured while moving it horizontally, the orientation detection direction in the horizontal (rotation) direction is selected.

[0121] Furthermore, as described above, when control is performed including translational movement in the XYZ directions so as to obtain an image in which orientation is maintained and in focus in multiple regions (for example, in mode setting), the orientation detection direction is not set in step S2201. The control unit 1924 estimates the position and orientation of the plane of the subject to be focused on, based on the distance information distribution acquired corresponding to multiple regions of the captured image as in the first embodiment, and controls the position and orientation of the pan head device 2000 (camera device 1900).

[0122] In the following, as an example of setting the facing detection direction to either the vertical direction (rotation axis = tilt axis) or the horizontal direction (rotation axis = pan axis), a case will be described in which the facing detection direction is set to the horizontal direction.

[0123] Next, in step S2202, since the facing detection direction is the horizontal direction, the determination unit 1922 sets two regions aligned horizontally within the imaging range of the camera device 1900 as "regions for obtaining a defocus amount." For example, as shown in FIG. 24(a), the determination unit 1922 sets a region 2400 near the left end and a region 2401 near the right end of an imaging region 2104 that falls within an imaging range 2402 of the camera device 1900 in a social infrastructure structure 2100 as "regions for obtaining a defocus amount." Furthermore, the present embodiment is not limited to this, and even when the facing detection direction is set, the defocus amount for the entire screen (entire image) may be obtained as in the first embodiment.

[0124] In step S2203, the control unit 1924 acquires the defocus amount at the position set in step S2202 (area 2400 and area 2401 in the case of FIG. 24(a)) as described above. At this time, the camera device 1900 does not need to perform a focusing operation on the inspection target surface, and acquires the defocus amount at the area set in step S2202.

[0125] In step S2204, the acquisition unit 1923 acquires the "defocus amount in the left region" and the "defocus amount in the right region" acquired in step S2203. Then, the difference calculation unit 1925 calculates the difference by subtracting the "defocus amount in the right region" from the "defocus amount in the left region."

[0126] In step S2206, the specifying unit 1926 acquires, as rotation instruction information (notification information), "information indicating the rotation direction and degree of the camera device 1900" corresponding to the difference between the defocus amounts calculated in step S2204.

[0127] 25, a table 2515 is registered in the ROM 1902, in which rotation instruction information corresponding to differences between defocus amounts is registered. A range of the differences between defocus amounts is registered in a column 2516. For example, in a row 2519 of the column 2516, a range of the differences between defocus amounts "+11 or more" is registered, and in a row 2524 of the column 2516, a range of the differences between defocus amounts "-5 to -10" is registered.

[0128] Icons corresponding to the rotation amount when rotating the camera device 1900 to the left are registered in column 2517. The icon registered in row 2519 in column 2517 represents a larger rotation amount than the icon registered in row 2520 in column 2517. The icon registered in row 2520 in column 2517 represents a larger rotation amount than the icon registered in row 2521 in column 2517. The icons registered in rows 2522 to 2525 in column 2517 represent that there is no need to rotate to the left.

[0129] Icons corresponding to the amount of rotation when rotating the camera device 1900 to the right are registered in column 2518. The icon registered in row 2525 in column 2518 represents a larger amount of rotation than the amount of rotation represented by the icon registered in row 2524 in column 2518. The icon registered in row 2524 in column 2518 represents a larger amount of rotation than the amount of rotation represented by the icon registered in row 2523 in column 2518. The icons registered in rows 2519 to 2522 in column 2518 represent that there is no need to rotate to the right.

[0130] Therefore, for example, if the difference between the defocus amounts calculated in step S2204 is "+7", the identification unit 1926 acquires, as rotation instruction information, two icons registered in row 2520 corresponding to the interval "+10 to +5" that includes the difference "+7".

[0131] For example, if the difference between the defocus amounts calculated in step S2204 is "-12", the determination unit 1926 acquires, as rotation instruction information, two icons registered in row 2525 corresponding to the interval "-11 or less" that includes the difference "-12".

[0132] In other words, the table in Figure 25 registers rotation instruction information for notifying the rotation direction according to the sign of the difference between the defocus amounts and the degree of rotation according to the absolute value of the difference between the defocus amounts.

[0133] In step S2214, the output unit 1927 outputs the rotation instruction information acquired in step S2206 to the display unit 1905 as "notification information for notifying the user of the rotation direction and degree of rotation of the camera device 1900." The display unit 1905 displays the notification information on a display screen on the back of the camera device 1900. For example, as shown in FIG. 24( a), an icon 2405 acquired from column 2517 is displayed on the lower left side of a live view image 2404 displayed on the display screen on the back of the camera device 1900. Furthermore, an icon 2406 acquired from column 2518 is displayed on the lower right side of the live view image 2404. Note that the display positions of the icons 2405 and 2406 are not limited to any particular display positions, and they may be displayed by being superimposed on the live view image 2404, for example. In addition, in FIG. 24(a), icons 2400a and 2401a are displayed superimposed on a live view image 2404 at positions corresponding to positions 2400 and 2401, respectively.

[0134] The user visually checks the displayed icons 2405 and 2406, recognizes the notification to rotate the camera device 1900 left, and rotates the camera device 1900 left. The state of the camera device 1900 after rotating it left in the state of Fig. 24(a) is shown in Fig. 24(b).

[0135] In the state of FIG. 24(b), icons 2409 and 2410 are still displayed, so the user similarly recognizes the notification to rotate the camera device 1900 left and rotates the camera device 1900 left. Here, it is indicated that neither icon 2406 nor icon 2410 needs to be rotated right. On the other hand, it is indicated that both icon 2405 and icon 2409 need to be rotated left, but icon 2409 is rotated less than icon 2405. FIG. 24(c) shows the state after the camera device 1900 in the state of FIG. 24(b) has been further rotated left.

[0136] 24(c), an icon 2413 indicating that there is no need to rotate left and an icon 2414 indicating that there is no need to rotate right are displayed. When the user visually views the displayed icons 2413 and 2414, he or she recognizes the notification that there is no need to rotate the camera device 1900 left or right, and does not rotate the camera device 1900.

[0137] 23 is a diagram showing a state in which the camera device 1900 is mounted on a camera platform device 2000, and a remote control device 2010 is connected to the camera platform device 2000 for performing pan / tilt operations and image capturing operations of the camera device 1900. In this case, the remote control device 2010 is connected to the camera device 1900 via an external I / F 1910 of the camera device 1900, thereby enabling the camera device 1900 to capture images.

[0138] Returning to Fig. 22, in step S2215, the CPU 1901 determines whether or not the termination condition for the processing according to the flowchart in Fig. 22 has been satisfied. For example, if the user operates the operation unit 1906 to input an instruction to terminate the processing or turns off the power of the camera device 1900, the CPU 1901 determines that the termination condition for the processing according to the flowchart in Fig. 22 has been satisfied.

[0139] As a result of such a determination, if the termination conditions for the processing according to the flowchart in Figure 22 are met, the processing according to the flowchart in Figure 22 ends, and if the termination conditions are not met, the processing proceeds to step S2203.

[0140] In this way, by installing the camera device 1900 mounted on the pan head device 2000 as shown in FIG. 23 facing the surface to be inspected, it is possible to notify the user of rotation and translation instruction information for orienting the camera device 1900 relative to the surface to be inspected. The user then operates the pan head device 2000 and other devices in accordance with the notification, thereby enabling accurate orientation of the camera device 1900 relative to the surface to be inspected, thereby enabling accurate inspection of the surface for abnormalities. At the same time, by accurately orienting the camera device 1900, it is possible to continuously capture images of the surface to be inspected under uniform conditions by translating the camera device 1900 when capturing images of adjacent areas of the surface to be inspected. Furthermore, even if the image sensor 1904 or lens device 1913 of the camera device 1900 deviates from its design position due to aging, accurate orientation relative to the surface to be inspected enables accurate inspection of the surface for abnormalities.

[0141] In this embodiment, the rotation direction for orientation is the horizontal (rotation) direction, and the pan axis of the camera platform device 2000 is operated, but the orientation detection direction may be switched to issue an orientation rotation instruction for the vertical (rotation) direction, and the tilt axis may be operated. Furthermore, the horizontal (rotation) direction and vertical (rotation) direction may be detected simultaneously, and rotation instruction information for each may be presented.

[0142] Furthermore, in this embodiment, an example of the defocus amount value is presented, and three types of rotation instruction information are defined. However, since the defocus amount value differs depending on the type of image plane phase difference sensor used, it may be multiplied by an appropriate coefficient or the like, and the types are not limited to these.

[0143] In addition, in this embodiment, icons indicating both the direction of rotation and the degree of rotation are displayed, but an icon indicating the direction of rotation and an icon indicating the degree of rotation may be displayed separately, or only one of them may be displayed. Furthermore, the information indicating the direction of rotation or the degree of rotation is not limited to icons, and may be, for example, text information. Furthermore, the method of notifying the direction of rotation or the degree of rotation is not limited to a specific notification method.

[0144] In this embodiment, icons are displayed for directions that do not require rotation, but icons do not need to be displayed for directions that do not require rotation. Furthermore, for directions that require rotation, other information such as text information may be displayed in addition to icons.

[0145] Furthermore, in this embodiment, the camera device 1900 is configured to be mounted on the camera platform device 200, but as described above, the camera device 1900 may be configured to be mounted on a UAV (unmanned aerial vehicle) such as a drone. By configuring it in this way, it is possible to capture a front-on image of the inspection target surface of a target structure in an environment where a camera platform cannot be installed.

[0146] Furthermore, in this embodiment, rotation and / or translation instruction information is notified to the user, but the rotation and / or translation instruction information may be output to the camera platform device 2000. The camera platform device 2000 may be configured to control the rotation of the camera device 1900 in accordance with the rotation and / or translation instruction information, and automatically orient the camera device 1900 directly toward the surface to be inspected. This configuration reduces the amount of work required for the user to operate the camera device, improving convenience.

[0147] Furthermore, in this embodiment, the defocus amount (distance information distribution) is calculated by the camera device 1900, but as in the first embodiment, the defocus amount may be calculated by a computer connected so as to be able to communicate via a communication circuit.

[0148] Furthermore, in this embodiment, the distance information distribution is calculated in order to control the position and orientation of the camera device 1900 by operating the pan head device 2000, but the use of the calculated distance information distribution is not limited to this.

[0149] For example, the CPU 1901 records a pair of image data with parallax captured by the image sensor 1904 and the shooting conditions, including at least the F-number and KX value, in association with the image data on a memory card 1909 or the like. Based on the recorded pair of image data and the shooting conditions, the CPU 1901 or a CPU in an external device to which each data is output generates and acquires a distance information distribution. Here, the acquired distance information distribution is treated as a defocus amount distribution, and a blur map is generated by converting each defocus amount based on the shooting conditions, such as the F-number (or effective F-number) and the conversion coefficient KX. The blur map may be used to evaluate the quality of the blur in the captured image. In particular, when photographing for social infrastructure inspections, when inspecting the surface to be inspected for defects, crack detection and crack width measurement cannot be performed correctly unless the surface to be inspected is not out of focus. Therefore, by referring to the defocus amount distribution (or blur map), for example, by limiting measurements to an area (imaging range) where no blur occurs, more accurate inspections can be performed. Furthermore, if the CPU 1901 determines that a predetermined percentage or more of a captured image contains blur with a blur amount above a standard, the CPU 1901 may notify the user that the captured image is NG (deformation detection is not possible). Possible notification methods include displaying an image or icon on the display unit 1905, or using light, sound, vibration, or the like from other devices. The CPU 1901 may also generate the blur map described above and simply generate an image visualizing each blur amount and display it on the display unit. The user can manually or automatically retake the image or move the camera device 1900 while referring to the blur map.

[0150] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0151] [Other embodiments] The object of the present invention can also be achieved as follows: A storage medium storing software program code describing procedures for realizing the functions of each of the above-described embodiments is supplied to a system or device, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code stored in the storage medium.

[0152] In this case, the program code itself read from the storage medium will realize the novel functions of the present invention, and the storage medium storing the program code and the program will constitute the present invention.

[0153] Furthermore, examples of storage media for supplying the program code include flexible disks, hard disks, optical disks, magneto-optical disks, etc. Also usable are CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0154] The functions of the above-described embodiments are realized by making the computer executable the read program code. Furthermore, the functions of the above-described embodiments may be realized by an operating system (OS) or the like running on the computer performing some or all of the actual processing based on the instructions of the program code.

[0155] The following case is also included: First, program code is read from a storage medium and written into memory on an expansion board inserted into a computer or on an expansion unit connected to the computer. Then, based on the instructions of the program code, a CPU or other device on the expansion board or unit performs some or all of the actual processing. [Explanation of symbols]

[0156] 100 Image processing device 101 Digital Camera 102 Computer 103 Communication Circuit 1900 Camera Equipment 1913 Lens equipment 2000 Panhead device 2010 Remote Control Device

Claims

1. an acquisition means for acquiring a distance information distribution calculated from an image captured using an optical system that forms an image of the object scene on an image sensor of the image capture means; an estimation means for estimating a depth direction in the image from the imaging conditions of the imaging means; determining means for determining an evaluation value indicating a degree of deviation in the depth direction of the subject in the image from the relationship between the distance information distribution and the estimated depth direction; The image processing apparatus is characterized in that the distance information distribution is information relating to a distribution of the defocus amount of the subject normalized by the F-number and the allowable circle of confusion diameter.

2. 2. The image processing apparatus according to claim 1, wherein the distance information distribution is obtained from a pair of images having a parallax therebetween.

3. The image processing device according to claim 1 or 2, characterized in that the shooting conditions are at least one of the following: posture information of the device when the image was shot; a vanishing point in the image; a change in texture density in the image; or a determination result as to whether or not a structure with a known shape is captured in the image.

4. The relationship in the depth direction is 4. The image processing device according to claim 1, wherein the angle is formed by a line where the defocus amount in the distance information distribution is zero and a line indicating the in-focus area calculated by the depth direction estimation means.

5. The relationship in the depth direction is 2. The image processing device according to claim 1, characterized in that the difference is between a vector of the gradient of the defocus amount in the distance information distribution and a vector of the direction toward the vanishing point in the image calculated by the depth direction estimation means, or a vector of the direction of change in texture density in the image.

6. 3. The image processing apparatus according to claim 1, wherein the degree of deviation of the subject in the image in the depth direction is a difference between defocus amounts at a plurality of positions in the distance information distribution.

7. 7. The image processing apparatus according to claim 1, further comprising a notification unit that notifies the user of the evaluation value.

8. The image processing device according to any one of claims 1 to 7, characterized in that the determination means determines an evaluation value indicating the degree of deviation when it determines that the image acquired by the acquisition means includes general objects including the ground, the water surface, and structures laid vertically to the ground or the water surface.

9. 9. The image processing apparatus according to claim 8, wherein the statistics of the distance information distribution are a histogram of the distance information distribution.

10. The image processing device according to any one of claims 1 to 9, characterized in that the degree of deviation is corrected to be smaller by controlling an IS mechanism, performing image processing on the image, or rotating the image processing device, depending on an evaluation value indicating the degree of deviation.

11. The image processing device according to any one of claims 1 to 10, characterized in that an evaluation value indicating the degree of deviation is associated with information about the imaging element and the optical system that acquired the image for which the evaluation value was calculated, and the information is output to an external device.

12. an acquisition step of acquiring a distance information distribution calculated from an image captured using an optical system that forms an image of the object scene on an image sensor of an image capturing means; an estimation step of estimating a depth direction in the image from an imaging condition of the imaging means; a determining step of determining an evaluation value indicating a degree of deviation in the depth direction of the subject in the image from the relationship between the distance information distribution and the estimated depth direction, The image processing method is characterized in that the distance information distribution is information relating to a distribution of the defocus amount of the subject normalized by the F-number and the permissible circle of confusion diameter.

13. 12. A computer-executable program in which procedures for realizing the functions of each means for controlling the image processing apparatus according to claim 1 are described.

14. 12. A computer-readable storage medium storing a program for causing a computer to execute the functions of each unit of the image processing apparatus according to claim 1.

Images