Image processing apparatus, image processing method, and computer executable program product

Abstract

The invention relates to an image processing apparatus, an image processing method, and a computer-executable program product. An image processing apparatus according to the present invention comprises: an input means for inputting a distance information distribution calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging means; an estimation means for estimating a depth direction in the image in accordance with an imaging condition of the imaging means; and a determination means for determining an evaluation value indicating the degree of deviation of the optical system and the imaging element with respect to the design position on the basis of the relationship between the distance information distribution and the estimated depth direction.

Description

[0001] (This application is a divisional application of application No. 2020800548518, filed on July 28, 2020, entitled “Image Processing Device, Image Processing Method, Program, and Storage Medium.”) Technical Field

[0002] The present invention relates to an image processing apparatus, and in particular, to information related to secular changes in an optical system and an imaging element and information related to the posture of the image processing apparatus. Background Art

[0003] Conventionally, there are known technologies for diagnosing changes in the relative positional relationship between a pair of stereo cameras due to long-term changes, etc., by referencing distance information acquired from these cameras, and for supporting calibration of these stereo cameras. For example, Patent Document 1 discloses the following method. A stereo camera mounted on the head of a robot captures an image of a subject at a predetermined positional relationship on a substantially flat surface provided with a diagnostic texture. The obtained parallax images are used to calculate distance information to determine flatness. The obtained flatness is then compared with a predetermined reference value to determine whether calibration is necessary.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2004-306249 Summary of the Invention

[0007] Solution to the problem

[0008] Furthermore, in digital cameras not mounted on robots and used by typical users, the lens or CMOS image sensor, which form the optical system, may shift from its original attachment position (designed position) at the time of manufacture due to long-term changes, etc. If the lens or image sensor tilts, the relationship between actual distance and depth of field deviates, resulting in images that are not intended by the user. Therefore, a method for determining whether lens or image sensor calibration is necessary, and a solution therefor, are desired for digital cameras used by typical users as well.

[0009] Furthermore, when capturing images using a digital camera, even if the lens or imaging element is calibrated, a tilted imaging device or an inappropriate imaging distance prevents a good captured image from being obtained when the digital camera is facing a desired subject and capturing the image. In particular, tilt in the depth direction or distance error causes the captured image (the object subject) to become blurred.

[0010] Accordingly, an object of the present invention is to provide an image processing apparatus that enables notification of at least one of information related to tilt of a lens or an imaging element and information related to position or orientation of an imaging apparatus based on a distance information distribution corresponding to a distance to a subject.

[0011] To solve the above problems, an image processing apparatus according to the present invention includes: an input unit configured to input a distance information distribution calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging unit; an estimation unit configured to estimate a depth direction in the image based on an imaging condition of the imaging unit; and a determination unit configured to determine an evaluation value indicating a degree of deviation of the optical system and the imaging element from a design position based on a relationship between the distance information distribution and the estimated depth direction.

[0012] In addition, an image processing apparatus according to the present invention includes: an input unit configured to input a distance information distribution calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging unit; an estimation unit configured to estimate a depth direction in the image based on an imaging condition of the imaging unit; and a determination unit configured to determine an evaluation value indicating a degree of deviation in a depth direction of a subject in the image based on a relationship between the distance information distribution and the estimated depth direction.

[0013] In addition, an image processing apparatus according to the present invention includes: a first acquisition unit configured to acquire an imaging condition related to an image captured by an imaging unit, the imaging condition including at least an F-number and a conversion coefficient for converting an image offset amount into a defocus amount; a second acquisition unit configured to acquire a distance information distribution that is a distribution of distance information corresponding to each region of the image captured by the imaging unit; and an image processing unit configured to normalize the distance information distribution based on the F-number and the conversion coefficient.

[0014] In addition, an image processing method according to the present invention includes: an input step of inputting a distance information distribution calculated from an image captured by using an optical system for forming a field image on an imaging element of an imaging unit; an estimation step of estimating a depth direction in the image based on an imaging condition of the imaging unit; and a determination step of determining an evaluation value indicating a degree of deviation of the optical system and the imaging element from a design position based on a relationship between the distance information distribution and the estimated depth direction.

[0015] In addition, an image processing method according to the present invention includes: an input step for inputting a distance information distribution, which is calculated from an image captured by using an optical system for forming a field image on an image sensor of an imaging component; an estimation step for estimating a depth direction in the image according to the imaging conditions of the imaging component; and a determination step for determining an evaluation value indicating a deviation degree in the depth direction of a subject in the image according to the relationship between the distance information distribution and the estimated depth direction.

[0016] In addition, an image processing method according to the present invention includes: a first acquisition step for acquiring imaging conditions related to an image captured by an imaging component, where the imaging conditions at least include an F-number and a conversion coefficient for converting an image offset amount into a defocus amount; a second acquisition step for acquiring a distance information distribution, which is a distribution of distance information corresponding to each region of the image captured by the imaging component; and an image processing step for normalizing the distance information distribution based on the F-number and the conversion coefficient.

[0017] According to the present invention, an image processing device capable of confirming a deviation degree of an optical system and an image sensor relative to a design position can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram showing an example of a functional structure of an image processing device according to an embodiment of the present invention.

[0019] Figure 2 is a block diagram showing an example of a functional structure of a digital camera according to an embodiment of the present invention.

[0020] Figure 3 is a block diagram showing an example of a functional structure of a computer according to an embodiment of the present invention.

[0021] Figure 4A shows an example of a structure of an imaging unit according to an embodiment of the present invention.

[0022] Figure 4B shows an example of a structure of an imaging unit according to an embodiment of the present invention.

[0023] Figure 5A is a flowchart showing an operation of an image processing device according to an embodiment of the present invention.

[0024] Figure 5B is a flowchart showing an operation of an image processing device according to an embodiment of the present invention.

[0025] Figure 5C is a flowchart showing an operation of an image processing device according to an embodiment of the present invention.

[0026] Figure 6 Displays an image for recording a still image according to an embodiment of the present invention.

[0027] Figure 7 Displays a defocus map according to an embodiment of the present invention.

[0028] Figure 8 Is a block diagram showing an example of the functional structure of the image processing unit 306 according to an embodiment of the present invention.

[0029] Figure 9 Displays a plane where the defocus amount becomes zero according to an embodiment of the present invention.

[0030] Figure 10 Displays a defocus map when the focus plane is normal according to an embodiment of the present invention.

[0031] Figure 11A Displays a phenomenon that occurs when the optical system and the imaging element are offset from the design position according to an embodiment of the present invention.

[0032] Figure 11B Displays a phenomenon that occurs when the optical system and the imaging element are offset from the design position according to an embodiment of the present invention.

[0033] Figure 12 Displays a defocus map when the focus plane is tilted according to an embodiment of the present invention.

[0034] Figure 13 Displays an evaluation value indicating the degree of deviation according to an embodiment of the present invention.

[0035] Figure 14 Displays a notification to the user according to an embodiment of the present invention.

[0036] Figure 15 Displays the estimated results of the vanishing point and the depth direction according to an embodiment of the present invention.

[0037] Figure 16 Displays a histogram of the defocus map according to an embodiment of the present invention.

[0038] Figure 17A Displays an image for recording a still image, a defocus map, and a histogram of the defocus map in a portrait scene according to an embodiment of the present invention.

[0039] Figure 17B Displays an image for recording a still image, a defocus map, and a histogram of the defocus map in a portrait scene according to an embodiment of the present invention.

[0040] Figure 17CAn image for recording a still image in a portrait scene according to an embodiment of the present invention, a defocus map, and a histogram of the defocus map are shown.

[0041] Figure 18 Optical vignetting characteristics of the optical system according to the first embodiment of the present invention are shown.

[0042] Figure 19A 19 is a block diagram showing a hardware configuration example of a camera apparatus 1900 and a lens apparatus 1913 according to a second embodiment of the present invention.

[0043] Figure 19B : is a block diagram showing an example of the functional structure of a camera apparatus 1900 according to the second embodiment of the present invention.

[0044] Figure 20 2 is a block diagram showing an example of the hardware structure of a pan / tilt platform device 2000 according to the second embodiment of the present invention.

[0045] Figure 21A A second embodiment of the present invention will be described as a method for capturing images of social infrastructure.

[0046] Figure 21B A second embodiment of the present invention will be described as a method for capturing images of social infrastructure.

[0047] Figure 22 is a flowchart of the operation of the imaging system according to the second embodiment of the present invention.

[0048] Figure 23 A switch 2007 according to a second embodiment of the present invention is shown.

[0049] Figure 24A 19 is a diagram related to rotation control of the camera apparatus 1900 according to the second embodiment of the present invention.

[0050] Figure 24B 19 is a diagram related to rotation control of the camera apparatus 1900 according to the second embodiment of the present invention.

[0051] Figure 24C 19 is a diagram related to rotation control of the camera apparatus 1900 according to the second embodiment of the present invention.

[0052] Figure 25 A structural example of the table 2515 according to the second embodiment of the present invention is shown. DETAILED DESCRIPTION

[0053] [First embodiment]

[0054] Hereinafter, an image processing apparatus, an image processing method, and an image processing program according to a first embodiment of the present invention will be described in detail with reference to some drawings.Figure 1 As shown, an example of applying the present invention to an image processing apparatus 100 will be described, in which a digital camera 101 as an example of a imaging apparatus and a computer 102 as an example of an image processing apparatus are communicably connected to each other via a communication circuit 103. However, in the following description, the processing performed by the computer 102 may also be performed by the digital camera 101. In addition, the digital camera 101 may be any given electronic device having an imaging function, and the computer 102 may be any given electronic device capable of performing the processing described below or a computer in a server device. The computer 102 may also be a mobile computer or a desktop computer.

[0055] Figure 2 is a block diagram showing an example of the functional structure of the digital camera 101 according to an embodiment of the present invention. The system control unit 201 is, for example, a CPU, reads the operation programs of the blocks included in the digital camera 101 from the ROM 202, loads these operation programs into the RAM 203, and executes these operation programs to control the operations of the blocks included in the digital camera 101. The ROM 202 is a rewritable non-volatile memory, and in addition to storing the operation programs of the blocks included in the digital camera 101, also stores parameters required for the operations of the blocks, etc. The RAM 203 is a rewritable volatile memory, and is used as a temporary storage area for data output during the operations of the blocks included in the digital camera 101.

[0056] The optical system 204 forms a field image on the imaging unit 205. The imaging unit 205 is, for example, an imaging element such as a CCD or CMOS sensor, photoelectrically converts the optical image formed on the imaging element of the imaging unit 205 by the optical system 204, and outputs the obtained analog image signal to the A / D conversion unit 206. In addition, an IS mechanism for reducing the influence of camera shake is provided in each of the optical system 204 and the imaging unit 205. The A / D conversion unit 206 applies A / D conversion processing to the input analog image signal, and outputs the obtained digital image data to the RAM 203 for storage.

[0057] The image processing unit 207 applies various types of image processing such as white balance adjustment, color interpolation, reduction / enlargement, and filtering to the image data stored in the RAM 203.

[0058] The recording medium 208 is a removable memory card or the like, on which the images processed by the image processing unit 207 stored in the RAM 203 and the images A / D-converted by the A / D conversion unit 206 are recorded as recorded images.

[0059] The communication unit 209 transmits the image data file and the like recorded on the recording medium 208 to an external device in a wired or wireless manner.

[0060] The display unit 210 displays the image data obtained by imaging or the image data read from the recording medium 208, etc., or displays various menu screens. The display unit 210 also serves as an electronic viewfinder by displaying a live view image.

[0061] The operation unit 211 is a group of input devices for the user to input various instructions or settings, etc. to the digital camera 101, and includes keys and buttons such as a shutter button, a menu button, arrow keys, and a decision key, etc., which are typically provided on a digital camera. In addition, when the display unit 210 is a touch display, the display unit 210 also serves as the operation unit 211. Note that the operation unit 211 can be configured to omit physical operations such as a combination of a microphone and a voice command recognition unit.

[0062] The detection unit 212 includes a gyro sensor or sensors, and acquires angular velocity information or posture information, etc., of the digital camera 101. Note that the posture information includes information related to the tilt of the digital camera 101 relative to the horizontal direction, etc.

[0063] Figure 3 FIG. is a block diagram showing an example of the functional structure of the computer 102 according to the present embodiment. The system control unit 301 is, for example, a CPU, reads programs from the ROM 302, loads these programs into the RAM 303, and executes these programs to control the operations of the blocks included in the computer 102. The ROM 302 is a rewritable non-volatile memory, and stores, in addition to the programs executed by the system control unit 301, parameters required for controlling the blocks, etc. The RAM 303 is a rewritable volatile memory, and uses the respective blocks included in the computer 102 as a temporary storage area for the output data.

[0064] The communication unit 304 communicates with an external device such as the digital camera 101 through wired or wireless communication. The recording device 305 is, for example, a hard disk, and stores the image data, etc., received by the communication unit 304 from the digital camera 101.

[0065] The image processing unit 306, for example, calculates the defocus amount (to be described later) of the image data loaded from the recording device 305 into the RAM 303, estimates the depth direction based on the image, or calculates information related to the degree of deviation of the optical system and the imaging element from the designed positions.

[0066] The display unit 307 is used to display the GUI or various types of data provided by the OS or applications operating in the computer 102. The display unit 307 may be included in the computer 102 or may be connected as an external device.

[0067] The operation unit 308 is a group of input devices for the user to input various instructions or settings, etc. to the computer 102, and generally includes a keyboard, a mouse, a touchpad, etc. In addition, when the display unit 307 is a touch display, the display unit 307 also serves as the operation unit 308. Note that the operation unit 308 may be configured to omit physical operations such as a combination of a microphone and a voice command recognition unit.

[0068] Figure 4A Shows Figure 2 the arrangement structure of the pixels in the imaging unit 205. As Figure 4A shown, a plurality of pixels 400 are arranged two-dimensionally and regularly in the imaging unit 205. Specifically, the plurality of pixels 400 are arranged, for example, in the form of a two-dimensional lattice. Note that the arrangement structure of the pixels 400 is not limited to the lattice form arrangement structure, and other arrangement structures may also be adopted.

[0069] Figure 4B Shows in an enlarged manner Figure 4A the pixel 400 shown. As Figure 4B shown, each pixel 400 includes a microlens 401 and a pair of photoelectric conversion units 402A and 403B (hereinafter referred to as pupil-split pixels 402A and 403B, respectively). Both of the pupil-split pixels 402A and 403B have the same planar shape, and the planar shape has a rectangular shape with the long side direction being the y-axis direction. In each pixel 400, the pupil-split pixels 402A and 403B are arranged to be axially symmetric with the vertical bisector along the y-axis direction of the microlens 401 as the axis of symmetry. Note that the planar shape of the pupil-split pixels 402A and 403B is not limited to this, and other planar shapes may also be adopted. In addition, the arrangement manner of the pupil-split pixels 402A and 403B is not limited to this, and other arrangement manners may also be adopted.

[0070] In the present embodiment, an A image and a B image are respectively output as parallax images from the two-dimensionally and regularly arranged pupil-split pixels 402A and 403B. In addition, an A + B image obtained by adding the A image and the B image is recorded as a still image on the recording medium 208. By as Figure 4A and Figure 4BThe configuration shown forms an imaging unit 205 that images a pair of light beams passing through different regions of the pupil of the optical system 204 into a pair of optical images, and can output these images as an A image and a B image. Note that the method for obtaining the A image and the B image is not limited to the above method, and various methods can be adopted. For example, a parallax image obtained by an imaging device such as a plurality of cameras installed at a spatial interval can be used as the A image and the B image. In addition, a parallax image obtained by an imaging device such as a single camera including a plurality of optical systems and imaging units can also be used as the A image and the B image.

[0071] The operation of the image processing device 100 will be described below. In response to an imaging instruction such as a full press of a shutter button input through the operation unit 211 of the digital camera 101, the image processing device 100 executes Figure 5A , Figure 5B and Figure 5C the processing shown. Note that Figure 5A and Figure 5C the processing in is executed by the digital camera 101, and Figure 5B the processing in is executed by the computer 102.

[0072] First, in step S500, the system control unit 201 detects the state of the camera when the shutter button is pressed from the detection unit 212. Here, as the state of the camera, the tilt of the digital camera 101 with respect to the horizontal direction and the orientation in the up and down directions are detected.

[0073] In the subsequent step S501, the system control unit 201 performs imaging processing according to the exposure conditions determined in the imaging preparation state, and obtains an A image and a B image as a pair of parallax images from the imaging unit 205. Note that the A image and the B image pre-recorded on the recording medium 208 can also be read and obtained. In addition, the A image and the B image can be added and recorded on the recording medium 208 as an image for recording a still image. In Figure 6 is shown the image for recording a still image in this embodiment. Figure 6 is an image obtained by adding the captured A image and B image. In addition, 600 is an autofocus frame.

[0074] In a subsequent step S502, the system control unit 201 controls the image processing unit 207 and outputs data representing the spatial (two-dimensional) defocus amount distribution in the imaging range based on the disparity image acquired in step S501. In the following description, the data representing the spatial defocus amount distribution will be referred to as a defocus map. The defocus amount is the amount of focus shift from the distance at which the optical system 204 is focused, and thus is a type of distance information. Regarding the method for obtaining the defocus amount, for example, a method for calculating the phase difference between disparity images as disclosed in Japanese Patent Application Laid-Open No. 2008-15754 can be used. Specifically, the relationship between the offset amount of the disparity image and the defocus amount is expressed by the following expression.

[0075] DEF = KX · PY · x ··· (1)

[0076] In Expression (1), DEF is the defocus amount, PY is the detection pitch (the pitch for arranging pixels of the same type), KX is a conversion coefficient determined by the opening angle degree of the centroid of a pair of light beams passing through the pupil, and x is the offset amount of the disparity image.

[0077] In addition, the present invention is not limited thereto, and a distribution of the offset amount as the offset amount of the disparity image can also be acquired as the distance information distribution.

[0078] In addition, the distance information distribution can also be information expressed in units of a length (such as micrometers) obtained by multiplying the offset amount of the disparity image by the detection pitch PY.

[0079] In addition, the present invention is not limited thereto, and the distance information distribution can be converted from the defocus amount to the distribution of the actual distance by further referring to the position of the focusing lens.

[0080] In addition, the present invention is not limited thereto, and a distribution of a value obtained by normalizing the defocus amount with Fδ (F is the f-number and δ is the diameter of the circle of confusion) can also be acquired as the distance information distribution. This distribution represents the amount of blur with respect to δ. Here, although the f-number used for imaging can be applied as the f-number F to the entire distribution, in order to obtain a more accurate distribution of the amount of blur, it is preferable to apply the effective f-number (effective f-value) that takes into account the optical vignetting characteristics of the optical system 204 under the imaging conditions. Figure 18 is a graph showing the optical vignetting characteristic V(h), where the horizontal axis represents the distance (image height) from the optical center, and the vertical axis represents the amount of light normalized by setting the amount of light at the center of each image height to 1. Vignetting occurs according to the lens frame or aperture frame, and Figure 18In [the situation], as the image height increases (towards the end in the imaging range), the amount of light decreases. The optical vignetting characteristic has unique characteristics depending on the lens. Here, by referring to the optical vignetting characteristic, the effective f-number F' at the image height h is expressed as the following expression.

[0081]

[0082] In Figure 7 is shown Figure 6 the defocus map of the image in [it]. The defocus map 700 is represented by a grayscale of continuous values that become whiter (higher pixel values) as the distance gets closer. Additionally, 701 is the autofocus frame, and the in-focus area (where the defocus amount is zero) is shown in gray. Additionally, 702 is the straight line connecting the in-focus areas.

[0083] Subsequently, in step S503, the system control unit 201 sends the following information mainly including image data to the computer 102 through the communication unit 209.

[0084] - The image for recording a still image

[0085] - The defocus map

[0086] - Camera state detection information (information related to detecting the tilt of the camera)

[0087] - Autofocus frame position information

[0088] - The identification number (ID) of the camera body and the identification number (ID) of the attached lens

[0089] - Imaging information such as the f-number and ISO sensitivity

[0090] Record or transmit the above information in relation to each other. For example, the above information can be recorded in Exif information when using the JPEG format, or can be recorded as image attachment information in a single file when using the RAW data format. Optionally, the necessary information can be recorded or transmitted together with the image as a container file, where multiple associated data can be stored together in the container file. Optionally, these information can be recorded or transmitted as different files without being collected together. For example, processing such as setting the same file name, storing them in the same folder, or sequentially transmitting these data in order (the receiver can recognize that these information are related to each other based on the order or type of the data, etc.) needs to be performed so that these data files can be grasped as being related to each other. Since the transmission control such as the file structure or transmission protocol related to recording or transmission is not directly related to the present invention and known methods can be used, the details will be omitted from the description. Note that the above information can be recorded on the recording medium 208 and then can be transmitted to the computer 102 through the communication unit 209, or the recording medium 208 can be removed from the digital camera 101 to read the image data in the computer 102. Additionally, instead of generating a defocus map (recording information distribution), the camera can record the paired parallax images together with the above associated information, and the computer 102 can generate a defocus map.

[0091] In this embodiment, the processing from step S504 to step S508 is performed by the computer 102. Since the processing from step S504 to step S508 is performed by a device different from the digital camera 101, the user can know the information related to the degree of deviation of the optical system and the imaging element from the designed positions without performing special camera operations. Additionally, although a defocus map is generated in the digital camera 101 in this example, the parallax images can be sent to the computer 102, and the computer 102 can generate a defocus map. When the computational load on the digital camera 101 is heavy during continuous imaging, by dispersing the calculation for generating the defocus map to the computer 102, the time for calculating the information related to the degree of deviation of the optical system and the imaging element from the designed positions can be reduced. Additionally, by further transmitting the information related to the transformation coefficient KX and the effective f-number F' uniquely determined by the lens in use and the imaging conditions, the computer 102 can generate various distance information distributions. Additionally, the information related to the transformation coefficient KX and the effective f-number F' can be pre-stored in the computer 102, and can be read from the stored information based on the received lens identification (ID) number and imaging information.

[0092] Figure 8is a block diagram schematically showing an example of the functional structure of the image processing unit 306 included in the computer 102 according to the present embodiment. Now, hereinafter, with further reference to Figure 5B the operation of the image processing unit 306 will be described. Note that the operation of the image processing unit 306 is implemented under the control of the system control unit 301.

[0093] Initially, the system control unit 301 receives the information transmitted in step S503 and loads the read data into the RAM 303 (step S504).

[0094] Subsequently, in step S505, the depth direction estimation unit 800 estimates the depth direction in the image based on the camera state detection information 803 (imaging condition) when acquiring the disparity image recorded on the acquisition RAM 303. In this example, the depth direction is estimated with reference to the plane where the defocus amount becomes zero. Here, with reference to Figure 9 the plane where the defocus amount becomes zero will be described.

[0095] Figure 9 shows an image of the planar object 901 on the ground being taken in a state where the overlooking optical system and the imaging element of the digital camera 101 have not deviated from the designed positions, and the inclination of the digital camera 101 with respect to the horizontal direction ( Figure 9 the x-axis direction in ) is zero. In addition, when the autofocus frame 903 is at the point where the optical axis 900 and the object 901 intersect each other, the plane connecting the focus areas (hereinafter referred to as the focus plane) is the plane 902 parallel to the imaging unit 205 and perpendicular to the optical axis 900. In addition, the focus plane 902 in the captured image of the object 901 can also be represented by the straight line 904 passing through the autofocus frame 903. Figure 10 shows the defocus map 1000 of the captured image of the object 901, the autofocus frame 1001 therein, and the focus plane 1002 therein. According to Figure 10 , in the case of taking an image in a state where the camera with the optical system and the imaging element not deviated from the designed positions is overlooking and its inclination with respect to the horizontal direction is zero, the focus plane 1002 is a horizontal straight line with respect to the image. In addition, since the vertical orientation of the camera during imaging is known, the part above the focus plane 1002 is far away, and the part below the focus plane 1002 is close. That is, it can be estimated that the depth direction changes from the lower part to the upper part of the captured image. The depth direction estimation unit 800 outputs the expression of the straight line representing the focus plane 1002 as the depth estimation information 804.

[0096] Based on the defocus map 802 and the depth estimation information 804 (the expression of the straight line representing the focal plane) obtained by the depth direction estimation unit 800, the deviation degree calculation unit 801 calculates information 805 related to the deviation degree of the optical system and the imaging element for capturing the parallax image with respect to the design position (step S506).

[0097] Here, reference will be made to Figure 11A and Figure 11B to describe the phenomena that occur when the optical system and the imaging element are displaced from the design position. Figure 11A FIG. shows the state of the optical system 204 and the imaging unit 205 at the design position. When imaging is performed in this state, the focal plane 1100 is parallel to the imaging unit 205. On the other hand, Figure 11B FIG. shows the state where the optical system 204 is displaced from the design position and eccentricity occurs. In this case, based on the Scheimpflug's law, the focal plane 1101 is inclined according to the angle θ formed by the optical system 204 and the imaging unit 205. In Figure 11B the state where the camera is overlooking and the inclination of the camera with respect to the horizontal direction is zero, in Figure 12 , 1200 represents the defocus map of the captured image of the planar object on the ground, 1201 represents the autofocus frame therein, and 1202 represents the focal plane therein. According to Figure 12 , it should be understood that the depth changes from the lower right to the upper left of the screen. Therefore, there is a deviation from the depth change direction ( ​ ) in the state where the optical system and the imaging element are not displaced from the design position. Therefore, the relationship between the user's depth perception and the focal plane deviates, resulting in an imaging result that the user does not expect.

[0098] The deviation degree calculation unit 801 calculates the angle θ_diff formed by the expression of the straight line representing the focal plane 1202 in the defocus map 802 and the straight line 1002 representing the focal plane estimated by the depth direction estimation unit 800, and causes the RAM 303 to store this angle θ_diff as the evaluation value 805 representing the deviation degree. ​ FIG. shows θ_diff. The larger θ_diff is, the greater the deviation of the optical system and the imaging element with respect to the design position (calibration is required). Note that when the camera is inclined with respect to the horizontal direction during imaging, this inclination angle can be subtracted from θ_diff for correction, and the effects of the present invention can also be obtained even for images captured in a state where the camera is inclined with respect to the horizontal direction.

[0099] In subsequent step S507, the system control unit 301 compares the calculated θ_diff with a threshold value stored in advance, and if θ_diff is greater than the threshold value, step S508 is performed, and if θ_diff is less than or equal to the threshold value, the process ends.

[0100] In step S508, the system control unit 301 sends the following information to the digital camera 101 through the communication unit 304 to notify the optical system and the imaging element of the deviation from the designed position in the camera used by the user.

[0101] - Identification number of the camera body where the deviation is detected

[0102] - Identification number of the lens where the deviation is detected

[0103] In step S509, the system control unit 201 in the digital camera 101 determines whether it has received the information sent from the computer 102. If these information are received, step S510 is performed; if these information are not received, the process ends.

[0104] In step S510, the system control unit 301 outputs a display as ​ such to the display unit 210, and recommends the user to repair the camera and the lens at the customer center. The customer center receives the ID information of the camera and the lens, etc. and the image data from the user (digital camera 101), which is useful for determining or counting repair / failure information, etc.

[0105] In the above manner, according to the present embodiment, information related to the degree of deviation of the optical system and the imaging element from the designed position can be calculated without hindering the convenience of the user, and this information can be notified to the user.

[0106] In addition, in the present embodiment, although text notifying the user of the deviation of the optical system and the imaging element from the designed position is displayed, in order to enable the user to more easily recognize the occurrence of the deviation, an image can be displayed on the display unit 210. Specifically, the display unit 210 can be configured to display the ​ defocus map using grayscale generated in step S502, or the defocus map whose color values have been converted through look-up table conversion or the like.

[0107] In addition, in the present embodiment, although information is displayed to the user in the case where the optical system or the imaging element deviates from the designed position, information can also be displayed in the case where the occurrence of the deviation is not detected or in both cases. According to this configuration, when the user immediately needs to know the judgment result, the user can know whether calibration is required.

[0108] In addition, in the present embodiment, although the case of generating a defocus map by calculating a disparity amount using paired disparity images is described as an example, the present invention is not limited thereto. As a method for generating a defocus map, for example, a DFD (Depth From DeFocus) method can be adopted, in which a defocus map is obtained based on the correlation between two images with different focus positions or f-numbers. Since information related to the degree of deviation of the optical system and the imaging element from the design position can be calculated by using the images obtained in the aperture bracket imaging mode, the opportunity for detecting the deviation increases, and information can be provided to the user at an appropriate timing.

[0109] In addition, in the present embodiment, although the depth direction in the image is estimated based on the camera state detection information, the present invention is not limited thereto. For example, the depth direction can also be estimated by using information related to the vanishing point. With this configuration, even if the camera does not include a gyroscope or a sensor for detecting the state of the camera, the effects of the present invention can be obtained, and the convenience of the user is increased. Now, a method for estimating the depth direction by using vanishing point detection and for calculating information related to the degree of deviation of the optical system and the imaging element from the design position will be described below.

[0110] The vanishing point is the point at which the straight lines on the picture plane corresponding to the parallel lines in the three-dimensional space converge when projected onto the image plane through a projective transformation. That is, the vanishing point is the "infinite far point" on the plane image onto which the space with actual depth is projected, and is recognized as the point at which the extension lines of the lines parallel to the depth direction intersect each other or the extension of the plane extending in the depth direction converges to the infinite far point. Therefore, a plurality of straight lines in the image are detected by a known method such as the Hough transform, and the point at which the maximum number of the detected straight lines converge can be detected as the vanishing point. The result of detecting the vanishing point is shown in ​ in ​ the result of detecting the vanishing point.

[0111] In ​ 1500 is the vanishing point. In addition, the depth direction can be estimated as the direction 1501 from the autofocus frame 600 toward the vanishing point. The depth direction estimation unit 800 outputs the direction 1501 toward the vanishing point as the depth direction estimation information 804.

[0112] The deviation degree calculation unit 801 calculates the gradient (change direction) of the defocus amount near the autofocus frame in the defocus map 802 by using known techniques. Subsequently, based on the difference from the depth direction estimation information 804, the deviation degree calculation unit 801 calculates an evaluation value representing the deviation degree of the optical system and the imaging element relative to the design position. Specifically, the direction toward the vanishing point and the gradient direction of the defocus amount are each regarded as vectors, and the difference between the vectors is the evaluation value. The greater the deviation of the optical system and the imaging element from the design point, the greater the evaluation value.

[0113] In addition, the method for estimating the depth direction in the image by referring to the features extracted from the image can use not only the above-described vanishing point detection but also the information related to the change in the density of the texture. A method for detecting the depth in the image by referring to the change in the density of the texture. As a method, for example, the method described in “Texture Structure Classification and Depth Estimation using Multi-Scale Local Autocorrelation Features”, KANG Y, HASEGAWA O, NAGAHASHI H (Tokyo Inst. Technol.), JST-PRESTO (non-patent literature) can be adopted.

[0114] Specifically, when referring to an image for recording a still image and there is a uniform texture in the image (for example, ​ a road), the depth direction estimation unit 800 uses the fact that the density of the texture decreases as the distance increases. That is, if a region where the density of the same texture gradually decreases is detected in the image, the depth direction estimation unit 800 determines that the plane covered by the predetermined texture is moving away from the imaging position. The direction from the front side to the opposite side is output as the depth direction estimation information 804. In particular, in the focused area, it is possible to detect fine textures, and by making the above determination near the autofocus frame, the depth direction can be estimated with high accuracy. In addition, by using known typical object detection, it is possible to pre-detect regions where there is likely to be a uniform texture (such as the ground like a road, a water surface, or a hedge which is a structure built in the vertical direction on the ground or water surface), and the target area can be restricted. Therefore, the processing time for estimating the distribution of positions in the depth direction can be reduced.

[0115] In addition, similar to the case of using the vanishing point, the deviation degree calculation unit 801 sets the difference between the vector of the gradient (change direction) of the defocus amount near the autofocus frame in the defocus map 802 and the vector of the depth direction estimated based on the change in the density of the texture as the evaluation value.

[0116] In addition, in the present embodiment, an evaluation value indicating the degree of deviation of the optical system and the imaging element from the designed position is calculated based on a single captured image, and it is determined whether an offset has occurred. However, the present invention is not limited thereto. The determination may be made when the number of images captured by the user reaches a certain reference number. By making the determination based on the evaluation values in a plurality of captured images, the reliability of the determination regarding whether an offset has occurred can be increased. In addition, regarding the condition related to whether to notify the user of the occurrence of the offset of the optical system and the imaging element from the designed position, by checking whether the number of images in which the offset is detected reaches a certain reference number, the reliability of the determination can be further improved.

[0117] In addition, instead of determining the degree of deviation of the optical system and the imaging element from the designed position for all captured images, it is preferable to perform the deviation degree determination process after pre-evaluating whether the images in a large number of images are suitable for the deviation degree determination. Specifically, by performing the above-described typical object detection, an evaluation is made regarding whether there is a uniform texture such as a road and whether the image is suitable for estimating the depth direction. With this configuration, the processing time for determining the degree of deviation can be reduced. In addition, a subject having a texture such as a road is also suitable for detecting the phase difference between parallax images in step S502, so a more accurate deviation degree evaluation value can be expected. In addition, by recording known GPS information as image attachment information, it is possible to determine whether a subject having a texture such as a road is likely to be included in the captured image. By pre-selecting images that are expected to include textures from a large number of images captured by the user, the time for calculating a highly reliable evaluation result can be reduced. In addition, in the case of a camera having a detachable optical system, statistics related to whether an offset of the optical system and the imaging element from the designed position can be detected for each attached lens are collected. This makes it possible to determine whether an offset has occurred in the optical system or the imaging element and provide the user with a more detailed determination result.

[0118] In addition, in the deviation degree calculation unit 801, by considering the following details to obtain the gradient of the defocus map 802 with high precision, a highly reliable evaluation result can be obtained. Specifically, a histogram (statistical information) of the calculated defocus map is acquired, and based on the shape of the histogram, it is determined whether the gradient of the defocus amount can be obtained with high precision. Although it will be described below, it is preferable to select an image in which the histogram has a large width and a smooth change. ​ Shows ​ The histogram of the defocus map in. According to ​ , the defocus amount is widely distributed from the front side to the opposite side, and in addition, the change in the defocus amount is smooth. Therefore, ​ The image in is suitable for evaluating the change direction of the defocus amount in the entire image. On the other hand,​ Shows a defocused image of a bust of a person in a portrait photograph as ​ shown, and ​ shows its histogram. Since the amount of defocus in the image is concentrated on the person in the bust, it should be understood that this image is not suitable for evaluating the direction of change in the amount of defocus in the entire image. In the deviation degree calculation unit 801, the histogram of the defocused image is confirmed before comparison with the depth direction estimation information 804, and if the image is not suitable for evaluating the deviation degree, the determination process is interrupted, thereby reducing the calculation time.

[0119] In addition, in order to obtain the phase difference between the parallax images with high precision in step S502, an image with a high S / N can be selected. Therefore, images taken at the lowest possible sensitivity are preferentially selected from a large number of captured images, thereby improving the reliability of the evaluation value.

[0120] In addition, since the phase difference between the parallax images is affected by the aberration in the optical system 204, it is preferable to correct the aberration known as the design information of the optical system before comparison with the estimation result using the depth direction estimation unit 800. In addition, the area to be compared can be limited to an area where the influence of the aberration is small. In this way, a more reliable evaluation result can be calculated.

[0121] In addition, in step S510, when detection information related to the occurrence of the deviation of the optical system and the imaging element from the designed position is received, a display encouraging the user to repair the camera and lens at the customer center is output to the display unit of the digital camera 101. However, other solutions are also possible. Specifically, in step S508, the evaluation value 805 indicating the deviation degree is further sent to the digital camera 101. According to the evaluation value 805 indicating the deviation degree, simple calibration can be performed by driving the IS mechanism mounted on the optical system and the imaging unit to approach the state where no deviation of the optical system and the imaging element from the designed position has occurred. Alternatively, referring to the evaluation value 805 indicating the deviation degree, the image can be processed to approach the image obtained in the state where no deviation of the optical system and the imaging element from the designed position has occurred by performing image processing (sharpening or blurring processing) on the area with the tilted focal plane. Specifically, compared with the state where no deviation has occurred, the area with a defocus amount close to the focused state is blurred. Conversely, the area close to the background or the front side compared with the original defocus amount is sharpened. With the above configuration, even in a situation where the user cannot repair the camera and lens, an image with the depth of field desired by the user can be obtained, and user convenience can be improved.

[0122] In addition to the user's camera, detection information related to the occurrence of an offset of the optical system and the imaging element from the designed position can also be sent to the customer center. The customer center manages customer information registered by the user himself / herself and information related to all owned devices, and records the number of occurrences of the offset of the optical system and the imaging element from the designed position and the number of maintenance times. Thereby, user convenience can be further improved, such as reducing repair time and the like.

[0123] In addition, in the case where the camera and the lens used by the user for imaging are integrated, the display unit 201 can be configured to present to the user whether to execute an operation mode for distinguishing the cause of occurrence between the optical system and the imaging element. In response to the user selecting the operation mode for distinguishing the cause, the display unit 201 is instructed to encourage the user to capture an image suitable for distinguishing the cause. As specific details of this instruction, the user pastes graph paper on the wall directly in front of the user, and captures images while changing the imaging conditions (focal length, focusing lens position, aperture) of the optical system. As a result of the analysis, if the determination result changes by changing the imaging conditions of the optical system, the cause is the lens; if the occurrence of the offset is detected regardless of the imaging conditions, the cause is the imaging element. With this configuration, the cause of the offset of the optical system and the imaging element from the designed position can be found, and more appropriate notification or repair can be performed.

[0124] [Second Embodiment]

[0125] Hereinafter, an image processing apparatus, an image processing method, and an image processing program according to a second embodiment of the present invention will be described in detail with reference to some drawings. Note that components that are substantially the same as those in the image processing apparatus according to the above-described first embodiment are denoted by the same reference numerals, and these components will be omitted from the description or briefly described.

[0126] The first embodiment illustrates an embodiment in which information related to the degree of deviation of the optical system and the imaging element from the designed position, which is an internal parameter of the camera device 100, is calculated and notified to the user. The second embodiment of the present invention will illustrate an embodiment in which calibration of the position or orientation of the image processing apparatus, which is an external parameter, is performed.

[0127] First, the imaging system according to the present embodiment will be described. The imaging system according to the present embodiment images an inspection target surface of a structure that is an object of social infrastructure inspection, and in particular, can easily face the inspection target surface and image it or evaluate the captured image. The imaging system according to the present embodiment includes a camera device that periodically / irregularly captures moving images or still images, a lens device to be attached to the camera device, and a pan-tilt device for rotating the camera device.

[0128] First, with reference to ​ the block diagram, an example of the hardware configuration of the camera device 1900 and the lens device 1913 according to this embodiment will be described. Note that the imaging device in this embodiment may also be configured as the digital camera 101 as in the first embodiment. ​ The state where the lens device 1913 is attached to the camera device 1900 is shown.

[0129] First, an example of the hardware configuration of the camera device 1900 will be described. The camera device 1900 according to this embodiment acquires the distance information distribution at a plurality of positions in the imaging range of the camera device 1900, obtains information related to an instruction for rotating or translating the camera device 1900 based on the difference between the acquired distance information, and outputs the acquired information. Here, as in the first embodiment, the distance information and the distance information distribution may be any one of the image offset amount and the image offset amount distribution between a pair of disparity images, the defocus amount and the defocus map obtained by any means, or the subject distance information and the subject distance map.

[0130] The CPU (Central Processing Unit) 1901 performs various processes by using computer programs or data stored in the ROM (Read Only Memory) 1902 or the RAM (Random Access Memory) 1903. Therefore, the CPU 1901 controls the operation of the entire camera device 1900, and also performs or controls the processes described later as the processes performed by the camera device 1900.

[0131] The ROM 1902 stores the setting data of the camera device 1900, computer programs or data related to the startup of the camera device 1900, and computer programs or data related to the basic operations of the camera device 1900, etc.

[0132] The RAM 1903 has areas for storing computer programs or data read from the ROM 1902 or computer programs or data read from the memory card 1909 via the recording medium I / F 1908. The RAM 1903 also has areas for storing the captured images output from the imaging element 1904, computer programs or data received from external devices via the external I / F 1910, or data received from the lens device 1913 through the camera communication unit 1907. The RAM 1903 also has a work area used when the CPU 1901 performs various processes. In this way, the RAM 1903 can appropriately provide various areas.

[0133] The pixel arrangement of the imaging element 1904 has the same as ​It has the same arrangement structure as the imaging unit 205, and generates and outputs a captured image corresponding to the light entering through the lens device 1913. The display unit 1905 is a liquid crystal display (LCD), an organic EL display (OLED), or the like, and is a device that displays images or text on a display screen or a viewfinder screen. Note that the display unit 1905 may not be included in the camera device 1900, and may be, for example, an external device that can communicate with the camera device 1900 wiredly and / or wirelessly.

[0134] The operation unit 1906 is a user interface such as buttons, dials, touch panels, or joysticks, and can input various instructions to the CPU 1901 through user operations.

[0135] The camera communication unit 1907 performs data communication between the camera device 1900 and the lens device 1913. The recording medium I / F 1908 is an interface for attaching the memory card 1909 to the camera device 1900, and the CPU 1901 reads data from and writes data to the memory card 1909 via the recording medium I / F 1908.

[0136] As the memory card 1909, for example, card-type recording media such as SD, CF, CFexpress, XQD, or CFast are known. In addition, the memory card 1909 can also record data on an external device via a wireless network.

[0137] The external I / F 1910 is a communication interface for data communication with an external device, and the CPU 1901 performs data communication with the external device via the external I / F 1910. The power supply unit 1911 supplies and manages the power in the camera device 1900.

[0138] The CPU 1901, ROM 1902, RAM 1903, imaging element 1904, display unit 1905, operation unit 1906, camera communication unit 1907, recording medium I / F 1908, external I / F 1910, and power supply unit 1911 are all connected to the system bus 1912.

[0139] Next, an example of the hardware structure of the lens device 1913 will be described. The CPU 1914 performs various processes by using computer programs or data stored in the ROM 1915 or the RAM 1916. Therefore, the CPU 1914 controls the operation of the entire lens device 1913, and also performs or controls the processes described later as the processes performed by the lens device 1913.

[0140] The ROM 1915 stores setting data of the lens device 1913, computer programs or data related to the startup of the lens device 1913, and computer programs or data related to the basic operations of the lens device 1913, etc.

[0141] The RAM 1916 has an area for storing computer programs or data read from the ROM 1915 or data received from the camera device 1900 via the lens communication unit 1919. The RAM 1916 also has a working area used when the CPU 1914 performs various processes. In this way, the RAM 1916 can appropriately provide various areas.

[0142] The lens communication unit 1919 performs data communication between the camera device 1900 and the lens device 1913. For example, the lens communication unit 1919 receives control information from the camera device 1900 to the lens device 1913, sends the operation state of the lens device 1913, etc. to the camera device 1900, or receives power supply from the camera device 1900.

[0143] The display unit 1917 is a liquid crystal display (LCD) or an organic EL display (OLED), etc., and is a device for displaying the operation state of the lens device 1913, etc. Note that the display unit 1917 may not be included in the lens device 1913, and may be, for example, an external device that can communicate with the lens device 1913 wired and / or wirelessly.

[0144] The operation unit 1918 is a user interface such as buttons, dials, touch panels, or joysticks, etc., and can input various instructions to the CPU 1914 through user operations. In addition, the instructions input by operating the operation unit 1918 through user operations can be sent to the camera device 1900 via the lens communication unit 1919.

[0145] The lens drive unit 1920 controls the optical lenses included in the lens device 1913 based on instructions from the CPU 1901 or the CPU 1914, and thereby controls the aperture, focusing, zoom focus, and camera shake correction, etc. The light that enters via the optical lens after being controlled by the lens drive unit 1920 for the aperture, focusing, zoom focus, and camera shake correction, etc. is received by the above-mentioned imaging element 1904, and the imaging element 1904 generates and outputs a captured image corresponding to the received light.

[0146] The CPU 1914, ROM 1915, RAM 1916, lens communication unit 1919, display unit 1917, operation unit 1918, and lens drive unit 1920 are all connected to the system bus 1921.

[0147] Next, reference will be made to​ The block diagram is used to illustrate an example of the hardware structure of the pan-tilt device 2000 according to this embodiment.

[0148] The CPU 2001 performs various processes by using computer programs or data stored in the ROM 2002 or the RAM 2003. Therefore, the CPU 2001 controls the operation of the entire pan-tilt device 2000, and also performs or controls the processes described later as the processes performed by the pan-tilt device 2000.

[0149] The ROM 2002 stores setting data of the pan-tilt device 2000, computer programs or data related to the startup of the pan-tilt device 2000, and computer programs or data related to the basic operations of the pan-tilt device 2000, etc.

[0150] The RAM 2003 has an area for storing computer programs or data read from the ROM 2002. The RAM 2003 also has a working area used when the CPU 2001 performs various processes. In this way, the RAM 2003 can appropriately provide various areas.

[0151] The external I / F 2004 is a communication interface for obtaining various instructions from the remote control device 2010 through wireless or wired communication. The remote control device 2010 is a device for inputting various instructions to the pan-tilt device 2000, and can input, for example, a change instruction for changing the pan angle or the tilt angle of the camera device 1900 mounted on the pan-tilt device 2000. The external I / F 2004 can also communicate with the camera device 1900 mounted on the pan-tilt device 2000.

[0152] The power supply unit 2005 supplies and manages the power in the pan-tilt device 2000. The display unit 2006 is a liquid crystal display (LCD) or an organic EL display (OLED), etc., and is a device for displaying the operation state of the pan-tilt device 2000, etc. Note that the display unit 2006 may not be included in the pan-tilt device 2000, and can be, for example, an external device that is communicable with the pan-tilt device 2000 in a wired and / or wireless manner.

[0153] The operation unit 2007 is a user interface such as buttons, dials, touch panels, or joysticks, etc., and can input various instructions to the CPU 2001 through user operations.

[0154] The drive unit 2008 includes a base (fixing member) for fixing the camera device 1900, and a drive mechanism for panning the base, tilting the base, or translating the base in the XYZ directions. By controlling the drive mechanism, the drive unit 2008 controls the panning angle, tilting angle, and position in the XYZ directions of the camera device 1900 based on an instruction received from the remote control device 2010 via the external I / F 2004 or the like. Further, in the present embodiment, the panning, tilting, and imaging position of the camera device 1900 are controlled by mounting the camera device 1900 on the above-described pan-tilt device 2000. However, the present invention is not limited thereto, and can be applied, for example, to devices such as drones, in which at least one of the panning, tilting, and imaging position of the camera device 1900 is controlled by the movement of the device itself.

[0155] The CPU 2001, ROM 2002, RAM 2003, external I / F 2004, power supply unit 2005, display unit 2006, operation unit 2007, and drive unit 2008 are all connected to the system bus 2009.

[0156] Next, an example of the functional structure of the camera device 1900 will be described with reference to ​ the block diagram. Although each functional unit shown in the following description ​ mainly performs processing, actually, the CPU 1901 executes a computer program corresponding to the functional unit, thereby performing the operation of the functional unit. Further, ​ at least a part of the functional units shown can be implemented by hardware.

[0157] The subject recognition unit 1928 identifies whether the inspection target surface of the structure, which is the object of social infrastructure inspection, is included in the captured image by using known typical object detection. Specifically, the subject recognition unit 1928 prestores the feature amounts related to the structure that is the object of infrastructure inspection, and compares the feature amounts of the image obtained by imaging with the stored image. The result is also used as information for estimating the depth direction in the image. As a result of the recognition, if the inspection target surface of the structure that is the object of social infrastructure inspection is imaged, a calibration process is performed so that the camera device 1900 is in a facing relationship with the inspection target surface of the structure. Here, if the inspection target surface is flat, since the camera and the structure are in a facing relationship, it is assumed that the defocus amount in the imaging range is almost uniform. As a calibration target, similar to the method for determining the deviation degree of the optical system and the imaging element from the designed positions described in the first embodiment, the position and posture of the camera device 1900 can be corrected so that the values in the distance information distribution in the imaging range become uniform (fall within a predetermined range). However, the present embodiment shows a method for simply setting the control in the pan or tilt direction based on the defocus amount in a partial area within the imaging range hereinafter.

[0158] The determination unit 1922 acquires setting information indicating the setting information of "the rotation direction and translation direction for operating the camera device 1900 so that the camera device 1900 faces the inspection target surface of the structure that is the object of social infrastructure inspection". This setting information is determined, for example, by the user operating the operation unit 1906. If the driving of the camera indicated by the setting information is the rotation direction and the horizontal direction (pan direction), the determination unit 1922 sets each of the two regions arranged in the left - right direction within the imaging range of the camera device 1900 as a "region for acquiring the defocus amount". (For example, the positions near the left end and the right end within the imaging range). Here, the minimum unit of each region is 1 pixel.

[0159] On the other hand, if the driving of the camera indicated by the setting information is the rotation direction and the vertical direction, the determination unit 1922 sets each of the two regions arranged in the up - down direction within the imaging range of the camera device 1900 as a "region for acquiring the defocus amount". (For example, the positions near the upper end and the lower end within the imaging range). Here, the minimum unit of each region is also 1 pixel.

[0160] In addition, if the driving of the camera indicated by the setting information is translation, the determination unit 1922 sets each of the four regions arranged in the up - down direction and the left - right direction within the imaging range of the camera device 1900 as a "region for acquiring the defocus amount". Here, the minimum unit of each region is also 1 pixel.

[0161] In addition, in this embodiment, instead of using the setting information set by the user (or regardless of the setting information), as in the first embodiment, distance information can be obtained in multiple regions. That is to say, a distance information distribution can be obtained, and the driving of the pan-tilt device 2000 (the position and posture of the camera) can be controlled based on the analysis result of the distribution information. At this time, the "region for obtaining the defocus amount" is, for example, the entire region where the defocus amount can be obtained. By obtaining the distance information distribution, for example, by obtaining the two-dimensional or three-dimensional inclination of the distance information through plane detection, the position and posture of the camera can be controlled so that the inclination is facing and becomes close to zero in each direction.

[0162] The control unit 1924 obtains the defocus amount from the "region for obtaining the defocus amount" determined by the determination unit 1922 within the imaging range of the camera device 1900. The acquisition unit 1923 acquires the defocus amount acquired by the control unit 1924. The difference calculation unit 1925 calculates the difference between the defocus amount acquired by the acquisition unit 1923 and another defocus amount.

[0163] The determination unit 1926 determines the notification information for notifying the "rotation degree or translation degree (including direction) for driving the camera device 1900" based on the difference calculated by the difference calculation unit 1925. The output unit 1927 outputs the notification information determined by the determination unit 1926 to the pan-tilt device 2000 via the external I / F 1910. The pan-tilt device 2000 acquires the notification information via the external I / F 2004 and controls the drive unit 2008 based on the notification information to set the camera device 1900 in the desired position and posture.

[0164] In this embodiment, such an imaging system is used to image the social infrastructure to be inspected, and the social infrastructure is inspected based on the captured image obtained through imaging. Reference will be made to ​ and ​ to describe the imaging method for imaging the social infrastructure by using the imaging system according to this embodiment.

[0165] ​ An example of the inspection object surface of the social infrastructure to be inspected is shown. ​The social infrastructure 2100 shown is a wall-like structure with a side surface 2101 and a long lateral dimension. Reference numeral 2102 denotes a joint portion that occurs when the social infrastructure 2100 is segmented based on a design drawing and constructed with construction joints. The portion 2102 is also referred to as a construction joint portion, but is referred to here as a joint for ease of understanding. The joint portion 2102 can be visually observed, and thus the joint portion 2102 also serves as a unit for inspection operations. Reference numeral 2103 denotes an area (inspection target area) that is the object of a single inspection, and the imaging system images an imaging area 2104 that includes the inspection target area 2103. In a captured image obtained by imaging the imaging area 2104, "a partial image corresponding to a peripheral area of the inspection target area 2103 in the imaging area 2104" is information for grasping the positional relationship with an adjacent inspection target area. Therefore, this partial image is used for alignment when combining images into a single image including the entire social infrastructure 2100. In addition, the partial image corresponding to the peripheral area is also used for inspection of deformations in a wide range that is not limited to a single inspection target area.

[0166] ​ Shows a state in which the imaging area 2104 is imaged using the imaging system according to the present embodiment. In ​ this, the camera device 1900 is attached to the pan-tilt device 2000 having a tripod 2108, and the lens device 1913 is attached to the camera device 1900. The width of the imaging range 2109 on the inspection target surface imaged by the combination of the camera device 1900 and the lens device 1913 (the size in the lateral direction in this figure) corresponds to the width of the imaging area 2104 (the size in the lateral direction in this figure).

[0167] When imaging of the inspection target area 2103 is completed, an inspection target area adjacent to the inspection target area 2103 and not yet imaged is imaged. The imaging system according to the present embodiment is moved to the position denoted by reference numeral 2110, and the inspection target area in the imaging range 2112 is imaged in substantially the same manner. When imaging at the position denoted by reference numeral 2110 is completed, in order to image an inspection target area adjacent to the inspection target area and not yet imaged, the imaging system according to the present embodiment is moved to the position denoted by reference numeral 2111, and the inspection target area in the imaging range 2113 is imaged in substantially the same manner. In the case where the camera device 1900 is mounted on a moving object such as a drone, the user manually or automatically moves the moving object to each imaging position and sequentially performs imaging.

[0168] Here, in the present embodiment, the camera device 1900 needs to face the inspection object area. In the present embodiment, it is determined whether the camera device 1900 faces the inspection object area. If the camera device 1900 does not face the inspection object area, a notification for rotating or translating the camera device 1900 to face the inspection object area is issued.

[0169] To issue this notification, as described above, the control unit 1924 obtains the defocus amount at the position determined by the determination unit 1922. The method for obtaining the defocus amount is the same as the method in step S502 of the first embodiment, and thus is omitted from the description here. Here, the obtained defocus amount has a continuous value, and the defocus amount corresponding to the focusing degree can be determined as "-11" for the front focus, "0" for the in-focus state, and "+7" for the rear focus. In addition, as in the first embodiment, data representing the spatial (two-dimensional) defocus amount distribution in the imaging range can be created, and the control unit 1924 can be configured to obtain the defocus amount at the position determined by the determination unit 1922 in the defocus amount distribution (distance information distribution).

[0170] Next, the operation of the imaging system according to the present embodiment will be described with reference to ​ the flowchart. As described above, the user installs the imaging system to face the inspection object surface to image the inspection object surface by using the imaging system according to the present embodiment. At this time, the user can install the camera device 1900 in a direction assuming that it basically faces the inspection object area. However, if there is no reference point for the structure or the installation position and there is no accurate measurement information on the surrounding environment, the camera device 1900 cannot be accurately installed to face the inspection object area. In response to the power-on of the camera device 1900 after the camera device 1900 is installed, the captured image captured by the imaging element 1904 is displayed as a live view image by the display unit 1905 on the display screen on the back of the camera device 1900. Subsequently, the processing according to ​ the flowchart starts.

[0171] In step S2200, the subject recognition unit 1928 performs typical object detection processing on the captured image. In the present embodiment, since the inspection object surface of the structure to be imaged is included in the objects to be detected as typical objects, information related to the feature amount representing the inspection object surface is stored in the ROM 1902 in advance.

[0172] In step S2216, the subject recognition unit 1928 determines whether the object detected in step S2200 is the inspection target surface of the structure to be imaged when the camera device 1900 is in a facing relationship. If it is determined that the object is the inspection target surface, the process continues and proceeds to step S2201. On the other hand, if it is determined that the object is not the inspection target surface, the processing ends according to the ​ flowchart.

[0173] In step S2201, the determination unit 1922 acquires setting information that represents "the drive of the camera device 1900 for making the camera device 1900 face the inspection subject surface".

[0174] For example, as ​ shown, the operation unit 1906 controls "the drive of the camera device 1900 for making the camera device 1900 face the inspection subject surface" (the facing detection direction). That is, the operation unit 1906 has a switch corresponding to the operation unit 2007, and this switch is used to set the facing detection direction to at least any one of "vertical direction", "horizontal direction", and "translation". By operating this switch, the user can set the facing detection direction to either the vertical direction (rotation axis = pitch axis) or the horizontal direction (rotation axis = roll axis). The determination unit 1922 acquires the facing detection direction set by using the switch as the setting information. As ​ and ​ shown, if the user images a horizontally long structure while moving horizontally, the facing detection direction in the horizontal (rotation) direction is selected.

[0175] In addition, as described above, in the case of performing control including translation in the XYZ directions to obtain images with multiple area focuses from the front (for example, through mode setting), the setting of the facing detection direction in step S2201 is not performed. The control unit 1924 estimates the position and posture of the plane of the subject to be focused based on the distance information distribution obtained in multiple areas in the captured image, and controls the position and posture of the pan-tilt device 2000 (camera device 1900) as in the first embodiment.

[0176] As an example of setting the facing detection direction to either the vertical direction (rotation axis = pitch axis) or the horizontal direction (rotation axis = roll axis), the case of setting the facing detection direction to the horizontal direction will be described below.

[0177] Subsequently, in step S2202, since the facing detection direction is the horizontal direction, the determination unit 1922 sets each of the two areas arranged in the left-right direction within the imaging range of the camera device 1900 as "the area for acquiring the defocus amount". For example, as ​As shown, the determination unit 1922 sets the area 2400 near the left end and the area 2401 near the right end of the imaging area 2104 of the social infrastructure 2100 that falls within the imaging range 2402 of the imaging device 1900 as the "area for obtaining the defocus amount". Additionally, this embodiment is not limited to this, and when the front is set to the detection direction, similar to the first embodiment, the defocus amount of the entire screen (entire image) can be obtained.

[0178] In step S2203, the control unit 1924 obtains the defocus amount at the position set in step S2202 (in the case of ​ being areas 2400 and 2401) as described above. At this time, the camera device 1900 does not have to focus on the inspection object surface, and obtains the defocus amount in the area set in step S2202.

[0179] In step S2204, the acquisition unit 1923 acquires the "defocus amount in the left area" and the "defocus amount in the right area" obtained in step S2203. Subsequently, the difference calculation unit 1925 calculates the difference by subtracting the "defocus amount in the right area" from the "defocus amount in the left area".

[0180] In step S2206, the determination unit 1926 obtains the "information indicating the rotation direction and rotation degree of the camera device 1900" corresponding to the difference between the defocus amounts calculated in step S2204 as the rotation instruction information (notification information).

[0181] Here, as ​ shown, the table 2515 is registered in the ROM 1902. In the table 2515, the rotation instruction information corresponding to the difference between the defocus amounts is registered. In the column 2516, the range of the difference between the defocus amounts is registered. For example, the range of the difference between the defocus amounts "+11 or more" is registered in the row 2519 of the column 2516, and the range of the difference between the defocus amounts "-5 to -10" is registered in the row 2524 of the column 2516.

[0182] In the column 2517, the icons corresponding to the rotation amount in the case of rotating the camera device 1900 counterclockwise are registered. The icon registered in the row 2519 of the column 2517 indicates a larger rotation amount than the icon indicated by the row 2520 of the column 251,7. The icon registered in the row 2520 of the column 2517 indicates a larger rotation amount than the icon indicated by the row 2521 of the column 2517. The icons registered in the rows 2522 to 2525 of the column 2517 indicate that no counterclockwise rotation is required.

[0183] In column 2518, icons corresponding to the amount of rotation in the case where the camera device 1900 is rotated clockwise are registered. The icon registered in row 2525 of column 2518 indicates an amount of rotation larger than that indicated by the icon registered in row 2524 of column 2518. The icon registered in row 2524 of column 2518 indicates an amount of rotation larger than that indicated by the icon registered in row 2523 of column 2518. The icons registered in rows 2519 to 2522 of column 2518 indicate that no clockwise rotation is required.

[0184] Therefore, for example, if the difference between the defocus amounts calculated in step S2204 is "+7", the determination unit 1926 acquires the two icons registered in row 2520 corresponding to the range "+10 to +5" including the difference "+7" as rotation instruction information.

[0185] Further, for example, if the difference between the defocus amounts calculated in step S2204 is "-12", the determination unit 1926 acquires the two icons registered in row 2525 corresponding to the range "-11 or less" including the difference "-12" as rotation instruction information.

[0186] That is, in the ​ table, rotation instruction information for notifying the rotation direction corresponding to the sign of the difference between the defocus amounts and the rotation degree corresponding to the absolute value of the difference between the defocus amounts is registered.

[0187] In step S2214, the output unit 1927 outputs the rotation instruction information acquired in step S2206 as "notification information for notifying the user of the rotation direction and rotation degree of the camera device 1900" to the display unit 1905. The display unit 1905 displays this notification information on the display screen on the back surface of the camera device 1900. For example, as ​ shown, the icon 2405 acquired from column 2517 is displayed at the lower left of the live view image 2404 displayed on the display screen on the back surface of the camera device 1900. Further, the icon 2406 acquired from column 2518 is displayed at the lower right of the live view image 2404. Note that the display positions of the icon 2405 and the icon 2406 are not limited to specific display positions, and for example, the icon 2405 and the icon 2406 may be displayed so as to be superimposed on the live view image 2404. Further, in ​ the icons 2400a and 2401a are displayed in a superimposed manner at positions corresponding to the positions 2400 and 2401 respectively on the live view image 2404.

[0188] A user who sees the displayed icons 2405 and 2406 recognizes the notification for counterclockwise rotation of the camera device 1900, and rotates the camera device 1900 counterclockwise. ​ Shows the state after the camera device 1900 rotates counterclockwise from the ​ state.

[0189] Similarly, in the ​ state, since the icons 2409 and 2410 are still displayed, the user similarly recognizes the notification for counterclockwise rotation of the camera device 1900, and rotates the camera device 1900 counterclockwise. Here, both the icon 2406 and the icon 2410 indicate that clockwise rotation is not required. On the other hand, both the icon 2405 and the icon 2409 indicate that counterclockwise rotation is required, but the icon 2409 indicates a rotation with a smaller rotation amount compared to the icon 2405. ​ Shows the state after the camera device 1900 further rotates counterclockwise from the ​ state.

[0190] In the ​ state, the icon 2413 indicating that counterclockwise rotation is not required and the icon 2414 indicating that clockwise rotation is not required are displayed. A user who sees the displayed icons 2413 and 2414 recognizes the notification indicating that the camera device 1900 does not need to be rotated clockwise or counterclockwise, and does not rotate the camera device 1900.

[0191] ​ Shows the state where the camera device 1900 is assembled on the pan-tilt device 2000, and the remote control device 2010 for the pan / tilt operation of the pan-tilt device 2000 and the imaging operation of the camera device 1900 is connected. At this time, by connecting to the camera device 1900 via the external I / F 1910 of the camera device 1900, the remote control device 2010 can perform imaging by using the camera device 1900.

[0192] Return to reference ​ , in step S2215, the CPU 1901 determines whether the conditions for ending the processing according to the ​ flowchart are satisfied. For example, when the user inputs an instruction for ending the processing by operating the operation unit 1906 or disconnects the power supply of the camera device 1900, the CPU 1901 determines that the conditions for ending the processing according to the ​ flowchart are satisfied.

[0193] As a result of this determination, if the conditions for ending the processing according to the ​ flowchart are satisfied, according to the ​The processing of the flowchart ends. If the end condition is not satisfied, the processing proceeds to step S2203.

[0194] In the above manner, by mounting the pan-tilt device 2000 equipped with the camera device 1900 as described above ​ towards the inspection object surface, it is possible to notify the user of the rotation / translation instruction information for aligning the camera device 1900 with the inspection object surface. In addition, since the user who has received the notification operates the pan-tilt device 2000 or the like based on this notification, the camera device 1900 can be accurately aligned with the inspection object surface, and accurate inspection of deformation becomes possible. At the same time, by accurately aligning the camera device 1900 with the inspection object surface, when imaging an area adjacent to the inspection object surface, by translating the camera device 1900, it is possible to continuously image the inspection object surface under unified conditions. In addition, even when the imaging element 1904 or the lens device 1913 of the camera device 1900 deviates from the designed position due to long-term changes, since the camera device 1900 is accurately aligned with the inspection object surface, accurate inspection of deformation also becomes possible.

[0195] In the present embodiment, although the rotation direction for aligning the camera device 1900 with the inspection object surface is the horizontal (rotation) direction and the pan axis of the pan-tilt device 2000 is operated, it is possible to issue a rotation instruction for aligning the camera device 1900 with the inspection object surface in the vertical (rotation) direction by switching the alignment detection direction, and it is possible to operate the tilt axis. In addition, it is possible to simultaneously perform detection in the horizontal (rotation) direction and the vertical (rotation) direction, and it is possible to present the rotation instruction information in these two directions.

[0196] In addition, in the present embodiment, although an example of the value of the defocus amount is presented and the rotation instruction information is defined as three types, since the value of the defocus amount varies depending on the type of the image plane phase difference sensor to be used, it is possible to appropriately multiply by a coefficient or the like for use, and the types are not limited to these.

[0197] In addition, in the present embodiment, although an icon indicating both the rotation direction and the rotation degree is displayed, it is possible to separately display an icon indicating the rotation direction and an icon indicating the rotation degree, or it is possible to display only either of the two. In addition, the information indicating the rotation direction or the rotation degree is not limited to an icon, and for example, it may be text information. In addition, the method for notifying the rotation direction or the rotation degree is not limited to a specific notification method.

[0198] In addition, in the present embodiment, although an icon is displayed for the direction that does not require rotation, it is not necessary to display an icon for the direction that does not require rotation. In addition, for the direction that requires rotation, in addition to displaying an icon, other information such as text information may also be displayed.

[0199] In addition, in the present embodiment, although the camera device 1900 is mounted on the pan-tilt device 200, as described above, the camera device 1900 may also be mounted on a UAV (unmanned aerial vehicle) such as a drone device. With this configuration, it is possible to directly face and image the inspection target surface of an object structure in an environment where a pan-tilt cannot be installed.

[0200] In addition, in the present embodiment, although rotation and / or translation instruction information is notified to the user, the rotation and / or translation instruction information may also be output to the pan-tilt device 2000. The pan-tilt device 2000 may be configured to control the rotation of the camera device 1900 according to the rotation and / or translation instruction information, and can automatically direct the camera device 1900 to face the inspection target surface. With this configuration, the operation load on the user is reduced, thereby improving convenience.

[0201] In addition, in the present embodiment, although the camera device 1900 calculates the defocus amount (distance information distribution), as in the first embodiment, a computer communicably connected via a communication circuit may be configured to calculate the defocus amount.

[0202] Furthermore, in the present embodiment, although the distance information distribution is calculated to control the position and posture of the camera device 1900 by operating the pan-tilt device 2000, the use of the calculated distance information distribution is not limited to this.

[0203] For example, the CPU 1901 records, in association with the image data, data of a pair of parallax images captured by the imaging element 1904 and imaging conditions including at least the F-number and the KX value on a memory card 1909 or the like. Based on the data of the pair of recorded images and the imaging conditions, the CPU 1901 or the CPU of an external device to which each data is output generates and acquires a distance information distribution. Here, the distance information distribution to be acquired is a defocus amount distribution, and a blur map is generated by converting each defocus amount based on the F-number (or effective F-number) as the imaging condition and the conversion coefficient KX. The blur map can be used for quality evaluation related to blur in a captured image. In particular, in imaging for social infrastructure inspection, when inspecting deformations or the like on an inspection target surface, crack detection and crack width measurement cannot be correctly performed unless the evaluation is made using an image in which the inspection target surface is not blurred. Therefore, by referring to the defocus amount distribution (or blur map), for example, by performing measurement only for a non-blurred area (imaging range), a more accurate inspection can be performed. Further, for example, if it is determined that blur having a blur amount greater than or equal to a reference occurs in a captured image at a predetermined ratio or more, the CPU 1901 can notify the user that the captured image is unusable (deformation detection cannot be performed). As a notification method, an image or an icon can be displayed on the display unit 1905, or notification can be made using light, sound, vibration, or the like from another device. Further, the CPU 1901 can generate the above-described blur map, can generate an image in which each blur amount is simply visualized, and can display the image on the display unit. For example, by referring to the blur map, the user can re-capture an image or move the camera device 1900 manually or automatically.

[0204] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, the appended claims are added to disclose the scope of the present invention.

[0205] [Other Embodiments]

[0206] The object of the present invention can also be achieved as follows. More specifically, a storage medium storing program code of software describing a process for implementing the functions of the above-described embodiments is provided to a system or device. Then, a computer (or a CPU, an MPU, etc.) of the system or device reads and executes the program code stored in the storage medium.

[0207] In this case, the program code read from the storage medium itself implements the novel functions of the present invention. The storage medium storing the program code and the program constitute the present invention.

[0208] As a storage medium for providing program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, etc. can be given. Additionally, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, non-volatile memory card, or ROM, etc. can be used.

[0209] In addition, the functions of the above-described embodiments are implemented by making the program code read by the computer executable. Furthermore, there is also the following case: an OS (operating system) or the like operating on the computer performs part or all of the actual processing according to the instructions of the program code, and the functions of the above-described embodiments are implemented through these processes.

[0210] Furthermore, there is also the following case. First, the program code read from the storage medium is written into the memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, a CPU or the like included in the function expansion board or the function expansion unit performs part or all of the actual processing according to the instructions of the program code.

[0211] This application claims the benefit of Japanese Patent Application No. 2019-140818 filed on July 31, 2019, and Japanese Patent Application No. 2020-124031 filed on July 20, 2020, the entire contents of both of which are incorporated herein by reference.

Claims

1. An image processing apparatus, comprising: a first acquisition component configured to acquire imaging conditions related to an image captured by an imaging component; a second acquisition component configured to acquire a distance information distribution, the distance information distribution being a distribution of distance information corresponding to different regions in a region of the image captured by the imaging component; a generation component configured to generate a blur map based on the imaging conditions, the blur map being used for quality evaluation related to blur in a captured image; and a notification component configured to notify a user based on the blur map.

2. The image processing apparatus according to claim 1, wherein, The notification component displays the blur map on a display unit.

3. The image processing apparatus according to claim 1, wherein, The distance information distribution is information related to a distribution obtained by normalizing a distribution of a defocus amount of a subject using an F-number and a circle of confusion.

4. The image processing apparatus according to claim 1, wherein, In a case where it is determined that blur having a blur amount greater than or equal to a reference occurs in a captured image at a predetermined ratio, the notification component notifies the user that the captured image is unavailable.

5. The image processing apparatus according to claim 1, wherein, The distance information distribution is any one of the following information: information related to a distribution of a parallax amount of a subject, information related to a distribution of a defocus amount of a subject, and information related to a distribution of an actual distance from an imaging position to a subject.

6. The image processing apparatus according to claim 5, wherein, The information related to the distribution of the parallax amount of the subject is obtained from a pair of images having parallax.

7. The image processing apparatus according to claim 1, wherein, The imaging conditions are at least one of the following: posture information of the imaging component when capturing the image, a vanishing point in the image, a change in density of a texture in the image, and a determination result related to whether a structure having a known shape is included in the image.

8. The image processing apparatus according to any one of claims 1 to 7, wherein, The imaging conditions at least include an F-number and a transformation coefficient for transforming an image offset amount into a defocus amount.

9. An image processing method, comprising: a first acquisition step of acquiring imaging conditions related to an image captured by an imaging component; a second acquisition step of acquiring a distance information distribution, the distance information distribution being a distribution of distance information corresponding to different regions in a region of the image captured by the imaging component; a generation step of generating a blur map based on the imaging conditions, the blur map being used for quality evaluation related to blur in a captured image; and a notification step of notifying a user based on the blur map.

10. The image processing method according to claim 9, wherein, In the notification step, the blur map is displayed on a display unit.

11. The image processing method according to claim 9, wherein, In a case where it is determined that blur having a blur amount greater than or equal to a reference occurs in a captured image at a predetermined ratio, in the notification step, the user is notified that the captured image is unavailable.

12. The image processing method according to any one of claims 9 to 11, wherein, The imaging conditions at least include an F-number and a transformation coefficient for transforming an image offset amount into a defocus amount.

13. A computer-executable program product, on which a process for implementing functions of steps of the image processing method according to claim 9 is described.

Images