Image processing device

Abstract

Embodiments of the present invention relate to an image processing apparatus. The image processing apparatus according to the embodiments includes a storage unit, an image acquisition unit, a correction unit, and a distance acquisition unit. The storage unit stores a statistical model generated by learning the blurring that occurs in a first image affected by the aberration of an optical system and varies non-linearly according to the distance to a subject in the first image. The image acquisition unit acquires a second image affected by the aberration of the optical system. The correction unit performs color correction on the second image to reduce the number of colors represented in the second image. The distance acquisition unit inputs a third image obtained by performing color correction on the second image to the statistical model and acquires distance information indicating the distance to the subject in the third image.

Description

[0001] This application is based on Japanese Patent Application No. 2021-028830 (filing date: February 25, 2021), and claims priority therefrom. This application incorporates the entire contents of that application by reference thereto. Technical Field

[0002] Embodiments of the present invention relate to an image processing apparatus. Background Art

[0003] Generally, in order to obtain the distance to a subject, there is known a technique of using images captured by two imaging devices (cameras) or a stereo camera (multi-camera), but in recent years, a technique of obtaining the distance to a subject using an image captured by one imaging device (monocular camera) has been developed.

[0004] However, it is necessary to improve the accuracy of the distance obtained from an image captured by one imaging device and the environmental robustness when obtaining the distance from the image. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an image processing apparatus capable of improving the accuracy of the distance obtained from an image and the environmental robustness when obtaining the distance from the image.

[0006] The image processing apparatus according to the embodiment includes a storage unit, an image acquisition unit, a correction unit, and a distance acquisition unit. The storage unit stores a statistical model generated by learning the blur that occurs in a first image affected by the aberration of an optical system and varies non-linearly according to the distance to a subject in the first image. The image acquisition unit acquires a second image affected by the aberration of the optical system. The correction unit performs color correction on the second image to reduce the number of colors represented in the second image. The distance acquisition unit inputs a third image obtained by performing color correction on the second image to the statistical model and acquires distance information indicating the distance to the subject in the third image. Brief Description of the Drawings

[0007] Figure 1 is a diagram showing an example of the configuration of a distance measurement system including the image processing apparatus according to the first embodiment.

[0008] Figure 2 is a diagram showing an example of the system configuration of the image processing apparatus.

[0009] Figure 3 is a diagram for explaining an outline of the operation of the distance measurement system.

[0010] Figure 4It is a diagram showing the relationship between the distance to the subject in the case of using a single lens and the blur generated in the image due to chromatic aberration.

[0011] Figure 5 It is a diagram showing the relationship between the distance to the subject in the case of using an achromatic lens and the blur generated in the image due to chromatic aberration.

[0012] Figure 6 It is a diagram showing the relationship between the size of the aperture opening of the aperture mechanism included in the optical system of the imaging device and the PSF shape.

[0013] Figure 7 It is a diagram showing an example of the PSF shape generated in the image of each channel.

[0014] Figure 8 It is a diagram showing another example of the PSF shape generated in the image of each channel.

[0015] Figure 9 It is a diagram showing an example of the PSF shape generated at each position in the image.

[0016] Figure 10 It is a diagram for specifically explaining the position dependence of the PSF shape corresponding to the type of lens.

[0017] Figure 11 It is a diagram showing the relationship between the non-linearity of the PSF shape and the shape of the aperture opening of the aperture mechanism.

[0018] Figure 12 It is a diagram showing the outline of the operation of obtaining distance information.

[0019] Figure 13 It is a diagram for explaining the first method of estimating the distance based on the captured image.

[0020] Figure 14 It is a diagram showing an example of the information input to the statistical model in the first method.

[0021] Figure 15 It is a diagram for explaining the second method of estimating the distance based on the captured image.

[0022] Figure 16 It is a diagram showing an example of the information input to the statistical model in the second method.

[0023] Figure 17 It is a diagram for explaining the third method of estimating the distance based on the captured image.

[0024] Figure 18 It is a diagram for specifically explaining the distance to the subject estimated based on the captured image.

[0025] Figure 19 It is a flowchart showing an example of the processing sequence of an image processing device when obtaining distance information from a captured image.

[0026] Figure 20 It is a diagram showing an example of a preview screen.

[0027] Figure 21 It is a diagram showing an example of a preview screen when the distance measurement area selection button is pressed.

[0028] Figure 22 It is a diagram for explaining the environment-dependent PSF.

[0029] Figure 23 It is a diagram for explaining the outline of white balance correction.

[0030] Figure 24 It is a diagram for explaining the correction coefficient calculated for the pixels included in the captured image.

[0031] Figure 25 It is a diagram for explaining the unreliability calculated by the statistical model.

[0032] Figure 26 It is a diagram showing an example of a preview screen when the reliability confirmation mode is set.

[0033] Figure 27 It is a diagram showing an example of the learning process of the statistical model.

[0034] Figure 28 It is a flowchart showing an example of the processing sequence of the process for generating a statistical model.

[0035] Figure 29 It is a diagram showing an example of the structure of a distance measurement system of an image processing device including the second embodiment.

[0036] Figure 30 It is a diagram for explaining an example of the processing sequence of an image processing device when obtaining distance information from a captured image.

[0037] Figure 31 It is a diagram showing an example of the functional structure of a moving body including a distance measurement device.

[0038] Figure 32 It is a diagram for explaining the case where the moving body is an automobile.

[0039] Figure 33 It is a diagram for explaining the case where the moving body is a drone.

[0040] Figure 34 It is a diagram for explaining the case where the moving body is an autonomous mobile robot.

[0041] Figure 35 This is a diagram for explaining the case where the moving body is a robotic arm.

[0042] (Explanation of reference numerals in the drawings)

[0043] 1... Distance measurement system, 2... Imaging device (imaging unit), 3... Image processing device (image processing unit), 21... Lens, 22... Image sensor, 31... Statistical model storage unit, 32... Display processing unit, 33... Distance measurement area selection unit, 34... Image acquisition unit, 35... Image correction unit, 36... Distance acquisition unit, 37... Reliability confirmation mode setting unit, 38... Output unit, 39... Evaluation unit, 221... First sensor, 222... Second sensor, 223... Third sensor, 301... CPU, 302... Non-volatile memory, 303... RAM, 303A... Image processing program, 304... Communication device, 305... Input device, 306... Display device, 307... Bus. Detailed implementation manners

[0044] Hereinafter, each implementation manner will be described with reference to the drawings.

[0045] (First implementation manner)

[0046] First, the first implementation manner will be described. Figure 1 This shows an example of the structure of a distance measurement system including the image processing device of this implementation manner. Figure 1 The shown distance measurement system 1 is used for a captured image, and uses the captured image to obtain (measure) the distance from the imaging location to the subject.

[0047] As Figure 1 shown, the distance measurement system 1 includes an imaging device 2 and an image processing device 3. In this implementation manner, it is described that the distance measurement system 1 includes the imaging device 2 and the image processing device 3 as independent devices, but this distance measurement system 1 can also be implemented as one device (hereinafter, referred to as a distance measurement device) in which the imaging device 2 functions as an imaging unit and the image processing device 3 functions as an image processing unit. In the case of the distance measurement system 1, for example, a digital camera or the like can be used as the imaging device 2, and a personal computer, a smart phone, or a tablet computer or the like can be used as the image processing device 3. In this case, the image processing device 3 can also operate as a server device that executes cloud computing services, for example. On the other hand, in the case of the distance measurement device 1, a digital camera, a smart phone, and a tablet computer or the like can be used as the distance measurement device 1.

[0048] The imaging device 2 is used to capture various images. The imaging device 2 includes a lens 21 and an image sensor 22. The lens 21 and the image sensor 22 correspond to the optical system (monocular camera) of the imaging device 2. In addition, in the present embodiment, the lens 21 is combined with a mechanism for controlling the focus position by adjusting the position of the lens 21, a lens drive circuit, etc., a diaphragm mechanism and a diaphragm control circuit having an aperture for adjusting the amount of light (light incident amount) taken into the optical system of the imaging device 2, and a control circuit on which a memory for pre-storing information related to the lens 21 (hereinafter referred to as lens information) is mounted (not shown) to form a lens unit.

[0049] In addition, in the present embodiment, the imaging device 2 may be configured to be able to manually replace the lens 21 (lens unit) with another lens. In this case, the user can, for example, mount one of a plurality of types of lenses such as a standard lens, a telephoto lens, and a wide-angle lens on the imaging device 2 for use. In addition, when the lens is replaced, the focal length and the F value (aperture value) change, and an image corresponding to the lens used in the imaging device 2 can be captured.

[0050] In the present embodiment, the focal length refers to the distance from the lens to the position where the light converges when the light is incident on the lens parallelly. In addition, the F value refers to a value obtained by quantifying the amount of light taken into the imaging device 2 according to the diaphragm mechanism. In addition, the F value indicates that as the value becomes smaller, the amount of light taken into the imaging device 2 becomes larger (that is, the size of the aperture becomes larger).

[0051] The light reflected by the subject is incident on the lens 21. The light incident on the lens 21 passes through the lens 21. The light that has passed through the lens 21 reaches the image sensor 22 and is received (detected) by the image sensor 22. The image sensor 22 generates an image composed of a plurality of pixels by converting the received light (photoelectric conversion) into an electrical signal.

[0052] In addition, the image sensor 22 is implemented by, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The image sensor 22 includes, for example, a first sensor (R sensor) 221 that detects light in a red (R) wavelength band, a second sensor (G sensor) 222 that detects light in a green (G) wavelength band, and a third sensor (B sensor) 223 that detects light in a blue (B) wavelength band. The image sensor 22 can receive light in the corresponding wavelength bands through the first to third sensors 221 to 223 and generate sensor images (R image, G image, and B image) corresponding to the respective wavelength bands (color components). That is, the image captured by the imaging device 2 is a color image (RGB image), and the R image, G image, and B image are included in this image.

[0053] In addition, in the present embodiment, the description is made assuming that the image sensor 22 includes the first to third sensors 221 to 223, but the image sensor 22 only needs to be configured to include at least one of the first to third sensors 221 to 223. In addition, the image sensor 22 may be configured to include a sensor for generating, for example, a monochrome image instead of the first to third sensors 221 to 223.

[0054] In the present embodiment, the image generated based on the light transmitted through the lens 21 is an image affected by the aberration of the optical system (including the lens 21) and includes blur caused by this aberration. In addition, the details of the blur generated in the image will be described later.

[0055] The image processing device 3 includes a statistical model storage unit 31, a display processing unit 32, a ranging area selection unit 33, an image acquisition unit 34, an image correction unit 35, a distance acquisition unit 36, a reliability confirmation mode setting unit 37, and an output unit 38 as functional structures.

[0056] A statistical model used to obtain the distance of the subject from the image captured by the imaging device 2 is stored in the statistical model storage unit 31. The statistical model stored in the statistical model storage unit 31 is generated by learning the blur that occurs in the image affected by the aberration of the optical system and that varies non-linearly according to the distance to the subject in the image. The statistical model storage unit 31 can store multiple statistical models.

[0057] In addition, the statistical model is set to be generated by applying various known machine learning algorithms such as neural networks or random forests. Additionally, the neural networks applicable in the present embodiment may include, for example, convolutional neural networks (CNN: Convolutional Neural Network), fully coupled neural networks, and recurrent neural networks.

[0058] The display processing unit 32 sequentially displays a plurality of images continuously captured by the above-described imaging device 2 as preview displays.

[0059] The ranging area selection unit 33 selects an area in the image displayed by the display processing unit 32 (i.e., the image captured by the imaging device 2). In addition, in the present embodiment, the distance (information) to the subject included in the area selected by the ranging area selection unit 33 is obtained. Additionally, the area selected by the ranging area selection unit 33 is determined based on, for example, a user operation.

[0060] That is, in the present embodiment, the above-described preview display means that an image is displayed in advance for the user to confirm the subject that becomes the object for which the distance is obtained.

[0061] The image acquisition unit 34 acquires the image when the ranging area selection unit 33 selects an area in the image displayed by the display processing unit 32.

[0062] The image correction unit 35 corrects the color of the image acquired by the image acquisition unit 34. In this case, the image correction unit 35 corrects the image acquired by the image acquisition unit 34 to reduce the number of colors represented in the image (the color variation can be reduced).

[0063] The distance acquisition unit 36 acquires distance information indicating the distance to the subject in the image (the area selected by the ranging area selection unit 33) that has been color-corrected by the image correction unit 35. In this case, as described later, the distance acquisition unit 36 acquires the distance information by inputting the image that has been color-corrected by the image correction unit 35 into the statistical model stored in the statistical model storage unit 31.

[0064] The reliability confirmation mode setting unit 37 sets the reliability confirmation mode according to a user operation. The reliability confirmation mode is a mode for the user to confirm the reliability (the degree of unreliability) with respect to the distance to the subject. When the reliability confirmation mode setting unit 37 sets the reliability confirmation mode, the user can, for example, confirm the reliability with respect to the distance indicated by the distance information acquired by the distance acquisition unit 36 in the above-described preview screen.

[0065] The output unit 38 acquires the distance information acquired by the distance acquisition unit 36. In addition, the distance information can be displayed via the display processing unit 32, or can be output to the outside of the image processing apparatus 3.

[0066] Figure 2 Indicates Figure 1 An example of the system configuration of the image processing apparatus 3 shown. As Figure 2 shown, the image processing apparatus 3 includes a CPU 301, a non-volatile memory 302, a RAM 303, a communication device 304, an input device 305, a display device 306, and the like. In addition, the image processing apparatus 3 has a bus 307 that connects the CPU 301, the non-volatile memory 302, the RAM 303, the communication device 304, the input device 305, and the display device 306 to each other.

[0067] The CPU 301 is a processor for controlling the operations of various components within the image processing apparatus 3. The CPU 301 can be either a single processor or composed of multiple processors. The CPU 301 executes various programs loaded from the non-volatile memory 302 to the RAM 303. These programs include an operating system (OS) and various application programs. The application programs include an image processing program 303A that uses the image captured by the imaging device 2 to acquire the distance from the imaging device 2 to the subject in the image.

[0068] The non-volatile memory 302 is a storage medium used as an auxiliary storage device. The RAM 303 is a storage medium used as a main storage device. In Figure 2 only the non-volatile memory 302 and the RAM 303 are shown, but the image processing apparatus 3 may also include other storage devices such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

[0069] In addition, in the present embodiment, Figure 1 the statistical model storage unit 31 shown is implemented, for example, by the non-volatile memory 302 or other storage devices.

[0070] In addition, in the present embodiment, Figure 1Part or all of the display processing unit 32, ranging area selection unit 33, image acquisition unit 34, image correction unit 35, distance acquisition unit 36, reliability confirmation mode setting unit 37, and output unit 38 shown are implemented by software, that is, by causing the CPU 301 (i.e., the computer of the image processing device 3) to execute the image processing program 303A. The image processing program 303A can be distributed by being stored in a computer-readable storage medium or downloaded to the image processing device 3 via a network. In addition, part or all of these units 32 to 38 can be implemented by hardware such as an IC (Integrated Circuit) or by a combination of software and hardware.

[0071] The communication device 304 is a device configured to perform wired communication or wireless communication. The communication device 304 performs communication with an external device via a network, etc. The external device includes the imaging device 2. In this case, the image processing device 3 receives an image from the imaging device 2 via the communication device 304.

[0072] The input device 305 includes, for example, a mouse or a keyboard, etc. The display device 306 includes, for example, a liquid crystal display (LCD: Liquid Crystal Display), etc. In addition, the input device 305 and the display device 306 can also be integrally configured, such as a touch screen display, for example.

[0073] Next, with reference to Figure 3 , an outline of the operation of the ranging system 1 in the present embodiment will be described.

[0074] In the ranging system 1, the imaging device 2 (image sensor 22) generates an image affected by the aberration of the optical system (lens 21) as described above.

[0075] The image processing device 3 acquires the image generated by the imaging device 2 and inputs the image to the statistical model stored in the statistical model storage unit 31. Although not shown in Figure 3 , in the present embodiment, the image input to the statistical model is an image after color correction. Details of the color correction performed on the image will be described later.

[0076] The image processing device 3 (distance acquisition unit 36) acquires distance information indicating the distance to the subject in the image output from the statistical model into which the image has been input.

[0077] In this way, in the present embodiment, distance information can be acquired from the image captured by the imaging device 2 using the statistical model.

[0078] Here, in the present embodiment, in the image captured by the imaging device 2, blurring caused by aberration (lens aberration) of the optical system of the imaging device 2 occurs as described above.

[0079] Hereinafter, the blurring that occurs in the image captured by the imaging device 2 will be described. First, chromatic aberration among the blurring caused by aberration of the optical system of the imaging device 2 will be described.

[0080] Figure 4 The relationship between the distance to the subject and the blurring generated in the image due to chromatic aberration is shown. In the following description, the position focused on in the imaging device 2 is referred to as the focal position.

[0081] Since the refractive index of light when passing through the lens 21 with aberration is different for each wavelength band, for example, when the position of the subject deviates from the focal position, the light of each wavelength band does not converge at one point but reaches different points. This appears as chromatic aberration (blurring) in the image.

[0082] Figure 4 The upper part of shows the case where the position of the subject relative to the imaging device 2 (image sensor 22) is farther than the focal position (that is, the position of the subject is located on the inner side of the focal position).

[0083] In this case, regarding the light 401 of the red wavelength band, an image including a relatively small blur b is generated in the image sensor 22 (first sensor 221). R On the other hand, regarding the light 402 of the blue wavelength band, an image including a relatively large blur b is generated in the image sensor 22 (third sensor 223). B In addition, regarding the light 403 of the green wavelength band, an image including a blur of an intermediate size between blur b and blur b is generated. R and blur b B Therefore, in the image captured in a state where the position of the subject is farther than the focal position, a blue blur is observed on the outside of the subject in the image.

[0084] On the other hand, Figure 4 the lower part of shows the case where the position of the subject relative to the imaging device 2 (image sensor 22) is closer than the focal position (that is, the position of the subject is located in a position closer to the front than the focal position).

[0085] In this case, regarding the light 401 of the red wavelength band, an image including a relatively large blur b is generated in the image sensor 22 (first sensor 221). R On the other hand, regarding the light 402 of the blue wavelength band, an image including a relatively small blur b is generated in the image sensor 22 (third sensor 223). Bimage. In addition, regarding the light 403 in the green wavelength band, an image including a blur b R and the blur b B of medium size is generated. Therefore, in an image captured in a state where the position of the subject is closer to the imaging device than the focal position, a red blur is observed outside the subject in the image.

[0086] Here, Figure 4 An example in which the lens 21 is a simple single lens is shown. Generally, in the imaging device 2, a lens that has been subjected to chromatic aberration correction (hereinafter referred to as an achromatic lens) is sometimes used. In addition, an achromatic lens is a lens formed by combining a convex lens with low dispersion and a concave lens with high dispersion, and is the lens with the fewest lens elements as a lens for correcting chromatic aberration.

[0087] Figure 5 The relationship between the distance to the subject and the blur generated in the image due to chromatic aberration is shown in the case where the above-mentioned achromatic lens is used as the lens 21. In the achromatic lens, a design is made such that the focal positions of the blue wavelength and the red wavelength coincide, but chromatic aberration cannot be completely removed. Therefore, when the position of the subject is farther than the focal position, as Figure 5 shown in the upper part, a green blur is generated, and when the position of the subject is closer than the focal position, as Figure 5 shown in the lower part, a purple blur is generated.

[0088] In addition, Figure 4 and Figure 5 the middle part shows the case where the position of the subject coincides with the focal position with respect to the imaging device 2 (image sensor 22). In this case, an image with less blur is generated in the image sensor 22 (first to third sensors 221 to 223).

[0089] Here, the optical system (lens unit) of the imaging device 2 is provided with an aperture mechanism as described above, but the shape of the blur generated in the image captured by the imaging device 2 varies depending on the size of the opening of the aperture mechanism. In addition, the shape of the blur is called the PSF (Point Spread Function) shape, and shows the light diffusion distribution generated when a point light source is imaged.

[0090] Figure 6 The upper part of shows the PSF shape generated at the center of the image captured by the imaging device 2 (optical system) with a focal length of 50 mm, a focal position of 1500 mm, and an F value (aperture) of F1.8, in the order from near to far from the imaging device 2 for the position of the subject. Figure 6The lower part of [Figure] shows, from left to right in the order of the distance from the subject to the imaging device 2 increasing from near to far, the shape of the PSF generated in the central part of the image captured by the imaging device 2 (optical system) with a lens having a focal length of 50 mm, where the focal position is set to 1500 mm and the F value (aperture) is set to F4. Additionally, Figure 6 The center of the upper and lower parts of [Figure] shows the shape of the PSF when the position of the subject coincides with the focal position.

[0091] Figure 6 The PSF shapes shown at the corresponding positions in the upper and lower parts of [Figure] are the PSF shapes when the position of the subject relative to the imaging device 2 is the same. However, even when the position of the subject is the same, the shape of the PSF in the upper part (the PSF shape generated in the image captured with the F value set to F1.8) and the shape of the PSF in the lower part (the PSF shape generated in the image captured with the F value set to F4) are different.

[0092] And, as Figure 6 shown by the leftmost and rightmost PSF shapes in [Figure], even when the distance from the position of the subject to the focal position is of the same degree, for example, the PSF shape is different when the position of the subject is closer to the focal position and when the position of the subject is farther from the focal position.

[0093] In addition, as described above, the phenomenon that the PSF shape varies according to the size of the opening of the aperture mechanism and the position of the subject relative to the imaging device 2 also occurs in each channel (RGB image, R image, G image, and B image). Figure 7 The PSF shapes generated in the images of each channel captured by the imaging device 2 with a lens having a focal length of 50 mm, where the focal position is set to 1500 mm and the F value is set to F1.8, are shown by dividing them into the case where the position of the subject is closer to the focal position (in the foreground) and the case where the position of the subject is farther from the focal position (in the background). Figure 8 The PSF shapes generated in the images of each channel captured by the imaging device 2 with a lens having a focal length of 50 mm, where the focal position is set to 1500 mm and the F value is set to F4, are shown by dividing them into the case where the position of the subject is closer to the focal position and the case where the position of the subject is farther from the focal position.

[0094] Furthermore, the PSF shape generated in the image captured by the imaging device 2 also varies according to the position in the image.

[0095] Figure 9The upper part of [description] shows the PSF shapes generated at each position in the image captured by the imaging device 2 with a lens having a focal length of 50 mm, when the focal position is set to 1500 mm and the F value is set to F1.8, divided into the cases where the position of the subject is closer than the focal position and the case where the position of the subject is farther than the focal position.

[0096] Figure 9 The middle part of [description] shows the PSF shapes generated at each position in the image captured by the imaging device 2 with a lens having a focal length of 50 mm, when the focal position is divided into 1500 mm and the F value is set to F4, divided into the cases where the position of the subject is closer than the focal position and the case where the position of the subject is farther than the focal position.

[0097] As Figure 9 shown in the upper and middle parts of [description], near the end of the image captured by the imaging device 2 (especially near the upper left isometric corner), for example, a PSF shape different from the PSF shape near the center of the image can be observed.

[0098] In addition, Figure 9 The lower part of [description] shows the PSF shapes generated at each position in the image captured by the imaging device 2 with a lens having a focal length of 105 mm, when the focal position is set to 1500 mm and the F value is set to F4, divided into the cases where the position of the subject is closer than the focal position and the case where the position of the subject is farther than the focal position.

[0099] The above Figure 9 The upper and middle parts of [description] show the PSF shapes generated in the image captured using the same lens. However, as Figure 9 shown in the lower part of [description], when using lenses with different focal lengths, different PSF shapes corresponding to those lenses are observed (PSF shapes different from Figure 9 the upper and middle parts of [description]).

[0100] Next, with reference to Figure 10 , the position dependence of the PSF shape (lens aberration) corresponding to the type of lens used in the optical system of the above imaging device 2 will be specifically described. Figure 10 The PSF shapes generated near the center (center of the screen) and near the end (end of the screen) of the images captured using multiple lenses with different focal lengths are divided into the cases where the position of the subject is closer than the focal position and the case where the position of the subject is farther than the focal position for representation.

[0101] As Figure 10As shown, the shape of the PSF generated near the center of the image is approximately circular and the same even when the types of lenses are different. However, the shape of the PSF generated near the edge of the image is different from the shape of the PSF generated near the center of the image, and the characteristics (features) are different according to the type of lens. In addition, when the position of the subject described in the above Figure 5 is closer to the imaging device than the focal position, purple blur is generated near the edge of the PSF. When the position of the subject is farther from the focal position than the focal position, green blur is generated near the edge of the PSF. This is common even when the types of lenses are different.

[0102] In addition, in Figure 10 , two examples (#1 and #2) are shown for a lens with a focal length of 50 mm, which means that the focal lengths are 50 mm and the same, but the manufacturers of the lenses are different (i.e., they are different products). The same applies to lenses with a focal length of 85 mm.

[0103] In the image processing device 3 (range finding system 1) of the present embodiment, a statistical model generated by focusing on the blur (color, size, and shape) that changes non-linearly according to the distance to the subject in the image (i.e., the position of the subject relative to the imaging device 2) is used to obtain the distance to the subject from the image.

[0104] In addition, the blur that changes non-linearly according to the distance to the subject in the present embodiment includes: the blur generated according to the chromatic aberration of the optical system of the imaging device 2 described in the above Figure 4 and Figure 5 , the blur generated according to the size of the opening (i.e., the F value) of the aperture mechanism that adjusts the amount of light incident on the optical system of the imaging device 2 described in Figures 6 - 8 , and the blur that changes according to the position in the image captured by the imaging device 2 described in Figure 9 and Figure 10 , etc.

[0105] In addition, the shape of the PSF also differs according to the shape of the opening of the aperture mechanism. Here, Figure 11 represents the relationship between the non-linearity (asymmetry) of the PSF shape and the shape of the opening of the aperture mechanism. The non-linearity of the above PSF shape is likely to occur when the shape of the opening of the aperture mechanism is other than a circle. In particular, the non-linearity of the PSF shape is more likely to occur when the shape of the opening is an odd-sided polygon or an even-sided polygon asymmetrically arranged with respect to the horizontal or vertical axis of the image sensor 22.

[0106] In addition, in the above Figure 9In this case, the distance to the subject in the image and the PSF shape (i.e., the blur with distance dependence and position dependence) depending on the position in the image are represented on the image, and such an image is respectively referred to as an aberration map. The aberration map is information in the form of a map representing the distribution of blur, which is the blur generated in the image affected by the aberration of the optical system and is the blur depending on the distance to the subject in the image and the position in the image. In other words, the aberration map is information representing the distance dependence and position dependence of the blur (i.e., the aberration of the optical system) in the image generated due to aberration. Such an aberration map can be utilized when estimating the distance to the subject in the image (i.e., obtaining distance information representing the distance).

[0107] Figure 12 This shows an outline of the operation of obtaining distance information in the present embodiment. In the following description, the image captured by the imaging device 2 for obtaining distance information (distance to the subject) is referred to as a captured image.

[0108] In Figure 12 The blur (blur information) 502 generated in the captured image 501 shown becomes a physical clue related to the distance to the subject 503 being imaged. Specifically, the color of the blur, the size and shape of the PSF become clues related to the distance to the subject 503 being imaged.

[0109] In the image processing device 3 (distance acquisition unit 36) of the present embodiment, the distance 504 to the subject 503 is estimated by analyzing (resolving) the blur 502 generated in the captured image 501 as a physical clue using a statistical model.

[0110] Hereinafter, an example of the method of estimating the distance from the captured image using a statistical model in the present embodiment will be described. Here, the first to third methods will be described.

[0111] First, with reference to Figure 13 The first method will be described. In the first method, the distance acquisition unit 36 extracts a local region (image patch) 501a from the captured image 501.

[0112] In this case, for example, the entire region (or a specified region) of the captured image 501 can be divided into a matrix, and the divided partial regions can be sequentially extracted as the local region 501a, or the captured image 501 can be recognized, and the local region 501a can be extracted in a manner that includes the region where the subject (image) being imaged is detected. In addition, the local region 501a can also partially overlap with other local regions 501a.

[0113] The distance acquisition unit 36 inputs information related to each extracted partial region 501a (information of the captured image 501) into the statistical model, thereby estimating the distance 504 of the subject in the partial region 501a.

[0114] The statistical model input with the information related to the partial region 501a estimates the distance for each pixel constituting the partial region 501a.

[0115] Here, for example, when a specific pixel belongs to both the first partial region 501a and the second partial region 501a (i.e., the region including the pixel overlaps between the first partial region 501a and the second partial region 501a), there is a case where the distance at which the pixel is estimated to belong to the pixel of the first partial region 501a is different from the distance at which the pixel is estimated to belong to the pixel of the second partial region 501a.

[0116] Therefore, for example, when a plurality of partially overlapping partial regions 501a are extracted as described above, the distance of the pixels constituting the overlapping region of the plurality of partial regions 501a can be set, for example, as the average of the distance estimated for a part of the region (pixels) of one of the overlapping partial regions 501a and the distance estimated for a part of the region (pixels) of the other partial region 501a. In addition, when three or more partially overlapping partial regions 501a are extracted, the distance of the pixels constituting the overlapping region of the three or more partial regions 501a can also be determined by a majority vote based on the distances estimated for each part of the three or more overlapping partial regions 501a.

[0117] Figure 14 An example of the information related to the partial region 501a input into the statistical model in the above first method is shown.

[0118] The distance acquisition unit 36 generates gradient data (gradient data of the R image, gradient data of the G image, and gradient data of the B image) of the partial region 501a extracted from the captured image 501 for each of the R image, G image, and B image included in the captured image 501. The gradient data generated by the distance acquisition unit 36 in this way is input into the statistical model.

[0119] In addition, the gradient data represents the difference (difference value) of the pixel values between each pixel and the adjacent pixel. For example, when the partial region 501a is extracted as a rectangular region of n pixels (in the X-axis direction) × m pixels (in the Y-axis direction), gradient data is generated by arranging, for example, the difference values calculated between each pixel in the partial region 501a and the adjacent pixel on the right in a matrix of n rows × m columns.

[0120] The statistical model uses the gradient data of the R image, the gradient data of the G image, and the gradient data of the B image to estimate the distance based on the blur generated in each of these images. In Figure 14 , the case where the gradient data of the R image, the G image, and the B image are each input to the statistical model is shown, but it may also be a structure in which the gradient data of the captured image 501 (RGB image) is input to the statistical model.

[0121] Next, a description will be given with reference to Figure 15 The second method will be described. In the second method, as the information related to the local area 501a in the first method, the gradient data of each such local area (image patch) 501a and the position information of the local area 501a in the captured image 501 are input to the statistical model.

[0122] The position information 501b can be, for example, information indicating the center point of the local area 501a, or information indicating a predetermined side such as the upper left side. In addition, as the position information 501b, the position information of each pixel constituting the local area (image patch) 501a on the captured image 501 can also be used.

[0123] As described above, by further inputting the position information 501b to the statistical model, for example, it is possible to estimate the distance 504 while considering the difference between the blur of the subject image formed by the light passing through the central part of the transmission lens 21 and the blur of the subject image formed by the light passing through the end part of the lens 21.

[0124] That is, according to this second method, it is possible to more reliably estimate the distance based on the captured image 501 based on the correlation between blur, distance, and position on the image.

[0125] Figure 16 An example of the information related to the local area 501a input to the statistical model in the above second method is shown.

[0126] For example, when extracting a rectangular area of n pixels (in the X-axis direction) × m pixels (in the Y-axis direction) as the local area 501a, the distance acquisition unit 36 acquires the X coordinate value (X coordinate data) on the captured image 501 corresponding to, for example, the center point of the local area 501a and the Y coordinate value (Y coordinate data) on the captured image 501 corresponding to, for example, the center point of the local area 501a.

[0127] In the second method, the X coordinate data and the Y coordinate data thus acquired by the distance acquisition unit 36 are input to the statistical model together with the gradient data of the above-mentioned R image, G image, and B image.

[0128] Furthermore, a description will be given with reference to Figure 17, the third method is described. In the third method, extraction of the local region (image patch) 501a from the captured image 501 as in the first and second methods described above is not performed. In the third method, the distance acquisition unit 36 inputs information (gradient data of the R image, G image, and B image) related to the entire region (or a specified region) of the captured image 501 to the statistical model, for example.

[0129] Compared with the first and second methods of estimating the distance 504 for each local region 501a, there is a possibility that the uncertainty of the estimation based on the statistical model becomes higher in the third method, but the load on the distance acquisition unit 36 can be reduced.

[0130] In the following description, for convenience, the information input to the statistical model in the above first to third methods is referred to as information related to the image.

[0131] Here, with reference to Figure 18 , the distance to the object to be captured estimated from the captured image is specifically described.

[0132] In Figure 18 , the size of the blur generated when the object is closer to the focal position (in the front) is represented by a negative value on the X-axis, and the size of the blur generated when the object is farther from the focal position (in the back) is represented by a positive value on the X-axis. That is, in Figure 18 , the color and size of the blur are represented by positive and negative values.

[0133] In Figure 18 , it is shown that in any case where the position of the object is closer to the focal position and where the position of the object is farther from the focal position, the absolute value of the size (pixels) of the blur increases as the object moves farther away from the focal position.

[0134] In Figure 18 In the example shown, it is assumed that the focal position in the optical system that captured the image is about 1500 mm. In this case, for example, a blur of about -4.8 pixels corresponds to a distance of about 1000 mm from the optical system, a blur of 0 pixels corresponds to a distance of 1500 mm from the optical system, and a blur of about 4.8 pixels corresponds to a distance of about 750 mm from the optical system.

[0135] Here, for convenience, the case of representing the size (pixels) of the blur on the X-axis has been described, but as described in the above Figures 6 - 10 , regarding the shape (PSF shape) of the blur generated in the image, it is different between the case where the object is closer to the focal position and the case where the object is farther from the focal position, and also varies depending on the position in the image. Therefore, in Figure 18Among them, the value shown on the X-axis actually reflects the value of the blur shape (PSF shape).

[0136] The distance to the subject described above has a correlation with the color, size, and shape of the blur as shown by, for example, Figure 18 the line segment d1, so the estimated distance and the color, size, and shape of the estimated blur (the blur value expressed with positive and negative signs) are synonymous.

[0137] In addition, compared with the case of directly estimating the distance by the statistical model, when the statistical model estimates the color, size, and shape of the blur, the accuracy of the estimation based on the statistical model can be improved more. In this case, the statistical model is set, for example, for each local area of n pixels (in the X-axis direction) × m pixels (in the Y-axis direction), the information related to the local area (image) is input into the statistical model, and thus the distance obtained by arranging the color, size, and shape of the blur (the expressed blur value) estimated for each pixel constituting the local area in n rows × m columns is output.

[0138] Next, with reference to Figure 19 the flowchart of, an example of the processing sequence of the image processing device 3 when obtaining distance information from the captured image will be described.

[0139] First, the imaging device 2 (image sensor 22) generates a plurality of captured images including the subject by continuously imaging the subject. In addition, when the focal position of the imaging device 2 is fixed, the light transmitted through the lens 21 has a response shape of a point spread function (PSF) or a point image distribution function that varies according to the distance to the subject. By detecting such light with the image sensor 22, a captured image affected by the aberration of the optical system of the imaging device 2 is generated.

[0140] The display processing unit 32 sequentially displays the plurality of captured images generated in the imaging device 2 as described above on, for example, the display device 306, thereby performing preview display (step S1).

[0141] Here, Figure 20 an example of the screen of the image processing device 3 (display device 306) when performing preview display in step S1 is shown (hereinafter, referred to as the preview screen).

[0142] As Figure 20 shown, an image display area 306b, a ranging area selection button 306c, and a reliability confirmation mode button 306d are provided on the preview screen 306a.

[0143] The image display area 306b is an area for sequentially displaying the plurality of captured images generated in the imaging device 2 described above.

[0144] The distance measurement area selection button 306c is a button for indicating the selection of an area in the captured image displayed in the image display area 306b, that is, the area for obtaining distance information (hereinafter referred to as the distance measurement area).

[0145] The reliability confirmation mode button 306d is a button for indicating the setting of the reliability confirmation mode.

[0146] When the user presses (designates) the distance measurement area selection button 306c while the captured image is being displayed in the above-mentioned image display area 306b, Figure 20 the preview screen 306a shown migrates to Figure 21 the preview screen 306e shown, and the user can perform an operation of designating the distance measurement area on the image display area 306b provided in this preview screen 306e.

[0147] Specifically, for example, a rectangular frame 306f is displayed on the image display area 306b provided in the preview screen 306e, and the user can designate the area included in the frame 306f as the distance measurement area by operating the input device 305 to change the size of the frame 306f. Thus, the user can designate an area including the subject for which distance information is to be obtained as the distance measurement area.

[0148] Returning again to Figure 19 , the distance measurement area selection unit 33 determines whether the distance measurement area is selected based on the above-mentioned user operation (step S2). In step S2, when the user presses the distance measurement area selection button provided on the above-mentioned preview screen, it is determined that the distance measurement area is selected.

[0149] When it is determined that the distance measurement area is not selected (No in step S2), the process returns to step S1 and the processing is repeated.

[0150] On the other hand, when it is determined that the distance measurement area is selected (Yes in step S2), the distance measurement area selection unit 33 selects the area designated by the above-mentioned user operation (for example, the operation of changing the size of the frame 306f) as the distance measurement area. In addition, when the processing of step S2 is executed, the distance measurement area selection unit 33 obtains information indicating the distance measurement area selected in this step S2 (hereinafter referred to as distance measurement area information). The distance measurement area information includes information (such as coordinate values, etc.) that can determine the distance measurement area represented by the distance measurement area information on the captured image.

[0151] In addition, although it has been described above that the area designated by the user is selected as the ranging area, for example, it may be configured such that when the user performs an operation of designating a position on the image display area 306b (captured image), semantic segmentation is performed to classify each pixel around the pixel corresponding to the designated position based on the pixel value or the like of the pixel. In this case, it is possible to automatically detect the area including the specific subject corresponding to the position designated by the user, and select the detected area as the ranging area. That is, in the present embodiment, it may also be configured such that the area including the subject detected based on the position in the captured image designated by the user's operation is selected as the ranging area.

[0152] In addition, for example, when the imaging device 2 (ranging device 1) has an autofocus (AF) function for focusing on a specific subject, the area including the subject extracted by the autofocus function may be selected as the ranging area. The ranging area may also be selected according to other image processing performed on the captured image.

[0153] In addition, the ranging area selected in step S3 may be one or more. In addition, the ranging area may not be a partial area of the captured image, but the entire area of the captured image. In addition, the shape of the ranging area may not be a rectangular shape, and may be, for example, a circular shape.

[0154] Next, the image acquisition unit 34 acquires the captured image in which the ranging area is selected in step S3 (step S4). Hereinafter, the captured image acquired in step S4 is referred to as the target captured image.

[0155] When the process of step S4 is executed, the image correction unit 35 performs color correction on the target captured image (step S5).

[0156] Hereinafter, the color correction performed by the image correction unit 35 will be specifically described. In addition, in the present embodiment, it is described that white balance correction is performed as the color correction.

[0157] First, the target captured image in the present embodiment is an image affected by the aberration of the optical system as described above, but it is known that the color represented in the target captured image varies (deviates) according to the influence of the color temperature of the light source. In addition, the color temperature is a scale representing the color of the light emitted by the light source, and regarding this color temperature, for example, it is different between sunlight that is the light source outdoors and fluorescent lamps that are the light sources indoors. That is, it can be said that the color (represented in the target captured image) is affected by the environment around the imaging device 2. Furthermore, the color represented in the captured image also varies, for example, according to the characteristics (sensor RGB characteristics) of the image sensor 22 (first to third sensors 221 to 223).

[0158] Specifically, Figure 22 The PSF (point spread function) of the white dots shown is expressed as the lens characteristic h(λ) × the light S(λ) reflected from the object surface, and the pixel values of RGB (that is, the R image, G image, and B image included in the color image) are the same. However, as Figure 22 shown, the PSF of the white dots is affected by the light source E(λ) and the sensor RGB characteristic (sensor sensitivity) q c (λ). Therefore, in the object captured image, a PSF that depends on the environment represented by the lens characteristic h(λ) × the light S(λ) reflected from the object surface × the light source E(λ) × the sensor RGB characteristic q c (λ) is observed. In addition, Figure 22 the light source E(λ) shown represents the spectral characteristic of the wavelength (λ) of the light from the light source, Figure 22 the sensor RGB characteristic q c (λ) represents the ratio of the maximum value of the pixel values in the RGB image.

[0159] That is, if there is no influence of the light source and the sensor RGB characteristic, the result of the PSF of the white dots with the same RGB pixel values should be obtained. In the object captured image, a PSF with a deviation in RGB pixel values (that is, a PSF that depends on the environment) is observed.

[0160] As described above, in the object captured image, the color deviates due to the color temperature of the light source and the sensor RGB characteristic, and a PSF that depends on the environment is observed. However, when obtaining the distance (information) from such an object captured image, there is a case where the accuracy of the distance is low. Specifically, for example, if it is assumed that even the same color is displayed (expressed) differently depending on the color temperature of the light source and the sensor RGB characteristic, it is difficult to obtain (estimate) a distance with high accuracy. In such a case, in order to improve the accuracy of the distance, for example, considering generating a statistical model of a blur that has learned all the colors (color variations) considering the influence brought by the color temperature of the light source and the sensor RGB characteristic, but such learning requires a huge amount of labor, so it is difficult to improve the environmental robustness when obtaining the distance from the image.

[0161] Therefore, in the present embodiment, as Figure 23 shown, a correction coefficient for the object captured image (PSF that depends on the environment) is calculated, and the above-mentioned white PSF is restored using this correction coefficient. Thus, the influence brought by the color temperature of the light source, etc. is absorbed, and the number of colors (color variations) expressed in the object captured image is reduced. In addition, the correction coefficient is, for example, a gain value (gain(R, G, B)) that is multiplied by the R image, G image, and B image included in the color image in order to make a color such as gray expressed in the color image white.

[0162] In addition, the above white balance correction is performed for each of the predetermined regions included in the object captured image. In this case, for each pixel included in the object captured image (range finding region), a correction coefficient is calculated based on the pixel values of the multiple pixels included in a region of a predetermined size (hereinafter referred to as a unit region) that includes this pixel.

[0163] Here, Figure 24 the correction coefficient calculated for the pixel P included in the object captured image 501 as shown i will be described.

[0164] First, the image correction unit 35 determines a unit region A of a predetermined size that includes the pixel P i . In this case, for example, a rectangular unit region A i with the pixel P i at the center is determined, but it is also possible to determine a unit region A i with the pixel P i at a corner, or a unit region A i with a shape other than rectangular. i .

[0165] Next, the image correction unit 35 acquires the pixel values of the multiple pixels included in the determined unit region A i from the object captured image 501. Among the multiple pixels included in this unit region A i , the pixel P i is also included. In addition, in the case where the object captured image 501 is a color image as described above, the pixel values acquired by the image correction unit 35 respectively include the pixel values of the R image, the G image, and the B image included in this color image.

[0166] The image correction unit 35 calculates the average value of the pixel values of the R image (hereinafter referred to as the average pixel value of the R image) based on the pixel values of the R image included in the multiple pixels acquired. Similarly, the image correction unit 35 calculates the average value of the pixel values of the G image (hereinafter referred to as the average pixel value of the G image) based on the pixel values of the G image included in the multiple pixels acquired. Further, the image correction unit 35 calculates the average value of the pixel values of the B image (hereinafter referred to as the average pixel value of the B image) based on the pixel values of the B image included in the multiple pixels acquired.

[0167] Furthermore, the image correction unit 35 determines the maximum value among the calculated average pixel value of the R image, the average pixel value of the G image, and the average pixel value of the B image (hereinafter referred to as the maximum value of the RGB average pixel values).

[0168] In this case, for pixel P i the correction coefficient is calculated by the following equation (1).

[0169] [Equation 1]

[0170]

[0171] In addition, in Equation (1), gain(R, G, B) represents the correction coefficient (gain value) for pixel P i (the pixel values of the R image, G image, and B image respectively). R ave represents the average pixel value of the R image, G ave represents the average pixel value of the G image, B ave represents the average pixel value of the B image. In addition, max(R ave , G ave , B ave ) represents the maximum value of the RGB average pixel values.

[0172] That is, in this embodiment, the value obtained by dividing the maximum value of the RGB average pixel values by the average pixel value of the R image is the correction coefficient for the pixel value of the R image included in the pixel value of pixel P i (hereinafter referred to as gain(R)). In addition, the value obtained by dividing the maximum value of the RGB average pixel values by the average pixel value of the G image is the correction coefficient for the pixel value of the G image included in the pixel value of pixel P i (hereinafter referred to as gain(G)). In addition, the value obtained by dividing the maximum value of the RGB average pixel values by the average value of the B image is the correction coefficient for the pixel value of the B image included in the pixel value of pixel P i (hereinafter referred to as gain(B)).

[0173] In the case where the correction coefficient for pixel P i is calculated as described above, this correction coefficient is used to correct the pixel value of this pixel P i . Specifically, when the pixel values of the R image, G image, and B image included in the pixel value of pixel P i are set to P i (R, G, B), and the pixel value of pixel P i after white balance correction is set to P i ′(R, G, B), P i ′(R, G, B) is calculated by the following equation (2).

[0174] [Equation 2]

[0175] pi′ = pi(R, G, B) × gain(R, G, B) Equation (2)

[0176] That is, when performing white balance correction, pixel P i has a pixel value P i (R, G, B) that is corrected to a pixel value P i ′(R, G, B), and this pixel value P i ′(R, G, B) includes a value obtained by multiplying the pixel value of the R image included in the pixel value of pixel P i by gain(R), a value obtained by multiplying the pixel value of the G image included in the pixel value of pixel P i by gain(G), and a value obtained by multiplying the pixel value of the B image included in the pixel value of pixel P i by gain(B).

[0177] Here, pixel P i included in the target captured image 501 has been described, but the same processing is also performed on other pixels included in the target captured image 501.

[0178] In addition, as described above, the pixels whose pixel values are corrected are at least the pixels included in the ranging area, but the pixel values of all pixels included in the target captured image may also be corrected.

[0179] In addition, here, it has been described that the unit area is a partial area of the target captured image including the pixels whose pixel values are corrected, but the unit area may also be the entire area of the target captured image.

[0180] Furthermore, for example, in the case where an image with rich color variations is the target captured image, more effective white balance correction can be performed with a small area as the unit area. However, in this embodiment, since the structure is to obtain (estimate) the distance to the subject using the blur generated in the target captured image as a physical clue, it is preferable to set as the unit area an area that does not affect the relationship between the aberration of the optical system and the distance to the subject (that is, does not change the color of the blur, etc.). In addition, an area that does not affect the relationship between the aberration of the optical system and the distance to the subject includes, for example, an area of a size including the blur generated in the target captured image. However, at the time of performing the processing of step S5 described above, the distance to the subject included in the ranging area (that is, the size of the blur) is unknown. Therefore, for example, an area of a size that can include the blur that may be generated in the target captured image (ranging area) is used as the unit area. In addition, since the blur generated in the target captured image has position dependence, the size of the unit area can also be determined based on the position of the ranging area, etc.

[0181] Return again to Figure 19 ,the distance acquisition unit 36 acquires distance information indicating the distance to the subject included in the distance measurement area based on the above-described distance measurement area information and the object captured image after white balance correction (color correction) is performed in step S5 (step S6).

[0182] In this case, the distance acquisition unit 36 inputs information (gradient data) related to the distance measurement area indicated by the distance measurement area information to the statistical model stored in the statistical model storage unit 31. Thereby, the distance to the subject in the distance measurement area is estimated in the statistical model, and the estimated distance is output by the statistical model. In addition, the distance to the subject is estimated and output for each pixel constituting the distance measurement area. Thereby, the distance acquisition unit 36 can acquire distance information indicating the distance output from the statistical model.

[0183] Here, for example, as Figure 25 shown, when the statistical model estimates the distance 504 based on information related to the captured image (object captured image) 501, sometimes the degree of unreliability (hereinafter, referred to as unreliability) 601 of the estimation is calculated for each pixel, and the unreliability 601 is output together with the distance 504. The calculation method of the unreliability 601 is not limited to a specific method, and various known methods can be applied.

[0184] In this case, the distance acquisition unit 36 acquires the unreliability together with the distance information acquired in step S6 (step S7).

[0185] However, in the case where the user performs an operation of pressing the reliability confirmation mode button 306d set in the preview screen 306a shown above Figure 20 ,the reliability confirmation mode setting unit 37 sets the reliability confirmation mode. In addition, the timing at which the user performs the operation of pressing the reliability confirmation mode button 306d can be, for example, the timing at which the preview display is performed in step S1 above, or can be, for example, after the processing of step S7 is executed. In the case where the reliability confirmation mode is set by the reliability confirmation mode setting unit 37, information indicating that the reliability confirmation mode is set (hereinafter, referred to as the reliability confirmation mode setting flag) is held inside the image processing apparatus 3.

[0186] When step S7 described above is executed, it is determined whether the reliability confirmation mode is set based on the above-described reliability confirmation mode setting flag (step S8).

[0187] In the case where it is determined that the reliability confirmation mode is set (Yes in step S8), the display processing unit 32 performs preview display of the unreliability acquired in step S7 (step S9).

[0188] In this case, the above-mentioned Figure 21 shown preview screen 306e is migrated to Figure 26 the shown preview screen 306g.

[0189] As Figure 26 shown, in the image display area 306b set in the preview screen 306g, the unreliability calculated for each pixel as described above is overlapped and displayed on the object captured image as reliability.

[0190] In addition, in the case where the unreliability calculated for a specific pixel in the present embodiment is high, it indicates that the reliability of the distance to the subject corresponding to the specific pixel is low. On the other hand, in the case where the unreliability calculated for a specific pixel is low, it indicates that the reliability of the distance to the subject corresponding to the specific pixel is high.

[0191] In addition, in the image display area 306b, the unreliability is displayed in different colors corresponding to the unreliability. Thus, the user can intuitively grasp the unreliability (reliability) displayed on the object captured image.

[0192] In addition, the unreliability can be represented, for example, by the shade of the color corresponding to the unreliability, or by performing processing (for example, changing the thickness of the contour) on the distance measurement area or the contour (edge part) of the subject included in the distance measurement area. In addition, the unreliability can be displayed only as a numerical value or in other forms.

[0193] Here, a decision button 306h and a re-execution button 306i are also provided in the preview screen 306g. The decision button 306h is a button for instructing the output of the distance information obtained in step S6. The re-execution button 306i is a button for instructing the re-execution of the acquisition of the distance information.

[0194] The user refers to the preview screen 306g (image display area 306b) that displays the unreliability (reliability) as described above, and when it can be judged that the reliability of the distance to the subject included in the distance measurement area is high (that is, the unreliability is low), the user performs an operation of pressing the decision button 306h (that is, an operation of instructing the output of the distance information). On the other hand, when the user can judge that the reliability of the distance to the subject included in the distance measurement area is low (that is, the unreliability is high), the user performs an operation of pressing the re-execution button 306i (that is, an operation of instructing the re-execution of the acquisition of the distance information).

[0195] In this case, the distance acquisition unit 36 determines whether to output the distance information acquired in step S6 based on the user's operation (step S10). In step S10, it is determined to output the distance information when the operation of pressing the decision button 306h is performed in the preview screen 306g, and it is determined not to output the distance information when the operation of pressing the execute again button 306i is performed.

[0196] When it is determined to output the distance information (Yes in step S10), the output unit 38 outputs the distance information acquired in step S6 (step S11). In this case, the output unit 38 can output the distance information as image data in a mapping form configured to correspond in position to the distance measurement area in the object captured image (image data composed of pixels in which the distance represented by the distance information is set as the pixel value). In addition, the distance information can be output only as a numerical value or the like, or in other forms.

[0197] In addition, the distance information can be displayed on the preview screen or on a screen different from the preview screen. Also, the distance information can be displayed in combination with the object captured image, for example. In addition, the distance information can be stored in a server device or the like inside or outside the image processing device 3.

[0198] On the other hand, when it is determined not to output the distance information (No in step S10), the process returns to step S5 and the process is repeated. In this case, in step S5 above, the size of the unit area for which white balance correction (used in the calculation of the correction coefficient) is performed is changed, and the process of step S5 is executed again.

[0199] Here, the unit area for which white balance correction is performed as described above is preferably an area that does not affect the relationship between the aberration of the optical system and the distance to the subject (that is, does not change the blurred color, etc.). However, at the time of executing the process of step S5, the distance to the subject (that is, the size of the blur) is unknown. Therefore, in step S5, an area with a size that can include the blur that may occur in the captured image (distance measurement area) is used as the unit area for white balance correction. In contrast, in the process of step S5 executed again, by executing the process of step S6 above, the size of the blur is estimated based on the statistical model. Therefore, in step S5 executed again, an area including the size of the blur estimated by the statistical model (that is, the blur generated according to the distance to the subject included in the distance measurement area) is used as the unit area. As a result, more effective white balance correction can be performed in a unit area (minute area) suitable for the size of the blur actually generated in the object captured image, and thus it is possible to expect to acquire distance information representing a distance with a low degree of unreliability (that is, a high degree of reliability) in step S6 executed later.

[0200] That is, in the present embodiment, when the process of step S5 is repeatedly executed, the correction coefficient is calculated in such a way that the unreliability (or the ratio of pixels at a distance where low reliability is estimated) is reduced (that is, the unit area is changed).

[0201] Here, it has been described that when the operation of pressing the re-execution button 306i is performed, the process returns to step S5 for execution. However, when the operation of pressing the re-execution button 306i is performed, for example, the process may also return to step S3 to re-select a different distance measurement area.

[0202] In addition, in the present embodiment, it has been mainly described that one statistical model is stored in the statistical model storage unit 31. However, a plurality of statistical models may also be stored in the statistical model storage unit 31. Specifically, as described above, due to the type of lens, especially the PSF shape near the end of the image varies greatly. Therefore, in a structure where distance information is obtained using one statistical model generated without considering the type of lens, the accuracy of the distance obtained from the image may sometimes be low. Thus, the statistical model storage unit 31 may store, for example, a statistical model for each lens. In such a structure, when the operation of pressing the re-execution button 306i is performed as described above, the statistical model used in the process of step S6 may be changed to, for example, the statistical model corresponding to the lens 21 determined by the lens information (specification value of the lens 21). In addition, when changing the statistical model in this way, for example, an aberration map obtained by analyzing the object captured image may also be used. In this case, by pre-associating the aberration map and the statistical model suitable for the image from which the aberration map is obtained, the statistical model can be changed to the one corresponding to the aberration map obtained by analyzing the object captured image. In addition, the statistical model may also be changed according to the distance information obtained in step S6.

[0203] For example, it is changed to the statistical model learned from an image that has been color-corrected in a unit area including the blur size after obtaining the distance information once.

[0204] In addition, although not shown in Figure 19 , when changing the statistical model as described above, the process after step S6 may also be executed without changing the unit area (correction coefficient).

[0205] On the other hand, when it is determined in step S8 that the reliability confirmation mode is not set (No in step S8), the processes of steps S9 and S10 are not executed, and the process of step S11 is executed. In this case, for example, it is also possible to output only the distance information indicating the distances with unreliability below the threshold (reliability above the threshold) among the distance information obtained in step S6 (that is, discard the distance information indicating the distances with unreliability below the threshold), but it is also possible to output all the distance information obtained in this step S6.

[0206] According to the above Figure 19 processing shown, the ranging area is selected based on the user's operation. After performing white balance correction on the object captured image, a statistical model is used to obtain distance information and unreliability, and the unreliability (reliability) is displayed on the preview screen. However, such a series of operations can also be performed interactively. For example, when the image processing device 3 (ranging device 1) is implemented by a smartphone or the like, the user can perform the following operations: while referring to the preview screen, specify the ranging area, obtain the distance information again when the reliability of the distances included in the ranging area is low, and output the distance information when the desired distance information is obtained.

[0207] In addition, in the present embodiment, it is described that the ranging area is selected according to the user's operation on the preview screen (image display area). However, the ranging area may be set to the entire area of the captured image without depending on the user's operation, or may be set to an area including the subject automatically extracted by image processing. In this case, it can be set that Figure 1 the ranging area selection unit 33 shown is omitted. Figure 19 The processes of steps S2 and S3 shown can also be omitted.

[0208] Furthermore, in the present embodiment, it is described that the reliability confirmation mode is set according to the user's operation. However, it is also possible to perform operations with the reliability confirmation mode always set or operations with the reliability confirmation mode never set. In this case, it can also be set that Figure 1 the reliability confirmation mode setting unit 37 shown is omitted. In addition, when performing operations with the reliability confirmation mode always set, Figure 19 the process of step S8 shown is omitted. On the other hand, when performing operations with the reliability confirmation mode never set, Figure 19 the processes of S8 to S10 shown are omitted.

[0209] In addition, in the present embodiment, the case where the statistical model is constructed (generated) to calculate the unreliability is described. However, the statistical model can also be constructed so as not to calculate the unreliability when estimating the distance to the subject. In this case, for example, it is possible to omitFigure 19 The processes of steps S7 to S10 shown above.

[0210] Furthermore, in the present embodiment, it is assumed that the user operates the image processing apparatus 3 (range measuring apparatus 1) while referring to the preview screen. However, the present embodiment may also be configured to perform a simpler process of not performing preview display but only outputting distance information indicating the distance to the subject included in the captured image captured by the imaging device 2. In this case, it is only necessary to execute Figure 19 The processes of steps S4 to S6 and S11 shown above.

[0211] As described above, in the present embodiment, the distance to the subject can be obtained from the captured image using a statistical model, but this statistical model is generated by executing a learning process.

[0212] Hereinafter, the learning process for generating the statistical model (hereinafter simply referred to as the learning process of the statistical model) will be described. Figure 27 An example of the learning process (learning method) of the statistical model in the present embodiment is shown. Here, the learning process of the statistical model using the image captured by the imaging device 2 will be described, but this learning process of the statistical model may also be performed using, for example, an image captured by another device (such as a camera) having the same optical system as the optical system of the imaging device 2.

[0213] In addition, in the above description, the image captured by the imaging device 2 is used as the captured image in order to obtain distance information. However, in the present embodiment, for convenience, the image used for learning the blur that changes non-linearly according to the distance in the statistical model is referred to as the learning image.

[0214] When using any of the first method described above with reference to Figure 13 the description, the second method described with reference to Figure 15 the description, and the third method described with reference to Figure 17 the description, the learning process of the statistical model is basically performed by inputting information related to the learning image 701 into the statistical model and feeding back the error between the distance (distance information) 702 estimated by the statistical model and the correct value 703 to the statistical model. In addition, feedback means updating the parameters (such as weight coefficients) of the statistical model to reduce the error.

[0215] When the first method is applied as the method for estimating distance from a captured image as described above, even during the learning process of the statistical model, for each local region (image patch) extracted from the learning image 701, information (gradient data) related to the local region is input to the statistical model, and the distance 702 of each pixel within each local region is estimated by the statistical model. The error obtained by comparing the distance 702 thus estimated with the correct value 703 is fed back to the statistical model.

[0216] Similarly, when the second method is applied as the method for estimating distance from a captured image, even during the learning process of the statistical model, for each local region (image patch) extracted from the learning image 701, the gradient data and the position information are input to the statistical model as information related to the local region, and the distance 702 of each pixel within each local region is estimated by the statistical model. The error obtained by comparing the distance 702 thus estimated with the correct value 703 is fed back to the statistical model.

[0217] In addition, when the third method is applied as the method for estimating distance from a captured image, even during the learning process of the statistical model, information (gradient data) related to the entire region of the learning image 701 is input to the statistical model, and the distance 702 of each pixel within the learning image 701 is estimated by the statistical model. The error obtained by comparing the distance 702 thus estimated with the correct value 703 is fed back to the statistical model.

[0218] In addition, when information related to the learning image 701 is input to the statistical model as described above, in the same manner as in the case described in the above Figure 25 the uncertainty 602 with respect to the distance 702 is calculated. In the learning process of the statistical model in this case, the error obtained by dividing the error between the distance 702 and the correct value 703 by the square of the uncertainty 602 is fed back. In this case, when the uncertainty 602 is set to infinity, the error becomes zero, so the square of the uncertainty 602 is added to the error as a penalty.

[0219] Based on the above learning process of the statistical model, the parameters (e.g., weight coefficients) of the statistical model are updated in such a way that the value obtained by correcting the error between the distance 702 and the correct value 703 by the uncertainty 602 decreases.

[0220] Here, for example, when there is no error between the distance 702 estimated by the statistical model and the correct value 703, and on the other hand, the uncertainty 602 is high, it can be speculated that the distance 702 may have been estimated accidentally. In this case, the lack of learning of the distance 702 (correct value 703) can be identified.

[0221] Even when using the unreliability calculated by the statistical model in this way, it is possible to reduce the learning bias.

[0222] In addition, the statistical model in the present embodiment is generated, for example, by repeatedly executing a learning process using learning images captured while changing the distance from the imaging device 2 to the subject while fixing the focus position. In addition, when the learning process for one focus position is completed, the learning process is similarly executed for other focus positions, whereby a statistical model with higher accuracy can be generated.

[0223] In addition, in the above Figure 18 it was described that the estimated distance is synonymous with the color, size, and shape of the estimated blur, but when information related to the learning image is input to the statistical model during the learning of the statistical model, the color, size, and shape of the blur (the blur value represented by positive and negative) corresponding to the actual distance to the subject when the learning image was captured are used as the correct values. Based on the statistical model learned in this way, the above blur value is output as the distance to the subject in the image.

[0224] Next, with reference to Figure 28 the flowchart, an example of the processing order of the process for generating the statistical model used in the image processing apparatus 3 in the present embodiment (that is, the learning process of the statistical model) will be described. In addition, Figure 28 the processing shown can be executed, for example, in the image processing apparatus 3 or in other apparatuses.

[0225] First, a pre-prepared learning image is acquired (step S21). This learning image is, for example, an image generated by the image sensor 22 based on the light transmitted through the lens 21 provided in the imaging device 2, and is an image affected by the aberration of the optical system (lens 21) of the imaging device 2. Specifically, in the learning image, the Figures 4 - 10 blur that varies non-linearly according to the distance to the subject as described in

[0226] In addition, in the learning process of the statistical model, learning images obtained by imaging the subject at each distance with as fine a granularity as possible from the lower limit value (near) to the upper limit value (inside) of the distance that can be obtained (estimated) in the image processing apparatus 3 are prepared in advance. In addition, as the learning image, it is preferable to prepare various images of different subjects.

[0227] Here, in the present embodiment, the distance to the subject (the represented distance information) is obtained from the captured image that has been color-corrected (white balance correction) as described above. Therefore, the learning image obtained in step S1 is also color-corrected in the same manner (step S22). In addition, the process of step S22 is not essential, but in order to improve the accuracy of the distance estimated in the above statistical model, it is preferable to color-correct the learning image as well. The process of step S22 is the same as the process of step S5 shown above, so the detailed description thereof is omitted here. Figure 19 The process is the same as the process of step S5 shown above, so the detailed description thereof is omitted here.

[0228] When the process of step S22 is executed, information related to the learning image that has been color-corrected in this step S22 is input to the statistical model (step S23).

[0229] In the case where the above-described first method is applied as the method for estimating the distance from the captured image, as information related to the learning image, the gradient data of the R image, G image, and B image are input to the statistical model for each local region of the learning image.

[0230] In the case where the above-described second method is applied as the method for estimating the distance from the captured image, as information related to the learning image, the gradient data of the R image, G image, and B image and the position information on the learning image of the local region are input to the statistical model for each local region of the learning image.

[0231] In the case where the above-described third method is applied as the method for estimating the distance from the captured image, as information related to the learning image, the gradient data of the R image, G image, and B image of the entire region of the learning image are input to the statistical model.

[0232] In addition, in the present embodiment, it has been described that the gradient data of the R image, G image, and B image are input to the statistical model. However, in the case of estimating the distance from the viewpoint of the blurred shape (PSF shape) generated from the above learning image, it is sufficient to input at least one of the gradient data of the R image, G image, and B image to the statistical model. On the other hand, in the case of estimating the distance from the viewpoint of the blurred color and size generated in the learning image due to chromatic aberration, it is sufficient to input at least two of the gradient data of the R image, G image, and B image to the statistical model. In addition, in the case of using the gradient data of two colors, color correction that makes the two colors the same balance can also be performed. According to such a configuration, it is possible to contribute to cost reduction of processing.

[0233] When information related to the learning image is input to the statistical model, the distance to the subject is estimated by the statistical model (step S24). In this case, the statistical model extracts the blur generated in the learning image from the learning image and estimates the distance corresponding to the blur.

[0234] In addition, when the process of step S24 is executed, the statistical model calculates the degree of unreliability with respect to the distance estimated in step S24 (step S25).

[0235] The distance estimated in step S24 is compared with the correct value obtained at the time of shooting the imaging image (step S26).

[0236] The comparison result (error) in step S25 is corrected using the degree of unreliability calculated in step S25 and is fed back to the statistical model (step S27). Thus, in the statistical model, the parameters are updated in such a way as to reduce the error (i.e., the blur generated in the learning image is learned).

[0237] By repeatedly executing the above Figure 28 shown process for each learning image, a statistical model that has learned the blur is generated, and the blur varies non-linearly according to the distance to the subject in the learning image. The statistical model thus generated is stored in the statistical model storage unit 31 included in the image processing apparatus 3.

[0238] Here, the learning process for one statistical model has been described. However, in the case where statistical models for each lens (i.e., multiple statistical models) are pre-stored in the statistical model storage unit 31, for example, as long as the above Figure 28 shown process is executed for each such lens (the learning image shot using the lens).

[0239] As described above, in the present embodiment, an imaging image (second image) affected by the aberration of the optical system is acquired, color correction for reducing the number of colors (i.e., color variation) expressed in the imaging image is performed on the imaging image, the imaging image (third image) after the color correction is input to the statistical model, and distance information (first distance information) representing the distance to the subject in the imaging image is acquired.

[0240] The imaging image captured by the imaging device 2 is affected by the color temperature of the light source (i.e., the environment around the imaging device 2) and the like. However, in the present embodiment, with the above structure, by reducing the color variation of the imaging image, it is possible to improve the accuracy of the distance (the distance estimated in the statistical model) obtained from the imaging image and the environmental robustness when obtaining the distance from the imaging image.

[0241] In addition, in the present embodiment, since the distance information indicating the distance to the subject is obtained from the captured image in which the color (i.e., pixel value) that has been corrected for the influence of the color temperature of the light source and the like, it is not necessary to learn the blur of all colors (color changes) considering the influence of the color temperature of the light source and the like, and the burden brought by the learning process of the statistical model can be reduced.

[0242] In addition, in the present embodiment, white balance correction is performed in units of a region (first region) of a predetermined size included in the captured image. However, since the present embodiment is configured to estimate (obtain) the distance by using the blur generated in the captured image as a physical clue related to the distance to the subject, the region (unit region) for performing the white balance correction is preferably a region that does not affect the relationship between the aberration of the optical system and the distance to the subject. A region that does not affect the relationship between the aberration of the optical system and the distance to the subject is, for example, a region having a size that includes at least the blur generated according to the distance to the subject. Thus, by performing white balance correction, the color change of the blur is not caused, and therefore it is possible to prevent the white balance correction from becoming a main cause of reducing the accuracy of the distance.

[0243] In addition, in the present embodiment, for example, a correction coefficient (first correction coefficient) for the first pixel is calculated based on the pixel values of a plurality of second pixels included in the unit region including the first pixel, and color correction (white balance correction) for the pixel value of the first pixel is performed using the correction coefficient. Thus, the pixel values can be appropriately corrected for each pixel (pixels included in the distance measurement region) constituting the captured image.

[0244] Furthermore, in the present embodiment, when the reliability confirmation mode is set, white balance correction is performed again in a unit region (second region) having a different size from the unit region (first region) at the time of performing white balance correction, according to the user's operation based on the unreliability (degree of unreliability) with respect to the distance estimated by the statistical model. According to such a configuration, there is a possibility of obtaining distance information indicating a higher accuracy (lower unreliability) distance.

[0245] In addition, in the present embodiment, it has been described that, for example, a region having a size including the size of the blur estimated by the statistical model is used as the unit region to perform white balance correction again, but the unit region may be changed as long as the unreliability is reduced. In this case, it may also be configured to arbitrarily change the unit region and repeat the process until the unreliability is reduced (for example, the user performs an operation of pressing a decision button set in the preview screen).

[0246] In addition, the user refers to the unreliability (second unreliability) of the distance indicated by the distance information (second distance information) obtained from the captured image (fourth image) that has been white balance corrected again. When the unreliability is low, the obtained distance information is output according to the user's operation based on the unreliability. With such a configuration, it is possible to output the distance information desired by the user (for example, the distance information indicating a low unreliability).

[0247] In addition, in the case where the above reliability confirmation mode is not set, it may be configured to output only the distance information whose unreliability is below the threshold (that is, discard the distance information whose unreliability is not below the threshold).

[0248] Furthermore, in the present embodiment, a distance measurement area (third area) in the captured image is selected, color correction is performed on the distance measurement area, and distance information indicating the distance to the subject in the distance measurement area is obtained. Thus, it is possible to output the distance information desired by the user indicating the distance to the subject. In this case, the area specified by the user's operation may be selected as the distance measurement area, or the area including the subject detected based on the position in the captured image specified by the user's operation may be selected as the distance measurement area.

[0249] In addition, in the present embodiment, the case of performing white balance correction as color correction has been mainly described. However, the present embodiment may have a configuration that performs color correction to reduce the color change generated in the captured image corresponding to the surrounding environment of the imaging device 2 including the color temperature of the above light source and the like.

[0250] (Second Embodiment)

[0251] Next, the second embodiment will be described. In addition, in the present embodiment, the detailed description of the same parts as those in the foregoing first embodiment is omitted, and mainly the parts different from the first embodiment will be described.

[0252] Figure 29 An example of the configuration of the distance measurement system including the image processing device of the present embodiment is shown. In Figure 29 , the same parts as those in the above Figure 1 are labeled with the same reference numerals and their detailed descriptions are omitted, and the parts different from the Figure 1 will be described.

[0253] As Figure 29 shown, the difference between the image processing device 3 of the present embodiment and the first embodiment is that it includes an evaluation unit 39 instead of the reliability confirmation mode setting unit 37 described in the first embodiment.

[0254] In addition, the image processing apparatus 3 of the present embodiment has the above-described Figure 2 hardware configuration, and a part or all of the evaluation unit 39 is implemented by causing the CPU 301 (i.e., the computer of the image processing apparatus 3) to execute the image processing program 303A, that is, by software. In addition, a part or all of the evaluation unit 39 can be implemented by hardware such as an IC (Integrated Circuit), or can be implemented by a combination of software and hardware.

[0255] The evaluation unit 39 evaluates the distance information acquired by the distance acquisition unit 36 based on the unreliability described in the above-described first embodiment. In the present embodiment, based on the evaluation result of the evaluation unit 39, color correction of the image correction unit 35 is performed again.

[0256] Next, with reference to Figure 30 the flowchart of, an example of the processing sequence of the image processing apparatus 3 when acquiring distance information from a captured image will be described.

[0257] First, processing of steps S31 to S33 corresponding to the processing of steps S1 to S3 described above is executed. Figure 19 shown.

[0258] Here, when executing the processing of step S33, for example, a threshold value of unreliability is set according to a user operation (step S34).

[0259] When executing the processing of step S33, processing of steps S35 to S38 corresponding to the processing of steps S4 to S7 shown in Figure 19 is executed.

[0260] Next, in order to evaluate the distance information acquired by the distance acquisition unit 36, the evaluation unit 39 determines whether the unreliability acquired in step S38 is less than or equal to the threshold value set in step S34 (step S39).

[0261] In addition, in the above step S38, the unreliability with respect to the distance estimated for each pixel included in the distance measurement area (i.e., the unreliability of each pixel) is acquired, but in step S39, for example, it is determined whether the representative value (e.g., average value, etc.) of the unreliabilities of the plurality of pixels included in the distance measurement area is less than or equal to the threshold value.

[0262] In the case where it is determined that the unreliability (representative value) is not less than or equal to the threshold value (NO in step S39), the process returns to step S36 and the processing is repeated (i.e., the processing of step S36 is executed again). In addition, the processing of the re-executed step S36 is the same as the processing of the re-executed step S5 described in Figure 19 above, and thus the detailed description thereof is omitted here.

[0263] On the other hand, when it is determined that the representative value of the unreliability is below the threshold (Yes in step S39), the process of step S40, which is equivalent to the process of step S11 shown above, is executed. Figure 19 The process of step S40 is executed.

[0264] In the above-described first embodiment, distance information is output according to the operation of the user who has referred to (confirmed) the unreliability (reliability). However, as described above, in the present embodiment, when the unreliability is not below the threshold set by the user, white balance correction is performed again, and when the unreliability is below the threshold, distance information is automatically output. With such a configuration, distance information with the reliability desired by the user can be further obtained together with the accuracy of the distance obtained from the image and the environmental robustness when obtaining the distance from the image, and thus the usability can also be improved.

[0265] In addition, in step S38 shown above, it has been described that it is determined whether the representative value of the unreliability of each pixel constituting the ranging area is below the threshold. However, for example, it may also be determined whether the ratio of the pixels whose unreliability with respect to all the pixels constituting the ranging area is below the first threshold is above the second threshold. With such a configuration, when the ratio of the pixels whose unreliability with respect to all the pixels included in the ranging area is below the first threshold is not above the second threshold, the process returns to step S35 and the process is repeated, and when the ratio of the pixels whose unreliability with respect to all the pixels included in the ranging area is below the first threshold is above the second threshold, the process of step S39 is executed. Figure 30 The process of step S38 may also be executed for each small area (hereinafter referred to as a divided area) obtained by dividing the ranging area. Specifically, the process of step S38 is executed for each divided area, and the process is repeated by returning only to step S35 for the divided areas determined to have an unreliability not below the threshold. With such a configuration, it is possible to perform optimal white balance correction while referring to the unreliability in units of the divided areas obtained by dividing the ranging area, and the results (distance information) of the divided area units are combined and output.

[0266] Furthermore, in the process shown above, for example, the processes of steps S35 to S37 may be repeatedly executed a predetermined number of times, and in step S38, it is determined whether the process has been executed a predetermined number of times. In this case, a plurality of distance information is obtained by executing the processes of steps S35 to S37 a predetermined number of times, but in step S39, the distance information with the smallest unreliability among the plurality of distance information is output. With such a configuration, there is a possibility that distance information indicating a more accurate distance can be output.

[0267] Furthermore, in Figure 30 the process shown above, for example, the processes of steps S35 to S37 may be repeatedly executed a predetermined number of times, and in step S38, it is determined whether the process has been executed a predetermined number of times. In this case, a plurality of distance information is obtained by executing the processes of steps S35 to S37 a predetermined number of times, but in step S39, the distance information with the smallest unreliability among the plurality of distance information is output. With such a configuration, there is a possibility that distance information indicating a more accurate distance can be output.

[0268] In addition, similar to the first embodiment described above, this embodiment may be configured not to perform preview display, or may be configured to omit the ranging area selection unit 33.

[0269] (Application Example)

[0270] Hereinafter, an application example of the ranging system 1 having the configuration as in the first embodiment and the second embodiment described above will be described. Here, for convenience, the case where the ranging system 1 is implemented as one device (ranging device) having a imaging unit equivalent to the imaging device 2 shown in Figure 1 and Figure 29 and an image processing unit equivalent to the image processing device 3 will be described. In the following drawings, it is assumed that the ranging device 1 includes the imaging unit 2 and the image processing unit 3 for explanation.

[0271] Figure 31 FIG. shows an example of the functional configuration of the moving body 800 equipped with the ranging device 1. The moving body 800 can be realized, for example, as an automobile with an autonomous driving function, an unmanned aerial vehicle, an autonomous mobile robot, or the like. An unmanned aerial vehicle is an aircraft, a rotary-wing aircraft, a glider, or an airship that cannot be occupied by a person and can fly by remote control or automatic control. For example, it includes drones (multi-rotor helicopters), radio receivers, helicopters for pesticide spraying, etc. Autonomous mobile robots include mobile robots such as automated guided vehicles (AGVs), cleaning robots for cleaning the ground, and communication robots for guiding visitors in various ways. In the moving body 800, it includes not only robots that move the robot body but also industrial robots such as robotic arms having a drive mechanism for moving or rotating a part of the robot.

[0272] As Figure 31 shown, the moving body 800 has, for example, a ranging device 1, a control signal generation unit 801, and a drive mechanism 802. The ranging device 1 is arranged such that, for example, the imaging unit 2 can image an object in the traveling direction of the moving body 800 or a part thereof.

[0273] As Figure 32 shown, when the moving body 800 is an automobile 800A, the ranging device 1 is arranged as a so-called front camera for imaging the front. In addition, the ranging device 1 may be arranged as a so-called rear camera for imaging the rear when reversing. In addition, a plurality of ranging devices 1 may be arranged as a front camera and a rear camera. Furthermore, the ranging device 1 may also be arranged as a device having the function of a so-called driving recorder. That is, the ranging device 1 may also be a video recording device.

[0274] Figure 33 An example is shown in which the moving body 800 is a drone 800B. The drone 800B includes a drone main body 811 corresponding to the drive mechanism 802 and four propeller units 812 to 815. Each of the propeller units 812 to 815 has a propeller and a motor. By transmitting the drive of the motor to the propeller, the propeller rotates, and the drone 800B floats by the lift based on this rotation. A distance measuring device 1 is mounted, for example, on the lower part of the drone main body 811.

[0275] In addition, Figure 34 An example is shown in which the moving body 800 is an autonomous mobile robot 800C. A power unit 821 including a motor, wheels, etc., corresponding to the drive mechanism 802, is provided at the lower part of the mobile robot 800C. The power unit 821 controls the rotational speed of the motor and the orientation of the wheels. The mobile robot 800C transmits the drive of the motor, so that the wheels provided on the road surface or the ground rotate, and by controlling the orientation of the wheels, it can move in any direction. In Figure 34 the example shown, the distance measuring device 1 is provided, for example, at the head of the humanoid mobile robot 800C so that the imaging unit 2 images the front of the mobile robot 800C. In addition, the distance measuring device 1 can be provided so as to image the rear, left, and right of the mobile robot 800C, or a plurality of distance measuring devices 1 can be provided so as to image a plurality of directions. In addition, by providing the distance measuring device 1 in a small robot with little space for mounting sensors and the like, the own position, posture, and the position of the object are estimated, and thus dead reckoning can also be performed.

[0276] In addition, as Figure 35 shown, when the moving body 800 is a robotic arm 800D and the movement and rotation of a part of the robotic arm 800D are controlled, the distance measuring device 1 can also be provided at the tip of the robotic arm 800D or the like. In this case, the imaging unit 2 included in the distance measuring device 1 images the object held by the robotic arm 800D, and the image processing unit 3 can estimate the distance to the object to be held by the robotic arm 800D. Thus, in the robotic arm 800D, an accurate grasping operation of the object can be performed.

[0277] The control signal generation unit 801 outputs a control signal for controlling the drive mechanism 802 based on the distance information indicating the distance to the subject, which is output from the distance measurement device 1 (image processing unit 3). The drive mechanism 802 drives the moving body 800 or a part of the moving body 800 according to the control signal output from the control signal generation unit 801. The drive mechanism 802 performs at least one of, for example, the movement, rotation, acceleration, deceleration, addition and subtraction of thrust (lift), conversion of the traveling direction, switching between the normal operation mode and the autonomous driving mode (collision avoidance mode), and operation of safety devices such as airbags of the moving body 800 or a part of the moving body 800. For example, when the distance to the subject is less than the threshold, the drive mechanism 802 may perform at least one of movement, rotation, acceleration, addition and subtraction of thrust (lift), direction conversion toward the approaching object, and switching from the autonomous driving mode (collision avoidance mode) to the normal operation mode.

[0278] In addition, Figure 32 The drive mechanism 802 of the shown automobile 800A is, for example, a tire. Figure 33 The drive mechanism 802 of the shown drone 800B is, for example, a propeller. Figure 34 The drive mechanism 802 of the shown mobile robot 800C is, for example, a leg. Figure 35 The drive mechanism 802 of the shown robotic arm 800D is, for example, a support portion that supports the front end where the distance measurement device 1 is provided.

[0279] The moving body 800 may further include a speaker and a display that are input with information (distance information) related to the distance to the subject, which is output from the distance measurement device 1. The speaker or the display is connected to the distance measurement device 1 by wire or wirelessly, and is configured to output sound or an image related to the distance to the subject. Further, the moving body 800 may have a light emitting portion that is input with information related to the distance to the subject, which is output from the distance measurement device 1, and can be lit and extinguished according to the distance to the subject, for example.

[0280] In addition, for example, when the moving body 800 is the drone 800B, when making a map (three-dimensional shape of an object), conducting a structural survey of a building or terrain, or inspecting cracks, wire breaks, etc. from above, the imaging unit 2 acquires an image obtained by imaging the object, and determines whether the distance to the subject is equal to or greater than the threshold. The control signal generation unit 801 generates a control signal for controlling the thrust of the drone 800B based on the determination result so as to keep the distance to the inspection object constant. Here, the thrust also includes lift. The drive mechanism 802 operates the drone 800B based on the control signal, whereby the drone 800B can fly parallel to the inspection object. When the moving body 800 is a surveillance drone, a control signal for controlling the thrust of the drone to keep the distance from the object to be monitored constant may also be generated.

[0281] In addition, when the moving body 800 (e.g., the drone 800B) is used for maintenance inspections of various infrastructures (hereinafter simply referred to as infrastructures), by using the imaging unit 2 to image the image of the part to be repaired (hereinafter referred to as the repair part) including the cracked part or the rusted part in the infrastructure, the distance to the repair part can be obtained. In this case, by using the distance to the repair part, the size of the repair part can be calculated from the image. Thus, for example, by displaying the repair part on the map showing the entire infrastructure, the maintenance inspector of the infrastructure can identify the repair part. In addition, it is also useful to pre-transmit the size of the repair part to the maintenance inspector for a smooth repair operation.

[0282] In addition, when the drone 800B is flying, the imaging unit 2 acquires an image obtained by imaging the ground direction and determines whether the distance from the ground is above a threshold value. Based on this determination result, the control signal generation unit 801 generates a control signal for controlling the thrust of the drone 800B so that the height from the ground becomes a specified height. The drive mechanism 802 operates the drone 800B based on this control signal, so that the drone 800B can fly at the specified height. If the drone 800B is a drone for pesticide spraying, by keeping the height of the drone 800B from the ground constant in this way, it is easy to spray pesticides evenly.

[0283] In addition, when the moving body 800 is the vehicle 800A or the drone 800B, during the convoy driving of the vehicle 800A and the cooperative flight of the drone 800B, the imaging unit 2 images the vehicle in front and the drones around and determines whether the distance to the vehicle and the drones is above a threshold value. Based on this determination result, the control signal generation unit 801 generates a control signal for controlling the speed of the vehicle 800A and the thrust of the drone 800B so that the distance from the vehicle in front and the drones around is constant. The drive mechanism 802 operates the vehicle 800A and the drone 800B based on this control signal, so that it is easy to perform the convoy driving of the vehicle 800A and the cooperative flight of the drone 800B.

[0284] Furthermore, when the moving body 800 is the vehicle 800A, it can also be configured to be able to receive the driver's instruction via the user interface so that the driver of the vehicle 800A can set (change) the threshold value. Thus, the driver can drive the vehicle 800A with a good inter-vehicle distance. In addition, in order to maintain a safe inter-vehicle distance from the vehicle in front, the threshold value can be changed according to the speed of the vehicle 800A. The safe inter-vehicle distance varies according to the speed of the vehicle 800A. Therefore, the faster the speed of the vehicle 800A, the larger (longer) the threshold value can be set.

[0285] In addition, when the moving body 800 is the vehicle 800A, a predetermined distance in the traveling direction may be set as a threshold value, and when an object appears near the threshold value, the brake is operated or a control signal for operating a safety device such as an airbag is generated. In this case, safety devices such as an automatic brake and an airbag are provided in the drive mechanism 802.

[0286] According to at least one of the above-described embodiments, it is possible to provide an image processing apparatus, a distance measuring apparatus, a method, and a program that can improve the accuracy of the distance obtained from an image and the environmental robustness when obtaining the distance from the image.

[0287] In addition, the various functions described in the present embodiment and this modification can also be implemented by a circuit (processing circuit). In an example of the processing circuit, it includes a programmed processor such as a central processing unit (CPU). The processor executes the functions described respectively by executing a computer program (instruction set) stored in a memory. The processor may also be a microprocessor including a circuit. In an example of the processing circuit, it also includes a digital signal processor (DSP), an application specific integrated circuit (ASIC), a microcontroller, a controller, and other circuit components. Each of the other components other than the CPU described in the present embodiment can also be implemented by a processing circuit.

[0288] In addition, since the various processes of the present embodiment can be implemented by a computer program, by simply installing the computer program on a computer from a computer-readable storage medium storing the computer program and executing it, the same effects as the present embodiment can be easily achieved.

[0289] Several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope or gist of the invention, and are equally included in the invention described in the claims and its equivalents.

[0290] In addition, the above-described embodiments can be summarized into the following technical solutions.

[0291] [Technical Solution 1]

[0292] An image processing apparatus includes:

[0293] a storage unit that stores a statistical model generated by learning a blur that is generated in a first image affected by an aberration of an optical system and that non-linearly varies according to a distance to a subject in the first image;

[0294] An image acquisition unit that acquires a second image affected by the aberration of the optical system;

[0295] A correction unit that performs color correction on the second image, which reduces the number of colors exhibited in the second image; and

[0296] A distance acquisition unit that inputs a third image obtained by performing color correction on the second image to the statistical model and acquires first distance information indicating the distance to the subject in the third image.

[0297] [Technical solution 2]

[0298] According to the above technical solution 1,

[0299] The correction unit performs white balance correction as the color correction.

[0300] [Technical solution 3]

[0301] According to the above technical solution 2,

[0302] The correction unit performs the white balance correction on the second image in units of a first region of a predetermined size included in the second image,

[0303] The first region is a region that does not affect the relationship between the aberration and the distance to the subject.

[0304] [Technical solution 4]

[0305] According to the above technical solution 3,

[0306] A region that does not affect the relationship between the aberration and the distance to the subject is a region in the second image that includes blurring generated according to the distance to the subject.

[0307] [Technical solution 5]

[0308] According to the above technical solution 3 or 4,

[0309] The correction unit calculates a first correction coefficient for the first pixel based on the pixel values of a plurality of second pixels included in the first region including the first pixel, and performs color correction on the pixel value of the first pixel using the first correction coefficient.

[0310] [Technical solution 6]

[0311] According to the above technical solutions 1 to 5,

[0312] It further includes an output unit,

[0313] The statistical model estimates the distance to the subject in the third image and calculates a first degree of unreliability, which represents the degree of unreliability with respect to the estimated distance.

[0314] The output unit outputs first distance information, which represents a distance at which the first degree of unreliability is equal to or less than a threshold value.

[0315] [Technical solution 7]

[0316] According to the above technical solution 3,

[0317] The statistical model estimates the distance to the subject in the third image and calculates a first degree of unreliability, which represents the degree of unreliability with respect to the estimated distance.

[0318] The correction unit performs white balance correction on the second image in units of a second region having a size different from that of the first region according to an operation of the user based on the first degree of unreliability.

[0319] The distance acquisition unit inputs a fourth image obtained by performing white balance correction on the second image in units of the second region to the statistical model, and acquires second distance information, which represents the distance to the subject in the fourth image.

[0320] [Technical solution 8]

[0321] According to the above technical solution 7,

[0322] An output unit is further provided.

[0323] The statistical model estimates the distance to the subject in the fourth image and calculates a second degree of unreliability, which represents the degree of unreliability with respect to the estimated distance.

[0324] The output unit outputs the second distance information according to an operation of the user based on the second degree of unreliability.

[0325] [Technical solution 9]

[0326] According to the above technical solution 3,

[0327] The statistical model estimates the distance to the subject in the third image and calculates a first degree of unreliability, which represents the degree of unreliability with respect to the estimated distance.

[0328] When the first unreliability is not below the threshold, the correction unit performs white balance correction on the second image in units of a second region having a different size from the first region.

[0329] The distance acquisition unit inputs a fourth image obtained by performing white balance correction on the second image in units of the second region to the statistical model, and acquires second distance information indicating the distance to the subject in the fourth image.

[0330] [Technical solution 10]

[0331] According to the above technical solution 9,

[0332] An output unit is further provided.

[0333] The statistical model estimates the distance to the subject in the fourth image and calculates a second unreliability indicating the degree of unreliability with respect to the estimated distance.

[0334] When the second unreliability is below the threshold, the output unit outputs the second distance information.

[0335] [Technical solution 11]

[0336] According to the above technical solutions 7 to 10,

[0337] The second region is a region obtained by changing the first region in such a way that the first unreliability is reduced.

[0338] [Technical solution 12]

[0339] According to the above technical solutions 1 to 11,

[0340] A region selection unit for selecting a third region in the second image is further provided.

[0341] The correction unit performs the color correction on the third region.

[0342] [Technical solution 13]

[0343] According to the above technical solution 12,

[0344] The region selection unit selects the region specified by the user's operation as the third region.

[0345] [Technical solution 14]

[0346] According to the above technical solution 12,

[0347] The area selection unit selects, as the third area, an area including a subject detected based on a position in the second image specified by a user operation.

[0348] [Technical solution 15]

[0349] A distance measurement device includes:

[0350] An imaging unit that images an image;

[0351] A storage unit that stores a statistical model generated by learning a blur that is generated in a first image affected by an aberration of an optical system and that varies non-linearly according to a distance to a subject in the first image;

[0352] An image acquisition unit that acquires a second image affected by the aberration of the optical system of the imaging unit;

[0353] A correction unit that performs color correction on the second image, the color correction reducing the number of colors exhibited in the second image; and

[0354] A distance acquisition unit that inputs a third image obtained by performing color correction on the second image into the statistical model and acquires distance information indicating a distance to a subject in the third image.

[0355] [Technical solution 16]

[0356] A method performed by an image processing device, the image processing device including a storage unit that stores a statistical model generated by learning a blur that is generated in a first image affected by an aberration of an optical system and that varies non-linearly according to a distance to a subject in the first image,

[0357] The method performed by the image processing device includes the following steps:

[0358] Acquiring a second image affected by the aberration of the optical system;

[0359] Performing color correction on the second image, the color correction reducing the number of colors exhibited in the second image; and

[0360] Inputting a third image obtained by performing color correction on the second image into the statistical model and acquiring distance information indicating a distance to a subject in the third image.

[0361] [Technical solution 17]

[0362] A program executed by a computer of an image processing apparatus, the image processing apparatus including a storage unit that stores a statistical model generated by learning a blur that is generated in a first image affected by an aberration of an optical system and that varies non-linearly according to a distance to a subject in the first image.

[0363] The program causes the computer to execute the following steps:

[0364] Obtain a second image affected by the aberration of the optical system;

[0365] Perform color correction on the second image, the color correction reducing the number of colors exhibited in the second image; and

[0366] Input a third image obtained by performing color correction on the second image into the statistical model and obtain distance information indicating a distance to a subject in the third image.

Claims

1. An image processing apparatus, comprising: a storage unit that stores a statistical model generated by learning a blur that is generated in a first image affected by an aberration of an optical system and that varies non-linearly according to a distance to a subject in the first image; an image acquisition unit that acquires a second image affected by the aberration of the optical system; a correction unit that performs white balance correction on the second image in units of a first region having a predetermined size included in the second image; and a distance acquisition unit that inputs a third image obtained by performing white balance correction on the second image to the statistical model and acquires first distance information indicating a distance to a subject in the third image, wherein the first region is a region that does not affect the relationship between the aberration and the distance to the subject.

2. The image processing apparatus according to claim 1, wherein the region that does not affect the relationship between the aberration and the distance to the subject is a region having a size that includes a blur generated according to the distance to the subject in the second image.

3. The image processing apparatus according to claim 2, wherein the correction unit calculates a first correction coefficient for a first pixel based on pixel values of a plurality of second pixels included in the first region including the first pixel, and performs white balance correction on the pixel value of the first pixel using the first correction coefficient.

4. The image processing apparatus according to any one of claims 1 to 3, wherein it further comprises an output unit, the statistical model estimates a distance to a subject in the third image and calculates a first unreliability indicating a degree of unreliability with respect to the estimated distance, and the output unit outputs first distance information indicating a distance at which the first unreliability is equal to or less than a threshold value.

5. The image processing apparatus according to claim 1, wherein the statistical model estimates a distance to a subject in the third image and calculates a first unreliability indicating a degree of unreliability with respect to the estimated distance, the correction unit performs white balance correction on the second image in units of a second region having a size different from that of the first region according to a user operation based on the first unreliability, and the distance acquisition unit inputs a fourth image obtained by performing white balance correction on the second image in units of the second region to the statistical model and acquires second distance information indicating a distance to a subject in the fourth image.

6. The image processing apparatus according to claim 5, wherein it further comprises an output unit, the statistical model estimates a distance to a subject in the fourth image and calculates a second unreliability indicating a degree of unreliability with respect to the estimated distance, and the output unit outputs the second distance information according to the user operation based on the second unreliability.

7. The image processing apparatus according to claim 1, wherein, the statistical model estimates the distance to the subject in the third image and calculates a first unreliability degree, which represents the degree of unreliability with respect to the estimated distance; the correction unit performs white balance correction on the second image in units of a second region having a size different from that of the first region when the first unreliability degree is not below the threshold; the distance acquisition unit inputs the fourth image obtained by performing white balance correction on the second image in units of the second region to the statistical model, and acquires second distance information, which represents the distance to the subject in the fourth image.

8. The image processing apparatus according to claim 7, wherein, an output unit is further provided; the statistical model estimates the distance to the subject in the fourth image and calculates a second unreliability degree, which represents the degree of unreliability with respect to the estimated distance; the output unit outputs the second distance information when the second unreliability degree is below the threshold.

Images