Image processing apparatus, method, and program

Abstract

To provide an image processing device, method and a program each of which enables the accuracy of distance measured using an image to be improved.SOLUTION: According to an embodiment, an image processing device is provided, which is used when measuring an imaging distance from an imaging apparatus to a subject by using an image captured by the imaging apparatus. The image processing device comprises first acquisition means, second acquisition means, third acquisition means, creation means, and output processing means. The first acquisition means acquires an image captured by the imaging apparatus. The second acquisition means acquires configuration information on the optical system of the imaging apparatus. The third acquisition means acquires an imaging distance to an image on the basis of the image. The creation means creates a first imaging condition for an image that is suitable for measuring the distance to a subject, on the basis of the configuration information and the imaging distance. The output processing means outputs the first imaging condition.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to an image processing apparatus, method, and program.

Background Art

[0002] Generally, in order to measure (acquire) the distance to a subject, it has been known to use images captured by two imaging devices (cameras) or a stereo camera (compound-eye camera). In recent years, however, techniques for measuring the distance to a subject from an image captured by a single imaging device (monocular camera) have been disclosed.

[0003] However, measuring the distance to a subject from an image captured by a single imaging device (that is, performing distance measurement with a single image captured by a monocular camera) is highly convenient, but it is necessary to adjust the imaging conditions in order to improve the accuracy of the distance measurement.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Therefore, the problem to be solved by the present invention is to provide an image processing apparatus, method, and program capable of improving the accuracy of the distance measured using an image.

Means for Solving the Problems

[0007] According to an embodiment, there is provided an image processing apparatus used when measuring an imaging distance from the imaging apparatus to a subject in an image using an image affected by aberration of an optical system of the imaging apparatus imaged by the imaging apparatus. The image processing apparatus includes a first acquisition unit, a second acquisition unit, a third acquisition unit, a creation unit, and an output processing unit. The first acquisition unit acquires an image imaged by the imaging apparatus. The second acquisition unit acquires configuration information regarding the optical system of the imaging apparatus. The third acquisition unit acquires an imaging distance with respect to the acquired image based on the acquired image. The creation unit creates first imaging conditions of an image suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance. The output processing unit outputs the created first imaging conditions. The creation means creates a first imaging condition including a direction and a distance in which the imaging device should move in order to realize an imaging distance suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Embodiments for Carrying Out the Invention

[0009] Hereinafter, each embodiment will be described with reference to the drawings. (First Embodiment) First, the first embodiment will be described. FIG. 1 shows an example of the configuration of the image processing system in this embodiment. As shown in FIG. 1, the image processing system 1 includes an imaging device 2 and an imaging condition presentation device (image processing device) 3.

[0010] The imaging condition presentation device 3 according to this embodiment is used to present the imaging conditions of the image when measuring the distance from the imaging point to the subject using the image captured by the imaging device 2 (that is, to propose the imaging conditions of the image suitable for measuring the distance).

[0011] In addition, in the present embodiment, the image processing system 1 is described as including a separate imaging device 2 and an imaging condition presentation device 3. However, the image processing system 1 may be realized as a single device in which the imaging device 2 functions as an imaging unit and the imaging condition presentation device 3 functions as an imaging condition improvement unit. In the case where the image processing system 1 includes a separate imaging device 2 and an imaging condition presentation device 3, for example, a digital camera or the like can be used as the imaging device 2, and a personal computer, a smartphone, a tablet computer, or the like can be used as the imaging condition presentation device 3. In this case, the imaging condition presentation device 3 may operate as a server device that executes, for example, a cloud computing service. On the other hand, when the image processing system 1 is realized as a single device, a digital camera, a smartphone, a tablet computer, or the like can be used as the image processing system 1.

[0012] Furthermore, when the imaging conditions of an image are improved using the imaging condition presentation device 3 as described above, the distance from the image captured by the imaging device 2 according to the improved imaging conditions to the subject is measured. The process of measuring the distance to the subject is assumed to be executed by a device separate from the imaging condition presentation device 3 (hereinafter, a distance measuring device). Note that the distance measuring device may be realized as a single device equipped with the imaging device 2, or may be integrated with the imaging condition presentation device 3.

[0013] The imaging device 2 is used to capture various images. The imaging device 2 includes a lens 21 and an image sensor 22. The lens 21 and the image sensor 22 correspond to the optical system (monocular camera) of the imaging device 2. Further, in the present embodiment, the lens 21 includes a mechanism for controlling the focus position (focal length) by adjusting the position of the lens 21, a lens drive circuit, etc., a diaphragm mechanism having an aperture for adjusting the amount of light (light incident amount) taken into the optical system of the imaging device 2, a diaphragm control circuit, etc., and a control circuit etc. on which a memory for holding configuration information regarding the optical system of the imaging device 2 such as information regarding the lens 21 and other parameters (hereinafter referred to as the configuration information of the imaging device 2) is mounted, and together they constitute a lens unit.

[0014] Further, in the present embodiment, the imaging device 2 may be configured such that the lens 21 (lens unit) can be manually replaced with another lens. In this case, a user using the imaging device 2 can mount and use, for example, one of a plurality of types of lenses such as a standard lens, a telephoto lens, and a wide-angle lens on the imaging device 2. Note that when the lens is replaced, the focal length and the F value (diaphragm value) change, and an image corresponding to the lens used in the imaging device 2 can be captured.

[0015] In the present embodiment, the focal length refers to the distance from the lens to the position where light converges when the light is incident parallel to the lens. The F value is a quantification of the amount of light taken into the imaging device 2 according to the diaphragm mechanism. Note that the F value indicates that the amount of light taken into the imaging device 2 increases (that is, the size of the aperture increases) as the value decreases.

[0016] Light reflected by the subject is incident on the lens 21. The light incident on the lens 21 passes through the lens 21. The light that has passed through the lens 21 reaches the image sensor 22 and is received (detected) by the image sensor 22. The image sensor 22 generates an image composed of a plurality of pixels by converting the received light into an electrical signal (photoelectric conversion).

[0017] Note that the image sensor 22 is realized by, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The image sensor 22 includes, for example, a first sensor (R sensor) 221 that detects light in a red (R) wavelength band, a second sensor (G sensor) 222 that detects light in a green (G) wavelength band, and a sensor (B sensor) 223 that detects light in a blue (B) wavelength band. The image sensor 22 can receive light in the corresponding wavelength bands by the first to third sensors 221 to 223 and generate sensor images (R image, G image, and B image) corresponding to each wavelength band (color component). That is, the image captured by the imaging device 2 is a color image (RGB image), and the R image, G image, and B image are included in the image.

[0018] In the present embodiment, the image sensor 22 is described as including the first to third sensors 221 to 223. However, the image sensor 22 may be configured to include at least one of the first to third sensors 221 to 223. Further, the image sensor 22 may be configured to include a sensor for generating a monochrome image, for example, instead of the first to third sensors 221 to 223.

[0019] The image generated based on the light transmitted through the lens 21 in the present embodiment is an image affected by the aberration of the optical system (the lens 21 included therein) and includes the blur caused by the aberration. Details of the blur generated in the image will be described later.

[0020] The imaging condition presentation device 3 includes, as a functional configuration, a storage unit 31, an image acquisition unit 32, a configuration information acquisition unit 33, an imaging distance acquisition unit 34, an imaging condition creation unit 35, and an output processing unit 36.

[0021] The storage unit 31 stores a statistical model (distance estimation model) used to measure the distance (depth) to the subject from the image captured by the imaging device 2. In the present embodiment, it is assumed that the above-described distance measuring device is configured to measure the distance from the image to the subject using the same statistical model stored in this storage unit 31 (that is, the statistical model used in the imaging condition presentation device 3).

[0022] The statistical model is generated by learning the blur that occurs in the image affected by the aberration of the above-described optical system and that changes non-linearly according to the distance to the subject in the image.

[0023] Note that the statistical model can be generated by applying various known machine learning algorithms such as a neural network or a random forest. Also, neural networks applicable in the present embodiment may include, for example, a convolutional neural network (CNN), a fully-connected neural network, and a recurrent neural network.

[0024] The image acquisition unit 32 acquires the image captured by the above-described imaging device 2 from the imaging device 2 (image sensor 22).

[0025] The configuration information acquisition unit 33 acquires the configuration information of the imaging device 2 when the image acquired by the image acquisition unit 32 is captured. Note that the configuration information of the imaging device 2 is embedded in the image captured by the imaging device 2 as a meta-file (EXIF information) of the image, such as EXIF (Exchangeable Image File Format), which is a format for a digital camera that can store the date and time when the image was captured and other setting data in the image. In this case, the configuration information acquisition unit 33 can acquire the configuration information of the imaging device 2 from the image (EXIF information) acquired by the image acquisition unit 32.

[0026] The imaging distance acquisition unit 34 acquires the imaging distance for the image based on the image acquired by the image acquisition unit 32. In the present embodiment, the imaging distance is the distance between the subject in the image captured by the image acquisition unit 32 and the imaging device 2 when the image is captured, and corresponds to the position of the imaging device 2 with respect to the subject as a reference. This imaging distance is acquired, for example, using the statistical model stored in the storage unit 31 and the configuration information of the imaging device 2 acquired by the configuration information acquisition unit 33.

[0027] The imaging condition creation unit 35 creates imaging conditions for an image suitable for measuring the distance from the imaging device 2 to the subject based on the configuration information of the imaging device 2 acquired by the configuration information acquisition unit 33 and the imaging distance acquired by the imaging distance acquisition unit 34.

[0028] The output processing unit 36 outputs the imaging conditions created by the imaging condition creation unit 35. In this case, the imaging conditions are presented (proposed) to, for example, the user who uses the imaging device 2 (the photographer who captures an image using the imaging device 2) as the imaging conditions for the image to realize the measurement of the appropriate distance to the subject.

[0029] FIG. 2 shows an example of the system configuration of the imaging condition presentation device 3 shown in FIG. 1. As shown in FIG. 2, the imaging condition presentation device 3 includes a CPU 301, a nonvolatile memory 302, a RAM 303, a communication device 304, an input device 305, an output device 306, and the like. Further, the imaging condition presentation device 3 has a bus 307 that interconnects the CPU 301, the nonvolatile memory 302, the RAM 303, the communication device 304, the input device 305, and the output device 306.

[0030] The CPU 301 is a processor for controlling the operations of various components within the imaging condition presentation device 3. The CPU 301 may be a single processor or may be composed of multiple processors. Here, the CPU (Central Processing Unit) is used as the processor for controlling the operations of the components for explanation purposes, but the processor may be a GPU (Graphics Processing Unit). The CPU 301 executes various programs loaded from the non-volatile memory 302 into the RAM. Programs executed by the CPU 301 include an operating system and various application programs, and the application programs include an imaging condition presentation program (image processing program) 303A.

[0031] The non-volatile memory 302 is a storage medium used as an auxiliary storage device. The RAM 303 is a storage medium used as a main storage device. Although only the non-volatile memory 302 and the RAM 303 are shown in FIG. 2, the imaging condition presentation device 3 may include other storage devices such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

[0032] In addition, in the present embodiment, the storage unit 31 shown in FIG. 1 is realized by, for example, the non-volatile memory 302 or other storage devices.

[0033] Also, in the present embodiment, some or all of the image acquisition unit 32, configuration information acquisition unit 33, imaging distance acquisition unit 34, imaging condition creation unit 35, and output processing unit 36 shown in FIG. 1 are realized by causing the CPU 301 (i.e., the computer of the imaging condition presentation device 3) to execute an imaging condition presentation program 303A, that is, by software. This imaging condition presentation program 303A may be stored in a computer-readable storage medium and distributed, or may be downloaded to the imaging condition presentation device 3 through a network. Note that some or all of these units 32 to 36 may be realized by hardware such as an IC (Integrated Circuit), or may be realized by a combination of software and hardware.

[0034] The communication device 304 is a device configured to execute wired communication or wireless communication. The communication device 304 executes communication with an external device via a network. This external device includes the imaging device 2. In this case, the imaging condition presentation device 3 receives an image and configuration information of the imaging device 2 from the imaging device 2 via the communication device 304.

[0035] The input device 305 includes, for example, a mouse or a keyboard. The output device 306 includes, for example, a display and a speaker. Note that the input device 305 and the output device 306 may be integrally configured, such as a touch screen display.

[0036] Here, in the present embodiment, a configuration is assumed in which the imaging distance is acquired using the statistical model stored in the storage unit 31, and the imaging conditions are created based on the imaging distance. Hereinafter, with reference to FIG. 3, an outline of the statistical model used in the present embodiment will be described.

[0037] In this embodiment, the statistical model is constructed to input an image captured by the imaging device 2 (the image generated by the image sensor 22). Note that the image input to the statistical model is an image affected by the aberration of the optical system (lens 21) of the imaging device 2 as described above.

[0038] The statistical model in this embodiment is generated by learning the blur that changes non-linearly according to the distance to the subject in the image as described above. According to this statistical model, a blur value (blur information) indicating the blur generated in the image according to the distance to the subject in the image input to the statistical model is estimated, and the blur value is output. As will be described later, since there is a correlation between the distance to the subject in the image and the color, size, and shape of the blur generated in the image according to the distance, the distance to the subject can be obtained by converting the blur value output from the statistical model into a distance.

[0039] In this embodiment, the distance to the subject thus obtained using the statistical model is used as the imaging distance described above.

[0040] Here, the blur generated in the image captured by the imaging device 2 in this embodiment (the blur caused by the aberration of the optical system of the imaging device 2) will be described.

[0041] First, among the blurs caused by the aberration of the optical system of the imaging device 2, chromatic aberration will be described. FIG. 4 shows the relationship between the distance to the subject and the blur generated in the image due to chromatic aberration. In the following description, the position where the image captured by the imaging device 2 is in focus is referred to as the in-focus position.

[0042] Since the refractive index of light when passing through the lens 21 with aberration is different for each wavelength band, for example, when the position of the subject is deviated from the in-focus position, the light in each wavelength band does not converge to one point but reaches different points. This appears as chromatic aberration (blur) on the image.

[0043] The upper part of FIG. 4 shows the case where the position of the subject with respect to the imaging device (image sensor 22) is farther than the in-focus position (that is, the position of the subject is behind the in-focus position).

[0044] In this case, with respect to the light 401 in the red wavelength band, an image including a relatively small blur b is generated in the image sensor 22 (first sensor 221). R On the other hand, with respect to the light 402 in the blue wavelength band, an image including a relatively large blur b is generated in the image sensor 22 (third sensor 223). B Regarding the light 403 in the green wavelength band, an image including a blur of a size intermediate between the blur b R and the blur b B is formed. Therefore, in an image captured in such a state where the position of the subject is farther than the in-focus position, a blue blur is confirmed outside the subject in the image.

[0045] On the other hand, the lower part of FIG. 4 shows the case where the position of the subject with respect to the imaging device 2 (image sensor 22) is closer than the in-focus position (that is, the position of the subject is in front of the in-focus position).

[0046] In this case, with respect to the light 401 in the red wavelength band, an image including a relatively large blur b is generated in the image sensor 22 (first sensor 221). R On the other hand, with respect to the light 402 in the blue wavelength band, an image including a relatively small blur b is generated in the image sensor 22 (third sensor 223). B Regarding the light 403 in the green wavelength band, an image including a blur of a size intermediate between the blur b R and the blur b B is generated. Therefore, in an image captured in such a state where the position of the subject is closer than the in-focus position, a red blur is observed outside the subject in the image.

[0047] Here, FIG. 4 shows an example where the lens 21 is a simple single lens. Generally, in the imaging device 2, for example, a lens that has been subjected to chromatic aberration correction (hereinafter referred to as an achromatic lens) may be used. Note that an achromatic lens is a lens that combines a convex lens with low dispersion and a concave lens with high dispersion, and it is the lens with the fewest number of lenses as a lens for correcting chromatic aberration.

[0048] FIG. 5 shows the relationship between the distance to the subject and the blur generated in the image due to chromatic aberration when the above-described achromatic lens is used as the lens 21. In the achromatic lens, the design is such that the focal positions of the blue wavelength and the red wavelength are aligned, but chromatic aberration cannot be completely removed. Therefore, when the position of the subject is farther than the in-focus position, green blur occurs as shown in the upper part of FIG. 5, and when the position of the subject is closer than the in-focus position, purple blur occurs as shown in the lower part of FIG. 5.

[0049] Note that the middle part of FIGS. 4 and 5 shows the case where the position of the subject with respect to the imaging device 2 (image sensor 22) coincides with the in-focus position. In this case, an image with less blur is generated in the image sensor 22 (first to third sensors 221 to 223).

[0050] Here, as described above, the optical system (lens unit) of the imaging device 2 is provided with a diaphragm mechanism. However, the shape of the blur generated in the image captured by the imaging device 2 also varies depending on the size of the aperture of the diaphragm mechanism. Note that the shape of the blur is referred to as the PSF (Point Spread Function) shape and indicates the light diffusion distribution that occurs when a point light source is imaged.

[0051] The upper part of FIG. 6 shows, from left to right in the order of the subject's position being closer to the imaging device 2, the PSF shape generated in the central part of the image captured by the imaging device 2 when the focal length is 50 mm, the focus position is 1500 mm, and the F value (aperture) is F1.8 in the imaging device 2 (optical system). The lower part of FIG. 6 shows, from left to right in the order of the subject's position being closer to the imaging device 2, the PSF shape generated in the central part of the image captured by the imaging device 2 when the focal length is 50 mm, the focus position is 1500 mm, and the F value (aperture) is F4 in the imaging device 2 (optical system). The center of the upper and lower parts of FIG. 6 shows the PSF shape when the subject's position coincides with the focus position.

[0052] The PSF shapes shown at the corresponding positions in the upper and lower parts of FIG. 6 are the PSF shapes when the position of the subject with respect to the imaging device 2 is the same. However, even when the position of the subject is the same, the shape of the PSF in the upper part (the PSF shape generated in the image captured with an F value of F1.8) is different from the shape of the PSF in the lower part (the PSF shape generated in the image captured with an F value of F4).

[0053] Furthermore, as shown in the leftmost and rightmost PSF shapes in FIG. 6, even when the distance from the subject's position to the focus position is approximately the same, the PSF shape is different when the subject's position is closer to the focus position and when the subject's position is farther from the focus position.

[0054] Note that, as described above, the phenomenon that the PSF shape varies according to the size of the aperture of the aperture mechanism and the position of the subject with respect to the imaging device 2 also occurs in the same manner in each channel (RGB image, R image, G image, and B image). FIG. 7 shows the PSF shapes generated in the images of each channel captured by the imaging device 2 when the focal length is 50 mm, the focus position is 1500 mm, and the F value is F1.8, divided into the case where the position of the subject is closer (in front) than the focus position and the case where the position of the subject is farther (behind) than the focus position. FIG. 8 shows the PSF shapes generated in the images of each channel captured by the imaging device 2 when the focal length is 50 mm, the focus position is 1500 mm, and the F value is F4, divided into the case where the position of the subject is closer than the focus position and the case where the position of the subject is farther than the focus position.

[0055] Furthermore, the PSF shape generated in the image captured by the imaging device 2 also varies depending on the position in the image.

[0056] The upper part of FIG. 9 shows the PSF shapes generated at each position in the image captured by the imaging device 2 when the focal length is 50 mm, the focus position is 1500 mm, and the F value is F1.8, divided into the case where the position of the subject is closer than the focus position and the case where the position of the subject is farther than the focus position.

[0057] The middle part of FIG. 9 shows the PSF shapes generated at each position in the image captured by the imaging device 2 when the focal length is 50 mm, the focus position is 1500 mm, and the F value is F4, divided into the case where the position of the subject is closer than the focus position and the case where the position of the subject is farther than the focus position.

[0058] As shown in the upper and middle parts of FIG. 9, in the vicinity of the edge of the image captured by the imaging device 2 (particularly, in the vicinity of the corners such as the upper left), a PSF shape different from, for example, the PSF shape located near the center of the image can be observed.

[0059] Also, the lower part of FIG. 9 shows the PSF shapes generated at each position in the image captured by the imaging device 2 when the imaging device 2 uses a lens with a focal length of 105 mm, the focus position is 1500 mm, and the F value is F4, divided into the case where the subject position is closer than the focus position and the case where the subject position is farther than the focus position.

[0060] The upper and middle parts of FIG. 9 described above show the PSF shapes generated in the images captured using the same lens. However, as shown in the lower part of FIG. 9, when lenses with different focal lengths are used, different PSF shapes (PSF shapes different from those in the upper and middle parts of FIG. 9) corresponding to the lenses are observed.

[0061] Next, with reference to FIG. 10, the position dependence of the PSF shape (lens aberration) according to the type of lens used in the optical system of the imaging device 2 described above will be specifically described. FIG. 10 shows the PSF shapes generated near the center (center of the screen) and near the edge (edge of the screen) of the images captured using each of a plurality of lenses with different focal lengths, divided into the case where the subject position is closer than the focus position and the case where the subject position is farther than the focus position.

[0062] As shown in FIG. 10, the PSF shapes generated near the center of the image are generally circular and the same even when the types of lenses are different. However, the PSF shapes generated near the edge of the image have shapes different from those generated near the center of the image, and their characteristics (features) are different according to the type of lens. Regarding the fact that purple blur occurs near the edge of the PSF shape when the subject position is closer than the focus position and green blur occurs near the edge of the PSF shape when the subject position is farther than the focus position, as described in FIG. 5 above, this is common even when the types of lenses are different.

[0063] Also, in FIG. 10, two examples (#1 and #2) are shown for a lens with a focal length of 50 mm, which indicates that the focal lengths are the same at 50 mm but the lens manufacturers are different (i.e., they are different products). The same applies to the lens with a focal length of 85 mm.

[0064] As described above, the blur that changes non-linearly according to the distance to the subject in the present embodiment includes the blur caused by chromatic aberration of the optical system of the imaging device 2 described in FIGS. 4 and 5 above, the blur that occurs according to the size of the aperture (i.e., the F value) of the aperture mechanism that adjusts the amount of light taken into the optical system of the imaging device 2 described in FIGS. 6 to 8, the blur that changes according to the position in the image captured by the imaging device 2 described in FIGS. 9 and 10, and the like.

[0065] Note that the PSF shape also varies depending on the shape of the aperture of the aperture mechanism. Here, FIG. 11 shows the relationship between the non-linearity (asymmetry) of the PSF shape and the shape of the aperture of the aperture mechanism. The non-linearity of the PSF shape described above is likely to occur when the shape of the aperture of the aperture mechanism is other than a circle. In particular, the non-linearity of the PSF shape is more likely to occur when the shape of the aperture is an odd-sided polygon or an even-sided polygon arranged asymmetrically with respect to the horizontal or vertical axis of the image sensor 22.

[0066] In the present embodiment, a blur value indicating the blur occurring in the image is estimated (predicted) using a statistical model generated by focusing on the fact that the blur (color, size, and shape) occurring in the above-described image serves as a physical clue regarding the distance to the subject. Note that the blur value estimated by the statistical model in the present embodiment (i.e., output from the statistical model) is a scalar quantity representing the amount of blur including the color, size, and shape of the blur occurring in the image.

[0067] Hereinafter, an example of a method for estimating blur (a blur value indicating the blur) from an image using a statistical model in the present embodiment will be described. Here, the first to third methods will be described.

[0068] First, referring to FIG. 12, the first method will be described. In the first method, a local region (image patch) 501a is extracted from the image 501.

[0069] In this case, for example, the entire region of the image 501 may be divided in a matrix form, and the divided partial regions may be sequentially extracted as the local regions 501a, or the image 501 may be recognized, and the local regions 501a may be extracted so as to cover the detected region of the subject (image). Also, the local region 501a may partially overlap with other local regions 501a.

[0070] Next, for each of the extracted local regions 501a, by inputting the information regarding the local region 501a (information of the image 501) into the statistical model, a blur value indicating the blur generated according to the distance to the subject in the local region 501a is estimated.

[0071] The statistical model into which the information regarding the local region 501a is input in this way estimates a blur value 502 for each pixel constituting the local region 501a.

[0072] Here, for example, when a specific pixel belongs to both the first local region 501a and the second local region 501a (that is, the regions including the pixel overlap between the first local region 501a and the second local region 501a), the blur value estimated assuming that the pixel belongs to the first local region 501a and the blur value estimated assuming that the pixel belongs to the second local region 501a may be different.

[0073] Therefore, for example, when a plurality of local regions 501a, some of which overlap as described above, are extracted, the blur value of the pixels constituting the overlapping region of the plurality of local regions 501a may be, for example, the average value of the blur value estimated for a part of the region (pixels) of one of the overlapping local regions 501a and the blur value estimated for a part of the region (pixels) of the other local region 501a. Also, it may be determined by a majority vote based on the blur values estimated for each part of three or more local regions 501a that partially overlap.

[0074] FIG. 13 shows an example of information regarding the local region 501a input to the statistical model in the first method described above.

[0075] As shown in FIG. 13, gradient data of the local region 501a extracted from the image 501 is input to the statistical model. The gradient data of the local region 501a is generated from each of the R image, G image, and B image included in the image 501, and includes the gradient data of the R image, the gradient data of the G image, and the gradient data of the B image.

[0076] Note that the gradient data indicates the difference (difference value) between the pixel value of each pixel and the pixel value of the pixel adjacent to the pixel. For example, when the local region 501a is extracted as a rectangular region of n pixels (in the X-axis direction) × m pixels (in the Y-axis direction), gradient data in which the difference values, for example, with the right adjacent pixel calculated for each pixel in the local region 501a are arranged in an n-row × m-column matrix is generated.

[0077] The statistical model estimates a blur value indicating the blur occurring in each of the images using the gradient data of the R image, the gradient data of the G image, and the gradient data of the B image. Although FIG. 13 shows the case where the gradient data of each of the R image, G image, and B image is input to the statistical model, a configuration in which the gradient data of the image 501 (RGB image) is input to the statistical model may also be used.

[0078] Next, referring to FIG. 14, the second method will be described. In the second method, as information regarding the local region 501a in the first method, the gradient data for each of the local regions (image patches) 501a and the position information of the local region 501a in the image 501 are input into the statistical model.

[0079] The position information 501b may indicate, for example, the center point of the local region 501a, or may indicate a predetermined side such as the upper left side. Also, as the position information 501b, the positions on the image 501 of each of the pixels constituting the local region 501a may be used.

[0080] By further inputting the position information 501b into the statistical model as described above, it is possible to estimate a blur value 502 that takes into account the difference between the blur of the subject image formed by the light passing through the central portion of the lens 21 and the blur of the subject image formed by the light passing through the end portion of the lens 21.

[0081] That is, according to this second method, it is possible to estimate the blur value from the image 501 based on the correlation with the position on the image.

[0082] FIG. 15 shows an example of the information regarding the local region 501a input into the statistical model in the second method described above.

[0083] For example, when a rectangular region of n pixels (in the X-axis direction) × m pixels (in the Y-axis direction) is extracted as the local region 501a, the X coordinate value (X coordinate data) on the image 501 corresponding to, for example, the center point of the local region 501a and the Y coordinate value (Y coordinate data) on the image 501 corresponding to, for example, the center point of the local region 501a are acquired.

[0084] In the second method, the X coordinate data and the Y coordinate data thus acquired are input into the statistical model together with the gradient data of the R image, G image, and B image described above.

[0085] Furthermore, referring to FIG. 16, the third method will be described. In the third method, extraction of the local region 501a from the image 501 as in the above-described first and second methods is not performed. In the third method, for example, information regarding the entire region of the image 501 (gradient data of the R image, G image, and B image) is input into the statistical model.

[0086] Compared with the first and second methods of estimating the blur value 502 for each local region 501a, the third method may have a higher uncertainty in estimation by the statistical model, but the processing load for the estimation can be reduced.

[0087] In the following description, the information input into the statistical model in the above-described first to third methods is, for convenience, referred to as information regarding the image.

[0088] Although the description has been made assuming that the blur value is estimated for each pixel here, the blur value may be estimated for each predetermined region including at least one pixel.

[0089] Hereinafter, referring to FIG. 17, the correlation between the blur occurring in the image in the present embodiment and the distance to the subject in the image will be specifically described.

[0090] In FIG. 17, the size of the blur occurring when the subject is closer (in front) than the in-focus position is shown as a negative value on the X-axis, and the size of the blur occurring when the subject is farther (behind) than the in-focus position is shown as a positive value on the X-axis. That is, in view of the fact that the colors observed in the blur occurring in the subject are different when the position of the subject is closer than the in-focus position and when the position of the subject is farther than the in-focus position as described above, it can be said that the color and size of the blur are shown as positive and negative values in FIG. 17.

[0091] Also, in FIG. 17, it is shown that in both the case where the subject position is closer than the in-focus position and the case where the subject position is farther than the in-focus position, the absolute value of the blur size (pixels) increases as the subject moves away from the in-focus position.

[0092] In the example shown in FIG. 17, it is assumed that the in-focus position in the optical system of the imaging device 2 that captured the image is about 1500 mm. In this case, for example, a blur of about -4.8 pixels corresponds to a distance of about 1000 mm from the optical system, a blur of 0 pixels corresponds to a distance of 1500 mm from the optical system, and a blur of about 4.8 pixels corresponds to a distance of about 750 mm from the optical system.

[0093] Here, for the sake of convenience, the case where the blur color and size (pixels) are shown on the X-axis has been described. However, as described in FIGS. 6 to 10 above, the shape of the blur (PSF shape) generated in the image also differs between the case where the subject is closer than the in-focus position and the case where the subject is farther than the in-focus position, and also differs depending on the position in the image. Therefore, the value shown on the X-axis in FIG. 17 (that is, the blur value) is actually a value that reflects the shape of the blur (PSF shape).

[0094] Since there is a correlation such as that shown by the line segment d1 in FIG. 17 between the distance to the subject described above and the blur color, size, and shape, estimating the distance is synonymous with estimating the blur color, size, and shape (the blur value indicating them).

[0095] Note that, for example, a configuration in which the distance is directly estimated by a statistical model can also be considered. However, a configuration in which the blur value is estimated by the statistical model can use the same statistical model even when the in-focus position (focal length) in the optical system is changed, and it can be said that the versatility is high.

[0096] In this embodiment, by using the above-described statistical model, it is possible to obtain (estimate) a blur value indicating the blur generated in the image according to the distance from the image to the subject in the image. The statistical model is generated by executing a process (hereinafter referred to as a learning process) of learning the blur (a blur that changes non-linearly according to the distance from the subject in the image) generated in an image affected by the aberration of the optical system of the imaging device 2 as described above.

[0097] Hereinafter, an outline of the learning process (hereinafter referred to as the learning process of the statistical model) executed to generate the statistical model will be described.

[0098] FIG. 18 shows an example of the learning process (learning method) of the statistical model in this embodiment. The learning process of the statistical model is executed using an image (hereinafter referred to as a learning image) prepared to learn the statistical model. The learning image may be, for example, an image captured by the imaging device 2, or may be an image captured by another device (such as a camera) having an optical system similar to the optical system of the imaging device 2.

[0099] In any of the first method described with reference to FIG. 12, the second method described with reference to FIG. 14, and the third method described with reference to FIG. 16, the learning process of the statistical model basically inputs the learning image 601 (information related thereto) into the statistical model, and feeds back the error between the blur value 602 estimated by the statistical model and the correct value 603 to the statistical model. Note that feedback means updating the parameters (for example, weight coefficients) of the statistical model so that the error decreases.

[0100] When the first method is applied as a method for estimating the blur value from the above-described image, even during the learning process of the statistical model, for each local region (image patch) extracted from the learning image 601, information (gradient data) regarding the local region is input into the statistical model, and the blur value 602 of each pixel in each local region is estimated by the statistical model. The error obtained by comparing the thus estimated blur value 602 with the correct value 603 is fed back to the statistical model.

[0101] Similarly, when the second method is applied as a method for estimating the blur value from the image, even during the learning process of the statistical model, for each local region (image patch) extracted from the learning image 601, gradient data and position information as information regarding the local region are input into the statistical model, and the blur value 602 of each pixel in each local region is estimated by the statistical model. The error obtained by comparing the thus estimated blur value 602 with the correct value 603 is fed back to the statistical model.

[0102] Also, when the third method is applied as a method for estimating the distance from the image, even during the learning process of the statistical model, information (gradient data) regarding the entire region of the learning image 601 is collectively input into the statistical model, and the blur value 602 of each pixel in the learning image 601 is estimated by the statistical model. The error obtained by comparing the thus estimated blur value 602 with the correct value 603 is fed back to the statistical model.

[0103] According to the above-described learning process of the statistical model, the parameters of the statistical model are updated so that the error between the blur value 602 and the correct value 603 decreases, and the statistical model can learn the blur occurring in the learning image 601.

[0104] Note that the statistical model in this embodiment is generated by repeatedly executing learning processing using learning images captured while changing the distance from the imaging device 2 to the subject with the focus position fixed, for example. Further, when the learning processing for one focus position is completed, the learning processing can be similarly executed for other focus positions to generate a more accurate statistical model.

[0105] Also, the correct value used in the learning processing of the statistical model in this embodiment shall be a blurred value (that is, a blurred value indicating the color, size, and shape of the blur corresponding to the actual distance) converted from the actual distance to the subject when the learning image was captured as described above.

[0106] Next, with reference to the flowchart of FIG. 19, an example of the processing procedure of the learning processing of the statistical model will be described. Note that the processing shown in FIG. 19 may be executed, for example, in the imaging condition presentation device 3, or may be executed in other devices or the like.

[0107] First, a learning image (information related thereto) prepared in advance is input to the statistical model (step S1). This learning image is, for example, an image generated by the image sensor 2 based on the light transmitted through the lens 21 provided in the imaging device 2, and is an image affected by the aberration of the optical system (lens 21) of the imaging device 2. Specifically, the learning image has a blur that changes non-linearly according to the distance to the subject described in FIGS. 4 to 10.

[0108] Note that in the learning processing of the statistical model, it is assumed that learning images in which the subject is imaged at each distance with as fine a granularity as possible from the lower limit value (near) to the upper limit value (far) of the distance measurable (estimable) using the statistical model are prepared in advance. Further, it is preferable to prepare various images with different subjects as the learning images.

[0109] When the first method described above is applied as a method for estimating the blur value from an image, as information regarding the learning image, gradient data of the R image, G image, and B image is input into the statistical model for each local region of the learning image.

[0110] When the second method described above is applied as a method for estimating the blur value from an image, as information regarding the learning image, gradient data of the R image, G image, and B image and position information on the learning image of the local region are input into the statistical model for each local region of the learning image.

[0111] When the third method described above is applied as a method for estimating the blur value from an image, as information regarding the learning image, gradient data of the R image, G image, and B image for the entire region of the learning image is input into the statistical model.

[0112] In the present embodiment, it is described that gradient data of the R image, G image, and B image is input into the statistical model. However, when the statistical model estimates the blur value from the viewpoint of the shape of the blur (PSF shape) generated in the image, at least one of the gradient data of the R image, G image, and B image may be input into the statistical model. On the other hand, when the statistical model estimates the blur value from the viewpoint of the color and size of the blur generated in the image due to chromatic aberration, at least two of the gradient data of the R image, G image, and B image may be input into the statistical model.

[0113] When the process of step S1 is executed, the blur value is estimated by the statistical model (step S2).

[0114] The blur value estimated in step S2 is compared with the correct value obtained at the time of imaging the learning image (step S3).

[0115] The comparison result (error) in step S3 is fed back to the statistical model (step S4). As a result, in the statistical model, the parameters are updated so that the error decreases (that is, the blur occurring in the learning image is learned according to the distance to the subject).

[0116] By repeatedly executing the process shown in FIG. 19 described above for each learning image, a statistical model with high estimation accuracy is generated. The statistical model generated in this way is stored in the storage unit 31 included in the imaging condition presentation device 3.

[0117] In the present embodiment, the above-described statistical model is also held in the distance measuring device, and the distance measuring device inputs the image captured by the imaging device 2 into the statistical model and converts the blur value output from the statistical model into a distance, thereby measuring (i.e., measuring the distance) to the subject in the image.

[0118] That is, the imaging condition presentation device 3 according to the present embodiment acquires the imaging distance using the statistical model (the same statistical model) in consideration of the fact that the distance measuring device measures the distance to the subject from the image using the statistical model, and creates and outputs imaging conditions (conditions for capturing an image suitable for measuring the distance to the subject) based on the acquired imaging distance, thereby improving the accuracy of the distance to the subject measured by the distance measuring device.

[0119] Hereinafter, with reference to the flowchart of FIG. 20, an example of the processing procedure of the imaging condition presentation device 3 according to the present embodiment will be described.

[0120] First, the image acquisition unit 32 acquires an image including a subject captured by the imaging device 2 (image sensor 22) (hereinafter referred to as a captured image) (step S11). This captured image is an image affected by the aberration of the optical system (lens 21) of the imaging device 2 as described above. The captured image acquired in step S11 is passed to the imaging distance acquisition unit 34.

[0121] Note that the captured image (image data) passed from the image acquisition unit 32 to the imaging distance acquisition unit 34 may be, for example, RAW data, or may be data after predetermined image processing (data processing) such as development processing for converting the RAW data into a predetermined format or processing for changing the resolution. Note that the image processing executed on the captured image includes, for example, preprocessing for inputting the captured image into a statistical model and processing for reducing the processing time (processing amount) in the imaging condition presentation device 3.

[0122] Next, the configuration information acquisition unit 33 acquires the configuration information of the imaging device 2 when the captured image is captured, for example, from the captured image (EXIF information thereof) (step S12). In the present embodiment, the configuration information of the imaging device 2 is information regarding the optical system of the imaging device 2 and includes, for example, the focus distance. In the present embodiment, the focus distance is the distance to the in-focus position in the captured image described above (the in-focus position in the captured image) and is determined by the optical system of the imaging device 2 (the distance between the lens 21 and the image sensor 22). Further, the configuration information of the imaging device 2 is assumed to further include the focal length, F value, position of the lens 21, and size of the image sensor 22 (size of one pixel in the image sensor 22) in the optical system of the imaging device 2. The configuration information of the imaging device 2 acquired in step S12 is passed to the imaging distance acquisition unit 34 and the imaging condition creation unit 35.

[0123] The imaging distance acquisition unit 34 acquires the imaging distance with respect to the captured image (that is, the current imaging distance) (step S13). The processing in this step S13 includes distance estimation processing using the statistical model stored in the storage unit 31.

[0124] The processing of step S13 will be described below. First, the imaging distance acquisition unit 34 inputs the captured image passed from the image acquisition unit 32 into the statistical model to obtain the blur value output from the statistical model (that is, the blur value estimated in the statistical model). Note that the process of obtaining this blur value corresponds to the processes of steps S1 and S2 shown in FIG. 19 described above, and thus the detailed description thereof is omitted here.

[0125] Next, the imaging distance acquisition unit 34 converts the obtained blur value into a distance. Note that the configuration information of the imaging device 2 passed from the configuration information acquisition unit 33 is used for the conversion from the blur value to the distance. Specifically, the distance u is represented by the following formula (1) using the blur value b.

Equation

[0126] In formula (1), f represents the focal length in the optical system of the imaging device 2 that captured the captured image. u f represents the distance to the in-focus position in the captured image (that is, the focus distance). F represents the F value (aperture value) in the optical system of the imaging device 2 that captured the captured image.

[0127] That is, in the present embodiment, the distance can be calculated by applying the blur value obtained using the statistical model as described above, the focus distance, the focal length, and the F value included in the configuration information of the imaging device 2 to formula (1).

[0128] In step S13, the distance calculated as described above (that is, the distance converted from the blur value) is obtained as the imaging distance.

[0129] Here, when the captured image is input into the statistical model, the statistical model outputs a blur value for each pixel constituting the captured image. Therefore, the imaging distance acquisition unit 34 converts each of the blur values output for each pixel into a distance (that is, calculates the distance for each pixel). In this case, the imaging distance acquisition unit 34 acquires (determines) the imaging distance based on the distances calculated for each pixel.

[0130] Specifically, the imaging distance acquisition unit 34 calculates a statistical value (statistic), such as an average value or a median value of the distances calculated for each pixel, and acquires the statistical value as the imaging distance.

[0131] Here, it has been described that the statistical value of the distances calculated for each pixel is acquired as the imaging distance. However, the distance calculated for a specific pixel (for example, a pixel located at the center of the captured image or a pixel designated by the user, etc.) among the plurality of pixels constituting the captured image may be acquired as the imaging distance.

[0132] Alternatively, for example, the imaging distance may be acquired based on the distances calculated for each pixel constituting a region of interest in the captured image. Specifically, for example, a region including a subject in the captured image is extracted as the region of interest, and the statistical value of the distances calculated for each pixel constituting the region of interest can be acquired as the imaging distance.

[0133] The region of interest in this case may be, for example, a region having a rectangular shape or the like, but may also be a region having a shape along the contour of the subject (that is, a region like a mask corresponding to the shape of the subject).

[0134] Note that the target area may be automatically extracted by performing image processing on the captured image. For example, it may be extracted using a pre-trained machine learning model. Examples of the pre-trained machine learning model used to extract such a target area include a machine learning model trained to extract (present) an area (such as a rectangular area) including the subject by detecting the subject (object) in the image, and a machine learning model trained to extract an area composed of pixels classified into the same subject (category corresponding thereto) by classifying each of a plurality of pixels constituting the image on a pixel-by-pixel basis. However, other machine learning models may be used. Further, the target area may be manually specified by the user.

[0135] Furthermore, in the present embodiment, the statistical model has been described as estimating the blur value for each pixel constituting the image. However, the statistical model may be constructed to estimate the blur value and calculate an uncertainty indicating the degree of uncertainty of the blur value (that is, output the blur value and the uncertainty when an image is input). When such a statistical model is used, since the uncertainty with respect to the distance (distance calculated for each pixel) converted from the blur value estimated in the statistical model can be obtained, the imaging distance may be obtained using the uncertainty. Specifically, for example, a distance having an uncertainty equal to or greater than a predetermined value (threshold value) may be excluded from the target for calculating the statistical value to be obtained as the imaging distance, and the statistical value of the distance having an uncertainty less than the threshold value may be obtained as the imaging distance.

[0136] Note that although the method for obtaining a plurality of imaging distances has been described here, the method for obtaining the imaging distance employed in the present embodiment may be, for example, preset by the user or dynamically changed according to the image.

[0137] When the process of step S13 is executed, the imaging distance obtained in step S13 is passed to the imaging condition creation unit 35.

[0138] Next, the imaging condition creation unit 35 creates imaging conditions based on the configuration information of the imaging device 2 passed from the configuration information acquisition unit 33 and the imaging distance passed from the imaging distance acquisition unit 34 (step S14). Note that the imaging conditions created in step S14 are the imaging conditions of the image for improving the ranging accuracy using the above-described statistical model.

[0139] Here, generally, the accuracy of image processing depends on the sharpness of the image. However, when the image includes a blur of a large size, it is considered that the accuracy of the image processing decreases. According to this, in the configuration of measuring the distance to the subject in the image using the above-described statistical model (that is, converting the blur value output from the statistical model into a distance by inputting the image into the statistical model), for example, when the subject to be ranged exists near the in-focus position, the measurement accuracy of the distance to the subject becomes high. On the other hand, when the blur is large, it becomes difficult to capture the characteristics of the blur. Therefore, as the subject moves away from the in-focus position (that is, the blur becomes larger), the measurement accuracy of the distance to the subject becomes lower.

[0140] Therefore, in step S14, based on such a viewpoint, for example, imaging conditions are created that propose changing the distance between the subject and the imaging device 2 so that the distance between the subject and the imaging device 2 approaches the in-focus distance (that is, the position of the subject is near the in-focus position).

[0141] Specifically, in step S14, imaging conditions including, for example, the in-focus distance are created as the imaging distance (the distance between the subject and the imaging device 2). According to such imaging conditions, it is possible to propose (instruct) to set the distance between the subject and the imaging device 2 to the in-focus distance.

[0142] In addition, in order for the user to more intuitively grasp the imaging conditions, the imaging conditions may include the direction and distance in which the imaging device 2 should move (hereinafter referred to as the moving direction and moving amount of the imaging device 2) in order to achieve the above-described imaging distance (that is, to image the subject at the focus distance). In this case, the moving direction and moving amount of the imaging device 2 are determined by comparing, for example, the focus distance included in the configuration information of the imaging device 2 acquired in step S12 with the imaging distance acquired in step S13.

[0143] Specifically, for example, when the imaging distance acquired in step S13 is longer than the focus distance, the moving direction of the imaging device 2 is the forward direction (that is, the direction approaching the subject). On the other hand, for example, when the imaging distance acquired in step S13 is shorter than the focus distance, the moving direction of the imaging device 2 is the backward direction (that is, the direction moving away from the subject).

[0144] In addition, when the attention area is extracted from the captured image when acquiring the imaging distance (that is, the subject is detected), depending on the position of the attention area (subject) on the captured image, a direction divided at a certain angle such as the left direction or the right direction may be used as the moving direction of the imaging device 2. Furthermore, the moving direction of the imaging device 2 may be an angle (for example, 30° etc.) corresponding to the position of the attention area on the captured image based on the direction of the imaging device 2 at the time of imaging. Also, for example, if it is possible to determine the orientation of the subject included in the attention area based on the captured image, the moving direction of the imaging device 2 may be, for example, the direction of the position where the subject can be imaged from the front (that is, the imaging angle with respect to the subject).

[0145] Also, the moving amount of the imaging device 2 may be an actual distance based on the difference between the focus distance and the imaging distance, but may also be a distance defined by a certain division such as the number of steps of the user using the imaging device 2 or a grid (that is, a unit serving as a guide when the user moves).

[0146] Also, although the moving direction and amount of movement of the imaging device 2 have been described as being included in the imaging conditions here, if the distance between the subject and the imaging device 2 approaches the focus distance, the imaging conditions may include, for example, the moving direction and amount of movement of the subject.

[0147] Also, the range (lower limit value and upper limit value) of the distance to the subject that can be measured with a certain accuracy using the statistical model changes based on the optical system (focus position, focal length, F value, etc.) of the imaging device 2. In step S14, the imaging condition creation unit 35 may estimate the range of the distance (hereinafter referred to as the distance measurement possible range) based on the configuration information of the imaging device 2 and create imaging conditions based on the distance measurement possible range. In this case, the imaging condition creation unit 35 can propose imaging of an image such that, for example, the entire subject is within the distance measurement possible range by creating imaging conditions including the estimated distance measurement possible range as the imaging distance. Also, such imaging conditions may further include an angle of view or the like necessary to fit the entire subject within the distance measurement possible range.

[0148] Furthermore, the imaging conditions created in step S14 may include a set value of the imaging device 2 determined based on the configuration information of the imaging device 2 and the imaging distance, instead of the above-described imaging distance. Note that the set value of the imaging device 2 may be, for example, the amount of change in the focus distance (i.e., the focus position) such that the focus distance approaches the distance between the current subject and the imaging device 2 (i.e., the imaging distance), or may be the focal length, F value, etc. for changing the distance measurement possible range so that the position of the subject with respect to the imaging device 2 is included in the distance measurement possible range.

[0149] In addition, the imaging condition creation unit 35 may be configured to create imaging conditions (conditions related to distance) including the above-described imaging distance and imaging conditions (conditions related to the optical system) including the set values of the imaging device 2, and determine which of the two imaging conditions to prioritize. In this case, for example, when the distance that the imaging device 2 needs to move based on the imaging distance (the difference between the focus distance and the current imaging distance) is less than a predetermined value, it can be determined to prioritize the conditions related to distance. On the other hand, for example, when the distance that the imaging device 2 needs to move based on the imaging distance is greater than or equal to a predetermined value, it can be determined to prioritize the conditions related to the optical system. Also, it may be configured to determine the imaging conditions to be prioritized based on the relationship between the angle of view in the optical system of the imaging device 2 and the subject (for example, whether the subject is appropriately included in the angle of view). When changing the set values of the imaging device 2 may have an adverse effect on image capture (for example, blooming due to opening the aperture), it may be determined to prioritize the conditions related to distance. Note that the imaging conditions to be prioritized may be specified by, for example, the user who uses the imaging device 2.

[0150] When the process of step S14 is executed, the output processing unit 36 outputs the imaging conditions created in step S14 (step S15).

[0151] Hereinafter, the process of step S15 will be described. In step S15, the output processing unit 36 presents the imaging conditions to the user (that is, presents the conditions for prompting imaging suitable for measuring the distance) by outputting the imaging conditions to the output device 306, for example.

[0152] For example, when the output device 306 is a display, the imaging conditions created in step S14 are displayed in a form that can be understood by the user.

[0153] Specifically, when the imaging conditions include the imaging distance (that is, the focus distance), the output processing unit 36 can display the imaging distance superimposed on, for example, the captured image.

[0154] Also, when the imaging conditions include the moving direction and the moving amount of the imaging device 2 for realizing the imaging distance, for example, when the moving direction of the imaging device 2 is the forward direction and the moving amount of the imaging device 2 is 1 m, the output processing unit 36 superimposes and displays an arrow indicating the forward direction (moving forward) on the captured image and displays "1 m" near the arrow. In this case, for example, a marker (a figure such as a line) may be arranged (displayed) at a position on the captured image where the distance estimated by a statistical model (the distance converted from the blur value) is 1 m. According to this, it becomes possible for the user to intuitively grasp the position where the imaging device 2 should move.

[0155] Here, the case where the imaging conditions include the imaging distance (the moving direction and the moving amount of the imaging device 2) has been described. However, when the imaging conditions include the set value of the imaging device 2, the set value of the imaging device 2 may be superimposed and displayed on the captured image.

[0156] Note that the imaging conditions do not necessarily need to be displayed on the captured image. For example, they may be displayed on a distance map (a map in which the distance calculated for each pixel is assigned to each pixel), or may be displayed in other ways.

[0157] Also, here, the case where the output device 306 is a display has been described. However, when the output device 306 is a speaker, the imaging conditions may be output as sound and presented to the user.

[0158] That is, in the present embodiment, the imaging conditions may be output (proposed) in a manner that can be grasped by the user.

[0159] Although omitted in FIG. 20, the processes of steps S14 and S15 described above may be executed when the difference between the focus distance included in the configuration information acquired in step S12, for example, and the imaging distance acquired in step S13 is equal to or greater than a predetermined value. In other words, in the present embodiment, when the position of the subject is not near the in-focus position and it is necessary to change the imaging conditions, a configuration may be adopted in which the imaging conditions are created and output. When the difference between the focus distance included in the configuration information acquired in step S12 and the imaging distance acquired in step S13 is less than the predetermined value, step S14 may be omitted, and it may be sufficient to present (output) to the user that there is no need to change the imaging conditions in step S15.

[0160] When the imaging conditions are presented to the user by executing the process of step S15 as described above, the user can refer to the imaging conditions and change the current imaging conditions (that is, change to imaging conditions suitable for measuring the distance).

[0161] Specifically, when the imaging distance is included in the imaging conditions, the user may move the imaging device 2 (or the subject) so as to achieve the imaging distance. When the focal length is included as a set value of the imaging device 2 in the imaging conditions, the user can change the focal length within the range set in the lens 21, for example, or may change the focal length by replacing the lens 21. When the F value is included as a set value of the imaging device 2 in the imaging conditions, the user can change the F value by driving (operating) the aperture mechanism described above.

[0162] When the current imaging conditions are thus changed to imaging conditions suitable for measuring the distance, the distance measuring device can measure the distance from the imaging device 2 to the subject in the image captured by the imaging device 2 under the imaging conditions. It is assumed that the distance measuring device measures the distance to the subject using the same statistical model as the statistical model stored in the storage unit 31 described above.

[0163] Here, it has been described that the imaging conditions are presented to the user so that the user can change (improve) the imaging conditions. However, the output processing unit 36 may output the imaging conditions so that the current imaging conditions are automatically changed to imaging conditions suitable for measuring the distance. That is, in the present embodiment, the imaging conditions may be automatically changed based on the output imaging conditions (proposal results). Specifically, when the imaging distance (the moving direction and moving amount of the imaging device 2) is included in the imaging conditions and the imaging device 2 is mounted on a moving mechanism that autonomously moves the imaging device 2, the imaging conditions are output to the moving mechanism, and the imaging device 2 may be automatically moved to a position where the imaging distance included in the output imaging conditions can be realized. Further, when the set value of the imaging device 2 is included in the imaging conditions, the imaging conditions are output to the imaging device 2, and the current set value of the imaging device 2 may be automatically changed to the set value of the imaging device 2.

[0164] By the way, when converting the blur value estimated in the above statistical model to a distance, it is necessary to calculate the distance using the focus distance. When the focus distance is changed based on the imaging conditions (the set value of the imaging device 2), it is assumed that the changed focus distance is reset inside the imaging device 2 (that is, as configuration information of the imaging device 2). Note that such a changed focus distance may be reset by the user who uses the imaging device 2, or may be automatically reset based on information regarding the focus position (for example, the position of the lens 21 or a control signal for driving the lens 21).

[0165] As described above, in the present embodiment, an image captured by the imaging device 2 (an image affected by the aberration of the optical system of the imaging device 2) is acquired, configuration information of the imaging device 2 (information regarding the optical system of the imaging device 2) is acquired, and based on the acquired image, an imaging distance for the image is acquired. Further, in the present embodiment, based on the configuration information of the imaging device 2 and the imaging distance, imaging conditions (imaging conditions of an image suitable for measuring the distance to the subject) are created, and the created imaging conditions are output.

[0166] When the imaging conditions are output in this way, since the distance from the subject in the image is measured from the image captured by the imaging device 2 according to the imaging conditions, it is possible to improve the accuracy of the distance measured using the image.

[0167] Note that in the present embodiment, it is assumed that the distance measuring device measures the distance from the subject in the image affected by the aberration of the optical system of the imaging device 2 using a statistical model (a statistical model generated by learning the blur that changes non-linearly according to the distance to the subject in the image affected by the aberration of the optical system of the imaging device 2). Therefore, the imaging condition presentation device 3 also creates (proposes) imaging conditions based on the imaging distance obtained using the statistical model. However, the distance measuring device and the imaging condition presentation device 3 may be configured not to use the statistical model (that is, to perform distance measurement using aberration not based on the statistical model).

[0168] Further, in the present embodiment, by creating imaging conditions including an imaging distance (the direction and distance in which the imaging device 2 should move to realize the imaging distance) suitable for measuring the distance based on the configuration information of the imaging device 2 and the imaging distance for the captured image, the position of the current imaging device 2 can be changed to an appropriate position (that is, the imaging device 2 can be moved) in order to capture an image suitable for measuring the distance to the subject.

[0169] Also, in the present embodiment, by creating imaging conditions including the setting value of the imaging device 2 based on the configuration information of the imaging device 2 and the imaging distance with respect to the captured image, the current setting value of the imaging device 2 can be changed to an appropriate setting value in order to capture an image suitable for measuring the distance to the subject.

[0170] Note that the above-described imaging distance and the setting value of the imaging device 2 can be determined based on the focus distance (the distance to the position where the image captured by the imaging device 2 is in focus) included in the configuration information of the imaging device 2.

[0171] Furthermore, in the present embodiment, a configuration may be adopted in which imaging conditions including the above-described imaging distance and imaging conditions including the setting value of the imaging device 2 are created, and one of the imaging conditions is preferentially output. According to such a configuration, appropriate imaging conditions are output (for example, presented to the user) as the imaging conditions, so that the convenience of the user who changes the imaging conditions can be improved.

[0172] In the present embodiment, mainly one of the imaging conditions including the imaging distance and the imaging conditions including the setting value of the imaging device 2 is described as being output. However, the present embodiment may be configured such that imaging conditions including both the imaging distance and the setting value of the imaging device 2 are output (presented). In the case of such a configuration, the user can select the imaging distance and the setting value of the imaging device 2 included in the presented imaging conditions and change the current imaging conditions.

[0173] Note that although the imaging distance and the setting value of the imaging device 2 have been described here, the imaging conditions may be created based on the estimated distance measurement range (the range of the distance from the imaging device 2 capable of measuring the distance to the subject to the subject) estimated based on the configuration information of the imaging device 2, and any imaging conditions that propose imaging conditions suitable for measuring the distance to the subject using a statistical model may be used.

[0174] Also, in the present embodiment, when the statistical model is constructed to output a blur value and the uncertainty with respect to the blur value by inputting an image, a configuration for obtaining the imaging distance based on the uncertainty may be adopted. According to such a configuration, it is possible to obtain a more accurate imaging distance, and thus it is possible to create more appropriate imaging conditions.

[0175] Furthermore, in the present embodiment, although mainly described as obtaining, as the imaging distance, the distance converted from the blur value output from the statistical model when the image captured by the imaging device 2 is input to the statistical model, the imaging distance may be, for example, the distance converted from the blur value output from the statistical model by inputting an area including a specific subject in the image captured by the imaging device 2 to the statistical model. According to such a configuration, for example, it is possible to obtain the imaging distance based on the subject intended by the user, and thus it is possible to create appropriate imaging conditions for measuring the distance to the subject based on the imaging distance.

[0176] Also, in the present embodiment, with a configuration for presenting imaging conditions to the user who uses the imaging device 2, the user can determine whether to measure the distance to the subject by adopting the imaging conditions, and can intuitively capture an image that enables appropriate distance measurement based on the presented imaging conditions.

[0177] Also, in the present embodiment, a configuration may be adopted in which the current imaging conditions are automatically changed to imaging conditions suitable for measuring the distance. According to such a configuration, it is possible to reduce the labor of the user for changing the current imaging conditions to imaging conditions suitable for measuring the distance.

[0178] In the present embodiment, the description has been made assuming that one statistical model is stored in the storage unit 31. However, as described above, when the imaging device 2 is configured to be able to replace the lens, a statistical model is prepared for each lens that can be used in the imaging device 2 (that is, a lens that can be attached to the imaging device 2) (that is, the distance measuring device is configured to measure the distance to the subject using the statistical model corresponding to the lens). In this case, the imaging condition presentation device 3 (and the distance measuring device) may be configured to select a statistical model corresponding to the lens attached to the imaging device 2 using the configuration information (such as the focal length) of the imaging device 2 and use the selected statistical model.

[0179] Also, in the present embodiment, the imaging condition presentation device 3 has been described as including the units 31 to 36. However, for example, the storage unit 31 may be arranged in an external device different from the imaging condition presentation device 3. In this case, the imaging condition presentation device 3 may operate to use the statistical model acquired from the external device. Further, in the present embodiment, for example, a part of the processing executed by the units 32 to 36 may be executed by an external device.

[0180] (Second Embodiment) Next, the second embodiment will be described. In the present embodiment, the description of the same parts as those in the above-described first embodiment will be omitted, and the parts different from the first embodiment will be mainly described.

[0181] FIG. 21 shows an example of the configuration of the image processing system in the present embodiment. In FIG. 21, the same parts as those in FIG. 1 described above are denoted by the same reference numerals and the detailed description thereof is omitted, and the parts different from FIG. 1 will be described.

[0182] In the above-described first embodiment, the case where the imaging condition presentation device 3 is used to improve the imaging conditions of the image when measuring the distance from the imaging point to the subject using the image captured by the imaging device 2 has been described. However, the distance to the subject measured from the image can be used to measure (estimate) the size of the subject in the image. Note that the process of measuring the size of the subject in the image may be executed by the above-described distance measuring device, or may be executed by another device.

[0183] Therefore, the imaging condition presentation device 3 according to the present embodiment is different from the above-described first embodiment in that, in addition to measuring the distance to the subject in the above-described image, it creates imaging conditions considering measuring the size of the subject.

[0184] As shown in FIG. 21, in the present embodiment, the imaging condition creation unit 35 includes a first creation unit 35a and a second creation unit 35b.

[0185] The first creation unit 35a is a functional unit similar to the imaging condition creation unit 35 included in the imaging condition presentation device 3 according to the above-described first embodiment. That is, the first creation unit 35a creates imaging conditions (hereinafter referred to as distance measurement conditions) of an image suitable for measuring the distance to the subject based on the configuration information of the imaging device 2 acquired by the configuration information acquisition unit 33 and the imaging distance acquired by the imaging distance acquisition unit 34.

[0186] The second creation unit 35b creates imaging conditions (hereinafter referred to as size measurement conditions) of an image suitable for measuring the size of the subject based on the configuration information of the imaging device 2 acquired by the configuration information acquisition unit 33 and the imaging distance acquired by the imaging distance acquisition unit 34.

[0187] In the present embodiment, the output processing unit 36 outputs at least one of the distance measurement conditions created by the first creation unit 35a and the size measurement conditions created by the second creation unit 35b.

[0188] Next, with reference to the flowchart of FIG. 22, an example of the processing procedure of the imaging condition presentation device 3 according to the present embodiment will be described.

[0189] First, the processes of steps S21 to S23 corresponding to the processes of steps S11 to S13 shown in FIG. 20 described above are executed.

[0190] Next, the first creation unit 35a included in the imaging condition creation unit 35 creates ranging conditions based on the configuration information of the imaging device 2 passed from the configuration information acquisition unit 33 and the imaging distance passed from the imaging distance acquisition unit 34 (step S24). Since the process of step S24 is the same as the process of step S14 shown in FIG. 20 described above, a detailed description thereof will be omitted here.

[0191] Also, the second creation unit 35b included in the imaging condition creation unit 35 creates size measurement conditions based on the configuration information of the imaging device 2 passed from the configuration information acquisition unit 33 or the imaging distance passed from the imaging distance acquisition unit 34 (step S25).

[0192] Here, when measuring the size (actual size) of a subject in an image, if the subject is too far from the imaging device 2, the area occupied by the subject in the image becomes small (that is, the subject is crushed), so the size of the subject cannot be measured with a desired resolution.

[0193] Therefore, in step S25, size measurement conditions that satisfy the required resolution are created based on such a viewpoint.

[0194] Here, assume a case of measuring the width of a crack from an image including the crack on the wall surface. Note that the subject (here, the crack) to be measured for the size in the image may be manually specified by the user, for example, or automatically detected (discriminated).

[0195] First, assuming that the pixels indicating two points corresponding to both ends of the crack are the end point a1 and the end point a2, the width w of the crack corresponding to the distance from the end point a1 to the end point a2 is calculated by the following formula (2). Note that the actual width of the crack is obtained by synthesizing the horizontal component and the vertical component of the width of the crack. However, in the following description, for the sake of convenience, the horizontal component of the width of the crack will be simply referred to as the width of the crack.

Equation

[0196] Z in the formula (2) represents the distance from the imaging device 2 to the end point a1 (the depth in the actual scale). s x represents the horizontal size of the entire image sensor 22 provided in the imaging device 2. f represents the focal length in the optical system of the imaging device 2. r x represents the number of pixels (resolution) in the horizontal direction of the image captured by the imaging device 2. Δx represents the number of pixels corresponding to the horizontal width of the crack. Note that since the width of the crack is very small, in the formula (2), the case where the distance to the end point a1 is equal to the distance to the end point a2 is assumed.

[0197] Note that as Z in the formula (2), the distance to the crack estimated using the statistical model is used. Also, s x , f and r x can be obtained from the configuration information of the imaging device 2. Furthermore, Δx can be obtained based on the image captured by the imaging device 2.

[0198] According to the above formula (2), it is possible to calculate the width of the crack in the image from the image captured by the imaging device 2 and the configuration information of the imaging device 2.

[0199] Here, in the present embodiment, for example, it is necessary to create size measurement conditions that satisfy the resolution required for the crack width calculated by the above-described formula (2). Here, it is assumed that the crack width for one pixel in the image is measured with a resolution of 0.1 mm. This means that when Δx = 1 (that is, when Δx is one pixel), the crack width w calculated by formula (2) becomes 0.1. Note that the horizontal size s x of the image sensor 22 and the number of horizontal pixels r x of the image captured by the imaging device 2 are fixed values.

[0200] That is, in order to achieve a resolution of 0.1 mm in the crack width measured from the image, it is necessary to adjust (determine) the distance Z to the crack and the focal length f in the optical system of the imaging device 2 so that w in formula (2) becomes 0.1 or less.

[0201] In this case, for example, when the focal length in formula (2) is fixed to the focal length in the optical system of the current imaging device 2 (that is, the focal length included in the configuration information of the imaging device 2 acquired in step S22), the maximum value of the distance Z (that is, the longest allowable distance to the crack) allowed to achieve a resolution of 0.1 mm can be calculated from the formula (2).

[0202] In this case, in step S25, size measurement conditions including the maximum value of the distance Z (hereinafter referred to as the longest distance) calculated as described above as the imaging distance are created.

[0203] Here, it has been described that size measurement conditions including the longest distance are created as the imaging distance. However, similar to the improved imaging conditions described in the first embodiment above, the size measurement conditions may include the direction and distance (the moving direction and moving amount of the imaging device 2) in which the imaging device 2 should move to achieve the imaging distance (that is, to image the subject from the longest distance). Note that since the moving direction and moving amount of the imaging device 2 are as described in the first embodiment above, the detailed description thereof is omitted here.

[0204] On the other hand, when the distance Z to the crack in the formula (2) is fixed to the imaging distance (the distance to the crack estimated using the statistical model) obtained in step S23, the minimum value of the focal length f (that is, the shortest focal length in the optical system of the imaging device 2 allowed) that can achieve a resolution of 0.1 mm can be calculated.

[0205] In this case, in step S25, size measurement conditions including the minimum value of the focal length f (hereinafter referred to as the shortest focal length) calculated as described above are created as the set value of the imaging device 2.

[0206] In the present embodiment, it is sufficient to create either the size measurement conditions including the above-described imaging distance (the longest distance) or the size measurement conditions including the set value (the shortest focal length) of the imaging device 2. However, similar to the first embodiment described above, the size measurement conditions including the imaging distance and the size measurement conditions including the set value of the imaging device 2 may be created, and a configuration may be adopted to determine which of the two size measurement conditions is to be prioritized.

[0207] Also, in the present embodiment, it has been described that if the subject (for example, a crack on a wall surface) is too far from the imaging device 2, the size of the subject cannot be measured with a desired resolution. However, if the imaging device 2 approaches the subject too closely in order to enlarge the subject in the image, the blur occurring in the subject in the image becomes large, and the measurement accuracy of the distance to the subject or the detection (discrimination) accuracy of the subject decreases. In this case, as a result, it also affects the measurement accuracy of the size of the subject. In other words, when measuring the size of a minute subject such as a crack, depending on the amount of blur occurring in the subject, the measurement accuracy of the size may decrease. Note that, for example, there may be cases where it is difficult to approach the subject due to the angle of view or the influence of obstacles. However, the same applies when the imaging device 2 is too far from the subject in such cases.

[0208] Therefore, in step S25, for example, by further considering the blur value (the blur value indicating the blur occurring on the subject) output from the statistical model that inputs the captured image when the process of step S23 is executed, the absolute value of the blur value (that is, the magnitude of the blur occurring on the subject) does not exceed the maximum value allowable for measuring the size of the subject, a size measurement condition including the longest distance or the shortest focal length may be created. Note that the maximum value of the absolute value of the blur value allowable for measuring the size of the subject varies depending on the type or size of the subject, etc., but it may be preset according to the subject.

[0209] When the process of step S25 is executed, the output processing unit 36 compares the distance measurement condition created in step S24 and the size measurement condition created in step S25, and selects one of the distance measurement condition and the size measurement condition (that is, the imaging condition) (step S26).

[0210] The output processing unit 36 outputs the distance measurement condition or the size measurement condition selected in step S26 as the imaging condition (step S27).

[0211] That is, in the present embodiment, one of the distance measurement condition created in step S24 and the size measurement condition created in step S25 is preferentially output.

[0212] Hereinafter, the processes of steps S26 and S27 will be described. First, in step S26, it is assumed that the distance measurement condition is selected when the distance measurement condition satisfies the size measurement condition. On the other hand, when the distance measurement condition does not satisfy the size measurement condition, it is assumed that the size measurement condition is selected.

[0213] Specifically, for example, when the distance measurement condition includes the focus distance and the size measurement condition includes the longest distance, if the focus distance is less than or equal to the longest distance, the distance measurement condition is selected in step S26, and the distance measurement condition is output in step S27. According to this, it is possible to improve the accuracy of measuring the distance to the subject and present imaging conditions that can appropriately measure the size of the subject.

[0214] On the other hand, when the focus distance is longer than the longest distance, the size measurement condition is selected in step S26, and the size measurement condition is output in step S27. According to this, although it may not be possible to greatly improve the accuracy of measuring the distance to the subject, at least imaging conditions that can appropriately measure the size of the subject can be presented.

[0215] Also, for example, when the distance measurement condition includes the focal length as a setting value of the imaging device 2 and the size measurement condition includes the shortest focal length as a setting value of the imaging device 2, if the focal length included in the distance measurement condition is longer than the shortest focal length, the distance measurement condition is selected in step S26, and the distance measurement condition is output in step S27. On the other hand, when the focal length included in the distance measurement condition is shorter than the shortest focal length, the size measurement condition is selected in step S26, and the size measurement condition is output in step S27.

[0216] Note that when the distance measurement condition is output in step S27, the same processing as step S15 shown in FIG. 20 described above may be executed.

[0217] Also, even when the size measurement condition is output in step S27, by executing the same processing as step S15 shown in FIG. 20 described above, the size measurement condition (imaging distance or setting value of the imaging device 2) can be output (presented) in various modes.

[0218] As described above, in the present embodiment, size measurement conditions (second imaging conditions of an image suitable for measuring the size of a subject), are created based on the configuration information of the imaging device 2 and the imaging distance with respect to the captured image. When the distance measurement conditions do not satisfy the size measurement conditions, the size measurement conditions are preferentially output, so that imaging conditions that satisfy the measurement accuracy of the size required when measuring the size of a minute subject such as a crack on a wall surface can be presented (proposed).

[0219] When the imaging conditions are output in this way, since the size of the subject in the image is measured from the image captured by the imaging device 2 according to the imaging conditions, it becomes possible to appropriately measure the size of the subject using the image.

[0220] Furthermore, in the present embodiment, by inputting the captured image into a statistical model and creating size measurement conditions based on a blur value indicating the blur occurring in the subject in the image output from the statistical model, it is possible to suppress a decrease in the measurement accuracy of the size of the subject due to an increase in the size of the blur occurring in the subject.

[0221] In the present embodiment, although one of the distance measurement conditions and the size measurement conditions has been described as being preferentially output, a configuration may be adopted in which both the distance measurement conditions and the size measurement conditions are output (presented) and an appropriate imaging condition is selected by the user.

[0222] Further, the distance measurement conditions and the size measurement conditions in the present embodiment may include either one of the imaging distance and the set value of the imaging device 2, or may include both the imaging distance and the set value of the imaging device 2.

[0223] Note that, as in the first embodiment described above, the imaging conditions (at least one of the distance measurement conditions and the size measurement conditions) may be presented to the user, or the current imaging conditions may be automatically changed to imaging conditions suitable for measuring the size of the subject.

[0224] (Application Example) In each of the above-described embodiments, it has been described that the imaging device 2 images an image according to the imaging conditions output from the imaging condition presentation device 3, thereby improving the measurement accuracy of the distance to the subject in the image and the measurement accuracy of the size of the subject. Hereinafter, an application example in which a distance measurement device that measures the distance from the image captured by the imaging device 2 to the subject and executes processing such as measuring the size of the subject is applied will be described.

[0225] FIG. 23 shows an example of the functional configuration of a moving body 700 in which the imaging device 2 and the distance measurement device 4 are incorporated.

[0226] The moving body 700 can be realized, for example, as an automobile having an automatic driving function, a drone, a self-standing mobile robot, or the like. A drone is an aircraft that cannot be boarded by a person, a rotary-wing aircraft, a glider, or an airship, and can be flown by remote control or automatic control. Examples include drones (multi-copters), radio-controlled aircraft, and helicopters for agricultural chemical spraying. Self-standing mobile robots include mobile robots such as automated guided vehicles (AGVs), cleaning robots for cleaning floors, and communication robots that provide various guides to visitors. The moving body 700 includes not only those in which the robot body moves, but also industrial robots having a drive mechanism that moves or rotates a part of the robot, such as a robot arm.

[0227] As shown in FIG. 23, the moving body 700 includes, for example, an imaging device 2, a distance measurement device 4, a control signal generation unit 701, and a drive mechanism 702. The imaging device 2 is installed so as to be able to image a subject in the traveling direction of the moving body 700 or a part thereof.

[0228] As shown in FIG. 24, when the moving body 700 is an automobile 700A, the imaging device 2 is installed as a so-called front camera that images the front. Note that the imaging device 2 may be installed as a so-called rear camera that images the rear during backing. Also, a plurality of imaging devices 2 may be installed as the front camera and the rear camera. Furthermore, the imaging device 2 may be installed with the function of a so-called drive recorder. That is, the imaging device 2 may be a recording device.

[0229] FIG. 25 shows an example when the moving body 700 is a drone 700B. The drone 700B includes a drone main body 711 corresponding to the drive mechanism 702 and four propeller units 712 to 715. Each of the propeller units 712 to 715 has a propeller and a motor. When the drive of the motor is transmitted to the propeller, the propeller rotates, and the drone 700B rises by the lift force generated by the rotation. For example, the imaging device 2 is mounted on the lower part of the drone main body 711.

[0230] Also, FIG. 26 shows an example when the moving body 700 is an autonomous mobile robot 700C. A power unit 721 including a motor, wheels, etc., corresponding to the drive mechanism 702, is provided at the lower part of the mobile robot 700C. The power unit 721 controls the rotation speed of the motor and the direction of the wheels. The mobile robot 700C can move in an arbitrary direction by rotating the wheels installed on the road surface or the floor surface when the drive of the motor is transmitted and by controlling the direction of the wheels. In the example shown in FIG. 26, the imaging device 2 is installed on the head of the humanoid mobile robot 700C so as to image the front of the mobile robot 700C, for example. Note that the imaging device 2 may be installed so as to image the rear or the left and right of the mobile robot 700C, or a plurality of imaging devices 2 may be installed so as to image a plurality of directions. Also, dead reckoning can be performed by providing the imaging device 2 in a small robot with little space for mounting sensors or the like and estimating the self-position, posture, and the position of the subject.

[0231] Note that, as shown in FIG. 27, when the moving body 700 is the robot arm 700D and the movement and rotation of a part of the robot arm 700D are controlled, the imaging device 2 may be installed at the tip or the like of the robot arm 700D. In this case, the object gripped by the robot arm 700D is imaged by the imaging device 2, and the distance measuring device 4 can measure the distance to the object that the robot arm 700D is about to grip. Thereby, in the robot arm 700D, an accurate gripping operation of the object can be performed.

[0232] The control signal generation unit 701 outputs a control signal for controlling the drive mechanism 702 based on the distance information indicating the distance to the subject output from the distance measuring device 4. The drive mechanism 702 drives the moving body 700 or a part of the moving body 700 by the control signal output from the control signal generation unit 701. The drive mechanism 702 performs at least one of, for example, the movement, rotation, acceleration, deceleration, addition and subtraction of thrust (lift), change of the traveling direction, switching between the normal operation mode and the automatic operation mode (collision avoidance mode), and the operation of a safety device such as an airbag. The drive mechanism 702 may perform at least one of, for example, movement, rotation, acceleration, addition and subtraction of thrust (lift), change of the direction toward the object, and switching from the automatic operation mode (collision avoidance mode) to the normal operation mode when the distance to the subject is less than the threshold value.

[0233] Note that the drive mechanism 702 of the automobile 700A shown in FIG. 24 is, for example, a tire. The drive mechanism 702 of the drone 700B shown in FIG. 25 is, for example, a propeller. The drive mechanism 702 of the mobile robot 700C shown in FIG. 26 is, for example, a leg. The drive mechanism 702 of the robot arm 700D shown in FIG. 27 is, for example, a support portion that supports the tip where the imaging device 2 is provided.

[0234] The mobile body 700 may further include a speaker or a display to which information (distance information) regarding the distance to the subject output from the distance measuring device 4 is input. This speaker or display is connected to the distance measuring device 4 by wire or wirelessly, and is configured to output audio or an image regarding the distance to the subject. Further, the mobile body 700 may have a light emitting unit that receives the information regarding the distance to the subject output from the distance measuring device 4 and can be turned on and off, for example, according to the distance to the subject.

[0235] Also, for example, when the mobile body 700 is the drone 700B, when creating a map (three-dimensional shape of an object), investigating the structure of buildings and terrain, inspecting for cracks, wire breaks, etc. from above, the imaging device 2 acquires an image of the target, and determines whether the distance to the subject is equal to or greater than a threshold value. The control signal generation unit 701 generates a control signal for controlling the thrust of the drone 700B so that the distance to the inspection target becomes constant based on this determination result. Here, it is assumed that the thrust includes lift. By operating the drone 700B by the drive mechanism 702 based on this control signal, the drone 700B can be made to fly parallel to the inspection target. When the mobile body 700 is the monitoring drone 700B, the control signal generation unit 701 may generate a control signal for controlling the thrust of the drone 700B so as to keep the distance to the object to be monitored constant.

[0236] In addition, when the moving body 700 (e.g., drone 700B) is used for maintenance inspection of various infrastructures (hereinafter simply referred to as infrastructure), an image of a repair-required location (hereinafter referred to as a repair location) including a cracked location or a rusted location in the infrastructure is captured by the imaging device 2, so that the distance to the repair location can be obtained. In this case, by using the distance to the repair location, it is possible to measure the size of the repair location from the image. According to this, for example, by displaying the repair location on a map representing the entire infrastructure, the maintenance inspector of the infrastructure can recognize the repair location. In addition, informing the maintenance inspector of the size of the repair location in advance is also useful for performing a smooth repair operation.

[0237] Here, the case where the moving body 700 (e.g., drone 700B) in which the distance measuring device 4 is incorporated is used for infrastructure maintenance inspection and the like has been described. However, for example, when the distance measuring device 4 is realized as a smartphone or the like equipped with the imaging device 2, for example, the maintenance inspector can perform infrastructure maintenance inspection by capturing an image of the repair location using the smartphone. In addition, when the imaging device 2 is realized as a smartphone or the like, a similar maintenance inspection can be realized by uploading an image of the repair location captured by the maintenance inspector using the smartphone to the distance measuring device 4.

[0238] When uploading an image, for example, by using a method of transferring the image to the distance measuring device 4 (server device) via a network, the inspection work can be easily performed at the maintenance inspection site.

[0239] Also, when the drone 700B is flying, the imaging device 2 acquires an image of the ground direction and determines whether the distance from the ground is equal to or greater than a threshold value. The control signal generation unit 701 generates a control signal for controlling the thrust of the drone 700B so that the height from the ground becomes a specified height based on this determination result. By operating the drone 700B based on this control signal by the drive mechanism 702, the drone 700B can be made to fly at the specified height. If the drone 700B is a pesticide spraying drone, by keeping the height of the drone 700B from the ground constant in this way, it becomes easier to evenly spray pesticides.

[0240] Also, when the moving body 700 is the automobile 700A or the drone 700B, during the platoon running of the automobile 700A or the cooperative flight of the drone 700B, the imaging device 2 images the preceding automobile or the surrounding drones and determines whether the distance to the automobile or drone is equal to or greater than a threshold value. The control signal generation unit 701 generates a control signal for controlling the speed of the automobile 700A or the thrust of the drone 700B based on this determination result so that the distance to the preceding automobile or the surrounding drones becomes constant. By operating the automobile 700A or the drone 700B based on this control signal by the drive mechanism 702, the platoon running of the automobile 700A or the cooperative flight of the drone 700B can be easily performed.

[0241] Furthermore, when the moving body 700 is the automobile 700A, it may be configured to be able to receive the driver's instructions via a user interface so that the driver of the automobile 700A can set (change) a threshold value. Thereby, the automobile 700A can be made to run at an inter-vehicle distance preferred by the driver. Also, in order to maintain a safe inter-vehicle distance from the preceding automobile, the threshold value may be changed according to the speed of the automobile 700A. The safe inter-vehicle distance varies depending on the speed of the automobile 700A. Therefore, the higher the speed of the automobile 700A, the larger (longer) the threshold value can be set.

[0242] Also, when the moving body 700 is the automobile 700A, a predetermined distance in the traveling direction may be set as a threshold value, and a control signal may be generated to activate the brake or activate a safety device such as an airbag when an object appears in front of the threshold value. In this case, safety devices such as an automatic brake and an airbag are provided in the drive mechanism 702.

[0243] According to at least one of the embodiments described above, it is possible to provide an image processing apparatus, method, and program capable of improving the accuracy of the distance measured using an image.

[0244] Each of the various functions described in this embodiment may be realized by a circuit (processing circuit). Examples of the processing circuit include a programmed processor such as a central processing unit (CPU). This processor executes each of the described functions by executing a computer program (instruction group) stored in the memory. This processor may be a microprocessor including an electric circuit. Examples of the processing circuit also include a digital signal processor (DSP), an application specific integrated circuit (ASIC), a microcontroller, a controller, and other electric circuit components. Each of the other components other than the CPU described in this embodiment may also be realized by a processing circuit.

[0245] Also, since the various processes of this embodiment can be realized by a computer program, the same effects as this embodiment can be easily realized by simply installing and executing this computer program on a computer through a computer-readable storage medium storing this computer program.

[0246] Although some embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, as well as in the invention described in the claims and the equivalent scope thereof.

Description of Reference Numerals

[0247] 1... Image processing system, 2... Imaging device, 3... Imaging condition presentation device, 21... Lens, 22... Image sensor, 31... Storage unit, 32... Image acquisition unit, 33... Configuration information acquisition unit, 34... Imaging distance acquisition unit, 35... Imaging condition creation unit, 35a... First creation unit, 35b... Second creation unit, 36... Output processing unit, 301... CPU, 302... Non-volatile memory, 303... RAM, 303A... Imaging condition presentation program, 304... Communication device, 305... Input device, 306... Output device.

Claims

1. In an image processing apparatus used for measuring an imaging distance from the imaging apparatus to a subject in an image using an image affected by aberration of an optical system of the imaging apparatus imaged by the imaging apparatus, a first acquisition means for acquiring an image captured by the imaging apparatus; a second acquisition means for acquiring configuration information regarding the optical system of the imaging apparatus; a third acquisition means for acquiring an imaging distance with respect to the acquired image based on the acquired image; a creation means for creating a first imaging condition of an image suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance; an output processing means for outputting the created first imaging condition comprising: The creation means creates a first imaging condition including a direction and a distance in which the imaging apparatus should move in order to realize an imaging distance suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance. An image processing apparatus.

2. The image processing apparatus according to claim 1, wherein the creation means creates a first imaging condition including a set value of the imaging apparatus suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance.

3. The creation means creates a first imaging condition including a direction and a distance in which the imaging apparatus should move based on the acquired configuration information and the acquired imaging distance, and a first imaging condition including a set value of the imaging apparatus, and one of the first imaging conditions is preferentially output The image processing apparatus according to claim 1.

4. The image processing apparatus according to claim 1, wherein the creation means estimates a range of a distance from the imaging apparatus capable of measuring the distance to the subject based on the acquired configuration information, and creates the first imaging condition based on the range of the distance.

5. The image processing apparatus according to any one of claims 1 to 4, wherein the acquired configuration information includes a distance to a focused position in an image captured by the imaging device.

6. Further comprising storage means for storing a statistical model generated by learning blurring that varies non-linearly according to the distance to a subject in an image affected by aberration of an optical system of the imaging device, The third acquisition means acquires the imaging distance as the distance to the subject in the image, which is converted from a blur value indicating blurring occurring in the subject in the image output from the statistical model by inputting the acquired image into the statistical model. The image processing apparatus according to any one of claims 1 to 5.

7. The statistical model is constructed to further output an uncertainty with respect to a blur value output from the statistical model by inputting the acquired image, The third acquisition means acquires the imaging distance based on the uncertainty output from the statistical model. The image processing apparatus according to claim 6.

8. The image processing apparatus according to claim 6 or 7, wherein the third acquisition means acquires the imaging distance as the distance to the subject in the image, which is converted from a blur value indicating blurring occurring in the subject in the image output from the statistical model by inputting a region including the subject in the acquired image into the statistical model.

9. In an image processing apparatus used for measuring an imaging distance from the imaging device to a subject in an image affected by aberration of an optical system of the imaging device, using an image captured by the imaging device, First acquisition means for acquiring an image captured by the imaging device; Second acquisition means for acquiring configuration information regarding the optical system of the imaging device; Third acquisition means for acquiring an imaging distance with respect to the image based on the acquired image. Creation means for creating first imaging conditions of an image suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance; Output processing means for outputting the created first imaging conditions; comprising: The creation means creates second imaging conditions of an image suitable for measuring the size of the subject based on the acquired configuration information and the acquired imaging distance, and the output processing means further outputs the created second imaging conditions. An image processing apparatus.

10. In an image processing apparatus used when measuring the imaging distance from an imaging apparatus to a subject in an image using an image affected by aberration of an optical system of the imaging apparatus imaged by the imaging apparatus, First acquisition means for acquiring an image captured by the imaging apparatus; Second acquisition means for acquiring configuration information regarding the optical system of the imaging apparatus; Third acquisition means for acquiring an imaging distance with respect to the acquired image based on the acquired image; Creation means for creating first imaging conditions of an image suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance; Output processing means for outputting the created first imaging conditions; comprising: The creation means creates first imaging conditions including the direction and distance in which the imaging apparatus should move to realize an imaging distance suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance, and second imaging conditions of an image suitable for measuring the size of the subject, The second imaging conditions include the longest imaging distance from the imaging apparatus capable of estimating the size of the subject in the acquired image to the subject, and when the first imaging conditions do not satisfy the second imaging conditions, the output processing means preferentially outputs the second imaging conditions. Image processing apparatus.

11. In an image processing apparatus used for measuring an imaging distance from the imaging apparatus to a subject in an image using an image affected by aberration of an optical system of the imaging apparatus imaged by the imaging apparatus, a first acquisition means for acquiring an image imaged by the imaging apparatus; a second acquisition means for acquiring configuration information regarding the optical system of the imaging apparatus; a third acquisition means for acquiring an imaging distance with respect to the acquired image based on the acquired image; a creation means for creating a first imaging condition of an image suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance; an output processing means for outputting the created first imaging condition and comprising: The creation means creates a first imaging condition including a set value of the imaging apparatus suitable for measuring the distance to the subject and a second imaging condition of an image suitable for measuring the size of the subject based on the acquired configuration information and the acquired imaging distance. The set value included in the first imaging condition includes a focal length in the optical system of the imaging apparatus. The second imaging condition includes the shortest focal length capable of measuring the size of the subject in the acquired image. When the first imaging condition does not satisfy the second imaging condition, the output processing means preferentially outputs the second imaging condition. Image processing apparatus.

12. Further comprising a storage means for storing a statistical model generated by learning a blur that changes non-linearly according to the distance to a subject in an image generated in an image affected by aberration of an optical system of the imaging apparatus, The creation means creates the second imaging condition based on a blur value indicating the blur occurring in the subject in the image output from the statistical model by inputting the acquired image into the statistical model. The image processing apparatus according to any one of claims 9 to 11.

13. In an image processing apparatus used for measuring an imaging distance from the imaging apparatus to a subject in an image by using an image affected by aberration of an optical system of the imaging apparatus imaged by the imaging apparatus, a first acquisition means for acquiring an image captured by the imaging apparatus; a second acquisition means for acquiring configuration information regarding the optical system of the imaging apparatus; a third acquisition means for acquiring an imaging distance with respect to the image based on the acquired image; a creation means for creating first imaging conditions for an image suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance; an output processing means for outputting the created first imaging conditions and comprising: wherein the output processing means presents the first imaging conditions to a user who uses the imaging apparatus image processing apparatus.

14. The image processing apparatus according to any one of claims 9 to 12, wherein the output processing means presents the second imaging conditions to a user who uses the imaging apparatus.

15. In an image processing apparatus used for measuring an imaging distance from the imaging apparatus to a subject in an image by using an image affected by aberration of an optical system of the imaging apparatus imaged by the imaging apparatus, a first acquisition means for acquiring an image captured by the imaging apparatus; a second acquisition means for acquiring configuration information regarding the optical system of the imaging apparatus; a third acquisition means for acquiring an imaging distance with respect to the image based on the acquired image; a creation means for creating first imaging conditions for an image suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance; an output processing means for outputting the created first imaging conditions comprising, the output processing means outputs the first imaging condition so that the current imaging condition is automatically changed to the first imaging condition An image processing apparatus.

16. The image processing apparatus according to any one of claims 9 to 12, wherein the output processing means outputs the second imaging condition so that the current imaging condition is automatically changed to the second imaging condition.

17. A method executed by an image processing apparatus used when measuring an imaging distance from the imaging apparatus to a subject in an image using an image affected by aberration of an optical system of the imaging apparatus imaged by the imaging apparatus, acquiring an image captured by the imaging device; acquiring configuration information regarding the optical system of the imaging device; acquiring an imaging distance for the image based on the acquired image; creating imaging conditions for an image suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance; outputting the created imaging conditions; comprising, The creating step includes creating a first imaging condition including a direction and a distance in which the imaging device should move in order to realize an imaging distance suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance. Method.

18. A program executed by a computer of an image processing apparatus used when measuring an imaging distance from the imaging apparatus to a subject in an image using an image affected by aberration of an optical system of the imaging apparatus imaged by the imaging apparatus, causing the computer to, acquire an image captured by the imaging device; A step of acquiring configuration information regarding an optical system of the imaging device; A step of acquiring an imaging distance for the acquired image based on the acquired image; A step of creating imaging conditions for an image suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance; A step of outputting the created imaging conditions; and execute; The creating step includes a step of creating first imaging conditions including a direction and a distance in which the imaging device should move in order to realize an imaging distance suitable for measuring the distance to the subject based on the acquired configuration information and the acquired imaging distance. Program.

Images