Distance measuring device, processing device, method, and program

Abstract

To provide distance measuring equipment, an image processing device, method, and a program which enable the accuracy of measurement distance, using images.SOLUTION: Distance measuring equipment according to an embodiment comprises storage means, first acquisition means, second acquisition means, third acquisition means and first conversion means. The storage means stores a statistical model having been generated by learning a blur that nonlinearly changes in accordance with the distance to a subject in a first image. The first acquisition means acquires a second image having been captured by an imaging unit. The second acquisition means acquires focus position information that pertains to a focus position used when the second image is captured. The third acquisition means acquires a blur value that indicates a blur generated in a subject in the second image that is output from the statistical model by inputting the second image to the statistical model. The first conversion means converts the blur value to the distance to the subject on the basis of the focus position information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Embodiments of the present invention relate to a distance measuring device, an image processing device, a method, and a program. [Background technology]

[0002] Generally, it was known that images captured by two imaging devices (cameras) or a stereo camera (a camera with multiple lenses) were used to measure (acquire) the distance to a subject. However, in recent years, techniques have been disclosed that measure the distance to a subject using images captured by a single imaging device (a camera with one lens).

[0003] However, when measuring the distance from an image captured by a single imaging device to a subject (the actual distance between the subject and the imaging device), the accuracy of the measured distance may be low. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-043115 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] Therefore, the problem that the present invention aims to solve is to provide a distance measuring device, an image processing device, a method, and a program that can improve the accuracy of distance measured using images. [Means for solving the problem]

[0006] According to one embodiment, a distance measuring device is provided that includes an imaging unit. The distance measuring device comprises a storage means, a first acquisition means, a second acquisition means, a third acquisition means, and a first conversion means. The storage means stores a statistical model generated by learning the blur that occurs in a first image affected by the aberrations of the optical system of the imaging unit, and which changes non-linearly according to the distance to the subject in the first image. The first acquisition means acquires a second image captured by the imaging unit, which is affected by the aberrations of the optical system of the imaging unit. The second acquisition means acquires focus position information relating to the focus position when the second image was captured. The third acquisition means inputs the acquired second image into the statistical model and acquires a blur value indicating the blur that occurs in the subject in the second image, which is output from the statistical model. The first conversion means converts the acquired blur value into the distance to the subject based on the acquired focus position information. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram showing an example of the configuration of a distance measuring device according to an embodiment. [Figure 2] A diagram showing an example of a system configuration for a distance measuring device. [Figure 3] A diagram illustrating the general operation of a distance measuring device. [Figure 4] This diagram shows the relationship between the distance to the subject and the blur caused by chromatic aberration when using a single lens. [Figure 5] This diagram shows the relationship between the distance to the subject and the blur caused by chromatic aberration when using an achromatic lens. [Figure 6] This diagram shows the relationship between the size of the aperture of the diaphragm mechanism in the optical system of the imaging unit and the shape of the PSF (Photon Firmware Filter). [Figure 7] This figure shows an example of the PSF shape that occurs in the image of each channel. [Figure 8] This figure shows another example of the PSF shape that occurs in the image of each channel. [Figure 9] A diagram showing an example of PSF shapes occurring at various locations in the image. [Figure 10] A diagram for specifically explaining the position dependence of the PSF shape according to the type of lens. [Figure 11] A diagram showing the relationship between the non-linearity of the PSF shape and the shape of the aperture of the aperture mechanism. [Figure 12] A diagram for explaining the first method of estimating blur from an image. [Figure 13] A diagram showing an example of information input into the statistical model in the first method. [Figure 14] A diagram for explaining the second method of estimating blur from an image. [Figure 15] A diagram showing an example of information input into the statistical model in the second method. [Figure 16] A diagram for explaining the third method of estimating blur from an image. [Figure 17] A diagram for specifically explaining the correlation between the blur occurring in the image and the distance to the subject in the image. [Figure 18] A diagram showing an example of the learning process of the statistical model. [Figure 19] A flowchart showing an example of the processing procedure of the learning process of the statistical model. [Figure 20] A sequence chart showing an example of the processing procedure of the distance measuring device when measuring the distance to the subject. [Figure 21] A diagram for explaining the focus position information. [Figure 22] A diagram showing the correspondence between the focus position information and the actual focus distance. [Figure 23] A diagram showing another example of the configuration of the distance measuring device. [Figure 24] A diagram showing an example of the functional configuration of a moving body in which the distance measuring device is incorporated. [Figure 25] A diagram for explaining the case where the moving body is an automobile. [Figure 26] A diagram for explaining the case where the moving body is a drone. [Figure 27] A diagram for explaining the case where the moving body is a self-propelled mobile robot. [Figure 28]A diagram illustrating the case where the moving object is a robotic arm. [Modes for carrying out the invention]

[0008] The embodiments will be described below with reference to the drawings. Figure 1 shows an example of the configuration of a distance measuring device according to this embodiment. The distance measuring device 1 shown in Figure 1 is used to capture an image and measure (acquire) the distance from the capture point to the subject using the captured image.

[0009] As shown in Figure 1, the distance measuring device 1 comprises an imaging unit 2 and an image processing unit 3. In this embodiment, the distance measuring device 1 is described as being implemented as a single device comprising an imaging unit 2 and an image processing unit 3. However, the distance measuring device 1 may also be a distance measuring system (image processing system) implemented as an imaging device and an image processing device, where the imaging unit 2 and the image processing unit 3 are separate devices. In the case of a distance measuring device 1 comprising an imaging unit 2 and an image processing unit 3, for example, a digital camera, a smartphone, and a tablet computer can be used as the distance measuring device 1. On the other hand, in the case of a distance measuring system implemented as an imaging device and an image processing device, where the imaging unit 2 and the image processing unit 3 are separate devices, for example, a digital camera can be used as the imaging device, and a personal computer, smartphone, or tablet computer can be used as the image processing device. In this case, the image processing device may operate as a server device that executes a cloud computing service, for example.

[0010] The imaging unit 2 is implemented by a camera (imaging device) incorporated into the distance measuring device 1, and includes a lens 21 and an image sensor 22. The lens 21 and image sensor 22 correspond to the optical system (monocular camera) of the imaging unit 2.

[0011] In this embodiment, the lens 21 constitutes a lens unit together with a signal processing unit (signal processing circuit) and a lens drive unit (lens drive circuit) for controlling the focus position by adjusting the position of the lens 21, an aperture mechanism and aperture control circuit having an aperture for adjusting the amount of light (incident light) taken into the optical system of the imaging unit 2, and a control circuit equipped with a memory that holds information about the lens 21 (hereinafter referred to as lens information).

[0012] Furthermore, in this embodiment, the lens 21 (lens unit) may be manually replaceable with other lenses. In this case, the user of the rangefinder 1 can use one of several types of lenses, such as a standard lens, a telephoto lens, and a wide-angle lens, attached to the rangefinder 1. When the lens is changed, the focal length and F-number (aperture value) change, and the rangefinder 1 can capture an image corresponding to the lens used.

[0013] In this embodiment, focal length refers to the distance from the lens to the position where light converges when light is incident parallel to the lens. The F-number is a numerical representation of the amount of light taken into the imaging unit 2 (image sensor 22) according to the aperture mechanism. A smaller F-number indicates that the amount of light taken into the imaging unit 2 increases (i.e., the size of the aperture increases).

[0014] Light reflected from the subject enters lens 21. The light that enters lens 21 passes through it. The light that has passed through lens 21 reaches image sensor 22 and is received (detected) by image sensor 22. Image sensor 22 generates an image composed of multiple pixels by converting the received light into an electrical signal (photoelectric conversion).

[0015] The image sensor 22 can be implemented using, for example, a CCD (Change Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 22 includes, for example, a first sensor (R sensor) 221 that detects light in the red (R) wavelength band, a second sensor (G sensor) 222 that detects light in the green (G) wavelength band, and a third sensor (B sensor) 223 that detects light in the blue (B) wavelength band. The image sensor 22 receives light in the corresponding wavelength bands using the first to third sensors 221 to 223 and can generate sensor images (R image, G image, and B image) corresponding to each wavelength band (color component). That is, the image captured by the imaging unit 2 is a color image (RGB image), and this image includes the R image, G image, and B image.

[0016] In this embodiment, the image sensor 22 is described as including the first to third sensors 221 to 223, but the image sensor 22 only needs to be configured to include at least one of the first to third sensors 221 to 223. Furthermore, the image sensor 22 may be configured to include, for example, a sensor for generating a monochrome image instead of the first to third sensors 221 to 223.

[0017] In this embodiment, the image generated based on the light transmitted through the lens 21 is an image affected by the aberrations of the optical system (lens 21) of the imaging unit 2, and includes blurring caused by these aberrations. Details of the blurring that occurs in the image will be described later.

[0018] As shown in Figure 1, the imaging unit 2 further includes a first image acquisition unit 23 and a first focus position information acquisition unit 24.

[0019] The first image acquisition unit 23 acquires images captured by the imaging unit 2 (camera). The images acquired by the first image acquisition unit 23 are transmitted from the imaging unit 2 to the image processing unit 3.

[0020] The first focus position information acquisition unit 24 acquires information (hereinafter referred to as focus position information) regarding the focus position (the position in focus in the image) when the image is captured by the imaging unit 2. The focus position information acquired by the first focus position information acquisition unit 24 is added to the image acquired by the first image acquisition unit 23 described above and transmitted from the imaging unit 2 to the image processing unit 3.

[0021] The image processing unit 3 is connected to the imaging unit 2 and includes a storage unit 31, a second image acquisition unit 32, a second focus position information acquisition unit 33, a blur value acquisition unit 34, a real distance conversion unit 35, and an output unit 36.

[0022] The storage unit 31 stores a statistical model used to measure the distance to the subject from the image captured by the imaging unit 2. The statistical model stored in the storage unit 31 is generated by learning the blur that occurs in images affected by the aberrations of the optical system of the imaging unit 2, and which changes non-linearly depending on the distance to the subject in the image.

[0023] The statistical model can be generated by applying various known machine learning algorithms, such as neural networks or random forests. Furthermore, the neural networks applicable in this embodiment may include, for example, convolutional neural networks (CNNs), fully connected neural networks, and recurrent neural networks.

[0024] The second image acquisition unit 32 acquires the image transmitted from the imaging unit 2 described above. The second focus position information acquisition unit 33 acquires the focus position information attached to the image acquired by the second image acquisition unit 32 (i.e., the image transmitted from the imaging unit 2).

[0025] The blur value acquisition unit 34 inputs the image acquired by the second image acquisition unit 32 into the statistical model stored in the storage unit 31, thereby acquiring a blur value that indicates the blur occurring in the subject in the image, which is output from the statistical model.

[0026] The actual distance conversion unit 35 converts the blur value acquired by the blur value acquisition unit 34 into the distance to the subject in the image acquired by the second image acquisition unit 32 (i.e., the actual distance from the distance measuring device 1 to the subject), based on the focus position information acquired by the second focus position information acquisition unit 33.

[0027] The output unit 36 ​​outputs distance information (indicating the distance to the subject) converted from the blur value by the actual distance conversion unit 35.

[0028] Figure 2 shows an example of the system configuration of the distance measuring device 1. The distance measuring device 1 comprises a CPU 101, a non-volatile memory 102, a RAM 103, and a communication device 104. The distance measuring device 1 also has a bus 105 that interconnects the CPU 101, the non-volatile memory 102, the RAM 103, and the communication device 104.

[0029] In this embodiment, the distance measuring device 1 is assumed to incorporate a camera equipped with the lens 21 and image sensor 22 described in Figure 1 above, but this camera is omitted in Figure 2.

[0030] The CPU 101 is a processor for controlling the operation of various components within the distance measuring device 1. The CPU 101 may be a single processor or may consist of multiple processors. Here, we will use a CPU (Central Processing Unit) as the processor that controls the operation of the components, but this processor may also be a GPU (Graphics Processing Unit). The CPU 101 executes various programs loaded from the non-volatile memory 102 into the RAM 103. These programs include the operating system (OS) and various application programs, such as the distance measuring program 103A.

[0031] The non-volatile memory 102 is a storage medium used as an auxiliary storage device. The RAM 103 is a storage medium used as the main storage device. Although only the non-volatile memory 102 and RAM 103 are shown in Figure 2, the distance measuring device 1 may also be equipped with other storage devices such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

[0032] In this embodiment, the storage unit 31 shown in Figure 1 is implemented by, for example, a non-volatile memory 102 or other storage device.

[0033] Furthermore, in this embodiment, some or all of the first image acquisition unit 23, first focus position information acquisition unit 24, second image acquisition unit 32, second focus position information acquisition unit 33, blur value acquisition unit 34, actual distance conversion unit 35, and output unit 36 ​​shown in Figure 1 are implemented by the CPU 101 (i.e., the computer of the distance measuring device 1), that is, by causing the CPU to execute the distance measuring program 103A, i.e., by software. This distance measuring program 103A may be stored and distributed on a computer-readable storage medium, or it may be downloaded to the distance measuring device 1 via a network. Note that some or all of these units 23, 24 and 32-36 may be implemented by hardware such as ICs (Integrated Circuits), or by a combination of software and hardware.

[0034] Here, it has been explained that each part 23, 24, and 32-36 included in the distance measuring device 1 are implemented by a single program (distance measuring program 103A). However, for example, the first image acquisition unit 23 and the first focus position information acquisition unit 24 included in the imaging unit 2 and the second image acquisition unit 32, the second focus position information acquisition unit 33, the blur value acquisition unit 34, the actual distance conversion unit 35, and the output unit 36 ​​included in the image processing unit 3 may be implemented by different programs (software).

[0035] The communication device 104 is a device configured to perform wired or wireless communication.

[0036] Although not shown in Figure 2, the distance measuring device 1 may further include other devices such as an input device and a display device.

[0037] Next, with reference to Figure 3, an overview of the operation of the distance measuring device 1 in this embodiment will be described. In the distance measuring device 1, the imaging unit 2 (image sensor 22) captures an image affected by the aberrations of the optical system (lens 21) of the imaging unit 2, as described above.

[0038] The image processing unit 3 (second image acquisition unit 32) acquires the image captured by the imaging unit 2 and inputs the image into the statistical model stored in the storage unit 31.

[0039] In this embodiment, the statistical model is generated by learning the blur that changes non-linearly according to the distance to the subject in the image, as described above. When an image is input to the statistical model, a blur value (blur information) indicating the blur that occurs in the image according to the distance to the subject in the image is output from the statistical model. As will be described later, there is a correlation between the distance to the subject in the image and the color, size, and shape of the blur that occurs in the image according to that distance. The image processing unit 3 (actual distance conversion unit 35) can obtain the distance to the subject by converting the blur value output from the statistical model into distance.

[0040] Thus, in this embodiment, a statistical model can be used to measure the distance (distance information) from the image captured by the imaging unit 2 to the subject.

[0041] In this embodiment, the image captured by the imaging unit 2 exhibits blurring due to aberrations (lens aberrations) in the optical system of the imaging unit 2, as described above. The blurring that occurs in the image captured by the imaging unit 2 will be explained below. First, chromatic aberration, one of the types of blurring caused by aberrations in the optical system of the imaging unit 2, will be explained.

[0042] Figure 4 shows the relationship between the distance to the subject and the blur caused by chromatic aberration in the image.

[0043] Because the refractive index of light passing through the aberrated lens 21 differs for each wavelength band, if, for example, the subject's position is shifted from the focal point, the light from each wavelength band will not converge at a single point but will reach different points. This appears as chromatic aberration (blur) in the image.

[0044] The upper part of Figure 4 shows the case where the position of the subject relative to the distance measuring device 1 (image sensor 22) is farther than the focal position (i.e., the position of the subject is behind the focal position).

[0045] In this case, with respect to light 401 in the red wavelength band, the image sensor 22 (first sensor 221) exhibits relatively small blur b R An image containing this is generated. On the other hand, with respect to light 402 in the blue wavelength band, the image sensor 22 (third sensor 223) produces a relatively large blur b. B An image containing this is generated. Note that for light 403 in the green wavelength band, blur b R Playing dumb b B An image containing blur of an intermediate size is generated. Therefore, in images taken when the subject is farther away from the focal point, a blue blur is observed outside the subject in the image.

[0046] On the other hand, the lower part of Figure 4 shows the case where the position of the subject is closer to the distance measuring device 1 (image sensor 22) than the focal position (i.e., the position of the subject is in front of the focal position).

[0047] In this case, with respect to light 401 in the red wavelength band, the image sensor 22 (first sensor 221) exhibits relatively large blur b R An image containing this is generated. On the other hand, with respect to light 402 in the blue wavelength band, the image sensor 22 (third sensor 223) produces a relatively small blur b B An image containing this is generated. Note that for light 403 in the green wavelength band, blur b R Playing dumb b B An image containing blur of an intermediate size is generated. Therefore, in images taken when the subject is closer than the focal point, a red blur is observed outside the subject in the image.

[0048] Here, Figure 4 shows an example where lens 21 is a simple single lens, but in some cases, a lens with chromatic aberration correction (hereinafter referred to as an achromatic lens) may be used as lens 21. An achromatic lens is a lens that combines a low-dispersion convex lens and a high-dispersion concave lens, and it is the lens with the fewest number of lens elements that corrects chromatic aberration.

[0049] Figure 5 shows the relationship between the distance to the subject and the blur caused by chromatic aberration when the achromatic lens described above is used as lens 21. Although achromatic lenses are designed to align the focal points of blue and red wavelengths, chromatic aberration cannot be completely eliminated. Therefore, when the subject is farther away than the focal point, a green blur occurs as shown in the upper part of Figure 5, and when the subject is closer than the focal point, a purple blur occurs as shown in the lower part of Figure 5.

[0050] The middle section of Figures 4 and 5 shows the case where the position of the subject and the focus position coincide with the distance measuring device 1 (image sensor 22). In this case, the image sensor 22 (first to third sensors 221 to 223) generates an image with less blur.

[0051] Here, the optical system (lens unit) of the imaging unit 2 is equipped with an aperture mechanism as described above, and the shape of the blur produced in the image captured by the imaging unit 2 also differs depending on the size of the aperture of the aperture mechanism. The shape of the blur is called the PSF (Point Spread Function) shape and represents the diffuse distribution of light that occurs when a point light source is imaged.

[0052] The upper part of Figure 6 shows the PSF shape that occurs in the center of an image captured by the imaging unit 2 (optical system) using a lens with a focal length of 50 mm, when the focus position is 1500 mm and the F-number (aperture) is F1.8, from left to right in order of proximity of the subject to the rangefinder 1. The lower part of Figure 6 shows the PSF shape that occurs in the center of an image captured by the imaging unit 2 (optical system) using a lens with a focal length of 50 mm, when the focus position is 1500 mm and the F-number (aperture) is F4, from left to right in order of proximity of the subject to the rangefinder 1. Note that the center of the upper and lower parts of Figure 6 shows the PSF shape when the subject's position coincides with the focus position.

[0053] The PSF shapes shown in the upper and lower sections of Figure 6, corresponding to each other, represent the PSF shapes when the position of the subject relative to the rangefinder 1 is the same. However, even when the position of the subject is the same, the PSF shape in the upper section (the PSF shape that occurs in an image captured with an F value of F1.8) and the PSF shape in the lower section (the PSF shape that occurs in an image captured with an F value of F4) are different.

[0054] Furthermore, as shown in the leftmost and rightmost PSF shapes in Figure 6, even when the distance from the subject to the focal point is approximately the same, the PSF shape differs depending on whether the subject is closer to the focal point or further away.

[0055] As mentioned above, the phenomenon in which the PSF shape differs depending on the size of the aperture of the aperture mechanism and the position of the subject relative to the rangefinder 1 also occurs similarly in each channel (RGB image, R image, G image, and B image). Figure 7 shows the PSF shape generated in the images of each channel captured by the imaging unit 2 using a lens with a focal length of 50 mm, with a focus position of 1500 mm and an F value of F1.8, divided into cases where the subject is closer to the focus position (in front) and cases where the subject is further away from the focus position (in the background). Figure 8 shows the PSF shape generated in the images of each channel captured by the imaging unit 2 using a lens with a focal length of 50 mm, with a focus position of 1500 mm and an F value of F4, divided into cases where the subject is closer to the focus position and cases where the subject is further away from the focus position.

[0056] Furthermore, the PSF shape generated in the image captured by the imaging unit 2 also differs depending on its position within the image.

[0057] The upper part of Figure 9 shows the PSF shapes that occur at various positions in the image captured by the imaging unit 2, which uses a lens with a focal length of 50 mm, when the focus position is 1500 mm and the F-number is F1.8, divided into cases where the subject is closer to the focus position and cases where the subject is further away from the focus position.

[0058] The middle section of Figure 9 shows the PSF shapes that occur at various positions in the image captured by the imaging unit 2, which uses a lens with a focal length of 50 mm, when the focus position is 1500 mm and the F-number is F4, divided into cases where the subject is closer to the focus position and cases where the subject is further away from the focus position.

[0059] As shown in the upper and middle sections of Figure 9, near the edges of the image captured by the imaging unit 2 (particularly near the upper left corner), a PSF shape different from the PSF shape located near the center of the image can be observed.

[0060] Furthermore, the lower part of Figure 9 shows the PSF shapes that occur at each position in the image captured by the imaging unit 2, which uses a lens with a focal length of 105 mm, when the focus position is 1500 mm and the F value is F4, separately for cases where the subject is closer than the focus position and cases where the subject is further away from the focus position.

[0061] The upper and middle sections of Figure 9 above show the PSF shape that occurs in images captured using the same lens. However, as shown in the lower section of Figure 9, when lenses with different focal lengths are used, different PSF shapes corresponding to those lenses (different from the PSF shapes in the upper and middle sections of Figure 9) are observed.

[0062] Next, with reference to Figure 10, the position dependence of the PSF shape (lens aberration) according to the type of lens used in the optical system of the imaging unit 2 described above will be explained in detail. Figure 10 shows the PSF shapes that occur near the center (center of the screen) and near the edges (edge ​​of the screen) of images captured using each of several lenses with different focal lengths, divided into cases where the subject is closer than the focal position and cases where the subject is further away than the focal position.

[0063] As shown in Figure 10, the PSF shape that occurs near the center of the image is generally circular and identical regardless of the type of lens used. However, the PSF shape that occurs near the edges of the image has a different shape from the PSF shape that occurs near the center of the image, and its characteristics differ depending on the type of lens used. It should be noted that, as explained in Figure 5 above, when the subject is closer than the focal point, a purple blur occurs near the edge of the PSF shape, and when the subject is farther than the focal point, a green blur occurs near the edge of the PSF shape. This is common regardless of the type of lens used.

[0064] Furthermore, Figure 10 shows two examples (#1 and #2) for a lens with a focal length of 50mm. This indicates that although the focal length is the same at 50mm, the lenses are manufactured by different companies (i.e., they are different products). The same applies to lenses with a focal length of 85mm.

[0065] As described above, the blur in this embodiment that changes nonlinearly depending on the distance to the subject includes the blur caused by chromatic aberration of the optical system of the imaging unit 2 as explained in Figures 4 and 5, the blur caused according to the size of the aperture (i.e., the F-number) of the aperture mechanism that adjusts the amount of light taken into the optical system of the imaging unit 2 as explained in Figures 6 to 8, and the blur that changes according to the position in the image captured by the imaging unit 2 as explained in Figures 9 and 10.

[0066] Furthermore, the PSF shape also differs depending on the shape of the aperture of the diaphragm mechanism. Here, Figure 11 shows the relationship between the nonlinearity (asymmetry) of the PSF shape and the shape of the aperture of the diaphragm mechanism. The nonlinearity of the PSF shape described above is more likely to occur when the shape of the aperture of the diaphragm mechanism is not a circle. In particular, the nonlinearity of the PSF shape is more likely to occur when the shape of the aperture is an odd-sided polygon, or an even-sided polygon arranged asymmetrically with respect to the horizontal or vertical axis of the image sensor 22.

[0067] In the distance measuring device 1 according to this embodiment, a blur value indicating the blur occurring in the image is estimated (predicted) using a statistical model generated by focusing on the fact that the blur (color, size, and shape) occurring in the image serves as a physical clue regarding the distance to the subject. In this embodiment, the blur value estimated by the statistical model (i.e., output from the statistical model) is a scalar quantity representing the amount of blur, including the color, size, and shape of the blur occurring in the image.

[0068] Below, we will describe an example of a method for estimating blur (or the blur value indicating blur) from an image using a statistical model in this embodiment. Here, we will describe the first to third methods.

[0069] First, the first method will be explained with reference to Figure 12. In the first method, a local region (image patch) 501a is extracted from image 501.

[0070] In this case, for example, the entire area of ​​image 501 may be divided into a matrix, and the resulting sub-regions may be sequentially extracted as local regions 501a, or image 501 may be recognized, and local regions 501a may be extracted to cover the area where the subject (image) is detected. Furthermore, local regions 501a may partially overlap with other local regions 501a.

[0071] Next, for each extracted local region 501a, information about that local region 501a (information from image 501) is input into the statistical model, and a blur value indicating the blur that occurs depending on the distance to the subject in that local region 501a is estimated.

[0072] The statistical model, having received information about the local region 501a in this manner, estimates a blur value 502 for each pixel that makes up the local region 501a.

[0073] Here, for example, if a particular pixel belongs to both the first local region 501a and the second local region 501a (i.e., the region containing the pixel overlaps between the first local region 501a and the second local region 501a), the blur value estimated for the pixel belonging to the first local region 501a may differ from the blur value estimated for the pixel belonging to the second local region 501a.

[0074] Therefore, for example, if multiple local regions 501a that partially overlap are extracted as described above, the blur value of the pixels constituting the overlapping region of the multiple local regions 501a may be, for example, the average of the blur value estimated for a portion of one of the overlapping local regions 501a (pixels) and the blur value estimated for a portion of the other local region 501a (pixels). Alternatively, it may be determined by majority vote of the blur values ​​estimated for each portion of three or more overlapping local regions 501a.

[0075] Figure 13 shows an example of information about the local region 501a that is input into the statistical model in the first method described above.

[0076] As shown in Figure 13, the statistical model is input with gradient data of the local region 501a extracted from image 501. The gradient data of the local region 501a is generated from the R image, G image, and B image contained in image 501, and includes gradient data for the R image, G image, and B image.

[0077] The gradient data represents the difference (difference value) between each pixel and its adjacent pixels. For example, if a local region 501a is extracted as a rectangular region of n pixels (X-axis direction) x m pixels (Y-axis direction), gradient data is generated by arranging the difference values ​​calculated for each pixel within the local region 501a, for example, between that pixel and the pixel to its right, in an n-row x m-column matrix.

[0078] The statistical model uses the gradient data of the R image, the G image, and the B image to estimate the blur value, which indicates the blur occurring in each image. Figure 13 shows the case where the gradient data of the R, G, and B images are input to the statistical model, but it is also possible to configure the model so that the gradient data of image 501 (RGB image) is input to the statistical model.

[0079] Next, the second method will be explained with reference to Figure 14. In the second method, the gradient data for each local region (image patch) 501a and the positional information of the local region 501a in image 501 are input into the statistical model as information related to the local region 501a in the first method.

[0080] The position information 501b may, for example, indicate the center point of the local region 501a, or it may indicate a predetermined side, such as the upper left edge. Alternatively, the position information 501b may be the position on image 501 of each pixel constituting the local region 501a.

[0081] As described above, by further inputting the position information 501b into the statistical model, it is possible to estimate a blur value 502 that takes into account the difference between the blur of the subject image formed by light passing through the center of the lens 21 and the blur of the subject image formed by light passing through the edges of the lens 21.

[0082] In other words, according to this second method, the blur value can be estimated from image 501 based on its correlation with its position on the image.

[0083] Figure 15 shows an example of information about the local region 501a that is input into the statistical model in the second method described above.

[0084] For example, if a rectangular region of n pixels (X-axis direction) x m pixels (Y-axis direction) is extracted as a local region 501a, then the X coordinate value (X-coordinate data) on image 501 corresponding to, for example, the center point of the local region 501a, and the Y coordinate value (Y-coordinate data) on image 501 corresponding to, for example, the center point of the local region 501a are obtained.

[0085] In the second method, the X-coordinate data and Y-coordinate data acquired in this way are input into the statistical model along with the gradient data of the R, G, and B images described above.

[0086] Furthermore, the third method will be explained with reference to Figure 16. In the third method, local region 501a is not extracted from image 501 as in the first and second methods described above. In the third method, for example, information about the entire region of image 501 (gradient data of the R image, G image, and B image) is input into the statistical model.

[0087] Compared to the first and second methods, which estimate the blur value 502 for each local region 501a, the third method may have a higher uncertainty in the estimation by the statistical model, but it can reduce the processing load associated with the estimation.

[0088] In the following explanation, the information input into the statistical model in the first to third methods described above will be referred to as image-related information for convenience.

[0089] Here, we have explained that the blur value is estimated for each pixel, but this blur value may also be estimated for each predetermined region containing at least one pixel.

[0090] The correlation between the blurring that occurs in the image in this embodiment and the distance to the subject in the image will be explained in detail below with reference to Figure 17.

[0091] In Figure 17, the size of the blur that occurs when the subject is closer to the focal point (in front) is shown as a negative value on the X-axis, and the size of the blur that occurs when the subject is farther away from the focal point (in the background) is shown as a positive value on the X-axis. In other words, considering that the color observed in the blur that occurs around the subject differs depending on whether the subject is closer to the focal point or farther away, as described above, it can be said that in Figure 17, the color and size of the blur are shown as positive and negative values.

[0092] Furthermore, Figure 17 shows that, in both cases—when the subject is closer to the focal point and when the subject is farther from the focal point—the absolute value of the blur size (pixels) increases as the subject moves further away from the focal point.

[0093] In the example shown in Figure 17, it is assumed that the focal point in the optical system of the image capture unit 2 is approximately 1500 mm. In this case, for example, a blur of approximately -4.8 pixels corresponds to a distance of approximately 1000 mm from the optical system, a blur of 0 pixels corresponds to a distance of 1500 mm from the optical system, and a blur of approximately 4.8 pixels corresponds to a distance of approximately 750 mm from the optical system.

[0094] Here, for convenience, we have explained the case where the color and size (pixels) of the blur are shown on the X-axis. However, as explained in Figures 6 to 10 above, the shape of the blur (PSF shape) that occurs in the image also differs depending on whether the subject is closer or further away than the focal point, and also depending on its position in the image. Therefore, the value shown on the X-axis in Figure 17 (i.e., the blur value) actually reflects the shape of the blur (PSF shape).

[0095] As mentioned above, there is a correlation between the distance to the subject and the color, size, and shape of the blur, as shown by line segment d1 in Figure 17, for example. Therefore, estimating the distance is synonymous with estimating the color, size, and shape (and the blur value) of the blur.

[0096] While it is possible to directly estimate distance using a statistical model, for example, a configuration in which the statistical model estimates the blur value is more versatile because it allows the same statistical model to be used even when the focus position (focus distance) in the optical system is changed.

[0097] In this embodiment, by using the statistical model described above, a blur value indicating the blur that occurs in the image according to the distance from the image to the subject in the image can be obtained (estimated). This statistical model is generated by performing a learning process (hereinafter referred to as the learning process) that learns the blur that occurs in an image affected by the aberrations of the optical system of the imaging unit 2 (blur that changes nonlinearly according to the distance to the subject in the image).

[0098] The following is an overview of the training process performed to generate a statistical model (hereinafter referred to as the statistical model training process).

[0099] Figure 18 shows an example of the training process (training method) of the statistical model in this embodiment. The training process of the statistical model is performed using images prepared for training the statistical model (hereinafter referred to as training images). The training images may be images captured by the imaging unit 2, for example, but may also be images captured by another imaging device (camera, etc.) having an optical system similar to that of the imaging unit 2.

[0100] Regardless of which of the three methods used—the first method described with reference to Figure 12, the second method described with reference to Figure 14, or the third method described with reference to Figure 16—the statistical model learning process is basically carried out by inputting information about the training image 601 into the statistical model and feeding back the error between the blur value 602 estimated by the statistical model and the correct value 603 to the statistical model. Feedback refers to updating the parameters of the statistical model (e.g., weight coefficients) so that the error decreases.

[0101] When the first method is applied to estimate the blur value from the image described above, during the training process of the statistical model, information (gradient data) about each local region (image patch) extracted from the training image 601 is input to the statistical model, and the statistical model estimates the blur value 602 of each pixel within each local region. The error obtained by comparing the estimated blur value 602 with the correct value 603 is fed back to the statistical model.

[0102] Similarly, when the second method is applied as a method for estimating blur values ​​from an image, during the training process of the statistical model, gradient data and positional information are input to the statistical model as information about each local region (image patch) extracted from the training image 601, and the statistical model estimates the blur value 602 of each pixel within each local region. The error obtained by comparing the estimated blur value 602 with the ground truth value 603 is fed back to the statistical model.

[0103] Furthermore, when the third method is applied as the method for estimating distance from an image, during the training process of the statistical model, information (gradient data) about the entire region of the training image 601 is input to the statistical model all at once, and the statistical model estimates the blur value 602 of each pixel in the training image 601. The error obtained by comparing the estimated blur value 602 with the ground truth value 603 is fed back to the statistical model.

[0104] According to the training process of the statistical model described above, the parameters of the statistical model are updated so that the error between the blur value 602 and the correct value 603 decreases, and the statistical model can learn the blur occurring in the training image 601.

[0105] In this embodiment, the statistical model is generated by repeatedly performing a learning process using training images captured while varying the distance from the imaging unit 2 to the subject, for example, with the focus position fixed. Furthermore, once the learning process for one focus position is completed, a more accurate statistical model can be generated by performing the same learning process for other focus positions.

[0106] Furthermore, the correct values ​​used in the training process of the statistical model in this embodiment are the blur values ​​converted from the actual distance to the subject when the training image was captured (i.e., blur values ​​that indicate the color, size, and shape of the blur corresponding to the actual distance), as described above.

[0107] Next, an example of the processing procedure for learning a statistical model will be described with reference to the flowchart in Figure 19. Note that the processing shown in Figure 19 may be performed, for example, in the distance measuring device 1 (image processing unit 3), or in other devices.

[0108] First, pre-prepared training images (information about them) are input into the statistical model (step S1). These training images are, for example, images generated by the image sensor 22 based on light transmitted through the lens 21 provided in the imaging unit 2, and are images affected by the aberrations of the optical system (lens 21) of the imaging unit 2. Specifically, the training images have a blur that changes non-linearly depending on the distance to the subject, as explained in Figures 4 to 10.

[0109] Furthermore, in the training process of the statistical model, training images are prepared in advance, taken of the subject at the finest possible granularity, from the lower limit (closer) to the upper limit (farther) of the distance that can be measured (estimated) by the distance measuring device 1. In addition, it is preferable to prepare various images of different subjects as training images.

[0110] When the first method described above is applied as a method for estimating blur value from an image, gradient data of the R image, G image, and B image are input into the statistical model as information about the training image for each local region of the training image.

[0111] When the second method described above is applied as a method for estimating blur value from an image, the statistical model receives, as information about the training image, gradient data of the R image, G image, and B image, and the position information of the local region on the training image for each local region of the training image.

[0112] When the third method described above is applied as a method for estimating blur value from an image, gradient data of the R, G, and B images for the entire region of the training image are input into the statistical model as information about the training image.

[0113] In this embodiment, the gradient data of the R, G, and B images are described as being input to the statistical model. However, if the statistical model estimates the blur value from the perspective of the shape of the blur (PSF shape) that occurs in the image, it is sufficient for at least one of the gradient data of the R, G, and B images to be input to the statistical model. On the other hand, if the statistical model estimates the blur value from the perspective of the color and size of the blur that occurs in the image due to chromatic aberration, it is sufficient for at least two of the gradient data of the R, G, and B images to be input to the statistical model.

[0114] Once the process in step S1 is executed, the statistical model estimates the blur value (step S2).

[0115] The blur value estimated in step S2 is compared with the ground truth value obtained when the training images were captured (step S3).

[0116] The comparison result (error) in step S3 is fed back into the statistical model (step S4). As a result, the parameters in the statistical model are updated to reduce the error (i.e., the blur occurring in the training images according to the distance to the subject is learned).

[0117] The process shown in Figure 19 above is repeatedly executed for each training image to generate a statistical model with high estimation accuracy. The statistical model thus generated is stored in the storage unit 31 included in the image processing unit 3.

[0118] The distance measuring device 1 according to this embodiment measures the distance to the subject in the image captured by the imaging unit 2 using the statistical model generated as described above.

[0119] The following describes an example of the processing procedure of the distance measuring device 1 (imaging unit 2 and image processing unit 3) when measuring the distance to the subject, referring to the sequence chart in Figure 20.

[0120] First, the first image acquisition unit 23 included in the imaging unit 2 acquires an image (hereinafter referred to as the captured image) that includes the subject captured by the imaging unit 2 (image sensor 22) (step S11). As described above, this captured image is affected by the aberrations of the optical system (lens 21) of the imaging unit 2. Although not shown in Figure 20, the captured image acquired in step S11 may be recorded within the imaging unit 2.

[0121] Next, the first focus position information acquisition unit 24 included in the imaging unit 2 acquires focus position information relating to the focus position when the above-mentioned image was captured (step S12).

[0122] The focus position information in this embodiment will be described below with reference to Figure 21. Figure 21 schematically shows the optical system of the imaging unit 2.

[0123] As shown in Figure 21, the focus position (the position where the image captured by the imaging unit 2 is in focus) depends on the distance between the lens 21 and the image sensor 22 (relative lens position). In other words, the focus position is adjusted (set) by moving the position of the lens 21 in a direction parallel to the optical axis of the optical system (i.e., driving the lens 21).

[0124] As mentioned above, the lens 21 constitutes a lens unit, which includes a signal processing unit 2a and a lens drive unit 2b. The signal processing unit 2a and the lens drive unit 2b operate to control the position of the lens 21. Specifically, the signal processing unit 2a generates a control signal value (adjustment value) for driving the lens 21 in response to instructions from a control circuit (not shown) that controls the operation of the imaging unit 2 (camera) (for example, a focus position specified by the autofocus function), and transmits the control signal value to the lens drive unit 2b. The lens drive unit 2b drives the lens 21 based on the control signal value transmitted from the signal processing unit 2a.

[0125] As described above, when the focus position is adjusted by driving the lens 21, the control signal value for driving the lens 21 can be used as focus position information.

[0126] Specifically, for example, when the position of the lens 21 is mechanically controlled by an electric drive, the signal processing unit 2a generates a voltage value corresponding to the amount of movement of the lens 21 as a control signal value, and transmits this voltage value to the lens drive unit 2b. In this case, the voltage value transmitted from the signal processing unit 2a to the lens drive unit 2b is used as focus position information.

[0127] Furthermore, in a configuration where a unique focus adjustment amount for a preset focus position (reference value) is stored within the camera shooting software (software that operates to capture images), this focus adjustment amount is transmitted to the signal processing unit 2a, and the voltage value converted from this focus adjustment amount is transmitted from the signal processing unit 2a to the lens drive unit 2b. In this case, the focus adjustment amount stored within the software (i.e., the focus adjustment amount transmitted to the signal processing unit 2a) may be considered to correspond to the control signal value for driving the lens 21, and this focus adjustment amount may be used as focus position information.

[0128] The control signal values ​​for driving the lens 21 described here are just examples; the focus position information may be other control signal values ​​related to driving the lens 21.

[0129] Furthermore, the focus position information may also be, for example, information relating to the position of the lens 21, which moves in a direction parallel to the optical axis of the optical system of the imaging unit 2.

[0130] Specifically, the focus position (i.e., the position of the lens 21) may be adjusted by manually turning, for example, the lens barrel 21a or other screws on which the lens 21 is located. In this case, for example, the number of rotations of the lens barrel 21a or screws can be used as focus position information.

[0131] Furthermore, the relative position of the lens 21 inside the lens barrel 21a (i.e., the position of the lens 21 relative to the image sensor 22) may be used as focus position information. This relative position of the lens 21 may be, for example, the distance between the principal point 21b of the lens 21 and the image sensor 22, or it may be the position of the principal point 21b of the lens 21 relative to the reference point of the lens barrel 21a. Here, the case using the principal point 21b of the lens 21 has been mainly described, but it is not necessary to use the principal point 21b of the lens 21; for example, the distance between the front end of the lens 21 and the end of the lens barrel 21a may be used as focus position information. The relative position of the lens 21 described above can be obtained, for example, using a predetermined sensor, but it may also be obtained by other methods.

[0132] In step S12, it was explained that focus position information regarding the focus position when the image is captured is acquired. However, this focus position information can be acquired when the focus position is adjusted (set). In other words, the process in step S12 may be performed after the focus position has been adjusted but before the image is captured (i.e., before the process in step S11 is executed).

[0133] Furthermore, although step S12 was described as acquiring focus position information, if there is no focus position information regarding the focus position at the time the captured image was taken (or if it is not possible to acquire such focus position information), pre-prepared focus position information (or manually set focus position information) may be acquired.

[0134] Next, the focus position information acquired in step S12 is added to the captured image acquired in step S11 (step S13). In this case, the focus position information is recorded in the captured image as image metadata, such as EXIF ​​(Exchangeable Image File Format), a digital camera format that can store the date and time the image was taken and other setting data (focal length, aperture value, etc.) in the image. As a result, an captured image is obtained in which the focus position information is embedded in the header as metadata in the metadata.

[0135] Although this explanation assumes that the focus position information is embedded in the captured image, this focus position information may also be added to the captured image as another electronic file.

[0136] When the process in step S13 is executed, the captured image with focus position information added in step S13 is transmitted from the imaging unit 2 to the image processing unit 3 (step S14).

[0137] The captured image transmitted in step S14 is received by the image processing unit 3. As a result, the second image acquisition unit 32 included in the image processing unit 3 acquires the captured image received by the image processing unit 3 (step S15).

[0138] Furthermore, the second focus position information acquisition unit 33 included in the image processing unit 3 acquires the focus position information attached to the captured image acquired in step S15 (step S16).

[0139] Next, the second focus position information acquisition unit 33 acquires the actual focus distance (i.e., the actual distance from the distance measuring device 1 to the focus position) based on the focus position information acquired in step S16 (step S17).

[0140] Figure 22 shows the correspondence between focus position information and actual focus distance. Note that, for convenience, Figure 22 shows normalized focus position information and actual focus distance.

[0141] In this embodiment, a mathematical model (actual focus distance conversion model) representing the correspondence (relationship) between focus position information and actual focus distance, as shown in Figure 22, is assumed to be stored within the image processing unit 3 (second focus position information acquisition unit 33). According to this, the second focus position information acquisition unit 33 can obtain the actual focus distance by converting the focus position information acquired in step S16 into the actual focus distance by referring to such a mathematical model.

[0142] Here, the correspondence between focus position information and actual focus distance is described as being stored in the image processing unit 3 in the form of a mathematical model, but this correspondence may also be stored in the form of a table.

[0143] It should be assumed that the correspondence between the focus position information and the actual focus distance described above has been obtained in advance, for example, by experimental means (i.e., by actually measuring the actual focus distance corresponding to the focus position information).

[0144] When the process in step S17 is executed, the blur value acquisition unit 34 inputs the captured image acquired in step S15 into the statistical model stored in the storage unit 31 and acquires the blur value output from the statistical model (i.e., the blur value estimated by the statistical model) (step S18). The blur value acquired in step S18 corresponds to relative distance information corresponding to the distance to the subject in the captured image, as described above. The process in step S18 corresponds to the processes in steps S1 and S2 shown in Figure 19 above, so a detailed explanation is omitted here.

[0145] Although Figure 20 describes the process as being executed in the order of steps S17 and S18, the order in which steps S17 and S18 are executed may be changed.

[0146] Next, based on the actual focus distance acquired in step S17, the actual distance conversion unit 35 converts the blur value acquired in step S18 into an actual distance (i.e., the actual distance to the subject in the captured image) (step S19).

[0147] Here, the actual distance u converted from the blur value in step S19 is represented by the following formula (1) using the blur value b.

Equation

[0148] In formula (1), f represents the focal length in the optical system (lens 21) of the imaging unit 2 that captured the captured image. u f represents the actual focus distance when the captured image was captured. F represents the F value (aperture value) in the optical system of the imaging unit 2 that captured the captured image.

[0149] That is, in the present embodiment, the actual distance can be calculated by applying the actual focus distance acquired in step S17, the blur value acquired in step S18, the focal length and the F value in the optical system of the imaging unit 2 to formula (1).

[0150] Note that since the focal length and the F value in the optical system of the imaging unit 2 are held in the memory mounted on the lens unit (control circuit) as the above-described lens information, they can be acquired from the imaging unit 2. Also, the lens information (focal length and F value) may be added (recorded) to the captured image as a metafile, similar to the above-described focus position information.

[0151] When the process in step S19 is executed, the output unit 36 ​​outputs distance information, which is converted from the blur value in step S19, in a map format, for example, that is arranged in a positional correspondence with the captured image (step S20). In this embodiment, the distance information has been described as being output in a map format, but the distance information may be output in other formats.

[0152] Here, it has been explained that the image processing unit 2 (first focus position information acquisition unit 24) converts the acquired focus position information to the actual focus distance by referring to, for example, an actual focus distance conversion model. However, if, for example, an external distance sensor of the distance measuring device 1 can be used to measure (acquire) the distance to a subject that is in focus in the captured image (for example, one subject among multiple subjects in the captured image), then that distance (measured value) can be used as the actual focus distance. As a distance sensor, for example, a Lidar (Light detection and ranging) that can measure the distance to a subject by receiving the reflected wave of light (electromagnetic wave) transmitted to the subject can be used. With such a configuration, the processing in the image processing unit 3 to convert the focus position information to the actual focus distance (actual focus distance conversion processing) can be omitted, thereby reducing processing costs (computation costs) while improving the accuracy of the actual distance based on the actual focus distance (the actual distance from the camera to the subject converted from the blur value using the actual focus distance).

[0153] Furthermore, although Figure 20 describes the case where one statistical model is stored in the storage unit 31, if the rangefinder 1 is configured to allow lens replacement as described above, it is conceivable to prepare a statistical model for each lens that can be used with the rangefinder 1 (i.e., a lens that can be attached to the rangefinder 1). In this case, for example, the system may be configured to select a statistical model corresponding to the lens attached to the rangefinder 1 using the lens information described above (information on the lens specifications or design values, including focal length, F-number, and camera model name), and then estimate the blur value using the selected statistical model.

[0154] As described above, in this embodiment, an image is acquired (a second image captured by the imaging unit 2, which is affected by the aberrations of the optical system of the imaging unit 2), focus position information regarding the focus position when the image was captured is acquired, the image is input into a statistical model (a statistical model generated by learning the blur that occurs in the first image affected by the aberrations of the optical system of the imaging unit 2, which changes nonlinearly according to the distance to the subject in the first image), a blur value (a blur value indicating the blur that occurs in the subject in the image) is output from the statistical model, and the blur value is converted to the actual distance to the subject based on the focus position information.

[0155] In this embodiment, as described above, when acquiring an image, the accuracy of the distance measured from the image can be improved by acquiring focus position information regarding the focus position of the image at the time of acquisition.

[0156] Specifically, in order to convert the blur value, which indicates the blurring that occurs in the subject in the captured image, into actual distance, the actual focus distance is required, as shown in equation (1) above. If such an actual focus distance is set manually, for example, there is a possibility of errors due to human factors (errors in setting the actual focus distance), and such errors in the actual focus distance lead to a decrease in the accuracy of the actual distance converted from the blur value.

[0157] Furthermore, if, for example, the subject is excessively far from the focal point (i.e., the blur is excessively large), the rangefinder 1 may not be able to measure the actual distance to the subject. In this case, it is possible to change (adjust) the focal point when capturing an image that includes the subject. For example, even if the accurate actual focal distance was set before the focal point was changed, if the focal point is changed in this way, it is necessary to set a new actual focal distance (the actual distance from the rangefinder 1 to the focal point) based on the changed focal point, which may result in errors due to the aforementioned human factors. Moreover, there is the added effort of measuring and setting the actual focal distance each time the focal point is changed.

[0158] In contrast, this embodiment acquires focus position information when the captured image is taken and converts the blur value to the actual distance based on that focus position information. This configuration avoids errors in setting the actual focus distance and also allows for easy adaptation to changes in the focus position when capturing the image.

[0159] In this embodiment, the focus position information can be, for example, a control signal value for driving the lens 21 or the position of the lens moving parallel to the optical axis of the optical system of the imaging unit 2.

[0160] Furthermore, in this embodiment, the focus position information is converted to the actual focus distance by referring to the actual focus distance conversion model, and the blur value is converted to the actual distance using the actual focus distance. Therefore, it is possible to measure (acquire) an appropriate actual distance to the subject.

[0161] Furthermore, in this embodiment, the focus position information acquired by the imaging unit 2 (first focus position information acquisition unit 24) that captures the image is added to the image and transmitted to the image processing unit 3. However, the focus position information may be embedded in the header portion of the image as metadata (metafile), or it may be added to the image as a separate electronic file from the image. In other words, in this embodiment, it is sufficient that the image acquired by the imaging unit 2 and the focus position information related to the focus position at which the image was captured are passed from the imaging unit 2 to the image processing unit 3.

[0162] Incidentally, generally speaking, there are individual differences in the cameras incorporated into the rangefinder 1 (cameras as a product), and the correspondence between the focus position information and the actual focus distance described above may differ from one camera to another.

[0163] Therefore, in this embodiment, the imaging unit 2 may further include a focus position information correction unit 25, as shown in Figure 23, for example.

[0164] The focus position information correction unit 25 corrects the focus position information acquired by the first focus position information acquisition unit 24 based on correction values ​​that have been held (prepared) in advance. The focus position information corrected by the focus position information correction unit 25 is added to the image acquired by the first image acquisition unit 23 and transmitted from the imaging unit 2 to the image processing unit 3.

[0165] In this embodiment, it has been explained that a real-focus distance conversion model representing the correspondence between focus position information and the actual focus distance is pre-held in the image processing unit 3 (second focus position information acquisition unit 33). The correction value held by the focus position information correction unit 25 is a value for correcting the focus position information so that the focus position information acquired by the first focus position information acquisition unit 24 is converted to an appropriate actual focus distance by referring to the real-focus distance conversion model (i.e., absorbs the individual differences described above).

[0166] This correction value (i.e., the amount of deviation in focus position information between the camera incorporated in the rangefinder 1 and the reference camera) is obtained by comparing the correspondence between the focus position information and the actual focus distance (the actual distance from the rangefinder 1 to the focus position) when an image is captured at an arbitrary focus position, for example, during product inspection at the time of shipment of the product (camera), with the actual focus distance conversion model (the correspondence between the focus position information and the actual focus distance represented by it) held in the image processing unit 3 described above.

[0167] With this configuration, even if there are individual differences in the cameras incorporated into the rangefinder 1, the actual distance to the subject can be appropriately measured based on focus position information that has been corrected to absorb these individual differences.

[0168] In this explanation, the imaging unit 2 (focus position information correction unit 25) corrects the focus position information, but this correction of focus position information may also be performed on the image processing unit 3 side, for example.

[0169] Furthermore, although this embodiment has described the image processing unit 3 as including each of the parts 31 to 36, for example, the storage unit 31 may be located in an external device different from the distance measuring device 1. In this case, the image processing unit 3 may operate to utilize a statistical model acquired from the external device. Also, in this embodiment, for example, a part of the processing performed by each of the parts 32 to 36 may be performed by an external device.

[0170] (Examples of application) The following describes application examples to which the distance measuring device 1 according to this embodiment is applied.

[0171] Figure 24 shows an example of the functional configuration of a mobile body 700 into which the range measuring device 1 is incorporated. The mobile body 700 can be realized as, for example, an automobile with an autonomous driving function, an unmanned aerial vehicle, or an autonomous mobile robot. Unmanned aerial vehicles are airplanes, rotary-wing aircraft, gliders, and airships that cannot carry people and can be flown by remote control or autopilot, and include, for example, drones (multicopters), radio-controlled aircraft, and helicopters for spraying agricultural chemicals. Autonomous mobile robots include mobile robots such as automated guided vehicles (AGVs), cleaning robots for cleaning floors, and communication robots that provide various information to visitors. The mobile body 700 includes not only robots whose main body moves, but also industrial robots that have a drive mechanism that moves or rotates a part of the robot, such as a robot arm.

[0172] As shown in Figure 24, the moving body 700 includes, for example, a distance measuring device 1, a control signal generation unit 701, and a drive mechanism 702. The distance measuring device 1 is installed such that, for example, an imaging unit 2 can image a subject in the direction of travel of the moving body 700 or a part thereof.

[0173] As shown in Figure 25, when the mobile body 700 is a car 700A, the distance measuring device 1 is installed as a so-called front camera that captures images of the area in front. The distance measuring device 1 may also be installed as a so-called rear camera that captures images of the area behind when reversing. Furthermore, multiple distance measuring devices 1 may be installed as front and rear cameras. In addition, the distance measuring device 1 may also be installed to function as a so-called drive recorder. That is, the distance measuring device 1 may also be a recording device.

[0174] Figure 26 shows an example where the mobile body 700 is a drone 700B. The drone 700B comprises a drone body 711 corresponding to the drive mechanism 702 and four propeller sections 712 to 715. Each propeller section 712 to 715 has a propeller and a motor. The drive from the motor is transmitted to the propeller, causing it to rotate, and the lift generated by this rotation causes the drone 700B to float. A rangefinder 1 is mounted, for example, on the lower part of the drone body 711.

[0175] Figure 27 also shows an example where the mobile body 700 is an autonomous mobile robot 700C. A power unit 721, which includes a motor and wheels, is provided at the bottom of the mobile robot 700C, corresponding to the drive mechanism 702. The power unit 721 controls the rotation speed of the motor and the direction of the wheels. The mobile robot 700C can move in any direction by having its wheels, which are placed on the road or floor, rotate when the motor drive is transmitted, and by controlling the direction of the wheels. In the example shown in Figure 27, the distance measuring device 1 is installed on the head of the mobile robot 700C, for example, so that the imaging unit 2 images the area in front of the humanoid mobile robot 700C. The distance measuring device 1 may also be installed to image the area behind or to the left and right of the mobile robot 700C, or multiple devices may be installed to image multiple directions. Furthermore, dead reckoning can also be performed by installing the distance measuring device 1 on a small robot with limited space for mounting sensors, etc., and estimating its own position, posture, and the position of the subject.

[0176] As shown in Figure 28, if the moving body 700 is a robot arm 700D, and the movement and rotation of a part of the robot arm 700D are controlled, the distance measuring device 1 may be installed at the tip of the robot arm 700D. In this case, the imaging unit 2 provided in the distance measuring device 1 captures an image of the object being grasped by the robot arm 700D, and the image processing unit 3 can measure the distance to the object that the robot arm 700D is trying to grasp. This enables the robot arm 700D to perform accurate grasping operations of objects.

[0177] The control signal generation unit 701 outputs a control signal for controlling the drive mechanism 702 based on distance information indicating the distance to the subject output from the distance measuring device 1 (image processing unit 3). The drive mechanism 702 drives the moving body 700 or a part of the moving body 700 based on the control signal output from the control signal generation unit 701. The drive mechanism 702 performs at least one of the following actions: moving, rotating, accelerating, decelerating, adjusting thrust (lift), changing direction, switching between normal driving mode and automatic driving mode (collision avoidance mode), and activating safety devices such as airbags. The drive mechanism 702 may also perform at least one of the following actions: moving, rotating, accelerating, adjusting thrust (lift), changing direction toward approaching the object, and switching from automatic driving mode (collision avoidance mode) to normal driving mode, for example, when the distance to the subject is less than a threshold.

[0178] For example, the drive mechanism 702 of the automobile 700A shown in Figure 25 is a tire. The drive mechanism 702 of the drone 700B shown in Figure 26 is a propeller. The drive mechanism 702 of the mobile robot 700C shown in Figure 27 is a leg. The drive mechanism 702 of the robot arm 700D shown in Figure 28 is a support part that supports the tip on which the distance measuring device 1 is installed.

[0179] The mobile unit 700 may further include a speaker or display to which distance information (distance information) to the subject output from the range measuring device 1 is input. This speaker or display is connected to the range measuring device 1 by wire or wireless and is configured to output audio or images related to the distance to the subject. Furthermore, the mobile unit 700 may have a light-emitting unit that receives distance information to the subject output from the range measuring device 1 and can be turned on and off, for example, according to the distance to the subject.

[0180] Furthermore, for example, if the mobile body 700 is a drone 700B, when tasks such as creating maps (three-dimensional shapes of objects), surveying the structure of buildings and terrain, and inspecting for cracks, broken power lines, etc., are performed from above, the imaging unit 2 acquires images of the target and determines whether the distance to the subject is above a threshold. Based on this determination, the control signal generation unit 701 generates a control signal to control the thrust of the drone 700B so that the distance to the inspection target remains constant. Here, thrust includes lift. The drive mechanism 702 operates the drone 700B based on this control signal, allowing the drone 700B to fly parallel to the inspection target. If the mobile body 700 is a surveillance drone 700B, the control signal generation unit 701 may generate a control signal to control the thrust of the drone 700B so that the distance to the object being monitored remains constant.

[0181] Furthermore, when the mobile unit 700 (for example, the drone 700B) is used for maintenance and inspection of various infrastructure (hereinafter simply referred to as "infrastructure"), the distance to the repair location can be obtained by capturing images of the repair location (hereinafter referred to as "repair location"), including cracks or rust, with the imaging unit 2. In this case, the size of the repair location can be calculated from the image using the distance to the repair location. This allows, for example, the repair location to be displayed on a map representing the entire infrastructure, thereby enabling maintenance personnel to recognize the repair location. In addition, informing maintenance personnel of the size of the repair location in advance is useful for carrying out repair work smoothly.

[0182] In this explanation, we have described a case where a mobile device 700 (for example, a drone 700B) incorporating the distance measuring device 1 is used for infrastructure maintenance and inspection. However, if the distance measuring device 1 is implemented as a smartphone or the like, for example, a maintenance worker can perform infrastructure maintenance and inspection by using the smartphone to take images of the repair area. Furthermore, if the imaging device provided in the distance measuring system described above is implemented as a smartphone or the like, similar maintenance and inspection can be performed by uploading images of the repair area taken by the maintenance worker using the smartphone to an image processing device.

[0183] When uploading images, for example, by using a method that transfers the images to an image processing device on a server via a network, inspection work can be easily performed at the maintenance and inspection site.

[0184] Furthermore, during the flight of the drone 700B, the imaging unit 2 acquires an image of the ground direction and determines whether the distance to the ground is above a threshold. Based on this determination, the control signal generation unit 701 generates a control signal to control the thrust of the drone 700B so that its height from the ground becomes a specified height. The drive mechanism 702 operates the drone 700B based on this control signal, thereby allowing the drone 700B to fly at the specified height. If the drone 700B is a drone for spraying pesticides, maintaining a constant height of the drone 700B from the ground in this way makes it easier to spray pesticides evenly.

[0185] Furthermore, when the mobile unit 700 is a car 700A or a drone 700B, the imaging unit 2 captures images of the car in front or the surrounding drones during the formation of the car 700A or coordinated flight of the drone 700B, and determines whether the distance to the car or drone is above a threshold. Based on this determination, the control signal generation unit 701 generates control signals to control the speed of the car 700A or the thrust of the drone 700B so that the distance to the car in front or the surrounding drones remains constant. The drive mechanism 702 operates the car 700A or the drone 700B based on these control signals, making it easy to perform formation driving of the car 700A or coordinated flight of the drone 700B.

[0186] Furthermore, if the mobile unit 700 is a car 700A, the system may be configured to receive driver instructions via a user interface so that the driver of the car 700A can set (change) the threshold. This allows the driver to drive the car 700A at a preferred following distance. In addition, the threshold may be changed according to the speed of the car 700A in order to maintain a safe following distance from the car in front. A safe following distance varies depending on the speed of the car 700A. Therefore, the faster the speed of the car 700A, the larger (longer) the threshold can be set.

[0187] Furthermore, if the moving object 700 is an automobile 700A, a predetermined distance in the direction of travel may be set as a threshold, and a control signal may be generated to activate the brakes or safety devices such as airbags when an object appears in front of that threshold. In this case, safety devices such as automatic brakes or airbags are provided in the drive mechanism 702.

[0188] Each of the various functions described in this embodiment may be implemented by a circuit (processing circuit). Examples of processing circuits include a programmed processor, such as a central processing unit (CPU). This processor performs each of the described functions by executing computer programs (sets of instructions) stored in memory. This processor may be a microprocessor including electrical circuits. Examples of processing circuits also include digital signal processors (DSPs), application-specific integrated circuits (ASICs), microcontrollers, controllers, and other electrical circuit components. Each of the components other than the CPU described in this embodiment may also be implemented by a processing circuit.

[0189] Furthermore, since the various processes of this embodiment can be implemented by a computer program, the same effects as this embodiment can be easily achieved simply by installing and executing this computer program on a computer via a computer-readable storage medium containing the computer program.

[0190] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0191] 1... Distance measuring device, 2... Imaging unit, 2a... Signal processing unit, 2b... Lens drive unit, 3... Image processing unit, 21... Lens, 22... Image sensor, 23... First image acquisition unit, 24... First focus position information acquisition unit, 25... Focus position information correction unit, 31... Storage unit, 32... Second image acquisition unit, 33... Second focus position information acquisition unit, 34... Blur value acquisition unit, 35... Actual distance conversion unit, 36... Output unit, 101... CPU, 102... Non-volatile memory, 103... RAM, 103A... Distance measuring program, 104... Communication device.

Claims

1. In a distance measuring device having an imaging unit, a storage means for storing a statistical model generated by learning blur that occurs in a first image affected by aberration of the optical system of the imaging unit and that changes nonlinearly depending on the distance to a subject in the first image; a first acquisition means for acquiring a second image captured by the imaging unit, the second image being affected by aberration of the optical system of the imaging unit; a second acquisition means for acquiring focus position information relating to a focus position when the second image is captured; a third acquisition means for inputting the acquired second image into the statistical model and acquiring a blur value indicating blur occurring in a subject in the second image, the blur value being output from the statistical model; a first conversion means for converting the acquired blur value into a distance to the subject based on the acquired focus position information; A ranging device comprising:

2. the optical system of the imaging unit includes a lens that moves in a direction parallel to an optical axis of the optical system, The focus position information includes a control signal value for driving the lens.

2. The distance measuring device according to claim 1.

3. the optical system of the imaging unit includes a lens that moves in a direction parallel to an optical axis of the optical system, The focus position information includes the position of the lens.

2. The distance measuring device according to claim 1.

4. The camera further includes a second conversion unit that converts the acquired focus position information into a distance to the focus position, The first conversion means converts the acquired blur value into a distance to the subject based on the distance to the focus position converted from the focus position information.

2. The distance measuring device according to claim 1.

5. 5. The distance measuring device according to claim 1, wherein the focus position information is added to the second image in the imaging unit and transmitted from the imaging unit.

6. 6. The distance measuring device according to claim 5, wherein the focus position information is embedded as metadata in a header portion of the second image.

7. 6. The distance measuring device according to claim 5, wherein the focus position information is added to the second image as an electronic file separate from the second image.

8. 8. The distance measuring device according to claim 1, further comprising a correction unit that corrects the acquired focus position information based on a correction value that is stored in advance.

9. In an image processing device connected to an imaging device, a storage means for storing a statistical model generated by learning blur that occurs in a first image affected by aberration of an optical system of an imaging device and that changes nonlinearly depending on the distance to a subject in the first image; a first acquisition means for acquiring, from the imaging device, a second image captured by the imaging device, the second image being affected by aberration of the optical system of the imaging device; a second acquisition means for acquiring, from the imaging device, focus position information relating to a focus position when the second image was captured; a third acquisition means for inputting the acquired second image into the statistical model and acquiring a blur value indicating blur occurring in a subject in the second image, the blur value being output from the statistical model; a conversion means for converting the acquired blur value into a distance to the subject based on the acquired focus position information; An image processing device comprising:

10. A method executed by a distance measuring device including an imaging unit and a storage means for storing a statistical model generated by learning blur that occurs in a first image affected by aberration of an optical system of the imaging unit and that changes nonlinearly depending on a distance to a subject in the first image, the method comprising: acquiring a second image captured by the imaging unit, the second image being affected by aberration of the optical system of the imaging unit; acquiring focus position information regarding a focus position when the second image was captured; a step of inputting the acquired second image into the statistical model to obtain a blur value that indicates blur occurring in an object in the second image and is output from the statistical model; converting the acquired blur value into a distance to the subject based on the acquired focus position information; A method comprising:

11. 1. A method executed by an image processing device connected to an imaging device, the image processing device including a storage means for storing a statistical model generated by learning blur that occurs in a first image affected by aberration of an optical system of the imaging device and that changes nonlinearly depending on a distance to a subject in the first image, the method comprising: acquiring, from the imaging device, a second image captured by the imaging device, the second image being affected by aberrations of an optical system of the imaging device; acquiring, from the imaging device, focus position information relating to a focus position when the second image was captured; a step of inputting the acquired second image into the statistical model to obtain a blur value that indicates blur occurring in an object in the second image and is output from the statistical model; converting the acquired blur value into a distance to the subject based on the acquired focus position information; A method comprising:

12. A program executed by a computer of a distance measuring device including an imaging unit and a storage means for storing a statistical model generated by learning blur that occurs in a first image affected by aberration of an optical system of the imaging unit and that changes nonlinearly depending on the distance to a subject in the first image, The computer, acquiring a second image captured by the imaging unit, the second image being affected by aberration of the optical system of the imaging unit; acquiring focus position information regarding a focus position when the second image was captured; a step of inputting the acquired second image into the statistical model to obtain a blur value that indicates blur occurring in an object in the second image and is output from the statistical model; converting the acquired blur value into a distance to the subject based on the acquired focus position information; A program to execute.

13. A program executed by a computer of an image processing device connected to an imaging device, the image processing device including a storage means for storing a statistical model generated by learning blur that occurs in a first image affected by aberration of an optical system of the imaging device and that changes nonlinearly depending on a distance to a subject in the first image, the program comprising: The computer, acquiring, from the imaging device, a second image captured by the imaging device, the second image being affected by aberrations of an optical system of the imaging device; acquiring, from the imaging device, focus position information relating to a focus position when the second image was captured; a step of inputting the acquired second image into the statistical model to obtain a blur value that indicates blur occurring in an object in the second image and is output from the statistical model; converting the acquired blur value into a distance to the subject based on the acquired focus position information; A program to execute.

Images