Image processing apparatus, imaging apparatus, and calculation method

Abstract

To provide an image processing apparatus, an imaging apparatus, and a calculation method for improving the accuracy of measuring a distance while suppressing an arithmetic amount for a wire-angle image.SOLUTION: In an imaging apparatus 11, an image processing apparatus 10 has an acquisition unit 16 and a controller 17. The acquisition unit 16 acquires a reference image and a referred image. The controller 17 calculates a distance to a subject by PatchMatch using the reference image and the referred image, and performs projection transformation on a first pixel using distortion characteristics of an optical system used for imaging of the reference image on a hypothetical plane in the PatchMatch. The first pixel is a pixel in the reference image. The controller 17 further calculates a position of a second pixel by projecting the first pixel subjected to projection transformation on the referred image. The controller 17 calculates a difference degree between the first pixel and the second pixel.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, and a calculation method. [Background technology]

[0002] It is known that the distance to a subject is estimated by using a plurality of images captured from different viewpoints. Known methods for estimating the distance using images include the plane sweep method (see Patent Document 1) and the patch match method. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-208204 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the plane sweep method, the amount of calculations must be increased to improve measurement accuracy, so there is a trade-off between measurement accuracy and processing speed. The patch match method can improve measurement accuracy while reducing the amount of calculations compared to the plane sweep method. However, with the patch match method, it is difficult to measure images with large optical distortion with high accuracy. Therefore, it is difficult to improve measurement accuracy while reducing the amount of calculations for wide-angle images.

[0005] Therefore, an object of the present disclosure, made in consideration of the problems with the conventional techniques as described above, is to provide an image processing device, an imaging device, and a calculation method that improve the accuracy of distance measurement while reducing the amount of calculations required for wide-angle images. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an image processing apparatus according to a first aspect comprises: an acquisition unit that acquires a standard image captured from a standard position and a reference image captured from a position different from the standard position; a controller that calculates a distance to a subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image; The controller performs projective transformation of a first pixel in the reference image onto an assumed plane on which an object point is assumed to exist in the patch matching method using the distortion characteristics of an optical system used to capture the reference image, calculates the position of a second pixel in the reference image that corresponds to the first pixel by projecting the projectively transformed first pixel onto the reference image, and calculates the degree of dissimilarity between the first pixel and the second pixel.

[0007] Moreover, an imaging apparatus according to a second aspect comprises: a reference imaging unit located at a reference position and configured to capture a reference image; a reference image capturing unit that is located at a reference position different from the reference position and captures a reference image; a controller that calculates a distance to a subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image; The controller performs projective transformation of a first pixel in the reference image onto an assumed plane on which an object point is assumed to exist in the patch matching method using the distortion characteristics of an optical system used to capture the reference image, calculates the position of a second pixel in the reference image that corresponds to the first pixel by projecting the projectively transformed first pixel onto the reference image, and calculates the degree of dissimilarity between the first pixel and the second pixel.

[0008] In addition, the calculation method according to the third aspect is as follows: A standard image captured from a standard position and a reference image captured from a position different from the standard position are obtained; calculating a distance to a subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image; In the patch matching method, a first pixel in the reference image is projected onto an assumed plane on which an object point is assumed to exist using the distortion characteristics of the optical system used to capture the reference image, and the position of a second pixel in the reference image that corresponds to the first pixel is calculated by projecting the projected transformed first pixel onto the reference image, and the degree of dissimilarity between the first pixel and the second pixel is calculated. Effect of the Invention

[0009] According to the image processing device, imaging device, and calculation method configured as described above according to the present disclosure, the accuracy of measuring the distance to a wide-angle image is improved while reducing the amount of calculation. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a schematic configuration of an imaging device including an image processing device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining a three-dimensional coordinate system of a normal vector of a virtual plane. [Diagram 3] 4 is a flowchart for explaining a distance calculation process executed by the controller of FIG. 1. [Figure 4] FIG. 11 is a block diagram showing a schematic configuration of an imaging device including an image processing device according to a second embodiment. [Diagram 5] FIG. 11 is a diagram showing a first arrangement pattern of neighboring pixels for use in verifying the hypothetical plane according to the second embodiment with respect to a first pixel. [Figure 6] FIG. 11 is a diagram showing a second arrangement pattern of neighboring pixels for use in verifying the hypothetical plane in the second embodiment with respect to the first pixel. [Figure 7] FIG. 11 is a diagram showing a third arrangement pattern of neighboring pixels for use in verifying the hypothetical plane in the second embodiment with respect to a first pixel. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of an image processing device to which the present disclosure is applied will be described with reference to the drawings.

[0012] 1, an imaging device 11 including an image processing device 10 of the first embodiment may include a stereo camera 12 and the image processing device 10. The imaging device 11 may be mounted on, for example, a moving object.

[0013] The moving object may include, for example, a vehicle with an automatic driving function and an aircraft. The vehicle may include, for example, an automobile, an industrial vehicle, a railroad vehicle, a vehicle for daily life, and a fixed-wing aircraft that runs on a runway. The automobile may include, for example, a passenger car, a truck, a bus, a motorcycle, and a trolley bus. The industrial vehicle may include, for example, an industrial vehicle for agriculture and construction. The industrial vehicle may include, for example, a forklift and a golf cart. The industrial vehicle for agriculture may include, for example, a tractor, a cultivator, a transplanter, a binder, a combine, and a lawnmower. The industrial vehicle for construction may include, for example, a bulldozer, a scraper, a shovel, a crane, a dump truck, and a road roller. The vehicle may include one that runs by human power. The classification of the vehicle is not limited to the above-mentioned examples. For example, the automobile may include an industrial vehicle that can run on a road. The same vehicle may be included in multiple classifications. The aircraft may include, for example, a fixed-wing aircraft and a rotary-wing aircraft.

[0014] The image processing device 10 may calculate the position of a subject corresponding to the subject image included in the stereo image in a three-dimensional coordinate system based on the imaging device 11, based on the captured stereo image. The imaging device 11 notifies the external device 13 of the position in the three-dimensional coordinate system. In a configuration in which the imaging device 11 is mounted on a moving body, the external device 13 is, for example, a driving support device such as an auto cruise control, a driving support device such as an automatic braking device, a notification device that notifies the distance to the subject, and a warning device that issues a warning.

[0015] The stereo camera 12 includes a plurality of imaging units having parallax with respect to each other. The stereo camera 12 can image an object from a plurality of directions by making the plurality of imaging units cooperate with each other. The stereo camera 12 may include a plurality of imaging units in a single housing. The stereo camera 12 may include two or more imaging units that are independent of each other and located apart from each other. The position and orientation of each of the plurality of imaging units in a three-dimensional coordinate system defined with respect to the imaging device 11 are known. The position of the imaging unit is, for example, the position of the intersection between the light receiving surface of the imaging unit and the optical axis of the optical system. The orientation of the imaging unit is, for example, the direction of the optical axis of the optical system of the imaging unit.

[0016] The stereo camera 12 includes, for example, a reference imaging section 14 and a reference imaging section 15. The reference imaging section 14 is located at a reference position in the imaging device 11. The reference imaging section 15 is located at a reference position. The reference position is different from the reference position. The reference imaging section 14 at the reference position and the reference imaging section 15 at the reference position may be located such that their optical axes face in directions capable of imaging the same subject.

[0017] The optical axes of the optical systems of the standard imaging section 14 and the reference imaging section 15 may be parallel to each other. The optical axes being parallel is not limited to being strictly parallel, and may allow for misalignment during assembly, misalignment during installation, and misalignment over time. The optical axes of the standard imaging section 14 and the reference imaging section 15 are not limited to being parallel, and may point in different directions.

[0018] The reference imaging section 14 and the reference imaging section 15 may have optical systems with the same optical characteristics. The reference position and the reference position may be the same in a direction parallel to the optical axis of the optical system. Furthermore, the reference imaging section 14 and the reference imaging section 15 may have imaging elements with the same characteristics.

[0019] The reference imaging unit 14 and the reference imaging unit 15 each include an optical system that defines an optical axis and an imaging element. The optical system includes a lens or a mirror, and forms an image of a subject on a light receiving surface of the imaging element. The imaging element is, for example, a charge-coupled device image sensor (CCD image sensor) or a complementary MOS image sensor (CMOS image sensor). The reference imaging unit 14 captures an image from a reference position and generates a reference image as a signal. The reference imaging unit 15 captures an image from a reference position and generates a reference image as a signal. Specifically, optical images formed on the imaging elements of the reference imaging unit 14 and the reference imaging unit 15 are generated as the reference image and the reference image. The reference imaging unit 14 and the reference imaging unit 15 may perform any image processing, such as brightness adjustment, contrast adjustment, gamma correction, etc., on the captured image, except for distortion correction of the optical system.

[0020] The image processing device 10 includes an acquisition unit 16 and a controller 17. The image processing device 10 may further include an assignment unit 18 and a memory 19.

[0021] The acquisition unit 16 is an input interface of the image processing device 10 that acquires information from the stereo camera 12 and an external device. For example, a physical connector and a wireless communication device are adopted for the acquisition unit 16. The physical connector includes an electric connector corresponding to transmission by an electric signal, an optical connector corresponding to transmission by an optical signal, and an electromagnetic connector corresponding to transmission by an electromagnetic wave. The electric connector includes a connector conforming to IEC60603, a connector conforming to the USB standard, a connector corresponding to an RCA terminal, a connector corresponding to an S terminal defined in EIAJ CP-1211A, a connector corresponding to a D terminal defined in EIAJ RC-5237, a connector conforming to the HDMI (registered trademark) standard, and a connector corresponding to a coaxial cable including BNC. The optical connector includes various connectors conforming to IEC 61754. The wireless communication device includes a wireless communication device conforming to each standard including Bluetooth (registered trademark) and IEEE802.11. The wireless communication device includes at least one antenna.

[0022] The acquisition unit 16 acquires, as signals, the standard image and the reference image generated by the stereo camera 12 through imaging. The acquisition unit 16 may provide the acquired standard image and reference image to the controller 17.

[0023] The assigning unit 18 is an output interface of the image processing device 10 that can assign the position in the three-dimensional coordinate system calculated by the image processing device 10 as a signal to the external device 13. Like the acquiring unit 16, the assigning unit 18 includes various interfaces compatible with wired and wireless communication.

[0024] The memory 19 includes any storage device, such as a random access memory (RAM) and a read only memory (ROM), etc. The memory 19 stores various programs that cause the controller 17 to function and various information that the controller 17 uses.

[0025] The controller 17 includes one or more processors and a memory. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The controller 17 may be either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together.

[0026] The controller 17 calculates the distance to the subject whose image is included in the reference image and the reference image by a patch match method using the standard image and the reference image. Specifically, the controller 17 calculates the distance to the subject by performing an initial assumption of an assumed plane, verification of the assumed plane, spatial propagation, and random selection. In verifying the assumed plane, the controller 17 performs a projective transformation from the standard image to the reference image, unlike the known patch match method. Hereinafter, the patch match method executed by the controller 17 will be described in detail in this disclosure.

[0027] First, the controller 17 assumes an initial assumed plane for the reference image as described below. Specifically, the controller 17 assumes an initial assumed plane for each of a plurality of pixels constituting the reference image. The assumed plane is a so-called plane hypothesis, and is a very small plane on which it is assumed that an object point corresponding to the pixel exists in real space.

[0028] The virtual plane may be assumed by provisionally setting a normal vector n and a depth d. The normal vector n is a so-called normal hypothesis, and may be expressed in a three-dimensional coordinate system with the optical axis of the optical system of the reference imaging unit 14 and two straight lines perpendicular to the optical axis and perpendicular to each other as axes. As shown in FIG. 2, the normal vector n is a unit vector determined by a first rotation angle φ about the optical axis from an axis other than the axis that coincides with the optical axis ox in the three-dimensional coordinate system, and a second rotation angle θ from the optical axis ox. Specifically, the normal vector n is [sinθcosΦ sinθsinΦ cosθ]. The depth d is a so-called depth hypothesis, and is the depth in the direction of the optical axis ox of the reference imaging unit 14. Specifically, the depth d may be the position of an object point from the optical center along the optical axis ox. The controller 17 may generate the normal vector n and the depth d by random numbers. The controller 17 may limit the range of the depth before assuming the initial assumed plane. The range of the limiting depth may be determined in advance.

[0029] Controller 17 then calculates the dissimilarity of the first pixel and the second pixel in verifying the hypothesized hypothetical plane, as described below.

[0030] In verifying the assumed plane, the controller 17 projects each of the pixels constituting the reference image onto the assumed plane assumed for each pixel. For example, by calculating the following formula (1), an arbitrary first pixel constituting the reference image is projected onto the assumed plane. c is projected onto the hypothetical plane. η c -1 (K c -1 u c ) (1)

[0031] In equation (1), η c -1 is a transformation equation for projectively transforming an arbitrary pixel (hereinafter also referred to as a first pixel) in the reference image using the distortion characteristics of the optical system of the reference imaging unit 14 used to capture the reference image. c -1 Specifically, K is a transformation equation that transforms the direction of the chief ray passing through the first pixel in the camera coordinate system of the reference imaging unit 14 into a direction bent on the object side relative to the optical system. c -1 is a conversion formula for converting the image coordinates of the reference image into the camera coordinates of the reference imaging unit 14. c is the position in image coordinates of the reference image.

[0032] In verifying the hypothetical plane, the controller 17 further converts the position of the first pixel projected onto the hypothetical plane from the camera coordinates of the base imaging unit 14 to the camera coordinates of the reference imaging unit 15 by calculating the following equation (2). H c -1 (K c -1 u c ) (2)

[0033] In equation (2), the conversion equation H satisfies the following equation (3). H=R+tn T / d (3)

[0034] In equation (3), R is a rotational transformation formula from the standard imaging unit 14 to the reference imaging unit 15, and t is a translational transformation formula from the standard imaging unit 14 to the reference imaging unit 15.

[0035] In the verification of the hypothetical plane, the controller 17 further calculates the position, in the image coordinates of the reference image, of a second pixel obtained by projecting onto the reference image the first pixel that has been projected onto the hypothetical plane by calculating the following formula (4). The second pixel is a pixel that corresponds to the first pixel in the reference image. K r η r (H c -1 (K c -1 u c )) (4)

[0036] In equation (4), η r is a transformation equation for projectively transforming an object point onto a reference image using the distortion characteristics of the optical system of the reference image pickup unit 15 used to capture the reference image. r Specifically, K is a transformation equation that transforms the direction of the principal ray passing through an object point in the camera coordinate system of the reference imaging unit 15 into a direction bent on the image side of the optical system. r is a transformation formula for transforming the camera coordinates of the reference imaging unit 15 into the image coordinates of the reference image.

[0037] In the verification of the hypothetical plane, the controller 17 further calculates the dissimilarity between the first pixel and the second pixel as a provisionally recognized dissimilarity. The dissimilarity is, for example, a difference between the pixel values ​​of the first pixel and the second pixel. The pixel value may be a luminance value. Alternatively, the dissimilarity is a statistical value of the differences of each color component of the first pixel and the second pixel. The statistical value is, for example, an arithmetic mean, a geometric mean, a median, a mode, etc. Furthermore, the controller 17 may calculate the dissimilarity for the standard image and the reference image from which feature extraction has been performed. The feature extraction is, for example, edge detection and addition of the detected edge components.

[0038] Controller 17 may then re-assume the assumed plane by performing spatial propagation, as described below.

[0039] In the spatial propagation, the controller 17 may determine whether a pixel having a dissimilarity less than the dissimilarity of an arbitrary pixel constituting the reference image is located within a region surrounding the arbitrary pixel. The region surrounding the arbitrary pixel may be arbitrarily determined. The region surrounding the arbitrary pixel is a region including a pixel of interest. The region surrounding the arbitrary pixel is, for example, a region including arbitrary pixels such as 2×2, 3×3, 4×4, etc.

[0040] In the spatial propagation, if a pixel having a dissimilarity less than the dissimilarity of the given pixel is located in the region surrounding the given pixel, the controller 17 may select the pixel having the smallest dissimilarity less than the dissimilarity of the given pixel. The controller 17 may replace the assumed plane assumed for the given pixel with the assumed plane assumed for the selected pixel to verify the assumed plane.

[0041] Specifically, the controller 17 may project an arbitrary pixel of the standard image onto the replaced assumed plane. Furthermore, the controller 17 may perform coordinate conversion of the position of the arbitrary pixel projected onto the replaced assumed plane from the camera coordinate of the standard imaging unit 14 to the camera coordinate of the reference imaging unit 15. Furthermore, the controller 17 may calculate the position of a pixel in the reference image projected onto the arbitrary pixel projected onto the replaced assumed plane. Furthermore, the controller 17 may calculate a degree of difference for comparison between the arbitrary pixel and the projected pixel.

[0042] In the spatial propagation, the controller 17 may further compare the dissimilarity of the provisional recognition at the arbitrary pixel with the calculated dissimilarity for comparison. If the dissimilarity for comparison is less than the dissimilarity of the provisional recognition, the controller 17 may re-assume the replaced assumed plane as the assumed plane for the arbitrary pixel. The controller 17 may recognize the dissimilarity corresponding to the re-assumed assumed plane, in other words, the dissimilarity for comparison, as the dissimilarity of the provisional recognition. The controller 17 may re-assume the assumed plane for each pixel constituting the reference image.

[0043] Controller 17 then re-assumes the assumption plane by performing a random selection, as described below.

[0044] In the random selection, the controller 17 may perturb the hypothetical plane re-assumed for any pixel constituting the reference image by random number selection. Perturbation of the hypothetical plane means giving a minute amount of deviation from the normal vector n and depth d of the hypothetical plane re-assumed by a random number. For example, the normal vector n is changed by a uniform random number in a minute range for each of the first rotation angle Φ and the second rotation angle θ. Also, the depth d is changed by a uniform random number in a minute range. The minute range may be determined by a ratio of, for example, 1 / 100 to the current value. The controller 17 may perform the above-mentioned verification of the hypothetical plane for the perturbed hypothetical plane.

[0045] Specifically, the controller 17 may project an arbitrary pixel of the base image onto the perturbed hypothetical plane. Furthermore, the controller 17 may perform coordinate transformation of the position of the arbitrary pixel projected onto the hypothetical plane from the camera coordinate of the base imaging unit 14 to the camera coordinate of the reference imaging unit 15. Furthermore, the controller 17 may calculate the position of a pixel in the reference image projected onto the reference image of the arbitrary pixel projected onto the hypothetical plane. Furthermore, the controller 17 may calculate a degree of difference for comparison between the arbitrary pixel and the projected pixel.

[0046] In the random selection, the controller 17 may further compare the dissimilarity of the provisional recognition at the arbitrary pixel with the calculated dissimilarity for comparison. If the dissimilarity for comparison is less than the dissimilarity for provisional recognition, the controller 17 may re-assume the perturbed assumed plane as the assumed plane for the arbitrary pixel. The controller 17 may recognize the dissimilarity corresponding to the re-assumed assumed plane, in other words, the dissimilarity for comparison, as the dissimilarity for provisional recognition. The controller 17 may re-assume the assumed plane for each pixel constituting the reference image.

[0047] The controller 17 may execute a repetitive process including spatial propagation and random selection so that the degree of dissimilarity is reduced. For example, the controller 17 may repeat the execution of a sequence of spatial propagation and random selection so that the degree of dissimilarity is reduced. Also, for example, the controller 17 may repeat the execution of a sequence of spatial propagation once and random selection combinations repeated a plurality of times so that the degree of dissimilarity is reduced so that the degree of dissimilarity is reduced.

[0048] The controller 17 may repeat the process until the degree of dissimilarity is equal to or less than a discrimination threshold, which is the difference at which the images represented by the first pixel and the second pixel are deemed to be identical. An upper limit may be set for the number of times the process is repeated.

[0049] The controller 17 may determine that the hypothetical plane of an arbitrary pixel for which the repeated processing has been completed is the real plane on which the object point corresponding to the arbitrary pixel is located. The controller 17 may calculate the position of the object point corresponding to the arbitrary pixel based on the real plane determined for the arbitrary pixel in the reference image and the coordinate position of the arbitrary pixel in the reference image. The controller 17 may calculate the distance to the subject based on the positions of the object points corresponding to each pixel constituting the reference image.

[0050] Next, a distance calculation process executed by the controller 17 in this embodiment will be described with reference to the flowchart of Fig. 3. The distance calculation process starts every time one frame of a standard image and a reference image are acquired from the stereo camera 12.

[0051] In step S100, the controller 17 makes an assumption of an initial assumption plane for each pixel that constitutes the reference image. After the assumption, the process proceeds to step S101.

[0052] In step S101, the controller 17 performs verification of the hypothetical plane for each pixel constituting the reference image. After verification, the process proceeds to step S102.

[0053] In step S102, the controller 17 performs spatial propagation using the dissimilarity provisionally determined by the verification in step S101 for pixels for which spatial propagation and random selection have not been performed or for pixels for which the dissimilarity of the provisional determination is equal to or less than the discrimination threshold value. After performing spatial propagation, the process proceeds to step S103.

[0054] In step S103, the controller 17 performs random selection for each pixel constituting the reference image based on the hypothetical plane re-hypothesized by the spatial propagation in step S102. After performing random selection, the process proceeds to step S104.

[0055] In step S104, the controller 17 determines whether or not there is a pixel whose dissimilarity after random selection in step S103 is equal to or less than the discrimination threshold. If there is a pixel whose dissimilarity is not equal to or less than the discrimination threshold, the process proceeds to step S105. If the dissimilarity of all pixels is equal to or less than the discrimination threshold, the process proceeds to step S106.

[0056] In step S105, the controller 17 determines whether the number of iterations of the spatial propagation in step S102 and the random selection in step S103 is less than the upper limit. If it is less than the upper limit, the process returns to step S102. If it is not less than the upper limit, in other words, if the upper limit is reached, the process proceeds to step S106.

[0057] In step S106, the controller 17 recognizes the hypothetical plane after random selection in step S103 for each pixel constituting the reference image as an actual plane. After the recognition, the process proceeds to step S107.

[0058] In step S107, the controller 17 calculates the positions of object points corresponding to each pixel in the reference image, based on the positions of each pixel in the reference image and the real plane identified in step S106. After the calculation, the process proceeds to step S108.

[0059] In step S108, the controller 17 calculates the distance to each object point based on the position of each object point calculated in step S107. After the calculation, the distance calculation process ends.

[0060] The image processing device 10 of the first embodiment configured as above includes a controller 17 that uses a standard image and a reference image to calculate the distance to a subject whose image is included in the standard image and the reference image by a patch match method, and the controller 17 performs projective transformation of a first pixel in the standard image on an assumed plane on which an object point is assumed to exist in the patch match method using the distortion characteristics of the optical system used to capture the standard image, calculates the position of a second pixel corresponding to the first pixel in the reference image by projecting the projectively transformed first pixel onto the reference image, and calculates the dissimilarity between the first pixel and the second pixel. With this configuration, the image processing device 10 does not use the standard image and the reference image from which distortion has been removed in the patch match method, but calculates the dissimilarity by comparing the pixels of the reference image that have been projectively transformed from the pixels of the standard image that include distortion, and therefore can calculate the distance to the subject with high accuracy. Therefore, the image processing device 10 improves the measurement accuracy of the distance while suppressing the amount of calculation for a wide-angle image.

[0061] In the image processing device 10 of the first embodiment, the hypothetical plane is defined by a unit vector determined by a first rotation angle Φ around the optical axis ox and a second rotation angle θ from the optical axis ox of the imaging unit 14 that captures the reference image from a straight line perpendicular to the optical axis ox, and by a depth in the optical axis direction. With this configuration, the image processing device 10 can calculate the direction of a normal that satisfies the length constraint. With this configuration, the image processing device 10 can easily determine whether the plane is one that can be captured within the range of the angle of view of the reference imaging unit 14.

[0062] Moreover, in the image processing device 10 of the first embodiment, the patch matching method is repeated until the dissimilarity becomes equal to or less than the discrimination threshold. With such a configuration, the image processing device 10 can improve the accuracy of calculating the positions of object points corresponding to the pixels constituting the reference image.

[0063] In the image processing device 10 of the first embodiment, an upper limit is set for the number of times the iterative process in the patch matching method is repeated. With this configuration, the image processing device 10 can interrupt the re-assuming and calculate the distance of the entire subject moving to the reference image even if there is a first pixel at which the decrease in dissimilarity stops.

[0064] Next, a second embodiment of the present disclosure will be described. In the second embodiment, the verification method of the hypothetical plane is different from that of the first embodiment. The second embodiment will be described below, focusing on the differences from the first embodiment. Note that the same reference numerals are used to designate parts having the same configuration as the first embodiment.

[0065] 4, in the second embodiment, an imaging device 110 including an image processing device 100 may include a stereo camera 12 and an image processing device 100, similar to the first embodiment. In the second embodiment, the structure and function of the stereo camera 12 are the same as those in the first embodiment.

[0066] In the second embodiment, the image processing device 100 may be configured to include an acquisition unit 16 and a controller 170, similar to the first embodiment. The image processing device 100 may further be configured to include an assignment unit 18 and a memory 19. In the second embodiment, the structures and functions of the acquisition unit 16, the assignment unit 18, and the memory 19 are the same as those in the first embodiment. In the second embodiment, the structure of the controller 170 is the same as that in the first embodiment. In the second embodiment, the function of the controller 170, in other words, the processing executed, is different from that in the first embodiment.

[0067] The controller 170 calculates the distance to the subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image, similar to the first embodiment. Specifically, the controller 170 calculates the distance to the subject by performing assumption of an initial assumed plane, verification of the assumed plane, spatial propagation, and random selection, similar to the first embodiment. The controller 170 assumes the initial assumed plane in the same manner as the first embodiment.

[0068] In verifying the assumed plane, the controller 170 calculates the degree of difference between the first pixel and the second pixel, similar to the first embodiment. The controller 170 projects an arbitrary first pixel constituting the reference image onto the assumed plane assumed for the first pixel, as in the first embodiment. Furthermore, the controller 170 performs coordinate conversion of the position of the first pixel projected onto the assumed plane from the camera coordinate of the reference imaging unit 14 to the camera coordinate of the reference imaging unit 15, as in the first embodiment. Furthermore, the controller 170 calculates the position of the second pixel in the reference image projected onto the reference image of the first pixel projected onto the assumed plane, as in the first embodiment. The controller 170 selects at least some of the pixels constituting the reference image, for example, all pixels, as first pixels, and calculates the position of the second pixel corresponding to the first pixel, as in the first embodiment.

[0069] In verifying the hypothetical plane, unlike the first embodiment, the controller 170 uses neighboring pixels of the first pixel to calculate the dissimilarity between the first pixel and the second pixel. Specifically, the neighboring pixels may be located in a region that includes the first pixel in the reference image.

[0070] More specifically, the neighboring pixels may be selected based on a predetermined arrangement pattern for the first pixel. For example, as shown in Fig. 5, all pixels other than the first pixel p1 included in a 5 x 5 region centered on the first pixel p1 may be selected as neighboring pixels pn for the first pixel p1. Alternatively, for example, as shown in Fig. 6, in a 5 x 5 region centered on the first pixel p1, four pixels diagonally above and below the first pixel p1 within a 3 x 3 range centered on the first pixel p1, and four pixels above, below, left and right of the first pixel p1 within a range excluding the 3 x 3 may be selected as neighboring pixels pn.

[0071] Or, more specifically, the neighboring pixels pn may be selected randomly with respect to the first pixel p1. For example, as shown in Fig. 7, pixels located at random positions among pixels included in a 5 x 5 region centered on the first pixel p1 may be selected as the neighboring pixels pn.

[0072] In verifying the hypothetical plane, the controller 170 may calculate the dissimilarity between a pixel set of a first pixel p1 and its neighboring pixels pn and a pixel set of a second pixel and a pixel corresponding to the second pixel pn in the reference image as the dissimilarity between the first pixel p1 and the second pixel. The dissimilarity of the pixel set may be, for example, a statistical value of the combined dissimilarity including two corresponding pixels. The statistical value may be, for example, an arithmetic mean, a geometric mean, a median, a mode, or the like.

[0073] Next, the controller 170 re-assumes the assumption plane by performing spatial propagation, similar to the first embodiment.

[0074] In the spatial propagation, the controller 170 may determine whether or not a pixel having a dissimilarity less than the dissimilarity of an arbitrary pixel constituting the reference image is located within the region surrounding the arbitrary pixel, as in the first embodiment. When a pixel having a dissimilarity less than the dissimilarity of the arbitrary pixel is located within the region surrounding the arbitrary pixel, the controller 170 may select a pixel having a minimum dissimilarity less than the dissimilarity of the arbitrary pixel, as in the first embodiment.

[0075] Furthermore, the controller 170 may project an arbitrary pixel and a neighboring pixel pn of the reference image onto an assumed plane assumed for the selected pixel, similar to the first embodiment. The neighboring pixel pn may be the same as the neighboring pixel pn determined in the verification of the assumed plane. Furthermore, the controller 170 may perform coordinate conversion from the camera coordinates of the standard imaging unit 14 to the camera coordinates of the reference imaging unit 15 for the positions of the arbitrary pixel and the neighboring pixel pn projected onto the assumed plane, similar to the first embodiment. Furthermore, the controller 170 may calculate the positions of the pixels in the reference image projected onto the reference image of the arbitrary pixel and the neighboring pixel pn projected onto the assumed plane. Furthermore, the controller 170 may calculate a first degree of difference for comparison between the arbitrary pixel and the projected pixel, and a second degree of difference for comparison between the neighboring pixel pn and the projected pixel, similar to the first embodiment. Furthermore, unlike the first embodiment, the controller 170 may calculate a statistical value of the first and second comparative dissimilarity degrees as the comparative dissimilarity degrees. The statistical value may be, for example, an arithmetic mean, a geometric mean, a median, a mode, or the like.

[0076] In the spatial propagation, the controller 170 may further compare the dissimilarity of the provisional determination at the arbitrary pixel with the calculated dissimilarity for comparison, similar to the first embodiment. As in the first embodiment, when the dissimilarity for comparison is less than the dissimilarity of the provisional determination, the controller 170 may re-assume the assumed plane assumed for the selected pixel as the assumed plane for the arbitrary pixel. As in the first embodiment, the controller 170 may recognize the dissimilarity corresponding to the re-assumed assumed plane, in other words, the dissimilarity for comparison, as the dissimilarity of the provisional determination. As in the first embodiment, the controller 170 may re-assume the assumed plane for each pixel constituting the reference image.

[0077] Next, the controller 170 re-assumes the assumption plane by performing a random selection, similar to the first embodiment.

[0078] In the random selection, the controller 170 may perturb the hypothetical plane re-hypothesized for any pixel constituting the reference image by random number selection. Furthermore, the controller 170 may project the arbitrary pixel and neighboring pixels pn onto the hypothetical plane perturbed for the arbitrary pixel, similar to the first embodiment. The neighboring pixels pn may be the same as the neighboring pixels pn determined in the verification of the hypothetical plane. Furthermore, the controller 170 may convert the coordinates of the positions of the arbitrary pixel and neighboring pixels pn projected onto the hypothetical plane from the camera coordinates of the reference imaging unit 14 to the camera coordinates of the reference imaging unit 15, similar to the first embodiment. Furthermore, the controller 170 may calculate the position of the pixel in the reference image projected onto the reference image of the arbitrary pixel and neighboring pixels pn projected onto the hypothetical plane, similar to the first embodiment. Furthermore, the controller 170 may calculate a first difference for comparison between the arbitrary pixel and the projected pixel, and a second difference for comparison between the neighboring pixel pn and the projected pixel, similar to the first embodiment. Furthermore, unlike the first embodiment, the controller 170 may calculate a statistical value of the first difference for comparison and the second difference for comparison as the difference for comparison. The statistical value is, for example, an arithmetic mean, a geometric mean, a median, a mode, or the like.

[0079] In the random selection, the controller 17 may further compare the dissimilarity of the provisional recognition at the arbitrary pixel with the calculated dissimilarity for comparison, similar to the first embodiment. As in the first embodiment, the controller 170 may re-assume the perturbed assumed plane as the assumed plane for the arbitrary pixel when the dissimilarity for comparison is less than the dissimilarity of the provisional recognition. As in the first embodiment, the controller 170 may recognize the dissimilarity corresponding to the re-assumed assumed plane, in other words, the dissimilarity for comparison, as the dissimilarity of the provisional recognition. As in the first embodiment, the controller 170 may re-assume the assumed plane for each pixel constituting the reference image.

[0080] Similar to the first embodiment, the controller 170 may perform an iterative process including spatial propagation and randomization to reduce the dissimilarity.

[0081] The controller 170 may determine the hypothetical plane of an arbitrary pixel where the iterative process has been completed as a real plane on which an object point corresponding to the arbitrary pixel is located, similar to the first embodiment. The controller 170 may calculate the position of the object point corresponding to the arbitrary pixel based on the real plane determined for the arbitrary pixel in the reference image and the coordinate position of the arbitrary pixel in the reference image, similar to the first embodiment. The controller 170 may calculate the distance to the subject based on the positions of the object points corresponding to each pixel constituting the reference image, similar to the first embodiment.

[0082] The image processing device 100 of the second embodiment configured as above also includes a controller 170 that uses a standard image and a reference image to calculate the distance to a subject whose image is included in the standard image and the reference image by the patch match method, and the controller 170 performs projective transformation of a first pixel p1 in the standard image on an assumed plane on which an object point is assumed to exist in the patch match method using the distortion characteristics of the optical system used to capture the standard image, calculates the position of a second pixel corresponding to the first pixel p1 in the reference image by projecting the projectively transformed first pixel p1 onto the reference image, and calculates the degree of difference between the first pixel p1 and the second pixel. With this configuration, the image processing device 100 also improves the accuracy of distance measurement while suppressing the amount of calculation for wide-angle images, similar to the first embodiment.

[0083] Also, in the image processing device 100 of the second embodiment, the hypothetical plane is defined by a unit vector determined by a first rotation angle Φ around the optical axis ox and a second rotation angle θ from the optical axis ox of the optical system of the imaging unit 14 that captures the reference image from a straight line perpendicular to the optical axis ox, and by the depth in the optical axis direction. With this configuration, the image processing device 100 can also calculate the direction of a normal that satisfies the length constraint. With this configuration, the image processing device 10 can easily determine whether the plane is one that can be captured within the range of the angle of view of the reference imaging unit 14.

[0084] Also, in the image processing device 100 of the second embodiment, the controller 170 repeats the process in the patch matching method until the dissimilarity becomes equal to or less than the discrimination threshold. With this configuration, the image processing device 100 can also improve the calculation accuracy of the positions of object points corresponding to the pixels constituting the reference image, similar to the first embodiment.

[0085] Also, in the image processing device 100 of the second embodiment, an upper limit is set for the number of iterations of the iterative process in the patch matching method. With this configuration, the image processing device 100 can also interrupt the re-assuming and calculate the distance of the entire subject moving to the reference image, similar to the first embodiment, even if there is a first pixel p1 where the decrease in dissimilarity stops.

[0086] In the image processing device 100 of the second embodiment, the controller 170 uses the neighboring pixels pn of the first pixel p1 to calculate the degree of difference between the first pixel p1 and the second pixel. With this configuration, the image processing device 100 evaluates the verification of the hypothetical plane including the neighboring pixels pn, and therefore more appropriately evaluates the actual difference between the pixels in the standard image and the pixels in the reference image. Therefore, the image processing device 100 can further improve the accuracy of distance measurement.

[0087] In one embodiment, (1) an image processing device includes: an acquisition unit that acquires a standard image captured from a standard position and a reference image captured from a position different from the standard position; a controller that calculates a distance to a subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image; The controller performs projective transformation of a first pixel in the reference image onto an assumed plane on which an object point is assumed to exist in the patch matching method using the distortion characteristics of an optical system used to capture the reference image, calculates the position of a second pixel in the reference image that corresponds to the first pixel by projecting the projectively transformed first pixel onto the reference image, and calculates the degree of dissimilarity between the first pixel and the second pixel.

[0088] (2) In the image processing device of (1) above, The hypothetical plane is defined by a unit vector determined by a first rotation angle along the optical axis from a straight line perpendicular to the optical axis of the optical system of the imaging unit that captures the reference image and a second rotation angle from the optical axis, as well as a depth in the optical axis direction.

[0089] (3) In the image processing device of (1) or (2) above, The controller repeats the process of the patch matching method until the degree of dissimilarity becomes equal to or less than a discrimination threshold.

[0090] (4) In any one of the image processing devices according to (1) to (3), The number of repetitions of the patch matching method has an upper limit.

[0091] (5) In any one of the image processing devices according to (1) to (4), The controller uses neighboring pixels of the first pixel to calculate the dissimilarity between the first pixel and the second pixel.

[0092] (6) In the image processing device of (5) above, The neighboring pixels are selected based on a predetermined arrangement pattern relative to the first pixel.

[0093] (7) In the image processing device of (5) above, The neighboring pixels are randomly selected relative to the first pixel.

[0094] In one embodiment, (8) the imaging device comprises: a reference imaging unit located at a reference position and configured to capture a reference image; a reference image capturing unit that is located at a reference position different from the reference position and captures a reference image; a controller that calculates a distance to a subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image; The controller performs projective transformation of a first pixel in the reference image onto an assumed plane on which an object point is assumed to exist in the patch matching method using the distortion characteristics of an optical system used to capture the reference image, calculates the position of a second pixel in the reference image that corresponds to the first pixel by projecting the projectively transformed first pixel onto the reference image, and calculates the degree of dissimilarity between the first pixel and the second pixel.

[0095] In one embodiment, (9) the calculation method comprises: A standard image captured from a standard position and a reference image captured from a position different from the standard position are obtained; calculating a distance to a subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image; In the patch matching method, a first pixel in the reference image is projected onto an assumed plane on which an object point is assumed to exist using the distortion characteristics of the optical system used to capture the reference image, and the position of a second pixel in the reference image that corresponds to the first pixel is calculated by projecting the projected transformed first pixel onto the reference image, and the degree of dissimilarity between the first pixel and the second pixel is calculated.

[0096] The above describes an embodiment of the image processing device 10. However, the embodiment of the present disclosure can also be implemented as a method or program for implementing the device, or as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0097] The form of implementation of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may be a program module incorporated into an operating system, etc. Furthermore, the program may or may not be configured so that all processing is performed only by the CPU on the control board. The program may be configured so that a part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as necessary.

[0098] The drawings for explaining the embodiments according to the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0099] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions, etc. included in each component, etc. can be rearranged so as not to cause logical inconsistencies, and multiple components, etc. can be combined into one or divided.

[0100] All of the features described in this disclosure and / or all steps of any method or process disclosed herein may be combined in any combination except those combinations in which the features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is merely one example of a generic series of the same or equivalent features.

[0101] Furthermore, the embodiments of the present disclosure are not limited to any of the specific configurations of the embodiments described above, but rather extend to any novel feature or combination of features described herein, or any novel method or process step or combination of features described herein.

[0102] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In the configurations distinguished by descriptions such as "first" and "second" in this disclosure, the numbers in the configurations can be exchanged. For example, the first pixel can exchange the identifiers "first" and "second" with the second pixel. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the identifier exchange. The identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a code. The descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number. [Explanation of symbols]

[0103] 10, 100 Image processing device 11 Imaging device 12 Stereo Camera 13 External device 14 Reference imaging section 15 Reference imaging section 16 Acquisition Department 17, 170 Controller 18 Granting Department 19. Memory ox Optical axis of the optical system of the reference imaging unit p1 First pixel pn Neighboring pixels

Claims

1. an acquisition unit that acquires a standard image captured from a standard position and a reference image captured from a position different from the standard position; a controller that calculates a distance to a subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image; The controller performs projective transformation of a first pixel in the reference image onto an assumed plane on which an object point is assumed to exist in the patch matching method, using a distortion characteristic of an optical system used to capture the reference image, projects the projectively transformed first pixel onto the reference image, calculates a position of a second pixel in the reference image that corresponds to the first pixel, and calculates a degree of dissimilarity between the first pixel and the second pixel. Image processing device.

2. 2. The image processing device according to claim 1, The hypothetical plane is defined by a unit vector determined by a first rotation angle around the optical axis from a straight line perpendicular to the optical axis of the optical system of the imaging unit that captures the reference image and a second rotation angle from the optical axis, and by a depth in the optical axis direction. Image processing device.

3. 2. The image processing device according to claim 1, The controller repeats the process in the patch matching method until the degree of difference becomes equal to or less than a discrimination threshold. Image processing device.

4. 4. The image processing device according to claim 1, The number of times the patch matching process can be repeated is limited by an upper limit. Image processing device.

5. 4. The image processing device according to claim 1, The controller uses neighboring pixels of the first pixel to calculate the dissimilarity between the first pixel and the second pixel. Image processing device.

6. 6. The image processing device according to claim 5, The neighboring pixels are selected based on a predetermined arrangement pattern with respect to the first pixel. Image processing device.

7. 6. The image processing device according to claim 5, The neighboring pixels are randomly selected relative to the first pixel. Image processing device.

8. a reference imaging unit located at a reference position and configured to capture a reference image; a reference image capturing unit that is located at a reference position different from the reference position and captures a reference image; a controller that calculates a distance to a subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image; The controller performs projective transformation of a first pixel in the reference image onto an assumed plane on which an object point is assumed to exist in the patch matching method, using a distortion characteristic of an optical system used to capture the reference image, projects the projectively transformed first pixel onto the reference image, calculates a position of a second pixel in the reference image that corresponds to the first pixel, and calculates a degree of dissimilarity between the first pixel and the second pixel. Imaging device.

9. A standard image captured from a standard position and a reference image captured from a position different from the standard position are obtained; calculating a distance to a subject whose image is included in the standard image and the reference image by a patch matching method using the standard image and the reference image; A first pixel in the reference image is projected onto an assumed plane on which an object point is assumed to exist in the patch matching method using a distortion characteristic of an optical system used to capture the reference image, and the projected first pixel is projected onto the reference image to calculate a position of a second pixel in the reference image that corresponds to the first pixel, and a degree of dissimilarity between the first pixel and the second pixel is calculated. Calculation method.

Images