Image processing apparatus, imaging apparatus, and detection method

Abstract

To improve the accuracy of measuring a ground surface in various modes for a super-wide angle image.SOLUTION: An image processing apparatus 10 has an acquisition unit 16 and a controller 17. The acquisition unit 16 acquires a reference image and a referred image. The controller 17 detects a ground surface and a floor surface on the basis of observation parallax and ground surface parallax to be calculated by matching based on a difference degree between a first pixel and a second pixel. The first pixel is an arbitrary pixel in the reference image. The second pixel is a pixel in the referred image. The controller 17 performs projection transformation on the first pixel. The controller 17 calculates a position of the second pixel by projection. The controller 17 calculates a position of an object point corresponding to the first pixel based on the difference degree between the first pixel and the second pixel. The controller 17 calculates observation parallax based on the positions of the first pixel and the second pixel corresponding to the object point.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, and a detection method. [Background technology]

[0002] There is a demand for detecting road surfaces in images captured by imaging devices used outdoors for vehicle driving assistance, etc. It has been proposed to detect lane areas by detecting white lines that divide the road in the captured images (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-060842 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is difficult to apply the detection method described in Patent Document 1 to road surfaces on which white lines are not drawn. In addition, in a configuration using an ultra-wide-angle optical system such as a fisheye lens, the accuracy of detecting white lines decreases due to large distortion at the periphery of the acquired image, and it is expected that the accuracy of detecting the road surface will also decrease.

[0005] Therefore, in consideration of the problems with the conventional technology as described above, an object of the present disclosure is to provide an image processing device, an imaging device, and a detection method that improve the accuracy of detecting ground surfaces in various aspects in ultra-wide-angle images. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an image processing apparatus according to a first aspect comprises: an acquisition unit that acquires a standard image captured from a standard position and a reference image captured from a position different from the standard position; a controller for detecting the ground or floor surface based on an observation disparity between an arbitrary first pixel in the base image and a second pixel in the reference image, the observation disparity being calculated by matching based on a degree of disparity between the first pixel and the second pixel, and a ground disparity between pixels projected onto the base image and the reference image from a ground grid partition that is partitioned on the assumption that all object points in the base image are the ground, The controller calculates a position of a second pixel in the reference image that corresponds to the first pixel by projecting and projecting the first pixel, calculates a position of an object point corresponding to the first pixel based on a dissimilarity between the first pixel and the second pixel, and calculates the observation parallax based on the positions of the first pixel and the second pixel that correspond to the object point.

[0007] Moreover, an imaging apparatus according to a second aspect comprises: a reference imaging unit located at a reference position and configured to capture a reference image; a reference image capturing unit that is located at a reference position different from the reference position and captures a reference image; a controller for detecting the ground or floor surface based on an observation disparity between an arbitrary first pixel in the base image and a second pixel in the reference image, the observation disparity being calculated by matching based on a degree of disparity between the first pixel and the second pixel, and a ground disparity between pixels projected onto the base image and the reference image from a ground grid partition that is partitioned on the assumption that all object points in the base image are the ground, The controller calculates a position of a second pixel in the reference image that corresponds to the first pixel by projecting and projecting the first pixel, calculates a position of an object point corresponding to the first pixel based on a dissimilarity between the first pixel and the second pixel, and calculates the observation parallax based on the positions of the first pixel and the second pixel that correspond to the object point.

[0008] A detection method according to a third aspect of the present invention comprises: A standard image captured from a standard position and a reference image captured from a position different from the standard position are obtained; A first pixel in the reference image is projected onto a hypothetical plane; calculating a position of a second pixel in the reference image corresponding to the first pixel by projecting the second pixel; calculating a position of an object point corresponding to the first pixel based on a dissimilarity between the first pixel and the second pixel; Calculating an observation disparity between the first pixel and the second pixel based on positions of the first pixel and the second pixel corresponding to the object point; The ground or floor surface is detected based on the observation parallax and the ground parallax between pixels projected onto the standard image and the reference image from a ground grid partition that is partitioned assuming that all object points in the standard image are the ground. Effect of the Invention

[0009] According to the image processing device, imaging device, and detection method configured as described above according to the present disclosure, the measurement accuracy of various aspects of the ground surface in an ultra-wide-angle image is improved. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a schematic configuration of an imaging device including an image processing device according to a first embodiment. [Diagram 2] FIG. 2 is a conceptual perspective view for explaining the configuration of a ground grid in a first three-dimensional coordinate system. [Diagram 3] FIG. 2 is a diagram showing a first three-dimensional coordinate system viewed from the Y-axis direction to explain a ground angle. [Figure 4] 4 is a flowchart for explaining a ground detection process executed by the controller of FIG. 1. [Diagram 5] FIG. 11 is a block diagram showing a schematic configuration of an imaging device including an image processing device according to a second embodiment. [Figure 6] FIG. 2 is a diagram for explaining a three-dimensional coordinate system of a normal vector of a virtual plane. [Figure 7] 6 is a flowchart for explaining a ground detection process executed by the controller of FIG. 5. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of an image processing device to which the present disclosure is applied will be described with reference to the drawings.

[0012] 1, an imaging device 11 including an image processing device 10 of the first embodiment may include a stereo camera 12 and the image processing device 10. The imaging device 11 may be mounted on, for example, a moving object.

[0013] The moving object may include, for example, a vehicle with an automatic driving function and an aircraft. The vehicle may include, for example, an automobile, an industrial vehicle, a railroad vehicle, a vehicle for daily life, and a fixed-wing aircraft that runs on a runway. The automobile may include, for example, a passenger car, a truck, a bus, a motorcycle, and a trolley bus. The industrial vehicle may include, for example, an industrial vehicle for agriculture and construction. The industrial vehicle may include, for example, a forklift and a golf cart. The industrial vehicle for agriculture may include, for example, a tractor, a cultivator, a transplanter, a binder, a combine, and a lawnmower. The industrial vehicle for construction may include, for example, a bulldozer, a scraper, a shovel, a crane, a dump truck, and a road roller. The vehicle may include one that runs by human power. The classification of the vehicle is not limited to the above-mentioned examples. For example, the automobile may include an industrial vehicle that can run on a road. The same vehicle may be included in multiple classifications. The aircraft may include, for example, a fixed-wing aircraft and a rotary-wing aircraft.

[0014] The image processing device 10 may calculate the position of the road surface or ground in the stereo image in a first three-dimensional coordinate system defined based on the imaging device 11, based on the captured stereo image. In the present disclosure, the imaging device 11 is defined to have a normal attitude on the ground. It is assumed that the imaging device 11 is installed so as to have a normal attitude when, for example, a moving object is stopped on a horizontal plane. In the normal attitude, the first three-dimensional coordinate system is a coordinate system with a Y axis parallel to the vertical direction, a Z axis parallel to the direction away from the imaging device 11, and an X axis parallel to the Z axis and the Y axis as coordinate axes.

[0015] The imaging device 11 may notify the position of the road surface or the ground in the first three-dimensional coordinate system to the external device 13. In a configuration in which the imaging device 11 is mounted on a moving body, the external device 13 is, for example, a driving support device such as an auto-cruise control, a driving support device such as an automatic braking device, a notification device that notifies the distance to a subject, and a warning device that issues a warning.

[0016] The stereo camera 12 includes a plurality of imaging units having parallax with respect to each other. The stereo camera 12 can image an object from a plurality of directions by making the plurality of imaging units cooperate with each other. The stereo camera 12 may include a plurality of imaging units in a single housing. The stereo camera 12 may include two or more imaging units that are independent of each other and located apart from each other. The position and orientation of each of the plurality of imaging units in the first three-dimensional coordinate system are known. The position of the imaging unit is, for example, the position of the intersection between the light receiving surface of the imaging unit and the optical axis of the optical system. The orientation of the imaging unit is, for example, the direction of the optical axis of the optical system of the imaging unit.

[0017] The stereo camera 12 includes, for example, a reference imaging section 14 and a reference imaging section 15. The reference imaging section 14 is located at a reference position in the imaging device 11. The reference imaging section 15 is located at a reference position. The reference position is different from the reference position. The reference imaging section 14 at the reference position and the reference imaging section 15 at the reference position may be located such that their optical axes face in directions capable of imaging the same subject.

[0018] The optical axes of the optical systems of the standard imaging section 14 and the reference imaging section 15 may be parallel to each other. The optical axes being parallel is not limited to being strictly parallel, and may allow for misalignment during assembly, misalignment during installation, and misalignment over time. The optical axes of the standard imaging section 14 and the reference imaging section 15 are not limited to being parallel, and may point in different directions.

[0019] The reference imaging section 14 and the reference imaging section 15 may have optical systems with the same optical characteristics. The reference position and the reference position may be the same in a direction parallel to the optical axis of the optical system. Furthermore, the reference imaging section 14 and the reference imaging section 15 may have imaging elements with the same characteristics.

[0020] The standard imaging unit 14 and the reference imaging unit 15 each include an optical system that defines an optical axis and an imaging element. The optical system includes a lens or a mirror, and forms an image of a subject on a light receiving surface of the imaging element. The imaging element is, for example, a CCD image sensor (Charge-Coupled Device Image Sensor) or a CMOS image sensor (Complementary MOS Image Sensor).

[0021] The reference imaging unit 14 captures an image from a reference position and generates a reference image as a signal. The reference imaging unit 15 captures an image from a reference position and generates a reference image as a signal. Specifically, optical images formed on the imaging elements of the reference imaging unit 14 and the reference imaging unit 15 are generated as the reference image and the reference image. The reference imaging unit 14 and the reference imaging unit 15 may perform any image processing, such as brightness adjustment, contrast adjustment, gamma correction, etc., on the captured images except for distortion correction of the optical system.

[0022] The image processing device 10 includes an acquisition unit 16 and a controller 17. The image processing device 10 may further include an assignment unit 18 and a memory 19.

[0023] The acquisition unit 16 is an input interface of the image processing device 10 that acquires information from the stereo camera 12 and an external device. For example, a physical connector and a wireless communication device are adopted for the acquisition unit 16. The physical connector includes an electric connector corresponding to transmission by an electric signal, an optical connector corresponding to transmission by an optical signal, and an electromagnetic connector corresponding to transmission by an electromagnetic wave. The electric connector includes a connector conforming to IEC60603, a connector conforming to the USB standard, a connector corresponding to an RCA terminal, a connector corresponding to an S terminal defined in EIAJ CP-1211A, a connector corresponding to a D terminal defined in EIAJ RC-5237, a connector conforming to the HDMI (registered trademark) standard, and a connector corresponding to a coaxial cable including BNC. The optical connector includes various connectors conforming to IEC 61754. The wireless communication device includes a wireless communication device conforming to each standard including Bluetooth (registered trademark) and IEEE802.11. The wireless communication device includes at least one antenna.

[0024] The acquisition unit 16 acquires, as signals, the standard image and the reference image generated by the stereo camera 12 through imaging. The acquisition unit 16 may provide the acquired standard image and reference image to the controller 17.

[0025] The assigning unit 18 is an output interface of the image processing device 10 that can assign the position in the three-dimensional coordinate system calculated by the image processing device 10 as a signal to the external device 13. Like the acquiring unit 16, the assigning unit 18 includes various interfaces compatible with wired and wireless communication.

[0026] The memory 19 includes any storage device, such as a random access memory (RAM) and a read only memory (ROM), etc. The memory 19 stores various programs that cause the controller 17 to function and various information that the controller 17 uses.

[0027] The controller 17 includes one or more processors and a memory. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The controller 17 may be either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together.

[0028] The controller 17 performs matching based on the dissimilarity between an arbitrary first pixel in the base image and a second pixel in the reference image, and in the matching, the controller 17 estimates the second pixel corresponding to the first pixel.

[0029] Specifically, the controller 17 performs projective transformation on an arbitrary first pixel. More specifically, the controller 17 may perform projective transformation on the first pixel onto at least one hypothetical plane. The controller 17 may use distortion characteristics of the optical system of the reference imaging unit 14 for the projective transformation. Furthermore, the controller 17 may calculate a position where the projectively transformed first pixel is projected onto the reference image. The controller 17 recognizes the pixel at the calculated position as a second pixel that corresponds to the first pixel in the reference image. Furthermore, the controller 17 calculates the degree of difference between the first pixel and the second pixel.

[0030] The dissimilarity is, for example, a difference between pixel values ​​of a first pixel and a second pixel. The pixel value may be a luminance value. Alternatively, the dissimilarity is a statistical value of differences between each color component of the first pixel and the second pixel. The statistical value is, for example, an arithmetic mean, a geometric mean, a median, a mode, etc. Furthermore, the controller 17 may calculate the dissimilarity between the standard image and the reference image from which feature extraction has been performed. The feature extraction is, for example, edge detection and addition of the detected edge components.

[0031] In matching, the controller 17 calculates the position of the object point corresponding to the first pixel based on the calculated dissimilarity. In order to calculate the position of the object point corresponding to the first pixel, the controller 17 may estimate a plane on which the object point corresponding to the first pixel exists based on the dissimilarity. Furthermore, the controller 17 may calculate the position of the object point using the estimated plane. Furthermore, the controller 17 calculates the positions of a first pixel in the standard image and a second pixel in the reference image that correspond to the position of the object point. Furthermore, the controller 17 calculates an observation parallax that is the parallax between the first pixel and the second pixel based on the positions of the first pixel and the second pixel. Furthermore, the controller 17 may calculate an observation parallax for each of the first pixels constituting the standard image.

[0032] In the first embodiment, more specifically, the controller 17 calculates the observation parallax by matching using the plane sweep method. The calculation of the observation parallax by the plane sweep method will be described in detail below.

[0033] The controller 17 may assume a plurality of hypothetical planes that are parallel to each other and have different distances from the imaging device 11. The plurality of virtual planes may be assumed by provisionally setting the same normal vector n and a plurality of depths dp in the first three-dimensional coordinate system.

[0034] Furthermore, the controller 17 may perform projective transformation of any first pixel onto each of a plurality of hypothetical planes. For example, by calculating the following formula (1), any first pixel constituting the reference image is projected onto the hypothetical plane. ρc -1 (K c -1 u c ) (1)

[0035] In equation (1), ρ c -1 is a transformation equation for projectively transforming an arbitrary pixel (hereinafter also referred to as a first pixel) in the reference image using the distortion characteristics of the optical system of the reference imaging unit 14 used to capture the reference image. c -1 Specifically, K is a transformation equation that transforms the direction of the chief ray passing through the first pixel in the camera coordinate system of the reference imaging unit 14 into a direction bent on the object side relative to the optical system. c -1 is a conversion formula for converting the image coordinates of the reference image into the camera coordinates of the reference imaging unit 14. c is the location of the first pixel in image coordinates of the reference image.

[0036] Furthermore, the controller 17 may convert the position of the first pixel projected onto the hypothetical plane from the camera coordinates of the base imaging unit 14 to the camera coordinates of the reference imaging unit 15 by calculating the following formula (2). Hρ c -1 (K c -1 u c ) (2)

[0037] In equation (2), the conversion equation H satisfies the following equation (3). H=R+tn T / d (3)

[0038] In equation (3), R is a rotational transformation formula from the standard imaging unit 14 to the reference imaging unit 15, and t is a translational transformation formula from the standard imaging unit 14 to the reference imaging unit 15.

[0039] Furthermore, the controller 17 calculates the position, in the image coordinates of the reference image, of the second pixel obtained by projecting the first pixel, which has been subjected to projective transformation on the hypothetical plane, onto the reference image, by calculating the following formula (4). The second pixel is a pixel that corresponds to the first pixel in the reference image. K r ρ r (Hρ c -1 (K c -1 u c )) (4)

[0040] In equation (4), ρ r is a transformation equation for projectively transforming an object point onto a reference image using the distortion characteristics of the optical system of the reference image pickup unit 15 used to capture the reference image. r Specifically, K is a transformation equation that transforms the direction of the principal ray passing through an object point in the camera coordinate system of the reference imaging unit 15 into a direction bent on the image side of the optical system. r is a transformation formula for transforming the camera coordinates of the reference imaging unit 15 into the image coordinates of the reference image.

[0041] Since the position of the second pixel described above is a position where the first pixel, which has been subjected to projective transformation onto a plurality of hypothetical planes, is projected onto the reference image, the controller 17 calculates the position of the second pixel for each of the plurality of hypothetical planes.

[0042] Furthermore, the controller 17 may calculate the degree of difference between the position of the first pixel in the standard image and a plurality of positions of the second pixel in the reference image. Furthermore, the controller 17 may select the hypothetical plane in which the position of the second pixel with the lowest degree of difference has been calculated. The selected hypothetical plane is a plane in which an object point corresponding to the first pixel is estimated to exist.

[0043] Furthermore, the controller 17 may calculate the position of a second pixel obtained by projecting the first pixel, which has been subjected to projective transformation onto the selected hypothetical plane, onto the reference image. Furthermore, the controller 17 may calculate the disparity between the position of the first pixel and the calculated position of the second pixel as the above-mentioned observation disparity d m Calculate as follows: d mis a vector in the image coordinate system of the reference image. Therefore, the observation disparity d m The direction of the observation disparity d may vary depending on the position of the reference image selected as the first pixel. m is calculated using equation (5). d m =u c -K r ρ r (Hρ c -1 (K c -1 u c )) (5)

[0044] Furthermore, the controller 17 calculates the three-dimensional coordinates [X m , Y m , Z m The controller 17 may calculate an observation angle Φ m The observation angle Φ m is calculated using equation (6). Φ m =tan -1 (X m / Z m ) (6)

[0045] The controller 17 detects the observation parallax d m and the observation angle Φ m Along with the calculation of, a ground grid may be recognized for the reference image. The ground grid is obtained by dividing the assumed ground into a grid shape, assuming that all object points included in the reference image are the ground. As shown in FIG. 2, the ground grid gr may be formed by dividing a plurality of sections sc by straight lines parallel to the X-axis and straight lines parallel to the Z-axis of a three-dimensional coordinate system based on the imaging device 11 as coordinate axes. Note that the positions of the Y-axis in the three-dimensional coordinate system of the ground grid gr are all the same, e.g., zero.

[0046] The controller 17 detects the observation parallax d mand the ground grid gr, the ground or road surface is detected. The detection of the ground or road surface will be described in detail below.

[0047] The Z coordinate of the section sc is the distance from the imaging device 11. The minimum and maximum distances of the Z coordinate of the section sc may be determined by design. In other words, the minimum and maximum values ​​in the Z-axis direction of the ground grid gr may be determined. Therefore, the position of each section sc of the ground grid gr in the Z-axis is determined based on the position within the ground grid gr.

[0048] The X coordinate of the section sc is the horizontal position of the imaging device 11 fixed to the normal attitude and position. The X coordinate of the section sc may be determined by design. In other words, the minimum and maximum values ​​of the ground grid gr in the X-axis direction may be determined.

[0049] As shown in Figure 3, any partition sc ij The three-dimensional coordinates of the area sc ij The ground angle Φ i and Z-axis coordinate Z j Expressed as (Z j tanΦ i ,0,Z j ) The ground angle Φ i is the distance from the origin of the first three-dimensional coordinate system to any section sc when looking at the ground grid gr in the Y-axis direction. ij is the angle between the direction toward the center and the Z-axis direction.

[0050] In addition, any section sc ij The three-dimensional coordinates of the area sc ij The x-axis coordinate X is defined i and Z-axis coordinate Z j By expressing it as (X i ,0,Z j ) Any section sc ij The three-dimensional coordinates of (X i ,0,Z j ) in the configuration recognized as the ground angle Φ i is Φ i =tan -1 (Xi / Z i ) may be calculated.

[0051] The controller 17 is ij The three-dimensional coordinates of (Z j tanΦ i ,0,Z j ) to K c and ρ c The controller 17 may calculate the position of the image projected onto the reference image by using the above. ij The three-dimensional coordinates of (Z j tanΦ i ,0,Z j ) to K r and ρ r The controller 17 calculates the disparity between pixels corresponding to the positions projected onto the standard image and the reference image, respectively, by using the ground disparity d ij The ground disparity d ij is calculated by the formula (7). ij is any partition sc ij The controller 17 may calculate the ground parallax and the ground angle for all the sections sc that make up the ground grid gr. d ij =K c ρ c P ij -K r ρ r (RP ij +t) (7)

[0052] The three-dimensional coordinates of each section sc can be determined based on factors that are determined in advance, such as the optical system and design of the standard imaging unit 14 and the reference imaging unit 15. ij The three-dimensional coordinates of the ground disparity d ij and ground angle Φ i may be determined at the time of manufacture and stored in memory 19.

[0053] The controller 17 detects the observation parallax d m and ground disparity d ijThe controller 17 may further detect the ground or floor surface based on at least a comparison of the observation angle Φ m and ground angle Φ i The ground or floor surface may be detected based on the following. A specific method for detecting the ground or floor surface will be described below.

[0054] When an arbitrary section sc is included in the reference image as an image of the ground or floor surface, the ground disparity d of the first pixel at the position where the arbitrary section sc is projected onto the reference image is ij and the observed parallax d m are likely to be virtually identical. ij and the observed parallax d m Being substantially the same means that the difference between the two is so small that they can be considered to be the same even if they do not match.

[0055] On the other hand, if another object exists between the arbitrary section sc and the image capture device 11, the other object exists in the direction from the image capture device 11 to the arbitrary section sc, so the ground disparity d i and the observed parallax d m is likely to be different because the ground disparity d ij is the disparity assuming that the ground or floor image is included in the reference image, while the observed disparity d m is the parallax with respect to the image of the other object.

[0056] However, the ground disparity d i is calculated based on the three-dimensional coordinates of the block sc, while the observation disparity d m is calculated based on the position in the reference image of the first pixel constituting the reference image. Therefore, there is not necessarily a first pixel that perfectly matches any given section sc. Perfectly matching any given section sc means, for example, that the chief ray passing through the center of any given section sc passes through the center of any pixel on the light receiving surface of the reference imaging unit 14. Therefore, the controller 17 calculates the ground parallax d for each section sc. ij The difference between the observed parallax d and mIf there exists a first pixel where .gamma., then it may be estimated that the section sc is included in the reference image as the ground or floor surface.

[0057] However, if we judge only by the parallax, the observed parallax d of the first pixel at a position completely different from the projection position of the block sc onto the reference image is m and the ground disparity d of the area sc i It may happen that the difference between the ground parallax d and the ground parallax d is extremely small by chance. i and the observed parallax d m In addition to the difference between the ground angle Φ i and the observation angle Φ m If the difference between the first and second images is also extremely small, it may be estimated that the section sc is included in the reference image as the ground or road surface.

[0058] Specifically, the controller 17 is ij The first pixel-by-pixel difference ξ ijm =[Φ m -Φ i d m -d ij ] T Further, the controller 17 may calculate the difference ξ ijm Based on this, the following formula (8) is calculated to determine the area sc ij The likelihood function L for ij may be calculated.

[0059]

number

[0060] In equation (8), Γ is the covariance matrix. ijm T Gamma -1 ξ ijm ) is the difference ξ ijm is a Gaussian function whose probability density increases as it approaches zero. Therefore, the likelihood function L ijindicates that the larger it is, the higher the possibility that the image of the ground in the section sc is projected onto the reference image.

[0061] The controller 17 may calculate the likelihood function L for each of the sections sc. The controller 17 may detect the section sc having a large likelihood function L by estimating it to be the ground or floor surface. Furthermore, the controller 17 may calculate the likelihood function L as described below. ij Based on the cost function C ij The travelable area may be detected by calculating:

[0062] The controller 17 calculates a cost function C ij To calculate the likelihood function L ij The inverse of the section sc ij Evaluation function D for ij The controller 17 may calculate the evaluation function D for each of the sections sc. Furthermore, the controller 17 may calculate the evaluation function D ij Based on this, the following formula (9) is calculated to determine the area sc ij The cost function C ij may be calculated.

[0063]

number

[0064] In equation (9), C S is a smoothing term. In equation (9), Z j′ is the same as the ground angle Φ i If the position of the Z axis is an arbitrary section sc ij Different section from SC ij′ is the Z-axis position.

[0065] The controller 17 calculates a cost function C ij The path that minimizes the sum of the cost functions C may be estimated as the boundary between the ground or road surface and objects other than the ground such as vehicles, pedestrians, and buildings. ijThe controller 17 may detect the travelable area by connecting the boundary positions estimated for each position in the X-axis direction of the section sc.

[0066] Next, the ground detection process executed by the controller 17 in the first embodiment will be described with reference to the flowchart of Fig. 4. The ground detection process starts every time one frame of a standard image and a reference image are acquired from the stereo camera 12.

[0067] In step S100, the controller 17 selects an unselected pixel in the reference image. After selection, the process proceeds to step S101.

[0068] In step S101, the controller 17 performs a projective transformation of the pixels selected in step S100 onto a plurality of hypothetical planes. After the projective transformation, the process proceeds to step S102.

[0069] In step S102, the controller 17 projects the pixels that have been subjected to projective transformation onto the multiple hypothetical planes in step S102 onto the reference image. After the projection, the process proceeds to step S103.

[0070] In step S103, the controller 17 calculates the degree of dissimilarity between the pixels corresponding to the respective positions projected in step S102 and the pixels selected in step S100 as the degree of dissimilarity corresponding to each hypothetical plane. After the calculation, the process proceeds to step S104.

[0071] In step S104, the controller 17 selects the hypothetical plane corresponding to the dissimilarity degree that is the lowest value among the multiple dissimilarity degrees calculated in step S103. After the selection, the process proceeds to step S105.

[0072] In step S105, the controller 17 identifies pixels corresponding to positions where the object point on the hypothetical plane selected in step S104 is projected onto the standard image and the reference image. After identification, the process proceeds to step S106.

[0073] In step S106, the controller 17 calculates the observation disparity d based on the position of the pixel identified in step S105. m and the observation angle Φ m After the calculation, the process proceeds to step S107.

[0074] In step S107, the controller 17 calculates the observation parallax d m and the observation angle Φ m It is determined whether or not the calculation has been performed. If not, the process returns to step S100. If the calculation has been performed, the process proceeds to step S108.

[0075] In step S108, the controller 17 selects the unselected partitions sc in the ground grid gr. ij After the selection, the process proceeds to step S109.

[0076] In step S109, the controller 17 selects the partition sc selected in step S108. ij The ground disparity d ij and ground angle Φ i , and the observation disparity d m and the observation angle Φ m Based on the likelihood function L ij After the calculation, the process proceeds to step S110.

[0077] In step S110, the controller 17 calculates the ground disparity d for all the sections sc constituting the ground grid gr. ij and ground angle Φ i It is determined whether or not the calculation has been performed. If it has not been calculated, the process returns to step S108. If it has been calculated, the process proceeds to step S111.

[0078] In step S111, the controller 17 calculates the likelihood function L ij Based on each section SC ij The cost function C ij After the calculation, the process proceeds to step S112.

[0079] In step S112, the controller 17 calculates the distance between each section sc ij The cost function C ij After the detection, the ground detection process ends.

[0080] The image processing device 10 of the first embodiment configured as described above calculates an observation disparity d m and a ground grid gr that divides the ground, the controller 17 performs projective transformation of a first pixel onto at least one hypothetical plane, calculates the position of a second pixel in the reference image that corresponds to the first pixel by projecting the projectively transformed first pixel onto a reference image, calculates the position of an object point corresponding to the first pixel by estimating a plane on which the object point exists based on a dissimilarity between the first pixel and the second pixel, and calculates an observation disparity d m With this configuration, the image processing device 10 can detect ground surfaces in various forms, regardless of the presence or absence of white lines that are generally drawn on roads. In addition, with this configuration, the image processing device 10 calculates the positions of object points using a projective transformation onto a hypothetical plane, and calculates the observation parallax d m Since the above calculation is performed, the detection accuracy can be improved even for images with large distortion.

[0081] Next, a second embodiment of the present disclosure will be described. The second embodiment differs from the first embodiment in the method of calculating observation parallax by matching. The second embodiment will be described below, focusing on the differences from the first embodiment. Note that the same reference numerals are used to designate parts having the same configuration as the first embodiment.

[0082] 5, in the second embodiment, an imaging device 110 including an image processing device 100 may include a stereo camera 12 and an image processing device 100, similarly to the first embodiment. In the second embodiment, the structure and function of the stereo camera 12 are the same as those in the first embodiment.

[0083] In the second embodiment, the image processing device 100 may be configured to include an acquisition unit 16 and a controller 170, similar to the first embodiment. The image processing device 100 may further be configured to include an assignment unit 18 and a memory 19. In the second embodiment, the structures and functions of the acquisition unit 16, the assignment unit 18, and the memory 19 are the same as those in the first embodiment. In the second embodiment, the structure of the controller 170 is the same as that in the first embodiment. In the second embodiment, the function of the controller 170, in other words, the processing executed, is different from that in the first embodiment.

[0084] Similar to the first embodiment, the controller 170 performs matching based on the dissimilarity between an arbitrary first pixel in the base image and a second pixel in the reference image, and in the matching, the controller 17 estimates the second pixel corresponding to the first pixel.

[0085] In the second embodiment, more specifically, the controller 170 uses the patch matching method to calculate the observation disparity d m The observation disparity d m The calculation of is described in detail below.

[0086] The controller 170 calculates the observation disparity d for each first pixel constituting the reference image by a patch matching method using the reference image and the reference image. mmay be calculated. Specifically, the controller 170 may specify a plane on which a corresponding object point exists for each first pixel constituting the reference image by performing assumption of an initial assumption plane, verification of the assumption plane, spatial propagation, and random selection. Furthermore, the controller 170 may estimate a second pixel corresponding to the first pixel in the reference image by projecting the plane on which the corresponding object point exists onto the reference image for each first pixel constituting the reference image. Note that, in the second embodiment, the controller 170 may use the distortion characteristics of the optical system of the reference imaging unit 14 for projective transformation of the first pixel onto the assumption plane in the verification of the assumption plane, spatial propagation, and random selection of the patch match method, as described below.

[0087] First, the controller 170 assumes an initial assumed plane for the reference image as described below. Specifically, the controller 170 assumes an initial assumed plane for each of a plurality of pixels constituting the reference image. The assumed plane is a so-called plane hypothesis, and is a very small plane on which it is assumed that an object point corresponding to the pixel exists in real space.

[0088] The virtual plane may be assumed by provisionally setting a normal vector n and a depth dp. The normal vector n is a so-called normal hypothesis, and may be expressed in a three-dimensional coordinate system with the optical axis of the optical system of the reference imaging unit 14 and two straight lines perpendicular to the optical axis and perpendicular to each other as axes. As shown in FIG. 6, the normal vector n is a unit vector determined by a first rotation angle φ about the optical axis from an axis other than the axis coinciding with the optical axis ox in the three-dimensional coordinate system, and a second rotation angle θ from the optical axis ox. Specifically, the normal vector n is [sinθcosΦ sinθsinΦ cosθ]. The depth dp is a so-called depth hypothesis, and is the depth in the direction of the optical axis ox of the reference imaging unit 14. Specifically, the depth d may be the position of an object point from the optical center along the optical axis ox. The controller 170 may generate the normal vector n and the depth dp by random numbers. The controller 17 may limit the range of the depth dp before making the assumption of the initial assumption plane. The range of the depth dp to be limited may be determined in advance.

[0089] Controller 170 then calculates the dissimilarity of the first pixel and the second pixel in testing the hypothesized hypothetical plane, as described below.

[0090] In verifying the assumed plane, the controller 170 projects each of the pixels constituting the reference image onto the assumed plane assumed for the pixel. For example, by calculating the following equation (10), an arbitrary first pixel constituting the reference image is projected onto the assumed plane. c is projected onto the hypothetical plane. ρ c -1 (K c -1 u c ) (10)

[0091] In verifying the hypothetical plane, the controller 170 further converts the position of the first pixel projected onto the hypothetical plane from the camera coordinates of the base imaging unit 14 to the camera coordinates of the reference imaging unit 15 by calculating the following equation (11). Hρ c -1 (K c -1 u c ) (11)

[0092]

[0093] In the verification of the hypothetical plane, the controller 170 further calculates the position, in the image coordinates of the reference image, of a second pixel obtained by projecting onto the reference image the first pixel that has been projected onto the hypothetical plane by calculating the following formula (12). The second pixel is a pixel that corresponds to the first pixel in the reference image. K r ρ r (Hρ c -1 (K c -1 u c )) (12)

[0094] In further verifying the hypothetical plane, the controller 170 calculates a dissimilarity between the first pixel and the second pixel as a tentative dissimilarity.

[0095] Controller 170 may then re-assume the assumed plane by performing spatial propagation, as described below.

[0096] In the spatial propagation, the controller 170 may determine whether a pixel having a dissimilarity less than the dissimilarity of an arbitrary pixel constituting the reference image is located within a region surrounding the arbitrary pixel. The region surrounding the arbitrary pixel may be arbitrarily determined. The region surrounding the arbitrary pixel is a region including a pixel of interest. The region surrounding the arbitrary pixel is, for example, a region including arbitrary pixels such as 2×2, 3×3, 4×4, etc.

[0097] In the spatial propagation, if pixels with dissimilarity less than the dissimilarity of the given pixel are located in the region surrounding the given pixel, the controller 170 may select the pixel with the smallest dissimilarity less than the dissimilarity of the given pixel. The controller 170 may replace the assumed plane assumed for the given pixel with the assumed plane assumed for the selected pixel to perform the above-mentioned assumed plane verification.

[0098] Specifically, the controller 170 may project an arbitrary pixel of the reference image onto the replaced assumed plane. Furthermore, the controller 170 may perform coordinate conversion of the position of the arbitrary pixel projected onto the replaced assumed plane from the camera coordinate of the standard imaging unit 14 to the camera coordinate of the reference imaging unit 15. Furthermore, the controller 170 may calculate the position of a pixel in the reference image projected onto the arbitrary pixel projected onto the replaced assumed plane. Furthermore, the controller 170 may calculate a degree of difference for comparison between the arbitrary pixel and the projected pixel.

[0099] In the spatial propagation, the controller 170 may further compare the dissimilarity of the provisional determination at the arbitrary pixel with the calculated dissimilarity for comparison. If the dissimilarity for comparison is less than the dissimilarity of the provisional determination, the controller 170 may re-assume the replaced assumed plane as the assumed plane for the arbitrary pixel. The controller 170 may recognize the dissimilarity corresponding to the re-assumed assumed plane, in other words, the dissimilarity for comparison, as the dissimilarity of the provisional determination. The controller 170 may re-assume the assumed plane for each pixel constituting the reference image.

[0100] Controller 170 then re-assumes the assumption plane by performing a randomization, as described below.

[0101] In the random selection, the controller 170 may perturb the hypothetical plane re-assumed for any pixel constituting the reference image by random number selection. Perturbation of the hypothetical plane means giving a minute amount of deviation from the normal vector n and depth dp of the hypothetical plane re-assumed by a random number. For example, the normal vector n is changed by a uniform random number in a minute range for each of the first rotation angle Φ and the second rotation angle θ. Also, the depth d is changed by a uniform random number in a minute range. The minute range may be determined by a ratio of, for example, 1 / 100 to the current value. The controller 170 may perform the above-mentioned verification of the hypothetical plane for the perturbed hypothetical plane.

[0102] Specifically, the controller 170 may project an arbitrary pixel of the reference image onto the perturbed hypothetical plane. Furthermore, the controller 170 may perform coordinate transformation of the position of the arbitrary pixel projected onto the hypothetical plane from the camera coordinate of the reference imaging unit 14 to the camera coordinate of the reference imaging unit 15. Furthermore, the controller 170 may calculate the position of a pixel in the reference image projected onto the reference image of the arbitrary pixel projected onto the hypothetical plane. Furthermore, the controller 170 may calculate a degree of difference for comparison between the arbitrary pixel and the projected pixel.

[0103] In the random selection, the controller 170 may further compare the tentatively determined dissimilarity at the arbitrary pixel with the calculated comparison dissimilarity. If the comparison dissimilarity is less than the tentatively determined dissimilarity, the controller 170 may re-assume the perturbed assumed plane as the assumed plane for the arbitrary pixel. The controller 170 may determine the dissimilarity corresponding to the re-assumed assumed plane, in other words, the comparison dissimilarity, as the tentatively determined dissimilarity. The controller 170 may re-assume the assumed plane for each pixel constituting the reference image.

[0104] The controller 170 may repeat the spatial propagation and randomization so that the dissimilarity decreases. For example, the controller 170 may repeat the execution of sequences of spatial propagation and randomization so that the dissimilarity decreases.

[0105] The controller 170 may repeat the spatial propagation and randomization until the dissimilarity is equal to or less than a discrimination threshold, which is the difference at which the images represented by the first pixel and the second pixel are deemed to be the same. An upper limit may be set for the number of times the spatial propagation and randomization are repeated.

[0106] The controller 170 may determine that the hypothetical plane of an arbitrary pixel for which the re-hypothesization of the perturbed hypothetical plane and the calculation of the dissimilarity have been repeated is the real plane on which the object point corresponding to the arbitrary pixel is located. The controller 170 performs projective transformation of the arbitrary pixel in the standard image onto the determined real plane, and further calculates the disparity between the position of the pixel projected onto the reference image and the arbitrary pixel as the observed disparity d. m It may be calculated as:

[0107] The controller 170 calculates the three-dimensional coordinates [X m , Y m , Z m Similar to the first embodiment, the controller 170 may calculate the observation angle Φ based on the three-dimensional coordinates of the object point. m may be calculated.

[0108] The controller 170, like the first embodiment, controls the observation parallax d m and the ground grid gr, the controller 170 detects the ground or road surface. Furthermore, the controller 170 may detect a travelable area, as in the first embodiment.

[0109] Next, a ground detection process executed by the controller 170 in the second embodiment will be described with reference to the flowchart of Fig. 7. The ground detection process starts every time one frame of a standard image and a reference image are acquired from the stereo camera 12.

[0110] In step S200, controller 170 makes an initial assumption of a hypothetical plane for each pixel that constitutes the reference image. After assumption, the process proceeds to step S201.

[0111] In step S201, the controller 170 performs a hypothetical plane verification for each pixel that constitutes the reference image. After the verification, the process proceeds to step S202.

[0112] In step S202, the controller 170 performs spatial propagation using the dissimilarity provisionally determined by the verification in step S201 for pixels for which spatial propagation and random selection have not been performed or for pixels for which the dissimilarity of the provisional determination is equal to or less than the discrimination threshold value. After performing spatial propagation, the process proceeds to step S203.

[0113] In step S203, the controller 170 performs random selection for each pixel constituting the reference image based on the hypothetical plane re-hypothesized by the spatial propagation in step S202. After performing random selection, the process proceeds to step S204.

[0114] In step S204, the controller 170 determines whether there are any pixels whose dissimilarity after random selection in step S203 is equal to or less than the discrimination threshold. If there are any pixels whose dissimilarity is not equal to or less than the discrimination threshold, the process proceeds to step S205. If the dissimilarity of all pixels is equal to or less than the discrimination threshold, the process proceeds to step S206.

[0115] In step S205, the controller 170 determines whether the number of iterations of the spatial propagation in step S202 and the random selection in step S203 is less than the upper limit. If it is less than the upper limit, the process returns to step S202. If it is not less than the upper limit, in other words, if the upper limit is reached, the process proceeds to step S206.

[0116] In step S206, the controller 170 determines the hypothetical plane after random selection in step S203 for each pixel constituting the reference image as an actual plane. After determination, the process proceeds to step S207.

[0117] In step S207, the controller 170 calculates the positions of object points corresponding to each pixel in the reference image, based on the positions of each pixel in the reference image and the real plane identified in step S206. After the calculation, the process proceeds to step S208.

[0118] In step S208, the controller 170 identifies pixels corresponding to positions of projections from the object point on the plane identified in step S206 onto the standard image and the reference image. After identification, the process proceeds to step S209.

[0119] In step S209, the controller 170 calculates the observation disparity d based on the position of the pixel identified in step S208. m and the observation angle Φ m After the calculation, the process proceeds to step S210.

[0120] In steps S210 to S214, the controller 170 executes the same controls as in steps S108 to S112 of the ground detection process in the first embodiment. After the travelable area is detected in step S214, the ground detection process ends.

[0121] The image processing device 100 of the second embodiment having the above-mentioned configuration also calculates an observation disparity d between an arbitrary first pixel in a base image and a second pixel in a reference image by matching based on the degree of difference between the first pixel and the second pixel. m and a ground grid gr that divides the ground, the controller 170 performs projective transformation of a first pixel onto at least one hypothetical plane, calculates the position of a second pixel in the reference image that corresponds to the first pixel by projecting the projectively transformed first pixel onto a reference image, calculates the position of an object point corresponding to the first pixel by estimating a plane on which the object point exists based on a dissimilarity between the first pixel and the second pixel, and calculates an observation disparity d m With this configuration, the image processing device 100 can detect ground surfaces in various forms, regardless of the presence or absence of white lines that are generally drawn on roads. Also, with this configuration, the image processing device 100 can calculate the positions of object points using a projective transformation onto a hypothetical plane and calculate the observation parallax d m Since the above calculation is performed, the detection accuracy can be improved even in the case of an image with large distortion.

[0122] In one embodiment, (1) an image processing device includes: an acquisition unit that acquires a standard image captured from a standard position and a reference image captured from a position different from the standard position; a controller that detects the ground or floor surface based on an observation disparity between an arbitrary first pixel in the base image and a second pixel in the reference image, the observation disparity being calculated by matching based on a degree of disparity between the first pixel and the second pixel, and a ground disparity between pixels projected onto the base image and the reference image from a ground grid partition that is partitioned on the assumption that all object points in the base image are the ground, The controller calculates a position of a second pixel in the reference image that corresponds to the first pixel by projecting and projecting the first pixel, calculates a position of an object point corresponding to the first pixel based on a dissimilarity between the first pixel and the second pixel, and calculates the observation parallax based on the positions of the first pixel and the second pixel that correspond to the object point.

[0123] (2) In the image processing device of (1) above, The controller uses distortion characteristics of an optical system used to capture the reference image for projective transformation of the first pixel onto the hypothetical plane.

[0124] (3) In the image processing device of (1) or (2) above, The controller performs projective transformation of the first pixel onto a plurality of hypothetical planes that are parallel to each other and have different distances, and estimates a plane in which the object point exists for each of the first pixels that constitute the reference image by selecting from the plurality of hypothetical planes.

[0125] (4) In the image processing device of (1) or (2) above, The controller: In a patch matching method using the standard image and the reference image, a plane on which an object point corresponding to the first pixel exists is estimated, and the observation parallax is calculated based on a position of the second pixel in the reference image calculated based on the first pixel and the plane; In the patch matching method, the first pixel is projected onto an assumed plane on which an object point is assumed to exist, and the position of a second pixel corresponding to the first pixel in the reference image is calculated by projecting the projected transformed first pixel onto the reference image, and the dissimilarity between the first pixel and the second pixel is calculated.

[0126] In one embodiment, (5) the imaging device comprises: a reference imaging unit located at a reference position and configured to capture a reference image; a reference image capturing unit that is located at a reference position different from the reference position and captures a reference image; a controller that detects the ground or floor surface based on an observation disparity between an arbitrary first pixel in the base image and a second pixel in the reference image, the observation disparity being calculated by matching based on a degree of disparity between the first pixel and the second pixel, and a ground disparity between pixels projected onto the base image and the reference image from a ground grid partition that is partitioned on the assumption that all object points in the base image are the ground, The controller calculates a position of a second pixel in the reference image that corresponds to the first pixel by projecting and projecting the first pixel, calculates a position of an object point corresponding to the first pixel based on a dissimilarity between the first pixel and the second pixel, and calculates the observation parallax based on the positions of the first pixel and the second pixel that correspond to the object point.

[0127] In one embodiment, the detection method (6) comprises: A standard image captured from a standard position and a reference image captured from a position different from the standard position are obtained; A first pixel in the reference image is projected; calculating a position of a second pixel in the reference image corresponding to the first pixel by projecting the second pixel; calculating a position of an object point corresponding to the first pixel based on a dissimilarity between the first pixel and the second pixel; Calculating an observation disparity between the first pixel and the second pixel based on positions of the first pixel and the second pixel corresponding to the object point; The ground or floor surface is detected based on the observation parallax and the ground parallax between pixels projected onto the standard image and the reference image from a ground grid division that is divided by assuming that all object points in the standard image are the ground.

[0128] The above describes embodiments of the image processing devices 10, 100. However, embodiments of the present disclosure can also be embodied as a method or program for implementing the device, or as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0129] The form of implementation of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may be a program module incorporated into an operating system, etc. Furthermore, the program may or may not be configured so that all processing is performed only by the CPU on the control board. The program may be configured so that a part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as necessary.

[0130] The drawings for explaining the embodiments according to the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0131] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions, etc. included in each component, etc. can be rearranged so as not to cause logical inconsistencies, and multiple components, etc. can be combined into one or divided.

[0132] All of the features described in this disclosure and / or all steps of any method or process disclosed herein may be combined in any combination except those combinations in which the features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is merely one example of a generic series of the same or equivalent features.

[0133] Furthermore, the embodiments of the present disclosure are not limited to any of the specific configurations of the embodiments described above, but rather extend to any novel feature or combination of features described herein, or any novel method or process step or combination of features described herein.

[0134] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In the configurations distinguished by descriptions such as "first" and "second" in this disclosure, the numbers in the configurations can be exchanged. For example, the first pixel can exchange the identifiers "first" and "second" with the second pixel. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the identifier exchange. The identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a code. The descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number. [Explanation of symbols]

[0135] 10, 100 Image processing device 11, 110 Imaging device 12 Stereo Camera 13 External equipment 14 Reference imaging section 15 Reference imaging section 16 Acquisition Department 17, 170 Controller 18 Granting Department 19. Memory gr ground grid sc partition

Claims

1. an acquisition unit that acquires a standard image captured from a standard position and a reference image captured from a position different from the standard position; a controller that detects the ground or floor surface based on an observation disparity between an arbitrary first pixel in the base image and a second pixel in the reference image, the observation disparity being calculated by matching based on a degree of disparity between the first pixel and the second pixel, and a ground disparity between pixels projected onto the base image and the reference image from a ground grid partition that is partitioned on the assumption that all object points in the base image are the ground, The controller calculates a position of a second pixel in the reference image corresponding to the first pixel by projecting and projecting the first pixel, calculates a position of an object point corresponding to the first pixel based on a dissimilarity between the first pixel and the second pixel, and calculates the observation parallax based on the positions of the first pixel and the second pixel corresponding to the object point. Image processing device.

2. 2. The image processing device according to claim 1, The controller uses distortion characteristics of an optical system used to capture the reference image for projective transformation of the first pixel onto the hypothetical plane. Image processing device.

3. 3. The image processing device according to claim 1, The controller performs projective transformation of the first pixel onto a plurality of hypothetical planes that are parallel to each other and have different distances, and estimates a plane on which the object point exists for each of the first pixels constituting the reference image by selecting from the plurality of hypothetical planes. Image processing device.

4. 3. The image processing device according to claim 1, The controller: In a patch matching method using the standard image and the reference image, a plane on which an object point corresponding to the first pixel exists is estimated, and the observation parallax is calculated based on a position of the second pixel in the reference image calculated based on the first pixel and the plane; The first pixel is projected onto an assumed plane on which an object point is assumed to exist in the patch matching method, and the position of a second pixel corresponding to the first pixel in the reference image is calculated by projecting the projected first pixel onto the reference image, and a dissimilarity between the first pixel and the second pixel is calculated. Image processing device.

5. a reference imaging unit located at a reference position and configured to capture a reference image; a reference image capturing unit that is located at a reference position different from the reference position and captures a reference image; a controller that detects the ground or floor surface based on an observation disparity between an arbitrary first pixel in the base image and a second pixel in the reference image, the observation disparity being calculated by matching based on a degree of disparity between the first pixel and the second pixel, and a ground disparity between pixels projected onto the base image and the reference image from a ground grid partition that is partitioned on the assumption that all object points in the base image are the ground, The controller calculates a position of a second pixel in the reference image corresponding to the first pixel by projecting and projecting the first pixel, calculates a position of an object point corresponding to the first pixel based on a dissimilarity between the first pixel and the second pixel, and calculates the observation parallax based on the positions of the first pixel and the second pixel corresponding to the object point. Imaging device.

6. A standard image captured from a standard position and a reference image captured from a position different from the standard position are obtained; A first pixel in the reference image is projected; calculating a position of a second pixel in the reference image corresponding to the first pixel by projecting the second pixel; calculating a position of an object point corresponding to the first pixel based on a dissimilarity between the first pixel and the second pixel; calculating an observation disparity between the first pixel and the second pixel based on positions of the first pixel and the second pixel corresponding to the object point; The ground or floor surface is detected based on the observation parallax and the ground parallax between pixels projected onto the base image and the reference image from a ground grid section that is divided on the assumption that all object points in the base image are the ground. Detection methods.

Images