Image Processing Device

Abstract

The present invention addresses the problem of providing an image processing device which determines, on the basis of an image photographed by an in-vehicle camera, whether a parallax calculation result is correct. An image processing device (101) according to the present invention preserves a first image photographed by a first photograph unit (11) and a second image photographed by a second photograph unit (12), generates a plurality of third images obtained by moving, by a prescribed amount, pixels of the first image in a vertical direction, and obtains each parallax from each of the second image and the plurality of third images. In addition, the image processing device compares each of the parallaxes with a parallax expectation value, of a road surface, to be obtained from an image obtained by photographing a flat road surface in front of the vehicle, determines whether each of the parallaxes is valid, and detects, by using the valid parallaxes, an amount of deviation between the optical vertical axes of the first and second photograph units.

Description

[Technical Field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] A technology has been known in the past that analyzes images captured by a stereo camera mounted on a vehicle (own vehicle) to detect the relative vertical misalignment between the left and right cameras and correct the misalignment amount. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-105682 Summary of the Invention [Problem to be solved by the invention]

[0004] When an in-vehicle stereo camera captures an image of a road surface painted with repeated diagonal lines in the depth direction, such as a zebra strip, while there is a relative vertical misalignment between the optical axes of the left and right cameras, the area with the road paint may be mistakenly detected as a three-dimensional object.If the area with the road paint is mistakenly detected as a three-dimensional object, safety systems provided by the stereo camera, such as AEB and ACC, may be activated, causing unnecessary alarms or braking, which may cause discomfort to the occupants.

[0005] The conventional method described in Patent Document 1 is intended to correct the relative vertical deviation between the left and right cameras, and corrects the relative vertical deviation that occurs between the left and right cameras based on the parallax value calculated from the horizontal imaging position deviation between the left and right cameras.

[0006] However, the method of Patent Document 1 uses the calculated parallax value as is to correct the vertical misalignment between the left and right cameras, so there is a concern that erroneous correction may occur if an incorrect parallax value is calculated.If an image of a road surface painted with repeated diagonal lines in the depth direction, such as a flow guidance strip, is captured while the vertical misalignment has been erroneously corrected, there is a risk that the area with the road paint may be erroneously detected as a three-dimensional object.

[0007] If the area containing road paint is mistakenly detected as a three-dimensional object, safety systems provided by the stereo camera, such as AEB and ACC, may be activated, causing unnecessary alarms or braking, which may cause discomfort to the occupants. Therefore, a mechanism is required to determine whether the calculated disparity value is correct before using it. An object of the present invention is to provide an image processing device that determines whether a disparity calculation result is correct or not based on an image captured by an on-board camera. [Means for solving the problem]

[0008] The image processing device of the present invention that solves the above problems comprises: An image processing device that processes images captured by a first imaging unit and a second imaging unit that are arranged apart from each other in a horizontal direction and that capture overlapping areas, an image storage unit that stores a first image captured by the first imaging unit and a second image captured by the second imaging unit at the same time as the first image; a shifted image generating unit that generates a plurality of third images by shifting pixels of the first image in a vertical direction by a predetermined amount; a parallax calculation unit that calculates each parallax between the second image and each of the plurality of third images; a storage unit that stores an expected parallax value of a road surface, which is a parallax that should be obtained from an image of a flat road surface when the first imaging unit and the second imaging unit are installed at a predetermined position of a vehicle in a predetermined attitude state and the flat road surface ahead is imaged; a validity determination unit that compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface and determines whether each parallax is valid; an optical axis deviation detection unit that detects a deviation amount of the optical axes of the first imaging unit and the second imaging unit in the up-down direction by using the parallax determined to be valid from among the parallaxes; The present invention is characterized by comprising: [Effects of the Invention]

[0009] According to the present invention, by determining whether the parallax calculated from the images captured by the pair of image capturing units is correct, it is possible to prevent erroneous correction of the optical axis of the image capturing device.

[0010] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a functional block diagram illustrating an overall configuration of an image capturing apparatus according to an embodiment of the present invention to which an image processing apparatus of the present invention is applied; [Figure 2] 5 is a flowchart for explaining a parallax accuracy determination of the imaging device of the present embodiment. [Figure 3] 10A and 10B are diagrams illustrating simulated left and right captured images and a parallax image. [Figure 4] 10A and 10B are diagrams showing the difference between distant parallax and near parallax in left and right captured images; [Figure 5] FIG. 10 is a diagram illustrating a method for calculating parallax when a repeating pattern is captured by a stereo camera in which the optical axes of the left and right cameras are relatively offset in the vertical direction. [Figure 6] 10 is a graph showing the relationship between the vertical shift amount of the third image and the effective parallax number. [Figure 7] 10A and 10B are diagrams illustrating the relationship between a parallax image of a road surface and an expected parallax value. DETAILED DESCRIPTION OF THE INVENTION

[0012] Next, one embodiment of the present invention will be described. The imaging device of this embodiment to which the image processing device of the present invention is applied is an imaging device that is mounted on a vehicle and captures images of the outside world, and is configured by a stereo camera.

[0013] A stereo camera includes a pair of cameras positioned to capture an image of the front view, for example, through the windshield, and an image processing device that processes the images captured by the pair of cameras. The pair of cameras are installed in a predetermined position on the vehicle in a predetermined orientation, and are positioned horizontally apart from each other, capturing images of overlapping areas in front of the vehicle.

[0014] The image processing device is composed of an ECU (Electronic Control Unit) having a CPU (Central Processing Unit), memory, and software program. The image processing device, for example, calculates parallax based on the left and right captured images, determines whether the parallax is correct, and performs optical axis deviation detection processing using only the correct parallax. Then, using the amount of optical axis deviation, it performs correction processing to correct the relative vertical deviation between the optical axes of the pair of cameras.

[0015] First, a method for calculating parallax in a stereo camera and the reason why a stereo camera may calculate an erroneous parallax will be described with reference to FIGS.

[0016] Fig. 3 is a diagram showing simulated left and right captured images and parallax images. Fig. 3(a) is a schematic diagram of a captured image 301 captured by the left camera of the vehicle looking ahead, and Fig. 3(b) is a schematic diagram of a captured image 302 captured by the right camera of the vehicle looking ahead at the same time as the left camera. The left and right captured images 301 and 302 show a road surface 312 and a preceding vehicle 311 traveling on the road surface 312.

[0017] Figure 3(c) is a disparity image 303 created using the captured image 301 in Figure 3(a) and the captured image 302 in Figure 3(b), and shows an area 321 close to the vehicle, an area 323 far from the vehicle, and an intermediate area 322. From the relationship between distance and disparity as shown in Figure 3(c), the stereo camera can calculate the distance to an object based on the disparity value.

[0018] The stereo camera detects a three-dimensional object, such as a preceding vehicle 311, from a disparity image 303 as shown in Figure 3(c). When there is no three-dimensional object, the disparity indicates the distance on the road surface and gradually decreases as you move from near to far, and in the disparity image, it is also shown to gradually decrease as you move vertically from bottom to top. On the other hand, when there is a three-dimensional object, the disparity is the same for near and far objects, and is shown to be the same value along the vertical direction in the disparity image. Therefore, it is determined that a three-dimensional object is present in an area in the disparity image where the disparity is the same along the vertical direction.

[0019] FIG. 4 is a diagram showing the difference between the distant parallax and the near parallax in the left and right captured images. Two preceding vehicles 411 and 412 are captured in both the image 401 captured by the right camera and the image 402 captured by the left camera. Due to the characteristics of the stereo camera, the parallax for the preceding vehicle 411, which is close to the host vehicle, is large, and the parallax for the preceding vehicle 412, which is far from the host vehicle, is small, as shown in FIG.

[0020] Figure 5 shows the disparity when a three-dimensional object with a repeating pattern, such as a fence, is present in front of the vehicle. Figure 5(a) is a diagram showing a simulation of the method of calculating disparity when a repeating pattern is captured by a stereo camera in which the optical axes of the left and right cameras are not misaligned relative to each other in the vertical direction, and in which no vertical optical axis misalignment has occurred. Figure 5(b) is a diagram showing a simulation of the method of calculating disparity when a repeating pattern is captured by a stereo camera in which the optical axes of the left and right cameras are misaligned relative to each other in the vertical direction, and in which vertical optical axis misalignment has occurred.

[0021] For example, when calculating the parallax for a repeating pattern such as a fence with diagonally intersecting wire mesh as shown in Figure 5, the parallax of point 511 in left image 501 shown in Figure 5(a) is to be matched with point 512 in right image 502. In other words, point 512 in right image 502 that corresponds to point 511 in left image 501 is searched for. If there is a relative vertical positional shift δ between the left and right cameras of the stereo camera, the result will be as shown in Figure 5(b). Figure 5(b) is a diagram showing a state in which left image 503 is shifted upward by δ with respect to right image 504.

[0022] In this case, point 513 on the left image 503 should match point 514 on the right image, but because the left and right images are misaligned in the vertical direction, matching is not possible and it is impossible to calculate the parallax. However, in the case of a repeating pattern such as a fence, a matching event can occur even when the point is slightly misaligned from its original position, as in the case of point 515 on the right image. In this case, the calculated parallax value includes an error equal to the difference in the horizontal positions of points 514 and 515.

[0023] In the case of the method shown in Patent Document 1, if optical axis correction is performed when there is a repeating pattern such as a fence, barrier, or zebra stripe marked on the road ahead of the vehicle, there is a risk that a disparity value containing an error as described above will be calculated, and optical axis correction will be performed based on the disparity value containing the error.

[0024] To address this issue, the imaging device of this embodiment is characterized by performing a process to determine whether the parallax information used in the correction process is correct when correcting vertical misalignment that occurs between the left and right cameras.According to the imaging device of this embodiment, it is possible to perform vertical misalignment correction using only the correct parallax value from among multiple parallax values.

[0025] FIG. 1 is a functional block diagram showing the overall configuration of an image capturing apparatus according to this embodiment to which the image processing apparatus of the present invention is applied. 1, the imaging device 100, which is a stereo camera, is generally composed of a pair of imaging units 11 and 12, an image storage unit 13, a movement image generation unit 14, a parallax calculation unit 15, a memory unit 16, a validity determination unit 17, and an optical axis deviation detection unit 18. The pair of imaging units 11 and 12 are composed of left and right cameras, and the image storage unit 13, the movement image generation unit 14, the parallax calculation unit 15, the memory unit 16, the validity determination unit 17, and the optical axis deviation detection unit 18 are configured as internal functions of the image processing device 101.

[0026] The pair of imaging units 11, 12 (first imaging unit 11, second imaging unit 12) are used to capture an image in the direction of an imaging target in the imaging device 100 and acquire an image, and are arranged so that their optical axes and imaging directions face the same direction and are at the same height on the left and right sides of the imaging device 100. The first imaging unit 11 and the second imaging unit 12 are, for example, cameras having a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor and a lens.

[0027] The first imaging unit 11 and the second imaging unit 12 capture images simultaneously and periodically to obtain first and second images, respectively. The image storage unit 13 cuts out and stores images of predetermined overlapping regions from the first and second images captured simultaneously by the first imaging unit 11 and the second imaging unit 12. The shifted image generation unit 14 generates multiple third images by shifting the first images stored in the image storage unit 13 by a fixed amount in the up-down direction (vertical direction).

[0028] The parallax calculation unit 15 calculates the parallax for each third image from the second image and the multiple third images generated by the movement image generation unit. A commonly known method can be used to calculate the parallax. For example, using the second image as a reference, a pixel row in one third image at the same vertical height as the second image is scanned horizontally (horizontally) to find a matching point with the second image, and the horizontal displacement between the second and third images is calculated as the parallax, which is known as stereo matching. This is performed for all vertical height positions in a specified area. Then, the same process is performed for all remaining multiple third images.

[0029] The memory unit 16 stores the expected parallax value of the road surface. The expected parallax value of the road surface is the parallax that should be obtained from the captured image of the flat road surface when the first imaging unit 11 and the second imaging unit 12 are installed at a predetermined position of the vehicle in a predetermined attitude and the flat road surface ahead is imaged, and can be calculated geometrically. The memory unit 16 stores the expected parallax value calculated for each horizontal line from the first image or the second image captured by the first imaging unit 11 or the second imaging unit 12. The expected parallax value of the road surface is acquired each time the parallax between the second image and the third image is calculated. The expected parallax value for each horizontal line can be calculated from the height at which the imaging device 100 is installed and the width (baseline length) between the first imaging unit and the second imaging unit.

[0030] FIG. 7 is a diagram illustrating the relationship between the parallax image of the road surface and the parallax expectation value. 7 shows a disparity image 701 of a flat road surface 702 extending forward from the host vehicle, and disparity expectations d1-d3 corresponding to the disparity image 701. As shown in FIG. 7, the disparity expectations d1-d3 of the road surface 702 gradually decrease as the distance from the host vehicle increases, that is, as the distance increases upward along the Y-axis direction in the disparity image 701 (disparity expectations d1 of a region 711 near the host vehicle > disparity expectations d2 of a region 712 further away than the region 711 > disparity expectations d3 of a region 713 further away than the region 712).

[0031] The validity determination unit 17 compares each parallax calculated by the parallax calculation unit 15 with the expected value of the parallax calculated by the storage unit 16 to determine whether each parallax calculated by the parallax calculation unit 15 is valid. Of the parallaxes calculated by the parallax calculation unit 15, parallaxes (small parallaxes) indicating distances farther than the expected value of the parallax calculated by the storage unit 16 are parallaxes that should not appear when capturing an image of the road surface, and are therefore determined to be erroneous parallaxes and invalid parallaxes. Furthermore, of the parallaxes calculated by the parallax calculation unit 15, parallaxes (large parallaxes) indicating nearby distances equal to or less than the expected value of the parallax calculated by the storage unit 16 are determined to be correct parallaxes and valid parallaxes.

[0032] The comparison between the parallax calculated by the parallax calculation unit 15 and the parallax expected value calculated by the storage unit 16 is performed for each horizontal line. The parallax value calculated by the parallax calculation unit 15 for each horizontal line may be the average value or the median value for each horizontal line.

[0033] The optical axis shift detection unit 18 calculates the number of parallaxes (number of effective parallaxes) of the parallax values (effective parallaxes) determined to be valid by the validity determination unit 17. Here, the number of parallaxes of the effective parallaxes calculated from the second image calculated by the parallax calculation unit 15 and each of the plurality of third images is calculated. Then, the optical axis shift amount δ is detected based on the relationship between the amount of vertical shift of each third image and the number of parallaxes of the effective parallax.

[0034] FIG. 6 is a graph showing the relationship between the vertical shift amount of the third image and the effective number of parallaxes. In the graph of Figure 6, point 602 indicates the number of effective parallaxes between the first image and the second image, points 601a to 601d indicate the number of effective parallaxes between the second image and a third image obtained by shifting the first image upward by a predetermined number of pixels, and points 601e to 601i indicate the number of effective parallaxes between the second image and a third image obtained by shifting the first image downward by a predetermined number of pixels.

[0035] The technique of Patent Document 1 can be applied to a method for detecting the amount of optical axis shift from the amount of shift and the number of parallaxes. Specifically, as shown in FIG. 6, a distribution of the amount of vertical shift (vertical shift) from the first image in each third image and the number of effective parallaxes is created, and an approximation curve Ca is calculated from the distribution. The approximation curve Ca of the number of effective parallaxes is calculated using, for example, a least squares method. Then, the relative amount of optical axis shift δ in the up-down direction of the second imaging unit 12 with respect to the first imaging unit 11 is calculated from this approximation curve Ca.

[0036] Here, the optical axis shift amount δ is calculated by utilizing the fact that the number of parallaxes of a parallax image obtained from two images captured by a pair of image capture units increases or decreases depending on the amount of vertical shift between the optical axes of the pair of image capture units. That is, the number of parallaxes of a parallax image is maximized when there is no vertical shift between the optical axes of the pair of image capture units (minimum case), and tends to decrease as the amount of vertical shift between the optical axes increases. Since the number of parallaxes of a parallax image between the third image and the first image changes depending on the amount of vertical shift from the first image, and the number of parallaxes of a parallax image is maximized when there is no vertical shift between the optical axes (minimum case), the vertical shift amount δ of the optical axes of the pair of image capture units is obtained from the difference between the origin position 602 (i.e., when the third image is the same as the first image) in the horizontal direction (i.e., the amount of vertical shift between the third image) and the amount of shift at which the number of parallaxes is maximized (the peak value of the approximation curve Ca).

[0037] The image processing device 101 of the imaging device 100 uses the amount of optical axis shift δ detected by the optical axis shift detection unit 18 to perform vertical shift correction to align the vertical positions of the optical axis of the first imaging unit 11 and the optical axis of the second imaging unit 12. The amount of optical axis shift δ detected by the optical axis shift detection unit 18 is detected using only the effective parallax, so the image processing device 101 of the imaging device 100 can suppress erroneous correction when performing vertical shift correction.

[0038] FIG. 2 is a flowchart illustrating the contents of the determination of the parallax validity of the imaging device in this embodiment. In step S201, image acquisition is performed. Here, a pair of captured images (first image and second image) captured at the same time by the first imaging unit 11 and the second imaging unit 12 are acquired. In step S202, an image obtained by vertically shifting the first image is acquired as a third image. For example, an image obtained by shifting the first image relative to the second image by each vertical coordinate or by each divided section is acquired as the third image. Then, in step S203, parallax is calculated from the second image and the third image by stereo matching. The area for which parallax is calculated may be the entire image, or only an arbitrary area.

[0039] In step S204, the expected parallax value of the road surface is calculated from one of the first, second, or third image. The expected parallax value of the road surface is calculated from the same area as the area in which the parallax was calculated in step S203, and can be obtained for each horizontal line of the image. In this embodiment, the expected parallax value of the road surface is obtained each time the parallax between the second and third images is calculated, but this can also be achieved by having a disparity expected value table for each horizontal line in advance.

[0040] In step S205, the parallax calculated in step S203 is compared with the expected parallax calculated in step S204. The parallax calculated in step S203 can be compared with the expected parallax in units of one pixel in the horizontal direction for each horizontal line. Taking processing time into consideration, comparison may be made in units of multiple pixels. In step S205, it is determined whether the parallax calculated in step S203 is equal to or greater than the expected parallax calculated in step S204.

[0041] Here, if the parallax calculated in step S203 is equal to or greater than the expected parallax calculated in step S204, more specifically, if the parallax calculated in step S203 is greater than the expected parallax calculated in step S204 by at least one predetermined number (parallax indicating the distance of the vicinity) (YES in step S205), it is determined to be a valid parallax in step S206.

[0042] On the other hand, if the disparity calculated in step S203 is smaller than the expected disparity value calculated in step S204, more specifically, if the disparity calculated in step S203 is smaller than the expected disparity value calculated in step S204 by at least one predetermined number (a disparity indicating a far distance) (NO in step S205), it is determined to be an invalid disparity in step S207, and this invalid disparity is not used in subsequent processing.

[0043] In step S208, the number of parallaxes determined to be valid in step S205 is counted.

[0044] In step S209, it is determined whether multiple third images have been generated. If the number of third images is less than a predetermined number of two or more, it is determined that multiple third images have not been generated (NO in step S209), and the process returns to step S202. Then, in steps S202 and S203, the vertical shift amount of the first image is changed to generate another third image with a different shift amount, and the parallax between the generated third image and the second image is calculated.

[0045] Then, in step S209, a process of calculating the parallax between the second image and the third image with the shift amount changed is performed until it is determined that there are a specified number or more of third images (YES in step S209). The shift amount can be, for example, a shift of one pixel or a shift of multiple pixels. Furthermore, the direction in which the first image is shifted may be, for example, at least one of upward and downward. In this embodiment, the first image is shifted both upward and downward relative to the second image to generate multiple third images, and the parallax for each third image is calculated.

[0046] In step S210, a distribution of the vertical shift amount and the number of effective parallaxes is created, as shown in FIG.

[0047] In step S211, an approximation curve Ca is obtained from the distribution obtained in step S210, and the relative vertical optical axis shift amount δ is obtained. The position on the approximation curve Ca where the number of effective parallaxes is the largest is the position where there is no vertical shift between the first image and the third image. The optical axis shift amount δ obtained in step S211 is obtained from only the effective parallaxes remaining after removing invalid parallaxes from the multiple parallaxes. Therefore, by using this optical axis shift amount δ to correct the optical axis shift between the first image capturing unit 11 and the second image capturing unit 12, the relative vertical shift that has occurred between the first image capturing unit 11 and the second image capturing unit 12 can be accurately corrected.

[0048] If the number of effective parallaxes required to calculate the amount of optical axis deviation cannot be obtained from the distribution in step S210, the optical axis deviation detection process in step S211 is not performed until a captured image that satisfies a predetermined condition can be acquired. After the predetermined condition is satisfied, the optical axis deviation detection in step S211 is performed, and the process ends.

[0049] The image processing device 101 included in the imaging device 100 of this embodiment saves a first image captured by the first imaging unit 11 and a second image captured by the second imaging unit 12, generates multiple third images by shifting the pixels of the first image by a predetermined amount in the vertical direction, and calculates each parallax from the second image and each of the multiple third images.The image processing device 101 then compares each parallax with an expected parallax value of the road surface, which is the parallax that should be obtained from an image of a flat road surface ahead of the vehicle, to determine whether each parallax is valid, and detects the amount of misalignment of the optical axes of the first imaging unit 11 and the second imaging unit 12 in the vertical direction (up and down direction) using only the parallaxes determined to be valid.

[0050] According to the image processing device 101 of this embodiment, when correcting the vertical misalignment of the optical axes that occurs between the left and right cameras, it is possible to suppress erroneous correction of the optical axis misalignment that uses an erroneous parallax by determining whether the parallax information used in the correction process is correct. Therefore, for example, when an image of a road surface on which repeated diagonal lines in the depth direction, such as a zebra strip, are painted is captured, it is possible to suppress erroneous detection of the area on which the road paint is painted as a three-dimensional object, and to suppress the malfunction of safety systems provided by the stereo camera, such as AEB and ACC, which may cause unnecessary alarms or braking, giving discomfort to the occupants.

[0051] In the above embodiment, the image processing device 101 is configured as a stereo camera integral with the first imaging unit 11 and the second imaging unit 12. However, the configuration of the image processing device of the present invention is not limited to this embodiment. For example, the image processing device 101 may be provided separately from the pair of cameras.

[0052] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various design modifications can be made without departing from the spirit of the present invention as defined in the claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0053] 100 imaging device, 101 image processing device, 11 first imaging unit, 12 second imaging unit, 13 image storage unit, 14 movement image generation unit, 15 parallax calculation unit, 16 storage unit, 17 validity determination unit, 18 optical axis deviation detection unit

Claims

1. An image processing device that processes images captured by a first imaging unit and a second imaging unit that are arranged apart from each other in a horizontal direction and that capture overlapping areas, an image storage unit that stores a first image captured by the first imaging unit and a second image captured by the second imaging unit at the same time as the first image; a shifted image generating unit that generates a plurality of third images by shifting pixels of the first image in a vertical direction by a predetermined amount; a parallax calculation unit that calculates each parallax between the second image and each of the plurality of third images; a storage unit that stores an expected parallax value of a road surface, which is a parallax that should be obtained from an image of a flat road surface when the first imaging unit and the second imaging unit are installed at a predetermined position of a vehicle in a predetermined attitude state and the flat road surface ahead is imaged; a validity determination unit that compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface and determines whether each parallax is valid; an optical axis deviation detection unit that detects a deviation amount of the optical axes of the first imaging unit and the second imaging unit in the up-down direction by using the parallax determined to be valid from among the parallaxes; Equipped with the moving image generating unit generates the third image by shifting a relative vertical positional relationship between the first image and the second image for each vertical coordinate or for each divided section; the storage unit calculates and stores the expected parallax value for each vertical coordinate of the first image, the second image, or the third image; the storage unit stores the expected parallax value of the road surface as a value for each horizontal line of the captured image of the flat road surface; the validity determination unit compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface for each horizontal line; the validity determination unit determines, among the parallaxes calculated by the parallax calculation unit, a parallax that indicates a distance farther than an expected parallax value of the road surface as an invalid parallax, and determines a parallax that indicates a nearby distance equal to or less than the expected parallax value of the road surface as a valid parallax; the optical axis deviation detection unit calculates the number of parallaxes of the effective parallax determined to be valid by the validity determination unit, and detects the amount of optical axis deviation based on the relationship between the amount of vertical shift of each third image and the number of parallaxes of the effective parallax; When the number of parallaxes determined to be valid or the distribution of the number of parallaxes determined to be valid does not satisfy a predetermined condition, the optical axis deviation detection unit does not detect the amount of optical axis deviation until the predetermined condition is satisfied.

2. An imaging device mounted on a vehicle to capture an image of the outside world, a first imaging unit and a second imaging unit that are arranged apart from each other in a horizontal direction and capture images of overlapping areas; an image storage unit that stores a first image captured by the first imaging unit and a second image captured by the second imaging unit at the same time as the first image; a shifted image generating unit that generates a plurality of third images by shifting pixels of the first image in a vertical direction by a predetermined amount; a parallax calculation unit that calculates each parallax between the second image and each of the plurality of third images; a storage unit that stores an expected parallax value of a road surface, which is a parallax that should be obtained from an image of a flat road surface when the first imaging unit and the second imaging unit are installed at a predetermined position of the vehicle in a predetermined attitude state and the flat road surface ahead is imaged; a validity determination unit that compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface and determines whether each parallax is valid; an optical axis deviation detection unit that detects a deviation amount of the optical axes of the first imaging unit and the second imaging unit in the up-down direction by using the parallax determined to be valid from among the parallaxes; Equipped with the moving image generating unit generates the third image by shifting a relative vertical positional relationship between the first image and the second image for each vertical coordinate or for each divided section; the storage unit calculates and stores the expected parallax value for each vertical coordinate of the first image, the second image, or the third image; the storage unit stores the expected parallax value of the road surface as a value for each horizontal line of the captured image of the flat road surface; the validity determination unit compares each parallax calculated by the parallax calculation unit with an expected parallax value of the road surface for each horizontal line; the validity determination unit determines, among the parallaxes calculated by the parallax calculation unit, a parallax that indicates a distance farther than an expected parallax value of the road surface as an invalid parallax, and determines a parallax that indicates a nearby distance equal to or less than the expected parallax value of the road surface as a valid parallax; the optical axis deviation detection unit calculates the number of parallaxes of the effective parallax determined to be valid by the validity determination unit, and detects the amount of optical axis deviation based on the relationship between the amount of vertical shift of each third image and the number of parallaxes of the effective parallax; When the number of parallaxes determined to be valid or the distribution of the number of parallaxes determined to be valid does not satisfy a predetermined condition, the optical axis deviation detection unit does not detect the amount of optical axis deviation until the predetermined condition is satisfied. An imaging device characterized by:

Images