Image processing device and image processing method

Abstract

The purpose of the present invention is to accurately calculate and correct a parallax error caused by camera component deviation generated during traveling of a vehicle due to deterioration over time or temperature change. This image processing device comprises: a pixel deviation calculation unit that calculates a vertical pixel deviation between a set of images captured by a set of cameras; a component deviation estimation unit that, on the basis of the calculated vertical pixel deviation, estimates a pixel deviation between images that is caused by the component deviation; and a correction value determination unit that determines a correction value for the parallax error from the pixel deviation caused by the estimated component deviation. The component deviation estimation unit estimates pixel deviation caused by at least one component deviation from among: the angular deviation in the roll direction of one camera with respect to another camera; the angular deviation in the pitch direction of one camera with respect to another camera; and the positional deviation in the vertical direction of an imaging element with respect to an optical element.

Description

Image processing device and image processing method

[0001] The present invention relates to an image processing device and an image processing method.

[0002] A stereo camera is a device that uses multiple cameras to mimic human binocular vision and accurately measure the distance between an object and the cameras. The principle of stereo camera ranging is that a pair of cameras at a predetermined distance simultaneously capture images of the same object, and the distance between the object and the cameras is measured by calculating the difference between corresponding points in the pair of images based on the principle of triangulation. The distance between the pair of cameras is called the baseline length, and the difference between corresponding points in the pair of images is called parallax.

[0003] With the development of computer vision and artificial intelligence technology, stereo cameras have been widely used in automotive technology for various applications, such as automatic parking, adaptive cruise control (ACC), lane departure warning, traffic sign recognition, etc. For example, ACC systems use in-vehicle stereo camera technology to measure the distance to a vehicle ahead and the speed of the vehicle ahead in real time, and control the distance between vehicles to ensure safe driving.

[0004] Compared to other sensors such as lidar or millimeter-wave radar, stereo cameras have advantages such as versatility, high resolution, and low cost. The advantage of versatility is that stereo cameras can acquire detailed information, such as the color or shape of an object, by capturing image information, enabling more functions to be realized using image recognition. On the other hand, these other sensors are primarily used for distance measurement and cannot measure color. The advantage of high resolution is that they can acquire a larger number of pixels than the scanning methods used by these other sensors, thereby enabling highly accurate image information to be acquired. Therefore, stereo cameras have become a very important technology in automotive technology, and can provide highly accurate and reliable data for ADAS (Advanced Driver Assistance Systems) or autonomous driving, thereby improving the safety and sophistication of automobiles.

[0005] On the other hand, since many traffic accidents occur at intersections, in order to reduce traffic accidents, stereo cameras need to improve their recognition performance for pedestrians and bicycles at intersections. Conventional stereo cameras are primarily used to acquire distance information to assist in collision avoidance with forward objects, so they require a long-distance recognition angle (a field of view capable of image recognition, hereinafter also referred to as the "field of view"). A wider field of view is required for collision avoidance at intersections. To achieve both a long-distance and wide field of view, a fisheye lens, for example, can be used. However, fisheye lenses exhibit significant image distortion in the wide-angle range. This image distortion is usually corrected by geometric calibration during manufacturing. Furthermore, deviations in the relative position or orientation between a pair of cameras are corrected by this geometric calibration.

[0006] However, even if a camera is perfectly corrected during manufacturing, "component misalignment" may occur during user use due to aging or temperature changes. Component misalignment refers to misalignment in the position or orientation of components constituting a stereo camera. Component misalignment includes misalignment in the position or orientation between the imaging element and optical element in one camera, as well as misalignment in the position or orientation between a pair of cameras. Such component misalignment can lead to inaccurate image correction data used during manufacturing to correct image distortion, resulting in "pixel misalignment" between a pair of images. Stereo cameras typically calculate distance by calculating parallax horizontally. Therefore, pixel misalignment in the horizontal direction corresponds to a parallax error and causes ranging errors. Pixel misalignment in the horizontal direction is also called "horizontal pixel misalignment," and pixel misalignment in the vertical direction is also called "vertical pixel misalignment."

[0007] Because image distortion is small in narrow-angle areas with a small field of view, parallax error in narrow-angle areas can be corrected, for example, by using a constant value obtained from the parallax at infinity. Because image distortion is large in wide-angle areas, parallax error in wide-angle areas cannot be adequately corrected using the same method as parallax error in narrow-angle areas. Insufficient correction of parallax error can result in large ranging errors, potentially affecting safety. Therefore, it is important to determine how to calculate and correct parallax error across the entire field of view. Typically, this parallax error is corrected by periodic recalibration at a repair shop or dealer. However, due to aging or temperature changes, the effects of parallax error may appear in stereo cameras while the vehicle is in motion, potentially preventing accidents. Therefore, a function that automatically corrects parallax error while the vehicle is in motion is needed.

[0008] A known technique for correcting parallax errors due to component misalignment is disclosed in Patent Document 1. Patent Document 1 states that "relative vertical misalignment of images captured by left and right cameras is detected, the relative tilt angle of the imaging surfaces of the left and right cameras is calculated from the detected vertical misalignment, and the relative vertical and horizontal misalignment of the images captured by the left and right cameras is corrected according to the calculated tilt angle."

[0009] JP 2014-010100 A

[0010] Since the parallax changes with the distance to the object, the parallax error also changes with the distance to the object, making it difficult to calculate the parallax error directly. On the other hand, the vertical pixel shift between a pair of images is usually zero when the stereo camera is perfectly calibrated. However, if the stereo camera experiences component shift due to aging or temperature changes, vertical pixel shift occurs. Furthermore, the vertical pixel shift does not change with the distance to the object. Therefore, using the vertical pixel shift to calculate the parallax error due to component shift is an effective method.

[0011] In Patent Document 1, the relative vertical misalignment of images captured by left and right cameras is detected, the amount of misalignment between the sensor and lens in the camera is calculated, and the parallax error is corrected. However, in actual situations, component misalignment is not limited to the misalignment between the sensor and lens, but includes a mixture of misalignments between various types of components. Therefore, with the technology described in Patent Document 1, it is difficult to accurately calculate and correct the parallax error caused by various types of component misalignment that occur while the vehicle is traveling.

[0012] The present invention has been made in view of the above, and has as its object to accurately calculate and correct parallax errors caused by misalignment of camera components that occurs while a vehicle is running due to aging or temperature changes.

[0013] In order to solve the above problem, the image processing device of the present invention includes a pixel shift calculation unit that calculates a vertical pixel shift between a pair of images captured by a pair of cameras, each of which includes an optical element and an image sensor; a component shift estimation unit that estimates a pixel shift between the images caused by a component shift, which is a shift in position or orientation of at least one of the cameras, the optical element, and the image sensor, based on the calculated vertical pixel shift; and a correction value determination unit that calculates a correction value for a horizontal pixel shift between the images caused by the component shift, from the estimated pixel shift caused by the component shift, and determines the calculated correction value as a correction value for a parallax error between the images, wherein the component shift estimation unit estimates, as the pixel shift caused by the component shift, at least one of the pixel shift caused by an angular shift in the roll direction of one camera relative to the other camera, the pixel shift caused by an angular shift in the pitch direction of the other camera relative to the one camera, and the pixel shift caused by a vertical positional shift of the image sensor relative to the optical element.

[0014] In the present invention, "horizontal" means a direction along the baseline of a pair of cameras, and "vertical" means a direction perpendicular to the baseline of a pair of cameras and to the front-to-back direction. In other words, the horizontal and vertical in the present invention are relative concepts.

[0015] According to the present invention, it is possible to accurately calculate and correct the parallax error caused by the misalignment of camera components that occurs while the vehicle is running due to aging or temperature changes. Other problems, configurations, and effects will become clear from the description of the following embodiments.

[0016] FIG. 1 is a diagram explaining pixel misalignment when a horizontal sensor shift occurs as an example of component misalignment. FIG. 2 is a diagram explaining pixel misalignment after correcting the distortion of the image shown in FIG. 1. FIG. 3 is a diagram explaining seven types of component misalignment. FIG. 4 is a diagram showing the configuration of a camera system according to this embodiment. FIG. 5 is a diagram explaining the processing flow of the image processing device shown in FIG. 4. FIG. 6 is a diagram explaining a method for estimating component misalignment related to a first group of factors. FIG. 7 is a diagram explaining a method for estimating component misalignment related to a second group of factors. FIG. 8 is a diagram showing verification results by simulation.

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments and examples are merely illustrative of the present invention, and may be omitted or simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The position, size, shape, or range of each component shown in the drawings may not represent the actual position, size, shape, or range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, or range, etc., disclosed in the drawings. Note that components or functions denoted by the same reference numerals in each embodiment have the same configuration or function in each embodiment, and their description will be omitted unless otherwise specified. However, when there are multiple components having the same or similar functions, they may be described using the same reference numerals with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description.

[0018] FIG. 1 is a diagram illustrating pixel misalignment when a sensor horizontal shift occurs as an example of component misalignment.

[0019] The camera system 100 including the cameras 31 and 32 may experience "component misalignment" due to aging or temperature changes. The component misalignment is a misalignment in the position or orientation of the cameras 31 and 32 included in the camera system 100, or a misalignment in the position or orientation of the components that make up the cameras 31 and 32. The sensor horizontal shift is a misalignment in the horizontal direction of the image sensor 33 relative to the optical element 34, which is a component of the cameras 31 and 32. The optical element 34 forms an image of the object on the imaging surface of the image sensor 33.

[0020] In FIG. 1 , pixel points 10, 12, 14, and 16 each represent a pixel point before the horizontal sensor shift occurs. The positions of pixel points 10, 12, 14, and 16 can be expressed by a horizontal angle of view (hereinafter also referred to as the "horizontal angle of view") and a vertical angle of view (hereinafter also referred to as the "vertical angle of view"). The horizontal angle of view and the vertical angle of view are expressed as angles of displacement in the horizontal direction from an origin (horizontal angle of view 0°, vertical angle of view 0°) that is a pixel point intersecting the optical axis of the optical element 34 (a pixel point corresponding to the optical center of the optical element 34). The vertical angle of view is expressed as an angle of displacement in the vertical direction from the origin.

[0021] Pixel point 10 is a pixel point whose horizontal and vertical angles of view both coincide with the optical axis, i.e., a pixel point located at the origin. Pixel point 12 is a pixel point whose vertical angle of view coincides with the optical axis and whose horizontal angle of view is in a wide-angle region (region with a large angle of view). Pixel point 14 is a pixel point whose horizontal angle of view coincides with the optical axis and whose vertical angle of view is in a wide-angle region. Pixel point 16 is a pixel point whose horizontal and vertical angles of view both lie in the wide-angle region.

[0022] The first region is a region where the vertical angle of view is within approximately 1° from the optical axis. The second region is a region where the vertical angle of view is larger than that of the first region. In this embodiment, the second region is a region where the vertical angle of view is located in the wide-angle region. In the example of FIG. 1 , the region where the vertical angle of view coincides with the optical axis (region where the vertical angle of view is 0°) is shown as the first region. In the example of FIG. 1 , the region where the vertical angle of view is 14° is shown as the second region.

[0023] In Figure 1, pixel points 11, 13, 15, and 17 indicate positions to which pixel points 10, 12, 14, and 16 have been displaced due to horizontal sensor shift, respectively. SO0, SO1, SO2, and SO3 indicate the respective pixel shifts. Pixel shift refers to the shift in the positions of corresponding pixel points between a pair of images captured by a pair of cameras 31 and 32. In this embodiment, pixel shift in the horizontal direction is also referred to as "horizontal pixel shift," and pixel shift in the vertical direction is also referred to as "vertical pixel shift." Horizontal pixel shift corresponds to parallax error.

[0024] Since Figure 1 shows an image before distortion correction, SO0, SO1, SO2, and SO3 all have horizontal pixel shifts. The pixel shift amounts for all pixel points 10, 12, 14, and 16 are the same. Also, since only a horizontal sensor shift has occurred in Figure 1, horizontal pixel shifts occur, but vertical pixel shifts do not. In other words, with an image before distortion correction, it is not possible to estimate horizontal component shifts using vertical pixel shifts.

[0025] FIG. 2 is a diagram illustrating pixel shifts after the distortion of the image shown in FIG. 1 has been corrected.

[0026] In Figure 2, pixel points 20, 22, 24, and 26 correspond to pixel points 10, 12, 14, and 16 in Figure 1, respectively. That is, pixel points 20, 22, 24, and 26 indicate pixel points before the horizontal sensor shift occurs. In Figure 2, pixel points 21, 23, 25, and 27 indicate positions to which pixel points 20, 22, 24, and 26 have been displaced by the horizontal sensor shift, respectively. That is, pixel points 21, 23, 25, and 27 correspond to pixel points 11, 13, 15, and 17 in Figure 1, respectively.

[0027] 2 corresponds to pixel shift SO0 of pixel point 10 in FIG. 1, where both the horizontal and vertical angles of view coincide with the optical axis. Near the pixel point where both the horizontal and vertical angles of view coincide with the optical axis, there is almost no image distortion, so the pixel shift amounts of SS0 and SO0 are almost the same.

[0028] On the other hand, SO1 shown in FIG. 1 is a pixel misalignment in a wide-angle region where image distortion is significant, so the pixel misalignment amount of SS1 corresponding to SO1 is greater than that of SO1. This means that parallax error increases in wide-angle regions. SS2 is a pixel misalignment where the horizontal angle of view coincides with the optical axis and the vertical angle of view is in a wide-angle region. SS2 is a pixel misalignment at a position where the horizontal angle of view coincides with the optical axis, so the pixel misalignment amount is the same as that of the corresponding SO2. However, SO3 shown in FIG. 1 has both a horizontal and vertical angle of view in a wide-angle region, so image distortion is significant in both the horizontal and vertical directions, and the corresponding SS3 has large pixel misalignment amounts in both the horizontal and vertical directions. In other words, SS3 has vertical pixel misalignment. In this embodiment, component misalignment is estimated using vertical pixel misalignment in a wide-angle region, and parallax error is estimated and corrected from pixel misalignment in the horizontal direction (parallax error direction). FIG. 2 illustrates the focus of this embodiment.

[0029] 1 and 2, the horizontal sensor shift is described as an example of component misalignment, but in this embodiment, seven types of component misalignment are assumed. FIG. 3 is a diagram for explaining the seven types of component misalignment.

[0030] The seven types of component misalignment shown in FIG. 3 occur due to aging or temperature changes in the camera system 100 and are thought to significantly affect parallax error. The X-axis, Y-axis, and Z-axis in FIG. 3 are coordinate axes along the left-right, up-down, and front-back directions of the pair of cameras 31 and 32, respectively. As an example of component misalignment, the single camera roll rotation shown in FIG. 3( a) refers to the angular misalignment of one camera 31 around the Z-axis (roll direction) relative to the other camera 32. The sensor vertical shift shown in FIG. 3( b) refers to the positional misalignment of the image sensor 33 in the Y-axis direction (up-down direction, i.e., vertical direction) relative to the optical element 34. The single camera pitch rotation shown in FIG. 3( c) refers to the angular misalignment of the other camera 31 around the X-axis (pitch direction) relative to the one camera 32. The sensor horizontal shift shown in FIG. 3( d) refers to the positional misalignment of the image sensor 33 in the X-axis direction (left-right direction, i.e., horizontal direction) relative to the optical element 34. The optical axis change shown in FIG. 3( e) refers to the misalignment of the optical axis of the optical element 34 due to temperature changes, etc. The single camera yaw rotation shown in Fig. 3(f) refers to the angular deviation of one camera 31 around the Y axis (yaw direction) relative to the other camera 32. The sensor yaw rotation shown in Fig. 3(g) refers to the angular deviation of the image sensor 33 around the Y axis (yaw direction) relative to the optical element 34.

[0031] In this embodiment, the single camera roll rotation, sensor vertical shift, and single camera pitch rotation shown in Figures 3(a) to 3(c) are also referred to as "component misalignment related to the first group of factors." In this embodiment, the sensor horizontal shift, optical axis change, single camera yaw rotation, and sensor yaw rotation shown in Figures 3(d) to 3(g) are also referred to as "component misalignment related to the second group of factors." Component misalignment related to the first group of factors is component misalignment that is largely influenced by pixel misalignment in the vertical direction. Component misalignment related to the second group of factors is component misalignment that is largely influenced by pixel misalignment in the horizontal direction.

[0032] Component misalignment other than the seven types shown in Fig. 3, such as sensor pitch rotation or sensor yaw rotation, has little effect on the parallax error and is therefore not shown in Fig. 3. However, the component misalignment of this embodiment also includes component misalignment other than the seven types shown in Fig. 3.

[0033] FIG. 4 is a diagram showing the configuration of the camera system 100 of this embodiment.

[0034] The camera system 100 includes an imaging unit 120 and an image processing device 110. The image processing device 110 includes an external input unit 121, a preprocessing unit 122, a storage unit 123, a pixel shift calculation unit 124, a noise removal unit 125, component shift estimation units 126 and 128, a vertical pixel shift correction unit 127, a correction value determination unit 129, a correction unit 130, a distance measurement unit 131, and a control unit 132. The component shift estimation units 126 and 128 include a first component shift estimation unit 126 and a second component shift estimation unit 128.

[0035] The image processing device 110 includes a processor such as a CPU and a memory. The processor executes a program stored in the memory, thereby realizing the functions of an external input unit 121, a preprocessing unit 122, a pixel shift calculation unit 124, a noise removal unit 125, component shift estimation units 126 and 128, a vertical pixel shift correction unit 127, a correction value determination unit 129, a correction unit 130, a distance measurement unit 131, and a control unit 132.

[0036] The imaging unit 120 is connected to the pre-processing unit 122. The imaging unit 120 is a camera capable of acquiring images such as a visible light camera or an infrared camera. In this embodiment, the imaging unit 120 is configured as a stereo camera including a pair of cameras 31 and 32. However, the imaging unit 120 may be configured as a multi-camera including a set of three or more cameras, or may be configured as a set of cameras. The pair of cameras 31 and 32 simultaneously capture images of the same object to acquire a set of images, which are output to the pre-processing unit 122. The pixel value of each pixel in the acquired image includes a luminance value.

[0037] The external input unit 121 is connected to the storage unit 123. Pre-set image correction information is input to the external input unit 121 from an external device. The image correction information includes image correction data for correcting image distortion through geometric calibration performed during manufacturing and for correcting the effects of refraction of incident light by the vehicle's windshield. Furthermore, the image correction information includes a reference function for estimating pixel misalignment due to component misalignment. The reference function includes a reference function for estimating pixel misalignment due to component misalignment related to a first group of factors (hereinafter also referred to as a "reference function related to the first group of factors") and a reference function for estimating pixel misalignment due to component misalignment related to a second group of factors (hereinafter also referred to as a "reference function related to the second group of factors"). The reference function is calculated in advance through experiments, simulations, or the like. The external input unit 121 outputs the input image correction information to the storage unit 123.

[0038] The preprocessing unit 122 is connected to the imaging unit 120, the storage unit 123, and the pixel shift calculation unit 124. The preprocessing unit 122 performs preprocessing to correct the set of images acquired by the imaging unit 120, using the image correction data stored in the storage unit 123. The preprocessing unit 122 outputs the set of images that have been preprocessed to the pixel shift calculation unit 124.

[0039] The storage unit 123 is connected to the external input unit 121, the preprocessing unit 122, the pixel shift calculation unit 124, the noise removal unit 125, the component shift estimation units 126 and 128, and the correction unit 130. The storage unit 123 is configured with a non-volatile memory and can store each piece of information for a long period of time. The storage unit 123 stores the image correction information output from the external input unit 121. The storage unit 123 outputs image correction data included in the stored image correction information to the preprocessing unit 122. The storage unit 123 stores the vertical pixel shift and parallax output from the pixel shift calculation unit 124. The storage unit 123 outputs the stored vertical pixel shift to the noise removal unit 125. The storage unit 123 outputs the basis function included in the stored image correction information to the component shift estimation units 126 and 128. The storage unit 123 outputs the stored vertical pixel shift and parallax to the correction unit 130.

[0040] The pixel shift calculation unit 124 is connected to the preprocessing unit 122 and the storage unit 123. The pixel shift calculation unit 124 calculates the vertical pixel shift and disparity between a pair of images that have been preprocessed by the preprocessing unit 122. The pixel shift calculation unit 124 calculates the vertical pixel shift and disparity by performing stereo matching processing to search for corresponding points between the images. The pixel shift calculation unit 124 outputs the calculated vertical pixel shift and disparity to the storage unit 123.

[0041] The noise removal unit 125 is connected to the storage unit 123 and the first component shift estimation unit 126. The noise removal unit 125 removes noise from the vertical pixel shift stored in the storage unit 123 and outputs the result to the first component shift estimation unit 126. For example, the noise removal unit 125 acquires the vertical pixel shifts of multiple sets of images corresponding to multiple frames stored in the storage unit 123 and calculates the average value of the vertical pixel shifts. The noise removal unit 125 removes invalid vertical pixel shifts outside the predetermined range using a filter that passes vertical pixel shifts within a predetermined range including the calculated average value. The noise removal unit 125 then calculates the vertical pixel shift fitted to a smooth curve using a technique such as high-dimensional polynomial fitting. The noise removal unit 125 then calculates the vertical pixel shift after noise removal. The noise removal unit 125 outputs the calculated vertical pixel shift after noise removal to the first component shift estimation unit 126.

[0042] The first component misalignment estimation unit 126 is connected to the noise removal unit 125, the storage unit 123, and the vertical pixel misalignment correction unit 127. The first component misalignment estimation unit 126 estimates a pixel misalignment caused by a component misalignment related to the first group of factors (hereinafter also referred to as a "pixel misalignment related to the first group of factors") based on the vertical pixel misalignment output from the noise removal unit 125. That is, the first component misalignment estimation unit 126 estimates, as the pixel misalignment related to the first group of factors, at least one of a pixel misalignment caused by an angular misalignment in the roll direction of the one camera 31 relative to the other camera 32 (single camera roll rotation shown in FIG. 3A), a pixel misalignment caused by an angular misalignment in the pitch direction of the one camera 31 relative to the one camera 32 (single camera pitch rotation shown in FIG. 3C), and a pixel misalignment caused by a vertical positional misalignment of the image sensor 33 relative to the optical element 34 (sensor vertical shift shown in FIG. 3B). Estimating pixel shift related to the first group of factors means estimating which of the component shifts related to the first group of factors the component shift that is causing the parallax error corresponds to, and estimating the pixel shift (including the amount of pixel shift) caused by the corresponding component shift.

[0043] Specifically, the first component shift estimation unit 126 receives the vertical pixel shift output from the noise removal unit 125 and a criterion function related to the first group of factors included in the image correction information stored in the storage unit 123. The first component shift estimation unit 126 estimates the pixel shift related to the first group of factors by comparing the vertical pixel shift output from the noise removal unit 125 with the criterion function related to the first group of factors. The first component shift estimation unit 126 outputs the vertical pixel shift output from the noise removal unit 125, the criterion function related to the first group of factors, and the pixel shift related to the first group of factors to the vertical pixel shift correction unit 127.

[0044] The vertical pixel shift correction unit 127 is connected to the first component shift estimation unit 126 and the second component shift estimation unit 128. The vertical pixel shift correction unit 127 corrects the vertical pixel shift output from the noise removal unit 125 (i.e., the vertical pixel shift calculated by the pixel shift calculation unit 124) based on the pixel shift related to the first group of factors. Specifically, the vertical pixel shift correction unit 127 receives the vertical pixel shift output from the first component shift estimation unit 126, the basis function related to the first group of factors, and the pixel shift related to the first group of factors. The vertical pixel shift correction unit 127 corrects the pixel shift related to the first group of factors out of the received vertical pixel shift. The vertical pixel shift correction unit 127 outputs the corrected vertical pixel shift and the pixel shift related to the first group of factors to the second component shift estimation unit 128.

[0045] The second component misalignment estimation unit 128 is connected to the vertical pixel misalignment correction unit 127, the storage unit 123, and the correction value determination unit 129. The second component misalignment estimation unit 128 estimates pixel misalignments caused by component misalignments related to a second group of factors excluding the first group of factors (hereinafter also referred to as "pixel misalignments related to the second group of factors") based on the vertical pixel misalignment corrected by the vertical pixel misalignment correction unit 127. That is, the second component misalignment estimation unit 128 estimates, as pixel misalignments related to the second group of factors, at least one of the following: pixel misalignment caused by a horizontal positional misalignment of the image sensor 33 relative to the optical element 34 (sensor horizontal shift shown in FIG. 3D); pixel misalignment caused by an optical axis misalignment of the optical element 34 (optical axis change shown in FIG. 3E); pixel misalignment caused by an angular misalignment in the yaw direction of the one camera 31 relative to the other camera 32 (single camera yaw rotation shown in FIG. 3F); and pixel misalignment caused by an angular misalignment in the yaw direction of the image sensor 33 relative to the optical element 34 (sensor yaw rotation shown in FIG. 3G). Estimating pixel shift related to the second group of factors means estimating which of the component shifts related to the second group of factors the component shift that is causing the parallax error corresponds to, and estimating the pixel shift (including the amount of pixel shift) caused by the corresponding component shift.

[0046] Specifically, the second component shift estimation unit 128 receives the corrected vertical pixel shifts output from the vertical pixel shift correction unit 127, the pixel shifts related to the first group of factors, and the criterion functions related to the first and second group of factors included in the image correction information stored in the storage unit 123. The second component shift estimation unit 128 estimates the pixel shifts related to the second group of factors by comparing the corrected vertical pixel shifts with the criterion functions related to the second group of factors. The second component shift estimation unit 128 outputs the pixel shifts related to the first group of factors, the pixel shifts related to the second group of factors, the criterion functions related to the first and second group of factors to the correction value determination unit 129.

[0047] The correction value determination unit 129 is connected to the second component shift estimation unit 128 and the correction unit 130. The correction value determination unit 129 calculates a correction value for the horizontal pixel shift caused by the component shift from the estimated pixel shift caused by the component shift, and determines the calculated correction value as the correction value for the parallax error.

[0048] Specifically, the correction value determination unit 129 receives the pixel misalignment related to the first factor group, the pixel misalignment related to the second factor group, the criterion function related to the first factor group, and the criterion function related to the second factor group output from the second component misalignment estimation unit 128. The correction value determination unit 129 calculates a correction value for the horizontal pixel misalignment and a correction value for the vertical pixel misalignment caused by all component misalignments based on the pixel misalignment related to the first factor group, the pixel misalignment related to the second factor group, the criterion function related to the first factor group, and the criterion function related to the second factor group. The correction value determination unit 129 determines the correction value for the horizontal pixel misalignment as the correction value for the parallax error. The correction value determination unit 129 outputs each correction value to the correction unit 130.

[0049] The correction unit 130 is connected to the correction value determination unit 129, the storage unit 123, and the distance measurement unit 131. The correction unit 130 receives each correction value output from the correction value determination unit 129. The correction unit 130 receives the vertical pixel shift and parallax stored in the storage unit 123. The correction unit 130 corrects the vertical pixel shift and parallax with a correction value for the vertical pixel shift and a correction value for the parallax error, respectively. The correction unit 130 outputs the corrected parallax to the distance measurement unit 131.

[0050] The distance measurement unit 131 is connected to the correction unit 130 and the control unit 132. The distance measurement unit 131 calculates distance information from the parallax output from the correction unit 130 and outputs the information to the control unit 132. The control unit 132 outputs a control signal to issue a warning or activate the brakes, etc., to a vehicle control device based on the distance information output from the distance measurement unit 131.

[0051] The functions of the image processing device 110 have been described above, but the flow of each of processes 1 to 4 performed by the image processing device 110 will now be described with reference to Figures 5 to 7. The image processing method of this embodiment is realized by the image processing device 110 performing each of processes 1 to 4.

[0052] Fig. 5 is a diagram illustrating the flow of processing by the image processing apparatus 110 shown in Fig. 4. Fig. 6 is a diagram illustrating a method for estimating component misalignment related to the first group of factors. Fig. 7 is a diagram illustrating a method for estimating component misalignment related to the second group of factors.

[0053] First, the image processing device 110 captures an image of the object 52 (Process 1). A pair of images PL and PR are captured by a pair of cameras 31 and 32, respectively. The image processing device 110 performs stereo matching on the pair of images PL and PR in the vertical direction to search for corresponding points between the images PL and PR and calculates the vertical pixel shift (DPY) of the entire region (Process 2). In this embodiment, this vertical pixel shift is used to estimate pixel shifts caused by various component shifts (Processes 3-1, 3-2, and 3-3). P1, P2, and P3 are respectively the pixel shift amount caused by a single camera roll rotation shown in FIG. 3(a), the pixel shift amount caused by a sensor vertical shift shown in FIG. 3(b), and the pixel shift amount caused by a single camera pitch rotation shown in FIG. 3(c). Let P4, P5, P6, and P7 be the pixel shift amount due to the horizontal sensor shift shown in Fig. 3(d), the pixel shift amount due to the optical axis change shown in Fig. 3(e), the pixel shift amount due to the single camera yaw rotation shown in Fig. 3(f), and the pixel shift amount due to the sensor yaw rotation shown in Fig. 3(g), respectively. The image processing device 110 estimates the pixel shift amounts P1 to P7 using the vertical pixel shift, and determines a correction value for the parallax error based on the results.

[0054] The image processing device 110 estimates pixel shift amounts P1 to P3 related to the first group of factors and pixel shift amounts P4 to P7 related to the second group of factors separately. The component shift related to the first group of factors has a large effect on pixel shift in the vertical direction. The component shift related to the second group of factors has a large effect on pixel shift in the horizontal direction (parallax error direction). In this embodiment, the first component shift estimation unit 126 of the image processing device 110 estimates pixel shift amounts P1 to P3 related to the first group of factors based on vertical pixel shift (hereinafter also referred to as "first vertical pixel shift") calculated in a first region where the vertical angle of view is within approximately 1° from the optical axis (Process 3-1).

[0055] 6 indicates the vertical pixel shift in the first region when a component shift related to the first factor group and a component shift related to the second factor group occur simultaneously. That is, FDY0 indicates the vertical pixel shift (first vertical pixel shift) in the first region of DPY shown in FIG.

[0056] The first component shift estimation unit 126 of the image processing device 110 calculates the slope factor kT of the graph representing FDY0 and uses the slope factor kT to estimate the pixel shift amount P1 due to single camera roll rotation. Specifically, the image processing device 110 estimates the pixel shift amount P1 by calculating the ratio of the slope factor kB of the reference function ly to the slope factor kT, using the vertical pixel shift calculated in advance with the angular shift of the single camera roll rotation as a constant value as the reference function ly. For example, the image processing device 110 sets the vertical pixel shift calculated in advance with the angular shift PO1 of the single camera roll rotation as the reference function ly. The graph 64 shown in FIG. 6 shows the reference function ly(θ,0). The slope factor of the reference function ly(θ,0) is kB. Here, (θ,φ) indicates that the horizontal angle of view is θ and the vertical angle of view is φ. (θ,0) indicates that the horizontal angle of view is θ and the vertical angle of view is 0°, i.e., coincides with the optical axis. The gradient rate kT of the vertical pixel shift of the image of the object is also shown in Fig. 6. In this case, the pixel shift amount P1 can be calculated as P1 = (kT / kB) * PO1.

[0057] The image processing device 110 then corrects the pixel shift amount P1 caused by the single camera roll rotation. The correction for the pixel shift amount P1 is calculated as (P1 / PO1)*ly. FDY1 shown in FIG. 6 indicates the vertical pixel shift of the first region obtained by correcting the pixel shift amount P1 caused by the single camera roll rotation. OT and WT shown in FIG. 6 respectively represent the vertical pixel shift amount when the horizontal angle of view coincides with the optical axis and the vertical pixel shift amount when the horizontal angle of view is located in the wide-angle region.

[0058] Next, the image processing device 110 uses OT and WT to estimate pixel shift amounts P2 and P3 caused by the sensor vertical shift and single camera pitch rotation. Specifically, the image processing device 110 sets the vertical pixel shift calculated in advance using the position shift PO2 of the sensor vertical shift as the reference function my. The image processing device 110 sets the vertical pixel shift calculated in advance using the angle shift PO3 of the single camera pitch rotation as the reference function ny. The pixel shift amounts P2 and P3 can be calculated using equations (1) and (2), respectively. Graphs 65 and 66 show the reference functions my(θ,0) and ny(θ,0), respectively. P2 = (WT - OT) / (my(50,0) - my(0,0)) ... (1) P3 = (OT - my(0,0) * P2) / ny(0,0) ... (2)

[0059] In equations (1) and (2), (50,0) indicates that the horizontal angle of view is located in a wide-angle region where the horizontal angle of view is 50°, and the vertical angle of view is 0°, i.e., coincides with the optical axis. (0,0) indicates that both the horizontal and vertical angles of view are 0°, i.e., coincide with the optical axis. The vertical pixel shift amount my(0,0) of the reference function my when both the horizontal and vertical angles of view are 0° is shown as reference numeral 62 on graph 65. The vertical pixel shift amount my(50,0) of the reference function my when the horizontal angle of view is 50° and the vertical angle of view is 0° is shown as reference numeral 61 on graph 65. The vertical pixel shift amount ny(0,0) of the reference function ny when both the horizontal and vertical angles of view are 0° is shown as reference numeral 63 on graph 66.

[0060] The image processing device 110 then corrects the pixel shift amounts P2 and P3 caused by the sensor vertical shift and the single camera pitch rotation. The correction for pixel shift amount P2 is calculated as (P2 / PO2) * my. The correction for pixel shift amount P3 is calculated as (P3 / PO3) * my. ADY0 shown in FIG. 6 indicates the vertical pixel shift of the first region obtained by correcting the pixel shift amounts P2 and P3 caused by the sensor vertical shift and the single camera pitch rotation. That is, ADY0 shown in FIG. 6 indicates the vertical pixel shift of the first region after the pixel shift related to the first group of factors has been corrected (Process 3-2). The remaining pixel shift related to the second group of factors is a pixel shift that has a large impact in the horizontal direction and is therefore hardly reflected in the vertical pixel shift of the first region. Therefore, the vertical pixel shift amount of ADY0 shown in FIG. 6 is approximately 0 (zero).

[0061] Next, the image processing device 110 estimates pixel shift amounts P4 to P7 related to the second group of factors. In this embodiment, the second component shift estimation unit 128 of the image processing device 110 estimates pixel shift amounts P4 to P7 related to the second group of factors based on the vertical pixel shift (hereinafter also referred to as "second vertical pixel shift") calculated in a second region where the vertical angle of view is located in the wide-angle region (e.g., a vertical angle of view of 14°) (Process 3-3). The second vertical pixel shift can be expressed using the vertical pixel shift obtained by correcting the pixel shift related to the first group of factors. Here, the vertical pixel shift for the entire region after the pixel shift related to the first group of factors has been corrected is referred to as ADY. ADY represents DPY2 shown in FIG. 5 .

[0062] The image processing device 110 sets the vertical pixel shift calculated in advance using the sensor horizontal shift positional deviation PO4 as the reference function fy. The image processing device 110 sets the vertical pixel shift calculated in advance using the optical axis deviation PO5 due to optical axis change as the reference function gy. The image processing device 110 sets the vertical pixel shift calculated in advance using the single camera yaw rotation angle deviation PO6 as the reference function hy. The image processing device 110 sets the vertical pixel shift calculated in advance using the sensor yaw rotation angle deviation PO7 as the reference function jy. Graphs 71, 72, 73, and 74 shown in FIG. 7 respectively represent the reference function fy(θ,14), the reference function gy(θ,14), the reference function hy(θ,14), and the reference function jy(θ,14). The vertical pixel shift ADY corrected for pixel shifts related to the first factor group can be calculated using equation (3) when the vertical angle of view is 14°. ADY (θ, 14)≒fy (θ, 14) * (P4 / PO4) +gy (θ, 14) * (P5 / PO5) +hy (θ, 14) * (P6 / PO6) +jy (θ, 14) * (P7 / PO7) ... (3)

[0063] In equation (3), ADY(θ,14) represents the vertical pixel shift ADY at a vertical angle of view of 14° (second region) after the pixel shift related to the first factor group has been corrected. ADY(θ,14) is represented by the graph ADY14 shown in FIG. 7. PO4 to PO7 are preset values. The reference functions fy, gy, hy, and jy are also calculated in advance. The desired values ​​are pixel shift amounts P4 to P7. Simply put, graph ADY14 can be represented by combining graphs 71 to 74. In other words, the image processing device 110 simply estimates pixel shift amounts P4 to P7 using an arbitrary θ. The image processing device 110 can estimate pixel shift amounts P4 to P7 by searching for the respective optimal values ​​using an estimation method such as model fitting or the least squares method.

[0064] Finally, the image processing device 110 calculates the total parallax error using the pixel shift amounts P1 to P3 related to the first factor group and the pixel shift amounts P4 to P7 related to the second factor group, which becomes the correction value CDX for the parallax error (Process 4).

[0065] The image processing device 110 sets the basis function of horizontal pixel shift due to single camera roll rotation as lx. The image processing device 110 sets the basis function of horizontal pixel shift due to sensor vertical shift as mx. The image processing device 110 sets the basis function of horizontal pixel shift due to single camera pitch rotation as nx. The image processing device 110 sets the basis function of horizontal pixel shift due to sensor horizontal shift as fx. The image processing device 110 sets the basis function of horizontal pixel shift due to optical axis change as gx. The image processing device 110 sets the basis function of horizontal pixel shift due to single camera yaw rotation as hx. The image processing device 110 sets the basis function of horizontal pixel shift due to sensor yaw rotation as jx.

[0066] The image processing device 110 can calculate a correction value CDX for the parallax error using the pixel shift amounts P1 to P3 related to the first group of factors and the pixel shift amounts P4 to P7 related to the second group of factors using equation (4). Each term on the right side of equation (4) corresponds to a correction value for the horizontal pixel shift caused by each component shift. CDX(θ,φ)≈lx(θ,φ)*(P1 / PO1) +mx(θ,φ)*(P2 / PO2) +nx(θ,φ)*(P3 / PO3) +fx(θ,φ)*(P4 / PO4) +gx(θ,φ)*(P5 / PO5) +hx(θ,φ)*(P6 / PO6) +jx(θ,φ)*(P7 / PO7) ... (4)

[0067] Note that the image processing device 110 may appropriately integrate the reference functions fy, gy, hy, and jy because they are similar. For example, the reference function fx for the sensor horizontal shift and the reference function gy for the optical axis change are similar, so these may be integrated into a single reference function (e.g., fy). For example, the reference function hy for the single camera yaw rotation and the reference function jy for the sensor yaw rotation are similar, so these may be integrated into a single reference function (e.g., hy). In this way, the image processing device 110 can estimate the pixel shift amount related to the second group of factors using equation (5) instead of equation (3). ADY(θ,φ)≒fy(θ,φ)*(PM45 / PO4) +jy(θ,φ)*(PM67 / PO7) ... (5)

[0068] In equation (5), pixel shift amount PM45 is the pixel shift amount obtained by integrating pixel shift amount P4 due to horizontal sensor shift and pixel shift amount P5 due to optical axis change. Pixel shift amount PM67 is the pixel shift amount obtained by integrating pixel shift amount P6 due to single camera yaw rotation and pixel shift amount P7 due to sensor yaw rotation. This allows the image processing device 110 to reduce the component shift factors related to the second group of factors, thereby reducing the calculation load required to estimate pixel shift amounts related to the second group of factors. Note that the integration is not limited to two criterion functions, and integration into one criterion function or three criterion functions is also possible.

[0069] Furthermore, in the above description, when estimating pixel misalignment due to component misalignment related to the first group of factors, the image processing device 110 estimates the pixel misalignment using the characteristics of the vertical pixel misalignment in the first region. Because the amount of vertical pixel misalignment in the first region due to component misalignment related to the first group of factors and the amount of vertical pixel misalignment in the second region are substantially the same, it is also possible to directly correct the vertical pixel misalignment in the second region using the vertical pixel misalignment amount in the first region. Furthermore, in the above description, the image processing device 110 estimates pixel misalignment related to the first group of factors and pixel misalignment related to the second group of factors separately, but even if only pixel misalignment related to the first group of factors or only pixel misalignment related to the second group of factors is estimated and a correction value is determined, a correction effect on parallax error can be achieved.

[0070] In order to theoretically verify the effect of correcting the parallax error as described above, a simulation was performed using ray tracing. The simulation conditions (design values) were as follows: Calculation conditions: Lens focal length: approximately 5 mm Pixel pitch of the image sensor 33: approximately 3 μm Baseline length of the pair of cameras 31, 32: approximately 20 cm Distance to the object: 50 m Horizontal angle of view range: -50° to +50° (+ indicates rightward, - indicates leftward) Vertical angle of view range: -14° to +14° (+ indicates downward, - indicates upward) Angular deviation of single camera roll rotation: 0.05° Positional deviation of sensor vertical shift: 2 pixels Angular deviation of single camera pitch rotation: 0.05° Positional deviation of sensor horizontal shift: 2 pixels Optical axis deviation of optical axis change: 2 pixels Angular deviation of single camera yaw rotation: 0.05° Sensor yaw rotation: 0.05°

[0071] FIG. 8 is a diagram showing the verification results by simulation.

[0072] Graph 81 in Fig. 8 shows the simulation results of the parallax error at a vertical angle of view of 0°. Reference numeral 82 indicates the parallax error before correction, and reference numeral 83 indicates the parallax error after correction by the image processing device 110 of this embodiment. The parallax error before correction 82 reaches a maximum of 20 pixels in the wide-angle region of a horizontal angle of view of 50°. On the other hand, the parallax error after correction 83 is corrected to approximately 0 pixels in the wide-angle region of a horizontal angle of view of 50°.

[0073] Graph 84 in Figure 8 shows the results of converting graph 81, which shows the simulation results of parallax error, into a ranging error rate. Reference numeral 85 denotes the ranging error rate before correction, and reference numeral 86 denotes the ranging error rate after correction by the image processing device 110 of this embodiment. The ranging error rate 85 before correction reaches a maximum of approximately 73% in the wide-angle range of a horizontal angle of view of 50°. On the other hand, the ranging error rate 86 after correction is reduced to a maximum of approximately 4% in the wide-angle range of a horizontal angle of view of 50°.

[0074] 8, it was confirmed that the image processing device 110 of this embodiment can significantly improve the parallax error and the distance measurement error. Therefore, the image processing device 110 can achieve high-precision distance measurement even in a wide-angle field of view, and can achieve stable vehicle control.

[0075] As described above, the image processing device 110 of this embodiment includes a pixel shift calculation unit 124 that calculates the vertical pixel shift between a pair of images captured by a pair of cameras 31, 32, each of which includes an optical element 34 and an image sensor 33; component shift estimation units 126, 128 that estimate, based on the calculated vertical pixel shift, the pixel shift between the images caused by component shift, which is a shift in the position or orientation of at least one of the cameras 31, 32, the optical element 34, and the image sensor 33; and a correction value determination unit 129 that calculates a correction value for the horizontal pixel shift between the images caused by the component shift from the estimated pixel shift caused by the component shift, and determines the calculated correction value as a correction value for the parallax error between the images. The component misalignment estimation units 126, 128 estimate at least one of the following pixel misalignments caused by component misalignment: pixel misalignment caused by an angular misalignment in the roll direction of one camera 31 relative to the other camera 32; pixel misalignment caused by an angular misalignment in the pitch direction of the other camera 31 relative to the one camera 32; and pixel misalignment caused by a vertical positional misalignment of the image sensor 33 relative to the optical element 34.

[0076] As a result, the image processing device 110 can separate pixel shifts caused by various component shifts of the camera that occur while the vehicle is running due to aging or temperature changes into component shifts by using the vertical pixel shift, and estimate the amount of pixel shift for each component shift.The image processing device 110 can then calculate a correction value for each parallax error from the amount of each pixel shift, and determine a correction value for the total parallax error.As a result, the image processing device 110 can accurately calculate and correct the parallax error caused by component shifts of the camera that occur while the vehicle is running due to aging or temperature changes.

[0077] Furthermore, the image processing device 110 of this embodiment defines pixel misalignment caused by angular misalignment in the roll direction of the one camera 31 relative to the other camera 32, pixel misalignment caused by angular misalignment in the pitch direction of the one camera 31 relative to the one camera 32, and pixel misalignment caused by vertical positional misalignment of the image sensor 33 relative to the optical element 34 as pixel misalignment caused by component misalignment related to a first group of factors. The image processing device 110 further includes a vertical pixel misalignment correction unit 127 that corrects the vertical pixel misalignment calculated by the pixel misalignment calculation unit 124 based on the pixel misalignment caused by component misalignment related to the first group of factors. The component misalignment estimation units 126 and 128 estimate pixel misalignment caused by component misalignment related to a second group of factors excluding the first group of factors, based on the vertical pixel misalignment corrected by the vertical pixel misalignment correction unit 127.

[0078] As a result, the image processing device 110 can separate pixel misalignment caused by component misalignment into pixel misalignment caused by component misalignment related to a first group of factors that has a large influence on pixel misalignment in the vertical direction and pixel misalignment caused by component misalignment related to a second group of factors that has a large influence on pixel misalignment in the horizontal direction, and estimate the amount of pixel misalignment for each component misalignment. Therefore, the image processing device 110 can easily perform calculations to estimate each pixel misalignment amount, and can easily determine a correction value for parallax error. Therefore, the image processing device 110 can accurately and easily calculate and correct parallax error caused by component misalignment of the camera that occurs while the vehicle is traveling due to aging or temperature changes.

[0079] Furthermore, in the image processing device 110 of this embodiment, the component shift estimation units 126, 128 estimate pixel shifts caused by component shifts related to a first group of factors based on a first vertical pixel shift, which is a vertical pixel shift calculated in a first region in a set of images where the vertical angle of view is within approximately 1° from the optical axis, out of the vertical pixel shifts calculated by the pixel shift calculation unit 124, and estimate pixel shifts caused by component shifts related to a second group of factors based on a second vertical pixel shift, which is a vertical pixel shift calculated in a second region where the vertical angle of view is larger than that of the first region, out of the vertical pixel shifts corrected by the vertical pixel shift correction unit 127.

[0080] This allows the image processing device 110 to more easily perform calculations to estimate the amount of pixel shift caused by component shifts related to the first group of factors and the amount of pixel shift caused by component shifts related to the second group of factors, thereby enabling the image processing device 110 to accurately and more easily calculate and correct parallax errors caused by component shifts of the camera that occur while the vehicle is running due to aging or temperature changes.

[0081] Furthermore, in the image processing device 110 of this embodiment, the component shift estimation units 126, 128 use the gradient rate of the first vertical pixel shift calculated in the first region to estimate the pixel shift caused by the angular shift in the roll direction of one camera 31 relative to the other camera 32.

[0082] This allows the image processing device 110 to easily estimate pixel misalignment caused by angular misalignment in the roll direction of one camera 31 relative to the other camera 32. Therefore, the image processing device 110 can more easily perform calculations to estimate the amount of pixel misalignment caused by this type of component misalignment. Therefore, the image processing device 110 can accurately and more easily calculate and correct parallax error caused by component misalignment of the camera that occurs while the vehicle is running due to aging or temperature changes.

[0083] Furthermore, in the image processing device 110 of this embodiment, the component misalignment estimation units 126, 128 use the value of the first vertical pixel misalignment calculated in the first region, where the horizontal angle of view approximately coincides with the optical axis, to estimate the pixel misalignment caused by the angle misalignment in the pitch direction of one camera 31 relative to the other camera 32, and the pixel misalignment caused by the vertical positional misalignment of the image sensor 33 relative to the optical element 34.

[0084] This allows the image processing device 110 to easily estimate pixel shifts caused by angular shifts in the pitch direction of the one camera 31 relative to the other camera 32, and pixel shifts caused by vertical positional shifts of the image pickup element 33 relative to the optical element 34. Therefore, the image processing device 110 can more easily perform calculations to estimate the amount of pixel shift caused by this type of component shift. Therefore, the image processing device 110 can accurately and more easily calculate and correct parallax errors caused by component shifts of the cameras that occur while the vehicle is running due to aging or temperature changes.

[0085] The image processing method of this embodiment is realized by the image processing device 110 performing the above processes 1 to 4.

[0086] That is, the image processing method of this embodiment includes a pixel misalignment calculation process (process 2) that calculates a vertical pixel misalignment between a pair of images captured by a pair of cameras 31 and 32, each of which includes an optical element 34 and an image sensor 33, a component misalignment estimation process (processes 3-1 to 3-3) that estimates, based on the calculated vertical pixel misalignment, a pixel misalignment between the images caused by a component misalignment, which is a misalignment in the position or orientation of at least one of the cameras 31 and 32, the optical element 34, and the image sensor 33, and a correction value determination process (process 4) that calculates a correction value for a horizontal pixel misalignment between the images caused by the component misalignment from the estimated pixel misalignment caused by the component misalignment, and determines the calculated correction value as a correction value for a parallax error between the images. The component misalignment estimation process estimates, as the pixel misalignment caused by the component misalignment, at least one of the pixel misalignment caused by an angular misalignment in the roll direction of one camera 31 relative to the other camera 32, the pixel misalignment caused by an angular misalignment in the pitch direction of the other camera 31 relative to the camera 32, and the pixel misalignment caused by a vertical positional misalignment of the image sensor 33 relative to the optical element 34.

[0087] As a result, the image processing method of this embodiment can separate pixel shifts caused by various component shifts of the camera that occur while the vehicle is running due to aging or temperature changes into component shifts by using vertical pixel shifts, and estimate the amount of pixel shift for each component shift.The image processing method of this embodiment can then calculate a correction value for each parallax error from the amount of each pixel shift, and determine a correction value for the total parallax error.Therefore, the image processing method of this embodiment can accurately calculate and correct parallax errors caused by component shifts of the camera that occur while the vehicle is running due to aging or temperature changes.

[0088] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0089] Furthermore, the above-described configurations, functions, processing units, or processing means may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations or functions may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the program, table, or file that implements each function can be stored in a storage device such as a memory, a hard disk, or an SSD (solid state drive), or in a storage medium such as an IC card, an SD card, or a DVD.

[0090] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0091] 31, 32... Pair of cameras, 31... Other camera, 32... One camera, 33... Image pickup element, 34... Optical element, 110... Image processing device, 124... Pixel shift calculation unit, 126... First component shift estimation unit (component shift estimation unit), 127... Vertical pixel shift correction unit, 128... Second component shift estimation unit (component shift estimation unit), 129... Correction value determination unit

Claims

1. An image processing device comprising: a pixel shift calculation unit that calculates a vertical pixel shift between a pair of images captured by a pair of cameras, each of which includes an optical element and an imaging element; a component shift estimation unit that estimates a pixel shift between the images caused by a component shift, which is a shift in the position or attitude of at least one of the cameras, the optical element, and the imaging element, based on the calculated vertical pixel shift; and a correction value determination unit that calculates a correction value for a horizontal pixel shift between the images caused by the component shift from the estimated pixel shift caused by the component shift, and determines the calculated correction value as a correction value for a parallax error between the images, wherein the component shift estimation unit estimates at least one of the pixel shift caused by an angular shift in the roll direction of one camera relative to the other camera, the pixel shift caused by an angular shift in the pitch direction of the other camera relative to the one camera, and the pixel shift caused by a vertical positional shift of the imaging element relative to the optical element, as the pixel shift caused by the component shift.

2. The image processing device according to claim 1, characterized in that, if the pixel shift caused by an angular misalignment in the roll direction of the one camera relative to the other camera, the pixel shift caused by an angular misalignment in the pitch direction of the one camera relative to the other camera, and the pixel shift caused by a positional misalignment in the vertical direction of the imaging element relative to the optical element are defined as the pixel shift caused by the component misalignment related to a first group of factors, the image processing device further comprises a vertical pixel shift correction unit that corrects the vertical pixel shift calculated by the pixel shift calculation unit based on the pixel shift caused by the component misalignment related to the first group of factors, and the component misalignment estimation unit estimates the pixel shift caused by the component misalignment related to a second group of factors excluding the first group of factors based on the vertical pixel shift corrected by the vertical pixel shift correction unit.

3. The image processing device described in claim 2, characterized in that the component misalignment estimation unit estimates the pixel misalignment caused by the component misalignment related to the first group of factors based on a first vertical pixel shift, which is the vertical pixel shift calculated in a first region in the set of images where the vertical angle of view is within approximately 1° from the optical axis, among the vertical pixel shifts calculated by the pixel misalignment calculation unit, and estimates the pixel misalignment caused by the component misalignment related to the second group of factors based on a second vertical pixel shift, which is the vertical pixel shift calculated in a second region in the set of images where the vertical angle of view is larger than the first region, among the vertical pixel shifts corrected by the vertical pixel shift correction unit.

4. The image processing device described in claim 3, characterized in that the component shift estimation unit estimates the pixel shift caused by an angular shift in the roll direction of the one camera relative to the other camera using the gradient rate of the first vertical pixel shift calculated in the first region.

5. The image processing device described in claim 3, characterized in that the component misalignment estimation unit estimates the pixel misalignment caused by an angle misalignment in the pitch direction of the one camera relative to the other camera and the pixel misalignment caused by a vertical positional misalignment of the imaging element relative to the optical element, using the value of the first vertical pixel misalignment calculated in the first region where the horizontal angle of view approximately coincides with the optical axis.

6. An image processing method comprising: a pixel shift calculation process for calculating a vertical pixel shift between a pair of images captured by a pair of cameras, each of which includes an optical element and an imaging element; a component shift estimation process for estimating a pixel shift between the images caused by a component shift, which is a shift in position or attitude of at least one of the cameras, the optical element and the imaging element, based on the calculated vertical pixel shift; and a correction value determination process for calculating a correction value for a horizontal pixel shift between the images caused by the component shift from the estimated pixel shift caused by the component shift, and determining the calculated correction value as a correction value for a parallax error between the images, wherein the component shift estimation process estimates at least one of the pixel shift caused by an angular shift in the roll direction of one camera relative to the other camera, the pixel shift caused by an angular shift in the pitch direction of the other camera relative to the one camera, and the pixel shift caused by a vertical positional shift of the imaging element relative to the optical element, as the pixel shift caused by the component shift.

7. The image processing method according to claim 6, characterized in that, if the pixel shift caused by an angular shift in the roll direction of the one camera relative to the other camera, the pixel shift caused by an angular shift in the pitch direction of the one camera relative to the other camera, and the pixel shift caused by a positional shift in the vertical direction of the imaging element relative to the optical element are defined as the pixel shift caused by the component shift related to a first group of factors, the image processing method further comprises a vertical pixel shift correction process that corrects the vertical pixel shift calculated by the pixel shift calculation process based on the pixel shift caused by the component shift related to the first group of factors, and the component shift estimation process estimates the pixel shift caused by the component shift related to a second group of factors excluding the first group of factors based on the vertical pixel shift corrected by the vertical pixel shift correction process.

8. The image processing method according to claim 7, characterized in that the component shift estimation process estimates the pixel shift caused by the component shift related to the first group of factors based on a first vertical pixel shift, which is the vertical pixel shift calculated in a first region in the set of images where the vertical angle of view is within approximately 1° from the optical axis, among the vertical pixel shifts calculated by the pixel shift calculation process, and estimates the pixel shift caused by the component shift related to the second group of factors based on a second vertical pixel shift, which is the vertical pixel shift calculated in a second region in the set of images where the vertical angle of view is larger than the first region, among the vertical pixel shifts corrected by the vertical pixel shift correction process.

9. The image processing method according to claim 8, characterized in that the component shift estimation process estimates the pixel shift caused by an angular shift in the roll direction of the one camera relative to the other camera using a gradient rate of the first vertical pixel shift calculated in the first region.

10. The image processing method according to claim 8, characterized in that the component shift estimation process uses the value of the first vertical pixel shift calculated in the first region, the value of which is a first vertical pixel shift whose horizontal angle of view approximately coincides with the optical axis, to estimate the pixel shift caused by an angle shift in the pitch direction of the one camera relative to the other camera, and the pixel shift caused by a vertical positional shift of the imaging element relative to the optical element.

Images