Image processing device and image processing method

Abstract

To accurately calculate and correct a parallax error due to deviated parts of a camera generated by deterioration with time and temperature change while a vehicle travels.SOLUTION: An image processing device 110 comprises: a pixel deviation calculation unit 124 calculating vertical pixel deviation between a set of pixels imaged by a set of cameras 31, 32; parts deviation estimation units 126, 128 estimating inter-pixel pixel deviation due to parts deviation based on the calculated vertical pixel deviation; a correction value determination unit 129 determining a correction value with respect to the parallax error from pixel deviation due to the estimated parts deviation. The parts deviation estimation units 126, 128 estimate pixel deviation due to at least one parts deviation of angle deviation, to one camera 32, of an other camera 31 in a roll direction, angle deviation of the other camera 31 to the one camera 32 in a pitch direction, and positional deviation of an image pick-up device 33 to an optical element 34 in a vertical direction.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] A stereo camera is a device that uses multiple cameras to mimic human binocular vision and accurately measure the distance between an object and the camera. The principle of stereo camera distance measurement is that a pair of cameras with a certain distance simultaneously capture the same object, and based on the principle of triangulation, calculate the difference between corresponding points between the pair of images to measure the distance between the object and the camera. The distance between the pair of cameras is called the baseline length, and the difference between corresponding points between the pair of images is called the parallax.

[0003] With the development of computer vision and artificial intelligence technology, stereo cameras are widely used in automotive technology for various applications, such as automatic parking, adaptive cruise control (ACC), lane departure warning, or traffic sign recognition, etc. For example, ACC systems use in-vehicle stereo camera technology to measure the distance to the vehicle ahead and the speed of the vehicle ahead in real time, and control the distance between vehicles to ensure safe driving.

[0004] Compared to other sensors such as lidar or millimeter wave radar, stereo cameras have advantages such as diversity, high resolution, and low cost. The advantage of diversity is that stereo cameras can obtain detailed information such as the color or shape of an object by acquiring image information, and therefore more functions can be realized using image recognition. On the other hand, these other sensors are mainly used for distance measurement and cannot measure color. Also, the advantage of high resolution is that a larger number of pixels can be acquired compared to the scanning method adopted by these other sensors, so that highly accurate image information can be acquired. Therefore, stereo cameras have become one of the very important technologies in automotive technology, and can provide highly accurate and reliable data for ADAS (Advanced Driver Assistance Systems) or autonomous driving, thereby improving and sophisticating the safety of automobiles.

[0005] On the other hand, since many traffic accidents occur at intersections, in order to reduce traffic accidents, the stereo camera needs to improve the recognition performance for pedestrians or bicycles at intersections. Conventional stereo cameras are mainly used to obtain distance information to assist in collision avoidance of a forward object, so a long-distance recognition angle of view (an angle of view capable of image recognition, hereinafter also referred to as "angle of view"). A wider angle of view is required to avoid collisions at intersections. In order to realize both a long-distance angle of view and a wide angle of view, for example, a fisheye lens can be used. However, the fisheye lens causes large image distortion in the wide-angle region with a large angle of view. This image distortion is usually corrected by geometric calibration during manufacturing. In addition, the relative position or attitude deviation between a pair of cameras is corrected by this geometric calibration.

[0006] However, in reality, even if the camera is perfectly corrected during manufacturing, "component misalignment" may occur during use by a user due to aging or temperature changes. The component misalignment is a misalignment in the position or orientation of the components that make up the stereo camera. The component misalignment includes a misalignment in the position or orientation between the image sensor and the optical element in one camera, and a misalignment in the position or orientation between a pair of cameras. Such component misalignment makes the image correction data used during manufacturing to correct image distortion inaccurate, causing a "pixel misalignment" between a pair of images. A stereo camera usually calculates the parallax from the horizontal direction to calculate the distance. Therefore, the pixel misalignment in the horizontal direction corresponds to a parallax error and causes a ranging error. The pixel misalignment in the horizontal direction is also called "horizontal pixel misalignment", and the pixel misalignment in the vertical direction is also called "vertical pixel misalignment".

[0007] In a narrow-angle region with a small angle of view, image distortion is small, so the parallax error in the narrow-angle region can be corrected, for example, by using a constant value obtained from the parallax at infinity. In a wide-angle region, image distortion is large, so the parallax error in the wide-angle region cannot be sufficiently corrected by the same method as the parallax error in the narrow-angle region. If the correction of the parallax error is insufficient, the distance measurement error will be large, which may affect safety. Therefore, it is important how to calculate and correct the parallax error in the entire angle of view region. Usually, this parallax error is corrected by periodically recalibrating at a maintenance shop or dealer. However, due to deterioration over time or temperature changes, the influence of the parallax error may appear in the stereo camera while the vehicle is running, and there is a possibility that an accident cannot be prevented. Therefore, a function for automatically correcting the parallax error while the vehicle is running is required.

[0008] A known technology for correcting parallax error caused by component misalignment is disclosed in Patent Document 1. Patent Document 1 describes that "relative vertical shifts in images captured by left and right cameras are detected, and the relative tilt angles of the imaging planes of the left and right cameras are calculated from the detected vertical shifts, and the relative vertical and horizontal shifts of the images captured by the left and right cameras are corrected according to the calculated tilt angles." [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2014-010100 A Summary of the Invention [Problem to be solved by the invention]

[0010] Since the parallax changes with the change in the distance to the object, the parallax error also changes with the change in the distance to the object, so it is difficult to calculate the parallax error directly. On the other hand, the vertical pixel shift between a pair of images is usually zero when the stereo camera is perfectly calibrated. However, when the stereo camera is subjected to component shift due to aging or temperature change, the vertical pixel shift occurs. And the vertical pixel shift does not change with the distance to the object. Therefore, it is an effective method to calculate the parallax error due to the component shift using the vertical pixel shift.

[0011] In Patent Document 1, the relative vertical shift of the images captured by the left and right cameras is detected, the amount of shift between the sensor and lens in the camera is calculated, and the parallax error is corrected. However, in actual situations, the component shift includes not only the shift between the sensor and lens, but also a mixture of various types of component shifts. Therefore, with the technology described in Patent Document 1, it is difficult to accurately calculate the parallax error caused by various types of component shifts that occur while the vehicle is traveling and correct the parallax error.

[0012] The present invention has been made in consideration of the above, and has an object to accurately calculate and correct parallax error caused by misalignment of camera components that occurs while a vehicle is running due to deterioration over time or temperature changes. [Means for solving the problem]

[0013] In order to solve the above problem, the image processing device of the present invention includes a pixel shift calculation unit that calculates a vertical pixel shift between a pair of images captured by a pair of cameras, each of which includes an optical element and an imaging element; a component shift estimation unit that estimates a pixel shift between the images caused by a component shift, which is a shift in position or attitude of at least one of the cameras, the optical element, and the imaging element, based on the calculated vertical pixel shift; and a correction value determination unit that calculates a correction value for a horizontal pixel shift between the images caused by the component shift from the estimated pixel shift caused by the component shift, and determines the calculated correction value as a correction value for a parallax error between the images, wherein the component shift estimation unit estimates at least one of the pixel shift caused by an angular shift in the roll direction of one camera relative to the other camera, the pixel shift caused by an angular shift in the pitch direction of the other camera relative to the one camera, and the pixel shift caused by a vertical positional shift of the imaging element relative to the optical element, as the pixel shift caused by the component shift.

[0014] In the present invention, "horizontal" means a direction along the baseline of a pair of cameras, and "vertical" means a direction perpendicular to the baseline of a pair of cameras and to the front-rear direction. In other words, the terms "horizontal" and "vertical" in the present invention are relative concepts. Effect of the Invention

[0015] According to the present invention, it is possible to accurately calculate and correct parallax error caused by component misalignment of a camera that occurs while a vehicle is running due to aging or temperature changes. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0016] [Figure 1] 11A and 11B are diagrams for explaining pixel misalignment when a sensor horizontal shift occurs as an example of component misalignment. [Diagram 2] 2A to 2C are diagrams for explaining pixel shifts after the distortion of the image shown in FIG. 1 is corrected. [Diagram 3]A diagram explaining seven types of component misalignment. [Figure 4] FIG. 1 is a diagram showing the configuration of a camera system according to an embodiment of the present invention. [Diagram 5] 5 is a diagram for explaining the flow of processing in the image processing apparatus shown in FIG. 4. [Figure 6] FIG. 11 is a diagram for explaining a method for estimating component misalignment related to a first factor group. [Figure 7] FIG. 11 is a diagram for explaining a method for estimating component misalignment related to a second factor group. [Figure 8] FIG. 13 is a diagram showing the results of verification by simulation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiments or examples are illustrative for explaining the present invention, and are omitted and simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise limited, each component may be singular or plural. The position, size, shape, or range of each component shown in the drawings may not represent the actual position, size, shape, or range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, or range, etc., disclosed in the drawings. Note that, unless otherwise stated, the configuration or function to which the same symbol is attached in each embodiment has the same configuration or function in each embodiment, and the description thereof will be omitted. However, when there are multiple components having the same or similar functions, the same symbol may be attached with different subscripts to explain. In addition, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted to explain.

[0018] FIG. 1 is a diagram for explaining pixel misalignment when a sensor horizontal shift occurs as an example of component misalignment.

[0019] The camera system 100 including the cameras 31 and 32 may experience "component misalignment" due to aging or temperature changes. The component misalignment is a misalignment in the position or attitude of the cameras 31 and 32 included in the camera system 100, or a misalignment in the position or attitude of the components that make up the cameras 31 and 32. The sensor horizontal shift is a misalignment in the horizontal direction of the image sensor 33 relative to the optical element 34 that is a component of the cameras 31 and 32. The optical element 34 forms an image of the object on the imaging surface of the image sensor 33.

[0020] In FIG. 1, pixel points 10, 12, 14, and 16 each indicate a pixel point before the horizontal sensor shift occurs. The positions of pixel points 10, 12, 14, and 16 can be expressed by a horizontal angle of view (hereinafter also referred to as a "horizontal angle of view") and a vertical angle of view (hereinafter also referred to as a "vertical angle of view"). The horizontal angle of view and the vertical angle of view are expressed as an angle of displacement in the horizontal direction from a pixel point intersecting the optical axis of optical element 34 (a pixel point corresponding to the optical center of optical element 34) as the origin (horizontal angle of view 0°, vertical angle of view 0°). The vertical angle of view is expressed as an angle of displacement in the vertical direction from the origin.

[0021] Pixel point 10 is a pixel point whose horizontal and vertical angles of view both coincide with the optical axis, i.e., a pixel point located at the origin. Pixel point 12 is a pixel point whose vertical angle of view coincides with the optical axis and whose horizontal angle of view is in a wide-angle region (region with a large angle of view). Pixel point 14 is a pixel point whose horizontal angle of view coincides with the optical axis and whose vertical angle of view is in a wide-angle region. Pixel point 16 is a pixel point whose horizontal and vertical angles of view both lie in the wide-angle region.

[0022] The first region is a region where the vertical angle of view is within approximately 1° from the optical axis. The second region is a region where the vertical angle of view is larger than that of the first region. In this embodiment, the second region is a region where the vertical angle of view is located in a wide-angle region. In the example of FIG. 1, the region where the vertical angle of view coincides with the optical axis (region where the vertical angle of view is 0°) is shown as the first region. In the example of FIG. 1, the region where the vertical angle of view is 14° is shown as the second region.

[0023] In FIG. 1, pixel points 11, 13, 15, and 17 indicate positions to which pixel points 10, 12, 14, and 16 have been displaced due to the horizontal sensor shift. SO0, SO1, SO2, and SO3 indicate the respective pixel shifts. The pixel shift is a shift in the positions of corresponding pixel points between a pair of images captured by a pair of cameras 31 and 32. In this embodiment, the pixel shift in the horizontal direction is also referred to as the "horizontal pixel shift," and the pixel shift in the vertical direction is also referred to as the "vertical pixel shift." The horizontal pixel shift corresponds to a parallax error.

[0024] Figure 1 shows an image before the image distortion is corrected, so SO0, SO1, SO2, and SO3 all have horizontal pixel shifts. The pixel shift amounts for all pixel points 10, 12, 14, and 16 are the same. Also, in Figure 1, only the sensor horizontal shift has occurred, so horizontal pixel shifts occur, but vertical pixel shifts do not. In other words, for an image before distortion correction, it is not possible to estimate horizontal component shifts using vertical pixel shifts.

[0025] FIG. 2 is a diagram for explaining pixel shifts after the distortion of the image shown in FIG. 1 is corrected.

[0026] In Fig. 2, pixel points 20, 22, 24, and 26 correspond to pixel points 10, 12, 14, and 16 in Fig. 1, respectively. That is, pixel points 20, 22, 24, and 26 indicate pixel points before the horizontal sensor shift occurs. In Fig. 2, pixel points 21, 23, 25, and 27 indicate positions to which pixel points 20, 22, 24, and 26 have been displaced by the horizontal sensor shift, respectively. That is, pixel points 21, 23, 25, and 27 correspond to pixel points 11, 13, 15, and 17 in Fig. 1, respectively.

[0027] SS0 shown in Fig. 2 corresponds to pixel shift SO0 of pixel point 10 in Fig. 1, where both the horizontal and vertical angles of view coincide with the optical axis. Since there is almost no image distortion near the pixel point where both the horizontal and vertical angles of view coincide with the optical axis, the pixel shift amounts of SS0 and SO0 are almost the same.

[0028] On the other hand, SO1 shown in FIG. 1 is a pixel shift in a wide-angle region with large image distortion, so the pixel shift amount of SS1 corresponding to SO1 is larger than that of SO1. This means that the parallax error becomes large in the wide-angle region. SS2 is a pixel shift where the horizontal angle of view coincides with the optical axis and the vertical angle of view is located in the wide-angle region. SS2 is a pixel shift at a position where the horizontal angle of view coincides with the optical axis, so the pixel shift amount is the same as that of the corresponding SO2. However, SO3 shown in FIG. 1 has both the horizontal and vertical angles of view located in the wide-angle region, so image distortion is large in both the horizontal and vertical directions, and the corresponding SS3 has a large pixel shift amount in both the horizontal and vertical directions. That is, SS3 has a vertical pixel shift. In this embodiment, the component shift is estimated using the vertical pixel shift in the wide-angle region, and the parallax error is estimated from the pixel shift in the horizontal direction (parallax error direction) and corrected. FIG. 2 shows the focus of this embodiment.

[0029] 1 and 2, a horizontal sensor shift has been described as an example of component misalignment, but in this embodiment, seven types of component misalignment are assumed. FIG 3 is a diagram for explaining the seven types of component misalignment.

[0030] The seven types of component misalignment shown in FIG. 3 occur due to aging or temperature changes of the camera system 100, and are considered to have a large effect on the parallax error. The X-axis, Y-axis, and Z-axis in FIG. 3 are coordinate axes along the left-right direction, the up-down direction, and the front-back direction of a pair of cameras 31 and 32, respectively. As an example of component misalignment, a single camera roll rotation shown in FIG. 3(a) is an angular misalignment of one camera 31 around the Z axis (roll direction) relative to the other camera 32. A sensor vertical shift shown in FIG. 3(b) is a positional misalignment of the imaging element 33 in the Y axis direction (up-down direction, i.e., vertical direction) relative to the optical element 34. A single camera pitch rotation shown in FIG. 3(c) is an angular misalignment of the other camera 31 around the X axis (pitch direction) relative to the one camera 32. A sensor horizontal shift shown in FIG. 3(d) is a positional misalignment of the imaging element 33 in the X axis direction (left-right direction, i.e., horizontal direction) relative to the optical element 34. An optical axis change shown in FIG. 3(e) is a misalignment of the optical axis of the optical element 34 due to temperature changes or the like. The single camera yaw rotation shown in Fig. 3(f) refers to the angular deviation of one camera 31 around the Y axis (yaw direction) relative to the other camera 32. The sensor yaw rotation shown in Fig. 3(g) refers to the angular deviation of the imaging element 33 around the Y axis (yaw direction) relative to the optical element 34.

[0031] In this embodiment, the single camera roll rotation, sensor vertical shift, and single camera pitch rotation shown in Fig. 3(a) to Fig. 3(c) are also referred to as "component misalignment related to the first factor group." In this embodiment, the sensor horizontal shift, optical axis change, single camera yaw rotation, and sensor yaw rotation shown in Fig. 3(d) to Fig. 3(g) are also referred to as "component misalignment related to the second factor group." The component misalignment related to the first factor group is a component misalignment that is largely influenced by pixel misalignment in the vertical direction. The component misalignment related to the second factor group is a component misalignment that is largely influenced by pixel misalignment in the horizontal direction.

[0032] Component misalignment other than the seven types shown in Fig. 3, such as sensor pitch rotation or sensor yaw rotation, has a small effect on the parallax error and is therefore not shown in Fig. 3. However, the component misalignment of this embodiment also includes component misalignment other than the seven types shown in Fig. 3.

[0033] FIG. 4 is a diagram showing the configuration of a camera system 100 according to this embodiment.

[0034] The camera system 100 includes an imaging unit 120 and an image processing device 110. The image processing device 110 includes an external input unit 121, a preprocessing unit 122, a storage unit 123, a pixel shift calculation unit 124, a noise removal unit 125, component shift estimation units 126 and 128, a vertical pixel shift correction unit 127, a correction value determination unit 129, a correction unit 130, a distance measurement unit 131, and a control unit 132. The component shift estimation units 126 and 128 include a first component shift estimation unit 126 and a second component shift estimation unit 128.

[0035] The image processing device 110 includes a processor such as a CPU and a memory. The image processing device 110 realizes the functions of an external input unit 121, a preprocessing unit 122, a pixel shift calculation unit 124, a noise removal unit 125, component shift estimation units 126 and 128, a vertical pixel shift correction unit 127, a correction value determination unit 129, a correction unit 130, a distance measurement unit 131, and a control unit 132 by the processor executing a program stored in the memory.

[0036] The imaging unit 120 is connected to the pre-processing unit 122. The imaging unit 120 is a camera capable of acquiring images such as a visible light camera or an infrared camera. In this embodiment, the imaging unit 120 is configured by a stereo camera having a pair of cameras 31, 32. However, the imaging unit 120 may be configured by a multi-camera having three or more cameras as a set, or may be configured by multiple sets of cameras. The pair of cameras 31, 32 simultaneously captures an image of the same object to acquire a set of images, which are output to the pre-processing unit 122. The pixel value of each pixel of the acquired image includes a luminance value.

[0037] The external input unit 121 is connected to the storage unit 123. Pre-set image correction information is input to the external input unit 121 from the outside. The image correction information includes image correction data for correcting image distortion by geometric calibration during manufacturing and for correcting the effect of refraction of incident light by the windshield of the vehicle. Furthermore, the image correction information includes a reference function for estimating pixel shift caused by component misalignment. This reference function includes a reference function for estimating pixel shift caused by component misalignment related to a first factor group (hereinafter also referred to as a "reference function related to the first factor group") and a reference function for estimating pixel shift caused by component misalignment related to a second factor group (hereinafter also referred to as a "reference function related to the second factor group"). This reference function is calculated in advance by an experiment, a simulation, or the like. The external input unit 121 outputs the input image correction information to the storage unit 123.

[0038] The pre-processing unit 122 is connected to the imaging unit 120, the storage unit 123, and the pixel shift calculation unit 124. The pre-processing unit 122 performs pre-processing to correct a set of images acquired by the imaging unit 120, using the image correction data stored in the storage unit 123. The pre-processing unit 122 outputs the set of images that have been pre-processed to the pixel shift calculation unit 124.

[0039] The storage unit 123 is connected to the external input unit 121, the pre-processing unit 122, the pixel shift calculation unit 124, the noise removal unit 125, the component shift estimation units 126 and 128, and the correction unit 130. The storage unit 123 is configured by a non-volatile memory, and can store each piece of information for a long period of time. The storage unit 123 stores the image correction information output from the external input unit 121. The storage unit 123 outputs image correction data included in the stored image correction information to the pre-processing unit 122. The storage unit 123 stores the vertical pixel shift and parallax output from the pixel shift calculation unit 124. The storage unit 123 outputs the stored vertical pixel shift to the noise removal unit 125. The storage unit 123 outputs the reference function included in the stored image correction information to the component shift estimation units 126 and 128. The storage unit 123 outputs the stored vertical pixel shift and parallax to the correction unit 130.

[0040] The pixel shift calculation unit 124 is connected to the preprocessing unit 122 and the storage unit 123. The pixel shift calculation unit 124 calculates the vertical pixel shift and disparity between a pair of images preprocessed by the preprocessing unit 122. The pixel shift calculation unit 124 calculates the vertical pixel shift and disparity by performing stereo matching processing to search for corresponding points between the images. The pixel shift calculation unit 124 outputs the calculated vertical pixel shift and disparity to the storage unit 123.

[0041] The noise removal unit 125 is connected to the storage unit 123 and the first component shift estimation unit 126. The noise removal unit 125 removes noise from the vertical pixel shift stored in the storage unit 123 and outputs the vertical pixel shift to the first component shift estimation unit 126. For example, the noise removal unit 125 acquires each vertical pixel shift of a plurality of sets of images corresponding to a plurality of frames stored in the storage unit 123, and calculates an average value of each vertical pixel shift. The noise removal unit 125 removes invalid vertical pixel shifts outside a predetermined range using a filter that passes vertical pixel shifts within a predetermined range including the calculated average value. Then, the noise removal unit 125 calculates the vertical pixel shift that is fitted to a smooth curve using a method such as high-dimensional polynomial fitting. In this way, the noise removal unit 125 calculates the vertical pixel shift after noise removal. The noise removal unit 125 outputs the calculated vertical pixel shift after noise removal to the first component shift estimation unit 126.

[0042] The first component shift estimation unit 126 is connected to the noise removal unit 125, the storage unit 123, and the vertical pixel shift correction unit 127. The first component shift estimation unit 126 estimates a pixel shift caused by a component shift related to the first factor group (hereinafter also referred to as a "pixel shift related to the first factor group") based on the vertical pixel shift output from the noise removal unit 125. That is, the first component shift estimation unit 126 estimates at least one of the following pixel shifts as the pixel shift related to the first factor group: a pixel shift caused by an angle shift in the roll direction of the other camera 31 relative to the one camera 32 (single camera roll rotation shown in FIG. 3(a)), a pixel shift caused by an angle shift in the pitch direction of the other camera 31 relative to the one camera 32 (single camera pitch rotation shown in FIG. 3(c)), and a pixel shift caused by a vertical position shift of the image sensor 33 relative to the optical element 34 (sensor vertical shift shown in FIG. 3(b)). Estimating the pixel shift related to the first factor group means estimating which of the component shifts related to the first factor group the component shift that is the cause of the parallax error corresponds to, and estimating the pixel shift (including the amount of pixel shift) caused by the corresponding component shift.

[0043] Specifically, the first component shift estimation unit 126 receives the vertical pixel shift output from the noise removal unit 125, and a criterion function related to the first factor group included in the image correction information stored in the storage unit 123. The first component shift estimation unit 126 estimates the pixel shift related to the first factor group by comparing the vertical pixel shift output from the noise removal unit 125 with the criterion function related to the first factor group. The first component shift estimation unit 126 outputs the vertical pixel shift output from the noise removal unit 125, the criterion function related to the first factor group, and the pixel shift related to the first factor group to the vertical pixel shift correction unit 127.

[0044] The vertical pixel shift correction unit 127 is connected to the first component shift estimation unit 126 and the second component shift estimation unit 128. The vertical pixel shift correction unit 127 corrects the vertical pixel shift output from the noise removal unit 125 (i.e., the vertical pixel shift calculated by the pixel shift calculation unit 124) based on the pixel shift related to the first factor group. Specifically, the vertical pixel shift correction unit 127 receives the vertical pixel shift output from the first component shift estimation unit 126, the criterion function related to the first factor group, and the pixel shift related to the first factor group. The vertical pixel shift correction unit 127 corrects the pixel shift related to the first factor group out of the received vertical pixel shift. The vertical pixel shift correction unit 127 outputs the corrected vertical pixel shift and the pixel shift related to the first factor group to the second component shift estimation unit 128.

[0045] The second component shift estimation unit 128 is connected to the vertical pixel shift correction unit 127, the storage unit 123, and the correction value determination unit 129. The second component shift estimation unit 128 estimates a pixel shift caused by a component shift related to a second group of factors excluding the first group of factors (hereinafter also referred to as a "pixel shift related to the second group of factors") based on the vertical pixel shift corrected by the vertical pixel shift correction unit 127. That is, the second component shift estimation unit 128 estimates at least one of the following pixel shifts related to the second group of factors: a pixel shift caused by a horizontal position shift of the imaging element 33 relative to the optical element 34 (sensor horizontal shift shown in FIG. 3(d)), a pixel shift caused by an optical axis shift of the optical element 34 (optical axis change shown in FIG. 3(e)), a pixel shift caused by an angular shift in the yaw direction of the one camera 31 relative to the other camera 32 (single camera yaw rotation shown in FIG. 3(f)), and a pixel shift caused by an angular shift in the yaw direction of the imaging element 33 relative to the optical element 34 (sensor yaw rotation shown in FIG. 3(g)). Estimating the pixel shift related to the second factor group means estimating which of the component shifts related to the second factor group the component shift that is the cause of the parallax error corresponds to, and estimating the pixel shift (including the amount of pixel shift) caused by the corresponding component shift.

[0046] Specifically, the second component shift estimation unit 128 receives the corrected vertical pixel shift output from the vertical pixel shift correction unit 127, the pixel shift related to the first factor group, and each of the criterion functions related to the first factor group and the second factor group included in the image correction information stored in the storage unit 123. The second component shift estimation unit 128 estimates the pixel shift related to the second factor group by comparing the corrected vertical pixel shift with the criterion function related to the second factor group. The second component shift estimation unit 128 outputs the pixel shift related to the first factor group, the pixel shift related to the second factor group, the criterion function related to the first factor group, and the criterion function related to the second factor group to the correction value determination unit 129.

[0047] The correction value determination unit 129 is connected to the second component shift estimation unit 128 and the correction unit 130. The correction value determination unit 129 calculates a correction value for the horizontal pixel shift caused by the component shift, from the pixel shift caused by the estimated component shift, and determines the calculated correction value as the correction value for the parallax error.

[0048] Specifically, the correction value determination unit 129 receives the pixel shifts related to the first factor group, the pixel shifts related to the second factor group, the criterion function related to the first factor group, and the criterion function related to the second factor group output from the second component shift estimation unit 128. The correction value determination unit 129 calculates a correction value for a horizontal pixel shift and a correction value for a vertical pixel shift caused by all component shifts, based on the pixel shifts related to the first factor group, the pixel shifts related to the second factor group, the criterion function related to the first factor group, and the criterion function related to the second factor group. The correction value determination unit 129 determines the correction value for the horizontal pixel shift as a correction value for a parallax error. The correction value determination unit 129 outputs each correction value to the correction unit 130.

[0049] The correction unit 130 is connected to the correction value determination unit 129, the memory unit 123, and the distance measurement unit 131. The correction unit 130 receives each correction value output from the correction value determination unit 129. The correction unit 130 receives the vertical pixel shift and the parallax stored in the memory unit 123. The correction unit 130 corrects the vertical pixel shift and the parallax with a correction value for the vertical pixel shift and a correction value for the parallax error, respectively. The correction unit 130 outputs the corrected parallax to the distance measurement unit 131.

[0050] The distance measuring unit 131 is connected to the correction unit 130 and the control unit 132. The distance measuring unit 131 calculates distance information from the parallax output from the correction unit 130, and outputs the distance information to the control unit 132. The control unit 132 outputs a control signal for issuing a warning or activating the brakes, etc., to a vehicle control device based on the distance information output from the distance measuring unit 131.

[0051] The functions of the image processing device 110 have been described above, and the flow of each of the processes 1 to 4 performed by the image processing device 110 will be described with reference to Figs. 5 to 7. The image processing method of this embodiment is realized by the image processing device 110 performing each of the processes 1 to 4.

[0052] Fig. 5 is a diagram for explaining the flow of processing by the image processing device 110 shown in Fig. 4. Fig. 6 is a diagram for explaining a method for estimating component misalignment related to a first group of factors. Fig. 7 is a diagram for explaining a method for estimating component misalignment related to a second group of factors.

[0053] First, the image processing device 110 captures an image of the object 52 (Process 1). A pair of images PL and PR are images acquired by a pair of cameras 31 and 32, respectively. The image processing device 110 performs stereo matching processing on the pair of images PL and PR in the vertical direction to search for corresponding points between the images PL and PR, and calculates the vertical pixel shift (DPY) of the entire region (Process 2). In this embodiment, the vertical pixel shift is used to estimate pixel shifts caused by various component shifts (Processes 3-1, 3-2, and 3-3). P1, P2, and P3 are respectively the pixel shift amount caused by the single camera roll rotation shown in FIG. 3(a), the pixel shift amount caused by the sensor vertical shift shown in FIG. 3(b), and the pixel shift amount caused by the single camera pitch rotation shown in FIG. 3(c). Let P4, P5, P6, and P7 be the pixel shift amount caused by the sensor horizontal shift shown in Fig. 3(d), the pixel shift amount caused by the optical axis change shown in Fig. 3(e), the pixel shift amount caused by the single camera yaw rotation shown in Fig. 3(f), and the pixel shift amount caused by the sensor yaw rotation shown in Fig. 3(g), respectively. The image processing device 110 estimates the pixel shift amounts P1 to P7 using the vertical pixel shift, and determines a correction value for the parallax error based on the result.

[0054] The image processing device 110 estimates pixel shift amounts P1 to P3 related to the first factor group and pixel shift amounts P4 to P7 related to the second factor group separately. The component shift related to the first factor group has a large effect on pixel shift in the vertical direction. The component shift related to the second factor group has a large effect on pixel shift in the horizontal direction (parallax error direction). In this embodiment, the first component shift estimation unit 126 of the image processing device 110 estimates the pixel shift amounts P1 to P3 related to the first factor group based on the vertical pixel shift (hereinafter also referred to as "first vertical pixel shift") calculated in a first region in which the vertical angle of view is within approximately 1° from the optical axis (Process 3-1).

[0055] FDY0 in Fig. 6 indicates the vertical pixel shift in the first region when a component shift related to the first group of factors and a component shift related to the second group of factors occur simultaneously. That is, FDY0 indicates the vertical pixel shift (first vertical pixel shift) in the first region of DPY shown in Fig. 5.

[0056] The first component shift estimation unit 126 of the image processing device 110 calculates the slope rate kT of the graph representing FDY0, and estimates the pixel shift amount P1 caused by the single camera roll rotation using the slope rate kT. Specifically, the image processing device 110 estimates the pixel shift amount P1 by setting the vertical pixel shift calculated in advance with the angle shift of the single camera roll rotation as a constant value as the reference function ly and calculating the ratio of the slope rate kB of the reference function ly to the slope rate kT. For example, the image processing device 110 sets the vertical pixel shift calculated in advance with the angle shift PO1 of the single camera roll rotation as the reference function ly. The graph 64 shown in FIG. 6 shows the reference function ly(θ,0). The slope rate of the reference function ly(θ,0) is kB. Here, (θ,φ) indicates that the horizontal angle of view is θ and the vertical angle of view is φ. (θ,0) indicates that the horizontal angle of view is θ and the vertical angle of view is 0°, i.e., coincides with the optical axis. The gradient kT of the vertical pixel shift of the image of the object is also shown in Figure 6. In this case, the pixel shift amount P1 can be calculated as P1=(kT / kB)*PO1.

[0057] Then, the image processing device 110 corrects the pixel shift amount P1 caused by the single camera roll rotation. The correction to the pixel shift amount P1 is calculated as (P1 / PO1)*ly. FDY1 shown in Fig. 6 indicates the vertical pixel shift of the first region obtained by correcting the pixel shift amount P1 caused by the single camera roll rotation. OT and WT shown in Fig. 6 are the vertical pixel shift amount when the horizontal angle of view coincides with the optical axis, and the vertical pixel shift amount when the horizontal angle of view is located in the wide-angle region, respectively.

[0058] Next, the image processing device 110 estimates pixel shift amounts P2 and P3 caused by the sensor vertical shift and the single camera pitch rotation using OT and WT. Specifically, the image processing device 110 sets the vertical pixel shift calculated in advance using the position shift PO2 of the sensor vertical shift as a reference function my. The image processing device 110 sets the vertical pixel shift calculated in advance using the angle shift PO3 of the single camera pitch rotation as a reference function ny. The pixel shift amounts P2 and P3 can be calculated by equations (1) and (2), respectively. Graphs 65 and 66 show the reference functions my(θ,0) and ny(θ,0), respectively. P2=(WT-OT) / (my(50,0)-my(0,0)) ···(1) P3=(OT-my(0,0)*P2) / ny(0,0) ···(2)

[0059] In formulas (1) and (2), (50,0) indicates that the horizontal angle of view is 50° and the vertical angle of view is 0°, i.e., coincides with the optical axis. (0,0) indicates that both the horizontal angle of view and the vertical angle of view are 0°, i.e., coincides with the optical axis. The vertical pixel shift amount my(0,0) of the base function my when both the horizontal angle of view and the vertical angle of view are 0° is shown by reference number 62 in graph 65. The vertical pixel shift amount my(50,0) of the base function my when the horizontal angle of view is 50° and the vertical angle of view is 0° is shown by reference number 61 in graph 65. The vertical pixel shift amount ny(0,0) of the base function ny when both the horizontal angle of view and the vertical angle of view are 0° is shown by reference number 63 in graph 66.

[0060] Then, the image processing device 110 corrects the pixel shift amounts P2 and P3 caused by the sensor vertical shift and the single camera pitch rotation. The correction for the pixel shift amount P2 is calculated as (P2 / PO2)*my. The correction for the pixel shift amount P3 is calculated as (P3 / PO3)*my. ADY0 shown in FIG. 6 indicates the vertical pixel shift of the first region obtained by correcting the pixel shift amounts P2 and P3 caused by the sensor vertical shift and the single camera pitch rotation. That is, ADY0 shown in FIG. 6 indicates the vertical pixel shift of the first region in which the pixel shift related to the first factor group has been corrected (process 3-2). The remaining pixel shift related to the second factor group is a pixel shift that has a large influence in the horizontal direction, so it is hardly reflected in the vertical pixel shift of the first region, and therefore the vertical pixel shift amount of ADY0 shown in FIG. 6 becomes approximately 0 (zero).

[0061] Next, the image processing device 110 estimates pixel shift amounts P4 to P7 related to the second group of factors. In this embodiment, the second component shift estimation unit 128 of the image processing device 110 estimates pixel shift amounts P4 to P7 related to the second group of factors based on a vertical pixel shift (hereinafter also referred to as "second vertical pixel shift") calculated in a second region (e.g., vertical angle of view 14°) in which the vertical angle of view is located in a wide-angle region (process 3-3). The second vertical pixel shift can be expressed using a vertical pixel shift obtained by correcting the pixel shift related to the first group of factors. Here, the vertical pixel shift of the entire region in which the pixel shift related to the first group of factors has been corrected is defined as ADY. ADY indicates DPY2 shown in FIG. 5.

[0062] The image processing device 110 sets the vertical pixel shift calculated in advance using the positional deviation PO4 of the sensor horizontal shift as the reference function fy. The image processing device 110 sets the vertical pixel shift calculated in advance using the optical axis deviation PO5 of the optical axis change as the reference function gy. The image processing device 110 sets the vertical pixel shift calculated in advance using the angle deviation PO6 of the single camera yaw rotation as the reference function hy. The image processing device 110 sets the vertical pixel shift calculated in advance using the angle deviation PO7 of the sensor yaw rotation as the reference function jy. Graphs 71, 72, 73, and 74 shown in FIG. 7 respectively show the reference function fy(θ,14), the reference function gy(θ,14), the reference function hy(θ,14), and the reference function jy(θ,14). The vertical pixel shift ADY corrected for the pixel shift related to the first factor group can be calculated by Equation (3) when the vertical angle of view is 14°. ADY(θ,14)≒fy(θ,14)*(P4 / PO4) +gy(θ,14)*(P5 / PO5) +hy(θ,14)*(P6 / PO6) +jy(θ,14)*(P7 / PO7) (3)

[0063] In formula (3), ADY(θ,14) indicates ADY at a vertical angle of view of 14° (second region) among the vertical pixel shifts ADY in which the pixel shifts related to the first factor group have been corrected. ADY(θ,14) is represented by a graph ADY14 shown in FIG. 7. PO4 to PO7 are preset values. The reference functions fy, gy, hy, and jy are also calculated in advance. The values ​​to be obtained are the pixel shift amounts P4 to P7. Simply put, the graph ADY14 can be represented by combining the graphs 71 to 74. That is, the image processing device 110 only needs to estimate the pixel shift amounts P4 to P7 using an arbitrary θ. The image processing device 110 can estimate the pixel shift amounts P4 to P7 by searching for the respective optimal values ​​using an estimation method such as model fitting or the least squares method.

[0064] Finally, the image processing device 110 calculates a total parallax error using the pixel shift amounts P1 to P3 related to the first factor group and the pixel shift amounts P4 to P7 related to the second factor group. The parallax error becomes a correction value CDX for the parallax error (Process 4).

[0065] The image processing device 110 sets the reference function of the horizontal pixel shift caused by the single camera roll rotation as lx. The image processing device 110 sets the reference function of the horizontal pixel shift caused by the sensor vertical shift as mx. The image processing device 110 sets the reference function of the horizontal pixel shift caused by the single camera pitch rotation as nx. The image processing device 110 sets the reference function of the horizontal pixel shift caused by the sensor horizontal shift as fx. The image processing device 110 sets the reference function of the horizontal pixel shift caused by the optical axis change as gx. The image processing device 110 sets the reference function of the horizontal pixel shift caused by the single camera yaw rotation as hx. The image processing device 110 sets the reference function of the horizontal pixel shift caused by the sensor yaw rotation as jx.

[0066] The image processing device 110 can calculate a correction value CDX for the parallax error using the pixel shift amounts P1 to P3 related to the first group of factors and the pixel shift amounts P4 to P7 related to the second group of factors, using equation (4). Each term on the right side of equation (4) corresponds to a correction value for the horizontal pixel shift caused by each component shift. CDX(θ,φ)≒lx(θ,φ)*(P1 / PO1) +mx(θ,φ)*(P2 / PO2) +nx(θ,φ)*(P3 / PO3) +fx(θ,φ)*(P4 / PO4) +gx(θ,φ)*(P5 / PO5) +hx(θ,φ)*(P6 / PO6) +jx(θ,φ)*(P7 / PO7) (4)

[0067] In addition, since the reference functions fy, gy, hy, and jy are similar, the image processing device 110 may appropriately integrate them. For example, since the reference function fx of the sensor horizontal shift and the reference function gy of the optical axis change are similar, these may be integrated into one reference function (for example, fy). For example, since the reference function hy of the single camera yaw rotation and the reference function jy of the sensor yaw rotation are similar, these may be integrated into one reference function (for example, hy). In this way, the image processing device 110 can estimate the pixel shift amount related to the second factor group by using formula (5) instead of formula (3). ADY(θ,φ)≒fy(θ,φ)*(PM45 / PO4) +jy(θ,φ)*(PM67 / PO7)···(5)

[0068] In formula (5), the pixel shift amount PM45 is a pixel shift amount obtained by integrating the pixel shift amount P4 caused by the sensor horizontal shift and the pixel shift amount P5 caused by the optical axis change. The pixel shift amount PM67 is a pixel shift amount obtained by integrating the pixel shift amount P6 caused by the single camera yaw rotation and the pixel shift amount P7 caused by the sensor yaw rotation. This allows the image processing device 110 to reduce the factors of component shift related to the second factor group, and therefore the calculation load required for estimating the pixel shift amount related to the second factor group can be reduced. Note that the integration is not limited to two criterion functions, and integration may be performed into one criterion function or three criterion functions.

[0069] In the above, when estimating the pixel shift caused by the component shift related to the first factor group, the image processing device 110 estimates the pixel shift using the characteristics of the vertical pixel shift in the first region among the pixel shifts. Since the vertical pixel shift amount in the first region and the vertical pixel shift amount in the second region caused by the component shift related to the first factor group are substantially the same, it is also possible to directly correct the vertical pixel shift in the second region by the vertical pixel shift amount in the first region. In the above, the image processing device 110 estimates the pixel shift related to the first factor group and the pixel shift related to the second factor group separately, but even if only the pixel shift related to the first factor group or only the pixel shift related to the second factor group is estimated and a correction value is determined, a correction effect on the parallax error is obtained.

[0070] In order to theoretically verify the above-mentioned parallax error correction effect, a simulation was performed by ray tracing. The simulation conditions (design values) are as follows: ~Calculation conditions~ Lens focal length: Approx. 5mm -Pixel pitch of image sensor 33: approx. 3 μm Baseline length of a pair of cameras 31 and 32: approx. 20 cm Distance to object: 50m Horizontal angle of view: -50° to +50° (+ is right, - is left) Vertical angle of view: -14° to +14° (+ is downward, - is upward) - Angle deviation of single camera roll rotation: 0.05° - Sensor vertical shift misalignment: 2 pixels - Angle deviation of single camera pitch rotation: 0.05° - Sensor horizontal shift misalignment: 2 pixels Optical axis shift: 2 pixels -Single camera yaw rotation angle deviation: 0.05° Sensor Yaw Rotation: 0.05°

[0071] FIG. 8 is a diagram showing the results of verification by simulation.

[0072] Graph 81 in Fig. 8 shows the simulation results of the parallax error at a vertical angle of view of 0°. Reference numeral 82 indicates the parallax error before correction, and reference numeral 83 indicates the parallax error after correction by the image processing device 110 of this embodiment. The parallax error before correction 82 reaches a maximum of 20 pixels in the wide-angle region of a horizontal angle of view of 50°. On the other hand, the parallax error after correction 83 is corrected to approximately 0 pixels in the wide-angle region of a horizontal angle of view of 50°.

[0073] Graph 84 in Fig. 8 is a result of converting graph 81, which is a simulation result of parallax error, into a ranging error rate. Reference numeral 85 indicates the ranging error rate before correction, and reference numeral 86 indicates the ranging error rate after correction by the image processing device 110 of this embodiment. The ranging error rate 85 before correction reaches a maximum of approximately 73% in the wide-angle region of a horizontal angle of view of 50°. On the other hand, the ranging error rate 86 after correction is reduced to a maximum of approximately 4% in the wide-angle region of a horizontal angle of view of 50°.

[0074] 8, it was confirmed that the image processing device 110 of this embodiment can significantly improve the parallax error and distance measurement error. Therefore, the image processing device 110 can realize high-precision distance measurement up to the angle of view of the wide-angle region, and can realize stable vehicle control.

[0075] As described above, the image processing device 110 of this embodiment includes a pixel shift calculation unit 124 that calculates the vertical pixel shift between a pair of images captured by a pair of cameras 31, 32, each of which includes an optical element 34 and an imaging element 33; component shift estimation units 126, 128 that estimate the pixel shift between the images caused by component shift, which is a shift in the position or attitude of at least one of the cameras 31, 32, the optical element 34, and the imaging element 33, based on the calculated vertical pixel shift; and a correction value determination unit 129 that calculates a correction value for the horizontal pixel shift between the images caused by the component shift, from the estimated pixel shift caused by the component shift, and determines the calculated correction value as a correction value for the parallax error between the images. The component misalignment estimation units 126, 128 estimate at least one of the following pixel misalignments caused by component misalignment: pixel misalignment caused by an angular misalignment in the roll direction of one camera 31 relative to the other camera 32; pixel misalignment caused by an angular misalignment in the pitch direction of one camera 31 relative to the other camera 32; and pixel misalignment caused by a vertical positional misalignment of the image sensor 33 relative to the optical element 34.

[0076] As a result, the image processing device 110 can separate pixel shifts caused by various component shifts of the camera that occur while the vehicle is running due to aging or temperature changes into each component shift using the vertical pixel shift, and estimate the amount of pixel shift for each component shift. Then, the image processing device 110 can calculate a correction value for each parallax error from the amount of each pixel shift, and determine a correction value for the total parallax error. Therefore, the image processing device 110 can accurately calculate and correct the parallax error caused by the component shifts of the camera that occur while the vehicle is running due to aging or temperature changes.

[0077] Furthermore, the image processing device 110 of this embodiment regards the pixel shift caused by the angular shift in the roll direction of the one camera 31 relative to the other camera 32, the pixel shift caused by the angular shift in the pitch direction of the one camera 31 relative to the one camera 32, and the pixel shift caused by the vertical position shift of the image pickup element 33 relative to the optical element 34 as pixel shift caused by component shift related to the first group of factors. The image processing device 110 further includes a vertical pixel shift correction unit 127 that corrects the vertical pixel shift calculated by the pixel shift calculation unit 124 based on the pixel shift caused by the component shift related to the first group of factors. The component shift estimation units 126 and 128 estimate the pixel shift caused by the component shift related to the second group of factors excluding the first group of factors based on the vertical pixel shift corrected by the vertical pixel shift correction unit 127.

[0078] As a result, the image processing device 110 can separate the pixel shift caused by the component shift into a pixel shift caused by the component shift related to a first group of factors that are largely influenced by the pixel shift in the vertical direction, and a pixel shift caused by the component shift related to a second group of factors that are largely influenced by the pixel shift in the horizontal direction, and estimate the amount of pixel shift for each component shift. Therefore, the image processing device 110 can easily perform calculations for estimating the amount of pixel shift, and can easily determine a correction value for the parallax error. Therefore, the image processing device 110 can accurately and easily calculate and correct the parallax error caused by the component shift of the camera that occurs during vehicle travel due to aging or temperature changes.

[0079] Furthermore, in the image processing device 110 of this embodiment, the component misalignment estimation units 126, 128 estimate a pixel misalignment caused by a component misalignment related to a first group of factors based on a first vertical pixel shift, which is a vertical pixel shift calculated in a first region in a set of images where the vertical angle of view is within approximately 1° from the optical axis, among the vertical pixel shifts calculated by the pixel misalignment calculation unit 124, and estimate a pixel misalignment caused by a component misalignment related to a second group of factors based on a second vertical pixel shift, which is a vertical pixel shift calculated in a second region in a set of images where the vertical angle of view is larger than that of the first region, among the vertical pixel shifts corrected by the vertical pixel misalignment correction unit 127.

[0080] This allows the image processing device 110 to more easily perform calculations for estimating the amount of pixel shift caused by the component shift related to the first group of factors and the amount of pixel shift caused by the component shift related to the second group of factors. Therefore, the image processing device 110 can more easily and accurately calculate and correct the parallax error caused by the component shift of the camera that occurs while the vehicle is running due to aging or temperature change.

[0081] Furthermore, in the image processing device 110 of this embodiment, the component shift estimation units 126, 128 estimate the pixel shift caused by the angular shift in the roll direction of one camera 31 relative to the other camera 32, using the gradient rate of the first vertical pixel shift calculated in the first region.

[0082] This allows the image processing device 110 to easily estimate pixel shifts caused by angular shifts in the roll direction of the one camera 31 relative to the other camera 32. Therefore, the image processing device 110 can more easily perform calculations for estimating the amount of pixel shift caused by this type of component shift. Therefore, the image processing device 110 can accurately and more easily calculate and correct parallax errors caused by component shifts of the cameras that occur while the vehicle is running due to aging or temperature changes.

[0083] Furthermore, in the image processing device 110 of this embodiment, the component misalignment estimation units 126, 128 use the value of the first vertical pixel shift calculated in the first region, the value of which is the first vertical pixel shift for which the horizontal angle of view approximately coincides with the optical axis, to estimate the pixel misalignment caused by the angle misalignment in the pitch direction of the one camera 31 relative to the other camera 32, and the pixel misalignment caused by the vertical positional misalignment of the image sensor 33 relative to the optical element 34.

[0084] This allows the image processing device 110 to easily estimate pixel shifts caused by angle shifts in the pitch direction of the one camera 31 relative to the other camera 32, and pixel shifts caused by position shifts in the vertical direction of the image pickup element 33 relative to the optical element 34. Therefore, the image processing device 110 can more easily perform calculations for estimating the amount of pixel shift caused by this type of component shift. Therefore, the image processing device 110 can accurately and more easily calculate and correct parallax errors caused by component shifts of the cameras that occur while the vehicle is running due to aging or temperature changes.

[0085] Moreover, the image processing method of this embodiment is realized by the image processing device 110 performing the above-mentioned processes 1 to 4.

[0086] That is, the image processing method of this embodiment includes a pixel shift calculation process (process 2) for calculating a vertical pixel shift between a pair of images captured by a pair of cameras 31, 32 each including an optical element 34 and an imaging element 33, a component shift estimation process (processes 3-1 to 3-3) for estimating a pixel shift between images caused by a component shift, which is a shift in position or attitude of at least one of the cameras 31, 32, the optical element 34, and the imaging element 33, based on the calculated vertical pixel shift, and a correction value determination process (process 4) for calculating a correction value for a horizontal pixel shift between images caused by the component shift from the estimated pixel shift caused by the component shift, and determining the calculated correction value as a correction value for a parallax error between the images. The component shift estimation process estimates at least one of the pixel shift caused by an angular shift in the roll direction of one camera 31 relative to the other camera 32, the pixel shift caused by an angular shift in the pitch direction of the other camera 31 relative to the one camera 32, and the pixel shift caused by a vertical position shift of the imaging element 33 relative to the optical element 34, as the pixel shift caused by the component shift.

[0087] As a result, the image processing method of this embodiment can separate pixel shifts caused by various component shifts of the camera that occur while the vehicle is running due to aging or temperature changes into each component shift using the vertical pixel shift, and estimate the amount of each pixel shift for each component shift.The image processing method of this embodiment can then calculate a correction value for each parallax error from each pixel shift amount, and determine a correction value for the total parallax error.Therefore, the image processing method of this embodiment can accurately calculate and correct the parallax error caused by component shifts of the camera that occur while the vehicle is running due to aging or temperature changes.

[0088] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. In addition, it is possible to replace a part of the configuration of a certain embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of a certain embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0089] In addition, the above-mentioned configurations, functions, processing units, or processing means may be realized in part or in whole by hardware, for example, by designing them in an integrated circuit. The above-mentioned configurations or functions may be realized by software, in which a processor interprets and executes a program that realizes each function. Information such as a program, table, or file that realizes each function may be stored in a recording device such as a memory, a hard disk, or an SSD (solid state drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0090] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0091] 31, 32... pair of cameras, 31... other camera, 32... one camera, 33... imaging element, 34... optical element, 110... image processing device, 124... pixel shift calculation unit, 126... first component shift estimation unit (component shift estimation unit), 127... vertical pixel shift correction unit, 128... second component shift estimation unit (component shift estimation unit), 129... correction value determination unit

Claims

1. a pixel shift calculation unit that calculates a vertical pixel shift between a pair of images captured by a pair of cameras each including an optical element and an image sensor; a component shift estimation unit that estimates a pixel shift between the images caused by a component shift, which is a shift in position or orientation of at least one of the camera, the optical element, and the image sensor, based on the calculated vertical pixel shift; a correction value determination unit that calculates a correction value for a horizontal pixel shift between the images caused by the component shift from the estimated pixel shift caused by the component shift, and determines the calculated correction value as a correction value for a parallax error between the images, The component misalignment estimation unit estimates, as the pixel misalignment caused by the component misalignment, at least one of the pixel misalignment caused by an angular misalignment in a roll direction of one camera relative to the other camera, the pixel misalignment caused by an angular misalignment in a pitch direction of the other camera relative to the one camera, and the pixel misalignment caused by a positional misalignment in a vertical direction of the imaging element relative to the optical element.

13. An image processing device comprising:

2. The pixel shift caused by an angular shift in the roll direction of the one camera relative to the other camera, the pixel shift caused by an angular shift in the pitch direction of the one camera relative to the other camera, and the pixel shift caused by a positional shift in the vertical direction of the imaging element relative to the optical element are defined as the pixel shift caused by the component shift related to a first group of factors: the image processing device further includes a vertical pixel shift correction unit that corrects the vertical pixel shift calculated by the pixel shift calculation unit on the basis of the pixel shift caused by the component shift related to the first group of factors; The component shift estimation unit estimates the pixel shift caused by the component shift related to a second group of factors excluding the first group of factors, based on the vertical pixel shift corrected by the vertical pixel shift correction unit.

2. The image processing device according to claim 1,

3. The component deviation estimation unit is estimating the pixel shift caused by the component shift related to the first group of factors based on a first vertical pixel shift, which is the vertical pixel shift calculated in a first region in the set of images where a vertical angle of view is within approximately 1° from an optical axis, among the vertical pixel shifts calculated by the pixel shift calculation unit; The pixel shift caused by the component shift related to the second group of factors is estimated based on a second vertical pixel shift, which is the vertical pixel shift calculated in a second region in which the vertical angle of view is larger than that of the first region, out of the vertical pixel shifts corrected by the vertical pixel shift correction unit.

3. The image processing device according to claim 2.

4. The component shift estimation unit estimates the pixel shift caused by an angular shift in the roll direction of the one camera relative to the other camera, using a gradient rate of the first vertical pixel shift calculated in the first region.

4. The image processing device according to claim 3.

5. The component shift estimation unit estimates the pixel shift caused by an angle shift in the pitch direction of the one camera relative to the other camera and the pixel shift caused by a positional shift in the vertical direction of the imaging element relative to the optical element, using a value of the first vertical pixel shift calculated in the first region, the value of the first vertical pixel shift having a horizontal angle of view substantially coinciding with an optical axis.

4. The image processing device according to claim 3.

6. A pixel shift calculation process for calculating a vertical pixel shift between a pair of images captured by a pair of cameras each including an optical element and an image sensor; a component shift estimation process for estimating a pixel shift between the images caused by a component shift, which is a shift in position or orientation of at least one of the camera, the optical element, and the image sensor, based on the calculated vertical pixel shift; a correction value determination process for calculating a correction value for a horizontal pixel shift between the images caused by the component shift from the estimated pixel shift caused by the component shift, and determining the calculated correction value as a correction value for a parallax error between the images, The component misalignment estimation process estimates, as the pixel misalignment caused by the component misalignment, at least one of the pixel misalignment caused by an angular misalignment in a roll direction of one camera relative to the other camera, the pixel misalignment caused by an angular misalignment in a pitch direction of the other camera relative to the one camera, and the pixel misalignment caused by a positional misalignment in a vertical direction of the imaging element relative to the optical element.

13. An image processing method comprising:

7. The pixel shift caused by an angular shift in the roll direction of the one camera relative to the other camera, the pixel shift caused by an angular shift in the pitch direction of the one camera relative to the other camera, and the pixel shift caused by a positional shift in the vertical direction of the imaging element relative to the optical element are defined as the pixel shift caused by the component shift related to a first group of factors: the image processing method further includes a vertical pixel shift correction process that corrects the vertical pixel shift calculated by the pixel shift calculation process on the basis of the pixel shift caused by the component shift related to the first group of factors; The component shift estimation process estimates the pixel shift caused by the component shift related to a second group of factors excluding the first group of factors, based on the vertical pixel shift corrected by the vertical pixel shift correction process.

7. The image processing method according to claim 6.

8. The part deviation estimation process includes: estimating the pixel shift caused by the component shift related to the first group of factors based on a first vertical pixel shift, which is the vertical pixel shift calculated in a first region in the set of images where the vertical angle of view is within approximately 1° from the optical axis, among the vertical pixel shifts calculated by the pixel shift calculation process; The pixel shift caused by the component shift related to the second group of factors is estimated based on a second vertical pixel shift, which is the vertical pixel shift calculated in a second region in which the vertical angle of view is larger than that of the first region, out of the vertical pixel shifts corrected by the vertical pixel shift correction process.

8. The image processing method according to claim 7.

9. The component shift estimation process estimates the pixel shift caused by an angular shift in the roll direction of the one camera relative to the other camera, using a gradient rate of the first vertical pixel shift calculated in the first region.

9. The image processing method according to claim 8.

10. The component shift estimation process estimates the pixel shift caused by an angle shift in the pitch direction of the one camera relative to the other camera and the pixel shift caused by a position shift in the vertical direction of the image pickup element relative to the optical element, using a value of the first vertical pixel shift calculated in the first region, the value of the first vertical pixel shift having a horizontal angle of view substantially coinciding with an optical axis.

9. The image processing method according to claim 8.

Images