In-vehicle image processing device and calibration method thereof

Abstract

The present invention optimizes the implementation timing of calibration and limits the stoppage period of ADAS. The present invention is a vehicle-mounted image processing device that is mounted in a vehicle and processes an exterior image acquired by a camera, wherein the device includes: a temperature sensor that detects the current temperature; a temperature predicting unit that predicts a future temperature on the basis of time sequence data of temperatures; a distance measurement error predicting unit that predicts a future distance measurement error on the basis of the temperature prediction; a calibration timing determining unit that determines a timing at which calibration using the image can be implemented, on the basis of vehicle operation information or external world recognition information; and a calibration start determining unit that, when the calibration timing determining unit has determined that the calibration can be implemented in a remaining time until the point in time that the distance measurement error predicted by the distance measurement error predicting unit is predicted to exceed a threshold, evaluates whether or not to start the calibration using the image, on the basis of the current temperature or the remaining time.

Description

[Technical Field]

[0001] The present invention relates to an in-vehicle image processing device used in an advanced driver assistance system, and a calibration method thereof. [Background technology]

[0002] Some vehicles equipped with an advanced driver assistance system (hereinafter referred to as "ADAS") use a stereo camera as an external sensor to recognize the environment outside the vehicle. In this on-board stereo camera, the internal temperature rises due to internal heat generation after startup, and the structures such as the lens, circuit board, and housing are thermally deformed. This thermal deformation causes the optical axis shift of the lens. A leak occurs.

[0003] In recent years, in-vehicle stereo cameras have become increasingly miniaturized, with image sensors such as CMOS image sensors becoming smaller and smaller in size per pixel. Therefore, even a slight misalignment of the optical axis due to a slight temperature change can degrade the captured image and have a significant impact on the parallax image generated from the left and right images captured synchronously by the left and right cameras. In that case, a large error will occur in the distance measurement calculation of the environment outside the vehicle, which may ultimately have a negative impact on vehicle control by ADAS.

[0004] Therefore, conventional stereo cameras use image processing technology based on the stereo method to appropriately correct any optical axis misalignment that occurs.The stereo method is a method in which two cameras are used to capture images of the target object from different viewpoints, and the distance to the target object is calculated using the principle of triangulation.

[0005] Here, the abstract of Patent Document 1 describes the problem as "improving the accuracy of distance measurement to an object using a small stereo camera," and also states that the solution is to "acquire the temperature of the stereo camera and correct the base line length, which is a camera parameter, in accordance with the temperature. In one embodiment, sensor module 110 in camera head unit 100 is composed of stereo sensor 101 and thermistor 102. SEEPROM 104 stores calibration data, base line length B0, and temperature T0 at the time of calibration. When the stereo camera is in use, CPU 130 in image processing unit 120 acquires the calibration data, base line length B0, and temperature T0 from SEEPROM 104, and also stores the current temperature T0 as the output value of the thermistor 102. R The CPU 130 sets the calibration data directly in the correction circuit 122, but the base line length B0 is obtained by R The corresponding baseline length B R and set it in the distance measurement circuit 123." In this way, in Patent Document 1, the temperature of the stereo camera is acquired, and the base line length, which is one of the optical system parameters, is corrected according to the temperature.

[0006] Immediately after starting the vehicle (i.e., immediately after starting the stereo camera), the internal temperature of the stereo camera fluctuates significantly, making it more likely that the optical axis will shift due to thermal deformation of the structure. Furthermore, fluctuations in the internal temperature of the stereo camera are also affected by external factors such as the temperature inside the vehicle cabin, and do not necessarily follow a uniform trend each time. Therefore, it is not possible to uniformly schedule the timing of calibration to suppress distance measurement errors to a predetermined level or less. For this reason, as explained in paragraph 0048 and elsewhere in Patent Document 1, the baseline length, which is one of the optical system parameters, is updated as needed based on the periodically measured temperature inside the camera head unit. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-52335 Summary of the Invention [Problem to be solved by the invention]

[0008] When the calibration target is a baseline length whose fluctuation in response to temperature changes can be predicted, as in Patent Document 1, the baseline length can be calculated based on the measured temperature, so there is no need to stop the ADAS during calibration.

[0009] However, because the optical axis misalignment described above fluctuates in a complex manner in response to temperature changes, when the optical axis misalignment is to be calibrated, precise calibration using image processing technology must be performed at the appropriate time, which poses a problem in that the ADAS cannot be used during the calibration.

[0010] Therefore, an object of the present invention is to provide an in-vehicle image processing device and a calibration method thereof that can reduce the downtime of ADAS by optimizing the timing of calibration using image processing technology. [Means for solving the problem]

[0011] In order to solve the above problem, the present invention provides an in-vehicle image processing device that is mounted on a vehicle and processes images of the outside world captured by a camera, the in-vehicle image processing device having: a temperature sensor that detects the current temperature; a temperature prediction unit that predicts a future temperature based on time-series data of the temperature; a ranging error prediction unit that predicts a future ranging error based on the predicted temperature; a calibration timing determination unit that determines whether it is time to perform calibration using the image based on vehicle operation information or outside world recognition information; and a calibration start determination unit that determines whether to start calibration using the image based on the current temperature or the remaining time when the calibration timing determination unit determines that the calibration can be performed within the remaining time until the ranging error predicted by the ranging error prediction unit is predicted to exceed a threshold value. [Effects of the Invention]

[0012] According to the in-vehicle image processing device and the calibration method thereof of the present invention, the period during which the ADAS is stopped can be reduced by optimizing the timing of performing calibration using image processing technology. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 2 is a plan view of the in-vehicle stereo camera as seen from above. [Figure 2] FIG. 2 is a diagram illustrating one cause of optical axis misalignment in the left camera of FIG. 1. [Figure 3] 10 is a graph illustrating optical axis deviation in each temperature range. [Figure 4] FIG. 2 is a functional block diagram of the in-vehicle image processing device according to the first embodiment. [Figure 5] 1 is an example of a database stored in a memory. [Figure 6] FIG. 5 is a detailed configuration diagram of the calibration determination unit in FIG. 4. [Figure 7] 3 is a processing flowchart of the vehicle-mounted image processing device of the first embodiment. [Figure 8] 1 is an example of a method for detecting feature points in image data. [Figure 9] An example of how to track feature points in image data DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of an on-vehicle image processing device and a calibration method thereof according to the present invention will be described with reference to the drawings. [Example]

[0015] First, optical axis misalignment that occurs in the vehicle-mounted stereo camera 100 will be described with reference to FIGS.

[0016] 1 is a plan view of an in-vehicle stereo camera 100 equipped with a left camera 11 and a right camera 12, viewed from above, in which two vertical dashed lines on the left and right indicate the lens center axes of the left and right cameras. In addition, two pairs of dashed lines on the left and right illustrate the range of misalignment of the optical axes of the left and right cameras.

[0017] When the driver starts the vehicle engine, various circuits in the in-vehicle stereo camera 100 are energized and generate heat, causing the temperature inside the camera to gradually rise. As the temperature inside the camera rises, the optical axes of the left and right cameras become misaligned from the original lens center axis, causing optical axis misalignment.

[0018] FIG. 2 illustrates one cause of optical axis misalignment of the left camera 11 in FIG. 1. As shown here, when the internal temperature of the in-vehicle stereo camera 100 rises, each of the multiple lenses 11a built into the left camera 11 shifts from its reference position, deforming the lens central axis of the optical system and causing optical axis misalignment. This optical axis misalignment corresponds to a shift in the lens central axis position on the image sensor 11b, so an image captured after the occurrence of optical axis misalignment has a lower image quality than an image captured without the optical axis misalignment. Note that the cause of optical axis misalignment is not limited to the example in FIG. 2. For example, as shown in FIG. 1, expansion or twisting of the camera housing due to an increase in temperature of the camera housing, or fluctuations in the baseline length, can also be factors that cause optical axis misalignment.

[0019] 3 illustrates the relationship between the change in temperature inside the camera and the change in optical axis misalignment over time after the on-board stereo camera 100 is started up. As shown in the upper graph, in the unstable temperature region immediately after the on-board stereo camera 100 is started up, the expansion and twisting of the camera housing (see FIG. 1) and the lens central axis (see FIG. 2) change from moment to moment, causing the optical axis misalignment to fluctuate significantly up and down over time, as shown in the lower graph. In contrast, in the stable temperature region after a predetermined time has elapsed since the camera was started up (e.g., after 30 minutes), the temperature inside the camera is stable, so the expansion and twisting of the housing and the changes in the lens central axis converge, and the optical axis misalignment stabilizes at a predetermined amount.

[0020] 3, the appropriate interval for calibrating the optical axis misalignment differs between the temperature unstable region where the optical axis misalignment is unstable and the temperature stable region where the optical axis misalignment is stable. Based on the above explanation, the details of the in-vehicle image processing device 1 of this embodiment will be described.

[0021] <In-vehicle image processing device 1> 4 is a functional block diagram showing an example of the overall configuration of an in-vehicle image processing device 1 according to this embodiment. As shown in this figure, the in-vehicle image processing device 1 has a camera unit 10 and an image processing unit 20, and is also connected to a vehicle speed sensor 30. Each part will be explained in turn.

[0022] <<Camera Unit 10>> The camera unit 10 is disposed on the left side of the camera housing and outputs left image data P L a left camera 11 that captures an image of the right side of the camera housing, and a right camera 12 that captures an image of the left side of the camera housing. R The stereo camera has a right camera 12 that captures an image of the object, and a temperature sensor 13 that measures the temperature inside the camera housing and outputs temperature data T. L and right image data P R The imaging ranges of the left and right images are assumed to overlap at least partially, and a parallax image can be generated using the left and right images.

[0023] <<Image processing unit 20>> The image processing unit 20 includes a temperature prediction unit 21, a memory 22, an optical axis deviation prediction unit 23, a ranging error prediction unit 24, a calibration determination unit 25, a first calibration unit 26, and a second calibration unit 27. Specifically, the image processing unit 20 is a computer equipped with a calculation device such as a CPU, a storage device such as a semiconductor memory, and hardware such as a communication device. The calculation device executes a predetermined program to realize each functional unit such as the temperature prediction unit 21. Below, each unit will be described in detail, omitting such well-known techniques as appropriate.

[0024] The temperature prediction unit 21 uses the time-series temperature data T sequentially acquired from the temperature sensor 13 to calculate the gradient of the temperature inside the camera, and also finds the rate of change and rate of increase of the current temperature inside the camera.

[0025] A database showing how optical system parameters such as the baseline length and the central axes of the left and right lenses change with temperature is stored in memory 22. Fig. 5 shows an example of a temperature table that records the relationship between the deviation of the baseline length and the temperature inside the camera housing.

[0026] The optical axis deviation prediction unit 23 compares the temperature gradient calculated by the temperature prediction unit 21 with the database in the memory 22, and predicts a future optical axis deviation curve (optical axis deviation over time).

[0027] The distance measurement error prediction unit 24 converts the optical axis deviation predicted by the optical axis deviation prediction unit 23 into a distance measurement error, and predicts a future distance measurement error curve (a distance measurement error over time).

[0028] As shown in FIG. 6, the calibration determination unit 25 includes a calibration timing determination unit 25a and a calibration start determination unit 25b.

[0029] The calibration timing determination unit 25a determines an appropriate timing for calibrating the optical axis misalignment by image processing, based on the ranging error curve acquired from the ranging error prediction unit 24. For example, if the time when the ranging error will exceed an unacceptable range is predicted to be three minutes from now, the calibration timing determination unit 25a schedules the calibration to be performed within the remaining time of one minute immediately before that time (i.e., one minute from two to three minutes from now). This allows the calibration to be performed before the ranging error exceeds the allowable range. Note that the advantage of performing the calibration immediately before the time when the ranging error is predicted to exceed an unacceptable range is that by extending the period when calibration is not performed as long as possible, the number of times the ADAS is stopped and the duration of the stop of the ADAS can be reduced as much as possible.

[0030] The calibration start determination unit 25b commands the first calibration unit 26 or the second calibration unit 27 to start calibration based on the calibration timing determined by the calibration timing determination unit 25a and the vehicle speed information acquired from the vehicle speed sensor 30. Details of this processing will be described later with reference to FIG.

[0031] The first calibration unit 26 performs precise correction of optical axis misalignment using image processing based on the stereo method at the timing designated by the calibration determination unit 25 .

[0032] The second calibration unit 27 performs simple optical axis misalignment correction using predictive calculation at the timing designated by the calibration determination unit 25.

[0033] The processing results of the first calibration unit 26 or the second calibration unit 27 are stored in the memory 22, and the most recent processing result is used to perform distance measurement calculations for the environment outside the vehicle. This allows the environment to be recognized based on the most recent optical system parameters, making it possible to continue appropriate ADAS operation. Note that the processing results stored in the memory 22 may be averaged and used for distance measurement calculations for the environment outside the vehicle.

[0034] <Flowchart of calibration process according to this embodiment> Here, the calibration process according to this embodiment will be described with reference to the flowchart of FIG.

[0035] First, in step S1, the image processing unit 20 receives left image data P from the left camera 11. L The right image data P R Temperature data T is acquired from the temperature sensor 13. Although not shown in FIG. 1, the various data acquired in this step is temporarily stored in the memory 22.

[0036] In step S2, the temperature prediction unit 21 calculates the temperature gradient of the temperature inside the camera based on the time-series temperature data T stored in the memory 22.

[0037] In step S3, the optical axis deviation prediction unit 23 refers to the temperature gradient calculated in step S2 and the database acquired from the memory 22 to predict an optical axis deviation curve that indicates the amount of optical axis deviation over time.

[0038] In step S4, the distance measurement error predicting unit 24 predicts a distance measurement error curve that indicates the distance measurement error deviation amount with respect to time change, based on the optical axis deviation curve predicted in step S3.

[0039] In step S5, the calibration determination unit 25 determines an appropriate timing for calibrating the optical axis misalignment based on the distance measurement error curve predicted in step S4. In the above example, the appropriate timing is one minute between two and three minutes from the present time.

[0040] In step S6, the calibration determination unit 25 determines whether the vehicle is stopped. If the vehicle is stopped, the process proceeds to step S7, and if not (if the vehicle is moving), the process proceeds to step S10. When determining whether the vehicle is stopped, vehicle speed information acquired from the vehicle speed sensor 30 may be used, or information such as engine speed that is useful for estimating whether the vehicle is stopped may be used.

[0041] The reason for providing this step (vehicle stop determination) is that when performing precise calibration using image processing by the first calibration unit 26, the ADAS function needs to be stopped, and therefore, taking into consideration the risk of an accident due to the ADAS being stopped, calibration using image processing is performed as soon as possible when the vehicle is stopped.

[0042] In step S7, the calibration determination unit 25 determines whether the timing of the vehicle stop is outside the calibration period set in step S5. If it is outside the set period, the process proceeds to step S8, and if not (if it is within the set period), the process proceeds to step S9.

[0043] For example, if the timing of calibration by the first calibration unit 26 is set to one minute, two to three minutes after the timing setting, if the vehicle stops one minute and 30 seconds later, it is determined that calibration is not possible, and the process proceeds to step S8. On the other hand, if the vehicle stops two minutes and 30 seconds later, it is determined that calibration is possible, and the process proceeds to step S9.

[0044] In step S8, the calibration determination unit 25 does not issue a calibration command during this vehicle stop. This postpones the calibration. After step S8, the process returns to step S6, and the determination in step S7 is performed again at the next vehicle stop.

[0045] In step S9, the calibration determination unit 25 commands the first calibration unit 26 to perform calibration of optical axis misalignment by image processing. As described above, the ADAS must be stopped during precise calibration of optical axis misalignment by image processing, but if the process reaches this step after steps S6 and S7, the vehicle is stopped, so there is no disadvantage in terms of safety to stopping the ADAS and performing calibration.

[0046] In step S10, the calibration determination unit 25 determines whether the ADAS can be stopped even while the vehicle is traveling. If the ADAS can be stopped, the process proceeds to step S9; otherwise (if the ADAS cannot be stopped), the process proceeds to step S11. For example, if the vehicle is traveling slowly and there is no detected object within 5 meters ahead of the vehicle, stopping the ADAS for a while is considered to pose little disadvantage in terms of safety, so in step S9, the first calibration unit 26 is activated and a precise calibration of the optical axis misalignment is performed by image processing.

[0047] Details of the determination in step S10 will now be described with reference to Figures 8 and 9. One calibration technique is an automatic adjustment (calibration while driving) technique that uses feature point detection of an object.

[0048] In this method, first, feature points (corners with large brightness differences, etc.) of stationary objects such as signs, road shoulders, and white lines are detected based on the brightness information of each pixel, as shown in Fig. 8. Then, as shown in Fig. 9, the detected feature points are tracked until the feature point information is lost (moving objects such as vehicles are detected but not tracked), and feature point coordinate information, disparity information, etc. are stored as time-series information. Based on the acquired time-series information, the amount of movement of the vehicle and the amount of change in distance to the feature point ahead are calculated, and the disparity shift is calculated to determine the amount of correction.

[0049] If it is determined that there is no problem in stopping the camera while the vehicle is traveling using the feature point detection of the object in this automatic adjustment, the mode is instantly switched from automatic adjustment to calibration by image processing according to the timing of calibration determined by the calibration judgment unit 25 and the remaining time, and the processing of step S9 is carried out.

[0050] For example, if the vehicle's safety condition is ensured and the processing load rate of the object detection system is 20% or less (conditions are met, such as the vehicle moving parallel, not turning the steering wheel, traveling in a straight line, and there are no detectable three-dimensional objects ahead), it is determined that there is no problem with stopping the camera, and calibration using image processing (step S9) can be performed.

[0051] In step S11, the calibration determination unit 25 commands the second calibration unit 27 to perform a simple calibration of the optical axis misalignment by predictive calculation using the database registered in the memory 22. For example, even if the vehicle cannot stop within the calibration period set in step S5 and the ADAS cannot be stopped, such as when driving on a highway, the optical axis misalignment progresses due to temperature changes inside the camera housing immediately after starting the in-vehicle stereo camera 100. Therefore, in such a case, calibration of the optical axis misalignment by predictive calculation is performed, which can obtain a certain degree of correction accuracy, although the correction accuracy is inferior to that of a precise calibration using image processing.

[0052] Note that the calibration in this step has lower correction accuracy than the calibration in step S9, so it is desirable to set a predetermined limit (for example, up to two consecutive times) on the number of consecutive executions of the calibration in this step.

[0053] In this embodiment, the main cause of optical axis misalignment is explained as temperature characteristics, but strictly speaking, other factors can also be considered, such as manufacturing variations, deterioration over time, camera installation errors, etc. Therefore, since the condition of the vehicle (after shipment, long-term use, etc.) may affect optical axis misalignment, it is possible to perform calibration using image processing immediately after starting the camera and remove these errors before optical axis misalignment occurs due to temperature changes.

[0054] In this way, in this embodiment, it is possible to respond to various temperature changes by grasping future optical axis misalignment. For example, it is possible to perform local calibration due to sudden temperature changes, or conversely, it is possible to reduce the number of calibrations when the temperature is stable. This makes it possible to perform calibration at the optimal timing while taking into consideration the vehicle's driving environment. In addition, it is possible to manage the distance measurement error so that it does not exceed a threshold, and it is possible to perform calibration only the minimum number of times necessary, which also contributes to saving CPU resources.

[0055] According to the in-vehicle image processing device of this embodiment described above, the period during which the ADAS is stopped can be reduced by optimizing the timing of performing calibration using image processing technology. [Example]

[0056] Next, a second embodiment of the present invention will be described. Note that a duplicated description of points common to the first embodiment will be omitted.

[0057] In this embodiment, the determination of the remaining time after the vehicle is stopped as described in the first embodiment will be explained by replacing it with the temperature change. The determination is made based on the percentage change in the current temperature acquired by the temperature sensor 13 from the temperature acquired last time.

[0058] For example, if the vehicle is stopped within a set time remaining from the timing for performing calibration and a temperature change rate of 3% or more is set as the range in which calibration can be performed, calibration is determined to be possible if the change rate of the current temperature from the previously acquired temperature is 5% (5% ≥ 3%). Note that the previously acquired temperature information is stored in memory 22.

[0059] Furthermore, if it is determined that calibration is not possible based on the temperature change rate, the timing of calibration is postponed once, as in the operation shown in Example 1, and the process waits until the next time the vehicle stops, and then the process is repeated from the determination of the vehicle's running state using the vehicle speed sensor 30. [Explanation of symbols]

[0060] 100... In-vehicle stereo camera, 1... In-vehicle image processing device, 10... Camera unit, 11... Left camera, 11a... Lens, 11b... Image sensor, 12... Right camera, 13... Temperature sensor, 20... Image processing unit, 21... Temperature prediction unit, 22... Memory, 23... Optical axis deviation prediction unit, 24... Distance measurement error prediction unit, 25... Calibration determination unit, 26... First calibration unit, 27... Second calibration unit, 30... Vehicle speed sensor

Claims

1. An in-vehicle image processing device that is mounted on a vehicle and processes an image of the outside world acquired by a camera, a temperature sensor for detecting a current temperature; a temperature prediction unit that predicts a future temperature based on the time series data of the temperature; a ranging error prediction unit that predicts a future ranging error based on the predicted temperature; a calibration timing determination unit that determines whether it is a timing when calibration using the image can be performed based on vehicle operation information or external environment recognition information; and a calibration start determination unit that determines whether to start calibration using the image based on the current temperature or the remaining time when the calibration timing determination unit determines that the calibration can be performed within the remaining time until the time when the ranging error predicted by the ranging error prediction unit is predicted to exceed a threshold.

2. 2. The vehicle-mounted image processing device according to claim 1, an in-vehicle image processing device that, if it is not determined that calibration using the image can be performed within the remaining time, performs calibration based on the predicted distance measurement error;

3. 3. The vehicle-mounted image processing device according to claim 2, an in-vehicle image processing device that, when calibration based on the predicted ranging error has been performed consecutively a predetermined number of times or more, performs calibration using the image regardless of the judgment of the calibration start judgment unit;

4. 2. The vehicle-mounted image processing device according to claim 1, the calibration start determination unit determines whether to start calibration using the image based on the current temperature or the remaining time when the calibration timing determination unit determines that the calibration can be performed within a predetermined time before the time when the ranging error predicted by the ranging error prediction unit is predicted to exceed a threshold.

5. 2. The vehicle-mounted image processing device according to claim 1, The vehicle operation information is a vehicle speed or an engine rotation speed.

1. An in-vehicle image processing device comprising:

6. A calibration method for an in-vehicle image processing device that is mounted on a vehicle and processes an image of the external environment acquired by a camera, comprising: a temperature prediction step of predicting a future temperature based on time series data of temperatures detected by the temperature sensor; a ranging error prediction step of predicting a future ranging error based on the predicted temperature; a calibration timing determination step of determining whether it is time to perform calibration using the image based on vehicle operation information or external environment recognition information; and a calibration start determination step of determining whether to start calibration using the image based on the current temperature or the remaining time when it is determined that the calibration can be performed within the remaining time until the time when the predicted ranging error is predicted to exceed a threshold.

Images