Method of evaluating lighting settings for vehicle system calibration

Abstract

The invention relates to a method for evaluating lighting settings for vehicle system calibration. In one aspect, the invention utilizes a machine vision process that images a calibration target in an illumination setting using an imaging device and processes the image in order to identify an illumination problem. If a lighting problem is identified, the lighting settings are modified and / or the target is relocated relative to the vehicle and / or the lighting settings, and then the target is reimaged.

Description

Technical Field

[0001] The present invention relates to a method for evaluating lighting settings for vehicle system calibration. Background Art

[0002] Vehicle advanced driver assistance systems (ADAS), such as cameras mounted on the windshield, need to be recalibrated from time to time. For example, if the camera is mounted on a windshield that is replaced (due to breakage or other reasons), then after installing the replacement windshield and mounting the camera to the replacement windshield, the camera needs to be recalibrated. Generally, this may involve the camera imaging a specific calibration target (usually a target board) provided for the make and / or model of the vehicle.

[0003] During the ADAS camera calibration process, the calibration target board is used as a reference point to ensure that the camera is aligned with the driving axis, indicating that the camera can sense a complete view of the road and upcoming vehicles. Generally, if the calibration board is not well illuminated, the ADAS camera will not be able to correctly detect the target board pattern. This means that the calibration system will not be able to calibrate because it relies on the detected pattern and the respective positions of the patterns to calibrate the ADAS camera. Summary of the Invention

[0004] Therefore, in order to avoid calibration process failures, it is important to evaluate the target board for possible lighting problems such as uneven lighting, shadows, and saturation. The aim is to perform this process before calibration, and if there are lighting problems, reposition the board to a space where the lighting of the target board no longer poses a problem for the calibration system. The present invention provides a procedure for vehicle technicians to evaluate whether the positions of the vehicle and the calibration target are properly set in the lighting settings to ensure a greater likelihood of a successful recalibration procedure.

[0005] Therefore, in a broad aspect, the present invention provides a machine vision process for imaging a calibration target in a lighting setting to identify lighting problems, and if lighting problems are identified, modify the lighting settings or reposition the target and reimage the target. Repeat the procedure until the imaging process provides a result with satisfactory lighting such that the recalibration process can proceed. Subsequently, the recalibration procedure begins.

[0006] The process can be defined as:

[0007] i) Provide a calibration target and set the position of the calibration target relative to the vehicle;

[0008] ii) Provide a lighting setting for illuminating the calibration target;

[0009] iii) Operate the imaging device to image the target and process the image to evaluate whether the illumination is satisfactory for proceeding with the calibration phase;

[0010] iv) When the output of step iii) is that the illumination is satisfactory, continue with the recalibration phase.

[0011] Since the calibration target board may be illuminated by various environmental lights, there may be several light reflection characteristics on the target board. These are 1) single - area specular reflection, 2) multi - specular reflection, 3) light gradient, 4) shadow, 5) spotlight. Generally, illumination characteristics 1, 2, 4, and 5 result in uneven image illumination, while 3 results in poor image saturation quality. These illumination characteristics are described and shown in FIG. 1.

[0012] The imaging device includes a device having a camera and a processor, such as a smartphone.

[0013] Processing the image may involve the following steps:

[0014] i) Detect saturation and uneven illumination (SU) regions in the calibration target image;

[0015] ii) Quantify the degree of deviation of the calibration target of interest from a reference sample that has passed the calibration test.

[0016] It may include the following processing step: Process the image to remove background objects separated from the calibration target.

[0017] The processing steps may be performed according to an algorithm that takes different illumination characteristics into account.

[0018] The different illumination characteristics may be a combination of two or more of the following: the hue, saturation and value (HSV) color system, a threshold for determining the luminance formed by incident light for each pixel of the calibration target, and the reflection characteristics of the calibration target material.

[0019] The processing steps may use a reference image to calculate the deviation of the target image of a given calibration target from a satisfactory illumination reference image using the mean squared difference (MSD) function. The satisfactory illumination reference image may be an image of a well - illuminated target board that is known to have passed the calibration test. The reference image may be stored in a memory.

[0020] The processing steps may be performed according to an alternative contrast loss detection algorithm; preferably, in this alternative contrast loss detection algorithm, the distribution of illumination on the calibration target area is analyzed.

[0021] The process may include the following steps:

[0022] i) Using an object detection method to locate the calibration target area in the image;

[0023] ii) Calculating the non-illuminated image for the calibration target area; and

[0024] iii) Subsequently calculating the light distribution.

[0025] The process steps may apply metrics to describe the light non-uniformity, and a heat map may be generated to show the gradient of the light reflected on the calibration target area. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will now be further described in specific embodiments by way of example only and with reference to the drawings.

[0027] FIG. 1 schematically shows common problems related to calibration target illumination;

[0028] Figures 2 to 6 Relates to the implementation of a first method / algorithm according to the present invention using a processing system;

[0029] Figures 7 to 13 Relates to the implementation of a second method / algorithm according to the present invention using a processing system. DETAILED DESCRIPTION

[0030] Method 1

[0031] In the first method of the present invention, the program is as Figure 2 shown. Two main steps: detecting the saturation and uneven illumination (SU) areas in the target board image, and quantifying the degree of deviation of the target board of interest from a reference sample that has passed the calibration test. A preprocessing step may be added as a support tool to help clean up the image containing background objects. This is feasible when the focus of calibration is no longer based on the viewpoint of the ADAS camera. It should be noted that even in the presence of environmental clutter, this method can detect the SU (saturation and uneven illumination) on the target board image. Method 1 is outlined in Figure 2 and briefly discussed below.

[0032] The SU algorithm steps take into account the different lighting characteristics presented in Section 2. The different lighting characteristics are based on a strategic combination of the following: the hue, saturation and value (HSV) color system, the threshold for determining the luminance L(x,y) formed by the incident light l(x,y) with two unknowns for each pixel of the target board, and the reflection characteristic R(x,y) of the board material. This is expressed as:

[0033] L(x,y) = I(x,y)R(x,y)

[0034] Let be in the HSV space. To find the pixels corresponding to specular highlights (single and multiple points), saturation, shadows, and the presence of spotlighting, only sample the domain s L v (x,y) and value L

[0035]

[0036] Ω s (x,y) = L s (x,y) ≤ T s

[0037] Ω v (x,y) = L v (x,y) ≥ T v

[0038] where T s and T v are the thresholds of the saturation and value vectors L s (x,y) and L v (x,y) respectively, and C v and U Ωs are constants and control values respectively. It should be noted that after processing, Q(x,y) still retains the pixels within the range of [0, 255].

[0039] The reference image is used to calculate the degree of deviation of the target board image of interest from the general lighting standard of a given board using the mean squared difference (MSD) function. The concept behind the reference is based on the fact that there will always be an example of a well-illuminated target board that passes the calibration test. This reference will be stored in a file and used for each target board, otherwise it is difficult to quantify the degree to which a given target board image deviates from the standard of a well-illuminated target board. This is calculated mathematically as follows:

[0040]

[0041] Among them, N xy is the resolution of the image, Q(x, y) includes pixel regions with uneven illumination and poor saturation, and R(x, y) represents the reference example of the calibrated passing image.

[0042] The preprocessing includes two calculation schemes. The first method is to manually locate the region of interest (ROI) in the cluttered background image and automatically locate the target board. Manual location enables the user to select the ROI when capturing the image during real-time calibration.

[0043] The second method utilizes automatic target board location and is capable of using an instance segmentation model (Mask RCNN) that has been trained on all possible target board examples to locate the board of interest from the images captured during real-world calibration. If all the user is interested in is detecting the regions on the board with illumination problems, both methods can be ignored. The option to locate the board in the cluttered background can be activated according to the user's requirements.

[0044] Example

[0045] The results shown here demonstrate how effective the SU detection algorithm is in identifying pixels corresponding to specular highlights (single-point and multi-point), low saturation, shadows, and spotlighting in a given image. For each implementation scenario, a sample of the illumination problem is given, which describes the illumination challenge of the given target board input image to the SU detector, followed by the results of applying the SU detector, and then the value of the degree of deviation between the SU detection image and the reference image is calculated.

[0046] Implementation scenario one

[0047] In this scenario, the results of the SU detector processing illumination challenges from mild to severe illumination gradients and shadow gradients are as Figure 3 shown and discussed below. It can be observed that the SU detector is able to retrieve the regions in the target board image that are most likely to cause calibration failure, although severe illumination gradients may not occur because the calibration workshop follows illumination standards to ensure no excessive illumination imbalance. Through the calculation of the degree of deviation such as MSD (mean squared deviation), it can be easily observed that the output value corresponds to the observable illumination level problem existing in the target board image.

[0048] Implementation scenario two

[0049] In this scenario, an illumination challenge in the form of specular highlights appears. Due to the material of the target board and multiple light sources around the target board, specular highlights are an important issue that cannot be ignored. Compared with other illumination problems, this problem is more likely to be the root cause of calibration failure in the workshop.Figure 4 This is the performance of the SU detector for single - point specular reflection images (generated in the laboratory) and multi - specular reflection images. Even though an experienced technician might judge that calibration will fail based on the multi - specular reflection image, it is still necessary to be able to quantify the failure rate due to the deviation of the image from the standard. As Figure 4 shown, the SU detector can clearly show that the single - point specular highlight image may cause the calibration system to fail. This is of great concern because it is not apparent from the input of the calibration system that the circle of the left - most highlight may cause calibration failure. By using the MSD to quantify the problem, the problem of the high deviation of the single - point specular highlight from the expected standard is solved.

[0050] Pre - processing function

[0051] Figure 5 shows the results of the pre - processing step (the option to locate the board in a cluttered background) added to the SU detection algorithm. Pre - processing can provide advantages, but it may not necessarily be applicable to all implemented systems because the SU detector can detect areas of lighting problems on a given target board image with or without a cluttered background (see Figure 4 ). However, for the sake of completeness, Figure 5 shows the performance of the SU detector with the target board localization step added.

[0052] Figure 6 shows that: both the automatic localization method and the manual localization method can locate the target board and still produce closely related results. However, they both impose different levels of computational requirements on the SU detector. The automatic target board localization using the Mask - RCNN method requires more computational demand than the user - defined ROI method, even though it can be trained to detect and locate different target boards.

[0053] Method 2

[0054] In the second method of the present invention, the procedure is as Figure 7 shown. In this method, an alternative contrast loss detection algorithm is used. The concept of this algorithm is based on the analysis of the lighting distribution on the calibration target area. The general process of the algorithm is as Figure 7 shown. Initially, the object detection method is used to find the calibration target board area in the image. Then, the non - illuminated image is calculated for the board area. Subsequently, the light distribution is derived. Finally, a metric is applied to describe the light non - uniformity, and a heat map is generated to show the gradient of the light reflected on the board area.

[0055] Calibration target detection

[0056] To detect the calibration target board in the photos taken by smartphones with the least computing power, a process called template matching is used. The normalized cross-correlation (M) is a metric for calculating the similarity between an image I and a template t, which is defined as:

[0057]

[0058] where, is the average value of the template image, is the average value in the area under the template t. Once the normalized cross-correlation M is calculated, the best matching area can be found by returning the index of the largest element in M. However, due to the existence of multiple templates and the possible variation in the size of the board area in the image, this process should be executed multiple times to find the correct size and position of the board.

[0059] Calculating the locally unlit image

[0060] The unlit image is an ideal image (or pseudo-idealized board image) in which all areas are equally illuminated. Calculating the unlit image of the real scene is challenging, but due to the prior knowledge of the target calibration boards used by different vehicle manufacturers, the calculation process is simplified in this method. Most target calibration boards consist only of white and black areas. Therefore, based on this, the unlit image is obtained through a customized adaptive threshold. Figure 8 Shows the main steps of unlit image calculation.

[0061] Before applying the adaptive threshold, the background mode (white or black background) is determined according to the output i from the previous board detection process. Once the background is defined, a binary image as shown in the second image in Figure 8 is generated through the adaptive threshold. However, due to the lack of color information in black and white pixels, the binary image cannot be used as the unlit image. To retrieve the missing color, the average value of the corresponding white pixels in the original image is assigned to the white pixels in it. Similarly, the value obtained by averaging the dark pixels in the original image is reassigned to the black pixels in the intermediate image. Therefore, Figure 8 the third image in

[0062] Inferring the illumination distribution

[0063] Theoretically, in some practical cases, it can be achieved by subtracting from the original image Figure 8The illumination map D is simply obtained from the non-illuminated images. However, the patterns on the calibration target board may cause different reflectivities between the black and white areas, so interpolation is required to fix any incorrect predicted illumination in the foreground area (such as the black area in these cases). The interpolation algorithm fills all the incorrect pixels in the foreground area of the illumination map using an arithmetic sequence, where the arithmetic sequence is generated with reference to two neighboring end pixels in D. Figure 9 Shows a method for interpolating the illumination map.

[0064] Quantify local non-uniform illumination

[0065] Once the gray-scale interpolated illumination map N is calculated, the map is rendered using the Jet color map, as Figure 10 shown, which makes it more visible to the human eye to see the changes in illumination. To describe the non-uniformity of the illumination on the calibration board, it is quantified using variance, which is defined as:

[0066] where N is the interpolated illumination map, is the average pixel value. E(x,y) is the averaging operation on the image plane.

[0067] Results of Method 2

[0068] The experiment aims to evaluate the algorithm in a simulated calibration environment in the workshop. To construct the dataset for this evaluation, photos of the calibration board are taken using a smartphone from different perspectives and under various illuminations. During the image acquisition process, a projector and fill lights are used to illuminate the calibration board under different lighting conditions (i.e., different lighting settings). To execute the algorithm, binary templates of these calibration boards are provided. To speed up the calibration target board area detection process, the algorithm allows the user to define which calibration board he / she is looking for. Figure 11 Shows a typical output according to the algorithm, which includes multiple board regions in the gray-scale input image, labels of the identified calibration patterns, quantified non-uniformity of each board, illumination distribution on the board, and markings for indicating possible specular reflection points on the board. Similarly, Figure 12 and Figure 13 show more results of applying the algorithm to different levels of lighting problems for the same board and to images of different backgrounds and boards, respectively, which indicate that as the non-uniformity of the illumination increases, the value of the metric U also rises. The experiment is also extended to show the wide range of performance of different boards under mild to severe lighting problems.

[0069] The present invention provides different methods for detecting saturation and non-uniform illumination on an image of a calibration target board, so as to help calibration technicians identify areas on the target board that exhibit illumination characteristics that may cause calibration failure. Once the technician identifies such an area, the technician can decide to modify / change the illumination settings and / or reposition the calibration target board to an area with a more uniform / appropriate illumination distribution. The highlights of these methods are:

[0070] Method 1: The Saturation and Uneven Illumination (SU) detector program captures various illumination features such as single-point specular highlights, multi-point specular highlights, spotlighting, and shadows. But most importantly, it can capture images with poor saturation, especially those that show slight gradients in illumination.

[0071] Method 1: The SU detection method is even applicable to cluttered background images.

[0072] Method 1: The additional steps of preprocessing the cluttered background image to locate the target board with a user-defined ROI and automatically locating the target board with Mask-RCNN are useful additional functions of interest, but they impose varying degrees of computational cost on the SU detector processing.

[0073] Method 2: The contrast loss detection algorithm can detect the illumination on the target board.

[0074] Method 2: Use the covariance method to quantify the non-uniform illumination in the target board image.

[0075] Method 2: The template matching method is proven to be useful for locating the target board in a cluttered background.

[0076] The detection of saturation and non-uniform illumination is successfully achieved using a smartphone camera and appropriate processing. When the effect of the illumination settings ranges from slight to severe, the value of the metric quantifying this effect increases. This means that the smaller the illumination effect on the target board, the closer the value is to zero.

[0077] The present invention has been mainly described with respect to a calibration target board. However, it should be understood that other calibration target devices can be envisioned, such as an electronic / optical display screen or a light projection that implements the functions of such a target board.

Claims

1. A machine vision process that uses an imaging device to image a calibration target in an illumination setup, processes the image to identify illumination problems, and if an illumination problem is identified, modifies the illumination setup and / or relocates the target relative to the vehicle and / or the illumination setup, and re-images the target.

2. The machine vision process according to claim 1, wherein, Repeat the procedure until the imaging process provides a result with satisfactory illumination such that the recalibration process can proceed.

3. The machine vision process according to claim 1 or claim 2, wherein, The process steps include: i) Provide a calibration target and set the position of the calibration target relative to the vehicle; ii) Provide an illumination setup for illuminating the calibration target; iii) Operate the imaging device to image the target and process the image to evaluate whether the illumination is satisfactory for proceeding with the calibration phase; iv) When the output of step iii) is that the illumination is satisfactory, proceed with the recalibration phase.

4. The machine vision process according to any one of claims 1 to 3, wherein, The imaging device includes a camera and a processor for processing data from the camera.

5. The machine vision process according to any one of claims 1 to 4, wherein, The imaging device includes a smartphone having a camera and a processor.

6. The machine vision process according to any one of claims 1 to 5, wherein, Processing the image includes: i) Detecting saturation and uneven illumination SU regions in the calibration target image; ii) Quantifying the degree of deviation of the target of interest from a reference sample image that has passed a calibration test.

7. The machine vision process according to any one of claims 1 to 6, wherein, Includes a processing step in which the image is processed to remove background objects separated from the calibration target.

8. The machine vision process according to any one of claims 1 to 7, wherein, The processing step is performed according to an algorithm that takes different illumination characteristics into account.

9. The machine vision process according to claim 8, wherein, The different illumination characteristics are a combination of two or more of the following: hue, saturation, and value HSV color system, a threshold for determining the luminance formed by incident light for each pixel of the calibration target, and the reflection characteristics of the calibration target material.

10. The machine vision process according to any one of claims 1 to 9, wherein, The processing step uses a reference image to calculate the deviation of the target image of a given calibration target from a satisfactory illumination reference image using the mean square deviation MSD function.

11. The machine vision process according to claim 10, wherein, The satisfactory illumination reference image is an image of a well-illuminated target board that is known to have passed a calibration test.

12. The machine vision process according to any one of claims 1 to 11, wherein, The processing step is performed according to an alternative contrast loss detection algorithm.

13. The machine vision process according to claim 12, wherein, Analyze the distribution of illumination on the calibration target area.

14. The machine vision process according to claim 12 or claim 13, wherein: i) Use an object detection method to locate the calibration target area in the image; ii) Calculate a non-illuminated image for the calibration target area; iii) Subsequently calculate the distribution of light.

15. The machine vision process according to claim 14, wherein, The process steps apply a metric to describe the light non-uniformity and generate a heat map to show the gradient of the light reflected on the calibration target area.

16. The machine vision process according to claim 14 or claim 15, wherein, The non-illuminated image is calculated through an adaptive thresholding processing step.

17. The machine vision process according to any one of claims 14 to 16, wherein, Determine the background pattern (dark background or bright background) of the calibration target by processing the image, and generate a binary image through adaptive thresholding.

18. The machine vision process according to claim 17, wherein, To calculate the non-illuminated image, after determining the calibration target background and the adaptive threshold step: i) Assign the average value of the corresponding white pixels in the original image to the white pixels; ii) Reassign the value obtained by averaging the dark pixels in the original image to the black pixels in the intermediate image.

Images