A calibration image self-calibration method and device

Abstract

A calibration image self-calibration method includes the following steps: image preprocessing, converting the acquired initial image P1 and the image P2 after movement into grayscale images, and drawing the color distribution histogram of the image; feature point extraction, based on the local features of the image, finding extreme points (corner points) as feature points, and ensuring that they have rotation, scale scaling and brightness invariance; feature point screening, so that the selected feature points are all on the road surface; feature point matching, matching the feature points on the two images according to the similarity of the feature description histogram; outlier screening, screening outliers that do not match the local matching; calculating the homography matrix H2 from the image P2 to the image P1; calculating the new homography matrix H, H = H1*H2. The present invention can greatly reduce the workload and improve safety. It only needs to obtain two pictures taken before and after the camera changes, and the changed homography matrix can be automatically calculated through the original homography matrix.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a calibration image self-calibration method or device, which can automatically calculate the changed homography matrix through the original homography matrix by only obtaining two pictures taken before and after the camera changes, so as to achieve the technical purpose of automatically checking whether the camera moves and automatically completing the recalibration after the camera moves. Background Art

[0002] In the process of autonomous driving or vehicle-road collaboration, the image taken by the camera needs to be calibrated with other coordinates, such as GPS, millimeter-wave radar or lidar, through the homography matrix. Taking the conventional pixel coordinate and GPS calibration as an example, the homography matrix is ​​calculated by selecting multiple sets of pixel coordinates and GPS coordinate pairs to complete the image-to-map calibration.

[0003] However, in actual applications, the camera position and shooting angle may change over time and due to weather factors. Continuing to use the original homography matrix will cause positioning errors. The error at a distance may even exceed 10 meters. The original homography matrix needs to be updated.

[0004] The current solution is to re-acquire a new homography matrix to replace the original one. This is done by automatically selecting multiple sets of pixel coordinates and GPS coordinate pairs for recalibration, which has poor real-time performance. At the same time, since the time of camera movement is unknown, it is necessary to manually check whether the camera has moved regularly. Secondly, recalibration requires manpower and material resources. Furthermore, there are security issues when using RTK calibration. Summary of the invention

[0005] According to the problem raised by the background technology, the present invention provides a calibration image self-calibration method, which automatically checks whether the camera moves and automatically completes the recalibration after the camera moves. The scheme only needs to obtain two pictures taken before and after the camera changes, and the homography matrix after the change can be automatically calculated through the original homography matrix. The present invention is further explained below.

[0006] A calibration image self-calibration method comprises the following steps:

[0007] Step 1: Image preprocessing: convert the obtained initial image P1 and the moved image P2 into grayscale images, and draw a color distribution histogram of the images;

[0008] Step 2, feature point extraction, based on the local features of the image, find extreme points (corner points) as feature points, and ensure that they are rotation, scale and brightness invariant;

[0009] Step 3: feature point screening, so that all selected feature points are on the road surface;

[0010] Step 4: feature point matching: matching the feature points on the two images based on the similarity of the feature description histogram;

[0011] Step 5: Outlier point screening, screening out the outliers that do not match the local matching;

[0012] Step 6, calculating the homography matrix H2 from the image P2 to the image P1, specifically including the following steps:

[0013] Randomly select the first number of groups of matches from the filtered matches, and calculate a unique homography matrix;

[0014] Check whether the remaining matches conform to the mapping relationship corresponding to this matrix, and record the number of matches that conform;

[0015] Repeat several times, select the homography matrix with the largest number of matches, and extract all matches that match the matrix to obtain a matching subset;

[0016] Based on the matching subset, calculate the homography matrix H2 that meets the conditions and causes the minimum error;

[0017] Step 7, calculate the new homography matrix H, H = H1 * H2; wherein H1 represents the conversion relationship between the pixel coordinates on the original image P1 and the GPS coordinates, H2 represents the conversion relationship between the pixel coordinates on P2 and the pixel coordinates on the original image P1, and the homography matrix H represents the conversion relationship between the pixel coordinates on P2 and the GPS coordinates.

[0018] Furthermore, before converting to grayscale, the selected initial image P1 and the moved image P2 are adjusted to have the same size and no post-fonts;

[0019] After conversion to grayscale, in the obtained color distribution histogram, adjust the color distribution range and peak value, remove the parts with too large or too small values ​​and a small proportion, make the remaining colors occupy the range of 0-255, and the peak value appears in the range of 50-150; thereby achieving the purpose of adjusting contrast and brightness.

[0020] Furthermore, for the feature point extraction step, a first number of extreme points are found on the initial image P1 and the moved image P2, and the position, size, direction information of the extreme points and surrounding feature description factors are collected;

[0021] For the feature description factor, all pixels within a preset range around the feature point are selected and a histogram is drawn by pixel; the purpose is to describe the texture features.

[0022] Further, the screening step comprises the following steps:

[0023] Establish a trained neural network model to divide the content of P1 and P2;

[0024] Then use the expansion and erosion algorithm to remove noise, fill in the gaps, and ensure that the divided area is continuous and complete;

[0025] According to the divided road surface range, the extracted feature points are screened and only the feature points on the road surface are retained.

[0026] Furthermore, the feature point matching step includes the following steps:

[0027] For each feature point in P1, the difference between its histogram and the histograms of all feature points in P2 is calculated respectively, and the third number of points with the smallest difference are selected, and the obtained product number group of the first number and the third number is matched;

[0028] The matches are sorted by difference, and the top 80% with the smallest difference are retained.

[0029] Furthermore, the step of removing abnormal points includes:

[0030] For each local area in P1, check whether the feature points in the area are matched to the same area in P2;

[0031] If all feature points in the area that exceed the threshold ratio are matched to the same area of ​​P2, the abnormal points that are not matched to this area are screened out; if the matching of feature points in the area cannot reach a consensus, the area is divided into several blocks according to the distribution of feature points, and the matching of the segmented areas is checked to see whether a consensus is reached. The segmentation is continued until a consensus is reached, and the abnormal points are screened out.

[0032] Furthermore, after calculating the new homography matrix H, an inspection step is also included: setting the inspection algorithm and result judgment criteria of the homography matrix, and checking the quality of the homography matrix through an automatic detection mechanism. If the calculation result does not meet the standard, the system will automatically capture the image and recalculate until the result meets the standard.

[0033] Beneficial effects: Compared with the prior art, the present invention can greatly reduce the workload and improve safety. It only needs to obtain two pictures taken before and after the camera changes to automatically calculate the changed homography matrix through the original homography matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 : Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0035] Next, a specific embodiment of the present invention is described in detail.

[0036] A calibration image self-calibration method only needs to obtain two pictures taken before and after the camera changes, and can automatically calculate the changed homography matrix through the original homography matrix, so as to achieve the technical purpose of automatically checking whether the camera moves and automatically completing the recalibration after the camera moves. Therefore, after obtaining the initial homography matrix H1, the initial picture P1 and the picture P2 that needs to be moved, the new homography matrix can be automatically calculated.

[0037] Reference Figure 1 , specifically including the following steps:

[0038] Step 1: Image preprocessing: convert the obtained initial image P1 and the moved image P2 into grayscale images, and draw a color distribution histogram of the images;

[0039] Step 2, feature point extraction, based on the local features of the image, find extreme points (corner points) as feature points, and ensure that they are rotation, scale and brightness invariant;

[0040] Step 3: feature point screening, so that all selected feature points are on the road surface;

[0041] Step 4: feature point matching: matching the feature points on the two images based on the similarity of the feature description histogram;

[0042] Step 5: Outlier point screening, screening out the outliers that do not match the local matching;

[0043] Step 6, calculating the homography matrix H2 from the image P2 to the image P1, specifically including the following steps:

[0044] Randomly select the first number of groups of matches from the filtered matches, and calculate a unique homography matrix;

[0045] Check whether the remaining matches conform to the mapping relationship corresponding to this matrix, and record the number of matches that conform (reach consensus);

[0046] Repeat several times, select the homography matrix with the largest number of matches that meet (reach consensus), and extract all matches that meet this matrix to obtain a matching subset;

[0047] Based on the matching subset, calculate the homography matrix H2 that meets the conditions and causes the minimum error;

[0048] Step 7, calculate the new homography matrix H, H = H1*H2;

[0049] Among them, H1 represents the conversion relationship between the pixel coordinates on the original image P1 and the GPS coordinates.

[0050] H2 represents the conversion relationship between the pixel coordinates on P2 and the pixel coordinates on the original image P1.

[0051] The homography matrix H represents the conversion relationship between the pixel coordinates on P2 and the GPS coordinates.

[0052] For the image preprocessing step, the initial image P1 and the moved image P2 are selected to have the same size and similar brightness and contrast. First, ensure that there are no interference factors such as later fonts (shooting time, location, camera, etc.) in the image. If the fonts do not account for a large proportion, you can directly cut off the part with fonts; then, convert the initial image P1 and the moved image P2 into grayscale images, and draw the color distribution histogram of the image.

[0053] Furthermore, in the obtained color distribution histogram, the color distribution range and peak value are adjusted. For example but not limitation, the parts with values ​​that are too large or too small and account for a small proportion are removed, and the remaining colors occupy the entire range of 0-255, and the peak value appears in the range of 50-150, thereby achieving the purpose of adjusting contrast and brightness.

[0054] For the feature point extraction step, this embodiment is implemented by the SIFT local feature detection algorithm. Based on the image obtained by adjusting the contrast and brightness, extreme points are found in the grayscale value to ensure that they are rotation, scale and brightness invariant. This embodiment finds a first number of extreme points (i.e., feature points) on the initial image P1 and the moved image P2, for example, 8000, and collects the position, size, and direction information of the extreme points, as well as the surrounding feature description factors. For the feature description factor, 16*16 pixels around the feature point are selected, and 8 histograms are drawn by pixel division, in order to describe the texture features.

[0055] Since the 8000 feature points extracted by the SIFT algorithm are distributed throughout the image, including the road surface, sky, buildings, etc., which do not meet the requirements of the homography matrix algorithm for a single plane, we need to filter these points and only retain road surface points to improve the algorithm's adaptability to different scenes. This embodiment adds a feature point screening step before feature point matching to ensure that all feature points are on the road surface, further improving the algorithm's stability.

[0056] Feature point screening includes but is not limited to manual area restriction or automatic area restriction. This embodiment uses automatic area restriction to screen feature point areas, specifically using a segmentation algorithm, specifically including the following steps:

[0057] Establish a trained neural network model to divide the content of P1 and P2;

[0058] Then use the expansion and erosion algorithm to remove noise, fill in the gaps, and ensure that the divided area is continuous and complete;

[0059] According to the divided road surface range, the extracted feature points are screened and only the feature points on the road surface are retained.

[0060] The feature point matching step includes the following steps:

[0061] For each feature point in P1, the difference between its histogram and the histogram of all feature points in P2 is calculated respectively, and the third number of points with the smallest difference are screened out, for example, 5 groups of matches, and the total number of groups of matches obtained is the product of the first number and the third number, which is 5*8000 groups of matches in this embodiment;

[0062] Sort the matches from small to large according to the difference, and keep the first 80% with small difference, that is, only keep 4*8000 groups of matches.

[0063] By way of example and not limitation, the difference of the feature point histograms is determined based on the root mean square value of the two histograms, that is, the Euclidean distance (L2norm), which is a conventional determination method and will not be elaborated in detail in this embodiment.

[0064] For the outlier screening step, this embodiment adopts the AdaLAM outlier elimination algorithm to screen out outliers that do not match the local matching, thereby improving the performance of the algorithm when P1 and P2 differ greatly. The specific screening steps include:

[0065] For each local area in P1 (area with dense feature points), check whether the feature points in the area are all matched to the same area in P2, that is, whether the same homography matrix is ​​applicable;

[0066] If all feature points in the area that exceed the threshold ratio are matched to the same area of ​​P2, the abnormal points that are not matched to this area are screened out; if the matching of feature points in the area cannot reach a consensus, the area is divided into several blocks according to the distribution of feature points, and the matching of the segmented areas is checked to see whether a consensus is reached. The segmentation is continued until a consensus is reached, and the abnormal points are screened out.

[0067] It should be noted that in practice, if the feature points are not screened, when calculating the homography matrix, more road points are usually automatically selected to form a consensus matching subset, so ideal results can still be achieved in most cases. However, the role of feature point screening can greatly improve the adaptability of the algorithm to different scenarios, because when selecting feature points in step 2, ground points and non-ground points are treated equally. If no screening is performed, it is hoped that in the process of calculating the homography matrix in step 6, a large number of road points will reach a consensus, thereby excluding non-road points, and the final calculated matrix is ​​the conversion relationship from P1 road surface to P2 road surface. However, in actual applications, traffic light poles, road signs, flower beds, and certain details on buildings are easily marked as feature points. Therefore, when the zebra crossing lane lines on the road are blurred, or a large number of camera road signs are deployed at the intersection, the proportion of non-road points obtained becomes larger, thereby affecting the consensus of road points, resulting in the calculated matrix corresponding to a tilted plane in the air.

[0068] The role of outlier exclusion is to greatly improve the performance of the algorithm when P1 and P2 differ greatly. For example, if P1 is taken in the morning and P2 is taken in the afternoon, the road shadow situation is quite different, which increases the difference in texture features near the feature points, thus affecting the matching of feature points. Especially for similar feature points, such as a corner of a zebra crossing, it is very likely that there will be a chaotic situation of matching multiple points to one point when matching feature points, which will eventually lead to the calculated matrix corresponding to the conversion relationship between a certain plane in P1 and a certain point in P2. Outlier screening requires local matching to remain consistent. If zebra crossing 1 is matched to zebra crossing 1, other zebra crossings can no longer be matched to zebra crossing 1, and can only be matched to the corresponding zebra crossings. Otherwise, the matrix cannot reach a consensus, thus preventing this situation from happening.

[0069] After calculating the new homography matrix H, an inspection step is also included to set the homography matrix inspection algorithm and result judgment criteria. The homography matrix is ​​checked through an automatic detection mechanism. If the calculation result does not meet the standard, the system will automatically capture the image and recalculate until the result meets the standard.

[0070] In actual application, the original homography matrix is ​​updated after several months of operation or after a typhoon. Due to the offset of camera position or shooting angle, the points must be reselected each time, which is a lot of work. In addition, since the destination intersection usually has a large flow of people and vehicles, on-site calibration not only affects traffic, but also has certain safety issues. This method can greatly reduce the workload and improve safety. It only needs to obtain two pictures taken before and after the camera changes, and the changed homography matrix can be automatically calculated through the original homography matrix.

[0071] This solution has an automatic detection mechanism to check the quality of the homography matrix calculation. If the calculation result does not meet the standard, the system will automatically capture the image and calculate again until the result meets the standard; the matching result can also be checked manually.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A calibration image self-calibration method, characterized in that: The following steps are involved: Step 1: Image preprocessing: convert the obtained initial image P1 and the moved image P2 into grayscale images, and draw a color distribution histogram of the images; Step 2, feature point extraction, based on the local features of the image, find extreme points (corner points) as feature points, and ensure that they are rotation, scale and brightness invariant; Step 3: feature point screening, so that all selected feature points are on the road surface; Step 4: feature point matching: matching the feature points on the two images based on the similarity of the feature description histogram; Step 5: Outlier point screening, screening out the outliers that do not match the local matching; Step 6, calculating the homography matrix H2 from the image P2 to the image P1, specifically including the following steps: Randomly select the first number of groups of matches from the filtered matches, and calculate a unique homography matrix; Check whether the remaining matches conform to the mapping relationship corresponding to this matrix, and record the number of matches that conform; Repeat several times, select the homography matrix with the largest number of matches, and extract all matches that match the matrix to obtain a matching subset; Based on the matching subset, calculate the homography matrix H2 that meets the conditions and causes the minimum error; Step 7, calculate the new homography matrix H, H = H1*H2; Among them, H1 represents the conversion relationship between the pixel coordinates on the original image P1 and the GPS coordinates. H2 represents the conversion relationship between the pixel coordinates on P2 and the pixel coordinates on the original image P1. The homography matrix H represents the conversion relationship between the pixel coordinates on P2 and the GPS coordinates.

2. The calibration image self-calibration method according to claim 1, characterized in that: In the image preprocessing step, Before converting to grayscale, the selected initial image P1 and the moved image P2 are adjusted to the same size and without post-fonts; After conversion to grayscale, in the obtained color distribution histogram, adjust the color distribution range and peak value, remove the parts with large or small values ​​and small proportion, make the remaining colors occupy the range of 0-255, and the peak value appears in the range of 50-150; thereby achieving the purpose of adjusting contrast and brightness.

3. The calibration image self-calibration method according to claim 2, characterized in that: In the feature point extraction step, a first number of extreme points are found on the initial image P1 and the moved image P2, and the position, size, and direction information of the extreme points and surrounding feature description factors are collected; For the feature description factor, all pixels within a preset range around the feature point are selected and a histogram is drawn by pixel; the purpose is to describe the texture features.

4. The calibration image self-calibration method according to claim 3, characterized in that: The screening step includes the following steps: Establish a trained neural network model to divide the content of P1 and P2; Then use the expansion and erosion algorithm to remove noise, fill in the gaps, and ensure that the divided area is continuous and complete; According to the divided road surface range, the extracted feature points are screened and only the feature points on the road surface are retained.

5. The calibration image self-calibration method according to claim 4, characterized in that: The feature point matching step includes the following steps: For each feature point in P1, the difference between its histogram and the histograms of all feature points in P2 is calculated respectively, and the third number of points with the smallest difference are selected, and the obtained product number group of the first number and the third number is matched; The matches are sorted by difference, and the top 80% with the smallest difference are retained.

6. The calibration image self-calibration method according to claim 5, characterized in that: The steps for outlier screening include: For each local area in P1, check whether the feature points in the area are matched to the same area in P2; If all feature points in the area that exceed the threshold ratio are matched to the same area of ​​P2, the abnormal points that are not matched to this area are screened out; if the matching of feature points in the area cannot reach a consensus, the area is divided into several blocks according to the distribution of feature points, and the matching of the segmented areas is checked to see whether a consensus is reached. The segmentation is continued until a consensus is reached, and the abnormal points are screened out.

7. The calibration image self-calibration method according to claim 6, characterized in that: After calculating the new homography matrix H, there is also a checking step: setting the homography matrix inspection algorithm and result judgment criteria, and checking the homography matrix through an automatic detection mechanism. If the calculation result does not meet the standard, the system will automatically capture the image and recalculate until the result meets the standard.

8. A device, characterized in that The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, and the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the calibration image self-calibration method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the calibration image self-calibration method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product is run on a terminal device, the terminal device executes the calibration image self-calibration method according to any one of claims 1 to 6.

Images