A non-planar road surface calibration method and device

Abstract

The present invention relates to the field of computer image recognition, and provides a non-planar road surface calibration method, which only needs to obtain two photos of the intersection at different angles, and can automatically determine the number of planes contained in the road surface through image feature matching and homography matrix method, draw a segmentation line, and then calibrate each plane separately. It can reduce the inaccuracy factors caused by manual naked eye judgment during the calibration process, improve the automation ratio and accuracy of road surface segmentation; and avoid the problem of repeated calibration due to excessive errors found in subsequent applications, greatly reduce the workload, and improve safety.

Description

Technical Field

[0001] The invention relates to the field of computer image recognition, and in particular to a non-planar road surface calibration method and device. Background Art

[0002] In the process of vehicle-road collaboration, it is necessary to calibrate the image taken by the camera with other coordinates, such as maps, millimeter-wave radars or lidars. Among them, converting the pixel coordinates on the picture into GPS coordinates is the most common requirement. The image-to-map calibration can be completed by selecting multiple sets of pixel coordinates and GPS coordinate pairs to calculate the homography matrix. However, since the homography matrix method is only applicable to plane-to-plane transformation, it is not applicable to non-plane transformation. In the case where the road surface does not belong to the same plane (for example, there is a slope), the road surface needs to be divided into two or more areas to ensure that the road surface in each area is close to the plane, and then calculate the corresponding homography matrix separately.

[0003] The current methods for determining whether a road surface includes multiple planes include visual judgment and analysis of the matching error of a single homography matrix. However, the method of manually dividing the area has many defects: first, the quality of the dividing line directly affects a series of subsequent calculations, and the division by the naked eye alone is not reliable enough; second, after discovering a single matrix error, dividing the plane involves the problem of secondary calibration, which consumes manpower and material resources; third, this requires a high level of quality for on-site calibration personnel, and training requires time and financial costs.

[0004] Therefore, the present invention proposes a method for automatically dividing and identifying the number of road surface planes and dividing the planes according to a road surface picture taken by a camera, so as to solve the problem of non-planar road surface calibration. Summary of the invention

[0005] According to the problems raised in the background technology, the present invention provides a non-planar road surface calibration method, which only needs to obtain two photos of the intersection at different angles, and can automatically determine the number of planes contained in the road surface through image feature matching and homography matrix method, and draw a dividing line, and then calibrate each sub-plane separately. The method of the present invention aims to reduce the inaccuracy factors caused by manual naked eye judgment, improve the automation ratio and accuracy of road surface segmentation, and reduce the calibration workload. The present invention is further explained below.

[0006] A non-planar road surface calibration method comprises the following steps:

[0007] S1, image acquisition and preprocessing, obtains two images P1 and P2 with deviations in road shooting angles, and automatically preprocesses P1 and P2 to ensure that the two images have the same size and similar brightness and contrast.

[0008] S2, feature point extraction and screening, finds feature points based on local features of the image and ensures that they are rotation, scale and brightness invariant, divides the content of P1 and P2 based on a trained neural network, and only retains feature points on the road surface;

[0009] S3, feature point matching and outlier point screening, the feature points on the two images P1 and P2 are matched by brute force according to the similarity of the feature description histogram, and the outliers that do not match the local matching are screened out to obtain the matching point pair set M;

[0010] S4, extract a plane from the matching point pair set M, and obtain its corresponding point pair subset M1 and homography matrix H1:

[0011] Select a first number of matching point pairs from M, calculate a unique homography matrix, and check whether the remaining matching point pairs conform to the mapping relationship corresponding to this matrix, and record the number of matching point pairs that conform;

[0012] Repeat the above steps several times, select the homography matrix with the most consensus matching points from all homography matrices, extract all its corresponding matching points M1, calculate the number of matching point pairs contained in M1, record it as S1, and the matching set without consensus is recorded as M C (M C =M-M1);

[0013] Use this matching point pair subset M1 to calculate the homography matrix, which is H1;

[0014] S5, check whether there are other planes. If so, find the point pair subset and homography matrix corresponding to the plane:

[0015] Randomly select a second number of matching point pairs from the matching subset M1 and add them to M C Merge to form a new matching set M'; repeat step S4, extract a plane from M', obtain its corresponding point pair subset and homography matrix, and record the number of point pairs contained in the point pair subset;

[0016] Repeat the above steps several times, select the plane with the largest number of point pairs in the corresponding matching subset from all extracted planes, and record its corresponding point pair subset, the number of point pairs contained in the point pair subset and the homography matrix as M2', S2' and H2';

[0017] If S2' is greater than the preset ratio of S1, and the number of matches shared by M2' and M1 is less than the preset ratio S1, it is considered that there is a new plane in the image, and M2' and H2' are saved as M2 and H2; otherwise, it is considered that there is no new plane, and M2' and H2' are not saved;

[0018] S6, repeat step S5 until the algorithm determines that there is no new plane, and save the n planes obtained by the algorithm and the corresponding point pair subsets M1,…,M n With the homography matrix H1,…,H n ;

[0019] S7, the road surface is divided into subsets of n points:

[0020] If n = 1, the road surface is on the same plane and does not need to be divided;

[0021] If n = 2, a binary classification model is trained to divide the pixels on the P2 road into two regions;

[0022] If n>2, a multi-classification model is trained to divide the pixels on the P2 road surface into n regions.

[0023] S8, calibrating each sub-plane range respectively according to the divided sub-plane ranges.

[0024] Beneficial effects: The present invention only needs to obtain two photos of the intersection from different angles, and can automatically determine the number of planes contained in the road surface through image feature matching and homography matrix method, draw segmentation lines, and then calibrate each plane separately. It can reduce the inaccuracy caused by manual visual judgment during the calibration process, improve the automation ratio and accuracy of road segmentation; and avoid the problem of repeated calibration due to excessive errors in subsequent applications, greatly reducing workload and improving safety. DETAILED DESCRIPTION

[0025] Next, a specific embodiment of the present invention is described in detail.

[0026] A non-planar road surface calibration method comprises the following steps:

[0027] S1, image acquisition and preprocessing.

[0028] Get two pictures P1 and P2 with deviations in the shooting angle of the road surface, and automatically pre-process P1 and P2 to ensure that the two pictures have the same size and no interference factors such as later fonts; the brightness contrast is similar to improve the accuracy of later point matching.

[0029] In this embodiment, in order to obtain two pictures P1 and P2 with deviations in shooting angles of the same road surface, it is preferred to use the same camera in engineering applications, slightly change the position, and take two pictures continuously to achieve better matching effect and more accurate segmentation.

[0030] S2, feature point extraction and screening.

[0031] Obtain feature points: Use the SIFT local feature detection algorithm to find extreme points (i.e. feature points) based on the local features of the image, and ensure that they are rotation, scale and brightness invariant;

[0032] Obtain road feature points: Based on a trained neural network, such as the MMlab public neural network model MMSegmentation, perform content segmentation for P1 and P2 (including road surface, vehicles, buildings, pedestrians, sky, etc.), and only retain feature points on the road surface to eliminate the interference of non-road surface planes on the algorithm.

[0033] S3, feature point matching and outlier point removal.

[0034] Feature point matching: brute force matching is performed on the feature points of the two images P1 and P2, and the matching basis is the similarity of the feature description histogram;

[0035] Screen out the outliers to obtain the matching point pair set M: With the help of the AdaLAM outlier elimination algorithm, screen outliers that do not match the local matching (for each local area in P1, the feature points in the area should be matched to the same area of ​​P2, that is, the same homography matrix should be applicable), and finally obtain the matching point pair set M.

[0036] S4, extract a plane from the matching point pair set M, and obtain its corresponding point pair subset M1 and homography matrix H1.

[0037] Select a first number of groups, for example, 4 groups of matching point pairs, from M, calculate a unique homography matrix, and check whether the remaining matching point pairs conform to the mapping relationship corresponding to the matrix, and record the number of matching point pairs that conform (reach consensus);

[0038] Repeat the above steps several times, select the homography matrix with the most consensus matching points from all homography matrices, extract all its corresponding matching points M1, calculate the number of matching point pairs contained in M1, record it as S1, and the matching set without consensus is recorded as M C (M C =M-M1);

[0039] Use this matching point pair subset M1 to calculate the homography matrix, which is H1.

[0040] S5, check whether there are other planes. If so, find the point pair subset and homography matrix corresponding to the plane.

[0041] A second number of groups are randomly selected from the matching subset M1, for example, 1 / 3 of the total number of point pairs in the matching subset M1, and CMerge to form a new matching set M'; repeat step S4, extract a plane from M', obtain its corresponding point pair subset and homography matrix, and record the number of point pairs contained in the point pair subset;

[0042] Repeat the above steps several times, select the plane with the largest number of point pairs in the corresponding matching subset from all extracted planes, and record its corresponding point pair subset, the number of point pairs contained in the point pair subset and the homography matrix as M2', S2' and H2';

[0043] If S2' is greater than a preset ratio (e.g. 60%) of S1, and the number of matches shared by M2' and M1 is less than a preset ratio (e.g. 25%) of S1, it is considered that there is a new plane in the image, and M2' and H2' are saved as M2 and H2; otherwise, it is considered that there is no new plane, and M2' and H2' are not saved.

[0044] S6, repeat step S5 until the algorithm determines that there is no new plane, and save the n planes obtained by the algorithm and the corresponding point pair subsets M1,…,M n With the homography matrix H1,…,H n .

[0045] S7, dividing the road surface into subsets based on the n points.

[0046] If n = 1, the road surface is on the same plane and does not need to be divided;

[0047] If n=2, then train a binary classification model, such as a logistic regression model, a soft margin support vector machine or a neural network. Take the logistic regression model as an example: the coordinates of the points corresponding to M1 on P2 and the coordinates of the points corresponding to M2 on P2 are used as training sets, and the labels are set to 1 and -1 respectively; the logarithmic loss function (logloss) is used as the loss function, the model is trained using the stochastic gradient descent method, and the sigmoid function is used as the output. Through this model, the pixels on P2 (road surface) can be divided into two areas;

[0048] If n>2, then train a multi-classification model, such as a multi-classification logistic regression model, a multi-classification support vector machine or a neural network. Take the multi-classification logistic regression model as an example: transform the coordinates of the point on P2 corresponding to M1 to M n Corresponding to P n The coordinates of the points on are used as the training set, and the labels are set to 1,…,n respectively. The cross entropy is used as the loss function, the model is trained using the stochastic gradient descent method, and the softmax function is used as the output. Through this model, the pixels on P2 (road surface) can be divided into n regions.

[0049] In this step, the classification model can be selected according to the actual situation, and the logistic regression model is preferably used: compared with the support vector machine, the logistic regression model considers the influence of all training samples on the parameters in the training model, not just the support vector, so it has better stability; in addition, it is difficult for the neural network to extract the segmentation line that divides the plane, while the logistic regression model can return the segmentation line, and the interface is more flexible in engineering applications.

[0050] According to common sense, classification models usually support nonlinear situations, but in road calibration problems, nonlinear segmentation is usually not required. Selecting linear segmentation not only speeds up the calculation, but also avoids overfitting of the model.

[0051] S8, calibrating each sub-plane range respectively according to the divided sub-plane ranges.

[0052] A non-planar road surface calibration device, comprising: a shooting unit, a processor, a memory, a system bus

[0053] A shooting unit, including two sets of cameras with deviations in shooting angles or a single set of rotatable cameras, for obtaining two pictures P1 and P2 of the same road surface;

[0054] The processor and the memory are connected via the system bus;

[0055] The memory is used to store one or more programs, wherein the one or more programs include instructions, which, when executed by the processor, cause the processor to perform non-planar road surface calibration.

[0056] The present invention proposes a non-planar road surface calibration method. It only needs to obtain two photos of the intersection at different angles to automatically determine the number of planes contained in the road surface through image feature matching and homography matrix method, draw a segmentation line, and then calibrate each plane separately. This method can reduce the inaccuracy factors caused by manual visual judgment during the calibration process, improve the automation ratio and accuracy of road surface segmentation; and avoid the problem of repeated calibration due to excessive errors in subsequent applications, greatly reducing workload and improving safety.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A non-planar road surface calibration method, characterized in that: The following steps are involved: S1, image acquisition and preprocessing, obtain two images P1 and P2 with deviations in road shooting angles, and automatically preprocess P1 and P2 to ensure that the two images have the same size and similar brightness and contrast; S2, feature point extraction and screening, finds feature points based on local features of the image and ensures that they are rotation, scale and brightness invariant, divides the content of P1 and P2 based on a trained neural network, and only retains feature points on the road surface; S3, feature point matching and outlier point screening, the feature points on the two images P1 and P2 are matched by brute force according to the similarity of the feature description histogram, and the outliers that do not match the local matching are screened out to obtain the matching point pair set M; S4, extract a plane from the matching point pair set M, and obtain its corresponding point pair subset M1 and homography matrix H1: Select a first number of matching point pairs from M, calculate a unique homography matrix, and check whether the remaining matching point pairs conform to the mapping relationship corresponding to this matrix, and record the number of matching point pairs that conform; Repeat the above steps several times, select the homography matrix with the most consensus matching points from all homography matrices, extract all its corresponding matching points M1, calculate the number of matching point pairs contained in M1, record it as S1, and the matching set without consensus is recorded as M C , where M C =M-M1; Use this matching point pair subset M1 to calculate the homography matrix, which is H1; S5, check whether there are other planes. If so, find the point pair subset and homography matrix corresponding to the plane: Randomly select a second number of matching point pairs from the matching subset M1 and add them to M C Merge to form a new matching set M'; repeat step S4, extract a plane from M', obtain its corresponding point pair subset and homography matrix, and record the number of point pairs contained in the point pair subset; Repeat the above steps several times, select the plane with the largest number of point pairs in the corresponding matching subset from all extracted planes, and record its corresponding point pair subset, the number of point pairs contained in the point pair subset and the homography matrix as M2', S2' and H2'; If S2' is greater than the preset ratio of S1, and the number of matches shared by M2' and M1 is less than the preset ratio S1, it is considered that there is a new plane in the image, and M2' and H2' are saved as M2 and H2; otherwise, it is considered that there is no new plane, and M2' and H2' are not saved; S6, repeat step S5 until the algorithm determines that there is no new plane, and save the n planes obtained by the algorithm and the corresponding point pair subsets With the homography matrix ; S7, the road surface is divided into subsets of n points: If n = 1, the road surface is on the same plane and does not need to be divided; If n=2, a binary classification model is trained to divide the pixels on the P2 road into two regions; If n>2, a multi-classification model is trained to divide the pixels on the P2 road into n regions; S8, calibrating each sub-plane range respectively according to the divided sub-plane ranges.

2. The non-planar road surface calibration method according to claim 1, characterized in that: In step S2, feature points are found using the SIFT local feature detection algorithm, and the trained neural network is the neural network model MMSegmentation published by MMlab.

3. The non-planar road surface calibration method according to claim 1, characterized in that: In step S3, the outliers that do not match the local matching are screened out using the AdaLAM outlier elimination algorithm.

4. The non-planar road surface calibration method according to claim 1, characterized in that: In step S7, when n=2, the binary classification model trained is a logistic regression model, a soft margin support vector machine or a neural network; if n>2, the multi-classification model trained can be a multi-classification logistic regression model, a multi-classification support vector machine or a neural network.

5. The non-planar road surface calibration method according to claim 4, characterized in that: When n=2, the trained binary classification model is a logistic regression model: the coordinates of the points corresponding to M1 on P2 and the coordinates of the points corresponding to M2 on P2 are used as training sets, and the labels are set to 1 and -1 respectively; the logarithmic loss function is used as the loss function, the model is trained using the stochastic gradient descent method, and the sigmoid function is used as the output. Through this model, the pixels on P2 can be divided into two regions; When n>2, the multi-classification model trained is a multi-classification logistic regression model: the coordinates of the points corresponding to M1 on P2 are converted to M n Corresponding to P n The coordinates of the points on are used as the training set, and the labels are set as , using cross entropy as the loss function, training the model with stochastic gradient descent method, and softmax function as the output. Through this model, the pixels on P2 can be divided into n regions.

6. A non-planar road surface calibration device, characterized in that: include: Shooting unit, processor, memory, system bus; A shooting unit, including two sets of cameras with deviations in shooting angles or a single set of rotatable cameras, for obtaining two pictures P1 and P2 of the same road surface; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, and the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the non-planar road surface calibration method according to any one of claims 1-5.