Road surface image feature matching method and system based on region and motion information

Abstract

The present invention discloses a road surface image feature matching method and system based on region and motion information. The matching method is as follows: comprising the following steps: step 1, data input, inputting real-time road surface continuous frame images taken by a vehicle-mounted camera, current vehicle motion information, and vehicle-mounted camera parameters; step 2, using the FAST corner point detection algorithm to perform feature point partition detection on the input continuous frame images; step 3, preliminarily screening the road surface feature points after the partition detection based on the position and image information of the vehicle-mounted camera, and determining that the extracted feature points are located on the road surface; step 4, predicting the position of the road surface feature points of the first frame in the second frame based on the vehicle body motion information and the camera model; step 5, taking the world coordinates of the predicted point in step 4 as the center, matching the continuous frame feature points based on the local image information and motion information and outputting the pairing information.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a road surface image feature matching method and system based on region and motion information. Background Art

[0002] The on-board camera is an important sensor of the on-board vision system, which can provide the vehicle with driving environment perception capabilities and provide the basic data required for decision-making for the assisted driving system. The continuous frame images collected by the on-board camera are the information basis for environmental perception, and the feature information on the road surface is particularly important. The extraction and matching of road surface features is the basis for road target tracking, target motion estimation and visual odometer. Therefore, an efficient and accurate continuous frame road surface feature matching method is needed.

[0003] At present, the Chinese patent with the publication number of 111524192 discloses a calibration method, device, system and storage medium for the external parameters of a vehicle-mounted camera; specifically, the calibration of the camera external parameters includes preprocessing the two frames of images collected by the vehicle-mounted camera; matching the feature points of the two preprocessed frames to obtain the matched feature points; determining the motion posture of the vehicle-mounted camera according to the matched feature points; and determining the conversion relationship between the vehicle coordinate system and the vehicle-mounted camera coordinate system based on the motion posture to obtain the external parameters of the vehicle-mounted camera relative to the vehicle body. The above method involves a road feature point matching method, which directly preprocesses the two frames of images, then matches the feature points of the preprocessed two frames of images to obtain the matched feature points, and determines the motion posture of the camera according to the matched feature points. The above matching method can only be used to determine the motion posture of the vehicle, but cannot determine the next frame position of the feature points according to the vehicle running path. The matching calculation is large, time-consuming, and the probability of mismatch is large. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide a road image feature matching method and system that can predict the position of road feature points based on vehicle motion information, reduce the amount of feature matching calculations, and effectively improve the accuracy of road feature point matching.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A road surface image feature matching method based on area and motion information is characterized in that it includes the following steps: step 1, data input, inputting real-time road surface continuous frame images taken by a vehicle-mounted camera, current vehicle motion information, and vehicle-mounted camera parameters; step 2, using the FAST corner point detection algorithm to perform feature point partition detection on the input continuous frame images; step 3, preliminarily screening the road surface feature points after partition detection based on the vehicle-mounted camera position and image information, and determining that the extracted feature points are located on the road surface; step 4, predicting the position of the first frame road surface feature points in the second frame based on the vehicle body motion information and the camera model; step 5, with the world coordinates of the predicted point in step 4 as the center, matching the continuous frame feature points based on the local image information and motion information and outputting the pairing information. In this way, the vehicle-mounted camera image road surface feature matching method proposed in this application can realize efficient extraction and matching of road surface feature points based on the camera's own picture and vehicle body motion information. Compared with the traditional method based only on image information, this method adds vehicle motion information and camera calibration information, which can effectively screen out feature points located on the road surface, reduce the amount of matching calculations, and improve the matching accuracy.

[0007] Furthermore, in step 1, the current motion information of the vehicle includes the vehicle's own speed V and the vehicle's current angular velocity YawRate; the vehicle-mounted camera parameters include the vehicle-mounted camera internal parameters and the vehicle-mounted camera external parameters, the matrix K of the vehicle-mounted camera internal parameters includes the focal length, principal point and distortion coefficient of the camera, and the vehicle-mounted camera external parameters include the camera rotation matrix Re0 and the translation vector t relative to the world coordinates w .

[0008] Furthermore, in step 3, when initially screening the road feature points, the image width is set to W and the height is set to H. For the front and rear cameras, the feature point extraction area is For the side-view camera, the feature point extraction area is

[0009] Furthermore, in step 4, the specific method for predicting the position of the road feature point includes: establishing a camera model based on the parameters of the vehicle-mounted camera, and then converting the pixel coordinates of the road feature point in the first frame into world coordinates, and at the same time, taking the feature point in the first frame image as the center and d as the side length, intercepting a local image of the first frame image as the image to be matched, and calculating the coordinate position of the feature point of the first frame image in the second frame image based on the vehicle body motion information and the camera model.

[0010] Furthermore, when establishing the camera model, the spatial position of the road feature point in the first frame is P i , P i =K[R e0 t w ]P wi , where K is the camera internal parameter matrix, R e0is the camera rotation matrix, t w is the camera translation vector, is the vector from the camera installation position to the world coordinate origin,

[0011] Furthermore, the world coordinates of the feature points of the first frame image are calculated based on the camera model:

[0012] p wipre =[x wi -V COS (YawRate·dt)dt,y wi -Vsin(YawRate·dt)dt,z wi ,1]; where p wipre is the world coordinate of the feature point of the first frame image, V is the vehicle body speed, dt is the time interval between two frames, and YawRate is the current rotation rate of the vehicle; after calculating the world coordinate of the feature point of the first frame image, the pixel coordinate p of the predicted point in the second frame image can be inferred through the camera model. ipre ,

[0013] Furthermore, in step 5, the feature matching method is as follows:

[0014] 1) Taking the predicted point p ipre As the center, r is the radius to establish a search area, and search for feature points of the second frame image in the area; if there are no feature points in the area, the current matching fails and the next matching starts; if there are n feature points in the area, these feature points are taken as the center, d is the side length, and the local area Rect21~Rect2n of the second frame image is intercepted as the image to be matched;

[0015] 2) Calculate the absolute error and SAD between Rect21~Rect2n and Rect1. For the jth feature point in the region, 1≤j≤n, the calculation method is:

[0016] , calculate the SAD of all feature points to be matched using the above formula, and select the point with the smallest SAD value As the final matching result;

[0017] 3) Calculate the final matching point With the predicted point p ipre The pixel distance is calculated as:

[0018]

[0019] If the calculated pixel distance is greater than the threshold T, it means that it does not meet the road feature point motion condition, and the group of feature point pairs is discarded.

[0020] A road surface image feature matching system based on area and motion information, characterized in that it includes a vehicle-mounted camera, an image feature point detection module, a position prediction module and a feature matching module, wherein the feature point detection module is communicatively connected to the vehicle-mounted camera and the vehicle-machine controller, and is used to extract the continuous frame image feature points and vehicle-mounted camera parameters of the vehicle-mounted camera and the current vehicle motion information in the vehicle-machine controller, and at the same time, detects the extracted continuous frame image feature points and returns their pixel positions; the position prediction module is communicatively connected to the image feature point detection module, and is used to predict the position of the road surface feature points in the next frame according to the vehicle motion state and the external parameters of the vehicle-mounted camera; the feature matching module matches the continuous frame feature points based on local image information and motion information and outputs pairing information.

[0021] Furthermore, it also includes a road feature point preliminary screening module, which is bidirectionally connected to the image feature point detection module for preliminary screening of road feature points after partition detection, determines that the extracted feature points are located on the road surface, and feeds back the preliminary screening information to the image feature point detection module. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The basic process of the road surface image feature matching method in the embodiment;

[0023] Figure 2 This is the feature point prediction process in the embodiment;

[0024] Figure 3 This is the feature point matching process in the embodiment;

[0025] Figure 4 This is a typical matching condition in the embodiment. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0027] Example:

[0028] As shown in the figure, the road surface image feature matching system based on regional and motion information provided by the present embodiment includes a vehicle-mounted camera, an image feature point detection module, a road surface feature point preliminary screening module, a position prediction module and a feature matching module. The feature point detection module is communicatively connected with the vehicle-mounted camera and the vehicle-mounted controller, and is used to extract the continuous frame image feature points and vehicle-mounted camera parameters of the vehicle-mounted camera and the current vehicle motion information in the vehicle-mounted controller. At the same time, the extracted continuous frame image feature points are detected and their pixel positions are returned; the road surface feature point preliminary screening module is bidirectionally communicatively connected with the image feature point detection module, and is used to preliminarily screen the road surface feature points after the partition detection, determine that the extracted feature points are located on the road surface, and feed back the preliminary screening information to the image feature point detection module; the position prediction module is communicatively connected with the image feature point detection module, and is used to predict the position of the road surface feature points in the next frame according to the vehicle motion state and the external parameters of the vehicle-mounted camera; the feature matching module matches the continuous frame feature points based on the local image information and motion information and outputs the pairing information.

[0029] like Figure 1 As shown, the road surface image feature matching method based on region and motion information in this embodiment is as follows:

[0030] Step 1) Data input: The algorithm first reads in continuous frame images, the current vehicle motion state, and the camera internal and external parameters. The vehicle motion state includes the vehicle velocity V and angular velocity YawRate. The continuous frame images include their image information Im1, Im2 and corresponding timestamps. The time difference dt between the two frames can be calculated. The camera internal parameters are the matrix K, which includes the camera focal length and the main point pixel coordinates. The external parameters include the rotation matrix R of the camera relative to the world coordinates. e0 and the translation vector t w The world coordinates are established in the following way: the ground below the center of the rear axle of the vehicle is taken as the world coordinate origin, the vehicle forward is the positive direction of the X axis, and the vertical direction upward from the ground is the positive direction of the Z axis.

[0031] Step 2) FAST feature point extraction: First, extract the feature points in Im1 and Im2. The extraction algorithm uses FAST corner detection. FAST corner detection is a corner detection algorithm proposed by Edward Rosten and Tom Drummond in 2006. Its basic principle is: if a pixel point has different attributes from enough pixels in its surrounding area, then the pixel point may be a corner point. Considering a grayscale image, that is, if the grayscale value of the point is greater or less than the grayscale value of enough pixels in its surrounding area, then the point may be a corner point.

[0032] In order to make the extracted corner points distributed as evenly as possible on the image and prevent most of the corner points from being located at the junction of the sky and the land, the algorithm divides the image into regions and extracts each region separately. The specific method is: let the image width be W and the height be H, then take the area of ​​each local region as 30*30 pixels, and divide the image into W / 30*H / 30 square regions. The maximum number of corner points in each region is the same, which ensures that the feature points are evenly distributed on the image.

[0033] Step 3) Preliminary screening of road feature points: This step mainly performs preliminary screening of road feature points to reduce the amount of calculation for subsequent prediction and matching. The screening is based on the position and image information of the on-board camera. Specifically, the width of the on-board camera screen is set to W, the height is set to H, and for the front and rear cameras, the feature point extraction area is For the side-view camera, the feature point extraction area is This can ensure that the extracted feature points are located on the road surface as much as possible.

[0034] Step 4) Feature point position prediction: Figure 2 As shown, a camera model is established based on the parameters of the vehicle-mounted camera, and then the pixel coordinates of the feature points of the first frame of the road are converted into world coordinates. At the same time, with the feature point in the first frame of the image as the center and d as the side length, a partial image of the first frame of the image is intercepted as the image to be matched, and the coordinate position of the feature point of the first frame of the image in the second frame of the image is calculated based on the vehicle body motion information and the camera model. The specific method is as follows:

[0035] Taking the i-th feature point in the first frame image as an example, other feature points are similar:

[0036] 1) Build a camera model based on the initial values.

[0037] The spatial position of the road feature point in the first frame is P i , P i =K[R e0 t w ]P wi , where K is the camera internal parameter matrix, R e0 is the camera rotation matrix, t w is the camera translation vector, is the vector from the camera installation position to the world coordinate origin,

[0038] P i =[x i ,y i ,1] is the pixel homogeneous coordinate of the i-th feature point, P wi =[x wi ,y wi ,z wi ,1] is the corresponding world homogeneous coordinate.

[0039] 2) With [x i ,y i ] is the center, d is the side length, and the local area Rect1 in the first frame image is intercepted as the image to be matched.

[0040] 3) Calculate the world coordinates P of feature point i based on the camera model wi =[x wi ,y wi ,z wi ,1], since the feature point is located on the road surface, its Z coordinate z wi = 0. Then predict the position of the feature point in the next frame based on the vehicle body speed. The vehicle body speed is V, the time interval between two frames is dt, and the vehicle rotation rate is YawRate. Then the world coordinates of the predicted point are p wipre =[x wi -V COS (YawRate·dt)dt,y wi -Vsin(YawRate·dt)dt,z wi ,1]. In the formula, p wipre is the world coordinate of the feature point of the first frame image, V is the vehicle body speed, dt is the time interval between two frames, and YawRate is the current rotation rate of the vehicle; after calculating the world coordinate of the feature point of the first frame image, the pixel coordinate p of the predicted point in the second frame image can be inferred through the camera model. ipre ,

[0041] Step 5) Continuous frame feature matching: Figure 3 As shown, the details are as follows

[0042] 1) With p ipre As the center, r is the radius to establish a search area, and search for feature points of the second frame image in the area. If there are no feature points in the area, the current match fails and the next match begins. If there are n feature points in the area, these feature points are taken as the center, d is the side length, and the local area Rect21~Rect2 of the second frame image is intercepted. n as the image to be matched.

[0043] 2) Calculate Rect21~Rect2 n The absolute error and SAD between Rect1 and Rect1 are calculated as follows for the jth feature point in the region, 1≤j≤n: Calculate the SAD of all feature points to be matched using the above formula and select the point with the smallest SAD. as the final matching result.

[0044] 3) Calculate the final matching point With the predicted point p ipreThe pixel distance is calculated as: If the calculated pixel distance is greater than the threshold value T, it indicates that it does not meet the road feature point motion condition and the set of feature point pairs should be discarded. In this embodiment, the threshold value T is set to 5 pixels. If the pixel distance is greater than the threshold value T, it indicates that it does not meet the road feature point motion condition and should be discarded.

[0045] Step 7) Data output: After all feature points in the continuous frames are matched, the feature point pairs are output, including the corresponding pixel coordinates and numbers.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Although the applicant has described the present invention in detail with reference to the preferred embodiments, ordinary technicians in this field should understand that those modifications or equivalent substitutions of the technical solution of the present invention without departing from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A road image feature matching method based on region and motion information, characterized in that: It includes the following steps: Step 1, data input, input the real-time continuous frame images of the road surface taken by the vehicle-mounted camera, the current movement information of the vehicle, and the parameters of the vehicle-mounted camera; Step 2, use the FAST corner point detection algorithm to perform feature point partition detection on the input continuous frame images; Step 3, based on the position and image information of the vehicle-mounted camera, preliminarily screen the road surface feature points after the partition detection, and determine that the extracted feature points are located on the road surface; when preliminarily screening the road surface feature points, set the image width to W and the height to H. For the front and rear view cameras, the feature point extraction area is For the side-view camera, the feature point extraction area is Step 4, predicting the position of the road feature point in the first frame in the second frame based on the vehicle body motion information and the camera model; The specific method for predicting the position of the road feature point includes: establishing a camera model based on the parameters of the vehicle-mounted camera, then converting the pixel coordinates of the road feature point in the first frame into the world coordinates, taking the feature point in the first frame image as the center and d as the side length, intercepting a local image of the first frame image as the image to be matched, and calculating the coordinate position of the feature point of the first frame image in the second frame image based on the vehicle body motion information and the camera model; step 5, taking the world coordinates of the predicted point in step 4 as the center, matching the feature points of consecutive frames based on the local image information and motion information and outputting the pairing information; the feature matching method is as follows: 1) With p ipre Establish a search area with centered at r and radius r, and search for feature points of the second frame image in the area; If there are no feature points in the area, the current match fails and the next match begins; if there are n feature points in the area, these feature points are taken as the center and d is the side length to intercept the local area Rect21~Rect2 of the second frame image n As the image to be matched; the p ipre is the pixel coordinate of the second frame image in the camera model; 2) Calculate Rect21~Rect2 n The absolute error and SAD between Rect1 and Rect1 are calculated as follows for the jth feature point in the region, 1≤j≤n: ， Calculate the SAD of all feature points to be matched using the above formula and select the point with the smallest SAD value As the final matching result; 3) Calculate the final matching point With the predicted point p ipre The pixel distance is calculated as: If the calculated pixel distance is greater than the threshold T, it indicates that it does not meet the road feature point motion condition, and the group of feature point pairs is discarded.

2. The road surface image feature matching method based on region and motion information according to claim 1, characterized in that: In step 1, the vehicle's current motion information includes the vehicle's own speed V and the vehicle's current angular velocity YawRate; the vehicle-mounted camera parameters include the vehicle-mounted camera internal parameters and the vehicle-mounted camera external parameters. The matrix K of the vehicle-mounted camera internal parameters includes the camera's focal length, principal point, and distortion coefficient. The vehicle-mounted camera external parameters include the camera rotation matrix R relative to the world coordinates. e0 and the translation vector t w .

3. The road surface image feature matching method based on region and motion information according to claim 1 or 2, characterized in that: When establishing the camera model, the image pixel position of the road feature point in the first frame is P i , P i =K[R e0 t w ]P wi , where K is the camera internal parameter matrix, R e0 is the camera rotation matrix, t w is the camera translation vector, P wi The vector from the camera installation position to the world coordinate origin.

4. The road surface image feature matching method based on region and motion information according to claim 3 is characterized in that: Calculate the world coordinates of the feature points of the first frame image based on the camera model: p wipre =[x wi -V COS (YawRate·dt)dt,y wi -Vsin(YawRate·dt)dt,z wi ,1]; In the formula, p wipre is the world coordinate of the feature point of the first frame image, V is the vehicle body speed, dt is the time interval between two frames, and YawRate is the current rotation rate of the vehicle; after calculating the world coordinate of the feature point of the first frame image, the pixel coordinate p of the predicted point in the second frame image can be inferred through the camera model. ipre , 5. A road surface image feature matching system based on region and motion information, used to implement the road surface image feature matching method based on region and motion information as claimed in claim 1, characterized in that: It includes a vehicle-mounted camera, an image feature point detection module, a position prediction module and a feature matching module. The feature point detection module is connected to the vehicle-mounted camera and the vehicle controller, and is used to extract the continuous frame image feature points and vehicle camera parameters of the vehicle-mounted camera and the current vehicle motion information in the vehicle controller. At the same time, the extracted continuous frame image feature points are detected and their pixel positions are returned; The position prediction module is communicatively connected with the image feature point detection module, and is used to predict the position of the road feature point in the next frame according to the vehicle motion state and the external parameters of the vehicle-mounted camera; the feature matching module matches the feature points of consecutive frames based on local image information and motion information and outputs pairing information.

6. The road surface image feature matching system based on region and motion information according to claim 5, characterized in that: It also includes a road feature point preliminary screening module, which is bidirectionally connected to the image feature point detection module for preliminary screening of road feature points after partition detection, determines that the extracted feature points are located on the road surface, and feeds back the preliminary screening information to the image feature point detection module.

Images