A real-time occupancy grid update method for forward-looking assisted driving vehicles

Abstract

The present invention discloses a method for updating the forward-looking real-time occupancy grid of an assisted driving vehicle, comprising obtaining the real-time displacement of the vehicle dis t ; Acquire feature points of the on-board monocular image; Track and match the feature points to obtain a depth map of the on-board monocular image; Calculate the three-dimensional coordinates of the feature points using the depth map; Based on the three-dimensional coordinates, select feature points that are above the ground and below the vehicle height value to generate an occupancy grid map of the vehicle; Repeat the steps to continuously obtain the latest occupancy grid map of the vehicle. The method for updating the forward-looking real-time occupancy grid of an assisted driving vehicle of the present invention has the advantages of being able to solve the problem that the monocular vehicle initialization process cannot provide a large six-axis posture change to obtain a depth map, and solving the problem that the occupancy grid cannot be effectively generated based on traditional on-board monocular vision.

Description

Technical Field

[0001] The present invention relates to an environment perception technology for vehicle assisted driving, and in particular to a method for updating a forward-looking real-time occupancy grid of an assisted driving vehicle. Background Art

[0002] An occupancy grid map is a map representation that divides the environment into fixed-size grid cells and records the occupancy status of each cell (e.g., occupied by an obstacle, free, or unknown). This representation is simple and intuitive, suitable for modeling a variety of environments. Each cell in the occupancy grid has a value representing the probability of occupancy. A value close to 1 indicates a high degree of certainty that an obstacle exists in the cell. A value close to 0 indicates a high degree of certainty that the cell is unoccupied and free of obstacles.

[0003] Occupancy grid technology is a key environmental perception technology in artificial intelligence fields such as assisted driving and robotics. It uses visual or laser sensors to perceive surrounding obstacles and generate discrete grid occupancy information for more comprehensive environmental perception. Occupancy grids play a crucial role in the maneuvering and parking of assisted driving vehicles. Occupancy grid technology is primarily used for dynamic and static object detection and tracking, probabilistic estimation of occupancy maps around the vehicle, and for deriving various driving scenarios. It provides vehicles with real-time ground area occupancy information, providing input for subsequent vehicle planning, control, and obstacle avoidance, laying the foundation for assisted driving and is an indispensable key technology in the field.

[0004] Currently, vehicle-side occupancy grid algorithms are mainly divided into two directions. (1) The first is an occupancy grid algorithm based on three-dimensional point cloud coordinates, which is mainly based on LiDAR. This method has a good advantage in terms of accuracy, and the grid generation calculation logic is relatively simple, but the cost is relatively high. (2) The other direction is mainly based on the image depth information generated by visual images. It has a lower cost, but the calculation logic is more complex, and it often uses binocular SLAM or deep learning methods.

[0005] Assisted driving vehicles typically use a monocular camera to capture images of the vehicle's surroundings. However, vehicle-mounted monocular SLAM (Simultaneous Local Area Mapping) is subject to initialization requirements and significant pose variations, making it difficult for vision-based occupancy grid generation methods to effectively adapt to monocular assisted driving configurations. Summary of the Invention

[0006] In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a real-time occupancy grid update method for the forward vision of an assisted driving vehicle, so as to solve the problem that the occupancy grid cannot be effectively generated based on traditional on-board monocular vision in assisted driving vehicles.

[0007] The present invention adopts the following technical solutions to solve the technical problems.

[0008] The present invention provides a method for updating a forward-looking real-time occupancy grid of an assisted driving vehicle, comprising the following steps:

[0009] Step 1: Get the vehicle's real-time displacement dis t ;

[0010] Step 2: Obtain feature points of the vehicle-mounted monocular image;

[0011] Step 3: Tracking and matching the feature points of the vehicle-mounted monocular image to obtain a depth map of the vehicle-mounted monocular image;

[0012] Step 4: Calculate the three-dimensional coordinates of the feature points using the depth map of the vehicle-mounted monocular image;

[0013] Step 5: Based on the three-dimensional coordinates of each feature point, select feature points that are above the ground and below the vehicle height value H to generate a vehicle occupancy grid map;

[0014] Step 6: Repeat steps 1 to 5 to continuously obtain the latest vehicle occupancy grid map.

[0015] The structural features of the method for updating the forward-looking real-time occupancy grid of an assisted driving vehicle of the present invention are also as follows:

[0016] Furthermore, in step 1, the process of obtaining the real-time displacement of the vehicle includes the following steps:

[0017] Step 11: Obtain vehicle driving information and determine the validity of the vehicle driving information;

[0018] Step 12: Calculate the vehicle's real-time displacement dis based on the real-time speed in the vehicle's driving information t .

[0019] Furthermore, in step 12, the real-time displacement of the vehicle dis t The calculation formula is shown in the following formula (1);

[0020] (1)

[0021] In formula (1), the speed t is the vehicle speed at time t, the speed t-1 is the vehicle speed at time t-1, and △t is the time difference between time t and time t-1.

[0022] Furthermore, in step 2, obtaining feature points of the vehicle-mounted monocular image includes the following steps:

[0023] Step 21: Read the vehicle-mounted monocular image in the vehicle-mounted system, determine the validity of the vehicle-mounted monocular image, and retain the valid vehicle-mounted monocular image;

[0024] Step 22: Perform Shi-Tomasi feature point selection on the vehicle-mounted monocular image.

[0025] Furthermore, in step 22, the Shi-Tomasi feature point selection process includes the following steps:

[0026] Step 221: Read the grayscale value I (x, y) of the valid vehicle-mounted monocular image;

[0027] Step 221: Calculate the gradient I of the gray value I (x, y) in the x direction x and the gradient in the y direction I y ;

[0028] Step 223: constructing a structure tensor M for each pixel of the vehicle-mounted monocular image;

[0029] Step 224: Calculate the eigenvalue of the structure tensor M of each pixel, and calculate the corner point response value R of the pixel based on the eigenvalue;

[0030] Step 225: Determine whether the pixel is a corner point based on the corner point response value R.

[0031] Furthermore, in step 223, the structure tensor M is expressed as follows:

[0032] (3)

[0033] In formula (3), w(x, y) is the window function, x and y are the horizontal and vertical coordinates of the pixel (x, y) in the vehicle-mounted monocular image, respectively. x and I y are the gradient of I(x, y) in the x direction and the gradient in the y direction respectively.

[0034] Furthermore, in step 224, the calculation formulas of the eigenvalues ​​λ1 and λ2 are shown in the following formula (4);

[0035] (4)

[0036] In formula (4), trace(M) is the trace function of the two-dimensional matrix M; det(M) is the evaluation function of the two-dimensional matrix M.

[0037] Furthermore, in step 224, the corner point response value R is calculated as shown in the following formula (5):

[0038] (5)

[0039] In formula (5), min() is the function for finding the minimum value.

[0040] Furthermore, in step 3, the process of obtaining the depth map of the vehicle-mounted monocular image includes the following steps:

[0041] Step 31: Perform feature point tracking and matching based on Euclidean distance for multiple consecutive frames of the vehicle-mounted monocular image;

[0042] Step 32: Based on the tracked and matched feature points, calculate the optical flow change results of the feature points, and remove the feature points whose angle deviation does not meet the requirements according to the optical flow change results;

[0043] Step 33: Use vehicle real-time displacement dis t Perform scale calibration and obtain the depth map of the vehicle-mounted monocular image.

[0044] Furthermore, in step 4, when calculating the three-dimensional coordinates of the feature points, the pixel coordinates of the feature points are converted into three-dimensional coordinates.

[0045] Compared with the existing technology, the beneficial effects of the present invention are embodied in:

[0046] The present invention discloses a method for updating the forward-looking real-time occupancy grid of an assisted driving vehicle, comprising obtaining the real-time displacement of the vehicle dis t ; Obtain feature points of the on-board monocular image; Track and match the feature points to obtain a depth map of the on-board monocular image; Calculate the three-dimensional coordinates of the feature points using the depth map; Based on the three-dimensional coordinates, select feature points that are above the ground and below the vehicle height value to generate a vehicle occupancy grid map; Repeat the steps to continuously obtain the latest vehicle occupancy grid map.

[0047] The real-time occupancy grid update method for the forward vision of an assisted driving vehicle of the present invention has the advantages of being able to solve the problem that the monocular vehicle initialization process cannot provide large six-axis posture changes to obtain a depth map, and solving the problem that the occupancy grid cannot be effectively generated based on traditional vehicle-mounted monocular vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the method for updating the forward-looking real-time occupancy grid of an assisted driving vehicle according to the present invention.

[0049] Figure 2 Schematic diagram of the position and angle relationship between feature points and optical flow center points under multiple frames of the real-time occupancy grid update method for the assisted driving vehicle forward view of the present invention.

[0050] Figure 3 This is a pixel map of the real-time occupancy grid update method for the driver-assisted vehicle forward vision of the present invention.

[0051] The present invention will be further described below through specific implementation methods in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0052] See also Figures 1 to 3 The present invention provides a method for updating a forward-looking real-time occupancy grid of an assisted driving vehicle, comprising the following steps:

[0053] Step 1: Get the vehicle's real-time displacement dis t ;

[0054] Obtain vehicle driving information from the vehicle system and calculate the vehicle real-time displacement dis based on the vehicle driving information t ;

[0055] Step 2: Obtain feature points of the vehicle-mounted monocular image;

[0056] The vehicle-mounted monocular image is captured by the vehicle-mounted camera; the validity of the vehicle-mounted monocular image is judged, invalid images are eliminated, and then the feature points of the vehicle-mounted monocular image are obtained;

[0057] Step 3: Tracking and matching the feature points of the vehicle-mounted monocular image to obtain a depth map of the vehicle-mounted monocular image;

[0058] Track and match the feature points obtained in step 2, and obtain the depth map of the vehicle-mounted monocular image based on the optical flow change calculation results between multiple frames constrained by the vehicle's real-time displacement information;

[0059] Step 4: Calculate the three-dimensional coordinates of the feature points using the depth map of the vehicle-mounted monocular image;

[0060] The three-dimensional coordinates corresponding to each feature point of the vehicle-mounted monocular image are calculated through the depth map of the vehicle-mounted monocular image.

[0061] Step 5: Based on the three-dimensional coordinates of each feature point, select feature points that are above the ground and below the vehicle height value H to generate a vehicle occupancy grid map;

[0062] According to the three-dimensional coordinates, feature points within a range above the ground and below a maximum vehicle height H are screened for use in generating an occupancy grid map.

[0063] Step 6: Repeat steps 1 to 5 to continuously obtain the latest vehicle occupancy grid map.

[0064] Repeat the process of obtaining vehicle driving information and vehicle-mounted monocular images in steps 1 and 2, and process them through steps 3 to 5 to continuously obtain the latest occupancy grid map.

[0065] like Figure 1The present invention provides a real-time occupancy grid update method for the forward vision of an assisted driving vehicle. By using a vehicle-mounted forward-view monocular camera to calculate a depth map based on optical flow, the method solves the problem that the monocular vehicle initialization process cannot provide a large six-axis posture change to obtain a depth map. By combining the vehicle-mounted forward-view image with the depth map to generate an occupancy grid method, the method solves the problem that the occupancy grid cannot be effectively generated based on traditional vehicle-mounted monocular vision.

[0066] In specific implementation, in step 1, the process of obtaining the real-time displacement of the vehicle includes the following steps:

[0067] Step 11: Obtain vehicle driving information and determine the validity of the vehicle driving information;

[0068] Obtain vehicle driving information from the vehicle system and determine the validity of the vehicle driving information. Vehicle driving information mainly includes real-time vehicle speed and movement time t. Real-time vehicle speed is the real-time speed at the moment of reading vehicle driving information. Define movement time t=0 when the vehicle starts, and the vehicle system automatically records the timestamp of the standard time at time t=0; movement time t is the time the vehicle has been moving continuously from the start time to the current time t. In specific implementation, the read real-time vehicle speed and movement time t are one-to-one corresponding. Define the vehicle speed read at time t as speed t .

[0069] The validity of real-time speed is determined by checking whether its absolute value is less than the upper speed limit. The validity of movement time t is determined by checking whether it is greater than zero. The validity of real-time speed and movement time t is determined, and outliers such as Nan values ​​are eliminated.

[0070] Step 12: Calculate the vehicle's real-time displacement dis based on the real-time speed in the vehicle's driving information t .

[0071] In the specific implementation, in step 12, the vehicle real-time displacement dis t The calculation formula is shown in the following formula (1);

[0072] (1)

[0073] In formula (1), the speed t is the vehicle speed at time t, the speed t-1 is the vehicle speed at time t-1, and Δt is the time difference between time t and time t-1. In the vehicle system, the vehicle speed is collected and recorded every 50ms starting from the time t=0, that is, the value is generally taken as Δt=50ms.

[0074] In specific implementation, in step 2, obtaining feature points of the vehicle-mounted monocular image includes the following steps:

[0075] Step 21: Read the vehicle-mounted monocular image in the vehicle-mounted system, determine the validity of the vehicle-mounted monocular image, and retain the valid vehicle-mounted monocular image;

[0076] The vehicle-mounted monocular image is read in real time by the vehicle-mounted camera, and the validity of the vehicle-mounted monocular image is determined. The vehicle-mounted camera takes a picture every 50ms to obtain a frame of the vehicle-mounted monocular image. The validity determination process of each frame of the vehicle-mounted monocular image mainly includes: determining whether the grayscale value of each frame of the vehicle-mounted monocular image is abnormal, and eliminating overexposed or too dark grayscale values; performing image traversal on the acquired vehicle-mounted monocular image to obtain the average grayscale value gray_avg of the vehicle-mounted monocular image; when the gray_avg of a certain frame of the vehicle-mounted monocular image is greater than 200 or less than 50, that is, gray_avg>200 or gray_avg<50, the frame of the vehicle-mounted monocular image is considered to be abnormal, that is, there is overexposed or too dark conditions, and the frame of the vehicle-mounted monocular image in this case is eliminated.

[0077] Step 22: Perform Shi-Tomasi feature point selection on the vehicle-mounted monocular image.

[0078] In specific implementation, in step 22, the process of selecting the Shi-Tomasi feature points includes the following steps:

[0079] Step 221: Read the grayscale value I (x, y) of the valid vehicle-mounted monocular image;

[0080] The monocular image from the vehicle system is a single-channel grayscale image, such as Figure 3 As shown. The grayscale value I (x, y) is expressed using a matrix of n × m dimensions;

[0081] Step 221: Calculate the gradient I of the gray value I (x, y) in the x direction x and the gradient in the y direction I y ;

[0082] Gradient I x and gradient I y The expression of is shown in the following formula (2);

[0083] (2)

[0084] In formula (2), I (x, y) is the gray value function of the vehicle-mounted monocular image, The coordinates of the vehicle-mounted monocular image are The grayscale value of , x and y are the horizontal and vertical coordinates of the pixel (x, y) in the vehicle-mounted monocular image, u and v are the directional displacements of the horizontal and vertical coordinates, u and v are configurable values ​​and are set to 2 during implementation; To find the sign of partial derivative. Figure 3 Shown is a pixel map of I(x,y).

[0085] Step 223: constructing a structure tensor M for each pixel of the vehicle-mounted monocular image;

[0086] Step 224: Calculate the eigenvalue of the structure tensor M of each pixel, and calculate the corner point response value R of the pixel based on the eigenvalue;

[0087] Step 225: Determine whether the pixel is a corner point based on the corner point response value R.

[0088] In specific implementation, in step 223, the expression of the structure tensor M is shown in the following formula (3);

[0089] (3)

[0090] In formula (3), w(x, y) is the window function, x and y are the horizontal and vertical coordinates of the pixel (x, y) in the vehicle-mounted monocular image, respectively. x and I y are the gradient of I(x, y) in the x direction and the gradient in the y direction respectively.

[0091] The w(x, y) is a window function, preferably a Gaussian function, which is used to weight the gradient information of the local area.

[0092] In specific implementation, in step 224, the calculation formulas of the characteristic values ​​λ1 and λ2 are shown in the following formula (4);

[0093] (4)

[0094] In formula (4), trace(M) is the trace function of the two-dimensional matrix M; det(M) is the evaluation function of the two-dimensional matrix M.

[0095] In specific implementation, in step 224, the calculation formula of the corner point response value R is shown in the following formula (5);

[0096] (5)

[0097] In formula (5), min() is the function for finding the minimum value.

[0098] λ1 and λ2 are the eigenvalues ​​of the structure tensor M. When R is greater than a pre-set threshold, the pixel is considered a corner point. In one embodiment, the threshold is set to 0.01. That is, when R > 0.01, the pixel corresponding to R is considered a corner point and is considered a feature point in the vehicle-mounted monocular image. Otherwise, it is considered a non-feature point and discarded.

[0099] In specific implementation, in step 3, the process of obtaining the depth map of the vehicle-mounted monocular image includes the following steps:

[0100] Step 31: Perform feature point tracking and matching based on Euclidean distance for multiple consecutive frames of the vehicle-mounted monocular image;

[0101] Track and match the feature points in step 2, and based on the real-time displacement of the vehicle t Constrain the calculation results of optical flow changes between multiple frames of images to obtain the depth map of the vehicle-mounted monocular image.

[0102] Perform pixel distance-based feature point tracking and matching on three consecutive frames of images and retain the matching results. Calculate the pixel distance dis between the feature points of the two previous and next frames of the vehicle-mounted monocular image. pixel , retain the feature point P that is closest and can be tracked in three consecutive frames. For example, for three frames of images, such as the vehicle-mounted monocular image at time t-1, the vehicle-mounted monocular image at time t and the vehicle-mounted monocular image at time t+1, it is necessary to calculate the pixel distance dis between time t-1 and time t pixel and the pixel distance dis between time t and time t+1 pixel For three frames of images, it is necessary to calculate the two pixel distances dis pixel , each time the two results are calculated, the pixel distance dis is taken pixel The smaller result is the current tracking matching result (there are mismatches, which will be eliminated in subsequent steps).

[0103] The pixel distance dis pixel The calculation formula is shown in the following formula (6).

[0104] (6)

[0105] In formula (6), Px t and Py t are the x pixel coordinates and y pixel coordinates of the feature point P at time t, Px t-1 and Py t-1 are the x pixel coordinates and y pixel coordinates of the feature point P at time t-1 respectively.

[0106] In the specific calculation, three consecutive frames of vehicle-mounted monocular images are taken as a group. For each feature point in the first frame of the vehicle-mounted monocular image, a tracking match is performed with each feature point in the second frame of the vehicle-mounted monocular image. For example, the first frame of the vehicle-mounted monocular image includes n feature points, and the second frame of the vehicle-mounted monocular image includes m feature points. The first feature point P in the first frame of the vehicle-mounted monocular image is 11 , it is necessary to track and match all m feature points of the second frame of the vehicle-mounted monocular image, and calculate the m pixel distance dis pixel ; The m-pixel distance dis of the first feature point in the first frame of the vehicle-mounted monocular image pixel In the example, select the minimum pixel distance dis pixel The corresponding feature point P of the second frame of the vehicle-mounted monocular image 2min , feature point P 2min As the first feature point P of the first frame of the vehicle-mounted monocular image 11 The tracking and matching result is that the feature point P of the second frame of the vehicle-mounted monocular image is 2min The first feature point P of the first frame of the vehicle-mounted monocular image 11 Match each other, the first feature point P of the first frame of the vehicle-mounted monocular image 11 and the feature point P of the second frame of the vehicle-mounted monocular image 2min = is the image of the same point on the real object in the two vehicle-mounted monocular images. Repeat the calculation process of Formula 6 to track and match the n feature points of the first vehicle-mounted monocular image with all m feature points of the second vehicle-mounted monocular image one by one.

[0107] Similarly, for each feature point in the second frame of the vehicle-mounted monocular image, the same tracking and matching process is performed with each feature point in the third frame of the vehicle-mounted monocular image.

[0108] Step 32: Based on the tracked and matched feature points, calculate the optical flow change results of the feature points, and remove the feature points whose angle deviation does not meet the requirements according to the optical flow change results;

[0109] like Figure 2 As shown, the pixel angle angel of the two feature points that successfully match each other between the two frames of vehicle-mounted monocular images is calculated, and the angle angel of the central optical flow is compared with the angle angel of the central optical flow. center Compare and eliminate the mismatched points with large angle deviation △angel. The calculation formula of the pixel direction angel of the feature points after the front and back matching is shown in the following formula (7). Figure 2 Pixel angle angel and center optical flow angle angel center , the upward arrow direction is the reference direction.

[0110] (7)

[0111] The Angel center The calculation formula is shown in the following formula (8).

[0112] (8)

[0113] In formula (7) and formula (8), angel is the pixel angle of the feature point of the two frames of vehicle-mounted monocular images; Figure 2 Taking time t as an example, the pixel angle of Pt and Pt-1 is the angle between the vector direction from the feature point at time t-1 to the feature point at time t and the reference direction.

[0114] angel center From a certain feature point to the center point P of the vehicle-mounted monocular image of this frame center direction, Px t and Py t They are the x-coordinate and y-coordinate of the feature point P in the vehicle-mounted monocular image at time t, Px t-1 and Py t-1 They are the x-coordinate and y-coordinate of the feature point P in the vehicle-mounted monocular image at time t-1; for each frame of the vehicle-mounted monocular image, Pcenter x is the center point P of the vehicle-mounted monocular image of this frame center The x-coordinate, Pcenter y is the center point P of the vehicle-mounted monocular image of this frame center The x coordinate of . In specific implementation, Pcenter x and Pcenter y Set to half of the pixel length and pixel width of the image; express the grayscale image of the vehicle-mounted monocular image in a rectangular form, Pcenter x and Pcenter y is the coordinate of the center point of the rectangle. Figure 2 , Pcenter is the center point of the rectangular graph.

[0115] When eliminating the mismatched points with large angle deviation △angel, for angle deviation △angel=angel-angel center， beg The value of The value of is taken as the result of optical flow change. All relevant feature points greater than the preset value are eliminated. For example, when the preset value is 30%, Feature points with an angle deviation greater than 30% are identified as feature points that do not meet the requirements and are removed from both the previous and next two frames of vehicle-mounted monocular images.

[0116] Step 33: Use vehicle real-time displacement dis tPerform scale calibration and obtain the depth map of the vehicle-mounted monocular image.

[0117] Step 331: Use the vehicle real-time displacement dis t The pixel distance dis between the two feature points that successfully match each other in the two frames of vehicle-mounted monocular images pixel , the pixel distance dis between two feature points in the vehicle-mounted monocular image can be calculated pixel Real-time displacement of vehicles in the real world t Average scale pixel d , see the following formula (9).

[0118] (9)

[0119] Step 332: Set the edge depth of the vehicle-mounted monocular image to 0, and the scale of the edge point is also set to 0 by default. center Draw a circle with 5° as the center, and divide the whole image into 72 parts. Figure 2 As shown in the figure, the upward arrow direction is used as the reference direction and the starting direction, and the direction corresponding to the arrow is defined as 0°. All feature points in the vehicle-mounted monocular image are grouped into a group with a deflection angle of 5°, and an angle traversal is performed to obtain all feature points in each 5° angle area.

[0120] Calculate the angle of each feature point in each 5° area along the angel center The opposite direction to the edge point of the image edge. Figure 2 , the edge point corresponding to the feature point Pt is P center The starting point is the intersection of the Pt ray and the edge line of the image. Calculate the pixel distance d between each feature point and the edge point corresponding to the feature point edge , determine the distance d of all pixels edge The minimum value d in edgemin , select the minimum value d edgemin The corresponding feature point P1(x,y) and edge point P0; that is, select the feature point P1(x,y) closest to the edge, such as Figure 2 The point closest to the edge is Pt-1. edgemin Scale pixel of feature point P1 (x, y) d1 Multiply to obtain the depth value Depth1(x,y) of the feature point P1 (x,y), and assign the depth value Depth1(x,y) to the feature point P1 (x,y) as the depth value of the feature point P1 (x,y) in the vehicle-mounted monocular image.

[0121] The calculation formula for the depth value Depth1(x,y) of the feature point P1(x,y) closest to the edge is shown in the following formula (10).

[0122] (10)

[0123] In formula (10), P 1x and P 1y are the horizontal and vertical coordinates of the feature point P1 (x, y), P 0x and P 0y They are the horizontal and vertical coordinates of the edge point P0 of the feature point P1 (x, y), where the scale of the edge point P0 defaults to 0.

[0124] Calculate the feature points in each 5° angle area. Except for the feature point P1 (x, y) closest to the edge, calculate the feature point P first. j (x,y) and the previous feature point P j-1 The depth difference Valuej between (x,y).

[0125] (11)

[0126] In formula (11), P jx and P jy They are feature points P j The horizontal and vertical coordinates of (x,y), P j-1x and P j-1y They are feature points P j-1 (x,y) horizontal and vertical coordinates, pixel dj is the feature point P j (x,y) scale, pixels dj-1 is the feature point P j-1 The scale of (x,y).

[0127] Step 333: For each feature point in each 5° angle area, calculate the depth value Depth corresponding to the feature point, and assign the depth value Depth corresponding to the feature point to the feature point. When calculating specifically, first calculate the distance P in the 5° angle area. center The depth value Depth1 of the farthest feature point P1, and then calculate the distance P center The depth difference Value2 of the second farthest feature point P2 is calculated again until the distance P centerCalculate the depth difference Value for all feature points within each 5° angular region in order of distance. Assign the depth value Depth1 to feature point P1 as its depth value, Depth1 + Value2 to feature point P2 as its depth value Depth2, Depth2 + Value3 to feature point P3 as its depth value Depth3, and so on. Assign a depth value to all feature points within each 5° angular region.

[0128] Step 334: Assign depth values ​​to all feature points in the vehicle-mounted monocular image. After traversing all feature points, obtain the initial depth map of the vehicle-mounted monocular image. Since the above process only assigns depth values ​​to all feature points, the depth values ​​of non-feature points are not assigned, that is, the depth values ​​of non-feature points in the initial depth map are 0.

[0129] Then, the initial depth map is traversed based on each pixel from the upper left corner to the lower right corner of the image to determine whether the depth value of each pixel is null. For a pixel Pinter with a null depth value, if other pixels within four pixels of the pixel Pinter have depth values, the pixel P closest to the selected pixel Pinter is selected. org , pixel P org The depth value is Depth org .

[0130] Then based on P org Corresponding scale pixel dorg The linear interpolation is used to fill in the empty pixels in the initial depth map. The linear interpolation formula is shown in the following formula (12).

[0131] (12)

[0132] in, P org (x,y) to the center point The distance between Pixels (x,y) to the center point Vinter is the distance between the corresponding pixel Pinter and pixel P on the initial depth map. org The depth difference between (x,y), Porg x and Porg y The pixels P org The horizontal and vertical coordinates of (x, y), Pcenter x and Pcenter y The center point P center The x-coordinate is the horizontal and vertical coordinates of the Pinter x and Pintery are the horizontal and vertical coordinates of the pixel Pinter (x, y), pixel dorg For pixel P org The scale of (x,y).

[0133] Depth value of pixel Pinter inter For pixel P org Depth value org The sum of Vinter, namely: Depth inter = Depth org +Vinter.

[0134] Through step 334 , depth values ​​are supplemented and assigned to some pixels in the initial depth map to obtain a final depth map.

[0135] In specific implementation, in step 4, when calculating the three-dimensional coordinates of the feature points, the pixel coordinates of the feature points are converted into three-dimensional coordinates.

[0136] Convert the pixel coordinates (x, y) of the feature points in the depth map obtained in step 3 to the three-dimensional coordinates (X, Y, Z) in the camera system. The conversion formula is shown in the following formula (10).

[0137] (10)

[0138] In formula (10), x and y are the horizontal and vertical coordinates of the feature point pixel coordinates, respectively; X, Y, and Z are the physical world coordinates of the vehicle-mounted camera corresponding to the feature point in the image coordinate system, respectively; depth is the depth value corresponding to the feature point pixel coordinates (x, y) in the depth map; cx and cy are the focus pixel coordinates in the vehicle-mounted camera intrinsic parameters, and fx and fy are the focal lengths in the vehicle-mounted camera intrinsic parameters.

[0139] In step 5, the physical world height information Y of the vehicle-mounted camera calculated from the feature points is filtered, and only the calculation results between the height above the ground 0 and the vehicle height H are retained.

[0140] In step 5, when generating the vehicle occupancy grid map, the vehicle occupancy grid map is first constructed. The occupancy grid map is a two-dimensional grid map of the physical world of the vehicle-mounted camera without the height information. The horizontal and vertical coordinates of the physical world coordinates of the vehicle-mounted camera calculated for the feature points are the coordinates under the occupancy grid map. The occupancy grid map corresponding to the physical world coordinates of the vehicle-mounted camera calculated for the feature points is assigned a probability value.

[0141] Step 51: Construct a plane occupancy grid map, with the length and width of each grid being 0.5m, and set the value range of each grid to be between 0 and 1.0;

[0142] Step 52: Assign a value of 1.0 to feature points with a height between 0 and H in the grid;

[0143] Step 53: Assign a value of 0 to feature points with a height between 0 and H that do not exist in the grid;

[0144] Step 54: Traverse the zero-value area of ​​the grid. If more than two of the eight nearest neighboring grids around the grid have a value of 1.0, interpolate them. The interpolation value is the number of grids with a value of 1.0 divided by 9 (the eight surrounding grids plus the center grid).

[0145] The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle of the present invention has the following advantages:

[0146] 1. Provide a method for calculating depth maps based on optical flow for a vehicle-mounted forward-looking monocular camera, which can effectively solve the problem that the monocular vehicle initialization process cannot provide large six-axis posture changes to obtain depth maps.

[0147] 2. Provide a method for generating an occupancy grid by combining the vehicle-mounted front view image with the depth map, which can effectively solve the problem that the occupancy grid cannot be effectively generated based on traditional vehicle-mounted monocular vision.

[0148] The present invention provides a method for updating the forward-looking real-time occupancy grid of an assisted driving vehicle. With the development of assisted driving technology and breakthroughs in sensors, perception, and planning, it can control costs and achieve robust occupancy grid generation during the parking process of assisted driving vehicles. This will have a significant impact on the planning and control of assisted driving vehicles and provide passengers with a better user experience.

[0149] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0150] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for updating the forward-view real-time occupancy grid of an assisted driving vehicle, characterized by: The steps include: Step 1: Get the vehicle's real-time displacement dis t ; Step 2: Obtain feature points of the vehicle-mounted monocular image; Step 3: Tracking and matching the feature points of the vehicle-mounted monocular image to obtain a depth map of the vehicle-mounted monocular image; Track and match the feature points in step 2, and based on the real-time displacement of the vehicle in step 1 t Constrain the calculation results of optical flow changes between multiple frames of images to obtain the depth map of the vehicle-mounted monocular image; Step 4: Calculate the three-dimensional coordinates of the feature points using the depth map of the vehicle-mounted monocular image; Step 5: Based on the three-dimensional coordinates of each feature point, select feature points that are above the ground and below the vehicle height value H to generate a vehicle occupancy grid map; Step 6: Repeat steps 1 to 5 to continuously obtain the latest vehicle occupancy grid map.

2. The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle according to claim 1 is characterized in that: In step 1, the process of obtaining the real-time displacement of the vehicle includes the following steps: Step 11: Obtain vehicle driving information and determine the validity of the vehicle driving information; Step 12: Calculate the vehicle's real-time displacement dis based on the real-time speed in the vehicle's driving information t .

3. The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle according to claim 2 is characterized in that: In step 12, the vehicle real-time displacement dis t The calculation formula is shown in the following formula (1); （1） In formula (1), the speed t is the vehicle speed at time t, the speed t-1 is the vehicle speed at time t-1, and △t is the time difference between time t and time t-1.

4. The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle according to claim 1 is characterized in that: In step 2, obtaining feature points of the vehicle-mounted monocular image includes the following steps: Step 21: Read the vehicle-mounted monocular image in the vehicle-mounted system, determine the validity of the vehicle-mounted monocular image, and retain the valid vehicle-mounted monocular image; Step 22: Perform Shi-Tomasi feature point selection on the vehicle-mounted monocular image.

5. The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle according to claim 4 is characterized in that: In step 22, the process of selecting Shi-Tomasi feature points includes the following steps: Step 221: Read the grayscale value I (x, y) of the valid vehicle-mounted monocular image; Step 221: Calculate the gradient I of the gray value I (x, y) in the x direction x and the gradient in the y direction I y ; Step 223: constructing a structure tensor M for each pixel of the vehicle-mounted monocular image; Step 224: Calculate the eigenvalue of the structure tensor M of each pixel, and calculate the corner point response value R of the pixel based on the eigenvalue; Step 225: Determine whether the pixel is a corner point based on the corner point response value R.

6. The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle according to claim 5 is characterized in that: In step 223, the structure tensor M is expressed as follows: （3） In formula (3), w(x, y) is the window function, x and y are the horizontal and vertical coordinates of the pixel (x, y) in the vehicle-mounted monocular image, respectively. x and I y are the gradient of I(x, y) in the x direction and the gradient in the y direction respectively.

7. The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle according to claim 5 is characterized in that: In step 224, the calculation formulas of the eigenvalues ​​λ1 and λ2 are shown in the following formula (4); （4） In formula (4), trace(M) is the trace function of the two-dimensional matrix M; det(M) is the evaluation function of the two-dimensional matrix M.

8. The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle according to claim 7 is characterized in that: In step 224, the calculation formula of the corner point response value R is shown in the following formula (5); （5） In formula (5), min() is the function for finding the minimum value.

9. The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle according to claim 1 is characterized in that: In step 3, the process of obtaining the depth map of the vehicle-mounted monocular image includes the following steps: Step 31: Perform feature point tracking and matching based on Euclidean distance for multiple consecutive frames of the vehicle-mounted monocular image; Step 32: Based on the tracked and matched feature points, calculate the optical flow change results of the feature points, and remove the feature points whose angle deviation does not meet the requirements according to the optical flow change results; Step 33: Use vehicle real-time displacement dis t Perform scale calibration and obtain the depth map of the vehicle-mounted monocular image.

10. The method for updating the forward-view real-time occupancy grid of an assisted driving vehicle according to claim 1, characterized in that: In step 4, when calculating the three-dimensional coordinates of the feature points, the pixel coordinates of the feature points are converted into three-dimensional coordinates.

Images