Synchronous Localization and Real-time Generation Method of Three-dimensional Path for Panoramic Image of Driving Environment

Abstract

The present invention belongs to the field of intelligent driving technology, and particularly relates to a method for synchronous positioning and real-time generation of a three-dimensional path of a panoramic image of a driving environment. The present invention enhances the contrast of the collected images under low-light conditions, improves the accuracy of feature point extraction, and significantly improves the vehicle positioning accuracy by using an improved feature point matching algorithm. By combining the predicted position to update the feature point distribution in the panoramic image, the real-time generation and dynamic correction of the three-dimensional path map are realized, thereby effectively enhancing the adaptability and reliability of the system in complex and changing environments. This not only solves the problem of performance degradation of traditional methods in low-light environments, but also greatly improves the overall performance of the intelligent driving system, ensuring driving safety and the accuracy of path planning.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent driving, and particularly relates to a method for synchronous positioning of panoramic images of driving environments and real-time generation of three-dimensional paths. Background Art

[0002] Existing driving environment perception systems usually rely on sensor devices such as cameras and lidar (LiDAR) to obtain information about the vehicle's surrounding environment and process it through computer vision algorithms. The main steps of these systems include:

[0003] Image acquisition: Use multiple cameras or sensors to capture environmental images around the vehicle from different angles.

[0004] Image stitching: Stitch the multi-view images collected to form a panoramic image.

[0005] Feature point extraction and matching: Extract feature points in the panoramic image and use the feature point matching between adjacent frames to estimate the vehicle's motion state.

[0006] Path planning: Based on the feature point matching results, calculate the driving direction and speed of the vehicle and generate the corresponding path planning.

[0007] Although the existing technology can meet the basic requirements to a certain extent, under low light conditions, insufficient image brightness will lead to inaccurate feature point extraction, which in turn affects the vehicle's positioning accuracy and the reliability of path planning. In addition, existing systems often lack a dynamic update and correction mechanism for three-dimensional path maps and are difficult to adapt to complex dynamic environmental changes. Summary of the Invention

[0008] The purpose of the present invention is to provide a method for synchronous positioning of panoramic images of driving environments and real-time generation of three-dimensional paths, which not only improves the vehicle's positioning accuracy under low light conditions, but also realizes the accurate and real-time generation of three-dimensional paths, effectively improving the reliability and safety of the intelligent driving system, so as to solve the problems raised in the above background art.

[0009] To achieve the above purpose, the present invention adopts the following technical solution: A method for synchronous positioning of panoramic images of driving environments and real-time generation of three-dimensional paths, including the following steps:

[0010] Collect multi-view low-light images of the vehicle's surrounding environment; based on the multi-view low-light images, adjust the brightness of each view image to enhance the contrast, splice the adjusted images to form a panoramic image; mark feature points in the panoramic image and record their position information, and calculate the driving direction and speed of the vehicle using the position information of the feature points; according to the driving direction and speed, predict the vehicle position at the next moment, and update the feature point distribution in the panoramic image in combination with the predicted position; based on the updated feature point distribution, construct a three-dimensional path map, and continuously monitor and correct the three-dimensional path map.

[0011] Preferably, the collecting multi-view low-light images of the vehicle's surrounding environment includes:

[0012] Start multiple cameras set around the vehicle, and each camera is configured with a night vision enhancement mode;

[0013] Based on the night vision enhancement mode, adjust the exposure time of each camera , where ISO is the sensitivity and EV is the environmental brightness value;

[0014] Use the adjusted exposure time to capture images and record the timestamp of each frame of image , where t_start is the start time, n is the number of frames, and TI is the time interval.

[0015] Preferably, for the multi-view low-light images, adjusting the brightness of each view image to enhance the contrast includes:

[0016] Receive images with timestamps and calculate the average brightness value of each frame of image , where B(x,y) is the brightness value of the pixel point, and W and H are the image width and height respectively;

[0017] According to the average brightness value L_avg, determine the gain coefficient G = G_max - (L_avg / R) for each image, where G_max is the maximum gain value and R is the reference brightness value;

[0018] Use the gain coefficient G to adjust the brightness of each frame of image, and the updated brightness value ;

[0019] Remap the adjusted image brightness value to the range of 0 to 255, and the mapping formula is , where MIN_B_new and MAX_B_new are the minimum and maximum values in the adjusted image brightness value respectively, and B_final is the final brightness value.

[0020] Preferably, the splicing the adjusted images to form a panoramic image includes:

[0021] Receive the adjusted image corresponding to the mapped luminance value B_final(x,y), and calculate the set of feature points F={p1,p2,...,pn} in the overlapping area for each frame of image;

[0022] Based on the set of feature points F, use the matching point pairs between adjacent images to determine the transformation matrix M=[abc;def], where a to f are transformation parameters, and solve by minimizing the error where I1 and I2 are two adjacent images respectively;

[0023] Apply the transformation matrix M to each image and transform it to the global coordinate system to obtain a new position , ensuring that all images are aligned in the same coordinate system, and P_transformed is the aligned image;

[0024] Fuse the aligned image P_transformed(x,y), and calculate the final pixel value by the weighted average method: where w_i is the weight of the i-th image, and P(x,y) is the panoramic image.

[0025] Preferably, marking feature points in the panoramic image and recording their position information includes:

[0026] Receive the fused panoramic image P(x,y), and calculate the gradient magnitude GRAD=sqrt(dx^2+dy^2) of each frame of image, where dx and dy are the gradients in the x and y directions respectively;

[0027] Based on the gradient magnitude GRAD, determine the local maximum points as the set of feature points F={f1,f2,...,fm}, where the position of each feature point fi is represented by the coordinates (X_i,Y_i);

[0028] For each feature point fi in the set of feature points F, calculate its descriptor D_i=SUM((P(x+x_i,y+y_i)-AVG_P)^2), where AVG_P is the average value of the pixel values in the surrounding area, and (X_i,Y_i) is the center point coordinates;

[0029] Store the feature points and their descriptors D_i together with the position information (X_i,Y_i) into the database to form a feature point list:

[0030] L={(f1,D_1,X_1,Y_1),(f2,D_2,X_2,Y_2),...,(fm,D_m,X_m,Y_m)}.

[0031] Preferably, calculating the driving direction and speed of the vehicle using the position information of the feature points includes:

[0032] Receive the feature points in the feature point list L and their position information (X_i, Y_i), and match feature point pairs P between consecutive frames:

[0033] P = {(f1_t1, f1_t2), (f2_t1, f2_t2),..., (fn_t1, fn_t2)}, where t1 and t2 represent different time points;

[0034] Based on the feature point pairs P, calculate the displacement vector V = (X_2 - X_1, Y_2 - Y_1) of each feature point between two frames, where (X_1, Y_1) and (X_2, Y_2) are the position coordinates of the same feature point at two time points respectively;

[0035] Use the displacement vector V to calculate the vehicle driving direction A = atan2(V_y, V_x), where V_x and V_y are the components of the displacement vector on the x-axis and y-axis respectively, and atan2 is the arctangent function;

[0036] Combine the displacement vector V and the time interval TI, and calculate the vehicle speed S = sqrt(V_x^2 + V_y^2) / TI, where TI is the time difference between the shooting of two frames of images.

[0037] Preferably, predicting the vehicle position at the next moment according to the driving direction and speed includes:

[0038] Receive the vehicle driving direction A and the vehicle driving speed S, and determine the current vehicle position coordinates C = (X_cur, Y_cur), where (X_cur, Y_cur) is the position of the vehicle in the current frame;

[0039] Based on the vehicle driving direction A, calculate the direction vector DIR = (cos(A), sin(A)), where cos and sin are cosine and sine operations respectively;

[0040] Use the vehicle driving speed S and the time interval TI to calculate the displacement distance and combine it with the direction vector DIR to obtain the displacement vector ;

[0041] Add the displacement vector DELTA to the current vehicle position coordinates C to obtain the predicted vehicle position N at the next moment = (X_cur + DELTA_X, Y_cur + DELTA_Y), where DELTA_X and DELTA_Y are the components of the displacement vector on the x-axis and y-axis respectively.

[0042] Preferably, updating the distribution of feature points in the panoramic image in combination with the predicted position includes:

[0043] Receive the predicted vehicle position N=(X_next, Y_next) at the next moment, and obtain the list L of feature points in the current panoramic image:

[0044] L={(f1, D_1, X_1, Y_1), (f2, D_2, X_2, Y_2),..., (fm, D_m, X_m, Y_m)};

[0045] Based on the predicted vehicle position N at the next moment, calculate the position change POS_CHANGE=(X_next - X_i, Y_next - Y_i) of the vehicle relative to each feature point, where (X_i, Y_i) is the position coordinate of the feature point fi in the current frame;

[0046] Use the position change POS_CHANGE to update the new position coordinates (X'_i, Y'_i) of each feature point. The formula for the new position coordinates is X'_i = X_i + POS_CHANGE_X, Y'_i = Y_i + POS_CHANGE_Y, where POS_CHANGE_X and POS_CHANGE_Y are the components of POS_CHANGE on the x-axis and y-axis respectively;

[0047] Remap the updated feature point position coordinates (X'_i, Y'_i) into the panoramic image to form the updated feature point list L_updated:

[0048] L_updated={(f1, D_1, X'_1, Y'_1), (f2, D_2, X'_2, Y'_2),..., (fm, D_m, X'_m, Y'_m)}.

[0049] Preferably, constructing a three-dimensional path map based on the updated feature point distribution includes:

[0050] Receive the updated feature point list L_updated, and calculate the height value Z_i of each feature point on the z-axis;

[0051] Based on the position coordinates (X'_i, Y'_i, Z_i) of the feature points, establish a three-dimensional point cloud PC in the local coordinate system:

[0052] PC={(X'_1, Y'_1, Z_1), (X'_2, Y'_2, Z_2),..., (X'_m, Y'_m, Z_m)};

[0053] Using the points in the three-dimensional point cloud PC, calculate the distance ED_ij between adjacent feature points: ED_ij = sqrt((X'_j - X'_i)^2 + (Y'_j - Y'_i)^2 + (Z_j - Z_i)^2), where (i, j) is a pair of adjacent feature points;

[0054] Based on the distance ED_ij, construct edges connecting adjacent feature points to form a three-dimensional path graph G_path = {(f1, f2, ED_12), (f2, f3, ED_23),..., (fn-1, fn, ED_n-1n)}, where ED_ij represents the distance between feature points fi and fj.

[0055] Preferably, the continuous monitoring and correction of the three-dimensional path graph includes:

[0056] Based on the three-dimensional path graph G_path, set an initial threshold TH = TH_0 for detecting path changes. For each edge e_ij in the path graph belonging to the G_path set, calculate its edge length ED(e_ij) = dist_ij;

[0057] Update the position coordinates (X'_i, Y'_i, Z_i) of the feature points through sensor data, and recalculate the new length NEW_ED(e_ij) of each edge:

[0058] NEW_ED(e_ij) = sqrt((X'_j - X'_i)^2 + (Y'_j - Y'_i)^2 + (Z_j - Z_i)^2);

[0059] Compare the new length NEW_ED(e_ij) with the original length ED(e_ij). If |NEW_ED(e_ij) - ED(e_ij)| > TH, mark it as abnormal and record the abnormal information in the abnormal information list AN:

[0060] AN = {(e_1, |NEW_ED(e_1) - ED(e_1)|),..., (e_n, |NEW_ED(e_n) - ED(e_n)|)};

[0061] According to the abnormal information in the abnormal information list AN, correct the marked edges and update the edge lengths using the weighted average method. The formula is:

[0062] , where W_old and W_new are the old weight and the new weight respectively, and satisfy W_old + W_new = 1, CORRECTED_ED is the corrected edge length, and W is the weight.

[0063] Technical effects and advantages of the present invention: The method for synchronous positioning and real-time generation of a three-dimensional path for a panoramic image of a driving environment proposed by the present invention has the following advantages compared with the prior art:

[0064] By enhancing the contrast of the images collected under low-light conditions, the present invention improves the accuracy of feature point extraction and significantly enhances the vehicle positioning accuracy by using an improved feature point matching algorithm. By combining the predicted position to update the feature point distribution in the panoramic image, the real-time generation and dynamic correction of the three-dimensional path map are realized, thereby effectively enhancing the adaptability and reliability of the system in complex and changing environments. This not only solves the problem of performance degradation of traditional methods in low-light environments, but also greatly improves the overall performance of the intelligent driving system, ensuring driving safety and the accuracy of path planning. Description of the drawings

[0065] Figure 1 It is a flowchart of the method for synchronous positioning and real-time generation of a three-dimensional path for a panoramic image of a driving environment of the present invention. Detailed implementation manners

[0066] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0067] The present invention provides a method for synchronous positioning and real-time generation of a three-dimensional path for a panoramic image of a driving environment as Figure 1 shown, including the following steps:

[0068] Step 1: Collect multi-view low-light images of the vehicle surrounding environment; specifically including:

[0069] Start multiple cameras arranged around the vehicle. Each camera is configured with a night vision enhancement mode to ensure that high-quality environmental images can be captured under low-light conditions. The night vision enhancement mode improves the sensitivity of the camera in low-light environments, making the images clearer and more detailed.

[0070] Based on the night vision enhancement mode, adjust the exposure time of each camera , where ISO is the sensitivity and EV is the ambient brightness value; the principle of the formula is to calculate the appropriate exposure time based on the ambient brightness and sensitivity. A lower EV value indicates a darker environment and requires a longer exposure time to capture sufficient light; a higher ISO value indicates a higher sensitivity and can capture sufficient light in a shorter time. By dynamically adjusting the exposure time, the image quality can be optimized under different lighting conditions. The reasonable adjustment of the exposure time can avoid overexposure or underexposure of the image, thereby improving the accuracy of feature point extraction and subsequent processing.

[0071] Capture images using the adjusted exposure time and record the timestamp of each frame of the image. , where t_start is the starting time, n is the number of frames, and TI is the time interval. The principle of this formula is to calculate the specific shooting time of each frame of the image based on the starting time and the number of frames. For example, if the starting time is 0 seconds and the time interval is 0.033 seconds (i.e., 30 frames per second), then the timestamp of the 10th frame of the image is seconds.

[0072] By recording the timestamp of each frame of the image, the shooting time of each frame of the image can be accurately tracked, which is crucial for subsequent image matching, feature point tracking, and motion estimation. The accurate recording of the timestamp helps to improve the accuracy of positioning and path planning.

[0073] Step 2: Based on the multi-view low-light images, adjust the brightness of each view image to enhance the contrast; specifically including:

[0074] Receive the images with timestamps and calculate the average brightness value of each frame of the image (the average brightness of the entire image). , where B(x,y) is the brightness value of the pixel point, and W and H are the width and height of the image respectively;

[0075] This formula The principle is to sum the brightness values of all pixel points and divide by the total number of pixels to obtain the average brightness value of the entire image. This value is used to evaluate the overall brightness level of the image and provide a basis for subsequent gain adjustment. By calculating the average brightness value of each frame of the image, the overall brightness level of the image can be quantified. This step provides a benchmark reference for subsequent brightness adjustment and ensures the rationality of the brightness adjustment.

[0076] According to the average luminance value L_avg, determine the gain coefficient G = G_max - (L_avg / R) for each image, where G_max is the maximum gain value and R is the reference luminance value; the principle of this formula is to calculate the gain coefficient based on the difference between the average luminance of the image and the reference luminance. When the image is darker, L_avg is smaller and the G value is larger, thus increasing the image luminance; conversely, when the image is brighter, L_avg is larger and the G value is smaller, thus reducing the image luminance.

[0077] By calculating the gain coefficient, the gain can be dynamically adjusted according to the average luminance of the image, thereby optimizing the image luminance under different lighting conditions. The introduction of the gain coefficient makes the image luminance more balanced and improves the accuracy of feature point extraction.

[0078] Use the gain coefficient G to adjust the luminance of each frame of the image, and the updated luminance value ; the principle of this formula is to adjust the luminance value of each pixel point by multiplying the gain coefficient. When the gain coefficient is greater than 1, the luminance increases, and when it is less than 1, the luminance decreases, thereby achieving the adjustment of the overall luminance. By applying the gain coefficient to adjust the luminance of each frame of the image, the luminance and contrast of the image can be effectively improved. Especially under low-light conditions, the image quality can be significantly improved, facilitating subsequent processing.

[0079] Remap the adjusted image luminance value to the range of 0 to 255, and the mapping formula is , where MIN_B_new and MAX_B_new are respectively the minimum and maximum values in the adjusted image luminance value, and B_final is the final luminance value.

[0080] The principle of this formula is to normalize the adjusted luminance value and remap it to the standard range of 0 to 255. The specific steps are as follows:

[0081] Calculate the minimum and maximum values of the adjusted luminance value.

[0082] Use linear transformation to map the adjusted luminance value to between 0 and 255 to ensure that the luminance distribution of the image is uniform and meets the display standard.

[0083] By remapping the adjusted luminance value, ensure that the image luminance value falls within the standard range of 0 to 255, enhancing the contrast and visual effect of the image, and facilitating subsequent feature point extraction and matching.

[0084] Step three: Stitch the adjusted images to form a panoramic image; specifically including:

[0085] Receive the adjusted image corresponding to the mapped luminance value B_final(x,y), and calculate the set of feature points F={p1,p2,...,pn} in the overlapping region for each frame of image; by calculating the set of feature points in each frame of image, the overlapping region between adjacent images can be identified. These feature points provide key information for subsequent image alignment and stitching, ensuring the accuracy and stability of the stitching process.

[0086] Based on the set of feature points F, use the matching point pairs between adjacent images to determine the transformation matrix M=[abc;def], where a, b, c, d, e, f are transformation parameters, representing the specific values of the transformation matrix, and are solved by minimizing the error where I1 and I2 are two adjacent images respectively;

[0087] The principle of this formula is to solve the transformation matrix M by minimizing the difference of feature points between the two images. The specific steps are as follows:

[0088] Calculate the corresponding positions of each feature point p_i in the set of feature points F in the two images.

[0089] Use the transformation matrix M to map the feature point p_i in one image to the other image.

[0090] Optimize the transformation matrix M by minimizing the luminance difference of the corresponding feature points in the two images.

[0091] By calculating the transformation matrix, adjacent images can be aligned to the same coordinate system, ensuring that the stitched panoramic image is seamless and distortion-free. The accuracy of the transformation matrix directly affects the stitching quality.

[0092] Apply the transformation matrix M to each image to transform it to the global coordinate system to obtain a new position , ensuring that all images are aligned in the same coordinate system, and P_transformed is the aligned image; the principle of this formula is to apply the transformation matrix M to each pixel point in the image to transform it from the original coordinate system to the global coordinate system. The specific steps are as follows:

[0093] For each pixel point (x,y), calculate its new position in the global coordinate system.

[0094] Assign the transformed pixel value to the new position to ensure that all images are aligned in the same coordinate system.

[0095] By applying the transformation matrix, all images are transformed to the global coordinate system, ensuring the alignment and smooth transition between images. This step is the key to achieving seamless stitching.

[0096] Fuse the aligned image \(P_{transformed}(x, y)\) and calculate the final pixel value by the weighted average method:

[0097] , where \(w_i\) is the weight of the \(i\)-th image and \(P(x, y)\) is the panoramic image. The principle of this formula is to generate the final panoramic image through weighted averaging of the aligned images. The specific steps are as follows:

[0098] For each pixel point \((x, y)\), calculate its weighted average value among all aligned images.

[0099] The weight \(w_i\) is assigned according to the importance of the image or the size of the overlapping area to ensure a smooth transition in the overlapping area.

[0100] Assign the weighted average value to the corresponding position in the final panoramic image.

[0101] Fuse the aligned images by the weighted average method to generate a high-quality panoramic image. This method can smoothly process the overlapping area between images, reduce stitching artifacts, and improve the overall visual effect.

[0102] Step Four: Mark feature points in the panoramic image and record their position information; specifically including:

[0103] Receive the fused panoramic image \(P(x, y)\) and calculate the gradient magnitude of each frame of the image (used to measure the edge strength of each pixel point in the image) \(GRAD = \sqrt{dx^2 + dy^2}\), where \(dx\) and \(dy\) respectively represent the gradient values of the image in the \(x\) and \(y\) directions, reflecting the change rate of the image brightness along these two directions; the principle of this formula is to determine the edges in the image based on the magnitude of the gradient. The specific steps are as follows:

[0104] For each pixel point, calculate its gradient in the \(x\) direction (\(dx\)) and \(y\) direction (\(dy\)).

[0105] Use the above formula to calculate the gradient magnitude to quantify the edge strength at this point.

[0106] By calculating the gradients of the image in the \(x\) and \(y\) directions, the edges and details in the image can be identified.

[0107] Based on the gradient magnitude \(GRAD\), determine the local maximum points as the feature point set \(F = \{f1, f2,..., fm\}\), where the position of each feature point \(f_i\) is represented by the coordinates \((X_i, Y_i)\); by finding the local maximum points of the gradient magnitude, the significant feature points in the image can be effectively identified. These feature points provide key information for subsequent descriptor calculation and matching.

[0108] For each feature point fi in the feature point set F, calculate its descriptor Di = SUM((P(x + xi, y + yi) - AVG_P)^2), where AVG_P is the average of the pixel values in the surrounding area, and (Xi, Yi) is the center point coordinates; the principle of this formula is to calculate the sum of the squared differences relative to the average by statistically analyzing the pixel values in the surrounding area of the feature point, so as to characterize the local structure of this point. The specific steps are as follows:

[0109] For each feature point, define a window of a fixed size that includes the point and its surrounding pixels.

[0110] Calculate the average AVG_P of all pixel values within the window.

[0111] For each pixel within the window, calculate the squared difference between it and the average, and sum them up to obtain the descriptor Di.

[0112] By calculating the descriptors of each feature point, the local structure information around this point can be effectively represented. The similarity of the descriptors can be used for matching between feature points, thus realizing image stitching or recognition tasks.

[0113] Store the feature points and their descriptors Di together with the position information (Xi, Yi) into the database to form a feature point list:

[0114] L = {(f1, D_1, X_1, Y_1), (f2, D_2, X_2, Y_2),..., (fm, D_m, X_m, Y_m)}, where f_i, D_i, X_i, Y_i are the i-th feature point, its descriptor, and position coordinates respectively. By storing the feature points, descriptors, and their position information, it is convenient for subsequent retrieval and matching operations.

[0115] Step Five: Calculate the driving direction and speed of the vehicle using the position information of the feature points; specifically including:

[0116] Receive the feature points and their position information (Xi, Yi) in the feature point list L, and match the feature point pairs P between consecutive frames:

[0117] P = {(f1_t1, f1_t2), (f2_t1, f2_t2),..., (fn_t1, fn_t2)}, where t1 and t2 represent different time points; by matching the feature point pairs between consecutive frames, the movement trajectories of the feature points can be tracked. These matching results provide the basic data for subsequent calculation of displacement vectors, driving directions, and speeds.

[0118] Based on the feature point pair P, calculate the displacement vector V=(X_2 - X_1, Y_2 - Y_1) of each feature point between two frames, where (X_1, Y_1) and (X_2, Y_2) are the position coordinates of the same feature point at two time points respectively; the principle of this formula is to determine the displacement vector by calculating the coordinate difference of the feature point at the same position between two frames. The specific steps are as follows:

[0119] For each matched feature point pair, calculate its displacement components on the x-axis and y-axis.

[0120] Combine these components into a two-dimensional vector V.

[0121] By calculating the displacement vector of the feature point between two frames, the movement of the feature point can be quantified. These displacement vectors provide key data for subsequent calculation of the driving direction and speed of the vehicle.

[0122] Use the displacement vector V to calculate the driving direction A of the vehicle = atan2(V_y, V_x), where V_x and V_y are the components of the displacement vector on the x-axis and y-axis respectively, and atan2 is the arctangent function used to calculate the angle; the principle of this formula is to calculate the direction angle of the displacement vector through the arctangent function. The specific steps are as follows:

[0123] Use the atan2 function to calculate the angle between the x-axis and y-axis components of the displacement vector.

[0124] This angle represents the direction of movement of the feature point, and thus reflects the driving direction of the vehicle.

[0125] Combine the displacement vector V and the time interval (the time difference between two consecutive image captures) TI, and calculate the vehicle speed S = sqrt(V_x^2 + V_y^2) / TI, where TI is the time difference between two consecutive image captures. The principle of this formula is to determine the speed by calculating the magnitude of the displacement vector and dividing it by the time interval. The specific steps are as follows:

[0126] Calculate the magnitude of the displacement vector, that is, sqrt(V_x^2 + V_y^2).

[0127] Divide the magnitude by the time interval TI to obtain the vehicle speed S.

[0128] Through the above steps, the present invention realizes the function of calculating the driving direction and speed of the vehicle using the position information of the feature points.

[0129] Step Six: Predict the vehicle position at the next moment according to the driving direction and speed; specifically including:

[0130] Receive the vehicle driving direction A and the vehicle driving speed S, and determine the current vehicle position coordinates C=(X_cur, Y_cur), where (X_cur, Y_cur) is the position of the vehicle in the current frame; by receiving the driving direction, speed and current position coordinates of the vehicle, it provides the basic data for subsequent calculations.

[0131] Based on the vehicle driving direction A, calculate the direction vector DIR=(cos(A), sin(A)), where cos and sin are the cosine and sine operations respectively; the principle of this formula is to use trigonometric functions to convert the angle into a two-dimensional plane vector. The specific steps are as follows:

[0132] Calculate the cosine value cos(A) of the driving direction angle A, which represents the component of the direction vector on the x-axis.

[0133] Calculate the sine value sin(A) of the driving direction angle A, which represents the component of the direction vector on the y-axis.

[0134] By calculating the direction vector, the driving direction can be converted into a unit vector on the two-dimensional plane.

[0135] Use the vehicle driving speed S and the time interval TI to calculate the displacement distance , combined with the direction vector DIR, to obtain the displacement vector ; the principle of this formula is to calculate the displacement distance by multiplying the vehicle speed S by the time interval TI. Combining with the direction vector DIR, the displacement vector DELTA can be further calculated. The specific steps are as follows:

[0136] Calculate the displacement distance .

[0137] Multiply the displacement distance by the direction vector to obtain the displacement vector .

[0138] By calculating the displacement distance and combining with the direction vector, the displacement vector of the vehicle within a period of time can be determined.

[0139] Add the displacement vector DELTA to the current vehicle position coordinates C to obtain the predicted vehicle position N=(X_cur + DELTA_X, Y_cur + DELTA_Y) at the next moment, where DELTA_X and DELTA_Y are the components of the displacement vector on the x-axis and y-axis respectively. The principle of this formula is to add the components of the displacement vector to the corresponding components of the current position coordinates to obtain the position coordinates at the next moment. The specific steps are as follows:

[0140] Calculate the component of the displacement vector on the x-axis .

[0141] Calculate the component of the displacement vector on the y-axis .

[0142] Add these components to the current position coordinates to obtain the position coordinates at the next moment N = (X_cur + DELTA_X, Y_cur + DELTA_Y).

[0143] Step 7: Update the distribution of feature points in the panoramic image in combination with the predicted position; specifically including:

[0144] Receive the predicted vehicle position N = (X_next, Y_next) at the next moment, and obtain the list L of feature points in the current panoramic image:

[0145] L = {(f1, D_1, X_1, Y_1), (f2, D_2, X_2, Y_2),..., (fm, D_m, X_m, Y_m)}; By receiving the predicted vehicle position at the next moment and the list of feature points in the current panoramic image, it provides the basic data for subsequent calculations.

[0146] Based on the predicted vehicle position N at the next moment, calculate the position change POS_CHANGE = (X_next - X_i, Y_next - Y_i) of the vehicle relative to each feature point, where (X_i, Y_i) is the position coordinate of the feature point fi in the current frame; The principle of this formula is to determine the position change by calculating the difference between the new position of the vehicle and the current position of the feature point. The specific steps are as follows:

[0147] For each feature point fi, calculate its position change relative to the new position of the vehicle.

[0148] The component of the position change on the x-axis is POS_CHANGE_X = X_next - X_i.

[0149] The component of the position change on the y-axis is POS_CHANGE_Y = Y_next - Y_i.

[0150] By calculating the position change of the vehicle relative to each feature point, the new position of the feature point in the global coordinate system can be determined.

[0151] Use the position change POS_CHANGE to update the new position coordinates (X'_i, Y'_i) of each feature point. The formula for the new position coordinates is X'_i = X_i + POS_CHANGE_X, Y'_i = Y_i + POS_CHANGE_Y, where POS_CHANGE_X and POS_CHANGE_Y are the components of POS_CHANGE on the x-axis and y-axis respectively; The principle of this formula is to update the position of the feature point by adding the components of the position change to the current position coordinates of the feature point. The specific steps are as follows:

[0152] Calculate the new position coordinate X'_i of the updated feature point on the x-axis: X'_i = X_i + POS_CHANGE_X.

[0153] Calculate the new position coordinate Y'_i of the updated feature point on the y-axis: Y'_i = Y_i + POS_CHANGE_Y.

[0154] By adding the position change to the current position coordinates of the feature point, the new position of the feature point in the global coordinate system can be updated.

[0155] Remap the updated feature point position coordinates (X'_i, Y'_i) to the panoramic image to form an updated feature point list L_updated:

[0156] L_updated = {(f1, D_1, X'_1, Y'_1), (f2, D_2, X'_2, Y'_2),..., (fm, D_m, X'_m, Y'_m)}. By remapping the updated feature point position coordinates to the panoramic image, the correct distribution of the feature points in the new panoramic image can be ensured.

[0157] Step Eight: Construct a three-dimensional path map based on the updated feature point distribution; specifically including:

[0158] Receive the updated feature point list L_updated and calculate the height value Z_i of each feature point on the z-axis; by calculating the height value of each feature point, the two-dimensional feature points can be extended to three-dimensional space.

[0159] Based on the position coordinates (X'_i, Y'_i, Z_i) of the feature points, establish a three-dimensional point cloud PC in the local coordinate system:

[0160] PC = {(X'_1, Y'_1, Z_1), (X'_2, Y'_2, Z_2),..., (X'_m, Y'_m, Z_m)}; Use a stereo vision algorithm (such as a binocular camera) or a depth sensor (such as LiDAR) to obtain the height value Z_i of each feature point. Combine the height value Z_i with the position coordinates (X'_i, Y'_i) of the feature point to form three-dimensional coordinates (X'_i, Y'_i, Z_i).

[0161] Use the points in the three-dimensional point cloud PC to calculate the distance ED_ij between adjacent feature points: ED_ij = sqrt((X'_j - X'_i)^2 + (Y'_j - Y'_i)^2 + (Z_j - Z_i)^2), where (i, j) is a pair of adjacent feature points; The principle of this formula is based on the Euclidean distance formula in three-dimensional space and is used to calculate the straight-line distance between two points. The specific steps are as follows:

[0162] For each pair of adjacent feature points (i, j), calculate the coordinate differences between them on the x-axis, y-axis, and z-axis.

[0163] Calculate the three-dimensional distance ED_ij between the two points using the above formula.

[0164] Based on the distance ED_ij, construct edges connecting adjacent feature points to form a three-dimensional path graph G_path = {(f1, f2, ED_12), (f2, f3, ED_23),..., (fn-1, fn, ED_n-1n)}, where ED_ij represents the distance between feature points fi and fj. For each pair of adjacent feature points, record the distance ED_ij between them. Combine these feature points and their distances into a tuple (f_i, f_j, ED_ij), representing the connection between feature points fi and fj and their distance. Collect all the tuples to form the three-dimensional path graph G_path. By constructing the three-dimensional path graph, the relationship between feature points and their distances can be visualized.

[0165] Step Nine: Continuously monitor and correct the three-dimensional path graph; specifically including:

[0166] Based on the three-dimensional path graph G_path, set an initial threshold TH = TH_0 for detecting path changes. For each edge e_ij in the path graph belonging to the G_path set, calculate its edge length ED(e_ij) = dist_ij; by setting the initial threshold and calculating the initial length of each edge, a benchmark can be provided for subsequent path change detection.

[0167] Update the position coordinates (X'_i, Y'_i, Z_i) of the feature points through sensor data, and recalculate the new length NEW_ED(e_ij) of each edge:

[0168] NEW_ED(e_ij) = sqrt((X'_j - X'_i)^2 + (Y'_j - Y'_i)^2 + (Z_j - Z_i)^2); by using sensor data to update the position coordinates of the feature points and recalculating the new length of each edge, the actual position changes of the feature points can be dynamically reflected. The principle of this formula is based on the Euclidean distance formula in three-dimensional space, which is used to calculate the straight-line distance between two points. The specific steps are as follows:

[0169] Update the three-dimensional coordinates (X'_i, Y'_i, Z_i) of each feature point using sensor data.

[0170] For each edge e_ij, recalculate its new length NEW_ED(e_ij) using the above formula.

[0171] Compare the new length NEW_ED(e_ij) with the original length ED(e_ij). If |NEW_ED(e_ij) - ED(e_ij)| > TH, mark it as an anomaly and record the anomaly information in the anomaly information list AN:

[0172] AN = {(e_1, |NEW_ED(e_1) - ED(e_1)|),..., (e_n, |NEW_ED(e_n) - ED(e_n)|)}; By comparing the old and new side lengths and detecting abnormal changes, abnormal situations in the path graph can be discovered in a timely manner.

[0173] The process of data analysis and anomaly detection is as follows:

[0174] For each edge e_ij, calculate the side length difference |NEW_ED(e_ij) - ED(e_ij)|.

[0175] If the difference is greater than the set threshold TH, mark the edge as an anomaly and record its difference in the anomaly information list AN.

[0176] According to the anomaly information in the anomaly information list AN, correct the marked edges, and update the side lengths using the weighted average method. The formula is: , where W_old and W_new are the old weight and the new weight respectively, and satisfy W_old + W_new = 1. CORRECTED_ED is the corrected side length, W is the weight, W_old: old weight (OldWeight), representing the weight of the original side length. W_new: new weight (NewWeight), representing the weight of the new side length. W_old + W_new = 1: The sum of the weights is 1 to ensure the effectiveness of the weighted average.

[0177] The principle of this formula is to calculate the corrected side length by combining the old side length and the new side length using the weighted average method. The specific steps are as follows:

[0178] According to the anomaly information in the anomaly information list AN, determine the edges that need to be corrected.

[0179] For each edge e_ij marked as an anomaly, use the above formula to calculate the corrected side length CORRECTED_ED(e_ij).

[0180] Ensure that the weights W_old and W_new satisfy W_old + W_new = 1 to ensure the effectiveness of the weighted average.

[0181] In summary, the present invention enhances the contrast of the captured images under low light conditions, improves the accuracy of feature point extraction, and significantly improves the vehicle positioning accuracy by using an improved feature point matching algorithm. By combining the predicted position to update the feature point distribution in the panoramic image, the real-time generation and dynamic correction of the three-dimensional path map are achieved, thereby effectively enhancing the adaptability and reliability of the system in complex and changing environments. This not only solves the problem of performance degradation of traditional methods in low light environments, but also greatly improves the overall performance of the intelligent driving system, ensuring driving safety and the accuracy of path planning.

[0182] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for synchronous positioning and real-time generation of three-dimensional paths of panoramic images of driving environments, characterized in that, It includes the following steps: Collect multi-view low-light images of the vehicle's surrounding environment; Based on the multi-view low-light images, adjust the brightness of each view image to enhance the contrast, and splice the adjusted images to form a panoramic image; Mark feature points in the panoramic image and record their position information, and calculate the driving direction and speed of the vehicle using the position information of the feature points; Predict the vehicle position N at the next moment according to the driving direction and speed; Update the feature point distribution in the panoramic image in combination with the predicted position, specifically including: receiving the predicted vehicle position N=(X_next,Y_next) at the next moment, and obtaining the list L of feature points in the current panoramic image: L={(f1,D_1,X_1,Y_1),(f2,D_2,X_2,Y_2),...,(fm,D_m,X_m,Y_m)}; where fm is the feature point, D_m is the descriptor, and X_m, Y_m are the position coordinates of the feature point; Based on the predicted vehicle position N at the next moment, calculate the position change POS_CHANGE=(X_next-X_i,Y_next-Y_i) of the vehicle relative to each feature point, where (X_i,Y_i) is the position coordinate of the feature point fi in the current frame; Use the position change POS_CHANGE to update the new position coordinates (X'_i,Y'_i) of each feature point. The formula for the new position coordinates is X'_i=X_i+POS_CHANGE_X, Y'_i=Y_i+POS_CHANGE_Y, where POS_CHANGE_X and POS_CHANGE_Y are the components of POS_CHANGE on the x-axis and y-axis respectively; Remap the updated feature point position coordinates (X'_i,Y'_i) to the panoramic image to form an updated feature point list L_updated: L_updated={(f1,D_1,X'_1,Y'_1),(f2,D_2,X'_2,Y'_2),...,(fm,D_m,X'_m,Y'_m)}; Construct a three-dimensional path map based on the updated feature point distribution, specifically including: Receive the updated feature point list L_updated, and calculate the height value Z_i of each feature point on the z-axis; Based on the position coordinates (X'_i,Y'_i,Z_i) of the feature points, establish a three-dimensional point cloud PC in the local coordinate system: PC={(X'_1,Y'_1,Z_1),(X'_2,Y'_2,Z_2),...,(X'_m,Y'_m,Z_m)}; Use the points in the three-dimensional point cloud PC to calculate the distance ED_ij=sqrt((X'_j-X'_i)^2+(Y'_j-Y'_i)^2+(Z_j-Z_i)^2) between adjacent feature points, where (i,j) is a pair of adjacent feature points; Based on the distance ED_ij, construct edges connecting adjacent feature points to form a three-dimensional path graph G_path = {(f1, f2, ED_12), (f2, f3, ED_23),..., (fm-1, fm, ED_m-1m)}, where ED_m-1m represents the distance between feature points fm and fm-1; And continuously monitor and correct the three-dimensional path graph.

2. The method for synchronous positioning and real-time generation of a three-dimensional path of a panoramic image of a driving environment according to claim 1, wherein: The multi-view low-light images of the vehicle's surrounding environment are collected, including: Activate multiple cameras arranged around the vehicle, and each camera is configured with a night vision enhancement mode; Based on the night vision enhancement mode, adjust the exposure time of each camera E = 1 / (ISO * EV), where ISO is the sensitivity and EV is the ambient brightness value; Use the adjusted exposure time to capture images, and record the timestamp of each frame of the image T = t_start + n * TI, where t_start is the starting time, n is the number of frames, and TI is the time interval.

3. The method for synchronous positioning and real-time generation of a three-dimensional path of a panoramic image of a driving environment according to claim 2, wherein: Based on the multi-view low-light images, adjust the brightness of each view image to enhance the contrast, including: Receive the timestamped images, and calculate the average brightness value of each frame of the image L_avg = SUM(B(x, y)) / (W * H), where B(x, y) is the pixel brightness value, and W and H are the image width and height respectively; According to the average brightness value L_avg, determine the gain coefficient G of each image G = G_max - (L_avg / R), where G_max is the maximum gain value and R is the reference brightness value; Use the gain coefficient G to adjust the brightness of each frame of the image, and the updated brightness value B_new(x, y) = B(x, y) * G; Remap the adjusted image brightness value to the range of 0 to 255, and the mapping formula is B_final(x, y) = 255 * (B_new(x, y) - MIN_B_new) / (MAX_B_new - MIN_B_new), where MIN_B_new and MAX_B_new are the minimum and maximum values of the adjusted image brightness value respectively, and B_final is the final brightness value.

4. The method for synchronous positioning and real-time generation of a three-dimensional path of a panoramic image of a driving environment according to claim 3, wherein: The adjusted images are stitched to form a panoramic image, including: Receive the adjusted images corresponding to the mapped brightness value B_final(x, y), and calculate the feature point set F = {p1, p2,..., pα,..., pβ} of the overlapping area for each frame of the image; Based on the feature point set F, use the matching point pairs between adjacent images to determine the transformation matrix M = [abc; def], where a to f are transformation parameters, and are solved by minimizing the error ERR = SUM((I1(pα) - I2(M * pα))^2), where pα is the feature point in the feature point set, and I1 and I2 are two adjacent images respectively; Apply the transformation matrix M to each image, convert it to the global coordinate system, and obtain the new position P_transformed(x, y) = M * B_final(x, y), ensuring that all images are aligned in the same coordinate system, and P_transformed is the aligned image; Fuse the aligned image P_transformed(x, y), and calculate the final pixel value by the weighted average method: P(x, y) = SUM(w_s * P_transformed_s(x, y)) / SUM(w_s), where w_s is the weight of the s-th image, and P(x, y) is the panoramic image.

5. The method for synchronous positioning and real-time generation of a three-dimensional path of a panoramic image of a driving environment according to claim 4, wherein: Mark feature points in the panoramic image and record their position information, including: Receive the fused panoramic image P(x, y), and calculate the gradient magnitude GRAD = sqrt(dx^2 + dy^2) for each frame of the image, where dx and dy are the gradients in the x and y directions respectively; Based on the gradient magnitude GRAD, determine the local maximum points as the feature point set F = {f1, f2,..., fm}, where the position of each feature point fi is represented by the coordinates (X_i, Y_i); For each feature point fi in the feature point set F, calculate its descriptor D_i = SUM((P(x + X_i, y + Y_i) - AVG_P)^2), where AVG_P is the average pixel value in the surrounding area, and (X_i, Y_i) is the center point coordinates; Store the feature points, their descriptors D_i, and the position information (X_i, Y_i) into the database together to form a feature point list: L = {(f1, D_1, X_1, Y_1), (f2, D_2, X_2, Y_2),..., (fm, D_m, X_m, Y_m)}.

6. The method for real-time generation of synchronous positioning and three-dimensional path of panoramic images of driving environment according to claim 5, characterized in that: Calculate the driving direction and speed of the vehicle using the position information of the feature points, including: Receive the feature points and their position information (X_i, Y_i) in the feature point list L, and match the feature point pairs P between consecutive frames: P = {(f1_t1, f1_t2), (f2_t1, f2_t2),..., (fn_t1, fn_t2)}, where t1 and t2 represent different time points; Based on the feature point pairs P, calculate the displacement vector V = (X_2 - X_1, Y_2 - Y_1) of each feature point between two frames, where (X_1, Y_1) and (X_2, Y_2) are the position coordinates of the same feature point at two time points respectively; Use the displacement vector V to calculate the vehicle driving direction A = atan2(V_y, V_x), where V_x and V_y are the components of the displacement vector on the x-axis and y-axis respectively, and atan2 is the arctangent function; Combine the displacement vector V and the time interval TI to calculate the vehicle speed S = sqrt(V_x^2 + V_y^2) / TI, where TI is the time difference between the shooting of two frames of images.

7. The method for synchronous positioning and real-time generation of a three-dimensional path of a panoramic image of a driving environment according to claim 6, characterized in that: Predict the vehicle position at the next moment according to the driving direction and speed, including: Receive the vehicle driving direction A and the vehicle driving speed S, and determine the current vehicle position coordinates C = (X_cur, Y_cur), where (X_cur, Y_cur) is the position of the vehicle in the current frame; Based on the vehicle driving direction A, calculate the direction vector DIR = (cos(A), sin(A)), where cos and sin are the cosine and sine operations respectively; Using the vehicle driving speed S and the time interval TI, calculate the displacement distance DIST = S * TI. Combining with the direction vector DIR, obtain the displacement vector DELTA = DIST * DIR = (DIST * cos(A), DIST * sin(A)); Add the displacement vector DELTA to the current vehicle position coordinate C to get the predicted next moment vehicle position N = (X_cur + DELTA_X, Y_cur + DELTA_Y), where DELTA_X and DELTA_Y are the components of the displacement vector on the x-axis and y-axis respectively.

8. The method for synchronous positioning and real-time generation of a three-dimensional path of a panoramic image of a driving environment according to claim 7, wherein: The continuous monitoring and correction of the three-dimensional path map includes: Based on the three-dimensional path map G_path, set the initial threshold TH = TH_0 for detecting path changes. For each edge e_ij in the path map belonging to the G_path set, calculate its edge length ED(e_ij) = dist_ij; Update the position coordinates (X'_i, Y'_i, Z_i) of the feature points through sensor data, and recalculate the new length NEW_ED(e_ij) of each edge: NEW_ED(e_ij) = sqrt((X'_j - X'_i)^2 + (Y'_j - Y'_i)^2 + (Z_j - Z_i)^2); Compare the new length NEW_ED(e_ij) with the original length ED(e_ij). If |NEW_ED(e_ij) - ED(e_ij)| > TH, mark it as abnormal and record the abnormal information in the abnormal information list AN: AN = {(e_1, |NEW_ED(e_1) - ED(e_1)|),..., (e_k, |NEW_ED(e_k) - ED(e_k)|)}; According to the abnormal information in the abnormal information list AN, correct the marked edges, and update the edge length using the weighted average method. The formula is: CORRECTED_ED(e_ij) = [ED(e_ij) * W_old + NEW_ED(e_ij) * W_new] / (W_old + W_new), where W_old and W_new are the old weight and the new weight respectively, and satisfy W_old + W_new = 1. CORRECTED_ED is the corrected edge length, and W is the weight.

Images