Road boundary line detection method and device, storage medium, electronic device, computer program product

Abstract

The application discloses a road boundary line detection method and device, a storage medium, an electronic device and a computer program product, and relates to the computer field. The road boundary line detection method comprises the following steps: acquiring target image data of a visual sensor of a target vehicle and target point cloud data of a radar sensor of the target vehicle; and detecting road boundary lines on both sides of the target vehicle according to the target point cloud data and the target image data. The method solves the problem that the road boundary lines on both sides of the vehicle cannot be accurately detected.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method and apparatus for detecting road boundary lines, a storage medium, an electronic device, and a computer program product. Background Technology

[0002] In intelligent driving assistance systems, road boundary detection is one of the key technologies for ensuring the safe operation of autonomous vehicles. Traditionally, visual sensors (such as cameras) have been the primary means of road boundary detection, using image processing algorithms to identify boundary features such as lane lines, curbs, cones, and water-filled barriers. However, the performance of visual sensors is easily affected by environmental factors, such as changes in lighting, rain, snow, and poor visibility at night, leading to a decrease in the accuracy of boundary detection. Especially in complex traffic environments, such as scenarios with heavy traffic or road edges obscured by trees or shrubs, the detection capabilities of visual sensors are further limited.

[0003] On the other hand, millimeter-wave radar, unaffected by environmental factors such as weather and lighting, can provide stable target detection information, making it an indispensable sensor in intelligent driving systems. Especially with the development of 3D millimeter-wave radar technology, it can provide three-dimensional information of targets, including distance, azimuth, and velocity, giving it potential for road boundary detection. However, the point cloud of 3D millimeter-wave radar is relatively sparse, especially at long distances, where the distribution becomes even more scattered, limiting its application range and accuracy in boundary detection. Furthermore, radar signals are susceptible to multipath effects in complex scenarios, generating false point clouds, further affecting the stability of boundary detection.

[0004] To overcome these technical challenges, existing technologies typically employ high-cost LiDAR or 4D millimeter-wave radar. While these sensors offer richer point cloud data and longer detection ranges, they impose higher demands on cost and computational complexity, limiting their application in large-scale autonomous vehicles. Furthermore, processing visual and radar data usually requires sophisticated machine learning or deep learning algorithms, which demand significant computational resources during both training and inference phases, further increasing the overall system cost.

[0005] There is no effective solution yet to the problem that existing technologies cannot accurately detect the road boundary lines on both sides of a vehicle. Summary of the Invention

[0006] This application provides a method and apparatus for detecting road boundary lines, a storage medium, an electronic device, and a computer program product, to at least solve the problem of inaccurate detection of road boundary lines on both sides of a vehicle.

[0007] According to one aspect of the embodiments of this application, a method for detecting road boundary lines is provided, comprising: acquiring target image data from a visual sensor of a target vehicle and target point cloud data from a radar sensor of the target vehicle; and detecting road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data.

[0008] In an exemplary embodiment, detecting the road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data includes: determining a first set of points based on the target point cloud data, wherein the first set of points includes N subsets, the points in the first set of points are points determined based on N+M-1 frames of point cloud data, the N+M-1 frames are consecutive frames, the points in the i-th subset of the N subsets are points determined based on the i-th to i+M-1-th frames of point cloud data in the N+M-1 frames, N is an integer greater than or equal to 2, M is an integer greater than or equal to 2, and i is an integer greater than or equal to 1 and less than or equal to N, the target point cloud data includes the N+M-1 frames of point cloud data; determining the visual road boundary lines based on the target image data, and taking points at equal intervals along the visual road boundary lines in the longitudinal direction to obtain a second set of points, wherein the longitudinal direction is parallel to the driving direction of the target vehicle; and detecting the road boundary lines on both sides of the target vehicle based on the first set of points and the second set of points.

[0009] In an exemplary embodiment, determining a first set of points based on the target point cloud data includes: obtaining the i-th subset among the N subsets in the following manner to determine the first set of points; obtaining the j-th static point set in the following manner to obtain M static point sets, and determining the i-th subset based on the M static point sets, where j is an integer greater than or equal to 1 and less than or equal to M; when the j-th frame of point cloud data in M ​​frames of point cloud data is obtained, performing compensation processing on the points in the j-th frame of point cloud data according to a first time and a second time to obtain the j-th set of points, where the first time is the time when the j-th frame of point cloud data is obtained, the second time is the time corresponding to the point cloud timestamp of the j-th frame of point cloud data, and the M frames include the i-th frame to the (i+M-1)-th frame; and determining the j-th static point set from the j-th set of points.

[0010] In an exemplary embodiment, the compensation processing of points in the j-th frame point cloud data based on a first time moment and a second time moment includes: determining the time difference between the first time moment and the second time moment to obtain a first time difference; determining the distance the target vehicle travels in the longitudinal direction within the first time difference to obtain a first distance; determining the distance the target vehicle travels in the lateral direction within the first time difference to obtain a second distance, wherein the lateral direction is perpendicular to the longitudinal direction; determining the change value of the heading angle of the target vehicle within the first time difference to obtain a first value; and performing compensation processing on points in the j-th frame point cloud data based on the first distance, the second distance, and the first value.

[0011] In an exemplary embodiment, compensation processing is performed on the points in the j-th frame point cloud data based on the first distance, the second distance, and the first value, including: compensating the a-th point in the j-th frame point cloud data using the following formula to compensate the points in the j-th frame point cloud data, wherein a is an integer greater than or equal to 1 and less than or equal to W, and W is equal to the number of points in the j-th frame point cloud data:

[0012] x=xa*cos(Δheading)+ya*sin(Δheading)-Δx;

[0013] y=ya*cos(Δheading)-xa*sin(Δheading)-Δy;

[0014] Where xa is the vertical coordinate of the a-th point, ya is the horizontal coordinate of the a-th point, Δheading is the first value, Δx is the first distance, Δy is the second distance, x is the vertical coordinate of the a-th point after compensation, and y is the horizontal coordinate of the a-th point after compensation.

[0015] In an exemplary embodiment, determining the j-th static point set from the j-th point set includes: determining the absolute velocity of each point in the j-th point set relative to the ground; determining points in the j-th point set whose absolute velocities are less than a preset velocity to obtain a reference point set; and determining the j-th static point set based on the reference point set.

[0016] In an exemplary embodiment, determining the j-th static point set based on the reference point set includes: directly determining the reference point set as the j-th static point set; or deleting points in the reference point set that meet preset conditions to obtain the j-th static point set, wherein the points in the j-th static point set are the points in the reference point set that were not deleted, and the preset conditions include: points not located in a preset region of interest, points with abnormal radar cross section (RCS) and signal-to-noise ratio (SNR) within the preset region of interest, and points within the bounding boxes of visual targets and radar targets.

[0017] In an exemplary embodiment, determining the i-th subset based on the M sets of static points includes: performing kinematic compensation processing on the first M-1 sets of static points in the M sets of static points so that the time corresponding to the compensated M-1 sets of static points is synchronized with the time corresponding to the M-th set of static points; fusing the compensated M-1 sets of static points and the M-th set of static points to obtain a target set of static points; and filtering the points in the target set of static points to obtain the i-th subset.

[0018] In an exemplary embodiment, filtering the points in the target static point set to obtain the i-th subset includes: dividing the two regions to the left and right of the target vehicle along the longitudinal direction to obtain multiple regions; processing the points in the e-th region of the multiple regions in the following manner to determine the e-th point set, and determining the i-th subset based on the obtained E point sets, where e is an integer greater than or equal to 1 and less than or equal to E, and E is the number of regions in the multiple regions: if L points in the target static point set are in the e-th region, the first preset number of points among the L points that are closest to the lateral distance of the target vehicle constitute the e-th point set.

[0019] In an exemplary embodiment, determining the i-th subset based on the obtained set of E points includes: clustering the points in the set of E points and removing points in the set of E points that do not belong to any cluster to obtain a target set of points, wherein the target set of points includes points in the set of E points that belong to clusters; processing the points in the e-th region of the plurality of regions in the following manner to determine the e-th set of points again, and merging the re-determined set of E points to obtain the i-th subset of points: if U points in the target set are in the e-th region, selecting a second preset number of points among the U points that are closest to the lateral distance of the target vehicle to form the re-determined e-th set of points, wherein the second preset number is less than the first preset number.

[0020] In an exemplary embodiment, determining the visual road boundary line based on the target image data includes: acquiring a cubic curve of the boundary line corresponding to the target image data; kinematically compensating the cubic curve of the boundary line to the acquisition time corresponding to the N+M-1th frame to obtain the visual road boundary line; and detecting the road boundary lines on both sides of the target vehicle based on the first set of points and the second set of points, including: performing kinematic compensation processing on the first N-1 subsets of the first set of points so that the time corresponding to the compensated N-1 subsets is synchronized with the time corresponding to the Nth subset, wherein the time corresponding to the Nth subset... The acquisition time corresponding to the N+M-1th frame is defined as follows: The compensated N-1 subsets and the Nth subset are fused to obtain a first reference point set; each point in the first reference point set is filtered, and the points in the filtered first reference point set are further selected to obtain a second reference point set, wherein the points in the second reference point set are the points in the filtered first reference point set that were not deleted; the second reference point set and the second point set are fused to obtain a third reference point set; the road boundary lines on both sides of the target vehicle are detected based on the third reference point set.

[0021] In an exemplary embodiment, filtering each point in the first set of reference points includes: if there are S points in the first set of reference points located on one side of the target vehicle, filtering the S points in the following manner to filter each point in the first set of reference points: filtering the s-th point among the S points using the following formula:

[0022]

[0023] Where α is a preset weight value, α∈[0,1], y s Let be the horizontal coordinate of the s-th point, and y be the horizontal coordinate of the s-th point after filtering. Let be the horizontal coordinate of the (s-1)th point after filtering, assuming s equals 1. It is equal to the horizontal coordinate of the s-th point.

[0024] In an exemplary embodiment, filtering the points in the filtered first reference point set includes: if there are S points in the filtered first reference point set located on one side of the target vehicle, filtering the S points in the filtered first reference point set is performed as follows: using the least squares method to perform cubic curve fitting on the S points and determining the corresponding minimum average error; calculating an evaluation parameter value based on the minimum average error; if the evaluation parameter value is greater than or equal to an evaluation error threshold, deleting the S points whose lateral distance from the cubic curve fitted by the least squares method exceeds the filtering threshold; if the evaluation parameter value is less than the evaluation error threshold, using the random sampling consensus algorithm to perform cubic curve fitting on the S points and deleting the S points whose lateral distance from the cubic curve fitted by the random sampling consensus algorithm exceeds the filtering threshold.

[0025] In an exemplary embodiment, calculating the evaluation parameter value based on the minimum average error includes: calculating the evaluation parameter value using the following formula:

[0026]

[0027] Wherein, Sorce is the evaluation parameter value, leastSquareError is the minimum average error, and accuracyThr is the preset accuracy threshold.

[0028] In an exemplary embodiment, kinematic compensation is performed on the cubic curve of the boundary line to the acquisition time corresponding to the N+M-1th frame to obtain the visual road boundary line. This includes: determining the time difference between the third and fourth moments to obtain a second time difference, wherein the third moment is the moment corresponding to the image exposure stamp of the target image data, and the fourth moment is the acquisition time of the N+M-1th frame; calculating the distance traveled by the target vehicle and the change in heading angle within the second time difference to obtain a third distance and a second value; and updating the cubic curve of the boundary line according to the third distance and the second value to obtain the visual road boundary line.

[0029] In an exemplary embodiment, detecting the road boundary lines on both sides of the target vehicle based on the third reference point set includes: detecting the road boundary line on one side of the target vehicle by means of the following method to detect the road boundary lines on both sides of the target vehicle: determining multiple points located on one side of the target vehicle from the third reference point set to obtain a fourth reference point set; repeatedly performing the following operations until the fourth reference point set is empty, and determining the cubic curve y = c0 + c1x + c2x corresponding to the road boundary line according to the system state matrix. 2+c3x 3 c0, c1, c2, and c3: A point is randomly selected from the fourth reference point set, and the selected point is deleted from the fourth reference point set; Based on the Kalman prediction algorithm, the system state matrix and the prediction covariance matrix are calculated according to the third distance, the second value, the reference system state matrix, and the reference prediction covariance matrix; Based on the Kalman update algorithm, the system state matrix and the prediction covariance matrix are updated according to the coordinates of the selected point; The reference system state matrix is ​​updated to the updated system state matrix, and the reference prediction covariance matrix is ​​updated to the updated prediction covariance matrix.

[0030] According to another aspect of the embodiments of this application, a road boundary line detection device is provided, comprising: an acquisition module for acquiring target image data from a visual sensor of a target vehicle and target point cloud data from a radar sensor of the target vehicle; and a detection module for detecting road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data.

[0031] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which is configured to execute the above-described road boundary detection method when running.

[0032] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the road boundary detection method described above through the computer program.

[0033] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0034] By combining the target image data from the target vehicle's visual sensor and the target point cloud data from the target vehicle's radar sensor, this application enables multi-dimensional detection of road boundary lines on both sides of the target vehicle. This improves the detection accuracy of road boundary lines on both sides of the vehicle and solves the problem of inaccurate detection of road boundary lines on both sides of the vehicle. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of an optional road boundary line detection method according to an embodiment of this application;

[0038] Figure 2 This is a schematic diagram of an optional sensor arrangement according to an embodiment of this application;

[0039] Figure 3 This is a schematic diagram of an optional road boundary line output effect according to an embodiment of this application;

[0040] Figure 4 This is a schematic diagram of an optional S-shaped cubic curve according to an embodiment of this application;

[0041] Figure 5 This is a flowchart of an optional filtering method based on least squares and random sample consensus according to an embodiment of this application;

[0042] Figure 6 This is an optional Kalman calculation flowchart for one-sided boundary points according to an embodiment of this application;

[0043] Figure 7 This is a schematic diagram of an optional road real-time boundary line fitting and tracking method based on vision and radar fusion according to an embodiment of this application;

[0044] Figure 8 This is a structural block diagram of an optional road boundary detection device according to an embodiment of this application. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0046] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0047] To address the technical problems existing in related technologies, this embodiment provides a method for detecting road boundary lines. Figure 1 This is a flowchart of an optional road boundary line detection method according to an embodiment of this application, the process including the following steps S102-S104:

[0048] Step S102: Acquire target image data from the visual sensor of the target vehicle and target point cloud data from the radar sensor of the target vehicle.

[0049] Optionally, the radar sensor is a 3D millimeter-wave angle radar, such as... Figure 2 As shown, the target vehicle has four 3D millimeter-wave angle radars and one vision sensor. Figure 2 The forward-facing sensors are installed on the left front bumper, right front bumper, left rear bumper, right rear bumper, and behind the windshield of the vehicle.

[0050] Step S104: Detect the road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data.

[0051] The above steps, by combining the target image data from the target vehicle's visual sensor and the target point cloud data from the target vehicle's radar sensor, can detect the road boundary lines on both sides of the target vehicle from multiple dimensions, improving the detection accuracy of the road boundary lines on both sides of the vehicle and solving the problem of not being able to accurately detect the road boundary lines on both sides of the vehicle.

[0052] In an exemplary embodiment, step S104 above includes the following steps S11-S13:

[0053] Step S11: Determine a first point set based on the target point cloud data, wherein the first point set includes N subsets, the points in the first point set are points determined based on N+M-1 frames of point cloud data, the N+M-1 frames are consecutive frames, the points in the i-th subset of the N subsets are points determined based on the point cloud data from the i-th frame to the i+M-1-th frame in the N+M-1 frames, N is an integer greater than or equal to 2, M is an integer greater than or equal to 2, i is an integer greater than or equal to 1 and less than or equal to N, and the target point cloud data includes the N+M-1 frames of point cloud data;

[0054] To better understand, let's assume N equals 10 and M equals 3. We can obtain the first set of points based on 12 consecutive frames of point cloud data, assuming these 12 frames are 1 frame, 2 frames, ..., 12 frames. Then, the first subset is determined by subtracting 10 frames from the first frame, the second subset by subtracting 11 frames from the second frame, and the third subset by subtracting 12 frames from the third frame.

[0055] Step S12: Determine the visual road boundary line based on the target image data, and take points at equal intervals along the visual road boundary line in the longitudinal direction to obtain a second set of points, wherein the longitudinal direction is parallel to the driving direction of the target vehicle;

[0056] It's worth noting that visual sensors can provide clear boundary line information under good lighting conditions, and even in complex urban environments, they can distinguish boundary features such as curbs and traffic cones—details that radar point clouds struggle to accurately identify. By taking equally spaced points along the visual road boundary lines, a series of representative points can be obtained. These points not only contain the shape information of the boundary lines but also adapt to different road curvatures, ensuring more accurate subsequent data fusion and curve fitting.

[0057] Step S13: Detect the road boundary lines on both sides of the target vehicle based on the first set of points and the second set of points.

[0058] It should be noted that point cloud data from 3D millimeter-wave radar provides stable boundary detection, especially in situations where the performance of visual sensors is limited, such as at night or in rainy or snowy weather. Meanwhile, high-resolution images from visual sensors under good lighting conditions can compensate for the sparse point cloud limitations of radar in low light or at long distances. By fully utilizing the advantages of both sensors while mitigating their limitations, optimal boundary detection results can be achieved.

[0059] In an exemplary embodiment, step S11 above includes: obtaining the i-th subset among the N subsets to determine the first point set:

[0060] The j-th static point set is obtained through the following steps S111-S112, resulting in M ​​static point sets. The i-th subset is then determined based on the M static point sets, where j is an integer greater than or equal to 1 and less than or equal to M.

[0061] Step S111: When the j-th frame point cloud data in the M-frame point cloud data is obtained, the points in the j-th frame point cloud data are compensated according to the first time and the second time to obtain the j-th point set. The first time is the time when the j-th frame point cloud data is obtained, and the second time is the time corresponding to the point cloud timestamp of the j-th frame point cloud data. The M-frames include the i-th frame to the (i+M-1)-th frame.

[0062] Step S112: Determine the j-th static point set from the j-th point set.

[0063] The above steps ensure that each subset of the first point set contains a sufficient amount of static point cloud data synchronized with the current moment for subsequent road boundary detection, fitting, and tracking processes, thereby improving the detection accuracy and stability of the entire system. This method effectively overcomes the limitations of a single sensor when processing the fusion of visual sensor and millimeter-wave radar data. Especially in dynamic environments, by overlaying multiple frames of data and removing dynamic points, it can stably detect and track road boundary lines, maintaining good detection performance even in adverse weather conditions, strong lighting changes, or complex traffic scenarios.

[0064] In an exemplary embodiment, the above-described compensation processing of points in the j-th frame point cloud data based on the first time point and the second time point can be achieved through the following steps S21-S25:

[0065] Step S21: Determine the time difference between the first time point and the second time point to obtain the first time difference;

[0066] Step S22: Determine the distance the target vehicle travels in the longitudinal direction within the first time difference to obtain the first distance;

[0067] Step S23: Determine the distance the target vehicle travels in the lateral direction within the first time difference to obtain the second distance, wherein the lateral direction is perpendicular to the longitudinal direction;

[0068] Step S24: Determine the change in heading angle of the target vehicle within the first time difference to obtain the first value;

[0069] It should be noted that the execution of steps S22-S24 above is not in any particular order.

[0070] Step S25: Perform compensation processing on the points in the j-th frame point cloud data according to the first distance, the second distance, and the first value.

[0071] It should be noted that the above steps effectively compensate for radar point cloud data, eliminating coordinate offsets caused by time differences and vehicle movement. This provides a more accurate and reliable data foundation for subsequent road boundary detection, fitting, and tracking. This processing method not only improves detection accuracy but also enhances the system's adaptability to dynamic environments, which is of significant value for the safe operation of intelligent driving systems.

[0072] In an exemplary embodiment, step S25 includes: compensating the a-th point in the j-th frame point cloud data using the following formulas 1 and 2, to compensate the points in the j-th frame point cloud data, wherein a is an integer greater than or equal to 1 and less than or equal to W, and W is equal to the number of points in the j-th frame point cloud data:

[0073] x=xa*cos(Δheading)+ya*sin(Δheading)-Δx; (Formula 1)

[0074] y=ya*cos(Δheading)-xa*sin(Δheading)-Δy; (Formula 2)

[0075] Where xa is the vertical coordinate of the a-th point, ya is the horizontal coordinate of the a-th point, Δheading is the first value, Δx is the first distance, Δy is the second distance, x is the vertical coordinate of the a-th point after compensation, and y is the horizontal coordinate of the a-th point after compensation.

[0076] In an exemplary embodiment, determining the j-th static point set from the j-th point set includes the following steps S31-S33:

[0077] Step S31: Determine the absolute velocity of each point in the j-th point set relative to the ground;

[0078] Optionally, the absolute velocity of each point output by the radar relative to the ground can be calculated based on vehicle information (including but not limited to: the target vehicle's speed, steering ratio, and yaw angle), information of each point output by the radar (including but not limited to distance, speed, and azimuth), and radar installation location information.

[0079] It should be noted that the distance of the point output by the radar is the distance from the point to the radar sensor.

[0080] Step S32: Determine the points whose absolute velocity is less than the preset velocity from the j-th point set to obtain the reference point set;

[0081] It should be noted that points whose absolute speed exceeds the preset speed are designated as dynamic points, while points whose absolute speed does not exceed the preset speed are designated as static points.

[0082] Step S33: Determine the j-th static point set based on the reference point set.

[0083] It should be noted that the above steps can effectively separate static points from radar point cloud data to form a set of static points for road boundary line detection. This not only improves the accuracy of detection but also optimizes computational efficiency and enhances the stability and applicability of intelligent driving assistance systems in complex environments.

[0084] In an exemplary embodiment, determining the j-th static point set based on the reference point set includes: directly determining the reference point set as the j-th static point set; or deleting points in the reference point set that meet preset conditions to obtain the j-th static point set, wherein the points in the j-th static point set are the points in the reference point set that were not deleted, and the preset conditions include: points not located in a preset region of interest, points with abnormal radar cross section (RCS) and signal-to-noise ratio (SNR) within the preset region of interest, and points within the bounding boxes of visual targets and radar targets.

[0085] It should be noted that false static points may appear when the radar detection performance boundary or the target moves parallel to the radar plane, and these need to be filtered out. Additionally, expansion joints, manhole covers, and speed bumps on the road will also be classified as static points and require filtering. Therefore, to reduce computational load, a preset region of interest (ROI) is defined, with a range of ±30m in front and behind the target vehicle and ±10m to the left and right. Points within the preset ROI are initially filtered based on attributes such as RCS, SNR, and position, while points outside the preset ROI are discarded.

[0086] In addition, it is necessary to use the boxes of visual targets and radar targets to filter out static points within the boxes and prevent interference from false static points within the targets.

[0087] In an exemplary embodiment, determining the i-th subset based on the M static point sets includes the following steps S41-S43:

[0088] Step S41: Perform kinematic compensation processing on the first M-1 static point sets in the M static point sets so that the time corresponding to the compensated M-1 static point sets is synchronized with the time corresponding to the Mth static point set.

[0089] It should be noted that kinematic compensation processing can be performed on the m-th static point set among the M static point sets based on the m-th time and the M-th time. The m-th time is the acquisition time of the frame corresponding to the m-th static point set, and the M-th time is the acquisition time of the frame corresponding to the M-th static point set among the M static point sets. m is an integer greater than or equal to 1 and less than or equal to M-1.

[0090] It should be noted that the purpose of step S41 above is to ensure that all static point sets used to construct subsets are synchronized in time, which helps to accurately reflect the vehicle's surrounding environment at the current moment.

[0091] The specific compensation principle is similar to steps S21-S25 above. This compensation can eliminate the displacement deviation relative to the current moment caused by the movement of the target vehicle, ensuring that all point sets are aligned in space and time. Furthermore, through kinematic compensation, all static point sets reflect the environmental state at the same moment, avoiding detection errors caused by time deviations and laying a solid foundation for subsequent fusion processing.

[0092] Step S42: Merge the compensated M-1 sets of static points with the Mth set of static points to obtain the target set of static points;

[0093] It should be noted that the compensated first M-1 static point sets are merged with the uncompensated Mth static point set to form a target static point set containing richer information. The fused target static point set not only broadens the detection range but also improves the recognition of road boundary features. Especially when the point cloud is sparse or noisy, the superposition of information from multiple frames helps to improve the stability and accuracy of detection.

[0094] Step S43: Filter the points in the target static point set to obtain the i-th subset.

[0095] In this embodiment, because the single-frame point cloud of 3D millimeter-wave radar is sparse and it is difficult to extract effective features, multi-frame overlay of the point cloud is required. For multi-frame overlay of static point clouds, the required number of frames (M frames) can be set, and a dynamic array for storing multi-frame point cloud information can be created. The first frame of point cloud is directly stored in the dynamic array. Subsequently, based on the time difference between the newly acquired frame's point cloud and the previous frame's point cloud, and the motion information of the target vehicle speed, the historical points in the dynamic array are rotated and translated to compensate for the current time. If the number of point cloud frames stored in the dynamic array exceeds the set threshold, the earliest point cloud is removed from the array.

[0096] It should be noted that by implementing the above steps S41-S43, an accurate, stable, and information-rich subset can be constructed for the detection and fitting of road boundary lines, which is of great significance for the environmental perception and safe driving of intelligent driving assistance systems.

[0097] In an exemplary embodiment, step S43 above includes the following steps S431-S432:

[0098] Step S431: Divide the two regions on the left and right sides of the target vehicle along the longitudinal direction to obtain multiple regions;

[0099] Step S432 processes the points in the e-th region of the plurality of regions in the following manner to determine the e-th point set, and determines the i-th subset based on the obtained E point sets, where e is an integer greater than or equal to 1 and less than or equal to E, and E is the number of regions of the plurality of regions: when L points in the target static point set are in the e-th region, the first preset number of points among the L points that are closest to the lateral distance of the target vehicle constitute the e-th point set.

[0100] It should be noted that after multiple frames of static points are superimposed, the effective features of the road boundary become more obvious. However, as the number of point clouds increases, the number of interfering points also increases dramatically. Furthermore, the computational load for fitting and tracking the road boundary lines increases with the number of points. Therefore, a method based on prior knowledge and using grid partitioning to select points is adopted to minimize the removal of real road boundary points while reducing the overall number of point clouds. First, based on the prior knowledge that the road boundary mainly exists on the left and right sides of the target vehicle, the left and right sides of the target vehicle are divided into two regions. The two regions are then divided into several sub-regions at equal intervals (or non-equal intervals depending on the actual situation) in the longitudinal direction. The radar points in each sub-region are arranged in ascending order of their lateral distance from the vehicle. Several points with small distances (the number of points can be adjusted according to the computing power requirements) are selected as candidate points for subsequent operations, while other points are discarded.

[0101] In an exemplary embodiment, determining the i-th subset based on the obtained set of E points includes the following steps S51-S52:

[0102] Step S51: Cluster the points in the E point sets and remove the points in the E point sets that do not belong to any cluster to obtain the target point set, wherein the target point set includes the points in the E point sets that belong to the clusters;

[0103] Step S52: Process the points in the e-th region of the plurality of regions in the following manner to determine the e-th point set again, and merge the E-th point sets again to obtain the i-th subset: If there are U points in the target point set in the e-th region, select the second preset number of points among the U points that are closest to the target vehicle in the lateral distance to form the e-th point set again, wherein the second preset number is less than the first preset number.

[0104] It should be noted that there will still be many outliers in the E-point set, which are likely interference points and need to be filtered out. This application can use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method to set an appropriate distance threshold on the E-point set, cluster the non-outliers, and remove the outliers.

[0105] After removing outliers, follow steps S431-S432 above to divide the area into several grids vertically. The radar points in each grid area are arranged in ascending order of their lateral distance from the vehicle. Select several points with small distances (the number of points can be adjusted according to computing power requirements) as the original points of the road boundary for subsequent operations, and discard the other points.

[0106] It should be noted that the re-determined e-th point set not only includes points related to the road boundary line, but also further reduces the computational load by setting a second preset number, ensuring that the system maintains high accuracy while being highly efficient. Subsequently, the re-determined E point sets are merged to obtain the i-th subset, which will be directly used for fitting and tracking the road boundary line, ensuring the consistency of the detection process and the reliability of the output.

[0107] It should be noted that the technical features of steps S51 and S52, through cluster analysis, outlier removal, and further screening and fusion of point cloud data, significantly enhance the accuracy, stability, and computational efficiency of road boundary line detection. They also provide a solid data foundation for subsequent multi-sensor information fusion, which is of great value for improving the overall performance of intelligent driving assistance systems.

[0108] In an exemplary embodiment, the determination of the visual road boundary line based on the target image data includes the following steps S61-S62:

[0109] Step S61: Obtain the cubic curve of the boundary line corresponding to the target image data;

[0110] Step S62: Compensate the boundary line with cubic curve kinematics to the acquisition time corresponding to the N+M-1th frame to obtain the visual road boundary line.

[0111] Steps S61 and S62 described above illustrate how to extract the cubic curve of the boundary line from the visual image data and ensure its time synchronization with the radar data through kinematic compensation, providing key technical support for multi-sensor fusion road boundary line detection. This processing method not only improves the detection accuracy but also enhances the system's environmental adaptability and detection stability.

[0112] In an exemplary embodiment, step S62 above can be implemented by the following steps S621-S623:

[0113] Step S621: Determine the time difference between the third time and the fourth time to obtain the second time difference, wherein the third time is the time corresponding to the image exposure stamp of the target image data, and the fourth time is the acquisition time of the N+M-1th frame;

[0114] Step S622: Calculate the distance traveled by the target vehicle and the change in heading angle within the second time difference to obtain the third distance and the second value;

[0115] Step S623: Update the cubic curve of the boundary line according to the third distance and the second value to obtain the visual road boundary line.

[0116] In an exemplary embodiment, step S623 is implemented as follows: when the cubic curve of the boundary line is y = c0 + c1x + c2x 2 +c3x 3 In this case, c0, c1, c2, and c3 in the cubic curve of the boundary line are updated using the following formula three, and the updated cubic curve of the boundary line is represented as the visual road boundary line:

[0117] X = F*X + B*U; (Formula 3)

[0118] in, Δd is the third distance, and Δheading is the second value.

[0119] In an exemplary embodiment, detecting the road boundary lines on both sides of the target vehicle based on the first set of points and the second set of points can be achieved through the following steps S71-S75:

[0120] Step S71: Perform kinematic compensation processing on the first N-1 subsets in the first point set so that the time corresponding to the compensated N-1 subsets is synchronized with the time corresponding to the Nth subset, wherein the time corresponding to the Nth subset is the acquisition time corresponding to the N+M-1th frame;

[0121] It should be noted that kinematic compensation processing can be performed on the nth subset of the first point set based on the nth time and the Nth time. The nth time is the acquisition time of the last frame corresponding to the nth subset, and the Nth time is the acquisition time of the last frame (i.e., the N+M-1th frame) corresponding to the Nth subset of the first point set, where n is an integer greater than or equal to 1 and less than or equal to N-1.

[0122] The above ensures that the subsets in the first point set are synchronized with the data at the current moment, providing a consistent spatiotemporal framework for subsequent fusion and detection. Kinematic compensation processing ensures that all point cloud data used for detection are aligned on the time axis, eliminating point cloud misalignment caused by vehicle motion. This provides a time synchronization basis for the fusion of multi-sensor information and subsequent road boundary detection, improving the accuracy of the detection results.

[0123] Step S72: Merge the compensated N-1 subsets and the Nth subset to obtain the first reference point set;

[0124] Step S73: Filter each point in the first reference point set, and filter the points in the filtered first reference point set to obtain a second reference point set, wherein the points in the second reference point set are the points in the filtered first reference point set that were not deleted;

[0125] It should be noted that various algorithms can be used for filtering, such as Exponentially Weighted Moving Average (EWMA) filtering. Filtering and point selection ensure high-quality point cloud data, help avoid interference from points with non-road boundary features in subsequent detection, and improve the accuracy of road boundary line detection.

[0126] Step S74: Merge the second set of reference points and the second set of points to obtain the third set of reference points;

[0127] Step S75: Detect the road boundary lines on both sides of the target vehicle based on the third set of reference points.

[0128] Optionally, the Kalman prediction and update operation can be used to fit the road boundary lines of the points in the third reference point set, incorporating the observation noise (error) R of each point in the third reference point set. Since the third reference point set contains multi-sensor information, the resulting road boundary line detection results are more comprehensive and stable, and can adapt to detection needs under various environmental conditions.

[0129] The above steps can effectively detect road boundary lines, taking into account not only time synchronization, data fusion and point cloud quality optimization, but also making full use of the complementarity of multi-sensor information, providing intelligent driving vehicles with accurate and robust road environment perception capabilities.

[0130] In an exemplary embodiment, filtering each point in the first set of reference points includes: if there are S points in the first set of reference points located on one side of the target vehicle, filtering the S points in the following manner to filter each point in the first set of reference points: filtering the s-th point among the S points using the following formula:

[0131]

[0132] Where α is a preset weight value, α∈[0,1], y s Let be the horizontal coordinate of the s-th point, and y be the horizontal coordinate of the s-th point after filtering. Let be the horizontal coordinate of the (s-1)th point after filtering, assuming s equals 1. It is equal to the horizontal coordinate of the s-th point.

[0133] In an exemplary embodiment, the above-described filtering of points in the filtered first reference point set includes: if there are S points in the filtered first reference point set located on one side of the target vehicle, the S points are filtered through the following steps S81-S83 to filter the points in the filtered first reference point set:

[0134] Step S81: Perform cubic curve fitting on the S points using the least squares method and determine the corresponding minimum average error; calculate the evaluation parameter value based on the minimum average error;

[0135] In an exemplary embodiment, calculating the evaluation parameter value based on the minimum average error includes:

[0136] The evaluation parameter value is calculated using the following formula five:

[0137]

[0138] Wherein, Sorce is the evaluation parameter value, leastSquareError is the minimum average error, and accuracyThr is the preset accuracy threshold.

[0139] It should be noted that step S81 fits a cubic curve using the least squares method and calculates the evaluation parameter values, providing an important basis for subsequent point selection and curve optimization. If the accuracy of the fitting result is high enough (the evaluation parameter value is greater than or equal to the evaluation error threshold), then proceed directly to step S82 for point selection; otherwise, a more complex algorithm (such as the random sampling consensus algorithm mentioned in step S83) is needed for further optimization.

[0140] Step S82: If the evaluation parameter value is greater than or equal to the evaluation error threshold, delete the points among the S points whose horizontal distance from the cubic curve fitted by the least squares method exceeds the filtering threshold.

[0141] Step S83: If the evaluation parameter value is less than the evaluation error threshold, use the random sampling consensus algorithm to perform cubic curve fitting on the S points, and delete the points among the S points whose horizontal distance from the cubic curve obtained by the random sampling consensus algorithm exceeds the filtering threshold.

[0142] In other words, when the evaluation parameter value of the least squares fitting result is lower than the evaluation error threshold, the more robust RANSAC algorithm is used to fit the cubic curve in order to overcome noise and outliers in the point cloud and improve the accuracy of the fitted curve.

[0143] In an exemplary embodiment, detecting the road boundary lines on both sides of the target vehicle based on the third reference point set includes: detecting the road boundary line on one side of the target vehicle through the following steps S91-S92, so as to detect the road boundary lines on both sides of the target vehicle:

[0144] Step S91: Determine multiple points located on one side of the target vehicle from the third set of reference points to obtain a fourth set of reference points;

[0145] Step S92: Repeat the following operations until the fourth reference point set is empty, and determine the cubic curve y = c0 + c1x + c2x corresponding to the road boundary line according to the system state matrix. 2 +c3x 3c0, c1, c2, and c3: A point is randomly selected from the fourth reference point set, and the selected point is deleted from the fourth reference point set; Based on the Kalman prediction algorithm, the system state matrix and the prediction covariance matrix are calculated according to the third distance, the second value, the reference system state matrix, and the reference prediction covariance matrix; Based on the Kalman update algorithm, the system state matrix and the prediction covariance matrix are updated according to the coordinates of the selected point; The reference system state matrix is ​​updated to the updated system state matrix, and the reference prediction covariance matrix is ​​updated to the updated prediction covariance matrix.

[0146] It should be noted that the specific implementation of step S92 above is as follows: Repeatedly execute steps S921-S926 until the fourth reference point set is empty, and then... c0, c1, c2, and c3 are determined to be the cubic curve y = c0 + c1x + c2x corresponding to the road boundary line. 2 +c3x 3 c0, c1, c2, and c3 in the context of the first execution of the operation, and P k-1 For the preset matrix:

[0147] Step S921: Randomly select a point from the fourth reference point set, and delete the selected point from the fourth reference point set:

[0148] Step S922: Calculate the system state matrix using the following formula.

[0149] Step S923: Calculate the prediction covariance matrix using the following formula.

[0150] Step S924: Calculate the Kalman gain matrix using the following formula.

[0151] Step S925: Update the system state matrix using the following formula

[0152] Step S926: Update the predicted covariance matrix using the following formula

[0153] in, Z = y; x is the vertical coordinate of the selected point, y is the horizontal coordinate of the selected point, R is the observation noise matrix of the selected point, and Q is the process noise matrix of the selected point.

[0154] It should be noted that the output effect of the road boundary line in this application is as follows: Figure 3 The red dots represent dynamic radar points, the green dots represent static points, and the red curve represents the road boundary line that is finally fitted and tracked.

[0155] This application integrates roadside line detection from visual sensors and point cloud information from millimeter-wave radar to detect road boundaries (low curbs, rows of traffic cones, water-filled barriers, etc.) on both sides of the vehicle and utilizes a cubic helix curve (y = c0 + c1x + c2x). 2 +c3x 3 This method performs fitting and tracking. It expands the output range of road boundary line detection without adding additional high-level intelligent driving sensors (4D millimeter-wave radar / LiDAR) or using a high-performance computing platform, and improves the stability and accuracy of detection under different weather, lighting, and traffic conditions.

[0156] This application fuses road boundary lines output from existing forward-looking sensors with millimeter-wave radar point cloud information to stably detect, fit, and track road boundary lines. It leverages forward-looking sensors to compensate for the limited longitudinal detection range of 3D millimeter-wave radar and corner radar, the inaccuracy caused by radar multipath effects in complex traffic flow, and the sparse radar points at low speeds. Millimeter-wave radar compensates for the performance limitations of forward-looking sensors in adverse weather or low-light conditions, and compensates for the inability of visual sensors to detect irregular road boundaries (types not labeled and trained, such as trees across the road, fallen rocks, etc.) and provides blind spot coverage outside the field of view (FOV). Ultimately, this improves the output range of road boundary lines, enhances the generalization ability of boundary detection in different scenarios (reducing missed detections), and improves output accuracy and stability.

[0157] To better understand the above solution, the entire process implemented in this application is as follows, consisting of 14 steps:

[0158] (1) Synchronization of radar and visual sensor data:

[0159] The differences between the radar point cloud timestamp, the visual sensor image exposure timestamp, and the current timestamp are calculated separately. Then, kinematic compensation is performed on the radar and visual sensor data to achieve data synchronization. Specifically:

[0160] This application utilizes point clouds from four corner radars (front left, front right, rear left, and rear right), target information, and roadside and target information output from a vision sensor. Therefore, time synchronization between the corner radar data and the vision sensor data is required. The synchronization method involves the assisted driving domain controller's software transmitting radar and vision sensor data to the application via DDS (Data Distribution Service). The application then obtains the current timestamp, calculates the difference between the timestamp and the radar point cloud timestamp, and the timestamp of the vision sensor image exposure. Compensation is then performed based on the time difference and kinematic relationships to synchronize the acquired data to the current moment.

[0161] 1) Radar point cloud compensation principle:

[0162] Step 1: Calculate the difference between the current timestamp and the radar point cloud timestamp: timeDiff radar ;

[0163] Step 2: Calculate timeDiff using the linear interpolation method radar The distances Δx and Δy traveled by the vehicle in the x and y directions, and the change in heading angle Δheading;

[0164] Step 3: Use rotation and translation matrices to compensate the x and y coordinates of each point to the current time.

[0165] The equations for the rotation and translation matrices are shown in Formula 1 and Formula 2 above.

[0166] 2) Visual roadside compensation principle:

[0167] Step 1: Calculate the difference between the current timestamp and the image exposure timestamp as timeDiff. camera ;

[0168] Step 2: Calculate timeDiff using the linear interpolation method radar The distance traveled by the vehicle Δd and the change in heading angle Δheading;

[0169] Step 3: Use the prediction equation to compensate to the current time;

[0170] y = c0 + c1x + c2x 2 +c3x 3 If a cubic curve represents the roadside, then the prediction equation is X = F*X + B*U. The state variable X, state transition matrix F, control matrix B, and control variable U are detailed above.

[0171] (2) Dynamic and static separation of radar single-frame point cloud:

[0172] Based on information such as vehicle speed, steering ratio, yaw angle, and radar output point cloud distance, speed, azimuth angle, and radar installation location, the absolute velocity information of the radar output point cloud relative to the ground is calculated. Points with speeds exceeding a threshold are designated as dynamic points, while points with no speed or low speed are designated as static points. The specific threshold setting depends on the radar sensor performance (including but not limited to accuracy, precision, and resolution).

[0173] (3) Single-frame static point filtering based on ROI and radar attributes:

[0174] First, static points outside the defined Region of Interest (ROI) are filtered out. Then, based on prior knowledge, static points within the ROI that exhibit significantly abnormal properties such as Radar Cross-Section (RCS) and Signal Noise Ratio (SNR) are excluded. Specifically:

[0175] Because false static points may appear when the radar detection performance boundary or the target moves parallel to the radar plane, these need to be filtered out. Additionally, expansion joints, manhole covers, and speed bumps on the road are also classified as static points and require filtering. Therefore, to reduce computational load, a region of ±30m in front of and behind the vehicle, and ±10m to the left and right, is defined as the Region of Interest (ROI). Points within the ROI are initially filtered based on attributes such as RCS, SNR, and position, while points outside the ROI are discarded.

[0176] (4) Static point filtering based on visual and radar target bounding boxes:

[0177] Using boxes for visual and radar targets to filter out static points within the boxes prevents interference from false static points within the target.

[0178] (5) Multi-frame overlay of radar static point cloud:

[0179] Since the point cloud of a single frame of 3D millimeter-wave radar is relatively sparse and it is not easy to extract effective features, it is necessary to perform multi-frame overlay of the point cloud. For multi-frame overlay of static point cloud, firstly, the required number of frames to be overlaid is set, and a dynamic array is created to store the point cloud information of multiple frames. The first frame of point cloud is directly stored in the dynamic array. Then, based on the time difference between the point cloud of the newly acquired frame and the point cloud of the previous frame and the vehicle motion information, the historical points in the dynamic array are rotated and translated (same as the radar point cloud time compensation principle in step (1)) to compensate to the current time. If the number of point cloud frames stored in the dynamic array exceeds the set threshold, the earliest point cloud is removed from the array.

[0180] (6) Static point selection based on grid region division:

[0181] The vehicle's left and right sides are divided into two regions. These two regions are then further divided longitudinally into several sub-regions at equal intervals (though non-equal intervals may be necessary depending on the situation). The radar points within each sub-region are arranged in ascending order of their lateral distance from the vehicle. Several points with the smallest distances are selected (the number of points can be adjusted based on computing power requirements) as the points for subsequent operations. Specifically:

[0182] After stacking multiple frames of static points, the effective features of the road boundary become more apparent. However, as the number of point clouds increases, the number of interfering points also increases dramatically. Furthermore, the computational load for fitting and tracking the road boundary lines increases with the number of points. Therefore, a method based on prior knowledge and using grid partitioning to select points is proposed to minimize the removal of real road boundary points while reducing the overall number of point clouds. First, utilizing the prior knowledge that road boundaries mainly exist on the left and right sides of the vehicle, the left and right sides of the vehicle are divided into two regions. These two regions are then further divided into several sub-regions at equal intervals (or non-equal intervals depending on the actual situation). The radar points in each sub-region are arranged according to their lateral distance from the vehicle, from smallest to largest. Several points with the smallest distances (the number of points can be adjusted according to computational requirements) are selected as candidate points for subsequent operations, while other points are discarded. The region division is as follows: Figure 7 As shown.

[0183] (7) Radar static point cloud DBSCAN clustering filtering:

[0184] After step (6), there will still be many outliers among the selected candidate points, which are likely to be interference points and need to be filtered out. This application adopts the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, which sets an appropriate distance threshold on the point cloud obtained in step (6) to cluster out the non-outliers and remove the outliers.

[0185] (8) Extraction of candidate points for radar road boundaries based on grid region division:

[0186] Using the same method as step (6), the system is divided into several grids at equal intervals in the vertical direction. The radar points in each grid area are arranged in ascending order of their lateral distance from the vehicle. Several points with small distances (the number of points can be adjusted according to computing power requirements) are selected as the original points of the road boundary for subsequent operations, while other points are discarded. Finally, some of the output points in step (7) are extracted as candidate points for the radar road boundary.

[0187] (9) Multiple frames superimposed on candidate points at radar road boundaries:

[0188] The point cloud multi-frame overlay method of step (5) is used to overlay multiple frames on the road boundary candidate points obtained in step (8). See step S11 for details.

[0189] (10) Filtering of candidate points for radar road boundary based on EWMA:

[0190] The points obtained in step (9) are subjected to EWMA (Exponentially Weighted Moving Average) filtering operation. The EWMA formula is detailed in Formula 4.

[0191] (11) Curve fitting filtering fusing the least squares method and the random sample consensus method:

[0192] According to the set error threshold, the least squares method and the random sample consensus method are used to filter the points output in step (10), specifically:

[0193] The random sample consensus method is hereinafter referred to as RANSAC (Random Sample Consensus). The road boundary (curb) line of the present application adopts a cubic spiral curve (y=c0+c1x+c2x 2 +c3x 3 ). If the boundary points screened in the first 10 steps have the situation as shown Figure 4 , fitting each point is prone to "S"-shaped distortion, that is, overfitting. Therefore, it is necessary to filter out the points that cause the curve distortion (generally points with too far or too close lateral distance).

[0194] The present application adopts a method fusing least squares and RANSAC to perform the filtering operation. First, according to the accuracy requirement of the road boundary line, an accuracy threshold accuracyThr, a filtering threshold filterThr, and an error evaluation threshold scoreThr are set. Then, the idea of the least squares method is used to calculate the minimum average error leastSquareError of the cubic curve among the points provided in step (10), and the error evaluation formula is set as the above-mentioned Formula 5.

[0195] If the score Sorce ≥ sorceThr, the lateral distance between all points and the cubic curve fitted by the least squares method is calculated, and the points exceeding filterThr are filtered out; if Sorce < sorceThr, the RANSAC method is used for cubic curve fitting, the points whose lateral distance to the curve exceeds filterThr are filtered out, and the remaining points are taken as the final output points after filtering, and the flow of the filtering method is as shown Figure 5 .

[0196] (12) Visual road boundary point extraction:

[0197] A certain sampling interval is set, and points are taken at equal intervals in the longitudinal direction on the acquired visual road boundary (curb) line, and the set of points is the visual road boundary line points.

[0198] (13) Dynamic setting and fusion of multi-sensor boundary point observation errors:

[0199] Based on the scene information provided by the visual sensor, the weather and light information provided by the rain sensor, the wiper signal, and the photosensitive sensor, the observation noise (error) R value of the radar boundary points obtained in step (11) and the visual road boundary line points extracted by method (12) is dynamically set, and the point clouds of the two are merged into the same point cloud array.

[0200] (14) Road boundary fitting and tracking based on Kalman tracing:

[0201] The points output in step (13) are traversed sequentially. The coordinate values ​​(x, y) and observation noise (error) R of each point are used to perform Kalman prediction and update operations on the cubic curve representing the road boundary line. Finally, the road boundary line is fitted and tracked, and the road boundary lines on the left and right sides of the vehicle are output.

[0202] The Kalman calculation process for a single-sided boundary point is as follows: Figure 6 As shown, the formula is as follows:

[0203] 1) Kalman prediction formula:

[0204]

[0205] 2) Kalman update formula:

[0206]

[0207] Where R is the observation noise matrix, Q is the process noise matrix, F is the state transition matrix, H is the state observation transition matrix, B is the control input matrix, X is the system state matrix, P is the covariance matrix, and K... k This is the Kalman gain matrix.

[0208] It should be noted that, Figure 7 This is a schematic diagram illustrating a method for fitting and tracking real-time road boundary lines based on the fusion of vision and radar.

[0209] It should be noted that this application proposes a millimeter-wave radar anomaly point cloud filtering method that combines least squares method with RANSAC, which is used to extract the road boundary points used for fitting and tracking, and to prevent the cubic curve from exhibiting an abnormal "S" shape after overfitting.

[0210] It should be noted that this application integrates 3D millimeter-wave radar road boundary point cloud and visual roadside sampling points as the boundary points for road boundary fitting, thereby expanding the road boundary detection range and improving detection accuracy.

[0211] It should be noted that this application uses the Kalman filter method to fit and track cubic curves of selected multi-source (visual + radar) road boundary points. Specifically, based on the characteristics of millimeter-wave radar and visual sensors in different scenarios, the observation noise (error) R of road boundary points for different sensors is dynamically set to improve the generalization ability and performance boundary of road boundary line fitting in different scenarios. (For example, when the traffic flow is dense, the radar multipath effect is significant and easily interfered with by dense traffic flow; the radar R value should be increased, and the visual boundary points should be given more weight during fitting. In rainy or dark weather, the performance of the visual sensor is limited; the visual sensor R value should be increased, and the radar boundary points should be given more weight.)

[0212] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0213] This embodiment also provides a road boundary line detection device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0214] Figure 8 This is a structural block diagram of an optional road boundary detection device according to an embodiment of this application; as shown... Figure 8 As shown, the road boundary detection device includes:

[0215] The acquisition module 82 is used to acquire target image data from the visual sensor of the target vehicle and target point cloud data from the radar sensor of the target vehicle.

[0216] The detection module 84 is used to detect the road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data.

[0217] The aforementioned device, by combining target image data from the target vehicle's visual sensor and target point cloud data from the target vehicle's radar sensor, can detect the road boundary lines on both sides of the target vehicle from multiple dimensions, thereby improving the detection accuracy of the road boundary lines on both sides of the vehicle and solving the problem of not being able to accurately detect the road boundary lines on both sides of the vehicle.

[0218] In an exemplary embodiment, the detection module 84 is further configured to determine a first set of points based on the target point cloud data, wherein the first set of points includes N subsets, the points in the first set of points are points determined based on N+M-1 frames of point cloud data, the N+M-1 frames are consecutive frames, the points in the i-th subset of the N subsets are points determined based on the i-th to i+M-1-th frames of point cloud data in the N+M-1 frames, N is an integer greater than or equal to 2, M is an integer greater than or equal to 2, i is an integer greater than or equal to 1 and less than or equal to N, and the target point cloud data includes the N+M-1 frames of point cloud data; determine a visual road boundary line based on the target image data, and take points at equal intervals along the visual road boundary line in the longitudinal direction to obtain a second set of points, wherein the longitudinal direction is parallel to the driving direction of the target vehicle; and detect the road boundary lines on both sides of the target vehicle based on the first set of points and the second set of points.

[0219] In an exemplary embodiment, the detection module 84 is further configured to obtain the i-th subset among the N subsets to determine the first point set by: obtaining the j-th static point set to obtain M static point sets, and determining the i-th subset based on the M static point sets, where j is an integer greater than or equal to 1 and less than or equal to M: when the j-th frame point cloud data in M ​​frames of point cloud data is obtained, the points in the j-th frame point cloud data are compensated according to a first time and a second time to obtain the j-th point set, where the first time is the time when the j-th frame point cloud data is obtained, the second time is the time corresponding to the point cloud timestamp of the j-th frame point cloud data, and the M frames include the i-th frame to the (i+M-1)-th frame; and determining the j-th static point set from the j-th point set.

[0220] In an exemplary embodiment, the detection module 84 is further configured to: determine the time difference between the first time point and the second time point to obtain a first time difference; determine the distance the target vehicle travels in the longitudinal direction within the first time difference to obtain a first distance; determine the distance the target vehicle travels in the lateral direction within the first time difference to obtain a second distance, wherein the lateral direction is perpendicular to the longitudinal direction; determine the change value of the heading angle of the target vehicle within the first time difference to obtain a first value; and perform compensation processing on the points in the j-th frame point cloud data based on the first distance, the second distance, and the first value.

[0221] In an exemplary embodiment, the detection module 84 is further configured to perform compensation processing on the a-th point in the j-th frame point cloud data using the following formula, so as to compensate the points in the j-th frame point cloud data, wherein a is an integer greater than or equal to 1 and less than or equal to W, and W is equal to the number of points in the j-th frame point cloud data:

[0222] x=xa*cos(Δheading)+ya*sin(Δheading)-Δx;

[0223] y=ya*cos(Δheading)-xa*sin(Δheading)-Δy;

[0224] Where xa is the vertical coordinate of the a-th point, ya is the horizontal coordinate of the a-th point, Δheading is the first value, Δx is the first distance, Δy is the second distance, x is the vertical coordinate of the a-th point after compensation, and y is the horizontal coordinate of the a-th point after compensation.

[0225] In an exemplary embodiment, the detection module 84 is further configured to determine the absolute velocity of each point in the j-th point set relative to the ground; determine points in the j-th point set whose absolute velocity is less than a preset velocity to obtain a reference point set; and determine the j-th static point set based on the reference point set.

[0226] In an exemplary embodiment, the detection module 84 is further configured to directly determine the reference point set as the j-th static point set; or to delete points in the reference point set that meet preset conditions to obtain the j-th static point set, wherein the points in the j-th static point set are the points in the reference point set that have not been deleted, and the preset conditions include: points not located in a preset region of interest, points with abnormal radar cross section (RCS) and signal-to-noise ratio (SNR) in the preset region of interest, and points within the bounding boxes of visual targets and radar targets.

[0227] In an exemplary embodiment, the detection module 84 is further configured to perform kinematic compensation processing on the first M-1 static point sets in the M static point sets, so that the time corresponding to the compensated M-1 static point sets is synchronized with the time corresponding to the Mth static point set; merge the compensated M-1 static point sets and the Mth static point set to obtain a target static point set; and filter the points in the target static point set to obtain the i-th subset.

[0228] In an exemplary embodiment, the detection module 84 is further configured to divide the two regions to the left and right of the target vehicle in the longitudinal direction to obtain multiple regions; to process the points in the e-th region of the multiple regions in the following manner to determine the e-th point set, and to determine the i-th subset based on the obtained E point set, where e is an integer greater than or equal to 1 and less than or equal to E, and E is the number of regions of the multiple regions: when L points in the target static point set are in the e-th region, the first preset number of points among the L points that are closest to the lateral distance of the target vehicle constitute the e-th point set.

[0229] In an exemplary embodiment, the detection module 84 is further configured to cluster the points in the E point set and remove the points in the E point set that do not belong to any cluster to obtain a target point set, wherein the target point set includes the points in the E point set that belong to the clusters; the points in the e-th region of the plurality of regions are processed in the following manner to determine the e-th point set again, and the E point sets determined again are merged to obtain the i-th subset: if there are U points in the target point set in the e-th region, the second preset number of points among the U points that are closest to the lateral distance of the target vehicle constitutes the re-determined e-th point set, wherein the second preset number is less than the first preset number.

[0230] In an exemplary embodiment, the detection module 84 is further configured to acquire the cubic curve of the boundary line corresponding to the target image data; and to kinematically compensate the cubic curve of the boundary line to the acquisition time corresponding to the N+M-1th frame to obtain the visual road boundary line.

[0231] In an exemplary embodiment, the detection module 84 is further configured to perform kinematic compensation processing on the first N-1 subsets of the first point set, so that the time corresponding to the compensated N-1 subsets is synchronized with the time corresponding to the Nth subset, wherein the time corresponding to the Nth subset is the acquisition time corresponding to the N+M-1th frame; fuse the compensated N-1 subsets and the Nth subset to obtain a first reference point set; filter each point in the first reference point set, and filter the points in the filtered first reference point set to obtain a second reference point set, wherein the points in the second reference point set are the points in the filtered first reference point set that have not been deleted; fuse the second reference point set and the second point set to obtain a third reference point set; and detect the road boundary lines on both sides of the target vehicle based on the third reference point set.

[0232] In an exemplary embodiment, the detection module 84 is further configured to, when there are S points in the first set of reference points located on one side of the target vehicle, filter the S points in the following manner to filter each point in the first set of reference points: The s-th point among the S points is filtered using the following formula:

[0233] Where α is a preset weight value, α∈[0,1], y s Let be the horizontal coordinate of the s-th point, and y be the horizontal coordinate of the s-th point after filtering. Let be the horizontal coordinate of the (s-1)th point after filtering, assuming s equals 1. It is equal to the horizontal coordinate of the s-th point.

[0234] In an exemplary embodiment, the detection module 84 is further configured to, when there are S points in the filtered first reference point set located on one side of the target vehicle, filter the S points in the following manner to filter the points in the filtered first reference point set: perform cubic curve fitting on the S points using the least squares method and determine the corresponding minimum average error; calculate an evaluation parameter value based on the minimum average error; if the evaluation parameter value is greater than or equal to an evaluation error threshold, delete the S points whose lateral distance from the cubic curve fitted by the least squares method exceeds the filtering threshold; if the evaluation parameter value is less than the evaluation error threshold, perform cubic curve fitting on the S points using a random sampling consensus algorithm and delete the S points whose lateral distance from the cubic curve fitted by the random sampling consensus algorithm exceeds the filtering threshold.

[0235] In an exemplary embodiment, the detection module 84 is further configured to calculate the evaluation parameter value using the following formula:

[0236]

[0237] Wherein, Sorce is the evaluation parameter value, leastSquareError is the minimum average error, and accuracyThr is the preset accuracy threshold.

[0238] In an exemplary embodiment, the detection module 84 is further configured to determine the time difference between the third time and the fourth time to obtain a second time difference, wherein the third time is the time corresponding to the image exposure stamp of the target image data, and the fourth time is the acquisition time of the N+M-1th frame; calculate the distance traveled by the target vehicle and the change value of the heading angle within the second time difference to obtain a third distance and a second value; update the cubic curve of the boundary line according to the third distance and the second value to obtain the visual road boundary line.

[0239] In an exemplary embodiment, the detection module 84 is further configured to detect the road boundary line on one side of the target vehicle by: determining multiple points located on one side of the target vehicle from the third reference point set to obtain a fourth reference point set; repeatedly performing the following operations until the fourth reference point set is empty, and determining the cubic curve y = c0 + c1x + c2x corresponding to the road boundary line according to the system state matrix. 2 +c3x 3 c0, c1, c2, and c3: A point is randomly selected from the fourth reference point set, and the selected point is deleted from the fourth reference point set; Based on the Kalman prediction algorithm, the system state matrix and the prediction covariance matrix are calculated according to the third distance, the second value, the reference system state matrix, and the reference prediction covariance matrix; Based on the Kalman update algorithm, the system state matrix and the prediction covariance matrix are updated according to the coordinates of the selected point; The reference system state matrix is ​​updated to the updated system state matrix, and the reference prediction covariance matrix is ​​updated to the updated prediction covariance matrix.

[0240] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0241] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0242] S1, acquire target image data from the visual sensor of the target vehicle and target point cloud data from the radar sensor of the target vehicle;

[0243] S2, detect the road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data.

[0244] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0245] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0246] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0247] S1, acquire target image data from the visual sensor of the target vehicle and target point cloud data from the radar sensor of the target vehicle;

[0248] S2, detect the road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data.

[0249] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0250] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium storing the computer program product, wherein the computer program, when executed by a processor, implements the steps of the methods described in various embodiments of this application.

[0251] Optionally, in this embodiment, the computer program described above can be configured to perform the following steps when executed by a processor:

[0252] S1, acquire target image data from the visual sensor of the target vehicle and target point cloud data from the radar sensor of the target vehicle;

[0253] S2, detect the road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data.

[0254] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0255] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0256] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting road boundary lines, characterized in that, include: Acquire target image data from the visual sensor of the target vehicle and target point cloud data from the radar sensor of the target vehicle; The road boundary lines on both sides of the target vehicle are detected based on the target point cloud data and the target image data.

2. The method according to claim 1, characterized in that, Detecting the road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data includes: A first set of points is determined based on the target point cloud data. This first set comprises N subsets, and the points in the first set are determined based on N+M-1 frames of point cloud data, where N+M-1 frames are consecutive frames. The points in the i-th subset of the N subsets are determined based on point cloud data from the i-th frame to the i+M-1-th frame within the N+M-1 frames. N is an integer greater than or equal to 2. M is an integer greater than or equal to 2, i is an integer greater than or equal to 1 and less than or equal to N, and the target point cloud data includes the N+M-1 frames of point cloud data; The visual road boundary line is determined based on the target image data, and points are taken at equal intervals along the longitudinal direction of the visual road boundary line to obtain a second set of points, wherein the longitudinal direction is parallel to the driving direction of the target vehicle. The road boundary lines on both sides of the target vehicle are detected based on the first set of points and the second set of points.

3. The method according to claim 2, characterized in that, The first set of points is determined based on the target point cloud data, including: The first point set is determined by obtaining the i-th subset among the N subsets in the following manner: The j-th static point set is obtained in the following way to obtain M static point sets, and the i-th subset is determined based on the M static point sets, where j is an integer greater than or equal to 1 and less than or equal to M: When the j-th frame point cloud data in the M-frame point cloud data is obtained, the points in the j-th frame point cloud data are compensated according to the first time and the second time to obtain the j-th point set. The first time is the time when the j-th frame point cloud data is obtained, and the second time is the time corresponding to the point cloud timestamp of the j-th frame point cloud data. The M-frames include the i-th frame to the i+M-1-th frame. Determine the set of static points j from the set of points j.

4. The method according to claim 3, characterized in that, The points in the j-th frame point cloud data are compensated based on the first and second time points, including: Determine the time difference between the first moment and the second moment to obtain the first time difference; Determine the distance the target vehicle travels in the longitudinal direction within the first time difference to obtain a first distance; and Determine the distance the target vehicle travels laterally within the first time difference to obtain a second distance, wherein the lateral direction is perpendicular to the longitudinal direction; and Determine the change in heading angle of the target vehicle within the first time difference to obtain the first value; The points in the j-th frame point cloud data are compensated based on the first distance, the second distance, and the first value.

5. The method according to claim 4, characterized in that, The points in the j-th frame point cloud data are compensated based on the first distance, the second distance, and the first value, including: The a-th point in the j-th frame point cloud data is compensated using the following formula, where a is an integer greater than or equal to 1 and less than or equal to W, and W is equal to the number of points in the j-th frame point cloud data: x=xa*cos(Δheading)+ya*sin(Δheading)-Δx; y=ya*cos(Δheading)-xa*sin(Δheading)-Δy; Where xa is the vertical coordinate of the a-th point, and ya is the horizontal coordinate of the a-th point. Δheading is the first value, Δx is the first distance, Δy is the second distance, x is the longitudinal coordinate of the a-th point after compensation, and y is the lateral coordinate of the a-th point after compensation.

6. The method according to claim 3, characterized in that, Determining the j-th static point set from the j-th point set includes: Determine the absolute velocity of each point in the j-th point set relative to the ground; From the j-th point set, determine the points whose absolute velocity is less than the preset velocity to obtain the reference point set; The j-th static point set is determined based on the reference point set.

7. The method according to claim 6, characterized in that, Determining the j-th static point set based on the reference point set includes: The reference point set is directly determined as the j-th static point set; or Points that meet preset conditions in the reference point set are deleted to obtain the j-th static point set. The points in the j-th static point set are the points in the reference point set that were not deleted. The preset conditions include: points that are not located in a preset region of interest, points with abnormal radar cross section (RCS) and signal-to-noise ratio (SNR) in the preset region of interest, and points within the bounding boxes of visual targets and radar targets.

8. The method according to claim 3, characterized in that, Determining the i-th subset based on the M static point sets includes: Kinematic compensation is performed on the first M-1 static point sets of the M static point sets so that the time corresponding to the compensated M-1 static point sets is synchronized with the time corresponding to the Mth static point set. The compensated set of M-1 static points and the set of M static points are merged. Obtain the target static point set; The points in the target static point set are filtered to obtain the i-th subset.

9. The method according to claim 8, characterized in that, The points in the target static point set are filtered to obtain the i-th subset, which includes: The two regions to the left and right of the target vehicle are divided along the longitudinal direction to obtain multiple regions. Points in the e-th region of these multiple regions are processed in the following way to determine the e-th point set, and the i-th subset is determined based on the obtained E point sets, where e is an integer greater than or equal to 1 and less than or equal to E, and E is the number of regions in the multiple regions: If L points are in the target static point set in the e-th region, the first preset number of points among the L points that are closest to the target vehicle in lateral distance constitute the e-th point set.

10. The method according to claim 9, characterized in that, The i-th subset is determined based on the obtained set of E points, including: Cluster the points in the E point sets and remove the points in the E point sets that do not belong to any cluster to obtain the target point set, wherein the target point set includes the points in the E point sets that belong to the clusters; The points in the e-th region of the plurality of regions are processed in the following way to determine the e-th point set again. The e-th point set is then merged to obtain the i-th subset: when there are U points in the target point set in the e-th region, the second preset number of points among the U points that are closest to the target vehicle in lateral distance constitutes the e-th point set again, wherein the second preset number is less than the first preset number.

11. The method according to claim 2, characterized in that, Determining the visual road boundary line based on the target image data includes: acquiring the cubic curve of the boundary line corresponding to the target image data; kinematically compensating the cubic curve of the boundary line to the acquisition time corresponding to the N+M-1th frame to obtain the visual road boundary line. Detecting the road boundary lines on both sides of the target vehicle based on the first set of points and the second set of points includes: Kinematic compensation processing is performed on the first N-1 subsets in the first point set so that the time corresponding to the compensated N-1 subsets is synchronized with the time corresponding to the Nth subset, wherein the time corresponding to the Nth subset is the acquisition time corresponding to the N+M-1th frame; The compensated N-1 subsets and the Nth subset are merged to obtain the first reference point set; Each point in the first set of reference points is filtered, and the points in the filtered first set of reference points are further filtered to obtain a second set of reference points, wherein the points in the second set of reference points are the points in the filtered first set of reference points that were not deleted. The second set of reference points and the second set of points are merged to obtain a third set of reference points; the road boundary lines on both sides of the target vehicle are detected based on the third set of reference points.

12. The method according to claim 11, characterized in that, Each point in the first set of reference points is filtered, including: If there are S points in the first set of reference points located on one side of the target vehicle, the S points are filtered in the following way to filter each point in the first set of reference points: The s-th point among the S points is filtered using the following formula: y=αy (s-1) +(1-α)y s ; Where α is a preset weight value, α∈[0,1], y s Let y be the horizontal coordinate of the s-th point, and y be the horizontal coordinate of the s-th point after filtering. (s-1) Let y be the horizontal coordinate of the (s-1)th point after filtering. When s equals 1, y (s-1) It is equal to the horizontal coordinate of the s-th point.

13. The method according to claim 11, characterized in that, The points in the first set of reference points after filtering are selected, including: If there are S points in the filtered first reference point set located on one side of the target vehicle, the S points are filtered in the following way to filter the points in the filtered first reference point set: The least squares method is used to perform cubic curve fitting on the S points, and the corresponding minimum average error is determined; the evaluation parameter values ​​are calculated based on the minimum average error. If the evaluation parameter value is greater than or equal to the evaluation error threshold, delete the points among the S points whose horizontal distance from the cubic curve fitted by the least squares method exceeds the filtering threshold. If the evaluation parameter value is less than the evaluation error threshold, the random sampling consensus algorithm is used to perform cubic curve fitting on the S points, and points among the S points whose horizontal distance from the cubic curve obtained by the random sampling consensus algorithm exceeds the filtering threshold are deleted.

14. The method according to claim 13, characterized in that, The evaluation parameter values ​​are calculated based on the minimum average error, including: The evaluation parameter value is calculated using the following formula: Wherein, Sorce is the evaluation parameter value, leastSquareError is the minimum average error, and accuracyThr is the preset accuracy threshold.

15. The method according to claim 11, characterized in that, The boundary line cubic curve is kinematically compensated to the acquisition time corresponding to the N+M-1th frame to obtain the visual road boundary line, including: The time difference between the third and fourth moments is determined to obtain the second time difference, wherein the third moment is the moment corresponding to the image exposure stamp of the target image data, and the fourth moment is the acquisition moment of the N+M-1th frame; Calculate the distance traveled and the change in heading angle of the target vehicle within the second time difference to obtain the third distance and the second value; The boundary line cubic curve is updated based on the third distance and the second value to obtain the visual road boundary line.

16. The method according to claim 15, characterized in that, Detecting the road boundary lines on both sides of the target vehicle based on the third set of reference points includes: The road boundary line on one side of the target vehicle is detected using the following method, in order to detect the road boundary lines on both sides of the target vehicle: From the third set of reference points, multiple points located on one side of the target vehicle are determined to obtain a fourth set of reference points; The following operations are repeated until the fourth reference point set is empty, and the cubic curve y = c0 + c1x + c2x corresponding to the road boundary line is determined according to the system state matrix. 2 +c3x 3 c0, c1, c2, and c3 in the following: Randomly select a point from the fourth reference point set, and delete the selected point from the fourth reference point set; Based on the Kalman prediction algorithm, the system state matrix and the prediction covariance matrix are calculated according to the third distance, the second value, the reference system state matrix, and the reference prediction covariance matrix. Based on the Kalman update algorithm, the system state matrix and the prediction covariance matrix are updated according to the selected point coordinates; The reference system state matrix is ​​updated to the updated system state matrix, and the reference prediction covariance matrix is ​​updated to the updated prediction covariance matrix.

17. A road boundary line detection device, characterized in that, include: The acquisition module is used to acquire target image data from the visual sensor of the target vehicle and target point cloud data from the radar sensor of the target vehicle. The detection module is used to detect the road boundary lines on both sides of the target vehicle based on the target point cloud data and the target image data.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 16.

19. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 16 through the computer program.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 16.

Images