Multi-sensor fusion static road model determination method and device, and storage medium

Abstract

The invention provides a multi-sensor fusion static road model determination method and device and a storage medium, and the method comprises the steps: obtaining the road visual information at a first moment detected by a visual sensor in a vehicle, and the road point cloud data at the first moment detected by a radar sensor; according to the method, the road model in the static environment where the vehicle is located can be constructed by means of multiple sensors carried by the vehicle, the road attribute information and passable area lines of the road where the vehicle is located at the first moment are determined, and then the road passing curve in the static road model is determined from all passable area lines according to the road attribute information. The accuracy of model construction is improved by fusing the information of a plurality of sensors, so that accurate information is provided for the vehicle in real time without depending on a high-precision map, the construction cost of the static road model is greatly reduced, and meanwhile, the flexibility of a vehicle intelligent driving system can be ensured; the problem that the road model has errors due to the fact that the high-precision map is not updated in time is solved.

Description

Technical Field

[0001] The present application relates to the field of automobile intelligent driving technology, and in particular to a method, device and storage medium for determining a static road model using multi-sensor fusion. Background Art

[0002] With the widespread application of intelligent driving technology around the world, the static environment perception capability in the vehicle-mounted intelligent driving system is becoming increasingly important. Therefore, the static environment perception capability plays an increasingly important role in the perception, prediction, decision-making, planning and control modules of the intelligent driving system. Among them, static road model reconstruction is an indispensable part. This technology can help intelligent driving vehicles cope with traffic environments of different complexities and make the intelligent driving system drive vehicles more humanized.

[0003] In most current intelligent driving systems, the construction of static road models needs to rely on the elements provided by high-precision maps. However, there may be realistic problems such as untimely map updates. In addition, this method requires a lot of time and human resources to build map information in advance. At the same time, its performance is greatly affected by the vehicle positioning information. These factors not only lead to high costs, but also greatly reduce the flexibility of the entire intelligent driving system. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, the present application aims to provide a method, device and storage medium for determining a static road model based on multi-sensor fusion, so as to solve the problems of low accuracy, high cost and poor flexibility of static road models in the prior art.

[0005] The present application provides a method for determining a static road model using multi-sensor fusion, the method comprising:

[0006] Acquire road visual information detected by a visual sensor in a vehicle at a first moment, and road point cloud data detected by a radar sensor in the vehicle at a first moment;

[0007] Determine the road attribute information of the road where the vehicle is located at the first moment based on the road visual information at the first moment, and determine the passable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment;

[0008] Based on the road attribute information at the first moment, a road pass curve in the static road model at the first moment is determined from all passable area lines at the first moment.

[0009] Optionally, based on the road attribute information at the first moment, determining a road pass curve in the static road model at the first moment from all passable area lines at the first moment includes:

[0010] Determining a curve coefficient range based on the road geometry type in the road attribute information at the first moment;

[0011] Determine, among all the passable area lines at the first moment, the passable area lines that meet the curve coefficient range as candidate curves;

[0012] The degree of parallelism between the candidate curves is determined, and based on the degree of parallelism between the candidate curves, the road pass curve at the first moment is determined.

[0013] Optionally, determining the drivable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment includes:

[0014] Based on the distance between each target point in the road point cloud data at the first moment, clustering each target point to obtain a first clustering result;

[0015] Clustering each target point in the first clustering result based on the lateral distance between each target point in the first clustering result and the lane line or the roadside line to obtain a second clustering result;

[0016] For each target class in the second clustering result, curve fitting is performed on all target points belonging to the target class to obtain a traversable area line at the first moment.

[0017] Optionally, before determining the drivable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment, the method further includes:

[0018] For each target point in the road point cloud data at the first moment, determine a longitudinal distance between the target point and the vehicle, and a lateral distance between the target point and a nearest lane line or roadside line;

[0019] If the longitudinal distance exceeds a preset longitudinal range, or the lateral distance is greater than a preset minimum lateral threshold, the target point is removed from the road point cloud data at the first moment.

[0020] Optionally, before determining the drivable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment, the method further includes:

[0021] Acquire road point cloud data at a second moment and a posture change amount of the vehicle from the second moment to the first moment, wherein the second moment is a moment before the first moment;

[0022] Based on the posture change amount, converting the road point cloud data at the second moment to the first moment, to obtain the converted point cloud data at the first moment;

[0023] The road point cloud data at the first moment is merged with the converted point cloud data to update the road point cloud data at the first moment.

[0024] Optionally, after determining the road pass curve in the static road model at the first moment from all the passable area lines at the first moment based on the road attribute information at the first moment, the method further includes:

[0025] For each target point in the road point cloud data at the first moment, determining a lateral distance between the target point and the road pass curve at the first moment;

[0026] Based on the lateral distances corresponding to the target points, some target points in the road point cloud data at the first moment are eliminated.

[0027] Optionally, the method further includes:

[0028] Determine, based on the road visual information at the first moment, a lane line point set and a roadside line point set of the road where the vehicle is located at the first moment;

[0029] Based on the lane line point set and the roadside line point set at the first moment, the lane lines and the roadside lines in the static road model at the first moment are determined.

[0030] Optionally, determining the lane line and the roadside line in the static road model at the first moment based on the lane line point set and the roadside line point set at the first moment includes:

[0031] Acquire a lane line point set and a roadside line point set at a second moment, and a posture change amount of the vehicle from the second moment to the first moment, wherein the second moment is a moment before the first moment;

[0032] Based on the posture change amount, the lane line point set and the roadside line point set at the second moment are converted to the first moment to obtain the lane line conversion point set and the roadside point conversion point set at the first moment;

[0033] Merging the lane line point set at the first moment with the lane line conversion point set to update the lane line point set at the first moment, and merging the roadside line point set at the first moment with the roadside point conversion point set to update the roadside line point set at the first moment;

[0034] Based on the updated lane line point set and roadside line point set at the first moment, the lane lines and roadside lines in the static road model at the first moment are determined.

[0035] The present application also provides an electronic device, the electronic device comprising:

[0036] Processor and memory;

[0037] The processor is used to execute the steps of the static road model determination method of multi-sensor fusion provided in any embodiment of the present application by calling the program or instruction stored in the memory.

[0038] An embodiment of the present application also provides a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the steps of the static road model determination method of multi-sensor fusion provided in any embodiment of the present application.

[0039] In summary, the present application proposes a method for determining a static road model by multi-sensor fusion. The method obtains the road visual information detected by the visual sensor at the first moment in the vehicle and the road point cloud data detected by the radar sensor at the first moment, and determines the road attribute information of the road where the vehicle is located at the first moment according to the road visual information at the first moment, and determines the passable area line of the road where the vehicle is located at the first moment according to the road point cloud data at the first moment, and then determines the road traffic curve in the static road model from all the passable area lines at the first moment through the road attribute information at the first moment. The method fuses the road attribute information determined by the visual sensor with the passable area line determined by the radar sensor, and can construct a road model in the static environment where the vehicle is located with the help of multiple sensors carried by the vehicle, and improve the accuracy of model construction by fusing the information of multiple sensors, thereby providing accurate information to the vehicle in real time. The method does not need to rely on high-precision maps, which greatly reduces the construction cost of static road models. At the same time, it can also ensure the flexibility of the vehicle's intelligent driving system and avoid problems such as errors in the road model caused by untimely updates of high-precision maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is a flowchart of a method for determining a static road model by multi-sensor fusion provided in an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of a sensor detection provided by an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of determining a static road model provided in an embodiment of the present application;

[0044] Figure 4 It is a structural schematic diagram of a static road model determination device using multi-sensor fusion provided in an embodiment of the present application;

[0045] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0047] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0048] As mentioned in the background technology, in order to solve the problems in the prior art, the present application proposes a method for determining a static road model by multi-sensor fusion. Figure 1 is a flowchart of a method for determining a static road model using multi-sensor fusion provided in an embodiment of the present application. Figure 1 The method for determining the static road model of multi-sensor fusion specifically includes:

[0049] S110, obtaining road visual information detected by a visual sensor in the vehicle at the first moment, and road point cloud data detected by a radar sensor in the vehicle at the first moment.

[0050] The visual sensor may include a visual sensor mounted in front of the vehicle and a visual sensor mounted in the rear of the vehicle. Specifically, the visual sensor may be used to detect the road on which the vehicle is located, for example, to detect lane lines and roadside lines of the road on which the vehicle is located.

[0051] In the embodiment of the present application, the vehicle can be used as the origin of the coordinate system to establish a vehicle body coordinate system with the front of the vehicle as the positive X direction and the left side of the vehicle as the positive Y direction. In the vehicle body coordinate system, the road visual information detected by the visual sensor can include all visual lane lines and all visual roadside lines.

[0052] For example, Figure 2 is a schematic diagram of a sensor detection provided by an embodiment of the present application, such as Figure 2As shown, there are 8 visual lane lines detected by the visual sensor, including the current left lane line, the current right lane line, the left lane line of the current left lane line, and the right lane line of the current right lane line detected by the forward visual sensor, and the current left lane line, the current right lane line, the left lane line of the current left lane line, and the right lane line of the current right lane line detected by the backward visual sensor. There are 4 visual roadside lines detected by the visual sensor, including the current left roadside line and the current right roadside line detected by the forward visual sensor, and the current left roadside line and the current right roadside line detected by the backward visual sensor.

[0053] The visual lane line and the visual roadside line can be a cubic polynomial, such as y = c0 + c1 × x + c2 × x 2 +c3×x 3 ;

[0054] In the embodiment of the present application, the radar sensor may be a millimeter wave radar mounted on a vehicle. Specifically, the radar sensor may be used to collect point cloud data in each lane of the road where the vehicle is located. The road point cloud data may include multiple target points and information such as the coordinates, size, and relative motion attributes of each target point. For example, Figure 2 As shown, the road point cloud data may include target points detected by the millimeter wave radar.

[0055] S120. Determine the road attribute information of the road where the vehicle is located at the first moment based on the road visual information at the first moment, and determine the passable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment.

[0056] Specifically, after obtaining the road visual information at the first moment, the road lane lines and roadside lines in the road visual information may be processed.

[0057] For example, the 4 visual lane lines detected by the forward visual sensor and the 4 visual lane lines detected by the backward visual sensor are merged according to the position information to obtain 4 merged road lane lines, which are marked as Lane_Current Left Left, Lane_Current Left, Lane_Current Right, and Lane_Current Right Right. At the same time, the 2 visual curb lines detected by the forward visual sensor and the 2 visual curb lines detected by the backward visual sensor can be merged according to the position information to obtain 2 merged road curb lines, which are marked as Curb_Left and Curb_Right.

[0058] Furthermore, the road attribute information can be calculated based on the road lane lines and roadside lines obtained in the above steps. The road attribute information at least includes the road geometry type of the road where the vehicle is located, such as a straight line or a curved line. The road attribute information can also include the effective distance starting point and end point of the road where the vehicle is located, whether there is an intersection ahead, whether the lane lines are parallel, etc.

[0059] For example, the road attribute information Lane_attribute_t0 may be determined based on the curvature of the road lane line, the curvature of the road curb, the length of the road curb, the length of the road lane line, the extension direction of the starting point and the end point of the road lane line, and the like.

[0060] In addition to determining the road attribute information at the first moment based on the information detected by the visual sensor, the passable area line at the first moment can also be determined based on the information detected by the radar sensor. The passable area line can be a road curve representing the passable area of ​​the road where the vehicle is located, which is used to provide a data basis for the decision-making of the intelligent driving system in the vehicle.

[0061] In an embodiment of the present application, considering that there may be some invalid point cloud data in the road point cloud data detected by the radar sensor, in order to ensure the accuracy of the subsequent determination of the passable area line and improve the efficiency of the subsequent determination of the passable area line, the road point cloud data at the first moment may be preprocessed.

[0062] In one example, before determining the passable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment, the following steps are also included:

[0063] Step 11: for each target point in the road point cloud data at the first moment, determine the longitudinal distance between the target point and the vehicle, and the lateral distance between the target point and the closest lane line or roadside line;

[0064] Step 12: If the longitudinal distance exceeds the preset longitudinal range, or the lateral distance is greater than the preset minimum lateral threshold, the target point is removed from the road point cloud data at the first moment.

[0065] Among them, in step 11, the longitudinal distance between the target point and the vehicle can be determined based on the longitudinal coordinate of the target point. The lateral distance between the target point and the closest lane line or roadside line can be determined based on the horizontal coordinate of the target point and the lateral position of the closest lane line or roadside line. For example, the lateral distance between the target point and all lane lines and roadside lines can be calculated, and then the smallest lateral distance can be selected. It should be noted that the lane line and roadside line here can be determined based on the road visual information detected by the visual sensor.

[0066] In step 12, the preset longitudinal range may be a preset longitudinal distance range, and the preset minimum transverse threshold may be a preset minimum transverse distance value for eliminating target points.

[0067] Specifically, target points whose longitudinal distance from the vehicle exceeds a preset longitudinal range may be removed to filter out points that are too far from the vehicle in the longitudinal direction. Also, target points whose lateral distance from the nearest lane line or roadside line is greater than a preset minimum lateral threshold may be removed to filter out points that are too far from the lane line or roadside line in the lateral direction.

[0068] Through the above steps 11 and 12, the road point cloud data at the first moment can be preprocessed. Through preprocessing, invalid target points that are too far away from the vehicle in the longitudinal direction or too far away from the lane line or the roadside line in the lateral direction can be eliminated. While ensuring the accuracy of the subsequent determination of the passable area line, the efficiency of the subsequent determination of the passable area line can also be improved.

[0069] In an embodiment of the present application, taking into account the possibility that the radar sensor may miss collecting points in a certain area, in order to further improve the accuracy of the subsequent determination of the passable area line, the point cloud data of the second moment can be fused with the first moment as the new point cloud data of the first moment.

[0070] In one example, before determining the passable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment, the following steps are also included:

[0071] Step 13, obtaining the road point cloud data at the second moment and the posture change of the vehicle from the second moment to the first moment, wherein the second moment is the moment before the first moment;

[0072] Step 14: converting the road point cloud data at the second moment to the first moment based on the posture change amount to obtain the converted point cloud data at the first moment;

[0073] Step 15: Merge the road point cloud data at the first moment with the converted point cloud data to update the road point cloud data at the first moment.

[0074] In step 13, the road point cloud data at the second moment and the vehicle posture change from the second moment to the first moment may be obtained; the posture change may include the position difference between the second moment and the first moment. The posture change may be determined based on the odometer information, which may be collected by an inertial measurement unit sensor mounted on the vehicle.

[0075] It should be noted that the road point cloud data at the second moment may be the road point cloud data obtained after fusion at the second moment, or may be the road point cloud data after fusion at the second moment and after some points are removed according to the road passage curve. It is understandable that if the second moment is the first moment, the road point cloud data at the second moment may be the road point cloud data after some points are removed according to the road passage curve at the second moment.

[0076] Specifically, in step 14, the road point cloud data at the second moment can be converted to the first moment through the posture change, that is, based on the posture change, each point in the road point cloud data at the second moment is predicted, and its position at the first moment is predicted to obtain the converted point cloud data at the first moment.

[0077] Furthermore, in step 15, the road point cloud data at the first moment and the converted point cloud data may be merged, and the merged data may be used as the road point cloud data at the first moment.

[0078] Through the above steps 13 to 15, the road point cloud data at the second moment can be fused with the road point cloud data at the first moment, avoiding the situation where errors in the subsequent determination of the passable area line due to missed collection by the radar sensor, thereby ensuring the accuracy of the subsequent determination of the passable area line. In addition, in this process, the data collected by the inertial measurement unit sensor is also fused, which can further improve the accuracy of model construction.

[0079] In an embodiment of the present application, based on the road point cloud data at the first moment, the passable area line of the road where the vehicle is located at the first moment is determined. Specifically, the road point cloud data at the first moment can be clustered to obtain the passable area line.

[0080] In a specific implementation, determining the drivable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment includes the following steps:

[0081] Step 21: clustering the target points based on the distances between the target points in the road point cloud data at the first moment to obtain a first clustering result;

[0082] Step 22: clustering each target point in the first clustering result based on the lateral distance between each target point in the first clustering result and the lane line or the roadside line to obtain a second clustering result;

[0083] Step 23: For each target class in the second clustering result, curve fitting is performed on all target points belonging to the target class to obtain the traversable area line at the first moment.

[0084] In step 21, for the road point cloud data at the first moment, clustering can be performed based on the distances (such as Euclidean distances) between the target points therein to obtain a first clustering result. For example, DBSCAN density clustering can be used, and a clustering threshold Dis_min can be set to classify the target points whose distances are less than the clustering threshold into one category.

[0085] After the first clustering is performed according to the distance between each target point, considering that in the point cloud data collected by the radar sensor, the number of points will decrease as the distance increases, which will result in the points at a long distance not being classified in the first clustering process due to their small number, etc. Therefore, a second clustering can be performed to solve the problem of sparse points at a long distance not being classified.

[0086] Specifically, in step 22, based on the first clustering result, the lateral distance between each target point and each lane line or roadside line can be determined for the discrete target points that have not been clustered, and then the nearest lane line or roadside line is used as a label, and the target points whose lateral distance to the same lane line or roadside line meets the preset lateral range are determined as one category and labeled, thereby obtaining the second clustering result.

[0087] Furthermore, in step 23, curve fitting can be performed according to each target class obtained by clustering and all target points under each target class, such as performing cubic polynomial fitting on target points with the same label, and then the passable area line corresponding to each target class can be obtained, such as, it can be expressed by the coefficient equation: y = c0 + c1 × x + c2 × x 2 +c3×x 3 .

[0088] Through the above steps 21 to 23, the road point cloud data at the first moment can be processed by secondary clustering to obtain the passable area line at the first moment, avoiding clustering based only on the distance between target points, which results in the inability to classify points at a longer distance. At the same time, the first clustering can be performed based on the distance between target points to obtain the classification of most points, and then the second clustering can be performed based on the lateral distance between the lane line or the roadside line to obtain the classification of the remaining small number of points, thereby achieving the purpose of improving the overall clustering efficiency and ensuring the accuracy of clustering.

[0089] S130. Based on the road attribute information at the first moment, determine a road pass curve in the static road model at the first moment from all passable area lines at the first moment.

[0090] Specifically, after obtaining the road attribute information and all passable area lines at the first moment, all passable area lines at the first moment can be filtered in combination with the road attribute information at the first moment to obtain the passable area lines that meet the road attribute information and output them as road pass curves in the static road model at the first moment.

[0091] The static road model may be a map model describing the static environment of the road where the vehicle is located. The road traffic curve may be a line describing the area where the vehicle can pass in the road where the vehicle is located.

[0092] In a specific implementation, based on the road attribute information at the first moment, determining the road pass curve in the static road model at the first moment from all the passable area lines at the first moment includes the following steps:

[0093] Step 31: Determine a curve coefficient range based on the road geometry type in the road attribute information at the first moment;

[0094] Step 32: Determine the passable area lines that meet the curve coefficient range among all the passable area lines at the first moment as candidate curves;

[0095] Step 33: Determine the parallelism between the candidate curves, and determine the road pass curve at the first moment based on the parallelism between the candidate curves.

[0096] In step 31, the curve coefficient range that the road traffic curve should satisfy can be determined according to the road geometry type in the road attribute information at the first moment. For example, if the road geometry type is a straight line, the corresponding curve coefficient range can be: the quadratic term coefficient is less than a preset coefficient threshold, that is, c2 is less than a preset coefficient threshold c2_thr.

[0097] Furthermore, in step 32, the passable area lines that meet the curve coefficient range may be screened out from all the passable area lines as candidate curves.

[0098] In addition, considering that the number of road pass curves is generally 2, and the two road pass curves are highly parallel in theory, the parallelism between any two candidate curves can be calculated in step 33. The parallelism can describe the degree of parallelism between the two candidate curves, for example, it can be represented by the angle difference between the two candidate curves. The smaller the angle difference, the higher the parallelism.

[0099] Furthermore, two candidate curves with the highest degree of parallelism may be selected and output as the road pass curves at the first moment.

[0100] Through the above steps 31 to 33, the road geometry type detected by the visual sensor can be used to filter out the passable area lines that meet the road geometry type, and further selection can be performed in combination with parallelism, so as to ensure the accuracy of the final output road pass curve.

[0101] In an embodiment of the present application, in order to improve the accuracy of constructing a static road model at the next moment, the road point cloud data at the first moment can also be screened for the second time based on the road traffic curve output at the first moment, so that when the point cloud data is fused at the next moment, the point cloud data after the second screening at the first moment can be fused to ensure the accuracy of the static road model at the next moment.

[0102] In some implementations, after determining the road pass curve in the static road model at the first moment from all passable area lines at the first moment based on the road attribute information at the first moment, the following steps are further included:

[0103] Step 34: for each target point in the road point cloud data at the first moment, determine the lateral distance between the target point and the road traffic curve at the first moment;

[0104] Step 35: based on the lateral distances corresponding to the target points, some target points in the road point cloud data at the first moment are eliminated.

[0105] In step 34, the lateral distance between each target point in the road point cloud data at the first moment and the curve can be calculated according to the road passage curve determined at the first moment.

[0106] Furthermore, it can be determined whether the lateral distance corresponding to each target point is greater than a preset distance threshold. If so, it means that the distance between the target point and the road passage curve is large, and the target point is not a point that fits the road passage curve, that is, the target point does not belong to the category of the road passage curve. Therefore, the target point can be removed from the road point cloud data at the first moment.

[0107] Through the above steps 34 and 35, after determining the road passage curve at the first moment, the road point cloud data at the first moment can be screened again in combination with the road passage curve to improve the accuracy of constructing a static road model at the next moment.

[0108] In an embodiment of the present application, in addition to the road traffic curve, the static road model determined at the first moment may also have lane lines and roadside lines. For example, the lane lines and roadside lines in the static road model at the first moment may be determined based on the information detected by the visual sensor.

[0109] In a specific implementation, the method provided in the embodiment of the present application further includes the following steps:

[0110] Step 41: determining a lane line point set and a roadside line point set of the road where the vehicle is located at the first moment based on the road visual information at the first moment;

[0111] Step 42: Based on the lane line point set and the roadside line point set at the first moment, determine the lane line and the roadside line in the static road model at the first moment.

[0112] Among them, in step 41, all visual lane lines and all visual roadside lines in the road visual information at the first moment can be discretized to discretize the visual lane lines and visual roadside lines into separate points to obtain a lane line point set and a roadside line point set.

[0113] Furthermore, in step 42, the lane lines and the roadside lines in the static road model at the first moment can be obtained through the lane line point set and the roadside line point set at the first moment.

[0114] For example, each point in the lane line point set may be merged into four lane lines according to the position information, and each point in the roadside line point set may be merged into two roadside lines according to the position information.

[0115] With respect to the above step 42, in one example, based on the lane line point set and the roadside line point set at the first moment, determining the lane line and the roadside line in the static road model at the first moment includes the following steps:

[0116] Step 421, obtaining a lane line point set and a roadside line point set at a second moment, and a posture change amount of the vehicle from the second moment to the first moment, wherein the second moment is a moment before the first moment;

[0117] Step 422: based on the posture change amount, the lane line point set and the roadside line point set at the second moment are converted to the first moment, so as to obtain the lane line conversion point set and the roadside point conversion point set at the first moment;

[0118] Step 423: merge the lane line point set at the first moment with the lane line conversion point set to update the lane line point set at the first moment, and merge the roadside line point set at the first moment with the roadside point conversion point set to update the roadside line point set at the first moment;

[0119] Step 424: Determine the lane lines and roadside lines in the static road model at the first moment based on the updated lane line point set and roadside line point set at the first moment.

[0120] Specifically, in step 421, the lane line point set and the roadside line point set at the second moment, as well as the change in the vehicle's posture from the second moment to the first moment, may also be obtained. The lane line point set and the roadside line point set at the second moment may be fused point sets.

[0121] Furthermore, in step 422, the lane line point set and the roadside point set at the second moment can be converted to the first moment according to the posture change, that is, according to the posture change, the position of each point in the lane line point set and the roadside point set at the second moment at the first moment is predicted to obtain the lane line conversion point set and the roadside point conversion point set at the first moment.

[0122] Furthermore, in step 423, the lane line point set at the first moment and the lane line conversion point set can be merged, and the roadside line point set at the first moment and the roadside point conversion point set can be merged, and the merged point sets are respectively used as the new lane line point set and roadside line point set at the first moment.

[0123] Furthermore, in step 424, the lane line point set at the first moment may be fitted to obtain the lane line, and the roadside line point set at the first moment may be fitted to obtain the roadside line.

[0124] Through the above example, the data at the second moment can be fused with the data at the first moment, so that the fused data can be used as the data at the first moment to determine the lane line and the roadside line, avoiding the omission of some points caused by sensor collection errors, and further ensuring the accuracy of the static road model.

[0125] The method for determining a static road model by multi-sensor fusion provided in an embodiment of the present application obtains the road visual information detected by the visual sensor in the vehicle at the first moment and the road point cloud data detected by the radar sensor at the first moment, and determines the road attribute information of the road where the vehicle is located at the first moment according to the road visual information at the first moment, and determines the passable area line of the road where the vehicle is located at the first moment according to the road point cloud data at the first moment, and then determines the road traffic curve in the static road model from all the passable area lines at the first moment through the road attribute information at the first moment. The method fuses the road attribute information determined by the visual sensor with the passable area line determined by the radar sensor, and can construct a road model in the static environment where the vehicle is located with the help of multiple sensors carried by the vehicle, and improves the accuracy of model construction by fusing information from multiple sensors, thereby providing accurate information to the vehicle in real time. The method does not need to rely on high-precision maps, greatly reducing the construction cost of static road models. At the same time, it can also ensure the flexibility of the vehicle's intelligent driving system and avoid problems such as errors in the road model caused by untimely updates of high-precision maps.

[0126] For example, Figure 3 is a schematic diagram of determining a static road model provided in an embodiment of the present application, such as Figure 3As shown, first, the visual lane lines and roadside lines detected by the visual sensor can be processed, and then the road attributes can be calculated, and the millimeter wave point cloud can be preprocessed to eliminate invalid points; then, the vehicle running status can be recorded, that is, the vehicle posture can be recorded, so as to subsequently determine the change in the vehicle posture between two moments.

[0127] Furthermore, it can be determined whether the first moment has been initialized, that is, whether it is not the first moment for constructing the static road model. If so, the historical data is transformed (that is, the data of the second moment is converted to the first moment), and the point cloud data is merged, and point cloud screening and clustering are performed. If not, point cloud screening and clustering are performed directly.

[0128] After completing point cloud screening (i.e. preprocessing) and clustering, polynomial fitting can be performed to obtain all traversable area lines, and then filtered according to road attributes to output the road traffic curve of the static road model at the first moment; finally, the point cloud is screened again according to the road traffic curve.

[0129] This method can combine front and rear vision and radar information for all-round fusion and build a road model map in real time. It can obtain the equations of lane lines, roadside lines, and road traffic curves at a certain moment in history at any time without the need for high-precision maps, thus reducing costs and providing better flexibility.

[0130] Figure 4 4 is a schematic diagram of a static road model determination device for multi-sensor fusion provided in an embodiment of the present application. The device includes an information acquisition module 410, an information processing module 420 and a model construction module 430, wherein:

[0131] An information acquisition module 410 is used to acquire road visual information detected by a visual sensor in a vehicle at a first moment, and road point cloud data detected by a radar sensor in the vehicle at a first moment;

[0132] The information processing module 420 is used to determine the road attribute information of the road where the vehicle is located at the first moment based on the road visual information at the first moment, and determine the passable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment;

[0133] The model building module 430 is used to determine the road pass curve in the static road model at the first moment from all the passable area lines at the first moment based on the road attribute information at the first moment.

[0134] Based on the above implementations, optionally, the model building module 430 is specifically used to:

[0135] Based on the road geometry type in the road attribute information at the first moment, determine the curve coefficient range; among all the passable area lines at the first moment, the passable area lines that meet the curve coefficient range are determined as candidate curves; determine the parallelism between the candidate curves, and determine the road pass curve at the first moment based on the parallelism between the candidate curves.

[0136] On the basis of the above-mentioned embodiments, optionally, the information processing module 420 includes a clustering unit, wherein the clustering unit is used to cluster each target point based on the distance between each target point in the road point cloud data at the first moment to obtain a first clustering result; cluster each target point in the first clustering result based on the lateral distance between each target point and the lane line or the roadside line to obtain a second clustering result; for each target class in the second clustering result, curve fitting is performed on all target points belonging to the target class to obtain the passable area line at the first moment.

[0137] On the basis of the above-mentioned embodiments, optionally, the information processing module 420 also includes a preprocessing unit, wherein the preprocessing unit is used to determine, for each target point in the road point cloud data at the first moment, the longitudinal distance between the target point and the vehicle, and the lateral distance between the target point and the nearest lane line or roadside line; if the longitudinal distance exceeds a preset longitudinal range, or the lateral distance is greater than a preset minimum lateral threshold, the target point is removed from the road point cloud data at the first moment.

[0138] On the basis of the above-mentioned embodiments, optionally, the information processing module 420 also includes a point cloud fusion unit, wherein the point cloud fusion unit is used to obtain road point cloud data at a second moment and the change in posture of the vehicle from the second moment to the first moment; based on the change in posture, the road point cloud data at the second moment is converted to the first moment to obtain the converted point cloud data at the first moment; and the road point cloud data at the first moment is merged with the converted point cloud data to update the road point cloud data at the first moment.

[0139] On the basis of the above-mentioned implementation modes, optionally, the model building module 430 and the information processing module 420 are also used to determine, for each target point in the road point cloud data at the first moment, a lateral distance between the target point and the road traffic curve at the first moment; and to eliminate some target points in the road point cloud data at the first moment based on the lateral distances corresponding to each target point.

[0140] On the basis of the above-mentioned implementation modes, optionally, the information processing module 420 is also used to determine the lane line point set and the roadside line point set of the road where the vehicle is located at the first moment based on the road visual information at the first moment; the model construction module 430 is also used to determine the lane lines and roadside lines in the static road model at the first moment based on the lane line point set and the roadside line point set at the first moment.

[0141] On the basis of the above-mentioned implementation modes, optionally, the model building module 430 is also used to obtain the lane line point set and the roadside point set at the second moment, and the posture change of the vehicle from the second moment to the first moment; based on the posture change, the lane line point set and the roadside point set at the second moment are converted to the first moment to obtain the lane line conversion point set and the roadside point conversion point set at the first moment; the lane line point set and the lane line conversion point set at the first moment are merged to update the lane line point set at the first moment, and the roadside point set and the roadside point conversion point set at the first moment are merged to update the roadside point set at the first moment; based on the updated lane line point set and roadside point set at the first moment, the lane lines and roadside lines in the static road model at the first moment are determined.

[0142] The device for determining a static road model using multi-sensor fusion provided in the embodiment of the present application can execute the steps in the method for determining a static road model using multi-sensor fusion provided in the method embodiment of the present application, and the execution steps and beneficial effects are not repeated here.

[0143] Figure 5 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 400 includes one or more processors 401 and a memory 402 .

[0144] The processor 401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0145] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may run the program instructions to implement the static road model determination method for multi-sensor fusion of any embodiment of the present application described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0146] In one example, the electronic device 400 may further include: an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 403 may include, for example, a keyboard, a mouse, etc. The output device 404 may output various information to the outside, including early warning information, braking force, etc. The output device 404 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0147] Of course, to simplify, Figure 5 Only some of the components related to the present application in the electronic device 400 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 400 may also include any other appropriate components.

[0148] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the static road model determination method of multi-sensor fusion provided in any embodiment of the present application.

[0149] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0150] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the static road model determination method of multi-sensor fusion provided in any embodiment of the present application.

[0151] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0152] It should be noted that the terms used in this application are only for describing specific embodiments, rather than limiting the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0153] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0154] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and its core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression and the objective existence of infinite specific structures, ordinary technicians in this technical field can make several improvements, modifications or changes without departing from the principles of this application, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the concept and technical solution of the invention to other occasions without improvement, should be regarded as the scope of protection of this application.

Claims

1. A method for determining a static road model using multi-sensor fusion, characterized in that: include: Acquire road visual information detected by a visual sensor in a vehicle at a first moment, and road point cloud data detected by a radar sensor in the vehicle at a first moment; Determine the road attribute information of the road where the vehicle is located at the first moment based on the road visual information at the first moment, and determine the passable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment; Based on the road attribute information at the first moment, a road pass curve in the static road model at the first moment is determined from all passable area lines at the first moment.

2. The method according to claim 1, characterized in that: Based on the road attribute information at the first moment, determining a road pass curve in the static road model at the first moment from all passable area lines at the first moment includes: Determining a curve coefficient range based on the road geometry type in the road attribute information at the first moment; Determine, among all the passable area lines at the first moment, the passable area lines that meet the curve coefficient range as candidate curves; The degree of parallelism between the candidate curves is determined, and based on the degree of parallelism between the candidate curves, the road pass curve at the first moment is determined.

3. The method according to claim 1, characterized in that Determining a drivable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment includes: Based on the distance between each target point in the road point cloud data at the first moment, clustering each target point to obtain a first clustering result; Clustering each target point in the first clustering result based on the lateral distance between each target point in the first clustering result and the lane line or the roadside line to obtain a second clustering result; For each target class in the second clustering result, curve fitting is performed on all target points belonging to the target class to obtain a traversable area line at the first moment.

4. The method according to claim 1, characterized in that: Before determining the drivable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment, the method further includes: For each target point in the road point cloud data at the first moment, determine a longitudinal distance between the target point and the vehicle, and a lateral distance between the target point and a nearest lane line or roadside line; If the longitudinal distance exceeds a preset longitudinal range, or the lateral distance is greater than a preset minimum lateral threshold, the target point is removed from the road point cloud data at the first moment.

5. The method according to claim 1, characterized in that Before determining the drivable area line of the road where the vehicle is located at the first moment based on the road point cloud data at the first moment, the method further includes: Acquire road point cloud data at a second moment and a posture change amount of the vehicle from the second moment to the first moment, wherein the second moment is a moment before the first moment; Based on the posture change amount, converting the road point cloud data at the second moment to the first moment, to obtain the converted point cloud data at the first moment; The road point cloud data at the first moment is merged with the converted point cloud data to update the road point cloud data at the first moment.

6. The method according to claim 5, characterized in that After determining the road pass curve in the static road model at the first moment from all the passable area lines at the first moment based on the road attribute information at the first moment, the method further includes: For each target point in the road point cloud data at the first moment, determining a lateral distance between the target point and the road pass curve at the first moment; Based on the lateral distances corresponding to the target points, some target points in the road point cloud data at the first moment are eliminated.

7. The method according to claim 1, characterized in that The method further comprises: Determine, based on the road visual information at the first moment, a lane line point set and a roadside line point set of the road where the vehicle is located at the first moment; Based on the lane line point set and the roadside line point set at the first moment, the lane lines and the roadside lines in the static road model at the first moment are determined.

8. The method according to claim 7, characterized in that Determining the lane line and the roadside line in the static road model at the first moment based on the lane line point set and the roadside line point set at the first moment includes: Acquire a lane line point set and a roadside line point set at a second moment, and a posture change amount of the vehicle from the second moment to the first moment, wherein the second moment is a moment before the first moment; Based on the posture change amount, the lane line point set and the roadside line point set at the second moment are converted to the first moment to obtain the lane line conversion point set and the roadside point conversion point set at the first moment; Merging the lane line point set at the first moment with the lane line conversion point set to update the lane line point set at the first moment, and merging the roadside line point set at the first moment with the roadside point conversion point set to update the roadside line point set at the first moment; Based on the updated lane line point set and roadside line point set at the first moment, the lane lines and roadside lines in the static road model at the first moment are determined.

9. An electronic device, characterized in that: The electronic device comprises: Processor and memory; The processor is used to execute the steps of the static road model determination method of multi-sensor fusion as described in any one of claims 1 to 8 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, which enables a computer to execute the steps of the static road model determination method based on multi-sensor fusion as claimed in any one of claims 1 to 8.

Images