A multi-source lane line fusion method, device and storage medium

Abstract

The application discloses a multi-source lane line fusion method, equipment and a storage medium. The method comprises the following steps: collecting environment information collected by a plurality of sensors on a target vehicle at a current time, and determining a plurality of lane line point sets of the sensors at the current time; determining at least one target point set based on the plurality of lane line point sets of the sensors at the current time, wherein the target point sets respectively belong to different lane lines, and the target point set comprises points in a lane line point set corresponding to at least one sensor; and obtaining a fusion lane line of the target vehicle at the current time based on the target point sets and a historical fusion lane line, wherein the historical fusion lane line is a fusion lane line of the target vehicle at a historical time. Through the above method, the application can realize accurate fusion of multi-source lane lines.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a multi-source lane line fusion method, device and storage medium. Background Technology

[0002] Lane keeping systems primarily rely on lane markings for control. For example, when a vehicle changes lanes, the lane markings are crucial in determining whether or not to change lanes. As the requirements for the reliability and comfort of driving systems increase, people are placing higher demands on the accuracy of lane marking detection.

[0003] Therefore, accurate detection of lane markings is of great significance. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a multi-source lane line fusion method, device, and storage medium that can achieve accurate fusion of multi-source lane lines, so that the fused lane lines conform to the actual lane lines.

[0005] To address the aforementioned technical problems, this application provides a multi-source lane line fusion method, comprising: utilizing environmental information collected by several sensors on a target vehicle at the current moment to determine several lane line point sets of each sensor at the current moment; determining at least one target point set based on the several lane line point sets of each sensor at the current moment, wherein each target point set belongs to a different lane line, and the target point set contains points from the lane line point set corresponding to at least one sensor; and obtaining the fused lane line of the target vehicle at the current moment based on each target point set and the historical fused lane line, wherein the historical fused lane line is the fused lane line of the target vehicle at a historical moment.

[0006] The plurality of sensors include at least one first sensor and at least one second sensor, wherein the first sensor is located at a first position of the target vehicle and the second sensor is located at a second position of the target vehicle, and the first position and the second position are different.

[0007] The process of determining at least one target point set based on a set of lane line points from each sensor at the current time includes: determining a reference sensor from the set of sensors, and using the set of lane line points from the reference sensor at the current time as each target point set; for each target point set, selecting a set of lane line points belonging to the same lane line as the target point set from the set of lane line points from each of the remaining sensors, and adding the points from the selected set of lane line points to the target point set, wherein the remaining sensors are the sensors other than the reference sensor among the sensors.

[0008] Specifically, selecting a set of lane line points belonging to the same lane line as the target point set from the several lane line point sets of each remaining sensor includes: for each remaining sensor, selecting a set of lane line points that meets a first preset requirement from the several lane line point sets corresponding to the remaining sensor; the first preset requirement includes at least one of the slope difference reaching a first preset range and the distance reaching a second preset range.

[0009] Specifically, based on each set of target points and historical fused lane lines, the fused lane line of the target vehicle at the current moment is obtained, including: obtaining the predicted lane line at the current moment based on the historical fused lane lines and the driving parameters of the target vehicle; wherein the driving parameters include at least one of vehicle speed, yaw angle, time interval, and distance interval, the time interval is the interval between the current moment and the historical moment, and the distance interval is the driving distance corresponding to the time interval; obtaining the detected lane line at the current moment based on each set of target points; and obtaining the fused lane line at the current moment based on the predicted lane line and the detected lane line.

[0010] The process involves multiple fused lane lines, both predicted and detected. Based on historical fused lane lines and the target vehicle's driving parameters, a predicted lane line for the current moment is obtained, including: for each historical fused lane line, a Kalman filter algorithm is used to obtain a predicted lane line for the current moment based on the target vehicle's driving parameters and the historical fused lane lines; and / or, based on the predicted and detected lane lines, a fused lane line for the current moment is obtained, including: determining multiple lane line pairs, each lane line pair including a predicted lane line and a detected lane line belonging to the same lane line; for each lane line pair, a Kalman filter algorithm is used to obtain a fused lane line for the current moment based on the predicted and detected lane lines of the lane line pair.

[0011] This involves using environmental information collected by several sensors on the target vehicle at the current moment to determine a set of lane line points for each sensor at the current moment, including: for each sensor, using the environmental information collected by the sensor at the current moment to perform lane line detection, obtaining several lane line regions corresponding to the sensor; and determining the set of lane line points belonging to the lane line from each lane line region.

[0012] Specifically, determining the set of lane line points belonging to the lane lines from each lane line region includes: sampling each lane line region to obtain several points corresponding to the lane line region; performing clustering processing on the several points corresponding to each lane line region to obtain the set of lane line points corresponding to each lane line region; and performing filtering processing on each set of lane line points to obtain the filtered set of lane line points.

[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions; and the processor is used to execute the program instructions stored in the memory to implement any of the above methods.

[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing program instructions, which can be executed to implement the above-mentioned method.

[0015] The above scheme determines at least one set of target points based on a set of lane line points from each sensor at the current time. Then, based on each set of target points and historical fused lane lines, the fused lane line for the target vehicle at the current time is obtained. Different sets of target points belong to different lane lines, and each lane line's target point set contains points from at least one sensor's corresponding lane line point set. It is understood that this scheme can determine fused lane lines from several sensors based on target point sets (which fuse points from the lane line point sets corresponding to each sensor), ensuring that the obtained fused lane lines conform to the actual lane lines and achieving accurate fusion of multi-source lane lines. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an embodiment of the multi-source lane line fusion method provided in this application;

[0017] Figure 2 This is a schematic diagram of the sensor layout provided in this application;

[0018] Figure 3 yes Figure 1 The flowchart of step S13 shown is a schematic diagram of one embodiment.

[0019] Figure 4 This is a schematic diagram of the lane line coordinate system provided in this application;

[0020] Figure 5 yes Figure 3 The flowchart of step S33 shown is a schematic diagram of one embodiment.

[0021] Figure 6 This is a schematic diagram of the framework of an embodiment of the multi-source lane line fusion device provided in this application;

[0022] Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;

[0023] Figure 8 This is a schematic diagram of the structure of the computer-readable storage medium provided in this application. Detailed Implementation

[0024] To make the purpose, technical solution and effects of this application clearer and more explicit, the following describes this application in further detail with reference to the accompanying drawings and embodiments.

[0025] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multi-source lane line fusion method provided in this application. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily reflect that outcome. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment includes:

[0027] S11: Using the environmental information collected by several sensors on the target vehicle at the current moment, determine the set of several lane line points of each sensor at the current moment.

[0028] This embodiment is used to obtain the fused lane line of a target vehicle at the current moment based on the target point set (points that are fused from the lane line point set corresponding to several sensors) and the historical fused lane lines. The fused lane line consists of lane lines corresponding to several sensors, and each fused lane line corresponds to an actual lane line.

[0029] It should be noted that this application uses several sensors to collect current environmental information and determines several lane line point sets of each sensor at the current time. Therefore, for each actual lane line, a set of lane line points corresponding to each sensor will be determined. For each sensor, each set of lane line points corresponds to one lane line. Therefore, it is necessary to fuse the lane lines (the set of points corresponding to each lane line) corresponding to each sensor to obtain the fused lane line of the target vehicle with respect to several sensors. The multi-source lane line fusion described in this paper refers to fusing the lane lines corresponding to several sensors respectively.

[0030] For example, if the target vehicle is in an environment with two lane lines, X and Y, then by using the environmental information containing the actual lane lines X and Y collected by sensors A and B on the target vehicle at the current moment, we can determine the two lane line point sets A1 (corresponding to actual lane line X) and A2 (corresponding to actual lane line Y) for sensor A at the current moment, and the two lane line point sets B1 (corresponding to actual lane line X) and B2 (corresponding to actual lane line Y) for sensor B at the current moment. Then, based on the lane information collected by sensor A... Given line point sets A1 and A2, and lane line point sets B1 and B2 from sensor B, determine two target point sets (e.g., C1 and C2, corresponding to actual lane lines X and Y, respectively). Target point set C1 contains points from lane line point set A1 and point set B1, and target point set C2 contains points from lane line point set A2 and point set B2. After determining the two target point sets, based on each target point set and the historical fused lane lines, obtain the fused lane lines (2 lines, corresponding to actual lane lines X and Y, respectively) for the target vehicle at the current time.

[0031] The environmental information includes road information (including lane line information). The term "several sensors" as used herein refers to at least two sensors, which may be positioned at the same or different locations on the target vehicle to collect environmental information from the same or different directions. These sensors may be of the same type (e.g., all vision sensors or LiDAR), or they may be of different types (e.g., including both vision sensors and LiDAR).

[0032] In some embodiments, the plurality of sensors include at least one first sensor and at least one second sensor. The first sensor is located at a first position relative to the target vehicle, and the second sensor is located at a second position relative to the target vehicle. The first position and the second position are different. The specific positional information corresponding to the first position and the second position can be determined according to the actual situation, and is not specifically limited here. For example, as shown... Figure 2 As shown, Figure 2 This is a schematic diagram of the sensor layout provided in this application. During the target vehicle's movement, lane line information in front of the target vehicle is beneficial for its straight-line travel, while lane line information behind the target vehicle is beneficial for its decision on whether to change lanes. Therefore, as... Figure 2 As shown, the first sensor can be placed in front of the target vehicle to collect environmental information in front of the target vehicle, and the second sensor can be placed behind the target vehicle to collect environmental information behind the vehicle.

[0033] In this embodiment, for each sensor, the environmental information collected by the sensor at the current moment can be used to detect lane lines, obtain several lane line regions corresponding to the sensor, and then determine the set of lane line points belonging to the lane lines from each lane line region.

[0034] In one embodiment, relevant algorithms or lane detection models can be used to detect lane lines in the environmental information to obtain lane line regions containing lane lines. Alternatively, in another embodiment, a lane line semantic segmentation model can be used to segment the regions corresponding to lane lines, obtaining several lane line regions corresponding to each sensor.

[0035] In one embodiment, after determining the lane line regions corresponding to each sensor, it is necessary to determine the set of lane line points belonging to the lane lines from each lane line region, so as to determine the set of target points based on the set of points of each lane line.

[0036] For example, after determining the lane line regions corresponding to each sensor, each lane line region can be sampled to obtain several points corresponding to each lane line region; then, the several points corresponding to each lane line region are clustered (for example, by traversing the several points obtained from the sampling of each sensor) to obtain the set of lane line points corresponding to each lane line region. In order to reduce the influence of points with excessively large angles on subsequent processing, the set of lane line points corresponding to each lane line region can also be filtered to obtain the filtered set of lane line points.

[0037] Taking a visual sensor as an example, after the sensor acquires an image of the current environment, it can use relevant algorithms or lane line semantic segmentation models to detect lane lines in the image, determine several lane line regions corresponding to each sensor, where each region corresponds to an actual lane line, and then perform sampling, clustering and filtering on each lane line region to obtain a set of lane line points.

[0038] S12: Based on several lane line point sets of each sensor at the current time, determine at least one target point set, wherein each target point set belongs to a different lane line, and the target point set contains points from the lane line point set corresponding to at least one sensor.

[0039] In one embodiment, a reference sensor is first determined from a plurality of sensors, and several sets of lane line points from the reference sensor at the current time are respectively used as target point sets. For each target point set, a set of lane line points belonging to the same lane line as the target point set is selected from the several sets of lane line points from each of the remaining sensors, and points from the selected lane line point sets are added to the target point set. The remaining sensors are those other than the current sensor among the plurality of sensors. In other words, after determining the reference sensor and its target point sets, points from the lane line point sets of the remaining sensors that belong to the same lane line as each target point set are added to the target point set, thus fusing the points belonging to the same lane line from the reference sensor and the remaining sensors. It can be understood that each target point set is a fusion of the points belonging to the same lane line from the reference sensor and the remaining sensors.

[0040] In one embodiment, for each remaining sensor, a set of lane line points that satisfies a first preset requirement can be selected from a plurality of lane line point sets corresponding to the remaining sensors. The first preset requirement includes at least one of a slope difference reaching a first preset range and a distance reaching a second preset range. That is, whether each set of lane line points corresponding to each remaining sensor satisfies the first preset requirement with respect to the target point set can be determined based on the difference between the slope of each set of lane line points corresponding to each remaining sensor and the slope of the target point set, or the distance between each set of lane line points and the target point set. It should be noted that when the slope of a set of lane line points is the same as or similar to the slope of the target point set, the difference between the slopes of the set of lane line points and the target point set reaches the first preset range. In this case, it indicates that the lane line corresponding to the set of lane line points is parallel to the lane line corresponding to the target point set. When the distance between a set of lane line points and the target point set reaches the second preset range, it indicates that the lane line corresponding to the set of lane line points is very close to the lane line corresponding to the target point set, and is likely the same lane line. The first and second preset ranges can be determined based on the actual lane line fusion effect, and are not specifically limited here.

[0041] It should be noted that, since sensors may miss some data during the process of collecting current environmental information, when determining the reference sensor, the sensor that has collected information, has clear information, or is located in an important position can be used as the reference sensor. This makes the target point set, which combines the reference sensor and the remaining sensors that belong to the same lane line, more representative.

[0042] For example, one of the sensors collecting information about the environment in front of the vehicle can be used as a reference sensor. Points from several lane line point sets corresponding to the remaining sensors (including those collecting information about the environment behind the vehicle) are added to the target point sets of the reference sensors belonging to the same lane line. Then, based on the target point sets of the reference sensors that have fused the lane line point information from the remaining sensors, the fused lane line of the sensors at the current moment is determined. It is understandable that due to the influence of sensor intrinsic or extrinsic parameters, if the point sets of lane lines behind the vehicle and those in front of the vehicle are fused separately to determine the fused lane lines corresponding to the rear and front of the vehicle respectively, there will be a gap between the fused lane lines corresponding to the rear and front of the vehicle, resulting in discontinuous lines. However, this embodiment adds the point sets of lane lines behind the vehicle to the target point sets of the lane lines in front of the vehicle belonging to the same lane line, thus fusion of the point sets corresponding to the lane lines behind and in front of the vehicle. It is understood that by using the target point sets that include the point sets of the rear lane lines, the determined fused lane line makes the lane lines in front of and behind the vehicle appear as continuous lane lines. In other words, the merged lane lines in front of and behind the vehicle determined using the method of this embodiment appear more continuous.

[0043] In another embodiment, the lane line point sets of several sensors can be fused in other ways to determine at least one target point set. The specific method for determining at least one target point set can be determined according to actual needs and is not specifically limited here.

[0044] S13: Based on the set of target points and the historical fused lane lines, obtain the fused lane line of the target vehicle at the current moment. The historical fused lane lines are the fused lane lines of the target vehicle at historical moments.

[0045] In this embodiment, the merged lane line of the target vehicle at the current moment can be obtained based on the set of target points and the historical merged lane lines. The historical merged lane lines are the merged lane lines of the target vehicle at historical moments. The number of historical merged lane lines may be one or more, and each historical merged lane line corresponds to an actual lane line at that historical moment. Similarly, the current merged lane lines may be one or more, and each merged lane line corresponds to an actual lane line at the current moment.

[0046] Please see Figure 3 , Figure 3 yes Figure 1 The diagram shows a flowchart of one embodiment of step S13. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow that approach. Figure 3 The illustrated process sequence is limited. For example... Figure 3As shown, in this embodiment, based on the set of target points and historical fused lane lines, the fused lane line of the target vehicle at the current moment is obtained, including:

[0047] S31: Based on the historical fusion of lane lines and the driving parameters of the target vehicle, obtain the predicted lane line at the current moment. The driving parameters include at least one of vehicle speed, yaw angle, time interval, and distance interval. The time interval is the interval between the current moment and the historical moment, and the distance interval is the driving distance corresponding to the time interval.

[0048] This embodiment is used to obtain the fused lane line at the current time based on the predicted lane line of the historical fused lane line at the current time and the detected lane line at the current time.

[0049] In one embodiment, the predicted lane line of the historical fused lane line at the current moment can be predicted based on the target vehicle's driving parameters (time interval, vehicle speed, yaw angle, etc.).

[0050] In another implementation, for each historical merged lane line, a Kalman filter algorithm can be used to obtain a predicted lane line at the current time based on the target vehicle's driving parameters and the historical merged lane lines. For ease of understanding, lane lines can be described using lane line equations. (See also...) Figure 4 , Figure 4 This is a schematic diagram of the lane line coordinate system provided in this application. For example... Figure 4 As shown, a lane line coordinate system is established with the center of the rear axle of the target vehicle as the origin, the forward direction as the positive x-axis, and the leftward direction as the positive y-axis. Based on this, the lane line equation can be defined as:

[0051]

[0052] Where y0 represents the lateral offset, ε represents the angle of the target vehicle relative to the road curve, C0 represents the curvature of the curve, and C1 represents the rate of change of curvature. It should be noted that the lane line equation (i.e., the lane line) can be determined by y0, ε, C0, and C1. For the sake of convenience in subsequent expressions, the coefficients y0, ε, C0, and C1 are represented by lane line coefficients.

[0053] For example, the predicted lane line at the current time can be predicted using formula (1):

[0054] X(k|k-1)=AX(k-1|k-1)+BU(k) (1)

[0055] Where X(k|k-1) represents the lane line coefficients (including y0, ε, C0, and C1 mentioned above, i.e., the predicted lane lines) corresponding to the historical merged lane lines, A is the state matrix of the Kalman filter, X(k-1|k-1) is the lane line coefficients (i.e., the historical merged lane lines) of the historical merged lane lines, and U(k) is the external input. A, U(k), and B can be determined using the following formula:

[0056]

[0057]

[0058] Where Δd represents the distance interval, Δt represents the time interval, and yawrate represents the yaw angle. Δd can be determined based on the time interval and the target vehicle speed. After determining the above parameters, the state matrix of the Kalman filter can be determined using the distance interval. U(k) and B can be determined using parameters such as time interval, yaw angle, and distance interval. Then, the lane line coefficient X(k|k-1) of the predicted lane line at the current time can be determined using formula (1).

[0059] S32: Based on each set of target points, obtain the detection lane line at the current moment.

[0060] In one embodiment, for each set of target points, a point set fitting model or related algorithm can be used to fit each set of target points into a corresponding lane line. In this embodiment, the lane line obtained by fitting each set of target points is defined as the detection lane line corresponding to several sensors (corresponding to the detection lane line of several sensors for each actual lane line). It should be noted that the fitted detection lane line can be represented by the equation corresponding to the detection lane line. This equation includes lane line coefficients (the aforementioned coefficients such as y0, ε, C0, and C1). Refer to the content about the lane line equation and lane line coefficients in step S31; further details are omitted here.

[0061] S33: Based on the predicted lane lines and the detected lane lines, obtain the fused lane lines at the current moment.

[0062] In one embodiment, multiple lane line pairs can be determined first, wherein each lane line pair includes a predicted lane line and a detected lane line belonging to the same lane line; then, for each lane line pair, a Kalman filter algorithm can be used to obtain a fused lane line at the current time based on the predicted lane line and the detected lane line of the lane line pair.

[0063] Specifically, please refer to Figure 5 , Figure 5 yes Figure 3 The flowchart shown is a schematic diagram of one embodiment of step S33. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow the same pattern. Figure 5 The illustrated process sequence is limited. For example... Figure 5 As shown, in this embodiment, the fused lane line at the current moment is obtained based on the predicted lane line and the detected lane line, including:

[0064] S51: Identify multiple lane line pairs, each lane line pair including a predicted lane line and a detected lane line belonging to the same lane line.

[0065] In one embodiment, lane pairs can be determined based on the lane coefficients (or equations) of each predicted lane line and each detected lane line. It is understood that the purpose of determining lane pairs is to identify each predicted lane line and each detected lane line belonging to the same lane line, and then, based on one predicted lane line and one detected lane line from each lane pair belonging to the same lane line, to determine the merged lane line for the target vehicle. Therefore, for each predicted lane line, the detected lane line whose lane coefficients (or equations) are similar to those of the predicted lane line can be identified as a lane pair with the predicted lane line.

[0066] S52: For each lane pair, the Kalman filter algorithm is used to obtain a fused lane line at the current time based on the predicted lane line and the detected lane line of the lane pair.

[0067] In this embodiment, the predicted value of the covariance matrix at the current time can be obtained first based on the state matrix of the Kalman filter and the covariance matrix at historical time. Then, based on the predicted value of the covariance matrix at the current time and the preset observation matrix, the Kalman gain matrix is ​​obtained. Then, using the Kalman gain matrix, the lane line coefficient of the detected lane line in the lane line pair and the lane line coefficient of the predicted lane line, the final value of the fused lane line and the covariance matrix determined by the lane line pair at the current time is obtained. The final value of the covariance matrix is ​​used to predict the predicted value of the covariance matrix of the fused lane line at the current time at the next time.

[0068] For ease of explanation, the following text uses a single lane pair as an example to illustrate the process of obtaining a merged lane line corresponding to a single lane pair:

[0069] Specifically, for a lane pair, the predicted value of the covariance matrix at the current moment can be determined using formula (2):

[0070] P(k|k-1)=AP(k-1|k-1)A T +Q (2)

[0071] In formula (2), P(k|k-1) is the predicted value of the covariance matrix at the current time, A is the state matrix of the Kalman filter, P(k-1|k-1) is the covariance matrix at the historical detection time, and Q is the system noise covariance matrix (initialized with empirical values ​​obtained from actual experience).

[0072] After determining the predicted value of the covariance matrix at the current time, the Kalman gain matrix can be determined using formula (3):

[0073] K k =P(k|k-1)H T (HP(k|k-1)H T +R) -1 (3)

[0074] In formula (3), K k Here, H is the Kalman gain matrix, H is the preset observation matrix, which can be preset through experiments or empirical values, and R is the measurement noise covariance, which can be determined through experiments or empirical constants.

[0075] After determining the Kalman gain matrix, the lane line coefficients corresponding to the fused lane lines can be determined using formula (4):

[0076] X(k|k)=X(k|k-1)+K k (Z(k)-HX(k|k-1)) (4)

[0077] In formula (4), X(k|k-1) represents the lane line coefficient of the predicted lane line, Z(k) represents the lane line coefficient of the detected lane line, and X(k|k) represents the fused lane line coefficient (i.e., a fused lane line) of the target vehicle at the current time determined by a lane line pair (a predicted lane line and a detected lane line).

[0078] It should be noted that after determining the predicted value of the covariance matrix and the Kalman gain matrix at the current time, the final value of the covariance matrix can be determined using formula (5):

[0079] P(k|k=(IK) k H)P(k|k-1) (5)

[0080] Wherein, P(k|k) represents the final value of the covariance matrix, which is used to predict the covariance matrix prediction value of the fused lane line at the current time in the next time. By using the covariance matrix prediction value of the next time, the fused lane line at the next time can be obtained, thereby realizing the tracking of the fused lane line at different times.

[0081] It should be noted that, in some embodiments, in order to improve computational efficiency and enhance the numerical stability of the covariance matrix update calculation process at the current time, the Cholesky decomposition factor P(k|k-1)=Cp*Cp can be used. T The square root of the covariance matrix at the current time is directly propagated and updated in this form. The advantage of this embodiment is that it updates the covariance matrix in a more stable way.

[0082] Therefore, after Cholesky decomposition, formula (2) is updated to formula (6):

[0083] Cp(k)Cp T (k)=A(x)Cp(k-1)Cp T (k-1)A(x) T +CqCq T (6)

[0084] It should be noted that the Kalman filter algorithm described above updates the final covariance matrix and the lane coefficients of the final fused lane lines. Kalman filtering can achieve lane line tracking (from history to the present and even to the next moment). In addition, during the process of tracking and fusing lane lines, the Kalman filter algorithm can combine the driving parameters of the target vehicle. Therefore, the detected lane lines can be tracked and updated in combination with the vehicle kinematic model, which further improves the reliability of the system.

[0085] The above scheme determines at least one set of target points based on a set of lane line points from each sensor at the current time. Then, based on each set of target points and historical fused lane lines, the fused lane line for the target vehicle at the current time is obtained. Different sets of target points belong to different lane lines, and each lane line's target point set contains points from the lane line point set corresponding to at least one sensor. It can be understood that this embodiment can determine the fused lane line for several sensors based on the target point set (which fuses points from the lane line point sets corresponding to each sensor), ensuring that the obtained fused lane line conforms to the actual lane line and achieving accurate fusion of multi-source lane lines.

[0086] Please see Figure 6 , Figure 6This is a schematic diagram of a framework of an embodiment of the multi-source lane fusion device provided in this application. In this embodiment, the multi-source lane fusion device 60 includes: a point set determination module 61, a target point set determination module 62, and a lane fusion module 63. The point set determination module 61 is used to determine several lane line point sets of each sensor at the current time using environmental information collected by several sensors on the target vehicle at the current time; the target point set determination module 62 is used to determine at least one target point set based on the several lane line point sets of each sensor at the current time, wherein each target point set belongs to a different lane line target point set and contains points from at least one lane line point set corresponding to a sensor; the lane fusion module 63 is used to obtain the fused lane line of the target vehicle at the current time based on each target point set and the historical fused lane line, wherein the historical fused lane line is the fused lane line of the target vehicle at a historical time.

[0087] In some embodiments, the plurality of sensors include at least one first sensor and at least one second sensor, the first sensor being located at a first position of the target vehicle and the second sensor being located at a second position of the target vehicle, the first position being different from the second position.

[0088] In some embodiments, the target point set determination module 62 determines at least one target point set based on several lane line point sets of each sensor at the current time, including: determining a reference sensor from several sensors, and using several lane line point sets of the reference sensor at the current time as each target point set; for each target point set, selecting a lane line point set belonging to the same lane line as the target point set from several lane line point sets of each remaining sensor, and adding the points in the selected lane line point set to the target point set, wherein the remaining sensors are the sensors other than the reference sensor among the several sensors.

[0089] In some embodiments, selecting a set of lane line points belonging to the same lane line as the target point set from a plurality of lane line point sets of each remaining sensor includes: for each remaining sensor, selecting a set of lane line points that meets a first preset requirement from a plurality of lane line point sets corresponding to the remaining sensor; the first preset requirement includes at least one of slope difference reaching a first preset range and distance reaching a second preset range.

[0090] In some embodiments, the lane fusion module 63 obtains the fused lane line of the target vehicle at the current moment based on each set of target points and historical fused lane lines, including: obtaining the predicted lane line at the current moment based on historical fused lane lines and the driving parameters of the target vehicle; wherein the driving parameters include at least one of vehicle speed, yaw angle, time interval, and distance interval, the time interval is the interval between the current moment and the historical moment, and the distance interval is the driving distance corresponding to the time interval; obtaining the detected lane line at the current moment based on each set of target points; and obtaining the fused lane line at the current moment based on the predicted lane line and the detected lane line.

[0091] In some embodiments, the lane fusion module 63 obtains multiple fused lane lines, and the number of predicted lane lines and detected lane lines are both multiple. Based on historical fused lane lines and the driving parameters of the target vehicle, the predicted lane line at the current moment is obtained, including: for each historical fused lane line, a Kalman filter algorithm is used to obtain a predicted lane line at the current moment based on the driving parameters of the target vehicle and the historical fused lane lines; and / or, based on the predicted lane line and the detected lane line, the fused lane line at the current moment is obtained, including: determining multiple lane line pairs, each lane line pair including a predicted lane line and a detected lane line belonging to the same lane line; for each lane line pair, a Kalman filter algorithm is used to obtain a fused lane line at the current moment based on the predicted lane line and the detected lane line of the lane line pair.

[0092] In some embodiments, the point set determination module 61 uses environmental information collected by several sensors on the target vehicle at the current time to determine several lane line point sets of each sensor at the current time, including: for each sensor, using the environmental information collected by the sensor at the current time to perform lane line detection to obtain several lane line regions corresponding to the sensor; and determining the lane line point set belonging to the lane line from each lane line region.

[0093] In some embodiments, the point set determination module 61 determines the set of lane line points belonging to the lane lines from each lane line region, including: sampling each lane line region to obtain several points corresponding to the lane line region; performing clustering processing on the several points corresponding to each lane line region to obtain the set of lane line points corresponding to each lane line region; and performing filtering processing on each set of lane line points to obtain the filtered set of lane line points.

[0094] Please see Figure 7 , Figure 7 This is a schematic diagram of an embodiment of the electronic device provided in this application. In this embodiment, the electronic device 70 includes a processor 71 and a memory 72.

[0095] Processor 71 can also be referred to as CPU (Central Processing Unit). Processor 71 may be an integrated circuit chip with signal processing capabilities. Processor 71 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor, or processor 71 can be any conventional processor 71, etc.

[0096] The memory 72 in the electronic device 70 is used to store the program instructions required for the processor 71 to run.

[0097] The processor 71 is used to execute program instructions to implement the methods provided in any of the above embodiments and any non-conflicting combinations thereof.

[0098] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of the computer-readable storage medium provided in this application. The computer-readable storage medium 80 of this application embodiment stores program instructions 81, which, when executed, implement the methods provided in any of the above embodiments and any non-conflicting combinations. The program instructions 81 can form a program file and be stored in the computer-readable storage medium 80 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 80 includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0099] The above scheme determines at least one set of target points based on a set of lane line points from each sensor at the current time. Then, based on each set of target points and historical fused lane lines, the fused lane line for the target vehicle at the current time is obtained. Different sets of target points belong to different lane lines, and each lane line's target point set contains points from at least one sensor's corresponding lane line point set. It is understood that this scheme can determine fused lane lines from several sensors based on target point sets (which fuse points from the lane line point sets corresponding to each sensor), ensuring that the obtained fused lane lines conform to the actual lane lines and achieving accurate fusion of multi-source lane lines.

[0100] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0101] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multi-source lane line detection method, characterized in that, The method comprises: determining a plurality of lane line point sets of each sensor at the current time based on the environment information collected by a plurality of sensors on the target vehicle at the current time; the plurality of sensors comprises at least one first sensor and at least one second sensor, the first sensor is arranged in front of the target vehicle, and the second sensor is arranged behind the target vehicle; determining at least one target point set based on the plurality of lane line point sets of each sensor at the current time, wherein the target point set can fit a lane line, and each target point set belongs to a different lane line, and the target point set contains points in the lane line point set corresponding to at least one sensor; based on the predicted lane line and the detected lane line, a fusion lane line of the target vehicle at the current time is obtained; the predicted lane line is obtained based on a historical fusion lane line and driving parameters of the target vehicle; wherein the driving parameters include speed, yaw angle, time interval and distance interval; the detected lane line is fitted based on each target point set; and the historical fusion lane line is a fusion lane line of the target vehicle at a historical time; wherein, based on the plurality of lane line point sets of each sensor at the current time, determining at least one target point set comprises: determining a reference sensor from the plurality of sensors, and taking the plurality of lane line point sets of the reference sensor at the current time as each target point set; for each target point set, selecting a lane line point set belonging to the same lane line as the target point set from the plurality of lane line point sets of each remaining sensor, and adding points in the selected lane line point set to the target point set; wherein the remaining sensors are sensors other than the reference sensor in the plurality of sensors; the reference sensor is the first sensor arranged in front of the target vehicle, and the remaining sensors include the second sensor arranged behind the target vehicle, and the selected lane line point set includes the lane line point set of the second sensor arranged behind the target vehicle at the current time.

2. The method of claim 1, wherein, The selected lane line point set belonging to the same lane line as the target point set from the plurality of lane line point sets of each remaining sensor comprises: for each remaining sensor, selecting a lane line point set satisfying a first preset requirement from the plurality of lane line point sets corresponding to the remaining sensor; the first preset requirement comprises at least one of a slope difference reaching a first preset range and a distance reaching a second preset range.

3. The method of claim 1, wherein, The time interval is the interval between the current time and the historical time, and the distance interval is the driving distance corresponding to the time interval.

4. The method of claim 1, wherein, The number of fusion lane lines is multiple, and the number of predicted lane lines and detected lane lines is multiple; based on the historical fusion lane line and the driving parameters of the target vehicle, the predicted lane line at the current time is obtained, comprising: For each of the historical fused lane lines, a Kalman filtering algorithm is adopted to obtain a predicted lane line at the current time based on the driving parameters of the target vehicle and the historical fused lane lines. And / or, based on the predicted lane line and the detected lane line, a fused lane line at the current time is obtained, including: A plurality of lane line pairs are determined, each of the lane line pairs including a predicted lane line and a detected lane line belonging to the same lane line; For each of the lane line pairs, a Kalman filtering algorithm is adopted to obtain a fused lane line at the current time based on the predicted lane line and the detected lane line of the lane line pair.

5. The method of claim 1, wherein, The environmental information collected by the sensors on the target vehicle at the current time is used to determine a plurality of lane line point sets of each of the sensors at the current time, including: For each of the sensors, lane line detection is performed on the environmental information collected by the sensor at the current time to obtain a plurality of lane line regions corresponding to the sensor; Lane line point sets belonging to lane lines are determined from each of the lane line regions.

6. The method of claim 5, wherein, The lane line point sets belonging to lane lines are determined from each of the lane line regions, including: For each of the lane line regions, the lane line region is sampled to obtain a plurality of points corresponding to the lane line region; Each of the lane line regions is clustered to obtain a lane line point set corresponding to each of the lane line regions; Each of the lane line point sets is filtered to obtain a filtered lane line point set.

7. An electronic device, comprising: The memory and the processor are coupled to each other, The memory stores program instructions; The processor is configured to execute the program instructions stored in the memory to implement the method of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program instructions which can be executed to implement the method of any one of claims 1-6. The computer readable storage medium is configured to store program instructions which can be executed to implement the method of any one of claims 1-6.

Images