Method and apparatus for lane line fusion

Abstract

According to embodiments of the present disclosure, methods, apparatuses, devices and media for lane line fusion are provided. The method includes obtaining first lane line information associated with a first sensor and second lane line information associated with a second sensor; determining first matching information between a first set of lane lines indicated by the first lane line information and a set of reference lane lines indicated by reference lane line information; determining second matching information between a second set of lane lines indicated by the second lane line information and the set of reference lane lines indicated by the reference lane line information; and fusing the first lane line information and the second lane line information based on the first matching information and the second matching information. Thereby, the success rate of lane line matching can be improved in special road sections and lane line mis-detection, and the accuracy and robustness of lane line matching in all scenarios can be improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for lane line fusion. Background Technology

[0002] With the development of the automotive industry, electrification and intelligentization have become trends. Advanced Driving Assistance Systems (ADAS), as a crucial component of intelligent vehicles, can perceive dynamic and static objects based on environmental data collected from multiple sensors around the vehicle, thereby providing advance warnings of danger or controlling driving. Lane detection technology, as one of the fundamental technologies of ADAS, is core to functions such as Integrated Adaptive Cruise Control (ICC), Lane Departure Warning (LDW), Traffic Jam Assist (TJA), and Highway Assist (HWA). Summary of the Invention

[0003] In a first aspect of this disclosure, a method for lane line fusion is provided. The method includes: acquiring first lane line information associated with a first sensor and second lane line information associated with a second sensor; determining first matching information between a first set of lane lines indicated by the first lane line information and a set of reference lane lines indicated by reference lane line information; determining second matching information between a second set of lane lines indicated by the second lane line information and a set of reference lane lines indicated by reference lane line information; and fusing the first lane line information and the second lane line information based on the first matching information and the second matching information.

[0004] In a second aspect of this disclosure, an apparatus for lane line fusion is provided. The apparatus includes: an acquisition module configured to acquire first lane line information associated with a first sensor and second lane line information associated with a second sensor; a first determination module configured to determine first matching information between a first set of lane lines indicated by the first lane line information and a set of reference lane lines indicated by reference lane line information; a second determination module configured to determine second matching information between a second set of lane lines indicated by the second lane line information and a set of reference lane lines indicated by the reference lane line information; and a fusion module configured to fuse the first lane line information and the second lane line information based on the first matching information and the second matching information.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0008] It should be understood that the content described in this summary section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A flowchart illustrating a lane line fusion process according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating an example of lane line fusion according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic diagram illustrating a forward matching process according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A block diagram of an apparatus for lane line fusion according to some embodiments of the present disclosure is shown; and

[0015] Figure 6 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0020] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only under the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0021] As briefly mentioned above, lane detection technology is a fundamental and important technology for ADAS. Improving the accuracy of lane matching based on multi-sensor detection can enhance driving safety.

[0022] ADAS primarily uses sensors such as cameras, LiDAR, millimeter-wave radar, and ultrasonic radar to collect environmental data around the vehicle. However, these sensors have limitations in their ability to detect lane lines. For example, a wide-angle camera has a field of view (FOV) of approximately 120°, capable of detecting all lane lines laterally (perpendicular to the driving direction) and ensuring accurate detection of lane lines within a range of 0-60 meters from the vehicle longitudinally (parallel to the driving direction). Similarly, a telephoto camera has a field of view of approximately 30°, capable of detecting 2-3 lane lines close to the vehicle laterally and ensuring accurate detection of lane lines within a range of 20-120 meters longitudinally.

[0023] Due to the safety requirements of ADAS (Advanced Driver Assistance Systems), to ensure lane line detection accuracy in both lateral and longitudinal directions, lane line detection results from multiple sensors can be fused to output high-precision lane lines with a wide range and long distance. To fuse lane line information identified by multiple sensors, lane lines must first be accurately matched, and then lane lines belonging to the same target from different sensors are fused. On the one hand, the matching accuracy is affected by the complexity of road scenes and the diversity of lane line morphology. Lane lines have very complex parameters (morphology and attributes), such as solid lines, dashed lines, double solid lines, grid lines, yellow, white, etc. Furthermore, lane lines are easily affected by changes in lighting, vehicle occlusion, and road wear. These factors increase the challenge of lane line detection. On the other hand, the matching accuracy is limited by sensor inherent errors or detection errors. For example, regarding lateral errors, some lane matching techniques first calculate the lateral error between each pair of lane lines (e.g., lane line M1 identified by sensor M and lane line N1 identified by sensor N) using simple, equally spaced sampling based on the lane line fitting parameters. Then, a simple matching algorithm (e.g., the Hungarian algorithm) is used to match the lane lines, allowing two lane lines with smaller lateral errors to be successfully matched. This simple matching algorithm has several drawbacks, which will be analyzed further below.

[0024] In the matching algorithm described above, the lateral error calculation is based on anchor point sampling using the fitted parameters of the lane lines, which may introduce fitting errors. For example, for lane lines M1 and N1, 10 anchor points are sampled at equal intervals between 10 and 80 meters. Then, the mean of the lateral errors of the two lanes at these 10 anchor points is calculated and used as the final error. The fitted parameters referenced in this process may be misfitted for road scenarios such as curves or intersections, resulting in discrepancies between the actual lateral errors and the actual errors, thus leading to incorrect matching.

[0025] Lane matching accuracy is also affected by sensor errors. For example, lane lines identified by a wide-angle camera and those identified by a telephoto camera, when projected onto the vehicle's coordinate system, should theoretically overlap. However, due to intrinsic and extrinsic errors in cameras, there are significant lateral errors in the results of lane line matching using different sensors. In particular, vehicle vibrations during driving increase these intrinsic and extrinsic errors, thus increasing the probability of lane matching errors.

[0026] Lane matching techniques that only consider lateral errors are also prone to mismatches. For example, due to the diversity of lane line shapes, old lane lines (where the paint has been removed but lane line traces remain) or road cracks may be misdetected as short lane lines. These short lane lines, having smaller lateral errors compared to those detected by other sensors, may be successfully matched. However, longer lane lines with slightly higher lateral errors may be mismatched. Referring to the preceding information, firstly, due to projection errors, the lateral errors here cannot fully reflect the errors between lane lines in the real world. Secondly, lane matching actually requires long, precise lane lines; short, short lane lines do not meet the requirements. Therefore, simply using lateral error as a matching reference value can easily lead to mismatches.

[0027] In summary, due to sensor limitations, the complexity of road scenarios, and the diversity of lane line shapes, simple lateral error matching may result in low lane line matching accuracy and low matching precision, which could lead to serious traffic accidents.

[0028] This disclosure proposes a scheme for lane line fusion. A robust inter-sensor lane line matching method based on reference lane line information can solve the matching failure problem of existing technologies in situations such as curves, intersections, and false detections of short lane lines. Furthermore, it exhibits higher matching accuracy and robustness compared to existing technologies in other arbitrary scenarios. According to various embodiments of this disclosure, first lane line information associated with a first sensor and second lane line information associated with a second sensor are obtained. First matching information is determined between a first set of lane lines indicated by the first lane line information and a set of reference lane lines indicated by the reference lane line information. Second matching information is determined between a second set of lane lines indicated by the second lane line information and a set of reference lane lines indicated by the reference lane line information. Based on the first and second matching information, the first and second lane line information are fused.

[0029] In the embodiments of this disclosure, lane line information detected by multiple sensors is matched with reference lane line information, and then the matched lane line information is fused. This method can improve the lane line matching success rate in special road sections and in cases of lane line false detection, thereby improving the lane line matching accuracy and robustness across all scenarios.

[0030] Example Environment

[0031] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, electronic device 120 receives lane line information 110 to match and fuse multiple sets of lane lines, and outputs a lane line fusion result 130.

[0032] Lane line information 110 is obtained from data collected on the vehicle's surrounding environment by multiple sensors 102 mounted on the vehicle 101. Depending on the type of sensor 102, the collected lane line information includes, but is not limited to, the number, location, shape, size, color, texture, etc. of lane lines. Sensors 102 can be any suitable sensor capable of detecting lane lines, including but not limited to cameras, millimeter-wave radar, lidar, ultrasonic radar, etc. Although in Figure 1 The lane line information 110 is shown as two-dimensional image information acquired by a camera; however, it should be understood that the lane line information 110 can also be three-dimensional point cloud information acquired by a LiDAR. The source and format of the lane line information are merely exemplary and do not imply any limitation on the scope of this disclosure. The sensor 102 can be installed in the vehicle 101 or placed within the vehicle 101 as a stand-alone device.

[0033] The electronic device 120 can present the lane line fusion result 130 in a bird's-eye view (BEV) format on the display terminal. The electronic device 120 can also store the lane line fusion result 130 as reference lane line information for use as a reference when matching multiple sets of lane lines.

[0034] Electronic device 120 can be any type of computing device, including terminal devices or server devices. For example, electronic device 120 is a terminal device, integrated into the vehicle's infotainment system in vehicle 101, or it can be separate from vehicle 101. Terminal devices can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. As another example, electronic device 120 can be a server device. Server devices can include computing systems / servers, such as mainframe computers, edge nodes, computing devices in cloud environments, etc. In some embodiments, a portion of electronic device 120 is located in the cloud, and the portion in the cloud can communicate wirelessly with the portion within vehicle 101.

[0035] Vehicle 101 can be any type of vehicle capable of carrying people and / or goods and moving via a power system such as an engine, including but not limited to cars, trucks, buses, electric vehicles, RVs, trains, etc. In some cases, vehicle 101 can be an integrated system combining environmental perception, planning and decision-making, and multi-level assisted driving functions; such vehicles are also referred to as intelligent vehicles. Furthermore, vehicle 101 can be an autonomous vehicle.

[0036] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0037] Figure 2 A flowchart of a process 200 for lane line fusion according to some embodiments of the present disclosure is shown. Process 200 can be implemented in environment 100, for example by electronic device 120.

[0038] like Figure 2 As shown, in block 210, electronic device 120 acquires first lane line information and second lane line information. For example, to generate lane line fusion result 130, electronic device 120 can acquire lane line information 110, which may include multiple lane line information types, such as different lane line information collected using different sensors.

[0039] Figure 3 A schematic diagram of an example 300 of lane line merging according to some embodiments of the present disclosure is shown. Figure 3In the example, the first lane line information is lane line information 310 acquired by sensor 301 (also referred to as the first sensor). The second lane line information is lane line information 320 acquired by sensor 302 (also referred to as the second sensor). In some embodiments, vehicle 101 is equipped with multiple lane line detection sensors, and electronic device 120 can automatically or be designated to select the first and second sensors from them according to the scene. For example, in a daytime road scene, electronic device 120 selects a wide-angle camera as the first sensor and a telephoto camera as the second sensor to obtain a lane line image containing rich information (such as texture or color). As another example, in a nighttime road scene, electronic device 120 selects a lidar as the first sensor and a millimeter-wave radar as the second sensor to improve the resistance to interference from light and shadow. In some embodiments, electronic device 120 can also generate a lane line fusion result 130 based on lane line information acquired by all lane line detection sensors. For simplicity, the following description still uses the first lane line information and the second lane line information acquired by two sensors respectively as an example.

[0040] In box 220, electronic device 120 determines first matching information between a first set of lane lines indicated by first lane line information and a set of reference lane lines indicated by reference lane line information.

[0041] exist Figure 3 In the example, sensor 301 is, for example, a camera, and the captured image contains lane line information 310. Electronic device 120 receives the image captured by sensor 301 and detects a set of lane lines 311 from the lane line information 310 contained in the image. The set of lane lines 311 includes lane line A1, lane line A2, lane line A3, and lane line A4. Electronic device 120 performs a matrix transformation based on the extrinsic and intrinsic parameters of the camera to project the detected lane lines A1 to lane lines A4 onto a predetermined coordinate system (e.g., projected onto the vehicle coordinate system from a BEV perspective). It should be understood that for different types of sensors, electronic device 120 can utilize any known or future-developed techniques to perform matrix transformations to project the detected lane lines onto a predetermined coordinate system.

[0042] To obtain matching information, the electronic device 120 detects a set of reference lane lines 331 from the reference lane line information 330. This set of reference lane lines 331 includes lane line R1, lane line R2, lane line R3, and lane line R4. Further, the electronic device 120 projects lane lines R1 to R4 onto the same coordinate system, for example, the vehicle coordinate system. In the same coordinate system, the electronic device 120 compares lane lines A1 to A4 with lane lines R1 to R4, calculating the pairwise errors between lane lines to determine matching information 303 (also called first matching information). Matching information 303 includes the error value between all lane line pairs in the two sets of lane lines (e.g., a lane line pair is formed by one lane line in one set of lane lines 311 and one lane line in one set of reference lane lines 331), whether a matching lane line pair exists, and so on.

[0043] In some embodiments, to determine whether a detected lane line matches a reference lane line, the electronic device 120 first determines the lane line points for each lane line. When comparing lane lines pairwise, the electronic device 120 calculates the lateral error between the corresponding lane line points to determine whether the lane lines match. (Continuing...) Figure 3 For example, when comparing lane line A1 and lane line R1, electronic device 120 first extracts lane line points, for example, extracting multiple lane line points at equal intervals. Then, it compares the pixel coordinates of the lane line points on lane line A1 with the pixel coordinates of the corresponding lane line points on lane line R1, and determines whether lane line A1 and lane line R1 match based on the magnitude of the error between lane line point pairs (e.g., a point on one lane line and a point on another lane line that have the same horizontal or vertical coordinate). Similarly, electronic device 120 sequentially determines whether there are matching lane lines between a set of lane lines 311 and a set of reference lane lines 331. In some embodiments, electronic device 120 may extract multiple lane line points at non-equal intervals. For example, more lane line points may be extracted in the area closer to the vehicle, while fewer lane line points may be extracted in the area farther away from the vehicle.

[0044] In some embodiments, the electronic device 120 determines whether lane line pairs match based on the lateral distance (also known as lateral error) between multiple lane line point pairs. Continue Figure 3For example, taking a vehicle coordinate system, the vehicle's position is the origin, the direction of travel is the Y-axis, and the direction perpendicular to the direction of travel is the X-axis (i.e., lateral). The electronic device 120 takes multiple anchor points at equal intervals on the Y-axis and calculates the abscissas of the lane line points corresponding to these anchor points on lane line A1 and lane line R1 respectively. If the lateral distance between the corresponding lane line points is less than a threshold, then lane line A1 and lane line R1 are successfully matched. Alternatively, the electronic device 120 can determine the centerline of the lane line based on the lane line width and determine the lateral distance between lane line point pairs based on the centerline of the lane line pair. Alternatively, the electronic device 120 can determine the left and right lines of the lane line based on the lane line width and then determine the lateral distance between lane line point pairs based on the left and / or right lines of the lane line pair.

[0045] Continue to refer to Figure 2 In box 230, electronic device 120 determines second matching information between a second set of lane lines indicated by second lane line information and a set of reference lane lines indicated by reference lane line information.

[0046] exist Figure 3 In the example, sensor 302 is, for example, a camera, and the captured image contains lane line information 320. Electronic device 120 receives the image captured by sensor 302 and detects a set of lane lines 321 from the lane line information 320 contained in the image. The set of lane lines 321 includes lane line B1, lane line B2, lane line B3, and lane line B4. Electronic device 120 performs a matrix transformation based on the extrinsic and intrinsic parameters of the camera to project the detected lane lines B1 to B4 onto the same coordinate system as the reference lane lines R1 to R4. In the same coordinate system, electronic device 120 compares lane lines B1 to B4 with lane lines R1 to R4, calculates the pairwise error between the lane lines, and thus determines matching information 304 (also called second matching information). For details on how to determine whether the detected lane lines match the reference lane lines, please refer to the description above; it will not be repeated here.

[0047] According to various embodiments of this disclosure, when determining whether there are matching lane lines in the first set of lane lines and the second set of lane lines, the electronic device 120 first compares both with the same set of reference lane lines, and then indirectly determines whether they match. In some cases, the lane line sets being compared may have different distance ranges, which may result in lane line pairs having different lengths. For example, in Figure 3In the example, lane line A1 and lane line R1 have different lengths. In some embodiments, the electronic device 120 determines the distance range from the lane line to the vehicle based on first lane line information and second lane line information. Further, the lane line points of each lane line are determined based on the common range of both. In this way, the computational load can be reduced and the matching efficiency improved. For example, in Figure 3 In the example, the distance range corresponding to lane line information 310 is 0 to 60 meters, and the distance range corresponding to lane line information 320 is 10 to 30 meters, so the common range is 10 to 30 meters.

[0048] Alternatively, the electronic device 120 can further determine the common area based on the range of the Region of Interest (ROI). For example, if the distance range corresponding to lane line information 310 is 0 to 60 meters, the distance range corresponding to lane line information 320 is 20 to 120 meters, and the ROI range is 20 to 50 meters, then the common area is 20 to 50 meters.

[0049] When determining whether lane line pairs match, the electronic device 120 first determines sampled lane line points on each lane line, and then compares the corresponding sampled lane line points. In some cases, the lane line information only contains a set of discrete lane line points identified by sensors for the corresponding lane line. The electronic device 120 can fit these discrete lane line points into a single lane line. Then, the electronic device 120 samples at equal intervals on the fitted lane line, using the sampled points as sampled lane line points to calculate the lateral error between lane line pairs.

[0050] In some embodiments, the electronic device 120 uses an interpolation algorithm to determine the lateral error between lane line pairs to avoid introducing errors caused by fitting lane lines. In this way, for complex road scenarios such as intersections and curves, the lateral error between lane lines can be calculated more accurately, thereby improving the accuracy of lane line matching. Specifically, the electronic device 120 determines the number and sampling positions of a set of sampled lane line points for a common area, and determines whether there is a point among the lane line points identified by the sensor that corresponds to the position of the sampled lane line point. If such a point exists, the electronic device 120 identifies that point as the sampled lane line point. If no such point exists, the electronic device 120 interpolates between the two closest lane line points to determine the corresponding point. Alternatively or concurrently, the electronic device 120 uses a linear interpolation algorithm to determine the sampled lane line point, that is, it uses the straight line between the two closest lane line points to determine the point corresponding to the position of the sampled lane line point. The electronic device 120 may also use other interpolation algorithms, such as parabolic interpolation algorithms. In this way, fitting errors can be avoided, thereby improving the accuracy of lane line matching.

[0051] As mentioned above, the matching error between lane line pairs includes lateral error and longitudinal error (also known as length error). In some embodiments, to further improve the accuracy of lane line matching, the electronic device 120 determines the matching relationship of lane line pairs based on both length difference and lateral distance. In this way, the problem of mismatched small lane line segments caused by road surface wear or old lane lines can be solved. Continuing... Figure 3 For example, electronic device 120 determines the length of each lane line in a set of lane lines 311 and the length of each lane line in a set of reference lane lines 331. For instance, when determining whether lane line A1 and lane line R1 match, electronic device 120 makes the determination based on the difference in length and the difference in lateral distance between them.

[0052] In some embodiments, when determining the longitudinal error between lane pairs, if the length of the lane line identified by the sensor is greater than or equal to the length of the reference lane line, the electronic device 120 determines the longitudinal error as a preset value, such as 0. If the length of the lane line identified by the sensor is less than the length of the reference lane line, the electronic device 120 determines the length difference between the two as the longitudinal error. Furthermore, the electronic device 120 determines the matching relationship of the lane pairs based on the longitudinal and lateral errors.

[0053] In some embodiments, the electronic device 120 first determines the lateral and longitudinal errors of the lane line pairs, normalizes both respectively, and then sums them. Then, the electronic device 120 determines the matching relationship of the lane line pairs based on the normalized summed errors.

[0054] In some embodiments, the electronic device 120 determines the overall error between lane pairs based on the lateral error, longitudinal error, a preset lateral error weight, and a preset longitudinal error weight. Furthermore, the electronic device 120 determines the matching relationship between lane pairs based on the overall error.

[0055] The above describes how the matching relationship of lane line pairs is determined. In some embodiments, to further improve matching accuracy, the electronic device 120 employs a bidirectional matching strategy to determine whether lane line pairs match. Specifically, the electronic device 120 determines the matching relationship from a set of reference lane lines to a set of lane lines identified by sensors (also known as a forward matching relationship), and determines the matching relationship from a set of lane lines identified by sensors to a set of reference lane lines (also known as a reverse matching relationship). Furthermore, the electronic device 120 determines the matching information between the two based on the forward and reverse matching relationships.

[0056] Figure 4 A schematic diagram of a forward matching process 400 according to some embodiments of the present disclosure is shown. (See also:) Figure 3The example describes process 400. Electronic device 120 performs error calculations between a set of lane lines 311 and a set of reference lane lines 331 to obtain an error matrix M. AR The electronic device 120 performs error calculations between a set of lane lines 321 and a set of reference lane lines 331 to obtain an error matrix M. BR Error matrix M AR and error matrix M BR Both include lateral and longitudinal errors.

[0057] With regard to the error matrix M AR Taking the forward matching algorithm as an example, in process 400, the error matrix M is first... AR The calculation begins step-by-step from the first row and first column. If the current row number i is less than the number of rows in the matrix, it continues to check if the current column number j is less than the number of columns in the matrix, and then checks if the corresponding error value R(i, j) is less than a preset threshold. After determining the matching result based on each error value in the first column, the calculation is then performed on the next row. After completing the traversal calculation, the electronic device 120 finally outputs the matching results of a set of lane lines 311 and a set of reference lane lines 331, thereby determining the matching relationship between the two.

[0058] Alternatively, when applying the reverse matching algorithm to the error matrix, the electronic device 120 can perform row and column transformations on the initial error matrix and calculate it in conjunction with the forward matching process. The matching result output by the forward matching should be consistent with the matching result output by the reverse matching. For example, if lane line R1 successfully matches lane line A1 during forward matching, and lane line A1 successfully matches lane line R1 during reverse matching, then the electronic device 120 determines that the two are successfully matched. If either the forward or reverse matching is unsuccessful, then the electronic device 120 determines that the two are unsuccessful. In this way, the accuracy of lane line matching can be improved.

[0059] For the error matrix M BR The process of applying bidirectional matching can be referred to the above embodiment. Thus, the electronic device 120 determines whether the first set of lane lines and the second set of lane lines match based on a set of reference lane lines. That is, if the first set of lane lines matches a set of reference lane lines, and the second set of lane lines also matches that set of reference lane lines, then the first set of lane lines and the second set of lane lines are indirectly and successfully matched. In this way, accuracy can be improved compared to direct matching of the first set of lane lines and the second set of lane lines.

[0060] return Figure 2 For example, based on the matching results determined in the above embodiments, in box 240, the electronic device 120 fuses the first lane line information and the second lane line information based on the matching results. (Continue) Figure 3For example, if lane line A1 and lane line R1 are successfully matched, and lane line B2 and lane line R1 are successfully matched, then lane line A1 and lane line B2 are the same lane line. The electronic device 120 assigns the same label to both lane lines, thereby performing lane line fusion and generating the lane line fusion result 130 of the current frame.

[0061] In some embodiments, the electronic device 120 fuses a portion of a first lane line with a portion of a second lane line. For example, for matching lane line pairs, the electronic device 120 fuses both. For mismatched lane lines, such as the lane line furthest from the vehicle identified by the second sensor but not by the first sensor, the electronic device 120 can merge the lane lines. In this way, the electronic device 120 generates the lane line fusion result 130 for the current frame, which improves the matching accuracy of lane lines while ensuring the richness of lane line information.

[0062] The electronic device 120 can employ different fusion strategies for different lane line parameters or different road scenarios. For example, for two matched lane lines, the electronic device 120 can average the positions (pixel coordinates) before fusing them into a single lane line. Alternatively, it can determine the position weight of the lane lines identified by the sensor based on its accuracy, and then determine the position of the fused lane line based on the positions of the two matched lane lines and their corresponding weights. Furthermore, in a nighttime road scenario, the position weight of lane lines identified using LiDAR is greater than that of lane lines identified using a camera. The electronic device 120 determines the weights corresponding to the two matched lane lines based on the current scene, and then determines the position of the fused lane line. These specific fusion methods are merely exemplary and are not intended to limit the scope of this disclosure.

[0063] In some embodiments, the electronic device 120 uses historical lane information determined by sensor data collected at a first moment as reference lane information, and matches the first lane information and second lane information collected at a second moment after the first moment with the reference lane information, thereby indirectly determining the matching result of the first lane and the second lane. This scheme of using historical lane information as reference lane information for matching ensures that lane information identified by other sensors and reference lane information can still be successfully associated even in intersection scenarios or when individual sensors miss detection, thus continuously outputting lane fusion results. For example, at the first moment, the electronic device 120 determines and stores reference lane information. At the second moment, the electronic device collects first lane information using a wide-angle camera, but does not collect second lane information using a telephoto camera. The electronic device 120 determines matching information based on the first lane information and the reference lane information, and then outputs the lane matching result based solely on this matching information. To ensure that the lane information at the second moment and the reference lane information at the first moment include as many similar scenarios as possible, thereby improving the matching success rate, the difference between the second moment and the first moment should be less than a preset threshold.

[0064] Alternatively or additionally, the electronic device 120 may adjust the difference between the second moment and the first moment based on the speed at which the sensor acquires information and / or the processing speed of the electronic device 120. For example, if the camera takes 1 / 60 of a second to acquire one frame of image, the difference between the second moment and the first moment may be a multiple of 1 / 60 of a second. Alternatively or additionally, the electronic device 120 may also dynamically adjust a preset threshold based on the current vehicle speed. For example, when the vehicle speed is slow, the scene changes within the field of view are smaller, and the preset threshold may be larger.

[0065] In some embodiments, the electronic device 120 updates historical lane line information based on lane line fusion result 130, thereby using it as reference lane line information for the next moment.

[0066] In summary, according to the various embodiments of this disclosure, the electronic device 120 can utilize interpolation algorithms to calculate the lateral error of lane line pairs, thereby improving the accuracy of lane line matching in complex road scenarios such as intersections or curves. The electronic device 120 can also calculate the longitudinal error of lane line pairs, thus solving the problem of mismatched lane lines in small segments due to road surface wear or old lane lines. Furthermore, the electronic device 120 can also indirectly match lane line information identified by different sensors based on historical lane line information, thereby ensuring continuous output of lane line matching results even in intersection scenarios or when individual sensors miss detections. In addition to the above advantages, the various embodiments of this disclosure can also perform lane line matching more stably and accurately in other road scenarios, thereby improving the accuracy and robustness of lane line matching across all scenarios.

[0067] Example devices and equipment

[0068] Figure 5 A schematic structural block diagram of an apparatus 500 for autonomous driving according to certain embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in an electronic device 120. Various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0069] As shown in the figure, the device 500 includes an acquisition module 510 configured to acquire first lane line information associated with a first sensor and second lane line information associated with a second sensor. The device 500 also includes a first determination module 520 configured to determine first matching information between a first set of lane lines indicated by the first lane line information and a set of reference lane lines indicated by reference lane line information. The device 500 further includes a second determination module 530 configured to determine second matching information between a second set of lane lines indicated by the second lane line information and a set of reference lane lines indicated by reference lane line information. The device 500 also includes a fusion module 540 configured to fuse the first lane line information and the second lane line information based on the first matching information and the second matching information.

[0070] In some embodiments, the first determining module 520 is configured to: determine a first set of lane line points indicating a first lane line in a first set of lane lines based on first lane line information; determine a reference lane line point indicating a reference lane line in a set of reference lane lines based on reference lane line information; and determine a matching relationship between the first lane line and the reference lane line based on the distance between the first set of lane line points and the second set of lane line points.

[0071] In some embodiments, the first determining module 520 is further configured to: determine a first distance range from the first lane line to the vehicle based on the first lane line information; determine a second distance range from at least one third lane line to the vehicle based on the second lane line information; determine a common range of the first distance range and the second distance range; and determine a set of sampled lane line points for the common range based on the first lane line information, as a first set of lane line points.

[0072] In some embodiments, the first determining module 520 is further configured to: determine the number of a set of sampled lane line points to be sampled; determine a set of sampling positions for sampling a set of sampled lane line points based on the number and common range; in response to the presence of lane line points identified by the first sensor at a set of sampling positions, determine the lane line points identified by the first sensor as corresponding sampled lane line points; and in response to the absence of lane line points identified by the first sensor at a set of sampling positions, determine the corresponding sampled lane line points based on the first lane line information and using an interpolation algorithm.

[0073] In some embodiments, the first determining module 520 is further configured to: determine a first length of the first lane line based on the first lane line information; determine a second length of the reference lane line based on the reference lane line information; determine the length difference between the first length and the second length; and determine the matching relationship between the first lane line and the reference lane line based on the length difference and the distance.

[0074] In some embodiments, the first determining module 520 is further configured to: determine a length error as a preset value in response to a length difference indicating that the first length is greater than or equal to the second length; determine a length difference as a length error in response to a length difference indicating that the first length is less than the second length; and determine a matching relationship between the first lane line and the reference lane line based on the length error and the distance.

[0075] In some embodiments, the first determining module 520 is further configured to: determine a first matching relationship between a first set of lane lines and a set of reference lane lines, the first matching relationship indicating at most one reference lane line in the set of reference lane lines that matches a given lane line in the first set of lane lines; determine a second matching relationship between the first set of reference lane lines and the first set of lane lines, the second matching relationship indicating at most one lane line in the first set of lane lines that matches a given reference lane line in the set of reference lane lines; and determine first matching information between the first set of lane lines and the set of reference lane lines based on the first matching relationship and the second matching relationship.

[0076] In some embodiments, the fusion module 540 is configured to: determine, based on first matching information, that a first lane line in a first group of lane lines matches a target reference lane line indicated by reference lane line information; determine, based on second matching information, that a second lane line in a second group of lane lines matches the target reference lane line; determine that the first lane line matches the second lane line; and fuse a first portion of the first lane line information for the first lane line and a second portion of the second lane line information for the second lane line.

[0077] In some embodiments, the reference lane information includes historical lane information determined based on sensor data collected at a first moment, the first lane information and the second lane information correspond to a second moment, the second moment is later than the first moment, and the difference between the second moment and the first moment is less than a preset threshold.

[0078] In some embodiments, the device 500 further includes an update module configured to update historical lane line information based on the fusion result of the first lane line information and the second lane line information.

[0079] Figure 6A block diagram is shown illustrating an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to achieve Figure 1 Electronic device 120. Electronic device 600 may include or be implemented as Figure 5 The device 500.

[0080] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0081] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 600.

[0082] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0083] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0084] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0085] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0086] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0088] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0090] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method of lane fusion, comprising: obtaining first lane information associated with a first sensor and second lane information associated with a second sensor; determining first matching information between a first set of lane lines indicated by the first lane information and a set of reference lane lines indicated by reference lane information; determining second matching information between a second set of lane lines indicated by the second lane information and the set of reference lane lines indicated by the reference lane information; and fusing the first lane information and the second lane information based on the first matching information and the second matching information; wherein determining the first matching information between the first set of lane lines and the set of reference lane lines comprises: determining, based on the first lane information, a first set of lane line points indicating first lane lines in the first set of lane lines; determining, based on the reference lane information, reference lane line points indicating reference lane lines in the set of reference lane lines; and determining, based on distances between the first set of lane line points and the reference lane line points, a matching relationship between the first lane lines and the reference lane lines; wherein determining the first set of lane line points indicating first lane lines in the first set of lane lines comprises: determining, based on the first lane information, a first distance range of the first lane lines to a vehicle; determining, based on the second lane information, a second distance range of at least one third lane line to the vehicle; determining a common range of the first distance range and the second distance range; and determining, based on the first lane information, a set of sampled lane line points for the common range as the first set of lane line points; wherein determining the set of sampled lane line points for the common range comprises: determining a number of a set of sampled lane line points to be sampled; determining, based on the number and the common range, a set of sampling positions for sampling the set of sampled lane line points; in response to a lane line point recognized with the first sensor existing at the set of sampling positions, determining the lane line point recognized with the first sensor as a corresponding sampled lane line point; and in response to no lane line point recognized with the first sensor existing at the set of sampling positions, determining a corresponding sampled lane line point based on the first lane information and with an interpolation algorithm. 2.The method of claim 1, wherein determining the matching relationship between the first lane line and the reference lane line comprises: determining, based on the first lane information, a first length of the first lane line; determining, based on the reference lane information, a second length of the reference lane line; determining a length difference between the first length and the second length; and determining, based on the length difference and the distance, the matching relationship between the first lane line and the reference lane line. 3.The method of claim 2, wherein determining, based on the length difference and the distance, the matching relationship between the first lane line and the reference lane line comprises: ​ ​ in response to the length difference indicating that the first length is greater than or equal to the second length, determining a length error as a preset value; in response to the length difference indicating that the first length is less than the second length, determining the length difference as the length error; and based on the length error and the distance, determining a matching relationship between the first lane line and the reference lane line. 4.The method of claim 1, wherein determining the first matching information between the first set of lane lines and the set of reference lane lines comprises: determining a first matching relationship of the first set of lane lines to the set of reference lane lines, the first matching relationship indicating at most one reference lane line of the set of reference lane lines that matches a given lane line of the first set of lane lines; determining a second matching relationship of the set of reference lane lines to the first set of lane lines, the second matching relationship indicating at most one lane line of the first set of lane lines that matches a given reference lane line of the set of reference lane lines; and based on the first matching relationship and the second matching relationship, determining the first matching information between the first set of lane lines and the set of reference lane lines. 5.The method of claim 1, wherein fusing the first lane line information and the second lane line information comprises: determining, based on the first matching information, that a first lane line of the first set of lane lines matches a target reference lane line indicated by the reference lane line information; determining, based on the second matching information, that a second lane line of the second set of lane lines matches the target reference lane line; determining that the first lane line matches the second lane line; and fusing a first portion of the first lane line information for the first lane line and a second portion of the second lane line information for the second lane line. 6.The method of claim 1, wherein the reference lane line information comprises historical lane line information determined based on sensor data collected at a first time, the first lane line information and the second lane line information correspond to a second time, the second time is later than the first time, and a difference between the second time and the first time is less than a preset threshold. 7.The method of claim 6, further comprising: based on a fusion result of the first lane line information and the second lane line information, updating the historical lane line information. 8.An apparatus for lane line fusion, comprising: an obtaining module configured to obtain first lane line information associated with a first sensor and second lane line information associated with a second sensor; a first determining module configured to determine first matching information between a first set of lane lines indicated by the first lane line information and a set of reference lane lines indicated by reference lane line information; a second determining module configured to determine second matching information between a second set of lane lines indicated by the second lane line information and the set of reference lane lines indicated by the reference lane line information; and a fusing module configured to fuse the first lane line information and the second lane line information based on the first matching information and the second matching information. ​ a fusion module configured to fuse the first lane line information and the second lane line information based on the first matching information and the second matching information; a first determination module further configured to determine, based on the first lane line information, a first set of lane line points indicative of first lane lines in the first set of lane lines; determine, based on the reference lane line information, reference lane line points indicative of reference lane lines in the set of reference lane lines; and determine, based on distances between the first set of lane line points and the reference lane line points, a matching relationship between the first lane lines and the reference lane lines; the first determination module further configured to determine, based on the first lane line information, a first distance range of the first lane lines to a vehicle; determine, based on the second lane line information, a second distance range of at least one third lane line to the vehicle; determine a common range of the first distance range and the second distance range; and determine, based on the first lane line information, a set of sampling lane line points for the common range as the first set of lane line points; the first determination module further configured to determine a number of a set of sampling lane line points to be sampled; based on the number and the common range, determine a set of sampling positions for sampling the set of sampling lane line points; in response to a lane line point recognized by the first sensor existing at the set of sampling positions, determine the lane line point recognized by the first sensor as a corresponding sampling lane line point; and in response to no lane line point recognized by the first sensor existing at the set of sampling positions, determine, based on the first lane line information and by using an interpolation algorithm, a corresponding sampling lane line point.

9. An electronic device comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit causing the electronic device to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program executable by a processor to implement the method according to any one of claims 1-7.

11. A computer program product comprising computer executable instructions that, when executed by a processor, implement the method according to any one of claims 1-7. ​ ​ ​

Images