Lane line determination method and device, equipment and storage medium
By determining and matching the set of lane line sampling points in images acquired at different times, the problem of large errors in lane line reconstruction by a single camera is solved, and low-cost, widely adaptable 3D lane line reconstruction is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-03-13
AI Technical Summary
Existing technologies for low-cost driver assistance systems often result in significant errors when using a single camera to reconstruct lane lines, especially in curved scenarios, and methods that rely on the assumption of parallel lane lines have limited applicability.
By determining the lane line sampling point set from images acquired at different times and matching the sampling points, the three-dimensional coordinates of the lane lines are determined, avoiding reliance on the prior assumption that the lane lines are parallel, thus achieving three-dimensional reconstruction of the lane lines.
It achieves 3D reconstruction of lane lines with low cost and low computational cost, has a wide range of applications, and is not limited by the lane line parallel assumption.
Smart Images
Figure CN121661492A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer vision technology, and in particular to a method, apparatus, device and storage medium for determining lane lines. Background Technology
[0002] In intelligent driving scenarios, it is crucial to collect environmental information about the vehicle's surroundings through sensors such as cameras and radar, and to fit and reconstruct lane lines based on this information for downstream use.
[0003] In low-cost driver assistance systems, only one forward-facing camera is used. Therefore, it is necessary to remodel the lane lines in the real world based on the images captured by the forward-facing camera, generating a local map with lane lines. Thus, how to reconstruct lane lines using images captured by a single camera has become a hot research topic in the field of low-cost driver assistance. Summary of the Invention
[0004] Typically, lane line reconstruction using a single camera in related technologies is based on the assumption that lane lines are parallel. If this condition is not met (e.g., in curved or bifurcated scenarios), the calculated 3D (Three-Dimensional) lane lines will have significant errors or even be incorrect. To address these technical problems, this disclosure provides a method, apparatus, device, and storage medium for determining lane lines, which can solve the problem of large lane line reconstruction errors in curved scenarios in related technologies.
[0005] A first aspect of this disclosure provides a method for determining lane lines. The method includes: determining a first image and a second image acquired by an image sensor of a vehicle at different times; determining a first lane line sampling point set and a second lane line sampling point set based on the first image and the second image, respectively; performing sampling point matching between the first lane line sampling point set and the second lane line sampling point set to obtain sampling point matching pairs; and determining lane lines based on the sampling point matching pairs.
[0006] A second aspect of this disclosure provides a lane line determination device, the device comprising: an image determination module for determining a first image and a second image acquired by an image sensor of a vehicle at different times; and a lane line sampling module for determining a first lane line sampling point set and a second lane line sampling point set based on the first image and the second image, respectively.
[0007] The sampling point matching module is used to match the sampling points of the first lane line with the sampling points of the second lane line to obtain sampling point matching pairs; the lane line determination module is used to determine the lane line based on the sampling point matching pairs.
[0008] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program for performing the lane line determination method provided in the first aspect embodiment.
[0009] A fourth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the lane line determination method provided in the first aspect embodiment above.
[0010] A fifth aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs the lane line determination method provided in the first aspect of the present invention.
[0011] Based on the lane line determination method provided in this disclosure, a first image and a second image collected by the vehicle's image sensor at different times are determined; based on the first image and the second image, a first lane line sampling point set and a second lane line sampling point set are determined respectively; the first lane line sampling point set and the second lane line sampling point set are matched to obtain sampling point matching pairs; based on the sampling point matching pairs, the lane line is determined. Since the embodiments of this disclosure can determine the three-dimensional coordinates of the lane line by matching the lane line sampling points in the images collected at different times, the three-dimensional reconstruction of the lane line can be achieved with low cost, low computational load, and a single camera; furthermore, since the lane line determination method in the embodiments of this disclosure does not rely on the prior assumption that the lane lines are parallel, the method has a wide range of applications. Attached Figure Description
[0012] Figure 1 This is a schematic diagram of a vehicle in a driving scenario provided for an exemplary embodiment of this disclosure.
[0013] Figure 2 A schematic flowchart illustrating a method for determining lane lines provided as an exemplary embodiment of this disclosure.
[0014] Figure 3A A flowchart illustrating a method for determining lane lines provided as another exemplary embodiment of this disclosure.
[0015] Figure 3B A schematic diagram illustrating the principle of a lane line determination method provided for an exemplary embodiment of this disclosure.
[0016] Figure 4A A schematic flowchart of a method for determining lane lines provided as yet another exemplary embodiment of this disclosure.
[0017] Figure 4BA schematic diagram illustrating the principle of a lane line determination method provided for another exemplary embodiment of this disclosure.
[0018] Figure 5 A schematic flowchart of a method for determining lane lines provided as yet another exemplary embodiment of this disclosure.
[0019] Figure 6 A schematic flowchart of a method for determining lane lines provided as yet another exemplary embodiment of this disclosure.
[0020] Figure 7 A schematic diagram of the structure of a lane line determination device provided for an exemplary embodiment of this disclosure.
[0021] Figure 8 A schematic diagram of the structure of a lane line determination device provided for another exemplary embodiment of this disclosure.
[0022] Figure 9 A schematic diagram of the structure of a lane line determination device provided as yet another exemplary embodiment of this disclosure.
[0023] Figure 10 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation
[0024] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.
[0025] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of this disclosure.
[0026] Application Overview
[0027] First, the application scenarios of this disclosure will be introduced. The lane line determination method provided in the embodiments of this disclosure can be applied to, for example, autonomous driving scenarios, assisted driving scenarios, and any other feasible scenarios.
[0028] like Figure 1As shown, a single-view image sensor 11 (such as a front-view image sensor) is installed on the vehicle 10. This image sensor 11 can then capture images of the lane lines 12 (or 13) in front of the vehicle 10, obtaining images containing the lane lines captured by the image sensor 11 at different times. Lane lines 12 and 13 together constitute the lane in which the vehicle 10 is located. For example, a first image captured by the image sensor 11 at time t0 and a second image captured by the image sensor 11 at time t0+n can be determined; where n is a natural number greater than or equal to 1. Both the first and second images include lane lines 12 (or 13). It should be emphasized that, depending on the field of view (FOV) of the image sensor, the images captured by the image sensor may include lane lines from other lanes in addition to the lane in which the vehicle is located.
[0029] Furthermore, the lane line determination method provided in this embodiment can be used to process the first image and the second image to obtain the three-dimensional coordinates of multiple points in lane line 12 (or lane line 13). Then, the three-dimensional coordinates of the multiple points in lane line 12 (or lane line 13) can be used to perform 3D (Three-Dimensional) reconstruction of lane line 12 (or lane line 13). It should be noted that... Figure 1 This disclosure is merely an illustrative example and does not represent the actual situation. Figure 1 The location and type of the image sensor 11 are not limited. Of course, the vehicle 10 may also include other sensors besides the image sensor 11 (such as other image sensors, radar sensors, etc.), and this disclosure does not limit this.
[0030] In related technologies, when determining lane lines using a single-view image sensor, it is typically assumed that the left and right lane lines of the same lane are parallel (i.e., the width of the same lane must remain consistent within a local area). Then, using the assumption of constant width, combined with the height of the single-view image sensor and camera calibration parameters, the 3D coordinates of the lane lines are calculated. Because this method relies on the prior assumption of parallel lane lines, its applicability is very limited; furthermore, since actual road conditions are highly complex, the 3D lane lines calculated by this method often have significant errors or even obvious inaccuracies.
[0031] To address the aforementioned issues, this disclosure provides a method for determining lane lines. This method involves determining a first image and a second image captured by a vehicle's image sensor at different times; determining a first lane line sampling point set and a second lane line sampling point set based on the first and second images; matching the first and second lane line sampling point sets to obtain sampling point matching pairs; and determining lane lines based on these matching pairs. Since this disclosure can determine the three-dimensional coordinates of lane lines by matching lane line sampling points in images captured at different times, it enables three-dimensional reconstruction of lane lines with low cost, low computational load, and a single camera. Furthermore, because the lane line determination method in this disclosure does not rely on the prior assumption of lane line parallelism, it has a wide range of applicability.
[0032] Exemplary methods
[0033] Figure 2 This is a flowchart illustrating a lane line determination method provided as an exemplary embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices, such as... Figure 2 As shown, the method includes steps S201-S204.
[0034] Step S201: Determine the first and second images captured by the vehicle's image sensor at different times.
[0035] For example, such as Figure 1 As shown, a first image captured by the image sensor 11 on the vehicle 10 (automobile) at a first moment and a second image captured by the same image sensor 11 at a second moment can be acquired; wherein the first moment and the second moment are different moments. For example, during the driving of the vehicle 10, the image sensor 11 on the vehicle 10 acquires images of the lane line 12 (or lane line 13) ahead (acquiring images at preset time intervals or continuously), obtaining multiple images including the lane line 12 (or lane line 13). Then, the first image and the second image can be determined from the multiple images including the lane line 12 (or lane line 13). The first image and the second image can be two frames corresponding to adjacent acquisition moments, or two frames corresponding to any two acquisition moments; this embodiment does not impose any limitations on this.
[0036] It should be noted that the first image and the second image may include the same lane line (e.g., both include lane line 12 and / or both include lane line 13), or they may include different lane lines. When the first image and the second image include the same lane line, the 3D coordinates of the lane line can be obtained using the lane line determination method in this embodiment, thereby achieving lane line reconstruction. When the first image and the second image do not include the same lane line, the 3D coordinates of the lane line cannot be determined if the lane line determination method in this embodiment is executed.
[0037] Step S202: Based on the first image and the second image, determine the first lane line sampling point set and the second lane line sampling point set respectively.
[0038] For example, in this embodiment of the disclosure, a first lane line sampling point set can be determined based on a first image, and a second lane line sampling point set can be determined based on a second image. The first lane line sampling point set includes multiple first sampling points, and the second lane line sampling point set includes multiple second sampling points. Furthermore, the multiple first sampling points are the coordinates of multiple pixels of the lane line in the first image within a preset coordinate system; similarly, the multiple second sampling points are the coordinates of multiple pixels of the lane line in the second image within the preset coordinate system. In other words, in this embodiment of the disclosure, multiple pixels belonging to the lane line in the first image can be transformed into a preset coordinate system to obtain the first lane line sampling point set. Similarly, multiple pixels belonging to the lane line in the second image can also be transformed into a preset coordinate system to obtain the second lane line sampling point set.
[0039] In some examples, the preset coordinate system in which the first and second sampling points are located can be the VCS (Vehicle Coordinate System) coordinate system corresponding to when the vehicle starts moving. The VCS coordinate system refers to a coordinate system centered on the vehicle, with its origin at a specific reference point on the vehicle (such as the center of the rear axle), the X-axis pointing in the direction of the vehicle's movement, the Y-axis pointing to the left or right side of the vehicle, and the Z-axis pointing upwards towards the vehicle and perpendicular to the plane formed by the X and Y axes. Of course, the preset coordinate system in which the first and second sampling points are located can also be other predefined coordinate systems, and this disclosure does not impose any limitations on this.
[0040] For example, such as Figure 3B As shown, according to the principle of pinhole imaging, point d1 is the first pixel in the first lane line pixel set, and point D1 is the first sampling point obtained after coordinate transformation of the first pixel. This first sampling point is located in a preset coordinate system. Furthermore, multiple first sampling points can constitute the first lane line sampling point set in step S202.
[0041] Step S203: Match the sampling points of the first lane line with the sampling points of the second lane line to obtain a sampling point matching pair.
[0042] For example, in this embodiment of the disclosure, a plurality of first sampling points included in the first lane line sampling point set can be matched with a plurality of second sampling points included in the second lane line sampling point set to obtain a sampling point matching pair.
[0043] For example, the sampling point set for the first lane line includes {D1, D2, ... D...} m The sampling point set for the second lane line includes {R1, R2, ..., R}. s In this context, m and s can be the same or different (i.e., this embodiment does not limit the number of sampling points included in the first lane line sampling point set and the second lane line sampling point set). Furthermore, the first lane line sampling point set and the second lane line sampling point set can be matched to obtain sampling point matching pairs. For example, for a first sampling point D1, this embodiment can determine a second sampling point R2 that matches it from the second lane line sampling point set; then D1 and R2 constitute a sampling point matching pair. Thus, based on the matching results, a second sampling point that matches the first sampling point in the first lane line sampling point set can be determined, thereby obtaining multiple sampling point matching pairs.
[0044] In some examples, a sampling point matching operation can be performed on multiple first sampling points in the first lane line sampling point set to determine second sampling points that can form sampling point matching pairs with each of the first sampling points. Then, the lane lines in step S201 can be reconstructed using multiple sampling point matching pairs.
[0045] Step S204: Determine lane lines based on the sampling point matching pairs.
[0046] For example, through the operation in step S203, the coordinates of multiple sampling point matching pairs (i.e., the 3D coordinates of multiple sampling point matching pairs in a preset coordinate system) can be determined. Furthermore, based on the 3D coordinates of the multiple sampling point matching pairs in the preset coordinate system, multiple 3D coordinate points on the lane line in step S201 can be determined. Furthermore, based on the multiple 3D coordinate points on the lane line, the lane line can be determined. The multiple 3D coordinate points on the lane line are also located within the preset coordinate system.
[0047] For example, if multiple identified sampling point matching pairs include {(D1, R2), (D2, R3), (D3, R5), (D4, R6), (D5, R9)}, then based on the sampling point matching pair (D1, R2), 3D coordinate point A on the lane line can be determined; based on the sampling point matching pair (D2, R3), 3D coordinate point B on the lane line can be determined; based on the sampling point matching pair (D3, R5), 3D coordinate point C on the lane line can be determined; based on the sampling point matching pair (D4, R6), 3D coordinate point D on the lane line can be determined; and based on the sampling point matching pair (D5, R9), 3D coordinate point E on the lane line can be determined. Furthermore, the lane line can be determined using the 3D coordinate points A, B, C, D, and E on the lane line (e.g., for 3D reconstruction of the lane line).
[0048] The lane line determination method provided in this disclosure involves determining a first image and a second image captured by the vehicle's image sensor at different times; determining a first lane line sampling point set and a second lane line sampling point set based on the first and second images; matching the sampling points of the first lane line sampling point set and the second lane line sampling point set to obtain sampling point matching pairs; and determining the lane line based on the sampling point matching pairs. Since this disclosure method can determine the three-dimensional coordinates of the lane line by matching lane line sampling points in images captured at different times, it can achieve three-dimensional reconstruction of the lane line with low cost, low computational load, and a single camera. Furthermore, since the lane line determination method in this disclosure does not rely on the prior assumption of lane line parallelism, the method has a wide range of applications.
[0049] like Figure 3A As shown above, in the above Figure 2 Based on the illustrated embodiment, step S203 may include steps S2031-S2033.
[0050] Step S2031: Determine the first target sampling point in the first lane line sampling point set, as well as the odometer information of the vehicle at different times.
[0051] For example, a first target sampling point can be determined from a plurality of first sampling points included in the first lane line sampling point set. For instance, the plurality of first sampling points included in the first lane line sampling point set can be traversed, and each selected first sampling point during the traversal is the first target sampling point. Of course, the first target sampling point in the first lane line sampling point set can also be determined in other ways, and this disclosure does not limit this approach.
[0052] In some examples, it is also necessary to determine the vehicle's odometer information at different times. For example, the odometer frame sequence during the vehicle's operation can be determined, and then the vehicle's odometer information at the corresponding time can be determined from the odometer frame sequence based on different times. For example, the vehicle's odometer frame sequence includes vehicle odometer information at time t0, time t0+1, and so on up to time t0+f, and the first image is acquired at time t0, and the second image is acquired at time t0+2. In this embodiment of the present disclosure, it is necessary to determine the vehicle odometer information corresponding to time t0 and the vehicle odometer information corresponding to time t0+2.
[0053] Of course, other methods can also be used to determine the vehicle's odometer information at different times, and this disclosure does not limit this approach. The vehicle's odometer information refers to the motion state information calculated from motion-related data collected by the vehicle's own sensors (such as wheel encoders, IMUs, etc.), used to characterize the vehicle's relative displacement, heading change, and velocity relative to the initial moment. Odometer information typically includes the vehicle's relative position change, heading angle change, linear velocity, and angular velocity, as well as the timestamps corresponding to these data. An odometer frame sequence refers to a collection of odometer information arranged chronologically, used to reflect the vehicle's continuous relative motion state over a period of time. By accumulating the relative motion increments in the odometer frame sequence, the changes in the vehicle's relative position, orientation, and velocity over time during the entire motion process can be obtained. In some examples, the vehicle's odometer information during driving can be determined using wheel encoders, inertial measurement units (IMUs), visual odometers (VO), and laser odometers (LO).
[0054] Step S2032: Based on the odometer information, determine the first position information and the second position information of the image sensor in the preset coordinate system at different times.
[0055] For example, in this embodiment of the disclosure, the first position information and the second position information of the image sensor on the vehicle in a preset coordinate system at different times can be determined based on the odometer information at different times. The first position information is the camera optical center position of the image sensor at the first image acquisition time, and the second position information is the camera optical center position of the image sensor at the second image acquisition time.
[0056] For example, assuming the first image is acquired at time t0 and the second image is acquired at time t0+2, the first position information of the image sensor in the preset coordinate system at time t0 can be determined based on the vehicle's odometer information from the initial time to time t0. Similarly, the second position information of the image sensor in the preset coordinate system at time t0+2 can be determined based on the vehicle's odometer information from the initial time to time t0+2. Wherein, if the preset coordinate system is the VCS coordinate system corresponding to when the vehicle begins to move, then the initial time is the time when the vehicle begins to move.
[0057] In other words, one of the functions of the odometer information in this embodiment is to convert the position of the image sensor at different times to a preset coordinate system.
[0058] Step S2033: Based on the first location information, the second location information, and the third location information of the first target sampling point, perform sampling point matching between the first lane line sampling point set and the second lane line sampling point set to obtain the sampling point matching pair corresponding to the first target sampling point.
[0059] For example, assuming the acquisition time of the first image is the first moment and the acquisition time of the second image is the second moment, in this embodiment of the disclosure, the first lane line sampling point set and the second lane line sampling point set can be matched based on the first position information of the image sensor in the preset coordinate system at the first moment, the second position information of the image sensor in the preset coordinate system at the second moment, and the third position information of the first target sampling point in the preset coordinate system, to obtain the sampling point matching pair corresponding to the first target sampling point.
[0060] For example, such as Figure 3BAs shown, line 1 represents the lane line in the actual environment, and line 2 is the extension line of the plane where the image sensor is located. Point O1 represents the first position information of the image sensor in the preset coordinate system at the first moment (i.e., the position of the camera optical center of the image sensor in the preset coordinate system at the first moment), and point O2 represents the second position information of the image sensor in the preset coordinate system at the second moment (i.e., the position of the camera optical center of the image sensor in the preset coordinate system at the second moment). Image 31 is the first image acquired at the first moment, and image 32 is the second image acquired at the second moment. Point d1 is a pixel on the lane line in the first image 31, and point D1 is the point transformed from pixel d1 to the preset coordinate system, which is also the first target sampling point; that is, point d1 is the pixel of the 3D coordinate point A on the lane line projected onto the camera imaging plane (i.e., image 31) at the first moment. Point d1 is located on the lane line in image 31, and its projection onto the preset coordinate system is point D1. Based on this, the property that the projections of points on the lane line are coplanar in two consecutive frames can be utilized. Based on points O1, O2, and D1, sampling point matching is performed on the first target sampling point D1 to obtain the sampling point matching pair (D1, R1) corresponding to the first target sampling point D1. Here, R1 is the sampling point in the second lane line sampling point set that matches the first target sampling point D1. Similarly, point r1 is the image pixel corresponding to point R1. That is, point r1 is the pixel of the 3D coordinate point A on the lane line projected onto the camera imaging plane (i.e., image 32) at the second moment. The straight line containing point r1 is the lane line in image 32, and its projection onto the preset coordinate system is point R1. Thus, the 3D coordinate point A on the lane line can be determined based on points O1, O2, D1, and R1. Furthermore, based on the property that the projections of points on the lane line are coplanar in two consecutive frames, it can be known that… Figure 3B Points O1, O2, D1, R1, d1, and r1 in the image are all located in the same plane. Furthermore, according to the principles of camera imaging, Figure 3B Midpoint O1, point d1, and point D1 are on the same straight line. Similarly, points O2, r1, and R1 are also on the same straight line.
[0061] The lane line determination method provided in this embodiment determines a first target sampling point in a first lane line sampling point set, and the odometer information of the vehicle at different times; based on the odometer information, it determines the first position information and the second position information of the image sensor in a preset coordinate system at different times.
[0062] Based on the first location information, the second location information, and the third location information of the first target sampling point, the first lane line sampling point set and the second lane line sampling point set are matched to obtain sampling point matching pairs. Thus, the sampling point matching pairs of the first target sampling point can be determined using the location information of the image sensor at different times and the location information of the first target sampling point. This method of determining the sampling point matching pairs of the first target sampling point does not require feature point extraction, does not require descriptors, and has low computational cost. Therefore, the lane line determination method in this embodiment can achieve lane line reconstruction simply, quickly, and accurately.
[0063] like Figure 4A As shown above, in the above Figure 3A Based on the illustrated embodiment, step S2033 may include steps S21-S23.
[0064] Step S21: Determine the target plane based on the first position information, the second position information, and the third position information.
[0065] For example, assume that the acquisition time of the first image is time 1, and the acquisition time of the second image is time 2. Based on the principle of three points being coplanar, a target plane can be determined based on the first position information of the image sensor in the preset coordinate system at time 1, the second position information of the image sensor in the preset coordinate system at time 2, and the third position information of the first target sampling point in the preset coordinate system.
[0066] Step S22: Based on the second location information, the second lane line sampling point set, and the target plane, determine the second target sampling point from the second lane line sampling point set.
[0067] For example, based on the property that the projections of points on the lane line onto the images in the preceding and following frames are coplanar, it can be known that the lane line in the second image and the target plane determined in step S21 intersect. Therefore, the positional relationship between the vector formed by each second sampling point in the second lane line sampling point set and the second position information, and the target plane, can be determined. Thus, based on this positional relationship, the second target sampling point is determined from the second lane line sampling point set.
[0068] Step S23: Determine the first target sampling point and the second target sampling point as a sampling point matching pair.
[0069] For example, after determining the second target sampling point from the second lane line sampling point set, the first target sampling point and the second target sampling point can be identified as a sampling point matching pair. Then, based on this sampling point matching pair, the lane line is determined. For example, as... Figure 3BAs shown, the first straight line O1D1 and the second straight line O2R1 can be determined based on the sampling point matching pair (D1, R1), and then the 3D coordinate point A on the lane line can be determined based on the first straight line O1D1 and the second straight line O2R1.
[0070] In some examples, such as Figure 3B As shown, the first straight line O1D1 and the second straight line O2R1 can be determined based on the sampling point matching pair (D1, R1). Furthermore, the intersection point between the first straight line O1D1 and the second straight line O2R1 can be determined as the 3D coordinate point A on the lane line. Since the coordinates of points O1, D1, O2, and R1 in the preset coordinate system are already determined, the 3D coordinates of point A can be directly determined based on the coordinates of points O1, D1, O2, and R1 in the preset coordinate system.
[0071] The lane line determination method provided in this disclosure determines a target plane based on first, second, and third position information; a second target sampling point is determined from the second lane line sampling point set based on the second position information, a second lane line sampling point set, and the target plane; and the first and second target sampling points are identified as a sampling point matching pair. Thus, by utilizing the characteristic that the projections of points on the lane line onto consecutive frames are coplanar, lane line sampling point matching pairs at different times can be determined, thereby determining the lane line. This method does not rely on the prior assumption of lane line parallelism, therefore it has a wide range of applications.
[0072] In some embodiments, step S22, determining a second target sampling point from the second lane line sampling point set based on the second location information, the second lane line sampling point set, and the target plane, includes: determining the normal vector of the target plane; determining a plurality of first vectors based on the second location information and the second lane line sampling point set; determining a first target vector from the plurality of first vectors based on the positional relationship between the first vector and the normal vector; and determining the second target sampling point from the second lane line sampling point set based on the first target vector.
[0073] For example, assuming the acquisition time of the first image is a first moment and the acquisition time of the second image is a second moment. In this embodiment of the disclosure, the normal vector of the target plane can be determined first, and then multiple first vectors can be determined based on the second position information of the sensor in the preset coordinate system at the second moment and multiple second sampling points in the second lane line sampling point set. Furthermore, based on the positional relationship between each of the multiple first vectors and the normal vector of the target plane, a first target vector can be determined from the multiple first vectors, and based on the first target vector, a second target sampling point can be determined from the second lane line sampling point set.
[0074] In some examples, a first target vector is determined from multiple first vectors based on the positional relationship between each first vector and the normal vector of the target plane. This includes: determining a first vector whose positional relationship is perpendicular to the target plane based on the positional relationship between each first vector and the normal vector of the target plane, and determining the first vector whose positional relationship is perpendicular to the target plane as the first target vector.
[0075] For example, such as Figure 4B As shown, assume the acquisition time of the first image is the first moment, and the acquisition time of the second image is the second moment. Point O1 represents the first position information of the image sensor in the preset coordinate system at the first moment (i.e., the position of the camera optical center of the image sensor in the preset coordinate system at the first moment), and point O2 represents the second position information of the image sensor in the preset coordinate system at the second moment (i.e., the position of the camera optical center of the image sensor in the preset coordinate system at the second moment). Point d1 is the image pixel corresponding to the first target sampling point, which is also the pixel of the 3D coordinate point A on the lane line projected onto the camera imaging plane (i.e., the first image) at the first moment. Furthermore, the plane formed by points O1, O2, and d1 is the target plane in step S21, where n is the normal vector of the target plane. Thus, assume the second lane line sampling point set includes the second sampling points {R1, R2, ... R...} s Then, multiple first vectors O2R1, O2R2, ..., and so on, until O2R s Furthermore, the positional relationship between each first vector and the normal vector n can be determined. If the first vector O2R1 and the normal vector n are perpendicular, then the first vector O2R1 is determined as the first target vector, and the point R1 that constitutes this first target vector is the second target sampling point. Where r1 is the second target sampling point R1(…). Figure 4B The image pixel corresponding to (not shown) is also the pixel of the 3D coordinate point A on the lane line projected onto the camera imaging plane (i.e., the second image) at the second moment. After determining the second target sampling point, the 3D coordinate point A on the lane line can be determined based on the first target sampling point, the second target sampling point, the first position information, and the second position information.
[0076] like Figure 5 As shown above, in the above Figure 2 Based on the illustrated embodiment, step S204 may include steps S2041-S2044.
[0077] Step S2041: Determine the first straight line based on the first target sampling point in the first position information and the sampling point matching pair.
[0078] For example, assume the acquisition time of the first image is time 1, and the acquisition time of the second image is time 2. Based on the principle of two points forming a line, a first straight line can be determined based on the first position information of the sensor in the preset coordinate system at the first moment and the position of the first target sampling point. For example, as... Figure 3B As shown, point O1 is the first position information of the image sensor in the preset coordinate system at the first moment, and point D1 is the three-dimensional position information of the first target acquisition point in the preset coordinate system in the sampling point matching pair, thereby determining the first straight line O1D1; wherein, point O1, point d1 and point D1 are located on the same straight line.
[0079] Step S2042: Determine the second straight line based on the second location information and the second target sampling point in the sampling point matching pair.
[0080] For example, assume the first image is acquired at time 1, and the second image is acquired at time 2. Based on the principle of two points forming a line, the second straight line can be determined based on the second position information of the sensor in the preset coordinate system at time 2 and the position of the second target sampling point. For example, as... Figure 3B As shown, point O2 represents the second position information of the image sensor in the preset coordinate system at the second time, and point R1 represents the three-dimensional position information of the second target acquisition point in the preset coordinate system in the sampling point matching pair, thereby determining the second straight line O2R1; wherein, point O2, point r1 and point R1 are located on the same straight line.
[0081] Step S2043: Determine the intersection point of the first straight line and the second straight line, and define the intersection point as a three-dimensional lane line point on the lane line.
[0082] For example, in this embodiment of the disclosure, the intersection point between the first straight line and the second straight line can be determined, and this intersection point is a three-dimensional lane line point on the lane line. For example, as Figure 3B As shown, the intersection point A of lines O1D1 and O2R1 in the preset coordinate system can be calculated to obtain the three-dimensional coordinates of the intersection point A. Furthermore, in this embodiment of the present disclosure, multiple three-dimensional lane line points on the lane line can be determined by traversing each first sampling point in the first lane line sampling point set, thereby obtaining the 3D point set of the lane line. This achieves the three-dimensional reconstruction of the lane line.
[0083] Step S2044: Determine the lane lines based on the three-dimensional lane line points.
[0084] For example, by matching the sampling points of the first lane line sampling point set with those of the second lane line sampling point set, at least one sampling point matching pair can be obtained. Then, based on this at least one sampling point matching pair, a corresponding number of three-dimensional lane line points can be determined, thereby determining the lane line based on this corresponding number of three-dimensional lane line points. For instance, by matching the sampling points of the first lane line sampling point set with those of the second lane line sampling point set, 20 sampling point matching pairs are determined. Then, based on these 20 sampling point matching pairs, the three-dimensional lane line points corresponding to each sampling point matching pair can be determined, thus using these 20 three-dimensional lane line points to determine the lane line.
[0085] The lane line determination method provided in this embodiment determines a first straight line based on a first target sampling point in a first position and sampling point matching pair; determines a second straight line based on a second target sampling point in a second position and sampling point matching pair; determines the intersection point of the first and second straight lines and identifies the intersection point as a three-dimensional lane line point on the lane line; and determines the lane line based on the three-dimensional lane line point. Thus, since the three-dimensional vector coordinates of the first and second straight lines are fixed, the 3D coordinates of the three-dimensional lane line point determined by the intersection point of the first and second straight lines are also very accurate.
[0086] In some embodiments, when there are multiple first target sampling points, step S2044, determining the lane line based on the three-dimensional lane line points, includes: determining the three-dimensional lane line points corresponding to the multiple first target sampling points respectively; and reconstructing the lane line based on the multiple three-dimensional lane line points.
[0087] For example, in this embodiment of the disclosure, the first sampling points in the first lane line sampling point set can be traversed, so that in each traversal, the corresponding three-dimensional lane line points can be determined based on the currently used first target sampling points. In this way, the lane lines can be reconstructed based on the three-dimensional lane line points corresponding to each first target sampling point.
[0088] like Figure 6 As shown above, in the above Figure 2 Based on the illustrated embodiment, step S202 may include steps S2021-S2024.
[0089] Step S2021: Perform semantic segmentation processing on the first image and the second image respectively to obtain the first semantic segmentation map and the second semantic segmentation map.
[0090] For example, in this embodiment of the disclosure, semantic segmentation processing can be performed on the first image to obtain a first semantic segmentation map; similarly, semantic segmentation processing can also be performed on the second image to obtain a second semantic segmentation map. Semantic segmentation processing refers to the process of assigning a category label to each pixel in the image, and then, after clustering neighboring pixels in the image according to the same category, the image can be divided into several regions with different semantic meanings.
[0091] For example, deep learning methods can be used to process the first image to obtain a first semantic segmentation map labeled with lane line categories; similarly, deep learning methods can be used to process the second image to obtain a second semantic segmentation map labeled with lane line categories. Of course, other methods (such as traditional machine learning methods) can also be used to perform semantic segmentation processing on the first and second images, and this disclosure does not limit this.
[0092] Step S2022: Sample the lane lines in the first image based on the first semantic segmentation map to obtain the first lane line pixel set.
[0093] For example, since the first semantic segmentation map is an image labeled with lane line categories, lane lines in the first image can be sampled based on the first semantic segmentation map to obtain a first set of lane line pixels.
[0094] In some examples, the first semantic segmentation map obtained in step S2021 can be used to first extract the pixels containing lane lines in the first semantic segmentation map, and then the sampling points of the lane lines can be determined based on the pixels of the lane lines. For example, the sampling points of the lane lines can be extracted by RLE (Run-Length Encoding).
[0095] Step S2023: Sample the lane lines in the second image based on the second semantic segmentation map to obtain the second lane line pixel set.
[0096] It is understandable that the implementation of sampling the lane lines in the second image based on the second semantic segmentation map in step S2023 to obtain the second lane line pixel set can be referred to step S2022, and will not be repeated here.
[0097] Step S2024: Perform coordinate transformation on the first lane line pixel set and the second lane line pixel set respectively to obtain the first lane line sampling point set and the second lane line sampling point set located in the preset coordinate system.
[0098] For example, the multiple first pixels included in the first lane line pixel set sampled in step S2022 are located in the image coordinate system. Therefore, coordinate transformation processing can be performed on them based on the parameter information of the image sensor and the odometer information at different times, transforming the multiple first pixels to a preset coordinate system to obtain a first lane line sampling point set including multiple first sampling points. For example, as... Figure 4B As shown, according to the principle of pinhole imaging, point d1 is the first pixel in the set of pixels of the first lane line, and point D1 is the first sampling point obtained after coordinate transformation of the first pixel. The first sampling point is located in the preset coordinate system.
[0099] In some examples, the image sensor's parameter information includes at least camera intrinsic and extrinsic parameters. Camera intrinsic parameters describe the camera's own optical and geometric characteristics; based on these intrinsic parameters and pixel depth, pixel coordinates can be projected onto the camera coordinate system. Camera intrinsic parameters mainly include focal length, principal point coordinates, and distortion coefficients. Camera extrinsic parameters primarily describe the relative relationship between the camera coordinate system and the vehicle coordinate system, including the displacement of the coordinate system origin and the rotation of the coordinate system orientation. Based on the camera extrinsic parameters, 3D points in the camera coordinate system can be projected onto the vehicle coordinate system.
[0100] It is understandable that the implementation of coordinate transformation of the second lane line pixel set to obtain the second lane line sampling point set located in the preset coordinate system can refer to the implementation of the processing of the first lane line pixel set, and will not be elaborated here.
[0101] The lane line determination method provided in this disclosure involves performing semantic segmentation on a first image and a second image respectively to obtain a first semantic segmentation map and a second semantic segmentation map; sampling lane lines in the first image based on the first semantic segmentation map to obtain a first lane line pixel set; sampling lane lines in the second image based on the second semantic segmentation map to obtain a second lane line pixel set; and performing coordinate transformation on the first lane line pixel set and the second lane line pixel set respectively to obtain a first lane line sampling point set and a second lane line sampling point set located in a preset coordinate system. In this way, the first lane line pixel set and the second lane line pixel set sampled at different times can be transformed into a unified coordinate system for matching processing, thereby determining the three-dimensional lane line points.
[0102] Exemplary device
[0103] Figure 7 A lane line determination device provided in this disclosure embodiment, such as Figure 7 As shown, the lane line determination device 700 includes an image determination module 701, a lane line sampling module 702, a sampling point matching module 703, and a lane line determination module 704.
[0104] The image determination module 701 is used to determine the first image and the second image acquired by the vehicle's image sensor at different times.
[0105] The lane line sampling module 702 is used to determine a first lane line sampling point set and a second lane line sampling point set based on a first image and a second image, respectively.
[0106] The sampling point matching module 703 is used to match the sampling points of the first lane line sampling point set with the sampling point set of the second lane line to obtain a sampling point matching pair.
[0107] Lane line determination module 704 is used to determine lane lines based on sample point matching pairs.
[0108] In some embodiments, such as Figure 8 As shown, the sampling point matching module 703 includes an information determination unit 7031, a position determination unit 7032, and a sampling point matching unit 7033.
[0109] The information determination unit 7031 is used to determine the first target sampling point in the first lane line sampling point set, as well as the odometer information of the vehicle at different times;
[0110] The position determination unit 7032 is used to determine the first position information and the second position information of the image sensor in a preset coordinate system at different times based on the odometer information.
[0111] The sampling point matching unit 7033 is used to match the sampling points of the first lane line sampling point set with the sampling point set of the second lane line sampling point set based on the first location information, the second location information and the third location information of the first target sampling point, so as to obtain the sampling point matching pair corresponding to the first target sampling point.
[0112] In some embodiments, the sampling point matching unit 7033 is specifically used to determine a target plane based on the first location information, the second location information, and the third location information; determine a second target sampling point from the second lane line sampling point set based on the second location information, the second lane line sampling point set, and the target plane; and determine the first target sampling point and the second target sampling point as a sampling point matching pair.
[0113] In some embodiments, the sampling point matching unit 7033 is specifically configured to: determine a target plane based on first position information, second position information, and third position information; determine the normal vector of the target plane; determine a plurality of first vectors based on the second position information and the second lane line sampling point set; determine a first target vector from the plurality of first vectors based on the positional relationship between the first vector and the normal vector; determine a second target sampling point from the second lane line sampling point set based on the first target vector; and determine the first target sampling point and the second target sampling point as a sampling point matching pair.
[0114] In some embodiments, such as Figure 9 As shown, the lane line determination module 704 includes a first straight line determination unit 7041, a second straight line determination unit 7042, an intersection point determination unit 7043, and a lane line determination unit 7044.
[0115] The first straight line determination unit 7041 is used to determine the first straight line based on the first position information and the first target sampling point in the sampling point matching pair;
[0116] The second line determination unit 7042 is used to determine a second line based on the second position information and the second target sampling point in the sampling point matching pair;
[0117] The intersection point determination unit 7043 is used to determine the intersection point of the first straight line and the second straight line, and to determine the intersection point as a three-dimensional lane line point on the lane line;
[0118] Lane line determination unit 7044 is used to determine lane lines based on three-dimensional lane line points.
[0119] In some embodiments, when there are multiple first target sampling points, the lane line determination unit 7044 is specifically used to determine the three-dimensional lane line points corresponding to the multiple first target sampling points respectively; and to reconstruct the lane line based on the multiple three-dimensional lane line points.
[0120] In some embodiments, the lane line sampling module 702 is specifically used to perform semantic segmentation processing on the first image and the second image respectively to obtain a first semantic segmentation map and a second semantic segmentation map; to sample the lane lines in the first image based on the first semantic segmentation map to obtain a first lane line pixel set; to sample the lane lines in the second image based on the second semantic segmentation map to obtain a second lane line pixel set; and to perform coordinate transformation processing on the first lane line pixel set and the second lane line pixel set respectively to obtain a first lane line sampling point set and a second lane line sampling point set located in a preset coordinate system.
[0121] The beneficial technical effects corresponding to the exemplary embodiment of the lane line determination device 700 described above can be found in the corresponding beneficial technical effects in the exemplary method section above, and will not be repeated here.
[0122] Exemplary electronic devices
[0123] Figure 10 The present disclosure provides a structural diagram of an electronic device 100, which includes at least one processor 101 and a memory 102.
[0124] The processor 101 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.
[0125] The memory 102 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 101 may execute one or more computer program instructions to implement the lane line determination methods and / or other desired functions of the various embodiments of this disclosure described above.
[0126] In one example, the electronic device 100 may also include an input device 103 and an output device 104, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0127] The input device 103 may also include, for example, a keyboard, a mouse, etc.
[0128] The output device 104 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0129] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 100 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 100 may include any other suitable components depending on the specific application.
[0130] Exemplary computer program products and computer-readable storage media
[0131] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the lane line determination method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.
[0132] Computer program products can be written in any combination of one or more programming languages to perform the operations of embodiments of this disclosure. These programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0133] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the lane line determination method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.
[0134] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0135] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0136] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.
Claims
1. A method for determining lane lines, comprising: Determine the first and second images captured by the vehicle's image sensors at different times; Based on the first image and the second image, the first lane line sampling point set and the second lane line sampling point set are determined respectively; The first lane line sampling point set and the second lane line sampling point set are matched to obtain sampling point matching pairs; The lane line is determined based on the sampling point matching pairs.
2. The method according to claim 1, wherein, The step of matching the first lane line sampling point set with the second lane line sampling point set to obtain sampling point matching pairs includes: Determine the first target sampling point in the first lane line sampling point set, and the odometer information of the vehicle at different times; Based on the odometer information, the first position information and the second position information of the image sensor in the preset coordinate system at different times are determined respectively; Based on the first location information, the second location information, and the third location information of the first target sampling point, the first lane line sampling point set and the second lane line sampling point set are matched to obtain the sampling point matching pair corresponding to the first target sampling point.
3. The method according to claim 2, wherein, The step of matching the first lane line sampling point set with the second lane line sampling point set based on the first location information, the second location information, and the third location information of the first target sampling point to obtain the sampling point matching pair corresponding to the first target sampling point includes: Based on the first location information, the second location information, and the third location information, the target plane is determined; Based on the second location information, the second lane line sampling point set, and the target plane, a second target sampling point is determined from the second lane line sampling point set; The first target sampling point and the second target sampling point are determined as a sampling point matching pair.
4. The method according to claim 3, wherein, The step of determining the second target sampling point from the second lane line sampling point set based on the second location information, the second lane line sampling point set, and the target plane includes: Determine the normal vector of the target plane; Based on the second location information and the second lane line sampling point set, multiple first vectors are determined; Based on the positional relationship between the first vector and the normal vector, a first target vector is determined from a plurality of first vectors; Based on the first target vector, the second target sampling point is determined from the second lane line sampling point set.
5. The method according to any one of claims 2-4, wherein, Determining the lane line based on the sample point matching pairs includes: Based on the first location information and the first target sampling point in the sampling point matching pair, a first straight line is determined; Based on the second location information and the second target sampling point in the sampling point matching pair, a second straight line is determined; Determine the intersection point of the first straight line and the second straight line, and define the intersection point as a three-dimensional lane line point on the lane line; The lane line is determined based on the three-dimensional lane line points.
6. The method according to claim 5, wherein, When there are multiple first target sampling points, determining the lane line based on the three-dimensional lane line points includes: Determine the three-dimensional lane line points corresponding to the multiple first target sampling points respectively; The lane lines are reconstructed based on multiple three-dimensional lane line points.
7. The method according to claim 1, wherein, The step of determining the first lane line sampling point set and the second lane line sampling point set based on the first image and the second image respectively includes: Semantic segmentation is performed on the first image and the second image respectively to obtain a first semantic segmentation map and a second semantic segmentation map; Based on the first semantic segmentation map, the lane lines in the first image are sampled to obtain the first lane line pixel set; Based on the second semantic segmentation map, the lane lines in the second image are sampled to obtain the second lane line pixel set; The first lane line pixel set and the second lane line pixel set are respectively subjected to coordinate transformation processing to obtain the first lane line sampling point set and the second lane line sampling point set located in the preset coordinate system.
8. A lane marking device, comprising: An image determination module is used to determine the first and second images captured by the vehicle's image sensors at different times. The lane line sampling module is used to determine a first lane line sampling point set and a second lane line sampling point set based on the first image and the second image, respectively. The sampling point matching module is used to match the first lane line sampling point set with the second lane line sampling point set to obtain a sampling point matching pair. The lane line determination module is used to determine the lane line based on the sampling point matching pairs.
9. A computer-readable storage medium storing a computer program for performing the lane line determination method according to any one of claims 1-7.
10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the lane line determination method according to any one of claims 1-7.