Lane line determination method and device, equipment and storage medium

Abstract

The invention discloses a lane line determination method and device, equipment and a storage medium. The method comprises the steps that a first image and a second image collected by an image sensor of a vehicle at different moments are determined; based on the first image and the second image, respectively determining a first lane line key point set and a second lane line key point set; determining the displacement of the vehicle between the first moment and the second moment and the height information of the image sensor; and determining a lane line based on the displacement, the first lane line key point set, the second lane line key point set and height information of the image sensor. According to the technical scheme of the invention, the three-dimensional coordinates of the lane line are determined according to the lane line key points of the images collected by the same sensor at different moments, the displacement of the vehicle and the height information of the image sensor. The three-dimensional reconstruction of the lane line can be simply, quickly and accurately realized under the conditions of a single camera and a dotted ramp lane line.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a method, apparatus, device and storage medium for determining lane lines. Background Technology

[0002] In intelligent driving scenarios, collecting environmental information about the vehicle's surroundings using sensors such as cameras and radar, and then fitting and reconstructing lane lines based on this information for downstream use, is a crucial operation. Low-cost driver assistance systems typically use only a forward-facing camera. Therefore, it's necessary to remodel the real-world lane lines based on the images captured by the forward-facing camera, generating a local map with lane lines.

[0003] The most common method in existing technology is to calculate the 3D (three-dimensional) coordinates of lane lines based on the assumption of lane line parallelism (i.e., assuming that the width of the same lane remains unchanged within a local area) and the height and calibration parameters of the forward-looking camera. However, the lane line height determined by this method is often inaccurate in certain scenarios (such as curve scenarios, road forks, etc.).

[0004] Therefore, how to use images captured by a single camera to obtain more accurate lane line heights and thus improve the effect of traditional forward-looking lane line reconstruction of 3D lane lines has become a hot research topic in low-cost driver assistance systems. Summary of the Invention

[0005] Typically, lane line reconstruction using a single camera in related technologies is based on the assumption that lane lines are parallel. However, in specific scenarios (such as curves or road forks), the height error of the 3D lane lines calculated by this method is relatively large. To address the above-mentioned technical problems, this disclosure provides a method, apparatus, device, and storage medium for determining lane lines, which can solve the problem of large lane line reconstruction errors in specific scenarios in related technologies.

[0006] A first aspect of this disclosure provides a method for determining lane lines. The method includes: determining a first image captured by an image sensor of a vehicle at a first moment and a second image captured at a second moment; determining a first set of key points for lane lines and a second set of key points for lane lines based on the first image and the second image; determining the displacement of the vehicle between the first moment and the second moment, and the height information of the image sensor; and determining lane lines based on the displacement, the first set of key points for lane lines, the second set of key points for lane lines, and the height information of the image sensor.

[0007] A second aspect of this disclosure provides a lane line determination apparatus, comprising: an image determination module for determining a first image acquired by a vehicle's image sensor at a first moment and a second image acquired at a second moment; a key point determination module for determining a first set of lane line key points and a second set of lane line key points based on the first and second images; a parameter determination module for determining the vehicle's displacement between the first and second moments and the height information of the image sensor; and a lane line determination module for determining lane lines based on the displacement, the first set of lane line key points, the second set of lane line key points, and the height information of the image sensor.

[0008] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program for performing the lane line determination method provided in the first aspect embodiment.

[0009] A fourth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the lane line determination method provided in the first aspect embodiment above.

[0010] A fifth aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs the lane line determination method provided in the first aspect of the present invention.

[0011] Based on the lane line determination method provided in this disclosure, a first image captured by the vehicle's image sensor at a first moment and a second image captured at a second moment are determined; based on the first and second images, a first lane line key point set and a second lane line key point set are determined; the vehicle's displacement between the first and second moments and the image sensor's height information are determined; and based on the displacement, the first lane line key point set, the second lane line key point set, and the image sensor's height information, the lane line is determined. Since this embodiment can first determine the first lane line key point set corresponding to the first image and the second lane line key point set corresponding to the second image, and then combine the vehicle's displacement and the image sensor's height information at different moments to determine the three-dimensional coordinates of the lane line, it can achieve three-dimensional reconstruction of lane lines with low cost, low computational load, and a single camera. Furthermore, since the lane line determination method in this embodiment does not rely on the prior assumption of lane line parallelism, the method has a wide range of applications. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a vehicle in a driving scenario provided for an exemplary embodiment of this disclosure.

[0013] Figure 2 A schematic flowchart illustrating a method for determining lane lines provided as an exemplary embodiment of this disclosure.

[0014] Figure 3 A flowchart illustrating a method for determining lane lines provided as another exemplary embodiment of this disclosure.

[0015] Figure 4A A schematic flowchart of a method for determining lane lines provided as yet another exemplary embodiment of this disclosure.

[0016] Figure 4B A schematic diagram illustrating the principle of a lane line determination method provided for an exemplary embodiment of this disclosure.

[0017] Figure 4C A schematic diagram illustrating the principle of a lane line determination method provided for another exemplary embodiment of this disclosure.

[0018] Figure 4D A schematic diagram illustrating the principle of a lane line determination method provided as yet another exemplary embodiment of this disclosure.

[0019] Figure 5 A schematic flowchart of a method for determining lane lines provided as yet another exemplary embodiment of this disclosure.

[0020] Figure 6 A schematic flowchart of a method for determining lane lines provided as yet another exemplary embodiment of this disclosure.

[0021] Figure 7 A schematic flowchart of a method for determining lane lines provided as yet another exemplary embodiment of this disclosure.

[0022] Figure 8 A schematic flowchart of a method for determining lane lines provided as yet another exemplary embodiment of this disclosure.

[0023] Figure 9 A schematic diagram of the structure of a lane line determination device provided for an exemplary embodiment of this disclosure.

[0024] Figure 10 A schematic diagram of the structure of a lane line determination device provided for another exemplary embodiment of this disclosure.

[0025] Figure 11 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0026] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0027] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0028] Application Overview

[0029] First, the application scenarios of this disclosure will be introduced. The lane line determination method provided in the embodiments of this disclosure can be applied to, for example, autonomous driving scenarios, assisted driving scenarios, and any other feasible scenarios.

[0030] like Figure 1 As shown, the vehicle 10 is equipped with a single-view image sensor 11 (such as a front-view image sensor). This image sensor 11 can then capture images of the lane lines 12 in front of the vehicle 10, obtaining a first image captured by the image sensor 11 at a first moment and a second image captured by the image sensor 11 at a second moment. For example, the first image captured by the image sensor 11 at time t0 and the second image captured by the image sensor 11 at time t0+n can be determined; where n is a natural number greater than or equal to 1. Both the first and second images include the lane lines 12. It should be emphasized that, depending on the field of view (FOV) of the image sensor, the images captured by the image sensor may include lane lines from other lanes in addition to the lane line of the vehicle's lane.

[0031] Furthermore, the lane line determination method provided in this embodiment can be used to process the first image and the second image to obtain the three-dimensional coordinates of multiple points in the lane line 12. Then, the three-dimensional coordinates of the multiple points in the lane line 12 can be used to perform 3D reconstruction of the lane line 12. It should be noted that... Figure 1 This disclosure is merely an illustrative example and does not represent the actual situation. Figure 1 The location and type of the image sensor 11 are not limited. Of course, the vehicle 10 may also include other sensors besides the image sensor 11 (such as other image sensors, radar sensors, etc.), and this disclosure does not limit this.

[0032] In related technologies, when determining lane lines using a single-view image sensor, it is usually necessary to assume that the left and right lane lines of the same lane are parallel to each other (i.e., the width of the same lane must remain consistent within a local area). Then, using the assumption of constant width, combined with the height of the single-view image sensor and camera calibration parameters, the 3D coordinates of the lane line are calculated. Because this method relies on the prior assumption of parallel lane lines, in certain scenarios (such as curves or road forks), the 3D lane lines calculated by this method have significant errors, especially in the determined 3D lane line height, which may even contain obvious errors.

[0033] To address the aforementioned problems, this disclosure provides a method for determining lane lines. This method involves determining a first image captured by a vehicle's image sensor at a first moment and a second image captured at a second moment; determining a first set of key points for the lane line and a second set of key points for the lane line based on the first and second images; determining the vehicle's displacement between the first and second moments, as well as the image sensor's height information; and determining the lane line based on the displacement, the first set of key points for the lane line, the second set of key points for the lane line, and the image sensor's height information. Since this disclosure can first determine the first set of key points for the lane line corresponding to the first image and the second set of key points for the lane line corresponding to the second image, and then combine the vehicle's displacement at different moments and the image sensor's height information to determine the three-dimensional coordinates of the lane line, it can achieve three-dimensional reconstruction of lane lines with low cost, low computational load, and a single camera. Furthermore, since the lane line determination method in this disclosure does not rely on the prior assumption of lane line parallelism, the method has a wide range of applications.

[0034] Exemplary methods

[0035] Figure 2 This is a flowchart illustrating a lane line determination method provided as an exemplary embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices, such as... Figure 2 As shown, the method includes steps S201-S204.

[0036] Step S201: Determine the first image acquired by the vehicle's image sensor at the first moment and the second image acquired at the second moment.

[0037] For example, such as Figure 1As shown, a first image captured by the image sensor 11 on the vehicle 10 (automobile) at a first moment, and a second image captured by the same image sensor 11 at a second moment, can be acquired first. For example, during the driving of the vehicle 10, the image sensor 11 on the vehicle 10 acquires images of the lane lines 12 ahead (acquiring images at preset time intervals or continuously), obtaining multiple images including the lane lines 12. Then, the first image and the second image can be determined from the multiple images including the lane lines 12. The first image and the second image can be two frames corresponding to adjacent acquisition moments, or two frames corresponding to any two acquisition moments; this embodiment does not impose any limitations on this.

[0038] It should be noted that the first image and the second image may include the same lane line or different lane lines. When the first image and the second image include the same lane line 12, the 3D coordinates of the lane line 12 can be obtained using the lane line determination method in this embodiment, thereby achieving lane line reconstruction. When the first image and the second image do not include the same lane line, the 3D coordinates of the lane line cannot be determined if the lane line determination method in this embodiment is executed.

[0039] Step S202: Based on the first image and the second image, determine the first lane line key point set and the second lane line key point set.

[0040] For example, in this embodiment of the disclosure, a first set of key points for lane lines can be determined based on a first image, and a second set of key points for lane lines can be determined based on a second image. The first set of key points for lane lines includes multiple first lane line key points, and the second set of key points for lane lines includes multiple second lane line key points. It should be noted that if the first or second image does not include lane lines, performing the method in step S202 above will not yield the corresponding first set of key points or the corresponding second set of key points for lane lines, thus making it impossible to determine the 3D coordinates of the lane lines.

[0041] For example, lane line key points can be extracted from the first image acquired at a first time and the second image acquired at a second time, respectively, to determine the first set of lane line key points and the second set of lane line key points. For low-cost driver assistance systems, since the CPU (Central Processing Unit) has relatively low computing power, the FAST (Features from Accelerated Segment Test) algorithm can be used to extract lane line key points from the first and second images respectively. Of course, if the CPU has higher computing power, the SIFT (Scale-Invariant Feature Transform) algorithm can also be used to extract lane line key points from the first and second images respectively. That is to say, the algorithm for lane line key point extraction in this embodiment is not limited, and those skilled in the art can choose according to the actual application.

[0042] In some examples, to further reduce the workload of lane line key point extraction, embodiments of this disclosure can perform semantic segmentation on the first image and the second image respectively, obtaining a first semantic segmentation map corresponding to the first image and a second semantic segmentation map corresponding to the second image. Then, lane line key points can be extracted from the first image based on the first semantic segmentation map, and from the second image based on the second semantic segmentation map; thus, key points can be extracted only in the target region (such as the region including lane lines), thereby significantly narrowing the scope of lane line key point extraction and reducing the computational load.

[0043] Step S203: Determine the vehicle's displacement between the first and second moments, as well as the height information from the image sensor.

[0044] For example, in this embodiment of the disclosure, the odometer information of the vehicle at a first time point and a second time point can be determined first, and then the displacement of the vehicle between the first time point and the second time point can be determined based on the odometer information. The odometer information refers to the motion state information of the vehicle relative to its initial time point, calculated from motion-related data collected by its own sensors (such as wheel encoders, IMUs, etc.), and used to characterize the vehicle's relative displacement, heading change, and velocity. Odometer information typically includes the vehicle's relative position change, heading angle change, linear velocity, and angular velocity, as well as the timestamps corresponding to the above data. An odometer frame sequence refers to a collection of odometer information arranged chronologically, used to reflect the vehicle's continuous relative motion state over a period of time. By accumulating the relative motion increments in the odometer frame sequence, the changes in the vehicle's relative position, orientation, and velocity over time during the entire motion process can be obtained. In some examples, the odometer information of the vehicle during driving can be determined using wheel encoders, inertial measurement units (IMUs), visual odometers (VOs), and laser odometers (LOs). Furthermore, the method for determining odometer information is not limited in the embodiments disclosed herein.

[0045] For example, the odometer pose of the vehicle at the first moment is (x1, y1, θ1), and the odometer pose of the vehicle at the second moment is (x2, y2, θ2), where (x1, y1) are the position coordinates of the vehicle at the first moment, and θ1 is the orientation angle of the vehicle at the first moment. (x2, y2) are the position coordinates of the vehicle at the second moment, and θ2 is the orientation angle of the vehicle at the second moment. Then, the displacement d of the vehicle between the first and second moments can be determined by the following formula (1):

[0046]

[0047] In some examples, the vehicle's displacement between the first and second moments can also be determined based on carrier phase positioning technology (RTK). Of course, other methods can also be used to determine the vehicle's displacement between the first and second moments, and this disclosure does not limit this approach.

[0048] In some embodiments, it is also necessary to determine the height information of the vehicle's image sensor. The height information of the image sensor refers to the vertical distance between the image sensor and the ground plane it is located on, and this height information can be determined based on the parameters of the image sensor and the vehicle's parameters.

[0049] Step S204: Determine the lane lines based on displacement, the set of key points for the first lane line, the set of key points for the second lane line, and the height information from the image sensor.

[0050] For example, lane line keypoint matching pairs can be determined first based on displacement, a first set of lane line keypoints, and a second set of lane line keypoints; wherein the first and second lane line keypoints in the matching pairs correspond to the same physical point in the physical world. Then, the height of the lane line is determined based on the matching pairs, displacement, and height information from the image sensor. Finally, the 3D lane line can be determined by combining the lane line height with the lane line's two-dimensional coordinate position.

[0051] The lane line determination method provided in this disclosure involves determining a first image captured by a vehicle's image sensor at a first moment and a second image captured at a second moment; determining a first lane line key point set and a second lane line key point set based on the first and second images; determining the vehicle's displacement between the first and second moments, as well as the image sensor's height information; and determining the lane line based on the displacement, the first lane line key point set, the second lane line key point set, and the image sensor's height information. Since this disclosure can first determine the first lane line key point set corresponding to the first image and the second lane line key point set corresponding to the second image, and then combine the vehicle's displacement and the image sensor's height information at different moments to determine the three-dimensional coordinates of the lane line, it can achieve three-dimensional reconstruction of lane lines with low cost, low computational load, and a single camera. Furthermore, since the lane line determination method in this disclosure does not rely on the prior assumption of lane line parallelism, the method has a wide range of applications.

[0052] like Figure 3 As shown above, in the above Figure 2 Based on the illustrated embodiment, step S204 may include steps S2041-S2043.

[0053] Step S2041: Based on the displacement, the first lane line key point set, and the second lane line key point set, determine the lane line key point matching pair.

[0054] For example, in this embodiment of the disclosure, a first target key point in the first set of key points for the first lane line, and sensor parameters of the image sensor can be determined first. The first target key point can be any point among multiple first lane line key points, or it can be a specific point among multiple first lane line key points; this embodiment of the disclosure does not impose any limitation on this. Meanwhile, the sensor parameters of the image sensor include, but are not limited to, camera intrinsic parameters and camera extrinsic parameters. Camera intrinsic parameters are parameters used to describe the optical and geometric characteristics of the camera itself, mainly including focal length, principal point coordinates, and distortion coefficients. Camera extrinsic parameters are mainly used to describe the relative relationship between the camera coordinate system and the vehicle coordinate system, including the displacement of the coordinate system origin and the rotation of the coordinate system orientation.

[0055] Furthermore, based on the first target key point and the sensor parameters of the image sensor, the height range of the lane line point corresponding to the first target key point in the physical world can be determined; wherein, the height range of the lane line point corresponding to the first target key point in the physical world refers to the possible range of height values ​​of the lane line point corresponding to the first target key point in the physical world.

[0056] Finally, based on the first target key point, the height range corresponding to the first target key point, the vehicle's displacement between the first and second moments, the height information of the image sensor, and the sensor parameters of the image sensor, the target region in the second image can be determined. Then, based on the target region and the set of second lane line key points, the lane line key point matching pair of the first target key point can be determined. For example, if point p1 on the lane line in the first image is the first target key point, and point p1 and point p2 on the lane line in the second image correspond to the same point P on the lane line in the physical world, then the target region in the second image can be determined based on the first target key point p1, the height range of the lane line point P in the physical world, the vehicle's displacement between different moments, the height information of the image sensor, and the sensor parameters of the image sensor. Finally, based on the target region and the set of second lane line key points, the lane line key point matching pair (p1, p2) of the first target key point p1 can be determined.

[0057] Step S2042: Determine the height of the lane line based on the lane line key point matching pairs, displacement, and height information from the image sensor.

[0058] For example, a first distance can be determined based on lane line key point matching pairs; and the height of the lane line can be determined based on the first distance, the vehicle displacement between the first and second moments, and the height information of the image sensor above the ground plane.

[0059] For example, when the lane line key point matching pair includes a first target key point and a second target key point, coordinate transformation processing can be performed on the first and second target key points in the lane line key point matching pair to obtain a first three-dimensional lane line point and a second three-dimensional lane line point located in a preset coordinate system. Further, a first distance can be determined based on the first and second three-dimensional lane line points; and the height of the lane line can be determined based on the first distance, the vehicle's displacement between the first and second moments, and the height information of the image sensor above the ground plane. For example, if the lane line key point matching pair of the first target key point is (p1, p2), coordinate transformation processing is performed on p1 and p2 respectively (transforming them from the image coordinate system to the preset coordinate system) to obtain the coordinates of point p1 in the preset coordinate system, i.e., point C, and the coordinates of point p2 in the preset coordinate system, i.e., point D. Then, the distance between points C and D is the first distance.

[0060] The preset coordinate system can be the VCS (Vehicle Coordinate System) coordinate system corresponding to the vehicle when it starts moving. The VCS coordinate system refers to a coordinate system centered on the vehicle, with its origin at a specific reference point on the vehicle (such as the center of the rear axle), the X-axis pointing in the direction of the vehicle's forward movement, the Y-axis pointing to the left or right side of the vehicle, and the Z-axis pointing upwards towards the vehicle and perpendicular to the plane formed by the XY axes. Of course, the preset coordinate system in this embodiment can also be other predefined coordinate systems (such as the world coordinate system), and this embodiment does not impose any limitations on this.

[0061] Step S2043: Determine the lane lines based on their height.

[0062] For example, the two-dimensional coordinates of the lane line can be determined first, and then the 3D coordinates of the lane line can be obtained by combining the lane line height determined in step S2042.

[0063] The lane line determination method provided in this disclosure determines lane line key point matching pairs based on displacement, a first set of lane line key points, and a second set of lane line key points; determines the lane line height based on the lane line key point matching pairs, displacement, and height information from the image sensor; and determines the lane line based on the lane line height. In this way, the lane line height information can be determined using lane line key point matching pairs in images at different times, the vehicle displacement between the first and second times, and the height information from the image sensor, thereby obtaining the three-dimensional coordinates of the lane line and enabling accurate and rapid 3D reconstruction of the lane line.

[0064] In some embodiments, step S2042, determining the height of the lane line based on lane line key point matching pairs, displacement, and height information from the image sensor, includes: determining a first number of lane line key point matching pairs and a second number of first lane line key points included in the first lane line key point set; determining a key point matching rate based on the first number and the second number; and determining the height of the lane line based on the lane line key point matching pairs, displacement, and height information from the image sensor in response to the key point matching rate being greater than or equal to a preset threshold.

[0065] For example, since the set of key points for the first lane line includes multiple key points, these multiple key points can be traversed to determine possible lane line key point matching pairs. Then, a first number m of lane line key point matching pairs and a second number n of multiple key points can be counted, and a key point matching rate r can be determined based on the first and second numbers. If the key point matching rate r is less than a preset threshold, it is considered that the vehicle has experienced a bump, and subsequent 3D lane line calculations are stopped. Matching operations resume once the key point matching rate returns to normal. Conversely, if the key point matching rate r is greater than or equal to the preset threshold, the height of the lane line is determined based on the lane line key point matching pairs, the vehicle's position between the first and second time moments, and the height information from the image sensor.

[0066] In this embodiment of the disclosure, the key point matching rate r can be determined by the following formula (2):

[0067] r = m / n (2);

[0068] like Figure 4A As shown above, in the above Figure 3 Based on the illustrated embodiment, step S2041 may include steps S21-S24.

[0069] Step S21: Determine the first target key point in the first lane line key point set, and the sensor parameters of the image sensor.

[0070] Step S22: Based on the first target key point and sensor parameters, determine the height range of the lane line point corresponding to the first target key point in the physical world.

[0071] For example, such as Figure 4B As shown, line 41 is the lane line in the physical world (located on a ramp), and line 42 is the extension line of the plane where the image sensor is located. Point A is the position of the camera optical center of the image sensor in the preset coordinate system at the first moment, and point B is the position of the camera optical center of the image sensor in the preset coordinate system at the second moment. Point P is the lane line point corresponding to the first target key point in the physical world. Based on this, the height range z (i.e., line segment PG) of the lane line point P corresponding to the first target key point in the physical world can be determined by the following formula (3):

[0072] (z / d * )×100%=i (3);

[0073] Where i represents the slope range of the road where the lane line is located; d *The longitudinal relative distance (i.e., line segment DG) between the lane line point P corresponding to the first target key point in the physical world and the vehicle's position (i.e., point B) at the time of image acquisition in the second image can be determined using the first target key point, sensor parameters, and the vehicle's displacement between the first and second moments; z represents the range of height values. For example, since there are relevant design specifications for the slope of the ramp where the lane line is located in the physical world—for instance, urban road design specifications recommend a maximum longitudinal slope of 3%-4% for expressways and 4%-5% for arterial roads—the slope range of the road where the lane line is located can be determined according to the actual road design specifications.

[0074] It should be noted that, since the starting point of the ramp where the lane line is located cannot be determined in the actual environment, point D is used to replace the starting point of the ramp in this embodiment of the disclosure, and then the height range of the lane line point P corresponding to the first target key point in the physical world is determined by the line segment DG. Of course, point A can also be used to replace the starting point of the ramp, and this embodiment of the disclosure does not limit this.

[0075] Step S23: Based on the first target key points, height value range, displacement, image sensor height information and sensor parameters, determine the target area in the second image.

[0076] For example, such as Figure 4B As shown, the vehicle's displacement between the first and second moments is d (i.e., line segment AB), and the image sensor's height information is h (i.e., line segment AC or line segment BD). Point F is the image pixel corresponding to point P at the first moment projected onto the preset coordinate system through the image sensor's sensor parameters, and point E is the image pixel corresponding to point P at the second moment projected onto the preset coordinate system through the image sensor's sensor parameters. Since the first target key point ( Figure 4B Points A, P, and F (not shown in the image) are all located on the same straight line, and the second target key point ( Figure 4B Points B, P, and E (not shown) are all located on the same straight line. Therefore, based on the principle of similar triangles, and using the first target key point, the height range z of point P, the vehicle displacement d between the first and second time moments, the height information h of the image sensor, and the sensor parameters, the target region in the second image can be determined. The target region in the second image represents the image area range where the second target key point is located.

[0077] It should be noted that, Figure 4B The illustration only shows the scenario where the vehicle is on the ground plane at the first and second moments, and is for illustrative purposes. The lane line determination method provided in this disclosure is equally applicable to other scenarios (e.g., the vehicle is on a ramp where the lane line is located at the first and / or second moments).

[0078] Step S24: Based on the target area and the set of key points of the second lane line, determine the lane line key point matching pair of the first target key point.

[0079] For example, in this embodiment of the present disclosure, after determining the image region range where the second target key point is located, the second target key point can be determined based on the range (i.e., the target region) and multiple second lane line key points in the second lane line key point set; then, based on the first target key point and the second target key point, a lane line key point matching pair of the first target key point is determined.

[0080] The lane line determination method provided in this embodiment determines a first target key point in a first set of lane line key points, and sensor parameters of an image sensor; based on the first target key point and sensor parameters, it determines the height range of the lane line point corresponding to the first target key point in the physical world; based on the first target key point, height range, displacement, height information of the image sensor, and sensor parameters, it determines a target region in a second image; based on the target region and the second set of lane line key points, it determines lane line key point matching pairs of the first target key point; thus, based on the principle of similar triangles, it can determine the image region range where the second target key point is located, and then determine the lane line key point matching pairs.

[0081] In some embodiments, step S23, determining the target region in the second image based on the first target key point, height value range, displacement, height information of the image sensor, and sensor parameters, includes: determining first angle information and second angle information based on the first target key point and sensor parameters; determining multiple third angle information based on the first angle information, second angle information, height value range, height information of the image sensor, and displacement; determining multiple target pixels in the second image based on the multiple third angle information and sensor parameters; and determining the target region based on the position information of the multiple target pixels in the second image.

[0082] For example, such as Figure 4C As shown, triangle 43 represents the image sensor, point O is the camera optical center of the image sensor, and the two sides containing vertex O of triangle 43 represent the field of view of the image sensor. Furthermore, based on the first target keypoint and the sensor parameters of the image sensor, first angle information and second angle information can be determined. For example, Figure 4CThe angle β between the camera normal of the image sensor and the horizontal line is the first angle information. Simultaneously, point O and the lane line point P corresponding to the first target key point in the physical world form ray OP, and the angle between ray OP and the camera normal is α. If point O is the camera optical center point A at the first moment, then the angle α is the second angle information. Thus, based on angle β, the angle α corresponding to point A (at the first moment), the height value range z, the height information h and displacement d of the image sensor, multiple third angle information can be determined; and based on these multiple third angle information and sensor parameters, multiple target pixels are determined in the second image; and based on the position information of these multiple target pixels in the second image, the target region is determined. Each z value in the height value range corresponds to one third angle information.

[0083] For example, such as Figure 4B As shown, after determining angle β (the first angle information) and angle α (the second angle information) corresponding to point A, the included angle ∠CAP can be calculated. Then, using the principle of similar triangles (triangles ACF and PGF are similar triangles), the length of line segment CF can be determined using the height information h from the image sensor. Furthermore, the lengths of line segments GF and CG can be calculated based on the proportional relationship between the height range z (i.e., line segment PG) and AC. Further, the length of line segment DG can be calculated based on the displacement d, and the angle ∠DBP can be determined based on the proportional relationship between line segments PG and BD and line segment DG. That is, since points P, G, B, and D form a right trapezoid, and given the lengths of the two parallel sides (the lengths of line segments PG and BD) and the length of the right-angled side (the length of line segment DG), the angle of the other angle of the right trapezoid (angle ∠DBP) can be solved using trigonometric functions. It should be noted that since this method is a conventional mathematical calculation method, the specific calculation process will not be explained separately here.

[0084] Among them, such as Figure 4DAs shown, point A (or B) is the optical center of the image sensor's camera (point A is the optical center of the image sensor at the first moment, and point B is the optical center of the image sensor at the second moment), the dashed box 43 is the imaging plane of the image sensor, line segment AR (or BR) is the focal length of the image sensor (i.e., the perpendicular distance from the optical center to the imaging plane; line segment AR or line segment BR is also the camera normal of the image sensor), and point p1 (or p2) is the pixel point projected onto the imaging plane from point P on the lane line in the physical world (point p1 is the pixel point projected onto the imaging plane from point P at the first moment, and point p2 is the pixel point projected onto the imaging plane from point P at the second moment). Furthermore, since the pixel coordinates of point p1 and point R are known, the distance between point R and point p1 can be determined. Furthermore, in the right triangle formed by points A, R, and p1, the angle ∠RAp1 can be determined using the line segment AR (sensor parameter: focal length) and the distance between points R and p1, through the trigonometric function principle of the right triangle. This angle is the α angle corresponding to point A.

[0085] Furthermore, such as Figure 4B As shown, the angle α corresponding to point B (i.e., the third angle information) can be determined by the following formula (4):

[0086] α=90-∠DBP-β (4);

[0087] Furthermore, based on the multiple α angles corresponding to point B (at the second moment) (each z value corresponds to an α angle of point B) and the sensor parameters, multiple target pixels in the second image (i.e., the coordinates of multiple pixels projected onto the second image by point P with different height values) can be determined.

[0088] The target pixel point corresponding to angle α in the second image can be determined based on the angle α corresponding to point B and the sensor parameters in the following way:

[0089] like Figure 4D As shown, in the right triangle formed by points B, R, and p2, the distance between points R and p2 can be determined using the line segment BR (sensor parameter: focal length) and the angle α corresponding to point B, applying the trigonometric functions of a right triangle. Furthermore, since the pixel coordinates of point R are known, the pixel coordinates of point p2 can be determined based on the distance between R and p2, and the pixel coordinates of R. The pixel coordinates of point p2 are the target pixel point corresponding to angle α in the second image.

[0090] In this way, the coordinate positions of multiple target pixels in the second image can be used to determine an image region (i.e., the target region) in the second image.

[0091] like Figure 5 As shown above, in the above Figure 4ABased on the illustrated embodiment, step S24 may include steps S24a-S24c.

[0092] Step S24a: Based on the target area, determine the second candidate key points from the set of key points of the second lane line.

[0093] For example, after identifying the target region in the second image, the range of the projection of point P on the lane line in the physical world at the second moment onto the second image can be obtained. Then, the key points of the second lane line within this range can be directly obtained and used as second candidate key points.

[0094] Step S24b: In response to the number of second candidate key points being multiple, determine the first semantic information corresponding to the first target key point and the second semantic information corresponding to the second candidate key point.

[0095] For example, if the number of second candidate keypoints is 0, lane line keypoint matching fails. If the number of second candidate keypoints is 1, the second candidate keypoint can be used as the second target keypoint. If the number of second candidate keypoints is multiple (greater than or equal to 2), it is necessary to determine the first semantic information corresponding to the first target keypoint and the second semantic information corresponding to the second candidate keypoint.

[0096] In some examples, the first semantic information corresponding to the first target keypoint includes, but is not limited to, the semantic information of the pixels surrounding the first target keypoint. Similarly, the second semantic information corresponding to the second candidate keypoint includes, but is not limited to, the semantic information of the pixels surrounding the second candidate keypoint. For example, if the first target keypoint is the point with pixel coordinates (3,2) in the first image, then the first semantic information corresponding to the first target keypoint includes, but is not limited to, the semantic information of the pixel with coordinates (2,2), the pixel with coordinates (3,1), the pixel with coordinates (3,3), and the pixel with coordinates (4,2) in the first image.

[0097] Step S24c: Based on the first semantic information and the second semantic information, determine the lane line key point matching pair of the first target key point.

[0098] For example, the first semantic information and the second semantic information can be compared, and lane line key point matching pairs can be determined based on the comparison result. For instance, if the first semantic information and the second semantic information are the same, the second candidate key point corresponding to the second semantic information is determined as the second target key point, thereby determining the lane line key point matching pair of the first target key point.

[0099] The lane line determination method provided in this embodiment determines second candidate key points from a second set of lane line key points based on a target region; in response to the presence of multiple second candidate key points, it determines first semantic information corresponding to a first target key point and second semantic information corresponding to the second candidate key points; and determines lane line key point matching pairs of the first target key point based on the first and second semantic information. Thus, it is possible to determine second target key points matching the first target key point from within the target region based on the semantic information of neighboring pixels around the key point.

[0100] In some embodiments, step S24c, determining the lane line key point matching pair of the first target key point based on the first semantic information and the second semantic information, includes: comparing the first semantic information with the second semantic information to obtain a comparison result; determining the second target key point from a plurality of second candidate key points based on the comparison result; and determining the first target key point and the second target key point as a lane line key point matching pair.

[0101] For example, the first target keypoint p1 is a point with pixel coordinates (3,2) in the first image. For its surrounding neighboring pixels, the semantic information of points (3,1) and (2,2) is a non-lane line region (road surface), and the semantic information of points (3,3) and (4,2) is a lane line region (lane line point). If the semantic information of the neighboring pixels of a second candidate keypoint p2 among multiple second candidate keypoints is the same as the above semantic information, then the second candidate keypoint p2 is considered to be the second target keypoint, and the lane line keypoint matching pair of the first target keypoint p1 is (p1, p2).

[0102] It should be noted that the neighboring pixels around the first target key point mentioned above are only illustrative examples, and the position and number of neighboring pixels around the first target key point are not limited in this embodiment. For example, if the pixel coordinates of the first target key point are (3,2), the coordinates of its neighboring pixels can also be (2,1), (2,2), (2,3), (3,1), (3,3), (4,1), (4,2), and (4,3).

[0103] In some embodiments, determining a second target keypoint from a plurality of second candidate keypoints based on the comparison results includes: determining the second candidate keypoint as the second target keypoint in response to the comparison results showing that the second semantic information corresponding to a second candidate keypoint is the same as the first semantic information; or, determining the pixel difference between the second candidate keypoint and its adjacent pixels in response to the comparison results showing that the second semantic information corresponding to a plurality of second candidate keypoints is the same as the first semantic information; and determining the second target keypoint from the plurality of second candidate keypoints based on the pixel difference.

[0104] For example, in this embodiment of the disclosure, if there exists a second candidate key point whose second semantic information is the same as the first semantic information, then the second candidate key point is determined as the second target key point. If there exist at least two second candidate key points whose second semantic information is the same as the first semantic information, then the pixel difference between each of the at least two second candidate key points and its adjacent pixels is determined, and based on the pixel difference, the second target key point is determined from the at least two second candidate key points.

[0105] In some examples, a second target keypoint can be determined from at least two second candidate keypoints based on a maximum value suppression algorithm. For example, given a second candidate keypoint 1 and a second candidate keypoint 2, the pixel differences between the second candidate keypoint 1 and its neighboring pixels are determined, and the sum of these pixel differences is recorded as the first difference sum. Similarly, the pixel differences between the second candidate keypoint 2 and its neighboring pixels are determined, and the sum of these pixel differences is recorded as the second difference sum. If the second difference sum is greater than the first difference sum, then the second candidate keypoint 2 is determined as the second target keypoint. Thus, when there are a large number of keypoints, compared to the traditional method of pairwise matching based on descriptors, the above keypoint matching method can reduce the workload of keypoint matching and improve the efficiency of keypoint matching.

[0106] like Figure 6 As shown above, in the above Figure 4A Based on the illustrated embodiment, step S22 may include steps S22a-S22d.

[0107] Step S22a: Determine the slope parameters of the road.

[0108] For example, the slope parameters of the road where the lane lines are located can be determined based on the road type and relevant design specifications. In one instance, the road type of the road where the lane lines are located can be determined based on location information and / or image perception data. Then, based on the determined road type, the slope parameters of the road where the lane lines are located can be determined according to the relevant road design specifications. The slope parameters include the maximum value of the slope. For example, if the determined road where the lane lines are located is an urban expressway, the recommended maximum longitudinal slope value is 4%-5%.

[0109] Step S22b: Determine the slope range of the road based on the slope parameters.

[0110] For example, if the slope parameter is the maximum longitudinal slope recommended value of 4%-5%, then the slope range of the road is [0%, 5%].

[0111] Step S22c: Based on the first target key point and sensor parameters, determine the relative distance between the lane line point corresponding to the first target key point in the physical world and the vehicle.

[0112] For example, in this embodiment of the present disclosure, a first relative distance between the lane line point corresponding to the first target key point in the physical world and the vehicle at the first moment can be determined firstly based on the first target key point and the camera's intrinsic and extrinsic parameters. Then, based on the vehicle's displacement between the first and second moments and the first relative distance, a second relative distance between the lane line point corresponding to the first target key point in the physical world and the vehicle at the second moment can be determined. Thus, either the first or second relative distance can be determined as the relative distance in step S22c.

[0113] For example, the coordinates of the first target key point can be transformed using the camera's intrinsic and extrinsic parameters, converting it to the vehicle's coordinate system at the first moment. The transformed longitudinal coordinate is the first relative distance between the lane line point corresponding to the first target key point in the physical world and the vehicle at the first moment. Furthermore, the difference between the first relative distance and the displacement is the second relative distance between the lane line point corresponding to the first target key point in the physical world and the vehicle at the second moment.

[0114] Step S22d: Determine the range of height values ​​based on the slope range and relative distance.

[0115] For example, the height range of the lane line point corresponding to the first target key point in the physical world can be determined based on the above formula (3), which will not be elaborated here.

[0116] The lane line determination method provided in this embodiment determines the road slope parameters; based on the slope parameters, it determines the road slope range; based on the first target key point and sensor parameters, it determines the relative distance between the lane line point corresponding to the first target key point in the physical world and the vehicle; based on the slope range and the relative distance, it determines the height value range; thus, it is possible to determine the height of the lane line point corresponding to the first target key point in the physical world under different slopes, thereby determining the height value range of the lane line point.

[0117] like Figure 7 As shown above, in the above Figure 3 Based on the illustrated embodiment, step S2042 may include steps S25-S27.

[0118] Step S25: Perform coordinate transformation on the first target key point and the second target key point in the lane line key point matching pair to obtain the first three-dimensional lane line point and the second three-dimensional lane line point.

[0119] For example, such as Figure 4BAs shown, coordinate transformation can be performed on the first target key point in the lane line key point matching pair to obtain the first three-dimensional lane line point F located in the preset coordinate system. Similarly, coordinate transformation can be performed on the second target key point in the lane line key point matching pair to obtain the second three-dimensional lane line point E located in the preset coordinate system. Here, point A is the optical center of the camera of the image sensor at the first moment, and point B is the optical center of the camera of the image sensor at the second moment.

[0120] Step S26: Determine the first distance based on the first three-dimensional lane line points and the second three-dimensional lane line points.

[0121] For example, such as Figure 4B As shown, the length of line segment EF can be determined based on the first three-dimensional lane line point F and the second three-dimensional lane line point E. The length of line segment EF is the first distance.

[0122] Step S27: Determine the height of the lane line based on the first distance, displacement, and height information from the image sensor.

[0123] For example, after determining the first distance between the first three-dimensional lane line point and the second three-dimensional lane line point, the height of the lane line can be determined based on the first distance, displacement, and height information from the image sensor.

[0124] In some examples, such as Figure 4B As shown, d is the vehicle displacement between the first and second moments, h is the height information of the image sensor, and line segment EF is the first distance. Furthermore, the actual height Z of point P on the lane line can be determined using the following formula (5):

[0125]

[0126] In other words, after excluding vehicle bumps, since the vehicle travels a short distance on a flat surface (the time interval between the first and second moments is less than the preset interval), it is approximately assumed that the vehicle travels on the same plane within that distance. Therefore, the height of the lane line points in the physical world (the height of multiple lane line points is the lane line height) can be calculated based on the parameters in formula (5) using the geometric similar triangle relationship.

[0127] The lane line determination method provided in this embodiment of the present disclosure obtains first three-dimensional lane line points and second three-dimensional lane line points by performing coordinate transformation processing on the first target key point and the second target key point in the lane line key point matching pair. Based on the first three-dimensional lane line points and the second three-dimensional lane line points, a first distance is determined; based on the first distance, displacement, and height information of the image sensor, the height of the lane line is determined; thus, the height of the lane line points in the physical world can be accurately obtained by utilizing geometric relationships.

[0128] In some embodiments, a target range can be defined in the first image with a first target key point as the center and a preset value as the radius. Furthermore, it can be assumed that the height of lane line points within this target range is the same as the height of the first target key point. Similarly, a target range can be defined in the second image with a second target key point as the center and a preset value as the radius. Furthermore, it can be assumed that the height of lane line points within this target range is the same as the height of the second target key point.

[0129] like Figure 8 As shown above, in the above Figure 2 Based on the illustrated embodiment, step S202 may include steps S2021-S2024.

[0130] Step S2021: Divide the first image and the second image into regions to obtain multiple first sub-regions and multiple second sub-regions.

[0131] For example, in this embodiment of the disclosure, the first image can be divided into regions to obtain multiple first sub-regions. Similarly, the second image can be divided into regions to obtain multiple second sub-regions. For instance, the first image and the second image can be divided into three first sub-regions (upper, middle, and lower), and the second image can be divided into three second sub-regions (upper, middle, and lower).

[0132] Step S2022: Determine the first scale parameter based on the regional location of the first sub-region, and determine the second scale parameter based on the regional location of the second sub-region.

[0133] For example, a first scale parameter can be determined based on the relative position of a first sub-region in a first image; and a second scale parameter can be determined based on the relative position of a second sub-region in a second image. The first and second scale parameters can be the same or different, and this disclosure does not impose any limitations on this.

[0134] In some examples, embodiments of this disclosure may employ the FAST algorithm to extract key points from the first and second images. During key point extraction, taking into account the characteristics of roads appearing larger when closer and smaller when farther away, and the equal length of each segment of dashed lane lines in the images, key points are extracted from the first or second sub-region located at the bottom of the image (e.g., the bottom 1 / 3 of the image) using a radius r = 3; from the first or second sub-region located in the middle of the image (e.g., the middle 1 / 3 of the image) using a radius r = 2; and from the first or second sub-region located at the top of the image (e.g., the top 1 / 3 of the image) using a radius r = 1. Here, r is a parameter representing the size of the local neighborhood range used for corner detection in the FAST algorithm.

[0135] It should be noted that the granularity of the division of the first and second image regions in this embodiment is not limited, as are the values ​​of the first scale parameter of each first sub-region and the second scale parameter of each second sub-region. Those skilled in the art can set these values ​​according to actual application conditions.

[0136] Step S2023: Extract key points from the first sub-region based on the first scale parameter to obtain the key point set of the first lane line.

[0137] It should be noted that multiple key points of the first lane line in the set of key points of the first lane line can be located on the lane line or near the lane line.

[0138] Step S2024: Extract key points from the second sub-region based on the second scale parameter to obtain the key point set of the second lane line.

[0139] It should be noted that multiple key points of the second lane line in the key point set can be located on the lane line or near the lane line.

[0140] The lane line determination method provided in this embodiment divides a first image and a second image into multiple first sub-regions and multiple second sub-regions; determines a first scale parameter based on the location of the first sub-regions, and determines a second scale parameter based on the location of the second sub-regions; extracts key points from the first sub-regions based on the first scale parameter to obtain a first lane line key point set; and extracts key points from the second sub-regions based on the second scale parameter to obtain a second lane line key point set. This method improves the efficiency of key point extraction while reducing computational power.

[0141] Exemplary device

[0142] Figure 9 A lane line determination device provided in this disclosure embodiment, such as Figure 9 As shown, the lane line determination device 900 includes an image determination module 901, a key point determination module 902, a parameter determination module 903, and a lane line determination module 904.

[0143] The image determination module 901 is used to determine the first image acquired by the vehicle's image sensor at a first moment and the second image acquired at a second moment;

[0144] The key point determination module 902 is used to determine a first lane line key point set and a second lane line key point set based on the first image and the second image.

[0145] The parameter determination module 903 is used to determine the vehicle's displacement between the first and second moments, as well as the height information from the image sensor.

[0146] The lane line determination module 904 is used to determine the lane line based on displacement, a first set of key points for the lane line, a second set of key points for the lane line, and height information from the image sensor.

[0147] In some embodiments, such as Figure 10 As shown, the lane line determination module 904 includes a matching pair determination unit 9041, a height determination unit 9042, and a lane line determination unit 9043.

[0148] The matching pair determination unit 9041 is used to determine lane line key point matching pairs based on displacement, the first lane line key point set, and the second lane line key point set.

[0149] The height determination unit 9042 is used to determine the height of the lane line based on lane line key point matching pairs, displacement, and height information from image sensors.

[0150] Lane line determination unit 9043 is used to determine lane lines based on the height of the lane lines.

[0151] In some embodiments, the matching pair determination unit 9041 is configured to determine a first target key point in a first set of lane line key points, and sensor parameters of an image sensor; based on the first target key point and sensor parameters, determine the height range of the lane line point corresponding to the first target key point in the physical world; based on the first target key point, the height range, displacement, height information of the image sensor, and sensor parameters, determine a target region in a second image; and based on the target region and the second set of lane line key points, determine a lane line key point matching pair of the first target key point.

[0152] In some embodiments, the matching pair determination unit 9041 is configured to: determine a first target key point in a first set of lane line key points, and sensor parameters of an image sensor; determine the height range of the lane line point corresponding to the first target key point in the physical world based on the first target key point and the sensor parameters; determine first angle information and second angle information based on the first target key point and the sensor parameters; determine multiple third angle information based on the first angle information, the second angle information, the height range, the height information of the image sensor, and the displacement; determine multiple target pixels in a second image based on the multiple third angle information and the sensor parameters; determine a target region based on the position information of the multiple target pixels in the second image; and determine a lane line key point matching pair of the first target key point based on the target region and the second set of lane line key points.

[0153] In some embodiments, the matching pair determination unit 9041 is specifically configured to: determine a first target key point in a first set of lane line key points, and sensor parameters of an image sensor; determine the height range of the lane line point corresponding to the first target key point in the physical world based on the first target key point and the sensor parameters; determine first angle information and second angle information based on the first target key point and the sensor parameters; determine multiple third angle information based on the first angle information, the second angle information, the height range, the height information of the image sensor, and the displacement; determine multiple target pixels in a second image based on the multiple third angle information and the sensor parameters; determine a target region based on the position information of the multiple target pixels in the second image; determine a second candidate key point from the second set of lane line key points based on the target region; determine first semantic information corresponding to the first target key point and second semantic information corresponding to the second candidate key point in response to the number of second candidate key points being multiple; and determine a lane line key point matching pair of the first target key point based on the first semantic information and the second semantic information.

[0154] In some embodiments, the matching pair determination unit 9041 is specifically configured to: determine a first target key point in a first set of lane line key points, and sensor parameters of an image sensor; determine the height range of the lane line point corresponding to the first target key point in the physical world based on the first target key point and sensor parameters; determine first angle information and second angle information based on the first target key point and sensor parameters; determine multiple third angle information based on the first angle information, second angle information, height range, height information of the image sensor, and displacement; determine multiple target pixels in a second image based on the multiple third angle information and sensor parameters; determine a target region based on the position information of the multiple target pixels in the second image; determine a second candidate key point from the second set of lane line key points based on the target region; determine first semantic information corresponding to the first target key point and second semantic information corresponding to the second candidate key point in response to the number of second candidate key points being multiple; compare the first semantic information with the second semantic information to obtain a comparison result; determine a second target key point from the multiple second candidate key points based on the comparison result; and determine the first target key point and the second target key point as a lane line key point matching pair.

[0155] In some embodiments, the matching pair determination unit 9041 is specifically configured to: determine a first target key point in a first set of lane line key points, and sensor parameters of an image sensor; determine the height range of the lane line point corresponding to the first target key point in the physical world based on the first target key point and the sensor parameters; determine first angle information and second angle information based on the first target key point and the sensor parameters; determine multiple third angle information based on the first angle information, second angle information, height range, height information of the image sensor, and displacement; determine multiple target pixels in a second image based on the multiple third angle information and sensor parameters; determine a target region based on the position information of the multiple target pixels in the second image; and determine a lane line key point from the second set of lane line key points based on the target region. Second candidate keypoint; in response to the existence of multiple second candidate keypoints, determine the first semantic information corresponding to the first target keypoint and the second semantic information corresponding to the second candidate keypoints; compare the first semantic information and the second semantic information to obtain a comparison result; in response to the comparison result showing that there is a second candidate keypoint whose second semantic information is the same as the first semantic information, determine the second candidate keypoint as the second target keypoint; or, in response to the comparison result showing that there are multiple second candidate keypoints whose second semantic information is the same as the first semantic information, determine the pixel difference between the second candidate keypoint and its adjacent pixels; based on the pixel difference, determine the second target keypoint from multiple second candidate keypoints; determine the first target keypoint and the second target keypoint as a lane line keypoint matching pair.

[0156] In some embodiments, the matching pair determination unit 9041 is specifically configured to: determine a first target key point in a first set of lane line key points, and sensor parameters of an image sensor; determine road slope parameters; determine the road slope range based on the slope parameters; determine the relative distance between the lane line point corresponding to the first target key point in the physical world and the vehicle based on the first target key point and sensor parameters; determine a height value range based on the slope range and relative distance; determine first angle information and second angle information based on the first target key point and sensor parameters; determine multiple third angle information based on the first angle information, second angle information, height value range, image sensor height information, and displacement; determine multiple target pixels in a second image based on the multiple third angle information and sensor parameters; and determine a target based on the position information of the multiple target pixels in the second image. Region; Based on the target region, determine second candidate key points from the set of key points of the second lane line; In response to the number of second candidate key points being multiple, determine the first semantic information corresponding to the first target key point and the second semantic information corresponding to the second candidate key point; Compare the first semantic information and the second semantic information to obtain a comparison result; In response to the comparison result showing that there is a second candidate key point whose second semantic information is the same as the first semantic information, determine the second candidate key point as the second target key point; Or, In response to the comparison result showing that there are multiple second candidate key points whose second semantic information is the same as the first semantic information, determine the pixel difference between the second candidate key point and its adjacent pixels; Based on the pixel difference, determine the second target key point from multiple second candidate key points; Determine the first target key point and the second target key point as a lane line key point matching pair.

[0157] In some embodiments, the height determination unit 9042 is specifically used to perform coordinate transformation processing on the first target key point and the second target key point in the lane line key point matching pair to obtain a first three-dimensional lane line point and a second three-dimensional lane line point. Based on the first three-dimensional lane line point and the second three-dimensional lane line point, a first distance is determined; based on the first distance, displacement, and height information from the image sensor, the height of the lane line is determined.

[0158] In some embodiments, the height determination unit 9042 is specifically configured to determine a first number of lane line key point matching pairs and a second number of first lane line key points included in the first lane line key point set; determine a key point matching rate based on the first number and the second number; and determine the height of the lane line based on the lane line key point matching pairs, displacement, and height information from the image sensor in response to the key point matching rate being greater than or equal to a preset threshold.

[0159] In some embodiments, the key point determination module 902 is specifically used to divide the first image and the second image into regions to obtain multiple first sub-regions and multiple second sub-regions; determine a first scale parameter based on the region location of the first sub-region, and determine a second scale parameter based on the region location of the second sub-region; extract key points from the first sub-region based on the first scale parameter to obtain a first lane line key point set; and extract key points from the second sub-region based on the second scale parameter to obtain a second lane line key point set.

[0160] The beneficial technical effects corresponding to the exemplary embodiment of the lane line determination device 900 described above can be found in the corresponding beneficial technical effects of the exemplary method section described above, and will not be repeated here.

[0161] Exemplary electronic devices

[0162] Figure 11 The present disclosure provides a structural diagram of an electronic device 110, which includes at least one processor 111 and a memory 112.

[0163] The processor 111 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 110 to perform desired functions.

[0164] The memory 112 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 111 may execute one or more computer program instructions to implement the lane line determination methods and / or other desired functions of the various embodiments of this disclosure described above.

[0165] In one example, the electronic device 110 may also include an input device 113 and an output device 114, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0166] The input device 113 may also include, for example, a keyboard, a mouse, etc.

[0167] The output device 114 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0168] Of course, for the sake of simplicity, Figure 11 Only some of the components of the electronic device 110 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 110 may include any other suitable components depending on the specific application.

[0169] Exemplary computer program products and computer-readable storage media

[0170] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the lane line determination method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0171] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0172] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the lane line determination method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0173] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0174] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0175] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for determining lane lines, comprising: Determine the first image captured by the vehicle's image sensor at a first moment, and the second image captured at a second moment; Based on the first image and the second image, determine the first lane line key point set and the second lane line key point set; Determine the displacement of the vehicle between the first time moment and the second time moment, as well as the height information of the image sensor; The lane line is determined based on the displacement, the first set of key points of the lane line, the second set of key points of the lane line, and the height information of the image sensor.

2. The method according to claim 1, wherein, Determining the lane line based on the displacement, the first set of key points for the lane line, the second set of key points for the lane line, and the height information from the image sensor includes: Based on the displacement, the first set of key points of the lane line and the second set of key points of the lane line, a matching pair of key points of the lane line is determined; The height of the lane line is determined based on the lane line key point matching pair, the displacement, and the height information of the image sensor; The lane lines are determined based on their height.

3. The method according to claim 2, wherein, The step of determining lane line key point matching pairs based on the displacement, the first set of lane line key points, and the second set of lane line key points includes: Determine the first target key point in the first set of key points of the first lane line, and the sensor parameters of the image sensor; Based on the first target key point and the sensor parameters, determine the height range of the lane line point corresponding to the first target key point in the physical world. Based on the first target key point, the height value range, the displacement, the height information of the image sensor, and the sensor parameters, the target region in the second image is determined; Based on the target area and the second set of lane line key points, lane line key point matching pairs of the first target key point are determined.

4. The method according to claim 3, wherein, The step of determining the target region in the second image based on the first target key point, the height value range, the displacement, the height information of the image sensor, and the sensor parameters includes: Based on the first target key point and the sensor parameters, determine the first angle information and the second angle information; Based on the first angle information, the second angle information, the height value range, the height information of the image sensor, and the displacement, multiple third angle information are determined; Based on the multiple third-angle information and the sensor parameters, multiple target pixels are determined in the second image; The target region is determined based on the position information of the plurality of target pixels in the second image.

5. The method according to claim 3, wherein, The step of determining the lane line key point matching pair of the first target key point based on the target area and the second lane line key point set includes: Based on the target area, a second candidate key point is determined from the second lane line key point set; In response to the fact that there are multiple second candidate key points, the first semantic information corresponding to the first target key point and the second semantic information corresponding to the second candidate key points are determined. Based on the first semantic information and the second semantic information, lane line key point matching pairs of the first target key point are determined.

6. The method according to claim 5, wherein, The step of determining the lane line key point matching pair of the first target key point based on the first semantic information and the second semantic information includes: The first semantic information is compared with the second semantic information to obtain the comparison result; Based on the comparison results, a second target key point is determined from a plurality of second candidate key points; The first target key point and the second target key point are determined as the lane line key point matching pair.

7. The method according to claim 6, wherein, The step of determining the second target key point from multiple second candidate key points based on the comparison results includes: In response to the comparison result indicating that there exists a second candidate key point whose second semantic information is identical to the first semantic information, the second candidate key point is determined as the second target key point; or, In response to the comparison result showing that there are multiple second candidate key points whose second semantic information is the same as the first semantic information, the pixel difference between the second candidate key point and its adjacent pixel is determined. Based on the pixel difference, the second target key point is determined from the plurality of second candidate key points.

8. The method according to claim 3, wherein, The step of determining the height range of the lane line point corresponding to the first target key point in the physical world based on the first target key point and the sensor parameters includes: Determine the slope parameters of the road; Based on the slope parameters, the slope range of the road is determined; Based on the first target key point and the sensor parameters, determine the relative distance between the lane line point corresponding to the first target key point in the physical world and the vehicle. The range of height values ​​is determined based on the slope range and the relative distance.

9. The method according to any one of claims 2-8, wherein, Determining the height of the lane line based on the lane line key point matching pair, the displacement, and the height information of the image sensor includes: The first and second target key points in the lane line key point matching pair are subjected to coordinate transformation to obtain the first three-dimensional lane line point and the second three-dimensional lane line point. Based on the first three-dimensional lane line points and the second three-dimensional lane line points, determine the first distance; The height of the lane line is determined based on the first distance, the displacement, and the height information from the image sensor.

10. The method according to any one of claims 2-8, wherein, Determining the height of the lane line based on the lane line key point matching pair, the displacement, and the height information of the image sensor includes: Determine a first number of lane line key point matching pairs, and a second number of first lane line key points included in the first lane line key point set; Based on the first quantity and the second quantity, determine the key point matching rate; In response to the key point matching rate being greater than or equal to a preset threshold, the height of the lane line is determined based on the lane line key point matching pairs, the displacement, and the height information of the image sensor.

11. The method according to any one of claims 1-8, wherein, The step of determining the first lane line key point set and the second lane line key point set based on the first image and the second image includes: The first image and the second image are divided into regions to obtain multiple first sub-regions and multiple second sub-regions; Based on the location of the first sub-region, a first scale parameter is determined, and based on the location of the second sub-region, a second scale parameter is determined. Based on the first scale parameter, key points are extracted from the first sub-region to obtain the set of key points for the first lane line. Based on the second scale parameter, key points are extracted from the second sub-region to obtain the set of key points for the second lane line.

12. A lane marking device, comprising: An image determination module is used to determine a first image captured by the vehicle's image sensor at a first moment and a second image captured at a second moment; The key point determination module is used to determine a first lane line key point set and a second lane line key point set based on the first image and the second image; The parameter determination module is used to determine the displacement of the vehicle between the first time moment and the second time moment, as well as the height information of the image sensor; The lane line determination module is used to determine the lane line based on the displacement, the first set of lane line key points, the second set of lane line key points, and the height information of the image sensor.

13. A computer-readable storage medium storing a computer program for performing the lane line determination method according to any one of claims 1-11.

14. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the lane line determination method according to any one of claims 1-11.

Images