Lane line generation method and device, readable storage medium and electronic equipment

Abstract

This disclosure provides a method, apparatus, computer-readable storage medium, and electronic device for generating lane lines. The method includes: semantically segmenting a current image frame to obtain a first point cloud; acquiring a second point cloud corresponding to a target historical image frame; determining an incremental point cloud relative to the second point cloud; clustering the first point cloud to obtain at least one point cloud cluster; matching each of the at least one point cloud cluster with at least one preset historical lane line instance; generating at least one lane line instance corresponding to the current image frame using the incremental point cloud based on the matching results; and generating vectorized lane lines corresponding to each of the at least one lane line instance based on the point clouds corresponding to each lane line instance. This disclosure can help reduce the risk of incomplete lane line vectorization modeling in scenarios such as unclear lane lines or low lighting, thereby improving the integrity and stability of lane line vectorization modeling.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, computer-readable storage medium, and electronic device for generating lane lines. Background Technology

[0002] Current lane line recognition methods typically employ semantic segmentation techniques based on single-frame images to identify lane lines from the image. In scenarios such as underground parking garages, factors such as unclear lane lines and dim lighting can lead to unstable lane line segmentation. Furthermore, severe wear and tear on lane lines can cause discontinuous lane line segmentation, resulting in missed detections and incomplete lane line modeling.

[0003] Currently, lane line modeling often employs single-segment or multi-segment curve fitting. However, in complex scenarios such as underground parking garages, it is difficult to accurately and efficiently determine segmentation points. Furthermore, the granularity of segmentation is difficult to control, easily resulting in redundant segments or insufficient segments that fail to represent detailed structures. This modeling method requires continuous tracking of segmentation points, making lane line maintenance and updates cumbersome. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a method, apparatus, computer-readable storage medium, and electronic device for generating lane lines.

[0005] Embodiments of this disclosure provide a method for generating lane lines. The method includes: performing semantic segmentation on a current image frame to obtain a first point cloud representing lane lines; acquiring a second point cloud representing lane lines corresponding to a target historical image frame; determining an incremental point cloud relative to the second point cloud; clustering the first point cloud to obtain at least one point cloud cluster; matching the at least one point cloud cluster with at least one preset historical lane line instance to obtain a matching result; generating at least one lane line instance corresponding to the current image frame using the incremental point cloud based on the matching result; and generating vectorized lane lines corresponding to the at least one lane line instance based on the point clouds corresponding to the at least one lane line instance.

[0006] According to another aspect of the present disclosure, a lane line generation apparatus is provided. The apparatus includes: a segmentation module for semantically segmenting a current image frame to obtain a first point cloud representing a lane line; an acquisition module for acquiring a second point cloud representing a lane line corresponding to a target historical image frame; a first determination module for determining an incremental point cloud relative to the second point cloud; a clustering module for clustering the first point cloud to obtain at least one point cloud cluster; a matching module for matching the at least one point cloud cluster with at least one preset historical lane line instance to obtain a matching result; a first generation module for generating at least one lane line instance corresponding to the current image frame based on the matching result using the incremental point cloud; and a second generation module for generating vectorized lane lines corresponding to the at least one lane line instance based on the point clouds corresponding to the at least one lane line instance.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores a computer program for execution by a processor to implement the above-described lane line generation method.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the lane line generation method described above.

[0009] Based on the lane line generation method, apparatus, computer-readable storage medium, and electronic device provided in the above embodiments of this disclosure, by performing semantic segmentation on multiple consecutive frames of images, the incremental point cloud between the first point cloud corresponding to the current image frame and the second point cloud corresponding to the target historical image frame is determined. Then, the first point cloud is clustered, and the obtained point cloud clusters are matched with at least one preset historical lane line instance. Based on the matching result, at least one lane line instance corresponding to the current image frame is generated using the incremental point cloud. Finally, based on the point clouds corresponding to at least one lane line instance, vectorized lane lines corresponding to at least one lane line instance are generated. This disclosure effectively utilizes the semantic segmentation results of multiple frames of images, updates historical lane line instances in real time, and uses the point cloud corresponding to the updated lane line instance to update the vectorized lane line in real time, so that the updated vectorized lane line is linked with the historical lane line instance. This helps to reduce the risk of incomplete lane line vectorization modeling caused by unclear lane lines, low light, and other scenarios, thereby improving the integrity and stability of lane line vectorization modeling. Furthermore, since the embodiments of this disclosure are executed sequentially for each frame of multiple images in a preset order, when the number of image frames accumulated in the point cloud is large, the risk of misidentifying different lane line instances as the same lane line instance can be effectively reduced, thereby improving the accuracy of generating lane line instances and vectorized lane lines.

[0010] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 This is the system diagram to which this disclosure applies.

[0013] Figure 2 This is a flowchart illustrating a lane line generation method provided in an exemplary embodiment of this disclosure.

[0014] Figure 3A This is a schematic diagram of at least one historical lane line instance corresponding to a target historical image frame in an embodiment of this disclosure.

[0015] Figure 3B This is a schematic diagram illustrating the generation of at least one lane line instance corresponding to the current image frame based on incremental point cloud according to an embodiment of this disclosure.

[0016] Figure 3C This is a schematic diagram of the vectorized lane lines before and after the update of an embodiment of this disclosure.

[0017] Figure 3D This is a schematic diagram of at least one vectorized lane line corresponding to the current image frame of an embodiment of this disclosure.

[0018] Figure 4 This is a flowchart illustrating a method for generating lane lines provided in another exemplary embodiment of this disclosure.

[0019] Figure 5 This is a flowchart illustrating a method for generating lane lines provided in another exemplary embodiment of this disclosure.

[0020] Figure 6 This is a flowchart illustrating a method for generating lane lines provided in another exemplary embodiment of this disclosure.

[0021] Figure 7 This is a flowchart illustrating a method for generating lane lines provided in another exemplary embodiment of this disclosure.

[0022] Figure 8 This is a schematic diagram illustrating the merging of two lane line instances according to an embodiment of this disclosure.

[0023] Figure 9 This is a flowchart illustrating a method for generating lane lines provided in another exemplary embodiment of this disclosure.

[0024] Figure 10 This is a flowchart illustrating a method for generating lane lines provided in another exemplary embodiment of this disclosure.

[0025] Figure 11 This is a flowchart illustrating a method for generating lane lines provided in another exemplary embodiment of this disclosure.

[0026] Figure 12 This is a schematic diagram of the structure of a lane line generation apparatus provided in an exemplary embodiment of this disclosure.

[0027] Figure 13 This is a schematic diagram of the structure of a lane line generation apparatus provided in another exemplary embodiment of this disclosure.

[0028] Figure 14 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0029] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0030] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0031] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0032] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0033] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0034] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0035] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0036] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0037] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0039] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0040] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0041] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0042] Application Overview

[0043] Current lane line semantic segmentation methods based on single-frame images are prone to instability in scenarios such as underground parking garages due to factors such as unclear lane lines and low lighting, or discontinuous semantic segmentation due to factors such as severe wear and tear on lane lines. This can easily lead to missed detections and incomplete lane line modeling.

[0044] Traditional lane line modeling methods, which use piecewise curves to represent lane lines, typically leave a segment unchanged once it's determined, making it difficult to flexibly adjust lane line models based on the latest observation data. In scenarios like underground parking garages, lane line shapes and topologies are far more complex than on highways, and traditional polynomial curve and arc-length curve representations often fail to provide effective fitting in certain situations. Therefore, for common T-shaped and Y-shaped lane lines in underground parking garages, as well as irregular lane line bends, more flexible modeling methods are needed to improve modeling quality.

[0045] The embodiments disclosed herein aim to solve the above problems and propose a more flexible lane line vectorization modeling method. The method mainly adopts real-time semantic segmentation of multi-frame images, fuses the results of multi-frame semantic segmentation, and then generates vectorized lane lines in real time based on real-time updated lane line instances, so as to improve the integrity and stability of lane line modeling.

[0046] Exemplary System

[0047] Figure 1 An exemplary system architecture 100 is shown for a lane line generation method or lane line generation apparatus to which embodiments of the present disclosure may be applied.

[0048] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102, a server 103, and a vehicle 104. The terminal device may be installed on the vehicle 104. The network 102 serves as the medium for providing a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0049] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications, such as map applications and navigation applications, can be installed on terminal device 101.

[0050] Terminal device 101 can be various electronic devices, including but not limited to in-vehicle terminals, mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc. Terminal device 101 is typically installed in vehicle 104.

[0051] A camera 1041 is installed on the vehicle 104. The camera 1041 can capture images of the road that the vehicle 104 is traveling on in real time, and obtain image frames for semantic segmentation.

[0052] Server 103 can be a server that provides various services, such as a background image server that recognizes image frames uploaded by terminal device 101. The background image server can perform speech segmentation on the received image frames and generate vectorized lane lines based on the semantic segmentation results.

[0053] It should be noted that the lane line generation method provided in the embodiments of this disclosure can be executed by the server 103 or by the terminal device 101. Accordingly, the lane line generation device can be set in the server 103 or in the terminal device 101.

[0054] It should be understood that Figure 1 The number of terminal devices 101, network 102, server 103, and vehicle 104 in the diagram is merely illustrative. Depending on implementation needs, any number of terminal devices 101, network 102, server 103, and vehicle 104 can be included. If image frames do not need to be acquired remotely, the above system architecture may exclude the network and server, including only vehicle 104 and terminal device 101.

[0055] Exemplary methods

[0056] Figure 2 This is a schematic flowchart of a lane line generation method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices (such as...). Figure 1On the terminal device 101 or server 103 shown, such as Figure 2 As shown, the method includes the following steps:

[0057] Step 201: Perform semantic segmentation on the current image frame to obtain the first point cloud representing the lane lines.

[0058] In this embodiment, the electronic device can perform semantic segmentation on the current image frame to obtain a first point cloud representing the lane lines. The current image frame is as follows: Figure 1 The image shown is captured by camera 1041. The method for semantic segmentation of the image to obtain the first point cloud can be implemented using relevant techniques, such as the IPM (Inverse Perspective Mapping) method, which transforms the current image frame to a point cloud. Figure 3A and Figure 3B The VCS coordinate system (Vehicle Coordinate System) shown is a coordinate system with a top-down view of the ground. Existing point cloud generation methods (such as extracting three-dimensional point clouds from two-dimensional images based on neural networks) and point cloud segmentation methods are used to perform point cloud semantic segmentation on the image under the top-down view to obtain the first point cloud representing the lane lines.

[0059] Optionally, when performing semantic segmentation on the current image frame, a quadtree data structure can be used to store the point cloud. The principle is to recursively divide a rectangular area in the plane. Non-leaf nodes represent regions, with the regions represented by non-leaf nodes closer to the root node being larger. Leaf nodes represent the smallest rectangle within this recursive hierarchy, and each leaf node contains at least one point in the point cloud. Storing the point cloud in a quadtree format allows for direct determination of the corresponding node based on the point's coordinates, as well as obtaining the point's coordinates from its identifier. It also facilitates obtaining points surrounding a given point, making point cloud processing easier.

[0060] Step 202: Obtain the second point cloud representing the lane line corresponding to the target historical image frame.

[0061] In this embodiment, the electronic device can acquire a second point cloud representing lane lines corresponding to a target historical image frame. The target historical image frame can be any specified historical image frame; for example, it is typically the image frame preceding the current image frame. Since this method executes sequentially on each of multiple image frames according to a preset order, each historical image frame can correspond to a set of point clouds representing lane lines, thereby enabling the acquisition of the second point cloud corresponding to the target historical image frame.

[0062] Step 203: Determine the incremental point cloud of the first point cloud relative to the second point cloud.

[0063] In this embodiment, the electronic device can determine the incremental point cloud of the first point cloud relative to the second point cloud.

[0064] In this context, the incremental point cloud can be the point cloud that is added to or reduced from the first point cloud relative to the second point cloud. For example... Figure 3B As shown, 301 is the region containing the second point cloud in the image frame corresponding to time t (i.e., the target historical image frame), and 302 is the region containing the first point cloud in the image frame corresponding to time t+1 (i.e., the current image frame). The point clouds in regions 3021 and 3022 are both incremental point clouds. The point cloud in region 3021 is a positive incremental point cloud, that is, an added point cloud, while the point cloud in region 3022 is a negative incremental point cloud, that is, a decreased point cloud.

[0065] Step 204: Cluster the first point cloud to obtain at least one point cloud cluster.

[0066] In this embodiment, the electronic device can cluster the first point cloud to obtain at least one point cloud cluster.

[0067] Specifically, the method for clustering the first point cloud can be implemented using relevant technologies, such as the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm.

[0068] Step 205: Match at least one point cloud cluster with at least one preset historical lane line instance to obtain the matching result.

[0069] In this embodiment, the electronic device can match at least one point cloud cluster with at least one preset historical lane line instance to obtain a matching result. The at least one historical lane line instance is determined during semantic segmentation of the target historical image frame. Each historical lane line instance can represent a lane line, and each historical lane line instance can include a point cloud cluster composed of multiple points, such as... Figure 3A As shown, a′, b′, and c′ represent three historical lane line instances.

[0070] In this embodiment, the method for matching point cloud clusters with historical lane line instances can be performed using conventional means. For example, for a point cloud cluster, the distance between the point cloud cluster and each point cloud cluster included in each historical lane line instance is determined (the distance between each pair of point cloud cluster center points can be used as the distance between each pair of point cloud clusters). If the minimum distance among the determined distances is less than or equal to a preset distance threshold, then the point cloud cluster is determined to be matched with the historical lane line instance corresponding to the minimum distance. As another example, suppose a point cloud cluster has a total of N points and M target points. The minimum distance between each target point and each point in historical lane line instance a is less than a preset distance. If the number of target points M in the point cloud cluster is greater than or equal to a preset number, and / or the ratio of the number of target points in the point cloud cluster to the total number of points in the point cloud cluster (M / N) is greater than or equal to a preset ratio threshold, then the point cloud cluster is determined to be matched with historical lane line instance a.

[0071] Step 206: Based on the matching results, generate at least one lane line instance corresponding to the current image frame using incremental point cloud.

[0072] In this embodiment, the electronic device can generate at least one lane line instance corresponding to the current image frame based on the matching result using incremental point cloud.

[0073] Specifically, if the matching result indicates that the point cloud cluster containing the incremental point cloud matches the target historical lane line instance, the target historical lane line instance can be updated based on the incremental point cloud. For example, the added point cloud can be added to the target historical lane line instance, or the reduced point cloud can be deleted from the target historical lane line instance. Then, the updated historical lane line instance is determined as the lane line instance corresponding to the current image frame.

[0074] If the matching result indicates that the incremental point cloud does not match any of the target historical lane line instances, then the incremental point cloud can be used to create a new lane line instance as the lane line instance corresponding to the current image frame.

[0075] like Figure 3B As shown, the positive incremental point cloud in 3021 is added to the target historical lane line instances a′ and c′, and the negative incremental point cloud in 3022 is deleted from the target historical lane line instance c′, generating lane line instances a and c corresponding to the current image frame. Since historical lane line instance b′ does not contain incremental point cloud, it is directly determined as the lane line instance b corresponding to the current image frame.

[0076] Step 207: Generate vectorized lane lines corresponding to at least one lane line instance based on the point cloud corresponding to each lane line instance.

[0077] Lane vectorization refers to constructing a model representing lane lines by identifying lane lines and using a series of endpoints and directed line segments. Lane vectorization modeling can flexibly represent various complex lane line structures by adaptively adjusting the distribution of endpoints. The detailed structure of lane lines can be accurately described by increasing the density of local endpoints, and complex curves can be approximated by replacing curves with multiple straight lines.

[0078] In this embodiment, the electronic device can generate vectorized lane lines corresponding to at least one lane line instance based on the point cloud corresponding to each lane line instance. The point cloud corresponding to each lane line instance is as follows: Figure 3B The point cloud encompassed within the coverage area of ​​lane line instances a, b, and c shown. A vectorized lane line is a model representing a lane line, consisting of a series of endpoints and directed line segments connecting the endpoints. The direction of the directed line segments is determined according to the vehicle's travel direction.

[0079] like Figure 3D As shown, for lane line instances a, b, and c, each corresponds to a vectorized lane line. Each vectorized lane line consists of multiple endpoints (including endpoint AJ) and multiple directed line segments represented by dashed lines with arrows.

[0080] Optionally, at least one lane line instance can be recalculated to obtain the vectorized lane line corresponding to each lane line instance. Alternatively, based on the above incremental point cloud, the endpoints of the vectorized lane lines contained within the region of the incremental point cloud can be updated to obtain the vectorized lane line corresponding to each lane line instance.

[0081] As an example, image thinning or region growing methods of relevant technologies can be used to extract a series of points from each point cloud instance, and then these points can be reduced using methods such as Douglas-Puk to obtain the endpoints included in the vectorized lane lines, and then the endpoints can be connected by directed line segments.

[0082] The generated vectorized lane lines can accurately and flexibly describe complex lane line structures, providing a foundation for applications such as autonomous driving and advanced driver assistance systems.

[0083] The method provided in the above embodiments of this disclosure performs semantic segmentation on multiple consecutive frames of images to determine the incremental point cloud between the first point cloud corresponding to the current image frame and the second point cloud corresponding to the target historical image frame. Then, the first point cloud is clustered, and the resulting point cloud clusters are matched with at least one preset historical lane line instance. Based on the matching result, at least one lane line instance corresponding to the current image frame is generated using the incremental point cloud. Finally, based on the point clouds corresponding to at least one lane line instance, vectorized lane lines corresponding to at least one lane line instance are generated. This disclosure effectively utilizes the semantic segmentation results of multiple frames of images to update historical lane line instances in real time. The updated point cloud corresponding to the lane line instance is used to update the vectorized lane lines in real time, establishing a connection between the updated vectorized lane lines and historical lane line instances. This helps reduce the risk of incomplete lane line vectorization modeling caused by unclear lane lines or low light conditions, thereby improving the integrity and stability of lane line vectorization modeling. Furthermore, since the embodiments of this disclosure are executed sequentially for each frame of multiple images in a preset order, when the number of image frames accumulated in the point cloud is large, the risk of misidentifying different lane line instances as the same lane line instance can be effectively reduced, thereby improving the accuracy of generating lane line instances and vectorized lane lines.

[0084] In some alternative implementations, such as Figure 4 As shown, step 206 includes:

[0085] Step 2061: In response to the determined matching result indicating that the point cloud cluster where the incremental point cloud is located matches the target historical lane line instance, the target historical lane line instance is updated based on the incremental point cloud, and the updated target historical lane line instance is determined as the target lane line instance included in at least one lane line instance corresponding to the current image frame.

[0086] Specifically, when updating a target historical lane line instance, you can add the added point cloud to the target historical lane line instance or delete the reduced point cloud from the target historical lane line instance.

[0087] In this embodiment, when the incremental point cloud matches the target historical lane line instance, the target historical lane line instance is updated based on the incremental point cloud. This realizes the continuous updating of lane line instances based on multiple frames of images. In non-ideal conditions such as unclear actual lane lines or dim lighting, it helps to continuously build complete and clear lane line instances based on historical lane line instances.

[0088] In some alternative implementations, such as Figure 4 As shown, step 207 includes:

[0089] Step 2071: Using incremental point cloud, update the set of endpoints included in the target historical vectorized lane line corresponding to the target historical lane line instance.

[0090] The endpoint set is the set of endpoints of at least one directed line segment included in the target historical vectorized lane line.

[0091] As an example, existing image thinning or region generation methods can be used to determine a series of endpoints representing lane lines within the target range from the point cloud containing incremental point clouds, while deleting the original endpoints within the target range, thereby updating the endpoint set of the target's historical vectorized lane lines.

[0092] Step 2072: Based on the updated endpoint set, generate the target vectorized lane line corresponding to the target lane line instance.

[0093] Specifically, the target vectorized lane line can be obtained by connecting the endpoints included in the updated endpoint set using directed line segments.

[0094] This embodiment updates the target's historical vectorized lane lines based on incremental point clouds, enabling targeted adjustments to some endpoints of the target vectorized lane lines. This helps reduce the risk of abrupt changes in the updated vectorized lane lines and improves the stability and integrity of the generated vectorized lane lines.

[0095] In some alternative implementations, such as Figure 5 As shown, step 2071 includes:

[0096] Step 20711: Based on the incremental point cloud and the preset distance relationship, determine the target directed line segment from at least one directed line segment included in the target historical vectorized lane line.

[0097] Specifically, the aforementioned preset distance relationships can be set arbitrarily. For example, a directed line segment whose minimum distance to each point included in the incremental point cloud is within a preset distance threshold can be determined from at least one directed line segment as the target directed line segment; or a directed line segment whose distance to the center point of the incremental point cloud is within a preset distance threshold can be determined from at least one directed line segment as the target directed line segment.

[0098] Optionally, the incremental point cloud can be thinned first, that is, some points are extracted from the incremental point cloud as points representing the coverage area of ​​the incremental point cloud. Using the thinned points, the target directed line segment can be determined based on the above-mentioned preset distance relationship.

[0099] Step 20712: Based on the target directed line segment and the incremental point cloud, determine the updated endpoint corresponding to the target directed line segment using a preset loss function.

[0100] The loss function includes at least one of the following: the distance between the updated endpoint and the endpoint of the target directed line segment, the distance between the points corresponding to the target directed line segment in the incremental point cloud and the target directed line segment, and the angle between the target directed line segment and the updated directed line segment after updating the backend point.

[0101] Optionally, updating the endpoints of the target directed line segment can be achieved using a nonlinear optimization framework such as Ceres. The loss function used in nonlinear optimization can include the three items mentioned above.

[0102] For a target directed line segment, the variables to be optimized are the two endpoints of this target directed line segment. Before optimization, the coordinates of the two endpoints are p. a and p b Let S be the point set of the incremental point cloud, where S is the observation. Traverse the points in S and iteratively calculate the updated endpoints. Let p be the updated endpoint coordinates of the current iteration step. a ′ and p b ′, according to p a ′ and p b A line segment L can be obtained from these variables. Based on these variables, the loss function used for iterative updates is shown in equation (1) below:

[0103] Loss=∑ p∈S dist(p,L)+[dist(p a ,p a ′)+dist(p b ,p b ′)]+ang(l,l′) (1)

[0104] Here, `dist()` calculates the distance between the two elements within the parentheses, `ang(l,l′)` calculates the angle between line segments `l` and `l′`, and `l` represents point `p`. a Point P b The vector, l′ represents point p. a ′ to point p b The vector of '. Using the above loss function, the endpoint p is iteratively optimized. a ′ and p b The coordinates of the points are calculated until the loss converges. It should be noted that the loss function shown in equation (1) is only illustrative. In actual calculation of the loss value, it can be calculated based on the above variables. For example, the square root of the sum of squares of the three terms in equation (1), or the weighted sum of the three terms, etc. The last two terms in equation (1), namely the amount of movement of the updated endpoint relative to the original endpoint and the angle between the updated directed line segments, can be used as regularization terms to constrain the update range of the loss and avoid drastic fluctuations in the calculation results.

[0105] like Figure 3CAs shown, d′ is the vectorized lane line before the update, and d is the vectorized lane line after the update.

[0106] This embodiment constructs a loss function and iteratively calculates and updates the coordinates of the backend points, which can more accurately update the vectorized lane lines based on the incremental point cloud and improve the accuracy of determining the vectorized lane lines.

[0107] In some alternative implementations, such as Figure 6 As shown, step 206 includes:

[0108] Step 2062: In response to the matching result indicating that there is no matching historical lane line instance in the point cloud cluster where the incremental point cloud is located, determine whether the incremental point cloud meets the preset conditions for creating a new instance.

[0109] Among them, the conditions for creating a new instance are the conditions for generating a new lane line instance. These conditions can be set in any applicable way based on the actual situation.

[0110] Optionally, the conditions for creating a new instance include at least one of the following: the number of points in the point cloud cluster containing the incremental point cloud meets a point count threshold condition, and the cumulative number of image frames corresponding to the point cloud cluster containing the incremental point cloud meets a frame count threshold condition. The point count threshold condition is typically: the number of points in the point cloud cluster containing the incremental point cloud is greater than or equal to a preset point count threshold. The frame count threshold condition is typically: the cumulative number of image frames corresponding to the point cloud cluster containing the incremental point cloud is greater than or equal to a preset frame count threshold.

[0111] By setting this new instance condition, lane line instances can be created after the incremental point cloud reaches a certain number or after the incremental point cloud has accumulated for a certain period of time, reducing the error caused by short-term changes in the incremental point cloud when creating new lane line instances.

[0112] Step 2063: In response to the incremental point cloud meeting the conditions for creating a new instance, a new lane line instance is generated based on the incremental point cloud.

[0113] Generating new lane line instances involves constructing a region covering the incremental point cloud and establishing corresponding identification information. A new lane line instance is a lane line instance included in at least one lane line instance corresponding to an image frame.

[0114] This embodiment enables the generation of new lane line instances when no matching historical lane line instances exist in the point cloud cluster containing the incremental point cloud, provided that the conditions for creating new instances are met. This allows for the timely and accurate creation of new lane line instances, which helps improve the accuracy and efficiency of generating vectorized lane lines.

[0115] In some alternative implementations, such as Figure 6 As shown, step 207 includes:

[0116] Step 2073: Based on the point cloud corresponding to the newly created lane line instance, determine the set of newly created endpoints included in the newly created vectorized lane line to be generated.

[0117] Optionally, existing methods such as image thinning and region growing can be used to extract a series of points from each point cloud instance, and then these points can be reduced using methods such as Douglas-Puk to obtain the endpoints included in the vectorized lane lines, and the endpoints can be connected by directed line segments.

[0118] The image thinning method converts the point cloud into an image, uses an image thinning algorithm to extract the skeleton of the lane line region in the image, and then converts this skeleton from the image domain into a vectorized point sequence. The region growing method starts from a seed point, generates several search directions, determines which search direction allows continued searching of this cluster of points, and continuously searches forward to obtain a search path. The points along the longest search path constitute a vectorized point sequence. Further methods such as Douglas-Poké can be used to prune the points in the obtained vectorized point sequence, thus obtaining a new set of endpoints.

[0119] Step 2074: Based on the newly created endpoint set, generate the newly created vectorized lane line corresponding to the newly created lane line instance.

[0120] Specifically, the target vectorized lane line can be obtained by connecting the endpoints included in the updated endpoint set using directed line segments.

[0121] This embodiment generates new vectorized lane lines based on the point cloud corresponding to the newly created lane line instance. This can generate new vectorized lane lines in a timely and accurate manner, providing a basis for subsequent real-time updates of vectorized lane lines and improving the accuracy and efficiency of generating vectorized lane lines.

[0122] In some alternative implementations, such as Figure 7 As shown, after step 206, the method further includes:

[0123] Step 208: Determine the positional relationship information between at least one pair of lane line instances.

[0124] The location relationship information includes, but is not limited to, the distance between the two closest target points of the two lane line instances, and the location of the target area (e.g., the area formed by points within a preset distance from the target point) where the two closest target points of the two lane line instances are located.

[0125] Step 209: In response to determining that at least one obtained positional relationship information includes target positional relationship information that meets preset merging conditions, the two lane line instances corresponding to the target positional relationship information are merged into a new lane line instance.

[0126] The preset merging conditions can be set arbitrarily.

[0127] Optionally, the preset merging conditions include: the distance between the target endpoints of the two lane line instances meets the preset distance threshold condition; and / or, there is an overlapping area between the two lane line instances.

[0128] The target endpoint can be the closest point between two lane line instances, and the preset distance threshold condition can be that the distance between the target endpoints is less than or equal to a preset distance threshold. For example... Figure 8 As shown, if the distance between the endpoints 8011 and 8021 of lane line instance 801 and lane line instance 802 is less than or equal to a preset distance threshold, then lane line instance 801 and lane line instance 802 can be merged into a new lane line instance 803.

[0129] One way to determine whether there is an overlapping region between two lane line instances is to uniformly sample the point clouds included in the two lane line instances A and B respectively. For lane line instance A, it includes at least N target sampling points. If there are sampling points included in lane line instance B within a preset distance of these target sampling points, then it can be determined that there is an overlapping region between lane line instances A and B.

[0130] Figure 7 The corresponding embodiment determines the positional relationship information between lane line instances, and merges the lane line instances that meet the conditions based on the positional relationship information, which can make the lane line instances more complete and thus generate a more complete vectorized lane line.

[0131] In some alternative implementations, such as Figure 9 As shown, after step 207, the method further includes:

[0132] Step 210: Determine the location information representing the position of the target vehicle in the current image frame.

[0133] The target vehicle can be the vehicle where the camera that captured the current image frame is located, for example... Figure 1 The vehicle 104 shown. The position information of the target vehicle may be information such as the position of the center point of the rear axle of the target vehicle.

[0134] Step 211: Based on location information, determine the boundary line used to distinguish between non-updatable and updatable endpoints.

[0135] Typically, the points indicated by the above location information can be used as reference points for determining the boundary line. In the current image frame, the horizontal line passing through the reference point can be determined as the boundary line, or the straight line passing through the reference point and perpendicular to the driving direction of the target vehicle can be determined as the boundary line.

[0136] Step 212: Based on the boundary line, determine the non-updatable endpoints and updatable endpoints from the endpoints included in the vectorized lane lines corresponding to at least one lane line instance.

[0137] In this context, updatable endpoints are those used when generating vectorized lane lines using image frames following the current image frame, while non-updatable endpoints are those that cannot be used when generating vectorized lane lines using image frames following the current image frame. That is, when the method provided in this embodiment is executed again using image frames following the current image frame to update historical vectorized lane lines, only updatable endpoints can be used. The positions of updatable endpoints can change, while the positions of non-updatable endpoints remain fixed. Non-updatable endpoints can be used for target localization and target modeling, etc. Updatable endpoints, because they are updated in real time, have better timeliness than non-updatable endpoints and can be used for vehicle path planning, control, etc.

[0138] Typically, endpoints within the image region above the boundary line can be defined as updatable endpoints, while endpoints within the image region below the boundary line can be defined as non-updatable endpoints. For example... Figure 3D As shown, endpoint A below the boundary line is a non-updatable endpoint, while endpoints above the boundary line are updatable endpoints.

[0139] This embodiment determines non-updatable and updatable endpoints in at least one lane line instance, enabling the use of more timely updatable endpoints when updating vectorized lane lines, thereby effectively improving the efficiency and accuracy of real-time updating of vectorized lane lines.

[0140] In some alternative implementations, such as Figure 10 As shown, step 207 includes:

[0141] Step 2075: Based on the point cloud corresponding to at least one lane line instance, generate the initial vectorized lane line corresponding to at least one lane line instance.

[0142] The initial vectorized lane lines can be generated based on methods such as image thinning and region growing described in the above embodiments.

[0143] Step 2076: Smooth the initial vectorized lane lines to obtain vectorized lane lines corresponding to at least one lane line instance.

[0144] Smoothing is used to eliminate abrupt changes in the vectorized lane lines while reducing the number of endpoints. Optionally, smoothing methods can include the Douglas-Pokk method, digital filtering methods, etc. These methods can reduce the number of endpoints in the initial vectorized lane lines, retaining a portion of the endpoints that can reflect curvature changes, and deleting redundant endpoints that are basically collinear with these endpoints, so as to maintain a low redundancy in the final endpoint set and eliminate small fluctuations in the vectorized lane lines.

[0145] This embodiment effectively reduces fluctuations in vectorized lane lines by smoothing the initial vectorized lane lines, improving the accuracy of using vectorized lane lines to represent lane lines. At the same time, it reduces the number of endpoints of the vectorized lane lines, which helps to improve the efficiency of updating vectorized lane lines.

[0146] In some alternative implementations, such as Figure 11 As shown, step 201 includes:

[0147] Step 2011: Perform semantic segmentation on the current image frame to obtain the initial point cloud representing the lane lines.

[0148] Step 2012: Obtain the driving status information of the target vehicle.

[0149] The driving status information can be obtained through sensors on the target vehicle. For example, vibration sensors or pitch angle sensors can be used to collect driving status information indicating the degree of bumps in the target vehicle.

[0150] Step 2013: In response to the determination of driving status information indicating that the target vehicle is in a stable driving state, based on the position of the target vehicle, the point cloud within the first range is determined from the initial point cloud as the first point cloud.

[0151] The first range can be a range set with the location of the target vehicle as the reference point. For example, the first range can be a rectangular area, with the reference point being the center of the bottom edge of the rectangular area, and the length and width of the rectangular area being preset lengths.

[0152] Typically, driving status information can include a value indicating the degree of bumpiness of the target vehicle. If the value is greater than or equal to a preset value, the target vehicle is determined to be in an unstable driving state; if the value is less than the preset value, the target vehicle is determined to be in a stable driving state.

[0153] Step 2014: In response to the determination that the driving state information indicates that the target vehicle is in a non-stable driving state, based on the position of the target vehicle, the point cloud in the second range is determined from the initial point cloud as the first point cloud.

[0154] The area of ​​the second range is smaller than the area of ​​the first range.

[0155] This embodiment determines the point cloud within a certain range as the point cloud used to generate lane line instances, which can avoid the interference of point clouds far away from the target vehicle on the generation of lane line instances and vectorized lane lines. At the same time, when the vehicle is in a non-steady state, selecting point clouds within a smaller range can effectively reduce noise interference and improve the stability of real-time updates of vectorized lane lines.

[0156] Exemplary device

[0157] Figure 12 This is a schematic diagram of a lane line generation device provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 12 As shown, the lane line generation device includes: a segmentation module 1201, used to perform semantic segmentation on the current image frame to obtain a first point cloud representing the lane line; an acquisition module 1202, used to acquire a second point cloud representing the lane line corresponding to a target historical image frame; a first determination module 1203, used to determine the incremental point cloud of the first point cloud relative to the second point cloud; a clustering module 1204, used to cluster the first point cloud to obtain at least one point cloud cluster; a matching module 1205, used to match the at least one point cloud cluster with at least one preset historical lane line instance to obtain a matching result; a first generation module 1206, used to generate at least one lane line instance corresponding to the current image frame based on the matching result and using the incremental point cloud; and a second generation module 1207, used to generate vectorized lane lines corresponding to at least one lane line instance based on the point clouds corresponding to each of the at least one lane line instance.

[0158] In this embodiment, the segmentation module 1201 can perform semantic segmentation on the current image frame to obtain a first point cloud representing the lane lines. The current image frame is as follows: Figure 1 The image shown is captured by camera 1041. Semantic segmentation of the image to obtain the first point cloud can be achieved using relevant techniques, such as IPM (Inverse Perspective Mapping), which transforms the current image frame to a point cloud. Figure 3A and Figure 3B The VCS coordinate system (Vehicle Coordinate System) shown is a coordinate system with a top-down view of the ground. Existing point cloud generation methods (such as extracting three-dimensional point clouds from two-dimensional images based on neural networks) and point cloud segmentation methods are used to perform point cloud semantic segmentation on the image under the top-down view to obtain the first point cloud representing the lane lines.

[0159] In this embodiment, the acquisition module 1202 can acquire the second point cloud representing lane lines corresponding to the target historical image frame. The target historical image frame can be any specified historical image frame; for example, it is typically the image frame preceding the current image frame. Since this method executes sequentially on each of multiple image frames according to a preset order, each historical image frame can correspond to a set of point clouds representing lane lines, thereby enabling the acquisition of the second point cloud corresponding to the target historical image frame.

[0160] In this embodiment, the first determining module 1203 can determine the incremental point cloud of the first point cloud relative to the second point cloud.

[0161] In this context, the incremental point cloud can be the point cloud that is added to or reduced from the first point cloud relative to the second point cloud. For example... Figure 3B As shown, 301 is the region containing the second point cloud in the image frame corresponding to time t (i.e., the target historical image frame), and 302 is the region containing the first point cloud in the image frame corresponding to time t+1 (i.e., the current image frame). The point clouds in regions 3021 and 3022 are both incremental point clouds. The point cloud in region 3021 is a positive incremental point cloud, that is, an added point cloud, while the point cloud in region 3022 is a negative incremental point cloud, that is, a decreased point cloud.

[0162] In this embodiment, the clustering module 1204 can cluster the first point cloud to obtain at least one point cloud cluster.

[0163] Specifically, the method for clustering the first point cloud can be implemented using relevant technologies, such as the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm.

[0164] In this embodiment, the matching module 1205 can match at least one point cloud cluster with at least one preset historical lane line instance to obtain a matching result. The at least one historical lane line instance is determined during semantic segmentation of the target historical image frame. Each historical lane line instance can represent a lane line, and each historical lane line instance can include a point cloud cluster composed of multiple points, such as... Figure 3A As shown, a′, b′, and c′ represent three historical lane line instances.

[0165] In this embodiment, the first generation module 1206 can generate at least one lane line instance corresponding to the current image frame based on the matching result using incremental point cloud.

[0166] Specifically, if the matching result indicates that the point cloud cluster containing the incremental point cloud matches the target historical lane line instance, the target historical lane line instance can be updated based on the incremental point cloud. For example, the added point cloud can be added to the target historical lane line instance, or the reduced point cloud can be deleted from the target historical lane line instance. Then, the updated historical lane line instance is determined as the lane line instance corresponding to the current image frame.

[0167] If the matching result indicates that the incremental point cloud does not match any of the target historical lane line instances, then the incremental point cloud can be used to create a new lane line instance as the lane line instance corresponding to the current image frame.

[0168] In this embodiment, the second generation module 1207 can generate vectorized lane lines corresponding to at least one lane line instance based on the point cloud corresponding to each lane line instance. The point cloud corresponding to each lane line instance is as follows: Figure 3B The point cloud encompassed within the coverage area of ​​lane line instances a, b, and c shown. A vectorized lane line is a model representing a lane line, consisting of a series of endpoints and directed line segments connecting the endpoints. The direction of the directed line segments is determined according to the vehicle's travel direction.

[0169] Reference Figure 13 , Figure 13 This is a schematic diagram of the structure of a lane line generation apparatus provided in another exemplary embodiment of this disclosure.

[0170] In some optional implementations, the first generation module 1206 includes: a first update unit 12061, which, in response to a determined matching result indicating that the point cloud cluster where the incremental point cloud is located matches the target historical lane line instance, updates the target historical lane line instance based on the incremental point cloud, and determines the updated target historical lane line instance as the target lane line instance included in at least one lane line instance corresponding to the current image frame.

[0171] In some optional implementations, the second generation module 1207 includes: a second updating unit 12071, used to update the endpoint set of the target historical vectorized lane line corresponding to the target historical lane line instance using incremental point cloud, wherein the endpoint set is the set of endpoints of at least one directed line segment included in the target historical vectorized lane line; and a first generation unit 12072, used to generate the target vectorized lane line corresponding to the target lane line instance based on the updated endpoint set.

[0172] In some optional implementations, the second update unit 12071 includes: a first determining subunit 120711, used to determine a target directed line segment from at least one directed line segment included in the target historical vectorized lane line based on the incremental point cloud and a preset distance relationship; and a second determining subunit 120712, used to determine the updated endpoint corresponding to the target directed line segment based on the target directed line segment and the incremental point cloud, using a preset loss function, wherein the loss function includes at least one of the following: the distance between the updated endpoint and the endpoint of the target directed line segment, the distance between the points in the incremental point cloud corresponding to the target directed line segment and the target directed line segment, and the angle between the target directed line segment and the updated directed line segment after the updated backend point.

[0173] In some optional implementations, the first generation module 1206 includes: a first determining unit 12062, used to determine whether the incremental point cloud meets the preset conditions for creating a new instance in response to the matching result indicating that there is no matching historical lane line instance in the point cloud cluster where the incremental point cloud is located; and a second generation unit 12063, used to generate a new lane line instance based on the incremental point cloud in response to the incremental point cloud meeting the conditions for creating a new instance.

[0174] In some optional implementations, the conditions for creating a new instance include at least one of the following: the number of points in the point cloud cluster to which the incremental point cloud is located meets the point number threshold condition, and the cumulative number of image frames corresponding to the point cloud cluster to which the incremental point cloud is located meets the frame number threshold condition.

[0175] In some optional implementations, the second generation module 1207 includes: a second determining unit 12073, used to determine the set of newly created endpoints included in the newly created vectorized lane line to be generated based on the point cloud corresponding to the newly created lane line instance; and a third generation unit 12074, used to generate the newly created vectorized lane line corresponding to the newly created lane line instance based on the set of newly created endpoints.

[0176] In some optional implementations, the device further includes: a second determining module 1208, configured to determine positional relationship information between at least one pair of lane line instances; and a merging module 1209, configured to merge two lane line instances corresponding to the target positional relationship information into a new lane line instance in response to the determination that the at least one positional relationship information includes target positional relationship information that meets preset merging conditions.

[0177] In some optional implementations, the preset merging conditions include: the distance between the target endpoints of the two lane line instances meets a preset distance threshold condition; and / or, there is an overlapping area between the two lane line instances.

[0178] In some optional implementations, the device further includes: a third determining module 1210, configured to determine position information representing the position of the target vehicle in the current image frame; a fourth determining module 1211, configured to determine a boundary line for distinguishing between non-updatable endpoints and updatable endpoints based on the position information; and a fifth determining module 1212, configured to determine non-updatable endpoints and updatable endpoints based on the boundary line from the endpoints included in the vectorized lane lines corresponding to at least one lane line instance, wherein the updatable endpoints are the endpoints used when generating vectorized lane lines using image frames after the current image frame, and the non-updatable endpoints are the endpoints that cannot be used when generating vectorized lane lines using image frames after the current image frame.

[0179] In some optional implementations, the second generation module 1207 includes: a fourth generation unit 12075, used to generate initial vectorized lane lines corresponding to at least one lane line instance based on the point cloud corresponding to at least one lane line instance; and a smoothing processing unit 12076, used to smooth the initial vectorized lane lines to obtain vectorized lane lines corresponding to at least one lane line instance.

[0180] In some optional implementations, the segmentation module 1201 includes: a segmentation unit 12011, used to perform semantic segmentation on the current image frame to obtain an initial point cloud representing lane lines; an acquisition unit 12012, used to acquire driving state information of the target vehicle; a third determination unit 12013, used to determine, in response to the determination that the driving state information indicates that the target vehicle is in a stable driving state, to determine, based on the position of the target vehicle, a point cloud within a first range from the initial point cloud as a first point cloud; and a fourth determination unit 12014, used to determine, in response to the determination that the driving state information indicates that the target vehicle is in a non-stable driving state, to determine, based on the position of the target vehicle, a point cloud within a second range from the initial point cloud as a first point cloud, wherein the area of ​​the second range is smaller than the area of ​​the first range.

[0181] The lane line generation apparatus provided in the above embodiments of this disclosure performs semantic segmentation on multiple consecutive frames of images to determine the incremental point cloud between the first point cloud corresponding to the current image frame and the second point cloud corresponding to the target historical image frame. Then, it clusters the first point cloud and matches the resulting point cloud clusters with at least one preset historical lane line instance. Based on the matching result, it uses the incremental point cloud to generate at least one lane line instance corresponding to the current image frame. Finally, based on the point clouds corresponding to at least one lane line instance, it generates vectorized lane lines corresponding to at least one lane line instance. This disclosure effectively utilizes the semantic segmentation results of multiple frames of images to update historical lane line instances in real time. It uses the point cloud corresponding to the updated lane line instance to update the vectorized lane line in real time, so that the updated vectorized lane line establishes a connection with the historical lane line instance. This helps to reduce the risk of incomplete lane line vectorization modeling caused by unclear lane lines or low light conditions, thereby improving the accuracy of generating lane line instances and vectorized lane lines.

[0182] Exemplary electronic devices

[0183] Below, for reference Figure 14 To describe an electronic device according to embodiments of the present disclosure. The electronic device may be as follows: Figure 1 The terminal device 101 and server 103 shown, or either one or both, or a standalone device independent of them, can communicate with the terminal device 101 and server 103 to receive the collected input signals from them.

[0184] Figure 14 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0185] like Figure 14 As shown, the electronic device 1400 includes one or more processors 1401 and memory 1402.

[0186] The processor 1401 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1400 to perform desired functions.

[0187] The memory 1402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1401 may execute the program instructions to implement the lane line generation methods of the various embodiments of this disclosure described above and / or other desired functions. Various content such as image frames and point clouds may also be stored in the computer-readable storage medium.

[0188] In one example, the electronic device 1400 may also include an input device 1403 and an output device 1404, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0189] For example, when the electronic device is a terminal device 101 or a server 103, the input device 1403 can be a camera, mouse, keyboard, or other devices used to input image frames, various commands, and other content. When the electronic device is a standalone device, the input device 1403 can be a communication network connector used to receive input image frames, various commands, and other content from the terminal device 101 and the server 103.

[0190] The output device 1404 can output various information to the outside, including generated vectorized lane lines. The output device 1404 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0191] Of course, for the sake of simplicity, Figure 14 Only some of the components of the electronic device 1400 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1400 may include any other suitable components depending on the specific application.

[0192] Exemplary computer program products and computer-readable storage media

[0193] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the lane line generation methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0194] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0195] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the lane line generation methods according to various embodiments of this disclosure described in the "Exemplary Methods" section of this specification.

[0196] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0197] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0198] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0199] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0200] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0201] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0202] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0203] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for generating lane lines, comprising: Perform semantic segmentation on the current image frame to obtain the first point cloud representing the lane lines; Obtain the second point cloud representing the lane lines corresponding to the target historical image frame; Determine the incremental point cloud of the first point cloud relative to the second point cloud; Cluster the first point cloud to obtain at least one point cloud cluster; The at least one point cloud cluster is matched with at least one preset historical lane line instance to obtain the matching result; Based on the matching result, at least one lane line instance corresponding to the current image frame is generated using the incremental point cloud. Based on the point cloud corresponding to the at least one lane line instance, generate vectorized lane lines corresponding to the at least one lane line instance. The step of generating at least one lane line instance corresponding to the current image frame based on the matching result using the incremental point cloud includes: In response to the determined matching result indicating that the point cloud cluster where the incremental point cloud is located matches the target historical lane line instance, the target historical lane line instance is updated based on the incremental point cloud, and the updated target historical lane line instance is determined as the target lane line instance included in at least one lane line instance corresponding to the current image frame. or, In response to the matching result indicating that there is no matching historical lane line instance in the point cloud cluster where the incremental point cloud is located, it is determined whether the incremental point cloud meets the preset conditions for creating a new instance; in response to the incremental point cloud meeting the conditions for creating a new instance, a new lane line instance is generated based on the incremental point cloud.

2. The method of claim 1, wherein, The step of generating vectorized lane lines corresponding to the at least one lane line instance based on the point cloud corresponding to each lane line instance includes: Using the incremental point cloud, the endpoint set of the target historical vectorized lane line corresponding to the target historical lane line instance is updated, wherein the endpoint set is the set of endpoints of at least one directed line segment included in the target historical vectorized lane line. Based on the updated set of endpoints, a target vectorized lane line corresponding to the target lane line instance is generated.

3. The method of claim 2, wherein, The step of updating the endpoint set of the target historical vectorized lane line corresponding to the target historical lane line instance using the incremental point cloud includes: Based on the incremental point cloud and the preset distance relationship, the target directed line segment is determined from at least one directed line segment included in the target historical vectorized lane line; Based on the target directed line segment and the incremental point cloud, the updated endpoint corresponding to the target directed line segment is determined using a preset loss function. The loss function includes at least one of the following: the distance between the updated endpoint and the endpoint of the target directed line segment, the distance between a point in the incremental point cloud and the updated directed line segment corresponding to the updated endpoint, and the angle between the target directed line segment and the updated directed line segment passing through the updated endpoint.

4. The method of claim 1, wherein, The conditions for creating a new instance include at least one of the following: the number of points in the point cloud cluster to which the incremental point cloud is located meets the point number threshold condition, and the cumulative number of image frames corresponding to the point cloud cluster to which the incremental point cloud is located meets the frame number threshold condition.

5. The method of claim 1, wherein, The step of generating vectorized lane lines corresponding to the at least one lane line instance based on the point cloud corresponding to each lane line instance includes: Based on the point cloud corresponding to the newly created lane line instance, determine the set of newly created endpoints included in the newly created vectorized lane line to be generated; Based on the newly created endpoint set, a new vectorized lane line corresponding to the newly created lane line instance is generated.

6. The method of claim 1, wherein, After generating at least one lane line instance corresponding to the current image frame using the incremental point cloud based on the matching result, the method further includes: Determine the positional relationship information between each pair of the at least one lane line instance; In response to determining that at least one obtained positional relationship information includes target positional relationship information that meets preset merging conditions, the two lane line instances corresponding to the target positional relationship information are merged into a new lane line instance.

7. The method of claim 6, wherein, The preset merging conditions include: The distance between the target endpoints of the two lane line instances meets a preset distance threshold condition; and / or, There is an overlapping area between the two lane line instances.

8. The method according to claim 1, wherein, After generating vectorized lane lines corresponding to the at least one lane line instance based on the point cloud corresponding to each of the at least one lane line instance, the method further includes: In the current image frame, determine the location information representing the position of the target vehicle; Based on the location information, a boundary line is determined to distinguish between non-updatable endpoints and updatable endpoints; Based on the boundary line, from the endpoints of the vectorized lane lines corresponding to the at least one lane line instance, non-updatable endpoints and updatable endpoints are determined, wherein the updatable endpoints are the endpoints used when generating vectorized lane lines using image frames after the current image frame, and the non-updatable endpoints are the endpoints that cannot be used when generating vectorized lane lines using image frames after the current image frame.

9. The method according to claim 1, wherein, The step of generating vectorized lane lines corresponding to the at least one lane line instance based on the point cloud corresponding to each lane line instance includes: Based on the point cloud corresponding to the at least one lane line instance, generate the initial vectorized lane line corresponding to the at least one lane line instance; The initial vectorized lane lines are smoothed to obtain the vectorized lane lines corresponding to the at least one lane line instance.

10. The method according to claim 1, wherein, The step of semantic segmentation of the current image frame to obtain the first point cloud representing the lane lines includes: Perform semantic segmentation on the current image frame to obtain the initial point cloud representing the lane lines; Obtain the driving status information of the target vehicle; In response to determining that the driving state information indicates that the target vehicle is in a stable driving state, based on the position of the target vehicle, the point cloud within a first range is determined from the initial point cloud as the first point cloud; In response to determining that the driving state information indicates that the target vehicle is in a non-smooth driving state, based on the position of the target vehicle, a point cloud within a second range is determined from the initial point cloud as the first point cloud, wherein the area of ​​the second range is smaller than the area of ​​the first range.

11. A lane line generating device, comprising: The segmentation module is used to perform semantic segmentation on the current image frame to obtain the first point cloud representing the lane lines; The acquisition module is used to acquire the second point cloud representing the lane lines corresponding to the target historical image frame; The first determining module is used to determine the incremental point cloud of the first point cloud relative to the second point cloud; A clustering module is used to cluster the first point cloud to obtain at least one point cloud cluster; The matching module is used to match the at least one point cloud cluster with at least one preset historical lane line instance to obtain a matching result; A first generation module is configured to generate at least one lane line instance corresponding to the current image frame based on the matching result using the incremental point cloud. The first generation module includes a first update unit, which is configured to update the target historical lane line instance based on the incremental point cloud in response to the determined matching result indicating that the point cloud cluster where the incremental point cloud is located matches the target historical lane line instance, and determine the updated target historical lane line instance as the target lane line instance included in the at least one lane line instance corresponding to the current image frame. Alternatively, the first generation module includes a first determination unit and a second generation unit, which is configured to determine whether the incremental point cloud meets the preset new instance conditions in response to the matching result indicating that the point cloud cluster where the incremental point cloud is located does not have a matching historical lane line instance, and the second generation unit is configured to generate a new lane line instance based on the incremental point cloud in response to the incremental point cloud meeting the new instance conditions. The second generation module is used to generate vectorized lane lines corresponding to the at least one lane line instance based on the point cloud corresponding to the at least one lane line instance.

12. A computer-readable storage medium storing a computer program for execution by a processor to implement the method of any one of claims 1-10.

13. An electronic device, the electronic device comprising: processor; Memory for storing the executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-10.

Images