Lane line clustering method and apparatus, medium, and electronic device

Abstract

Disclosed are a lane line clustering method and apparatus, a medium, and an electronic device. The method comprises: determining a plurality of foreground pixels in a target image, and directional information and feature information corresponding to each foreground pixel; sorting the plurality of foreground pixels according to the feature information of each foreground pixel; clustering the sorted plurality of foreground pixels on the basis of the directional information and feature information of each foreground pixel to obtain a plurality of clustering instances; and merging the plurality of clustering instances on the basis of the feature information of each foreground pixel to obtain a clustered lane line instance. The technical solution provided by the present disclosure fuses the feature information and the directional information of each foreground pixel to perform clustering, thereby helping to increase lane line clustering speed and improving the accuracy of lane line clustering.

Description

Lane line clustering method and device, medium and electronic equipment

[0001] The present disclosure claims priority to the Chinese patent application No. CN202410868114.1, filed on June 28, 2024, and entitled "Lane line clustering method and device, medium and electronic equipment", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the technical fields of computer vision and assisted driving, and in particular, to a lane line clustering method, device, medium and electronic equipment. BACKGROUND

[0003] As an important traffic sign in vehicle driving, lane lines play an important role in driving scenarios such as assisted driving and autonomous driving, such as lane keeping, lane departure warning, lane changing decision, etc. Accurate and rapid identification of lane lines helps to determine the road situation, thereby improving the safety of vehicle driving. SUMMARY

[0004] The present disclosure provides a lane line clustering method, device, medium and electronic equipment.

[0005] According to an aspect of an embodiment of the present disclosure, a lane line clustering method is provided, comprising: determining a plurality of foreground pixel points in a target image, direction information and feature information corresponding to each foreground pixel point; sorting the plurality of foreground pixel points according to the feature information of each foreground pixel point; clustering the sorted plurality of foreground pixel points based on the direction information and the feature information of each foreground pixel point to obtain a plurality of clustering instances; and merging the plurality of clustering instances based on the feature information of each foreground pixel point to obtain a clustered lane line instance.

[0006] According to another aspect of an embodiment of the present disclosure, a lane line clustering device is provided, comprising: an acquisition module configured to determine a plurality of foreground pixel points in a target image, direction information and feature information corresponding to each foreground pixel point; a sorting module configured to sort the plurality of foreground pixel points according to the feature information of each foreground pixel point; a clustering module configured to cluster the sorted plurality of foreground pixel points based on the direction information and the feature information of each foreground pixel point to obtain a plurality of clustering instances; and a lane line generation module configured to merge the plurality of clustering instances based on the feature information of each foreground pixel point to obtain a clustered lane line instance.

[0007] According to another aspect of an embodiment of the present disclosure, a computer readable storage medium is provided, the storage medium storing a computer program for executing the lane line clustering method described in the above embodiments.

[0008] According to still another aspect of the embodiments of the present disclosure, an electronic device is provided, which includes a processor, a memory for storing executable instructions for the processor, and the processor configured to read the executable instructions from the memory and execute the instructions to implement the lane line clustering method according to the above embodiments.

[0009] Based on the above embodiments of the present disclosure, when lane line clustering is needed, a plurality of foreground pixel points in a target image, direction information and feature information corresponding to each foreground pixel point can be determined first, then the plurality of foreground pixel points are sorted, and the plurality of foreground pixel points after sorting are clustered based on the direction information and feature information corresponding to each foreground pixel point, to obtain a plurality of clustering instances, and then the plurality of clustering instances are merged based on the feature information of each foreground pixel point, to obtain the clustered lane line instances. As can be seen, the technical solution provided by the present disclosure can utilize the linear characteristics of the lane line, and fuse the feature information and direction information of each foreground pixel point for clustering. The clustering based on the direction information of the foreground pixel point when clustering is searched, is conducive to improving the lane line clustering speed, and improving the accuracy and effectiveness of lane line clustering. BRIEF DESCRIPTION OF DRAWINGS

[0010] FIG. 1 is a schematic diagram of a system architecture to which the present disclosure is applicable;

[0011] FIG. 2 is a schematic block diagram of lane line clustering according to an exemplary embodiment of the present disclosure;

[0012] FIG. 3 is a schematic flow diagram of a lane line clustering method according to an exemplary embodiment of the present disclosure;

[0013] FIG. 4 is a schematic flow diagram of clustering foreground pixel points according to an exemplary embodiment of the present disclosure;

[0014] FIG. 5 is a schematic flow diagram of determining target pixel points according to an exemplary embodiment of the present disclosure;

[0015] FIG. 6 is a schematic flow diagram of determining target pixel points according to another exemplary embodiment of the present disclosure;

[0016] FIG. 7 is a schematic flow diagram of determining target pixel points according to another exemplary embodiment of the present disclosure;

[0017] FIG. 8 is a schematic flow diagram of determining lane line instances according to clustering instances according to an exemplary embodiment of the present disclosure;

[0018] FIG. 9 is a schematic flow diagram of a lane line clustering method according to another exemplary embodiment of the present disclosure;

[0019] FIG. 10 is a schematic diagram of a second search space according to an exemplary embodiment of the present disclosure;

[0020] FIG. 11 is a structural schematic diagram of a lane line clustering device according to an example embodiment of the present disclosure;

[0021] FIG. 12 is a structural schematic diagram of a lane line clustering device according to another example embodiment of the present disclosure;

[0022] FIG. 13 is a structural schematic diagram of a lane line clustering device according to another example embodiment of the present disclosure;

[0023] FIG. 14 is a structural diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] For the purpose of explaining the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, and are not all the embodiments. It should be understood that the present disclosure is not limited by the example embodiments.

[0025] It should be noted that: unless otherwise specified, the relative arrangement, numerical expression and numerical value of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0026] SUMMARY OF THE DISCLOSURE

[0027] In the process of implementing the present disclosure, the inventors found that in the assisted driving scene, it is very important to identify lane lines according to external environment images for vehicle driving control.

[0028] In the related art, lane lines can be identified based on deep learning methods, such as methods based on parametric curves, image detection methods, or image segmentation methods.

[0029] Among them, the image segmentation-based method has a relatively large advantage in position accuracy, but needs complex post-processing to cluster the foreground pixel points obtained by segmentation into different lane line instances, so it often becomes a bottleneck in speed. The existing clustering methods mainly include mean-shift clustering and DBSCAN clustering. These clustering methods mainly achieve clustering through global clustering, so the clustering speed is slow and the special geometric shape information of the lane line is not considered, resulting in low lane line clustering efficiency.

[0030] To improve the lane line clustering speed, the inventors proposed the technical solutions of the present disclosure.

[0031] EXEMPLARY SYSTEM

[0032] FIG. 1 shows an example system architecture 100 of a lane line clustering method and device to which embodiments of the present disclosure can be applied.

[0033] As shown in FIG. 1, the system architecture 100 can include a terminal device 101, a network 102 and a server 103. The network 102 is used to provide a communication link medium between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0034] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as a shooting application, a navigation application, etc.

[0035] The terminal device 101 can be various electronic devices, including but not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet PCs), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc. and fixed terminals such as digital TVs, desktop computers, etc.

[0036] The server 103 can be a server that provides various services, such as a background server that clusters lane lines using data uploaded by the terminal device 101. The background server can perform image segmentation and recognition on the received vehicle external environment image to obtain direction information and feature information of the foreground pixel points, and obtain lane line instances through a lane line clustering method.

[0037] It should be noted that the lane line clustering method provided by the embodiments of the present disclosure can be executed by the server 103 or the terminal device 101, and correspondingly, the lane line clustering apparatus can be arranged in the server 103 or the terminal device 101.

[0038] It should be understood that the number of terminal devices 101, networks 102 and servers 103 in FIG. 1 is only illustrative. According to the needs of implementation, there can be any number of terminal devices 101, networks 102 and servers 103. In the case where external environment images and the like do not need to be obtained remotely, the above system architecture can not include a network, but only include a server or a terminal device.

[0039] Exemplarily, the lane line clustering process of the technical solution of the present disclosure is shown in FIG. 2. The acquired target image can be input into a lane line detection model 21. The lane line detection model 21 can perform lane line detection and binary classification processing (processing for separating foreground pixel points and background pixel points) on the target image in a bird's eye view (BEV) perspective, obtain all foreground pixel points 22 on the lane line, and then perform clustering on each foreground pixel point, including sorting, clustering, and deleting lane line instances that do not meet the conditions, to obtain lane line instances 23. The lane line detection model 21 can be a pre-trained neural network model. In the post-processing clustering, the foreground pixel points obtained through binary classification processing can be clustered to output lane line instances obtained through clustering of each foreground pixel point.

[0040] Exemplary method

[0041] FIG. 3 is a flowchart of a lane line clustering method according to an exemplary embodiment of the present disclosure. The present embodiment can be applied on an electronic device, as shown in FIG. 3, and includes the following steps:

[0042] Step 201: determining a plurality of foreground pixel points in a target image, direction information corresponding to each foreground pixel point, and feature information corresponding to each foreground pixel point.

[0043] The target image can be an external environment image collected by an image collection device on a current vehicle (hereinafter referred to as: the host vehicle) on which the electronic device is located, and the target image includes lane lines. The foreground pixel points are used to indicate foreground points belonging to the lane lines, and all foreground pixel points on the lane lines can be obtained through binary classification processing of the target image. The direction information of the foreground pixel points is used to indicate the local extension direction of the lane line at the position of the foreground pixel point, and the direction information of each foreground pixel point can be obtained according to the distribution of foreground pixel points in the neighborhood of each foreground pixel point. The feature information of the foreground pixel points is used to identify the inherent characteristics of the pixel points. Specifically, the feature information can include feature encoding, confidence, and position information of the foreground pixel points. The feature encoding is used to represent the characteristics of the instance information to which each foreground pixel point belongs, the confidence is used to indicate the credibility of the foreground pixel point belonging to a point on a certain lane line instance, and the position information can be used to indicate the coordinate information of the foreground pixel point in the image.

[0044] In the present embodiment, a pre-trained neural network model can be used to perform semantic segmentation on the target image to obtain a pixel-level segmentation result in the bird's eye view perspective corresponding to the target image, thereby obtaining a plurality of foreground pixel points of the lane line and the direction information and feature information corresponding to each foreground pixel point.

[0045] The initial model of the pre-trained neural network model can be a U-Net model, a SegNet model, or the like. The pre-trained neural network model can be pre-trained based on a large number of image samples, and can output a plurality of foreground pixel points in an image and direction information and feature information corresponding to each foreground pixel point for input image information.

[0046] In some optional embodiments, a large number of image samples can be pre-collected, each image sample can include a plurality of lane lines and carry foreground pixel points contained therein and direction information and a lane line instance to which each foreground pixel point belongs. The initial model is trained by using a large number of image samples, that is, a large number of image samples are input into the initial model, so that the initial model can learn the image samples. When learning each image sample, the distance between the feature encodings of the foreground pixel points of the same lane line can be set to be very small (for example, close to 0) by using a loss function, and the distance between the feature encodings of the foreground pixel points of different lane lines can be set to be very large, so that the feature encodings can represent the features of the instance information to which each foreground pixel point belongs.

[0047] In an optional example, the step 201 can be performed by a processor invoking corresponding instructions stored in a memory, or by an acquisition module running on the processor.

[0048] In step 202, the plurality of foreground pixel points are sorted according to the feature information of each foreground pixel point.

[0049] In this embodiment, the feature information includes confidence or position information. The sorting of the foreground pixel points includes sorting the plurality of foreground pixel points according to the confidence of each foreground pixel point. For example, the sorting is performed in an order from high to low confidence, or in an order from low to high confidence.

[0050] In some other embodiments of the present disclosure, the position of each foreground pixel point in the vehicle coordinate system corresponding to the vehicle can also be determined according to the position information of each foreground pixel point in the image coordinate system, so as to obtain the distance between the position of each foreground pixel point and the vehicle, and the plurality of foreground pixel points are sorted according to the distance, such as in an order from small to large distance.

[0051] In this implementation, by sorting each foreground pixel point according to the confidence or the distance from the vehicle, clustering search based on the sorted pixel points can be implemented, and repeated search or other situations that can cause low search efficiency in disordered search can be avoided.

[0052] In an optional example, the step 202 can be performed by a processor invoking corresponding instructions stored in a memory, or by a sorting module running on the processor.

[0053] In step 203, the sorted plurality of foreground pixel points are clustered based on the direction information and the feature information of each foreground pixel point, to obtain a plurality of clustering instances.

[0054] In this embodiment, when clustering, a search space for searching a target foreground point can be determined according to the direction information, and after the target pixel point is determined according to the feature information of each foreground pixel point in the search space, a search space is further determined according to the direction information of the determined target pixel point, thereby realizing iterative search. By searching the target pixel point in the search space determined according to the direction information, the lane line clustering speed can be greatly improved.

[0055] In an optional example, step 203 can be executed by a processor calling corresponding instructions stored in a memory, or by a clustering module running on the processor.

[0056] In step 204, the plurality of clustering instances are merged based on the feature information of each foreground pixel point, to obtain clustered lane line instances.

[0057] The lane line instance is used to represent a line of a driving lane of a vehicle.

[0058] In an optional example, step 204 can be executed by a processor calling corresponding instructions stored in a memory, or by a lane line generation module running on the processor.

[0059] The lane line clustering method provided in this embodiment determines a plurality of foreground pixel points in a target image, direction information and feature information corresponding to each foreground pixel point, then sorts the plurality of foreground pixel points, and clusters the sorted plurality of foreground pixel points based on the direction information and the feature information corresponding to each foreground pixel point, to obtain a plurality of clustering instances, and further merges the plurality of clustering instances based on the feature information of each foreground pixel point, to obtain clustered lane line instances. Therefore, the technical solution provided in this disclosure can utilize the line characteristics of lane lines, fuse the feature information and the direction information of each foreground pixel point for clustering, which is conducive to improving the lane line clustering speed, and improving the accuracy and effectiveness of lane line clustering.

[0060] As shown in FIG. 4, on the basis of the above-described embodiment shown in FIG. 3, step 203 can include the following steps:

[0061] In step 231, a current seed point for current search is determined from the plurality of foreground pixel points in sequence according to the order of the sorted plurality of foreground pixel points.

[0062] The current seed point is used to indicate a starting point of a clustering instance.

[0063] In this embodiment, one foreground pixel point which is not attributed to any instance can be selected as the current seed point from the plurality of foreground pixel points in sequence. For example, if the plurality of foreground pixel points are sorted from high to low according to the confidence, the foreground pixel point with the highest confidence among the foreground pixel points which are not attributed to any instance can be selected as the current seed point for the current search; or if the plurality of foreground pixel points are sorted from small to large according to the distance from the vehicle, the foreground pixel point with the smallest distance from the vehicle among the foreground pixel points which are not attributed to any instance can be selected as the current seed point for the current search.

[0064] In a specific implementation, whether a foreground pixel point has been attributed to an instance can be determined according to the attribution identifier of the foreground pixel point. If the attribution identifier of a foreground pixel point is an instance, it means that the foreground pixel point has been attributed to an instance, and one instance refers to one lane line instance. If a plurality of foreground pixel points are attributed to the same instance, it means that the plurality of foreground pixel points belong to the same lane line or the same lane.

[0065] To implement the present disclosure, in the process of clustering the foreground pixel points, after determining one foreground pixel point as a target pixel point each time, the attribution identifier of the foreground pixel point can be set as an attributed instance. For example, the initial value of each foreground pixel point is 0, and in the recursive search stage, the attribution identifier of the foreground pixel point can be used to identify which instance the foreground pixel point belongs to.

[0066] In an optional example, the step 231 can be performed by a processor invoking a corresponding instruction stored in a memory, or by a first determination submodule run by the processor.

[0067] In step 232, based on the direction information and the feature information of the current seed point, at least one target pixel point satisfying a first preset condition is determined by recursively searching in the plurality of foreground pixel points, and the attribution identifier of the at least one target pixel point is marked as the instance to which the target pixel point belongs.

[0068] The first preset condition is used to indicate a preset condition for determining the target pixel point from the plurality of foreground pixel points. The first preset condition can be that a distance between a feature code of the foreground pixel point and a feature code of the current seed point is less than a first set threshold, and the foreground pixel point has not been attributed to any instance. The attribution identifier is used to indicate whether each foreground pixel point has been attributed to a certain clustering instance. If a foreground pixel point is a target pixel point satisfying the first preset condition, the foreground pixel point can be attributed to the current clustering instance, and the attribution identifier of the foreground pixel point is set to the current clustering instance. If a foreground pixel point is not attributed to any clustering instance, the attribution identifier of the foreground pixel point is set to an unattributed instance. For example, the attribution identifiers of the plurality of foreground pixel points are 1, and the plurality of foreground pixel points belong to instance 1. For example, the attribution identifiers of the plurality of foreground pixel points are 2, and the plurality of foreground pixel points belong to instance 2.

[0069] It should be noted that, before clustering the foreground pixel points, the attribution identifiers of the foreground pixel points can be initialized as not attributed to any instance. For example, the attribution identifier of each foreground pixel point is set to 0. In the recursive search stage, the attribution identifier of the foreground pixel point is used to identify which instance the foreground pixel point belongs to. If the attribution identifier of the foreground pixel point has been marked as the instance to which the foreground pixel point belongs, the operation of taking the foreground pixel point as a seed point can be skipped, so as to avoid repeated search and further improve the clustering efficiency.

[0070] In this embodiment, the recursive search is performed according to the direction information and the feature information of the current seed point to determine the target pixel point, including: constructing a search region according to the direction information of the current seed point; based on the feature information of each foreground pixel point, recursively searching the plurality of foreground pixel points by using the search region, taking the plurality of foreground pixel points satisfying the first preset condition as the target pixel points, and marking the attribution identifiers of the target pixel points as the instances to which the target pixel points belong. By marking the attribution identifiers, it is helpful to avoid the case that the foreground pixel point is searched again in the next search, so as to reduce the number of searches and further improve the efficiency of the clustering search.

[0071] In an optional example, the step 232 can be performed by a processor calling corresponding instructions stored in a memory, or by a search submodule running on the processor.

[0072] In step 233, a current clustering instance is generated based on the current seed point and the at least one target pixel point.

[0073] In step 233, a current clustering instance is generated based on the current seed point and the at least one target pixel point.

[0074] In this embodiment, through the steps 231 to 233, one recursive search can be completed to generate one clustering instance, and then it can be determined (judged) through the step 234 whether each foreground pixel point in the plurality of foreground pixel points is marked as belonging to the instance, if not, the step 231 is iteratively executed to generate the next clustering instance, and if yes, the step 235 is executed to complete the iteration.

[0075] In an optional example, the step 233 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the generating submodule running on the processor.

[0076] In an optional example, the step 233 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the generating submodule running on the processor.

[0077] In an optional example, the step 234 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the clustering module running on the processor.

[0078] In an optional example, the step 235 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the clustering submodule running on the processor.

[0079] In this embodiment, after multiple iterations of search, if the ownership identifiers of the plurality of foreground pixel points are all marked as belonging to the instance, it means that all foreground pixel points have ownership instances and the clustering is completed.

[0080] In an optional example, the step 235 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the clustering submodule running on the processor.

[0081] In this embodiment, based on the direction information of the seed point, the implementation manner of generating each clustering instance through iterative search includes: sequentially determining a current seed point for current search from the plurality of foreground pixel points in order according to the sorted plurality of foreground pixel points; based on the direction information and the feature information of the current seed point, recursively searching in the plurality of foreground pixel points to determine at least one target pixel point satisfying a first preset condition, and marking the ownership identifier of the at least one target pixel point as the belonging instance; generating a current clustering instance based on the current seed point and the at least one target pixel point; iteratively performing the operation of sequentially determining the current seed point for current search from the plurality of foreground pixel points; and in response to the ownership identifiers of the plurality of foreground pixel points being all marked as the belonging instance, obtaining a plurality of clustering instances. By determining the current seed point for each search according to the sorting of the confidence or the distance from the vehicle, the point with high confidence or close distance from the vehicle can be preferentially selected as the seed point, and the noise point is avoided to be selected as the seed point to start a round of search, which helps to improve the clustering efficiency and clustering quality when clustering based on the seed point. In addition, by marking the ownership identifier of the target pixel as the belonging instance in the recursive search process, the situation that the foreground pixel point is searched again next time is avoided, thereby reducing the number of searches and further improving the efficiency of clustering search.

[0082] As shown in FIG. 5, on the basis of the embodiment shown in FIG. 4, step 232 can include the following steps:

[0083] Step 2321, constructing a first search space, the first search space including the current seed point.

[0084] Among the first search space, there can be more than one foreground pixel point. After completing the search started by the current seed point, the foreground pixel points in the first search space are used to generate a brand new clustering instance.

[0085] In this embodiment, when constructing the first search space, the current seed point can be added to the first search space. At this time, there is only one foreground pixel point in the first search space, i.e. only the current seed point.

[0086] In an optional example, this step 2321 can be executed by the processor calling the corresponding instructions stored in the memory, or by the first construction unit running on the processor.

[0087] Step 2322, determining a starting point for current search from the foreground pixel points in the first search space.

[0088] The first search space can be traversed, and one foreground pixel point in the first search space can be selected as a starting point of the current search. It can be understood that when the initial search is performed according to the first search space, there is only one current seed point in the first search space, and therefore the current seed point can be directly used as the starting point of the current search. In the iterative search, when multiple foreground pixel points have been added in the first search space, one foreground pixel point in the first search space that has not been used as the starting point to perform recursive search can be selected as the starting point of the current search.

[0089] In this embodiment, after one foreground pixel point is used as the starting point and one round of search is completed through subsequent steps 2323 to 2326, the foreground pixel point used as the starting point can be added with a starting mark, which is used to indicate that the foreground pixel point has been used as the starting point to perform one round of search. When the starting point is subsequently determined from the first search space, one foreground pixel point that has not been used as the starting point can be selected as the starting point.

[0090] In an optional example, the step 2322 can be performed by a processor calling corresponding instructions stored in a memory or by a first determination unit running on the processor.

[0091] In the step 2323, a second search space of the current search is determined according to the direction information of the starting point.

[0092] The second search space can be of any shape. In this embodiment, the second search space is a rotated rectangular region, and the direction of the rotated rectangular region is consistent with the direction of the starting point, as shown in FIG. 10. In FIG. 10, a rectangular region is created according to the direction of one starting point, and the direction of the rectangular region is consistent with the direction of the starting point. Whether each foreground pixel point in the rectangular region (second search space) is the target pixel point can be searched. (In the rectangular region shown in FIG. 10, there are only four foreground pixel points, but this is only an example, and the number of foreground pixel points in the second search space is not limited in the disclosure.) For example, if the direction of the starting point is the horizontal direction, the rotated rectangular region is also a horizontal rectangular region. If the direction of the starting point is a 45-degree angle with the horizontal direction, the direction of the rotated rectangular region is also a 45-degree angle with the horizontal direction, and the starting point can be the center point of the rotated rectangular region.

[0093] In the specific implementation, a rotated rectangular region consistent with the direction information of the starting point of the current search can be determined according to the direction information of the starting point, and the rotated rectangular region can be used as the second search space.

[0094] The length and width of the rotated rectangular region can be preset, and thus, when the second search space corresponding to the starting point is to be determined, a second search space can be automatically determined according to the direction information of the starting point and the preset length and width of the rotated region.

[0095] In an optional example, the step 2323 can be performed by a processor invoking corresponding instructions stored in a memory, or by a second determination unit running on the processor.

[0096] In an optional example, the step 2324 can be performed by a processor invoking corresponding instructions stored in a memory, or by a first traversal unit running on the processor.

[0097] In an optional example, the step 2324 can be performed by a processor invoking corresponding instructions stored in a memory, or by a first traversal unit running on the processor.

[0098] In an optional example, the step 2325 can be performed by a processor invoking corresponding instructions stored in a memory, or by a second determination unit running on the processor.

[0099] The first preset condition is used to indicate a preset condition for determining the target pixel point from the plurality of foreground pixel points. For example, the first preset condition can be that the foreground pixel point has not been attributed to any instance and the distance between the foreground pixel point and the starting point is less than a first set threshold.

[0100] In this embodiment, the distance between the foreground pixel point and the starting point can be determined based on the feature information of the foreground pixel point in the second search space. The distance can be the Euclidean distance between the feature code of the foreground pixel point and the feature code of the starting point. The foreground pixel point with a distance less than a first set threshold is taken as the target pixel point. Specifically, when the foreground pixel points in the second search space are traversed, it can be determined whether the attribution identifier of the foreground pixel point is the instance to which it belongs. If yes, the foreground pixel point is skipped. If not, the distance between the feature codes of the foreground pixel point and the starting point can be calculated, such as the Euclidean distance. If the distance between the feature codes of the foreground pixel point and the starting point is less than the first set threshold, the foreground pixel point is determined as the target pixel point. For example, the starting point can be denoted as P and the foreground pixel point can be denoted as Q. The distance between P and Q can be calculated by formula (1): g g ||P d -Q t d t

[0101] ​​​​In formula (1), Dist represents a distance between the pixel point P and the pixel point Q, the subscript g represents a feature code of the pixel point, the subscript d represents direction information of the pixel point, the subscript t represents position information of the pixel point, f represents a distance from the pixel point to a straight line, MAX represents calculating a maximum value, and a is a specified weight coefficient.

[0102] In an optional example, the step 2325 can be executed by a processor calling corresponding instructions stored in a memory or by a third determination unit running on the processor.

[0103] In step 2326, the target pixel point is added to the first search space, and a belonging identifier of the target pixel point is marked as a belonging instance.

[0104] The belonging identifier of the target pixel point is used to indicate whether each foreground pixel point has been attributed to a certain clustering instance. If a foreground pixel point is a target pixel point satisfying the first preset condition, the foreground pixel point can be attributed to a current clustering instance, and the belonging identifier of the foreground pixel point can be set as the belonging instance. If a foreground pixel point is not attributed to a certain clustering instance, the belonging identifier of the foreground pixel point is set as an unattributed instance.

[0105] In the embodiment, the search process performed by the starting point determined in step 2322 can be completed through the above steps 2322 to 2326. Then, whether each foreground pixel point in the first search space is completed recursive search as a starting point of a round of search is determined (judged) through step 2327. If not, step 2322 is iteratively executed. If each foreground pixel point is completed recursive search as a starting point of a round of search, step 2038 is executed.

[0106] In an optional example, the step 2326 can be executed by a processor calling corresponding instructions stored in a memory or by a first adding unit running on the processor.

[0107] In step 2327, whether each foreground pixel point in the first search space is completed recursive search as a starting point of a round of search is determined.

[0108] In step 2328, each target pixel point in the first search space is determined as a target pixel point satisfying the first preset condition.

[0109] In an optional example, the step 2328 can be executed by a processor calling corresponding instructions stored in a memory or by a fourth determination unit running on the processor.

[0110] In the embodiment, after multiple iteration operations, if each foreground pixel point in the first search space is searched as a starting point and no new target pixel point is added to the first search space, it is indicated that the search process of the target pixel point determined by the current seed point in step 2321 is completed, and each target pixel point in the first search space can be determined as a target pixel point satisfying the first preset condition, and a clustering instance can be generated according to each target pixel point in the first search space. The target pixel point determination method provided in the embodiment can construct a first search space when clustering is performed according to the current seed point, the foreground pixel points in the first search space are used to generate a clustering instance; a starting point of the current search can be determined from the foreground pixel points in the first search space, and then a second search space of the current search is determined according to the direction information of the starting point, and each foreground pixel point in the second search space is traversed; each foreground pixel point satisfying the first preset condition is determined as a target pixel point based on the direction information and the feature information of each foreground pixel point in the second search space; then the target pixel point is added to the first search space, and the affiliation identifier of the target pixel point is marked as the belonging instance; the operation of determining the starting point is iterated, and after each foreground pixel point in the first search space is searched as a starting point of a round of search, each target pixel point in the first search space is determined as a target pixel point satisfying the first preset condition. The target pixel point similar to the feature of the current seed point is determined through iterative search, and in the process of iterative search, the direction information of the foreground pixel point is fused to construct a second search space, which realizes clustering in a local search space combined with the line characteristics of the lane line itself, improves the clustering efficiency and improves the clustering quality.

[0111] FIG. 6 is a flowchart of determining a target pixel point according to another example embodiment of the present disclosure. As shown in FIG. 6, step 232 can further include the following steps:

[0112] Step 2321', a third search space is constructed, and the third search space includes the current seed point.

[0113] The third search space only contains the current seed point when it is initially created, and the target pixel point with a distance less than the second set threshold from the current seed point is added to the third search space. After the search started by the current seed point is completed, the foreground pixel points in the third search space are used to generate a new clustering instance.

[0114] The third search space in the embodiment has the same function as the first search space in the embodiment shown in FIG. 4.

[0115] In an optional example, step 2321' can be executed by the processor calling the corresponding instruction stored in memory, or by a second building unit run by the processor.

[0116] Step 2322': Based on the first direction of the current seed point, recursively search among multiple foreground pixels to determine the target pixel that meets the second preset condition, add the target pixel to the third search space, and mark the belonging identifier of the target pixel as its own instance.

[0117] In this embodiment, the direction information of the current seed point includes a first direction and a second direction, and the first direction and the second direction are opposite directions. For example, if the first direction is horizontal to the left, then the second direction is horizontal to the right; if the first direction is forward, then the second direction is backward.

[0118] The orientation information of the current seed point can be determined based on the distribution of foreground pixels in the neighborhood of the current seed point. The neighborhood of the current seed point refers to a region within a defined range around the current seed point, for example, a region within a range of 10 pixels.

[0119] In practice, the neighborhood of the current seed point can be determined first. For example, the area within 15 pixels around the current seed point can be determined as the neighborhood. Then, foreground pixels whose distance from the feature encoding of each foreground pixel is less than the fourth set threshold can be determined as reference foreground pixels in the neighborhood of each foreground pixel. Then, the orientation information of each foreground pixel can be determined based on the positional relationship between each foreground pixel and the reference foreground pixels in the neighborhood. For example, if the current seed point is P0, then each foreground pixel is traversed within the neighborhood of P0 (e.g., within a 15-pixel radius), and the distance (e.g., Euclidean distance) between each foreground pixel and the feature encoding of the current seed point is calculated. Foreground pixels with a distance less than a fourth set threshold are used as reference foreground pixels. Then, the positional relationship between the current seed point and each reference foreground pixel is calculated. Based on this positional relationship, the direction information of the current seed point is obtained. For example, if five reference foreground pixels are obtained by traversing within the neighborhood of P0, and these five reference foreground pixels are located to the right and left of P0 respectively, then the first direction and the second direction of P0 can be determined to be to the right and to the left, respectively.

[0120] In this embodiment, the second preset condition is used to indicate the pre-set conditions for determining the target pixel among multiple foreground pixels. The second preset condition may include one or more conditions, and foreground pixels that simultaneously satisfy each of the conditions in the second preset condition can be determined as target pixels.

[0121] The second preset condition specifically includes the conditions described in the embodiment shown in Figure 7 in conjunction with the specific process of determining the target pixel.

[0122] If a foreground pixel point is a target pixel point satisfying the second preset condition, the foreground pixel point can be attributed to the instance it belongs to, and the attribution identifier of the foreground pixel point is set as the instance it belongs to. If a foreground pixel point is not attributed to a cluster instance, the attribution identifier of the foreground pixel point is set as the unattributed instance.

[0123] The implementation manner of recursively searching for the target pixel point satisfying the second preset condition among the plurality of foreground pixel points can be referred to the embodiment shown in FIG. 7, which will not be described in detail here.

[0124] In response to the target pixel point satisfying the second preset condition not being searched based on the first direction, the operation of recursively searching for the target pixel point satisfying the second preset condition among the plurality of foreground pixel points based on the second direction of the current seed point is performed.

[0125] If the target pixel point satisfying the second preset condition cannot be continuously searched based on the first direction, the target pixel point can be continuously searched from the current seed point as a starting point along the second direction, that is, along the direction opposite to the first direction.

[0126] In this embodiment, the method of searching for the target pixel point based on the second direction is the same as the method of searching for the target pixel point based on the first direction.

[0127] In an optional example, the steps 2322' and 2323' can be executed by the processor invoking the corresponding instructions stored in the memory, or can be executed by the fifth determination unit running on the processor.

[0128] In response to the target pixel point satisfying the second preset condition not being searched based on the second direction, the operation of recursively searching for the target pixel point satisfying the second preset condition among the plurality of foreground pixel points based on the second direction of the current seed point is performed.

[0129] If the target pixel point satisfying the second preset condition cannot be continuously searched based on the second direction, it can be determined that the iterative search from the current seed point as a starting point has been completed, and each target pixel point in the third search space is determined as the target pixel point satisfying the second preset condition. Subsequently, a cluster instance can be generated according to each target pixel point in the third search space.

[0130] In an optional example, the step 2324' can be executed by the processor invoking the corresponding instructions stored in the memory, or can be executed by the sixth determination unit running on the processor.

[0131] The target pixel point determination manner provided by the embodiment comprises: constructing a third search space, the third search space comprising a current seed point; based on a first direction of the current seed point, recursively searching in a plurality of foreground pixel points to determine a target pixel point satisfying a second preset condition, adding the target pixel point to the third search space, and marking a belonging identifier of the target pixel point as a belonging instance; in response to failing to search for the target pixel point satisfying the second preset condition based on the first direction, performing an operation of recursively searching in the plurality of foreground pixel points based on a second direction of the current seed point to determine the target pixel point satisfying the second preset condition; and in response to failing to search for the target pixel point satisfying the second preset condition based on the second direction, determining each target pixel point in the third search space as at least one target pixel point satisfying a first preset condition. The embodiment of the disclosure can determine all foreground pixel points similar to the current seed point by iteratively searching based on the first direction and the second direction of the current seed point, thereby improving the robustness and accuracy of determining the target pixel point, and the bidirectional search combined with the line type characteristics of the lane line can also improve the clustering quality.

[0132] FIG. 7 is a flowchart of determining a target pixel point according to another example embodiment of the disclosure. As shown in FIG. 7, step 2322' comprises the following steps:

[0133] Step 23221', determining a current seed point as a starting point of the current search.

[0134] Step 23222', determining a fourth search space corresponding to the starting point according to direction information of the starting point.

[0135] The fourth search space can be a rectangular region, and the direction of the rectangular region is consistent with the direction of the starting point. For example, if the starting point is horizontally right, the rectangular region is also a horizontal rectangular region, and the starting point is located at the center point position of the left side.

[0136] In a specific implementation, a rectangular region consistent with the direction information of the starting point of the current search can be determined according to the direction information of the starting point, and the rectangular region is taken as the fourth search space.

[0137] The length and width dimensions of the rectangular region can be preset, so that when the fourth search space corresponding to the starting point needs to be determined, a fourth search space can be automatically determined according to the direction information of the starting point, the preset length and width dimensions of the rectangular region.

[0138] Step 23223', traversing each foreground pixel point in the fourth search space.

[0139] In step 23224', each foreground pixel point in the fourth search space satisfying the second preset condition and each foreground pixel point whose attribution identifier is a foreground pixel point not belonging to any instance is regarded as a target pixel point and added to the third search space.

[0140] The second preset condition is used to indicate a condition preset for determining the target pixel point from the plurality of foreground pixel points. The second preset condition can include more than one condition. When each foreground pixel point satisfies each condition in the second preset condition, the foreground pixel point can be determined as the target pixel point.

[0141] In the present disclosure, the first condition in the second preset condition can be "the distance between the foreground pixel point and the starting point is less than the second preset threshold (i.e., the Euclidean distance between the feature code of the foreground pixel point and the feature code of the starting point is less than the second preset threshold)"; and the second condition can be "the distance between the foreground pixel point and the reference line is less than the third preset threshold".

[0142] In order to ensure that each foreground pixel point in each clustering instance is distributed closer to a line, rather than being too scattered, after determining the foreground pixel point whose distance to the starting point is less than the second preset threshold (herein referred to as an optional foreground pixel point) according to the first condition in the second preset condition, each optional foreground pixel point can be filtered based on the reference line to filter out noise points.

[0143] In the present embodiment, the point farthest from the starting point can be selected as the reference pixel point from the optional foreground pixel points in the same direction as the starting point, and the line between the reference pixel point and the starting point is regarded as the reference line.

[0144] Based on the second preset condition, the present embodiment can first determine the distance between each foreground pixel point and the starting point based on the feature information (including the feature code and the position information) of each foreground pixel point in the fourth search space and the feature information (including the feature code and the position information) of the starting point. Then, each optional foreground pixel point whose distance to the starting point is less than the second preset threshold is determined. The point farthest from the starting point is selected as the reference pixel point from the optional foreground pixel points in the same direction as the starting point. The perpendicular distance between each optional foreground pixel point and the reference line is determined. The reference line is the straight line between the starting point and the reference pixel point. The optional foreground pixel point whose perpendicular distance to the reference line is less than the third preset threshold is determined as the target pixel point.

[0145] Exemplarily, it is assumed that 5 optional foreground pixel points are determined in the fourth search space according to the distance between each foreground pixel point and the starting point, that 3 points consistent with the direction of the starting point are determined in the 5 optional foreground pixel points, and that the point Q is farthest from the starting point in the 3 points. Then, the point Q can be determined as the reference pixel point, a straight line is constructed based on the starting point and the point Q as the reference line, and the vertical distance of the 5 optional foreground pixel points from the reference line is calculated. The point with a vertical distance less than the third preset threshold is taken as the target pixel point.

[0146] In other embodiments, if no target pixel point is determined in the fourth search space, the size of the fourth search space can be increased, such as adjusting the length of the rectangular region corresponding to the fourth search space to twice the original length, and further searching for the target pixel point satisfying the second preset condition in the fourth search space.

[0147] In yet other embodiments, if no target pixel point is determined in the fourth search space, the size of the fourth search space can be increased, such as adjusting the length of the rectangular region corresponding to the fourth search space to twice the original length, and further searching for the target pixel point satisfying the second preset condition in the fourth search space, or adjusting the second preset condition. For example, if the target pixel point is determined in the fourth search space generated in step 23222' according to the second preset condition (such as a distance less than 0.5 from the starting point), the target pixel point can be determined in the increased fourth search space according to the adjusted second preset condition (such as a distance less than 0.2 from the starting point). In the present implementation, further searching by increasing the size of the fourth search space helps to avoid missing some target pixel points, and setting the preset condition more strictly can avoid misidentifying some foreground pixel points as target pixel points.

[0148] Step 23225', determining whether there is a target pixel point satisfying the second preset condition in the current search process.

[0149] In some embodiments, the search process performed with the starting point in step 23222' can be completed through the above steps 23222' to 23224', and then it can be determined in step 23225' whether a target pixel point satisfying the second preset condition is searched in this search process. If no target pixel point is searched, the clustering process performed with the current seed point in step 23221' is completed in step 23227'; if a target pixel point is searched in this search process, the operation of determining the starting point of the next round of search is performed in step 23226'.

[0150] Step 23226', selecting the target pixel point farthest from the starting point of the current search from the fourth search space as the starting point of the next round of search.

[0151] In this embodiment, by taking the target pixel point farthest from the starting point of the current search as the starting point of the next round of search, the clustering speed is helped to be accelerated.

[0152] After the starting point is determined, step 23222' of determining a fourth search space corresponding to the starting point and performing recursive search can be performed.

[0153] The direction of the starting point of the next round of search can be the direction of the starting point of the previous round of search (i.e., the starting point of the current search in this embodiment) pointing to the starting point of the next round of search. For example, the starting point of the current search is P0, and the starting point of the next round of search is point q, and the direction of the starting point of the next round of search is P0->q.

[0154] Step 23227' of determining each target pixel point in the third search space as a target pixel point satisfying the second preset condition.

[0155] The above steps 23221' to 23226' can complete recursive search according to the first direction of the current seed point or the second direction of the current seed point. After multiple recursive operations, if no target pixel point satisfying the second preset condition can be searched in the fourth search space determined according to the direction information of the starting point, it is indicated that the search process of determining the target pixel point according to the first direction or the second direction of the current seed point in step 23221' has been completed. After recursive search according to the first direction and the second direction is completed, each target pixel point in the third search space can be determined as a target pixel point satisfying the second preset condition.

[0156] In an optional example, the above steps 23221' to 23227' can be executed by a processor calling corresponding instructions stored in a memory, or can be executed by a fifth determination unit running on the processor.

[0157] The embodiment provides another target pixel point determination manner, determines a current seed point as a starting point of current search; determines a fourth search space corresponding to the starting point according to direction information of the starting point; traverses each foreground pixel point in the fourth search space; regards each foreground pixel point in the fourth search space as a target pixel point, and adds the target pixel point to a third search space, where the foreground pixel point satisfies a second preset condition and the foreground pixel point is a foreground pixel point whose attribution identifier is not attributed to an instance; selects a target pixel point farthest from the starting point of current search from the fourth search space as a starting point of next round of search; recursively performs the operation of determining the fourth search space corresponding to the starting point according to the direction information of the starting point; and in response to the fact that there is no target pixel point satisfying the second preset condition in the fourth search space corresponding to the starting point, determines each target pixel point in the third search space as a target pixel point satisfying the second preset condition. Since the second preset condition contains two conditions that the distance between the foreground pixel point and the starting point is less than a second set threshold and the distance from the foreground pixel point to the reference line is less than a third set threshold when the target pixel point is determined in the determined local search space (the fourth search space), the noise point is avoided to be determined as the target pixel point, and when the starting point of next round of search is determined iteratively, the target pixel point farthest from the starting point of current search is selected as the starting point of next round of search, which helps to speed up the clustering.

[0158] FIG. 8 is a flowchart of determining a lane line instance according to a clustering instance, provided by an example embodiment of the present disclosure. As shown in FIG. 8, step 204 can include the following steps:

[0159] In step 241, the feature vector of the feature center of each clustering instance is determined according to the feature information of each foreground pixel point.

[0160] The feature center is used to indicate the mean value of the feature encoding of each foreground pixel point contained in each clustering instance. For example, if 20 foreground pixel points are contained in the clustering instance 1, the feature center of the clustering instance 1 can be obtained by calculating the mean value of the feature encoding of the 20 foreground pixel points.

[0161] In an optional example, the step 241 can be executed by the processor calling the corresponding instructions stored in the memory, or can be executed by the fourth determination submodule running on the processor.

[0162] In the embodiment, since the feature encoding is a feature vector, the feature center of each clustering instance is also a feature vector.

[0163] In step 242, the distance between the feature centers of any two clustering instances is determined according to the feature vectors of the feature centers of the clustering instances.

[0164] After the feature centers of the clustering instances are determined, the distance between the feature centers of any two clustering instances can be calculated, such as the Euclidean distance.

[0165] In an optional example, the step 242 can be executed by the processor invoking the corresponding instructions stored in the memory, or by the fifth determination submodule run by the processor.

[0166] In step 243, the plurality of clustering instances with a distance less than the fifth set threshold are merged to obtain the clustered lane line instance.

[0167] It can be understood that if the distance between the feature centers of the two clustering instances is small, such as less than the fifth set threshold, the two clustering instances can be merged. By merging the plurality of clustering instances with a distance less than the fifth set threshold, the clustered lane line instance can be obtained.

[0168] In an optional example, the step 243 can be executed by the processor invoking the corresponding instructions stored in the memory, or by the merging submodule run by the processor.

[0169] The target pixel point determination method provided in the embodiment determines the feature vector of the feature center of each clustering instance according to the feature information of each foreground pixel point, then determines the distance between the feature centers of any two clustering instances according to the feature vector of the feature center of each clustering instance, and then merges the plurality of clustering instances with a distance less than the fifth set threshold to obtain the clustered lane line instance. Therefore, after determining each clustering instance, the disclosure further calculates the distance between the clustering instances, which can merge the clustering instances, so that the obtained lane line instance is more complete and accurate.

[0170] FIG. 9 is a flowchart of a lane line clustering method according to another example embodiment of the disclosure. As shown in FIG. 9, the method can include the following steps:

[0171] In step 201, a plurality of foreground pixel points in a target image, direction information and feature information corresponding to each foreground pixel point are determined.

[0172] In step 202, the plurality of foreground pixel points are sorted according to the feature information of each foreground pixel point.

[0173] In step 203, the plurality of sorted foreground pixel points are clustered based on the direction information and feature information of each foreground pixel point to obtain a plurality of clustering instances.

[0174] In step 204, the plurality of clustering instances are merged based on the feature information of each foreground pixel point to obtain a clustered lane line instance.

[0175] In some embodiments, the implementation of steps 201 to 204 can refer to the embodiment shown in FIG. 3.

[0176] Step 205, determining the number of foreground pixel points in the clustered lane line instance.

[0177] In an optional example, the above step 205 can be executed by the processor calling the corresponding instructions stored in the memory, or by the number determination module running on the processor.

[0178] Step 206, deleting the lane line instance whose number is less than the sixth set threshold.

[0179] In an optional example, the above step 206 can be executed by the processor calling the corresponding instructions stored in the memory, or by the deletion module running on the processor.

[0180] Wherein, the sixth set threshold is a pre-set value for measuring the effectiveness of the lane line instance, which can be 3. If the lane line instance contains 3 or fewer foreground pixel points, it can be deleted.

[0181] In other examples, each foreground pixel point in the lane line instance can also be converted to the vehicle coordinate system, the length of the lane line instance is determined, and the lane line instance with a too small length, such as a distance less than 1 meter, is deleted, only to keep the effective lane line instance.

[0182] The lane line clustering method of the embodiment, by determining a plurality of foreground pixel points in the target image, direction information and feature information corresponding to each foreground pixel point, sorting the plurality of foreground pixel points according to the feature information of each foreground pixel point, clustering the sorted plurality of foreground pixel points based on the direction information and feature information of each foreground pixel point, obtaining a plurality of clustering instances, merging the plurality of clustering instances based on the feature information of each foreground pixel point, obtaining the clustered lane line instance, then determining the number of foreground pixel points in the clustered lane line instance, and deleting the lane line instance whose number is less than the sixth set threshold. Therefore, the present disclosure can utilize the linear characteristics of the lane line, fuse the feature information and direction information of each foreground pixel point for clustering, and perform clustering based on the direction information of the foreground pixel point during clustering search, thereby facilitating to improve the lane line clustering speed and improve the accuracy of the lane line. According to the number of foreground pixel points in the lane line instance, the lane line instance containing the foreground pixel points whose number is less than the sixth set threshold is deleted, which is helpful to improve the effectiveness of the lane line instance.

[0183] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed. The foregoing storage medium includes ROM, RAM, magnetic disc or optical disc and various storage medium capable of storing program codes.

[0184] Exemplary device

[0185] FIG. 11 is a structural schematic diagram of a lane line clustering device provided by an exemplary embodiment of the present disclosure. As shown in FIG. 11, the device can include:

[0186] The acquisition module 111 is configured to determine a plurality of foreground pixel points in a target image, direction information and feature information corresponding to each foreground pixel point.

[0187] The sorting module 112 is configured to sort the plurality of foreground pixel points according to the feature information of each foreground pixel point.

[0188] The clustering module 113 is configured to cluster the sorted plurality of foreground pixel points based on the direction information and the feature information of each foreground pixel point, to obtain a plurality of clustering instances.

[0189] The lane line generation module 114 is configured to merge the plurality of clustering instances based on the feature information of each foreground pixel point, to obtain a clustered lane line instance.

[0190] FIG. 12 is a structural schematic diagram of a lane line clustering device provided by another exemplary embodiment of the present disclosure. As shown in FIG. 12, on the basis of the embodiment shown in FIG. 11, in some embodiments, the clustering module 93 includes:

[0191] The first determination sub-module 1131 is configured to determine, in order of the sorted plurality of foreground pixel points, a current seed point for current search from the plurality of foreground pixel points.

[0192] The search sub-module 1132 is configured to recursively search in the plurality of foreground pixel points based on the direction information and the feature information of the current seed point, to determine at least one target pixel point satisfying a first preset condition, and mark the ownership identifier of the at least one target pixel point as a belonging instance.

[0193] The generation sub-module 1133 is configured to generate a current clustering instance based on the current seed point and the at least one target pixel point.

[0194] The iteration sub-module 1134 is configured to iteratively perform the operation of determining, in order, a current seed point for current search from the plurality of foreground pixel points.

[0195] The clustering submodule 1135 is configured to obtain a plurality of clustering instances in response to the belonging identifiers of the plurality of foreground pixel points being all marked as the belonging instance.

[0196] In some embodiments, the first search submodule 1132 includes:

[0197] The first construction unit 11321 is configured to construct a first search space, and the first search space includes a current seed point.

[0198] The first determination unit 11322 is configured to determine a starting point of the current search from the foreground pixel points in the first search space.

[0199] The second determination unit 11323 is configured to determine a second search space of the current search according to direction information of the starting point.

[0200] The first traversal unit 11324 is configured to traverse each foreground pixel point in the second search space.

[0201] The third determination unit 11325 is configured to determine each foreground pixel point satisfying a first preset condition as a target pixel point based on the direction information and the feature information of each foreground pixel point in the second search space.

[0202] The first adding unit 11326 is configured to add the target pixel point to the first search space, and mark a belonging identifier of the target pixel point as the belonging instance.

[0203] The first determination unit 11322 is configured to iteratively perform the operation of determining the starting point of the current search from the foreground pixel points in the first search space.

[0204] The fourth determination unit 11327 is configured to determine each target pixel point in the first search space as a target pixel point satisfying the first preset condition in response to each foreground pixel point in the first search space being recursively searched as the starting point of a round of search.

[0205] In some embodiments, the second determination unit 11323 is specifically configured to determine a rotated rectangular region consistent with the direction information of the starting point of the current search according to the direction information of the starting point, and take the rotated rectangular region as the second search space.

[0206] In some embodiments, the third determination unit 11325 is specifically configured to determine a distance between each foreground pixel point and the starting point based on the feature information of each foreground pixel point in the second search space, and take each foreground pixel point with a distance less than a first set threshold as a target pixel point.

[0207] In some embodiments, the direction information includes a first direction and a second direction, and the first direction and the second direction are two opposite directions.

[0208] The first search submodule 1132 comprises:

[0209] The second construction unit 11328 is configured to construct a third search space, and the third search space comprises a current seed point.

[0210] The fifth determination unit 11329 is configured to search recursively among the plurality of foreground pixel points based on a first direction of the current seed point, determine a target pixel point satisfying a second preset condition, add the target pixel point to the third search space, and mark a belonging identifier of the target pixel point as a belonging instance.

[0211] The fifth determination unit 11329 is further configured to, in response to failing to search for the target pixel point satisfying the second preset condition based on the first direction, perform the operation of searching recursively among the plurality of foreground pixel points based on a second direction of the current seed point to determine the target pixel point satisfying the second preset condition.

[0212] The sixth determination unit 11330 is configured to, in response to failing to search for the target pixel point satisfying the second preset condition based on the second direction, determine each target pixel point in the third search space as at least one target pixel point satisfying a first preset condition.

[0213] In some embodiments, the fifth determination unit 11329 is specifically configured to determine the current seed point as a starting point of the current search, determine a fourth search space corresponding to the starting point according to direction information of the starting point, traverse each foreground pixel point in the fourth search space, add, as target pixel points, each foreground pixel point in the fourth search space satisfying the second preset condition and having a belonging identifier that is not a belonging instance to the third search space, select, as a starting point of the next round of search, a target pixel point farthest from the starting point of the current search from the fourth search space, recursively perform the operation of determining the fourth search space corresponding to the starting point according to the direction information of the starting point, and in response to the fourth search space corresponding to the starting point being empty of the target pixel point satisfying the second preset condition, determine each target pixel point in the third search space as the target pixel point satisfying the second preset condition.

[0214] In some embodiments, the fifth determination unit 11329 is specifically configured to determine distances between each foreground pixel point in the fourth search space and the starting point based on feature information of each foreground pixel point and feature information of the starting point, determine each selectable foreground pixel point having a distance less than a second set threshold, select, as a reference pixel point, a point farthest from the starting point among the selectable foreground pixel points in the same direction as the starting point, determine a perpendicular distance between each selectable foreground pixel point and a reference line, the reference line being a straight line passing through the starting point and the reference pixel point, and determine, as the target pixel point, each selectable foreground pixel point having a perpendicular distance less than a third set threshold.

[0215] FIG. 13 is a structural schematic diagram of a lane line clustering device according to another example embodiment of the present disclosure. As shown in FIG. 13, in some embodiments, the obtaining module 111 comprises:

[0216] a second determining sub-module 1111, configured to determine direction information corresponding to each foreground pixel point based on the segmentation result of the target image; or,

[0217] a third determining sub-module 1112, configured to determine, based on the feature information of each foreground pixel point, reference foreground pixel points in the neighborhood of each foreground pixel point and having a distance less than a fourth preset threshold from each foreground pixel point; and determine the direction information of each foreground pixel point based on the positional relationship between each foreground pixel point and the reference foreground pixel points in the neighborhood.

[0218] In some embodiments, the lane line generation module 114 comprises:

[0219] a fourth determining sub-module 1141, configured to determine a feature vector of a feature center of each clustering instance according to the feature information of each foreground pixel point;

[0220] a fifth determining sub-module 1142, configured to determine a distance between the feature centers of any two clustering instances according to the feature vectors of the feature centers of the clustering instances;

[0221] a merging sub-module 1143, configured to merge a plurality of clustering instances having a distance less than a fifth preset threshold to obtain a clustered lane line instance.

[0222] In some embodiments, the device further comprises:

[0223] a number determining module 115, configured to determine the number of foreground pixel points in the clustered lane line instance;

[0224] a deleting module 116, configured to delete a lane line instance having a number less than a sixth preset threshold.

[0225] In some embodiments, the feature information comprises a confidence, and the sorting module 92 comprises:

[0226] a first sorting sub-module 1121, configured to sort the plurality of foreground pixel points according to the confidence of each foreground pixel point; or,

[0227] a second sorting sub-module 1122, configured to determine the distance between each foreground pixel point and the vehicle according to the feature information of each foreground pixel point, and sort the plurality of foreground pixel points according to the distance.

[0228] The beneficial technical effects of the device example embodiments can refer to the corresponding beneficial technical effects of the above-mentioned example method embodiments, which will not be repeated here.

[0229] Exemplary electronic device

[0230] FIG. 14 is a structural diagram of an electronic device according to an embodiment of the present disclosure. In this embodiment, the electronic device 14 includes at least one processor 11 and a memory 12.

[0231] The processor 11 can be a central processing unit (CPU) or other form of processor that has data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 14 to perform desired functions.

[0232] The memory 12 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory, for example, can include random access memory (RAM), cache, and / or the like. Non-volatile memory, for example, can include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 11 can execute the one or more computer program instructions to implement the lane line clustering method and / or other desired functions of various embodiments of the present disclosure described above.

[0233] In one example, the electronic device can further include an input device 13 and an output device 14, which are interconnected through a bus system and / or other form of connection mechanism (not shown).

[0234] The input device 13 can further include, for example, a keyboard, a mouse, and / or the like.

[0235] The output device 14 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0236] Of course, for simplicity, only some of the components in the electronic device related to the present disclosure are shown in FIG. 14, and components such as buses, input / output interfaces, and / or the like are omitted. In addition, the electronic device can further include any other appropriate components according to specific application cases.

[0237] Exemplary computer program product and computer-readable storage medium

[0238] In addition to the above-described method and device, an embodiment of the present disclosure can further provide a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the lane line clustering of various embodiments of the present disclosure described in the “Exemplary Method” section above.

[0239] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. The embodiments of the present disclosure can be a computer program product, which can include a computer program tangibly embodied in a machine readable storage medium.

[0240] Moreover, an embodiment of the present disclosure can also be a computer readable storage medium having stored thereon a computer program instructing, which can make a processor execute the steps of the lane line clustering of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section when the computer program is run on the processor.

[0241] The computer readable storage medium can be any combination of one or more computer readable medium(s). The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0242] The above describes the basic principles of the present disclosure in combination with specific embodiments, but the advantages, benefits, effects and the like mentioned in the present disclosure are only examples and are not limiting, and it cannot be considered that each embodiment of the present disclosure must have them. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above details do not limit the present disclosure to be necessarily implemented with the above specific details.

[0243] Those skilled in the art can make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A lane line clustering method, comprising: Determine multiple foreground pixels in the target image, as well as the orientation and feature information corresponding to each foreground pixel; The foreground pixels are sorted according to their feature information. Based on the orientation and feature information of each foreground pixel, the sorted foreground pixels are clustered to obtain multiple clustering instances. Based on the feature information of each foreground pixel, the multiple clustered instances are merged to obtain clustered lane line instances.

2. The method according to claim 1, wherein, The process involves clustering the sorted foreground pixels based on their orientation and feature information to obtain multiple clustering instances, including: According to the sorted order of the multiple foreground pixels, the current seed point for the current search is determined sequentially from the multiple foreground pixels; Based on the direction and feature information of the current seed point, a recursive search is performed among the multiple foreground pixels to determine at least one target pixel that meets the first preset condition, and the at least one target pixel is marked as the instance to which it belongs. Generate the current cluster instance based on the current seed point and the at least one target pixel point; Iteratively execute the operation of determining the current seed point from the plurality of foreground pixels; In response to the fact that the attribution identifiers of the multiple foreground pixels have been marked as belonging to the instance, multiple cluster instances are obtained.

3. The method according to claim 2, wherein, The step of recursively searching among the plurality of foreground pixels based on the direction and feature information of the current seed point to determine at least one target pixel that satisfies the first preset condition, and marking the at least one target pixel as its belonging instance, includes: Construct a first search space, which includes the current seed point; Determine the starting point of the current search from the foreground pixels within the first search space; Based on the direction information of the starting point, determine the second search space for the current search; Traverse each foreground pixel in the second search space; Based on the orientation and feature information of each foreground pixel in the second search space, each foreground pixel that meets the first preset condition is determined as the target pixel. Add the target pixel to the first search space and mark the target pixel's ownership identifier as its belonging instance; Iteratively execute the operation of determining the starting point of the current search from the foreground pixels in the first search space; In response to each foreground pixel in the first search space being used as the starting point for a round of search to complete a recursive search, each target pixel in the first search space is determined as a target pixel that satisfies the first preset condition.

4. The method according to claim 3, wherein, Determining the second search space for the current search based on the direction information of the starting point includes: Based on the direction information of the starting point, determine a rotating rectangular region that is consistent with the direction information of the current search starting point; The rotated rectangular region is used as the second search space.

5. The method according to claim 3, wherein, The step of determining the target pixel based on the orientation and feature information of each foreground pixel in the second search space, which satisfies the first preset condition, includes: Based on the feature information of each foreground pixel in the second search space, the distance between each foreground pixel and the starting point is determined. Each foreground pixel whose distance is less than a first set threshold is taken as the target pixel.

6. The method according to claim 2, wherein, The direction information includes a first direction and a second direction, wherein the first direction and the second direction are two opposite directions; The step of recursively searching among the plurality of foreground pixels based on the direction and feature information of the current seed point to determine at least one target pixel that satisfies the first preset condition, and marking the at least one target pixel as its belonging instance, includes: Construct a third search space, which includes the current seed point; Based on the first direction of the current seed point, a recursive search is performed among the multiple foreground pixels to determine the target pixel that meets the second preset condition. The target pixel is then added to the third search space, and the belonging identifier of the target pixel is marked as the instance to which it belongs. In response to the fact that no target pixel satisfying the second preset condition can be found based on the first direction, the operation of recursively searching among the plurality of foreground pixels to determine the target pixel satisfying the second preset condition is performed based on the second direction of the current seed point. In response to the fact that no target pixel satisfying the second preset condition can be found in the second direction, each target pixel in the third search space is determined to be at least one target pixel satisfying the first preset condition.

7. The method according to claim 6, wherein, The step of recursively searching among the plurality of foreground pixels to determine the target pixel that satisfies the second preset condition includes: The current seed point is determined as the starting point of the current search; Based on the direction information of the starting point, determine the fourth search space corresponding to the starting point; Traverse each foreground pixel in the fourth search space; Foreground pixels in the fourth search space that satisfy the second preset condition and whose attribution identifier is an unattributed instance are taken as target pixels and added to the third search space. Select the target pixel that is furthest from the current search starting point from the fourth search space as the starting point for the next round of search; The operation of determining the fourth search space corresponding to the starting point based on the direction information of the starting point is executed recursively. In response to the absence of a target pixel satisfying the second preset condition in the fourth search space corresponding to the starting point, each target pixel in the third search space is determined to be a target pixel satisfying the second preset condition.

8. The method according to claim 6, wherein, The step of selecting foreground pixels in the fourth search space that satisfy the second preset condition and whose attribution identifier is an unattributed instance as the target pixel includes: Based on the feature information of each foreground pixel in the fourth search space and the feature information of the starting point, the distance between each foreground pixel and the starting point is determined. Determine each selectable foreground pixel whose distance is less than a second preset threshold; Among the available foreground pixels in the same direction as the starting point, select the point that is furthest from the starting point as the reference pixel; Determine the vertical distance between each selectable foreground pixel and the reference line, where the reference line is the straight line between the starting point and the reference pixel. The optional foreground pixels whose vertical distance is less than a third set threshold are determined as the target pixels.

9. The method according to claim 1, wherein, Determining the orientation information corresponding to each foreground pixel includes: Based on the segmentation results of the target image, determine the orientation information corresponding to each foreground pixel; or, Based on the feature information of each foreground pixel, a reference foreground pixel whose distance from each foreground pixel is less than a fourth set threshold is determined in the neighborhood of each foreground pixel; based on the positional relationship between each foreground pixel and the reference foreground pixel in the neighborhood, the orientation information of each foreground pixel is determined.

10. The method according to claim 1, wherein, The step of merging the multiple clustered instances to obtain clustered lane line instances includes: Based on the feature information of each foreground pixel, the feature vector of the feature center of each cluster instance is determined respectively; Based on the feature vectors of the feature centers of each cluster instance, determine the distance between the feature centers of any two cluster instances; Multiple clustered instances whose distance is less than the fifth set threshold are merged to obtain clustered lane line instances.

11. The method according to claim 1, wherein, After obtaining the clustered lane line instances, the process further includes: Determine the number of foreground pixels in the clustered lane line instances; Delete lane line instances whose number is less than the sixth set threshold.

12. The method according to claim 1, wherein, The step of sorting the plurality of foreground pixels according to the feature information of each foreground pixel includes: The feature information includes confidence scores, and the multiple foreground pixels are sorted according to the confidence scores of each foreground pixel; or... Based on the feature information of each foreground pixel, the distance between each foreground pixel and the vehicle is determined, and the multiple foreground pixels are sorted according to the distance.

13. A lane line clustering device, comprising: The acquisition module is used to determine multiple foreground pixels in the target image, as well as the orientation and feature information corresponding to each foreground pixel; The sorting module is used to sort the multiple foreground pixels according to the feature information of each foreground pixel; The clustering module is used to cluster the sorted foreground pixels based on the orientation and feature information of each foreground pixel to obtain multiple clustering instances. The lane line generation module is used to merge the multiple clustered instances based on the feature information of each foreground pixel to obtain clustered lane line instances.

14. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-12.

15. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-12.

Images