Auxiliary label map generation method and device, equipment and storage medium

Abstract

The present application relates to the technical field of computer, disclose a kind of generation method, device, equipment and storage medium of auxiliary annotation graph, for by the key point of camera image, the image corresponding to point cloud data is supplemented with key point, improve the accuracy of the target auxiliary annotation graph generated.Auxiliary annotation graph generation method includes: obtaining point cloud lane key point set, camera lane key point set, camera traffic light position key point set and marked two-dimensional image;By point cloud lane key point set, the camera lane key point set is filtered, to obtain the camera lane key point set after screening;By the camera lane key point set after screening and traffic light position key point set, mark the key point of the marked two-dimensional image, to obtain target auxiliary annotation graph.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for generating auxiliary annotation diagrams. Background Technology

[0002] Auxiliary annotation maps are an important component of visual road environment perception. In autonomous driving scenarios, auxiliary annotation maps are generally used to sense target objects, enabling autonomous vehicles to drive normally.

[0003] Currently, the methods for generating auxiliary annotation maps generally involve acquiring point cloud data using LiDAR, analyzing the point cloud data to determine the target object to be identified, and then generating the auxiliary annotation map.

[0004] However, the above method only performs data analysis based on point cloud data, resulting in low accuracy of the generated auxiliary annotation map. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for generating auxiliary annotation maps. By supplementing key points of the image corresponding to the point cloud data with key points of the camera image, the accuracy of the generated target auxiliary annotation map is improved.

[0006] The first aspect of the present invention provides a method for generating an auxiliary annotation map, comprising: acquiring a point cloud lane line key point set, a camera lane line key point set, a camera traffic light location key point set, and a marked two-dimensional image; filtering the camera lane line key point set using the point cloud lane line key point set to obtain a filtered camera lane line key point set; and marking key points on the marked two-dimensional image using the filtered camera lane line key point set and the traffic light location key point set to obtain a target auxiliary annotation map.

[0007] In one feasible implementation, acquiring the point cloud lane line key point set, the camera lane line key point set, the camera traffic light location key point set, and the marked two-dimensional image includes:

[0008] Point cloud data is acquired and transformed to obtain a two-dimensional image. Key points are extracted from the point cloud data and the two-dimensional image to obtain a point cloud road edge key point set and a point cloud lane line key point set. Key points are marked on the two-dimensional image using the point cloud lane line key point set and the point cloud road edge key point set to obtain a marked two-dimensional image. Camera images of the region corresponding to the point cloud data are acquired, and key points are extracted from the camera images to obtain a camera lane line key point set and a camera traffic light location key point set.

[0009] In one feasible implementation, the step of extracting key points from the point cloud data and the two-dimensional image to obtain a set of key points for lane lines and a set of key points for road edges in the point cloud includes: performing road edge identification on the point cloud data using a preset density clustering algorithm to obtain point cloud road edge data; extracting key points from the point cloud road edge data to obtain a set of key points for road edges in the point cloud; and extracting key points from the two-dimensional image using a preset lane detection algorithm to obtain a set of key points for lane lines in the point cloud.

[0010] In one feasible implementation, the step of identifying road edges in the point cloud data using a preset density clustering algorithm to obtain point cloud road edge data includes: classifying the point cloud data into clusters with different densities using a preset density clustering algorithm to obtain at least one cluster; obtaining the bounding box size and point cloud entropy corresponding to each cluster, and matching the bounding box size and point cloud entropy corresponding to each cluster using a preset verification library to obtain identification data corresponding to each cluster; and extracting the target cluster from the at least one cluster using the identification data corresponding to each cluster to obtain point cloud road edge data.

[0011] In one feasible implementation, the step of extracting key points from the two-dimensional image using a preset lane detection algorithm to obtain a point cloud lane line key point set includes: converting the two-dimensional image into a grayscale image using a preset grayscale conversion function; identifying the grayscale image using a template matching algorithm, a position detection algorithm, and an edge detection algorithm in the preset lane detection algorithm to obtain a candidate lane line feature region set, wherein the lane detection algorithm includes a template matching algorithm, a position detection algorithm, an edge detection algorithm, and a non-maximum suppression algorithm; filtering the candidate lane line feature region set using the non-maximum suppression algorithm to obtain a target lane line feature region set; and extracting key points from the target lane line feature region set to obtain a point cloud lane line key point set.

[0012] In one feasible implementation, the step of acquiring camera images of the region corresponding to the point cloud data and extracting key points from the camera images to obtain a set of camera lane line key points and a set of camera traffic light location key points includes: acquiring camera images of the region corresponding to the point cloud data; performing lane line recognition on the camera images using a preset lane line detection model to obtain a set of camera lane lines; extracting key points from the set of camera lane lines to obtain a set of camera lane line key points; and extracting key points from the camera images using a preset deep learning algorithm to obtain a set of camera traffic light location key points.

[0013] In one feasible implementation, the step of identifying the camera image using a preset lane detection model to obtain a camera lane line set includes: performing lane line detection on the camera image using the preset lane line detection model to obtain lane line semantic segmentation results and pixel feature information; clustering the lane line pixels in the camera image based on the lane line semantic segmentation results and the pixel feature information to obtain clustering results; and obtaining the camera lane line set through the clustering results.

[0014] In one feasible implementation, the step of extracting key points from the camera image using a preset deep learning algorithm to obtain a set of key points for the camera traffic light location includes: recognizing the camera image using a preset deep learning algorithm to obtain at least one traffic light detection box; filtering the at least one traffic light detection box to obtain a set of target detection boxes; obtaining the center point of each target detection box in the set of target detection boxes, and determining the center point of each target detection box as a key point for the traffic light location to obtain a set of key points for the traffic light location.

[0015] In one feasible implementation, the step of filtering the camera lane line key point set using the point cloud lane line key point set to obtain filtered camera lane line key points includes: comparing the camera lane line key point set and the point cloud lane line key point set to determine whether there are overlapping key points, wherein the overlapping key points are used to indicate camera lane line key points in the camera lane line key point set that overlap with point cloud lane line key points in the point cloud lane line key point set; if there are overlapping key points, then the overlapping key points in the camera lane line key point set are deleted to obtain the filtered camera lane line key point set.

[0016] A second aspect of the present invention provides an apparatus for generating an auxiliary annotation map, comprising: an acquisition module for acquiring a point cloud set of lane line key points, a camera set of lane line key points, a camera set of traffic light location key points, and a marked two-dimensional image; a filtering module for filtering the camera set of lane line key points using the point cloud set of lane line key points to obtain a filtered set of camera set of lane line key points; and a marking module for marking key points on the marked two-dimensional image using the filtered set of camera set of lane line key points and the set of traffic light location key points to obtain a target auxiliary annotation map.

[0017] In one feasible implementation, the acquisition module includes: an acquisition unit for acquiring point cloud data and converting the point cloud data to obtain a two-dimensional image; an extraction unit for extracting key points from the point cloud data and the two-dimensional image respectively to obtain a point cloud road edge key point set and a point cloud lane line key point set; a marking unit for marking key points on the two-dimensional image using the point cloud lane line key point set and the point cloud road edge key point set to obtain a marked two-dimensional image; and a processing unit for acquiring camera images of the area corresponding to the point cloud data and extracting key points from the camera images to obtain a camera lane line key point set and a camera traffic light location key point set.

[0018] In one feasible implementation, the extraction unit includes: a first identification subunit, used to identify road edges in the point cloud data using a preset density clustering algorithm to obtain point cloud road edge data; a first extraction subunit, used to extract key points from the point cloud road edge data to obtain a set of point cloud road edge key points; and a second extraction subunit, used to extract key points from the two-dimensional image using a preset lane detection algorithm to obtain a set of point cloud lane line key points.

[0019] In one feasible implementation, the first identification subunit is specifically used to: classify the point cloud data into clusters with different densities using a preset density clustering algorithm to obtain at least one cluster; obtain the bounding box size and point cloud entropy corresponding to each cluster, and match the bounding box size and point cloud entropy corresponding to each cluster using a preset verification library to obtain identification data corresponding to each cluster; and extract the target cluster from the at least one cluster using the identification data corresponding to each cluster to obtain point cloud curb data.

[0020] In one feasible implementation, the second extraction subunit is specifically used for: converting the two-dimensional image into a grayscale image using a preset grayscale conversion function; identifying the grayscale image using a template matching algorithm, a position detection algorithm, and an edge detection algorithm in a preset lane detection algorithm to obtain a set of candidate lane line feature regions, wherein the lane detection algorithm includes a template matching algorithm, a position detection algorithm, an edge detection algorithm, and a non-maximum suppression algorithm; filtering the set of candidate lane line feature regions using the non-maximum suppression algorithm to obtain a set of target lane line feature regions; and extracting key points from the set of target lane line feature regions to obtain a set of point cloud lane line key points.

[0021] In one feasible implementation, the processing unit includes: an acquisition subunit for acquiring camera images of the region corresponding to the point cloud data; a second recognition subunit for performing lane line recognition on the camera images using a preset lane line detection model to obtain a camera lane line set; a third extraction subunit for extracting key points from the camera lane line set to obtain a camera lane line key point set; and a fourth extraction subunit for extracting key points from the camera images using a preset deep learning algorithm to obtain a camera traffic light location key point set.

[0022] In one feasible implementation, the second identification subunit is specifically used to: perform lane line detection on the camera image using a preset lane line detection model to obtain lane line semantic segmentation results and pixel feature information; cluster the lane line pixels in the camera image according to the lane line semantic segmentation results and the pixel feature information to obtain clustering results; and obtain a camera lane line set through the clustering results.

[0023] In one feasible implementation, the fourth extraction subunit is specifically used to: identify the camera image using a preset deep learning algorithm to obtain at least one traffic light detection box; filter the at least one traffic light detection box to obtain a set of target detection boxes; obtain the center point of each target detection box in the set of target detection boxes, and determine the center point of each target detection box as a key point of the traffic light location to obtain a set of key points of the traffic light location.

[0024] In one feasible implementation, the filtering module is specifically used to: compare the set of camera lane line key points with the set of point cloud lane line key points to determine whether there are overlapping key points, wherein the overlapping key points are used to indicate camera lane line key points in the set of camera lane line key points that overlap with point cloud lane line key points in the set of point cloud lane line key points; if there are overlapping key points, the overlapping key points in the set of camera lane line key points are deleted to obtain the filtered set of camera lane line key points.

[0025] A third aspect of the present invention provides an auxiliary annotation map generation device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the auxiliary annotation map generation device to perform the above-described auxiliary annotation map generation method.

[0026] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described method for generating auxiliary annotation maps.

[0027] The technical solution provided by this invention involves acquiring a point cloud set of lane line key points, a camera set of lane line key points, a camera set of traffic light location key points, and a marked 2D image. The point cloud set of lane line key points is used to filter the camera set of lane line key points, resulting in a filtered set of camera lane line key points. The filtered set of camera lane line key points and the traffic light location key points are then used to mark key points on the marked 2D image, resulting in a target auxiliary annotation map. In this embodiment, filtering the camera set of lane line key points based on the point cloud set of lane line key points, and then using the filtered set of camera lane line key points and traffic light location key points to mark key points on the marked 2D image, improves the accuracy of the target auxiliary annotation map. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of an embodiment of the method for generating auxiliary annotation maps in this invention;

[0029] Figure 2 This is a schematic diagram of another embodiment of the method for generating auxiliary annotation maps in this invention;

[0030] Figure 3 This is a schematic diagram of one embodiment of the auxiliary annotation diagram generation device in this invention;

[0031] Figure 4 This is a schematic diagram of another embodiment of the auxiliary annotation diagram generation device in this invention;

[0032] Figure 5 This is a schematic diagram of one embodiment of the auxiliary annotation map generation device in this invention. Detailed Implementation

[0033] This invention provides a method, apparatus, device, and storage medium for generating auxiliary annotation maps. By supplementing key points in the image corresponding to point cloud data using key points from camera images, the accuracy of the generated target auxiliary annotation map is improved.

[0034] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] It is understood that the executing entity of this invention can be an auxiliary annotation map generation device, a terminal, or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as the executing entity as an example.

[0036] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the method for generating auxiliary annotation maps in this invention includes:

[0037] 101. Obtain the point cloud lane line key point set, the camera lane line key point set, the camera traffic light position key point set, and the marked 2D image;

[0038] The process involves acquiring initial point cloud data, which includes multiple data points. Multiple reference points are then determined from these data points, and a reference ground is established based on these reference points. Ground points are then extracted from the data points based on this reference ground to obtain the target point cloud data. The specific steps for extracting ground points from the data points based on the reference ground are as follows: The vertical distance between each data point and the reference ground is calculated, resulting in a set of vertical distances. It is then determined whether there exists a target vertical distance less than or equal to a preset distance within this set. If such a distance exists, the data point corresponding to that target vertical distance is identified as a ground point, thus obtaining the target point cloud data. By determining the reference ground to filter the point cloud data, target point cloud data belonging to ground points is obtained, reducing interference from non-ground points, thereby reducing computational load and improving the processing efficiency of the target point cloud data.

[0039] The target point cloud data is clustered to obtain multiple clusters. Feature information is extracted from each cluster. Using this feature information, the target cluster is extracted from the multiple clusters to obtain point cloud road edge data. Feature information includes height and shape. Key points are then extracted from the point cloud road edge data to obtain a set of point cloud road edge key points. By using clustering and feature information extraction, the efficiency of point cloud road edge data extraction is improved.

[0040] The target point cloud data is converted into a two-dimensional image. The two-dimensional image is segmented to obtain multiple regions. The multiple regions are processed by a pre-set lane line convolutional neural network to obtain a set of lane line regions. The set of lane line regions is fitted to obtain a set of lane lines. The feature points of each lane line in the set of lane lines are identified to obtain a set of key points of the lane lines. The feature points include vertices, midpoints, inflection points and corners.

[0041] Obtain camera images of the region corresponding to the initial point cloud data, and obtain the set of camera lane line key points in a manner similar to that used in processing two-dimensional images.

[0042] The key points of the road edge and the key points of the lane line in the point cloud are marked in the two-dimensional image to obtain the marked two-dimensional image.

[0043] The camera image is subjected to bounding box detection to obtain an initial set of traffic light regions. The initial set of traffic light regions is then filtered by color detection to obtain a set of target traffic light regions. The center point of each target traffic light region in the target set is obtained, and the center point of each target traffic light region is determined as the key point of the traffic light location, thus obtaining a set of key points of the traffic light location.

[0044] In bounding box detection, the camera image is segmented into multiple instance regions. These regions are then compared with a preset traffic light bounding box image to obtain multiple similarity scores. It is determined whether a target similarity score greater than or equal to the preset score exists. If such a score exists, the instance region corresponding to the target similarity score is extracted, resulting in an initial traffic light region set. In color detection, color parameters (brightness and chroma) are extracted for each initial traffic light region. These parameters are compared with preset color parameters to obtain an initial traffic light region color set. It is then determined whether a target color (red, green, and yellow) exists within this color set. If a target color exists, the initial traffic light region corresponding to the target color is extracted, resulting in a target traffic light region set. By using bounding box detection and color detection to determine the target traffic light regions, the accuracy of target traffic light region detection is improved, thereby enhancing the accuracy of obtaining key points for the camera's traffic light location.

[0045] 102. By filtering the set of key points of the lane lines in the point cloud, the set of key points of the camera lane lines is obtained.

[0046] The process involves obtaining the 2D coordinates of each lane line keypoint in the point cloud keypoint set, resulting in multiple 2D coordinates of point cloud lane line keypoints. It also involves obtaining the 2D coordinates of each camera lane line keypoint in the camera lane line keypoint set, resulting in multiple 2D coordinates of camera lane line keypoints. The 2D coordinates of the multiple point cloud lane line keypoints are compared with those of the multiple camera lane line keypoints to determine if there exists a target camera lane line keypoint with different 2D coordinates. If such a target camera lane line keypoint exists, the corresponding camera lane line keypoint is extracted, resulting in a filtered set of camera lane line keypoints. If no target camera lane line keypoint exists, the marked 2D image is labeled using the traffic light location keypoint set, resulting in a labeled image with supplemented keypoints.

[0047] 103. By using the filtered set of key points for camera lane lines and key points for traffic light locations, key points are marked on the marked 2D image to obtain a target auxiliary annotation map.

[0048] The system obtains the 2D coordinates of each traffic light location key point in the set of traffic light location key points. Based on the corresponding positions of the 2D coordinates of each camera lane line key point in the filtered set of camera lane line key points and the 2D coordinates of each traffic light location key point in the marked 2D image, key points are marked to obtain a target auxiliary annotation map. When a trigger message is received, the location of the trigger point in the trigger message is obtained. The system then queries the key point closest to the trigger point to obtain the target key point and displays its marking content.

[0049] In this embodiment of the invention, the camera lane line key point set is filtered based on the point cloud lane line key point set, and the key points of the marked two-dimensional image are marked with the filtered camera lane line key point set and traffic light position key point set to obtain a target auxiliary annotation map, which improves the accuracy of the target auxiliary annotation map.

[0050] Please see Figure 2 Another embodiment of the method for generating auxiliary annotation maps in this invention includes:

[0051] 201. Acquire point cloud data, transform the point cloud data, and obtain a two-dimensional image;

[0052] The point cloud data is acquired, which includes multiple 3D coordinate points. The coordinates of the multiple 3D coordinate points are transformed using a preset world coordinate to camera coordinate transformation matrix to obtain multiple camera coordinate points. The multiple camera coordinate points are then converted into pixel coordinate points to obtain multiple pixel coordinate points. The image is then drawn using the multiple pixel coordinate points to obtain a 2D image.

[0053] 202. Key points are extracted from point cloud data and two-dimensional images respectively to obtain a set of key points for road edges in point cloud and a set of key points for lane lines in point cloud;

[0054] The point cloud data is used to identify road edges by a pre-set density clustering algorithm to obtain point cloud road edge data; key points are extracted from the point cloud road edge data to obtain a set of point cloud road edge key points; and key points are extracted from the two-dimensional image by a pre-set lane detection algorithm to obtain a set of point cloud lane line key points.

[0055] The specific steps for obtaining point cloud road edge data through a pre-defined density clustering algorithm are as follows: The point cloud data is classified into clusters with different densities using the pre-defined density clustering algorithm, resulting in at least one cluster. The bounding box size and point cloud entropy corresponding to each cluster are obtained, and these are matched against a pre-defined verification library to obtain the identification data for each cluster. The target cluster is extracted from the at least one cluster using the identification data, thus obtaining the point cloud road edge data. Determining the target cluster using the bounding box size and point cloud entropy improves the accuracy of the obtained point cloud road edge data.

[0056] Point cloud data comprises multiple data points. The density of each data point is obtained, and the density of each data point is compared with the cluster densities of multiple preset clusters. Each data point is classified into the cluster whose density is closest to its own, resulting in at least one cluster. The bounding box size indicates the length and width of an object, determining its size. Point cloud entropy indicates the distribution of data points; objects with uniformly distributed data points (e.g., pedestrians) have lower point cloud entropy, while objects with randomly distributed data points have higher entropy (e.g., tree branches). Corresponding recognition data is matched using the bounding box size and point cloud entropy. Each bounding box size and point cloud entropy corresponds to one recognition data point, and one recognition data point represents one target object class. For example, the first cluster includes the first bounding box size and first point cloud entropy, the second cluster includes the second bounding box size and second point cloud entropy, the third cluster includes the third bounding box size and third point cloud entropy, and the fourth cluster includes the fourth bounding box size and fourth point cloud entropy. Recognition data includes recognition data A, recognition data B, recognition data C, and recognition data D. In the diagram, identification data A indicates pedestrians, identification data B indicates road edges, identification data C indicates tree branches, and identification data D indicates roadblocks. If the size of the first frame model and the first point cloud entropy correspond to identification data A, the size of the second frame model and the second point cloud entropy correspond to identification data B, the size of the third frame model and the third point cloud entropy correspond to identification data C, and the size of the fourth frame model and the fourth point cloud entropy correspond to identification data D, then the first cluster corresponds to identification data A, the second cluster corresponds to identification data B, the third cluster corresponds to identification data C, and the fourth cluster corresponds to identification data D. By using identification data A, identification data B, identification data C, and identification data D, the cluster corresponding to identification data B is extracted from the first cluster, the second cluster, the third cluster, and the fourth cluster to obtain the second cluster. The second cluster indicates the point cloud road edge data.

[0057] The specific steps for extracting key points from a 2D image using a pre-defined lane detection algorithm to obtain a point cloud set of lane line key points are as follows: The 2D image is converted to a grayscale image using a pre-defined grayscale conversion function; the grayscale image is then identified using template matching, position detection, and edge detection algorithms within the pre-defined lane detection algorithm to obtain a set of candidate lane line feature regions. The lane detection algorithm includes template matching, position detection, edge detection, and non-maximum suppression; the candidate lane line feature region set is then filtered using the non-maximum suppression algorithm to obtain a set of target lane line feature regions; and key points are extracted from the target lane line feature region set to obtain a point cloud set of lane line key points.

[0058] The grayscale image is divided into multiple regions. In the template matching algorithm, the similarity of each region with multiple preset templates is compared to obtain the similarity score of each region with the multiple preset templates. The highest similarity score among the similarity scores of each region with the multiple preset templates is extracted to obtain the highest similarity set. If there is a target highest similarity score in the highest similarity set that is greater than a preset first similarity score, the lane line type of the corresponding region is determined by the preset template corresponding to the target highest similarity score. Lane line types include rectangles, triangles, and arrows, etc., and the region corresponding to the target highest similarity score is determined as the first candidate lane line feature region, obtaining the first candidate lane line feature region set. In the position detection method, Hough transform is used to detect multiple regions, and the first target region containing a straight line is extracted from the multiple regions to obtain the second candidate lane line feature region. Region set; In edge detection, the gray values ​​of multiple regions are identified to obtain a set of region gray values. It is determined whether there is a target region gray value in the set of region gray values ​​that is greater than the gray value of a preset region. If there is a target region gray value in the set of region gray values ​​that is greater than the gray value of a preset region, the second target region corresponding to the gray value of the target region is determined as the third candidate lane line feature region, and the third candidate lane line feature region set is obtained. The first candidate lane line feature region set, the second candidate lane line feature region set, and the third candidate lane line feature region set are merged to obtain a candidate lane line feature region set. The confidence of the candidate lane line feature regions in the candidate lane line feature region set is calculated to obtain a confidence set. Based on the confidence set, the candidate lane line feature region set is filtered to obtain the target lane line feature region set.

[0059] The specific steps for filtering the candidate lane line feature region set based on the confidence score set to obtain the target lane line feature region set are as follows: Target candidate lane line feature regions are extracted from the candidate lane line feature region set according to their confidence scores from high to low. After each extraction of a target candidate lane line feature region, a target processing procedure is executed. This procedure includes obtaining candidate lane line feature regions with confidence scores lower than the target candidate lane line feature regions from the candidate lane line feature region set to be processed, thus obtaining a set of candidate lane line feature regions to be processed. Each candidate lane line feature region to be processed in the set of candidate lane line feature regions is then compared with the target candidate lane line feature region. Similarity comparison: If there is a target candidate lane line feature region in the set of candidate lane line feature regions with a similarity greater than a preset second similarity, then the corresponding candidate lane line feature region in the set of candidate lane line feature regions is deleted based on the target candidate lane line feature region. After the target processing program is completed, the next target candidate lane line feature region is extracted and the target processing program is executed, and so on, until the last target candidate lane line feature region has completed the target processing program, resulting in a set of filtered candidate lane line feature regions. The set of filtered candidate lane line feature regions is determined as the set of target lane line feature regions.

[0060] The grayscale information of each pixel in the feature region of the target lane line is obtained to obtain a grayscale information set. The grayscale information includes grayscale gradient value and grayscale gradient direction feature value. If there is target grayscale information in the grayscale information set that meets the preset first condition, the preset first condition is that the grayscale gradient value in the grayscale information is greater than the preset grayscale gradient value and the grayscale gradient direction feature value is greater than the preset grayscale gradient direction value. Then, the pixel corresponding to the target grayscale information is extracted to obtain the first point cloud lane line key point set. The first point cloud lane line key point set includes corner points and intersection points. The position, length, and orientation of each lane line in the target lane line feature region set are obtained through Hough transform. The vertex and center point of each lane line are determined based on its position, length, and orientation, resulting in a second point cloud lane line key point set. Feature information of each pixel in the target lane line feature region is obtained, resulting in a feature information set. The feature information includes the grayscale values ​​of a predetermined first number of pixels on the neighborhood circumference of the pixel. It is determined whether the feature information set contains target feature information that meets a predetermined second condition: the difference between the grayscale values ​​of the predetermined first number of pixels on the neighborhood circumference of the pixel and the grayscale value of the current pixel is greater than a predetermined grayscale difference. If the feature information set contains target feature information that meets the predetermined second condition, the corresponding pixel is extracted, resulting in a third point cloud lane line key point set. The first, second, and third point cloud lane line key point sets are merged to obtain a final point cloud lane line key point set.

[0061] By combining multiple lane line detection methods to detect lane lines, a set of candidate lane line feature regions is obtained, which improves the accuracy of identifying candidate lane line feature regions and the efficiency of extracting candidate lane line feature regions. By deleting redundant candidate lane line feature regions from the set, the duplicate processing of the same candidate lane line feature regions is avoided, reducing the amount of data processing and improving the data processing efficiency. Furthermore, by retaining the candidate lane line feature regions with the highest confidence, the optimal candidate lane line feature regions are obtained. By extracting key points from the target lane line feature region set using multiple key point detection methods, the accuracy of key point identification is improved.

[0062] 203. Mark key points on the two-dimensional image using the key point set of lane lines and the key point set of road edges in the point cloud to obtain the marked two-dimensional image;

[0063] Obtain the two-dimensional coordinates of each lane line key point in the point cloud key point set and the two-dimensional coordinates of each road edge key point in the point cloud key point set. Mark the key points according to the corresponding positions of each lane line key point and road edge key point in the two-dimensional image to obtain the marked two-dimensional image.

[0064] 204. Obtain camera images of the regions corresponding to the point cloud data, extract key points from the camera images, and obtain the set of key points for the camera lane lines and the set of key points for the camera traffic light positions.

[0065] Acquire camera images of the corresponding areas of point cloud data; perform lane line recognition on the camera images using a pre-set lane line detection model to obtain a set of camera lane lines; extract key points from the set of camera lane lines to obtain a set of camera lane line key points; and extract key points from the camera images using a pre-set deep learning algorithm to obtain a set of key points for the camera traffic light locations.

[0066] The specific steps for performing lane line recognition on camera images using a pre-set lane line detection model to obtain a set of camera lane lines are as follows: Lane lines are detected in the camera images using the pre-set lane line detection model to obtain lane line semantic segmentation results and pixel feature information; based on the lane line semantic segmentation results and pixel feature information, lane line pixels in the camera images are clustered to obtain clustering results; and the set of camera lane lines is obtained through the clustering results.

[0067] The feature information of each pixel includes the lateral feature vector of the lane line pixel. Based on the semantic segmentation result of the lane line, the pixel row and pixel column of each lane line pixel in the camera image are determined. Based on the lateral feature vector of the lane line pixel, the lateral feature vector corresponding to the pixel row and pixel column of each lane line pixel is determined. The lane line pixels are clustered according to the lateral feature vector corresponding to the pixel row and pixel column of each pixel to obtain a set of lane line pixel clusters. Each lane line pixel cluster corresponds to one lane line.

[0068] The specific steps for extracting key points from camera images using a pre-set deep learning algorithm to obtain a set of key points for the traffic light location are as follows: The camera image is identified using a pre-set deep learning algorithm to obtain at least one traffic light detection box; the at least one traffic light detection box is filtered to obtain a set of target detection boxes; the center point of each target detection box in the target detection box set is obtained, and the center point of each target detection box is determined as a key point for the traffic light location, thus obtaining a set of key points for the traffic light location.

[0069] The pre-built deep learning algorithms include a traffic light detection model and a non-maximum suppression algorithm. The camera image is input into the traffic light detection model, which extracts bounding boxes from the image, resulting in multiple image bounding boxes. These bounding boxes are then compared with preset traffic light bounding boxes to obtain a similarity set. If a target bounding box with a similarity greater than or equal to a preset third similarity exists in the similarity set, the corresponding image bounding box is extracted, resulting in at least one traffic light bounding box. The non-maximum suppression algorithm is then used to remove redundant traffic light bounding boxes from this set, resulting in a target bounding box set. The center point of each target bounding box in the target bounding box set is obtained, and this center point is designated as a key point for the traffic light location, resulting in a set of key points for the traffic light location. By comparing the similarity of multiple image detection boxes with preset traffic light detection boxes, and then filtering out at least one traffic light detection box from multiple image detection boxes based on the similarity of the target detection boxes, the filtering of multiple image detection boxes can be performed quickly, improving the efficiency of filtering traffic light detection boxes. By deleting redundant traffic light detection boxes, a set of target detection boxes is obtained, reducing the storage space occupied and improving the data processing efficiency.

[0070] For example, assuming a preset third similarity of 80%, multiple image detection boxes, including a first image detection box, a second image detection box, a third image detection box, a fourth image detection box, and a fifth image detection box, are compared. The first image detection box is compared with a preset traffic light detection box to obtain its similarity score; the second image detection box is compared with the preset traffic light detection box to obtain its similarity score; the third image detection box is compared with the preset traffic light detection box to obtain its similarity score; the fourth image detection box is compared with the preset traffic light detection box to obtain its similarity score; and the fifth image detection box is compared with the preset traffic light detection box to obtain its similarity score. If the first detection box has a similarity score of 70% and the second detection box has a similarity score of 85%, then... The similarity of the third detection box is 82%, the similarity of the fourth detection box is 20%, and the similarity of the fifth detection box is 90%. Since the similarity of the second detection box (85%), the third detection box (82%), and the fifth detection box (90%) are all greater than the preset third similarity of 80%, the second image detection box corresponding to the second detection box similarity, the third image detection box corresponding to the third detection box similarity, and the fifth image detection box corresponding to the fifth detection box similarity are determined as traffic light detection boxes, resulting in multiple traffic light detection boxes. If the third image detection box is determined to be a redundant object by the non-maximum suppression algorithm, the third image detection box in the multiple traffic light detection boxes is deleted, resulting in a target detection box set, which includes the second image detection box and the fifth image detection box.

[0071] 205. By filtering the set of key points of the lane lines in the point cloud, the set of key points of the camera lane lines is obtained.

[0072] The camera lane line keypoint set and the point cloud lane line keypoint set are compared to determine whether there are overlapping keypoints. Overlapping keypoints are used to indicate camera lane line keypoints in the camera lane line keypoint set that coincide with point cloud lane line keypoints in the point cloud lane line keypoint set. If overlapping keypoints exist, they are deleted from the camera lane line keypoint set to obtain the filtered camera lane line keypoint set.

[0073] Obtain the 2D coordinates of each camera lane line keypoint in the camera lane line keypoint set, and obtain the 2D coordinate set of each point cloud lane line keypoint in the point cloud lane line keypoint set. Compare the 2D coordinate set of the camera lane line keypoints and the 2D coordinate set of the point cloud lane line keypoints to determine if there are any overlapping keypoint coordinates. If there are overlapping keypoint coordinates, delete the camera lane line keypoints corresponding to the overlapping keypoint coordinates in the camera lane line keypoint set, and obtain the filtered camera lane line keypoint set.

[0074] 206. By using the filtered set of key points for camera lane lines and key points for traffic light locations, key points are marked on the marked 2D image to obtain a target auxiliary annotation map.

[0075] The execution process of step 206 is similar to that of step 103 above, and will not be described again here.

[0076] In this embodiment of the invention, point cloud road edge key point set, point cloud lane line key point set, camera lane line key point set, and camera traffic light location key point set are obtained through point cloud data, two-dimensional images, and camera images. The camera lane line key point set is then filtered to obtain a filtered camera lane line key point set. The filtered camera lane line key point set and traffic light location key point set are used to mark key points on the marked two-dimensional image to obtain a target auxiliary annotation map, which improves the accuracy of the target auxiliary annotation map.

[0077] The above describes the method for generating auxiliary annotation maps in the embodiments of the present invention. The following describes the apparatus for generating auxiliary annotation maps in the embodiments of the present invention. Please refer to [link / reference]. Figure 3 One embodiment of the auxiliary annotation map generation device in this invention includes:

[0078] The acquisition module 301 is used to acquire a set of key points for lane lines in point cloud, a set of key points for lane lines in camera, a set of key points for traffic light positions in camera, and a marked two-dimensional image.

[0079] The filtering module 302 is used to filter the camera lane line key point set through the point cloud lane line key point set to obtain the filtered camera lane line key point set.

[0080] The marking module 303 is used to mark key points on the marked two-dimensional image using the filtered set of key points for camera lane lines and key points for traffic light positions, so as to obtain a target auxiliary annotation map.

[0081] In this embodiment of the invention, the camera lane line key point set is filtered based on the point cloud lane line key point set, and the key points of the marked two-dimensional image are marked with key points using the filtered camera lane line key points and traffic light position key points to obtain a target auxiliary annotation map, thereby improving the accuracy of the target auxiliary annotation map.

[0082] Please see Figure 4 Another embodiment of the auxiliary annotation map generation device in this invention includes:

[0083] The acquisition module 301 is used to acquire a set of key points for lane lines in point cloud, a set of key points for lane lines in camera, a set of key points for traffic light positions in camera, and a marked two-dimensional image.

[0084] The filtering module 302 is used to filter the camera lane line key point set through the point cloud lane line key point set to obtain the filtered camera lane line key point set.

[0085] The marking module 303 is used to mark key points on the marked two-dimensional image using the filtered set of key points for camera lane lines and key points for traffic light positions, so as to obtain a target auxiliary annotation map.

[0086] Optionally, the acquisition module 301 includes:

[0087] The acquisition unit 3011 is used to acquire point cloud data and convert the point cloud data to obtain a two-dimensional image;

[0088] Extraction unit 3012 is used to extract key points from point cloud data and two-dimensional images respectively, to obtain a set of key points for road edge in point cloud and a set of key points for lane line in point cloud.

[0089] The labeling unit 3013 is used to label key points of a two-dimensional image using a set of key points for lane lines in a point cloud and a set of key points for road edges in a point cloud, thereby obtaining a labeled two-dimensional image.

[0090] The processing unit 3014 is used to acquire camera images of the area corresponding to the point cloud data, extract key points from the camera images, and obtain a set of key points for camera lane lines and a set of key points for camera traffic light positions.

[0091] Optionally, the extraction unit 3012 includes:

[0092] The first identification subunit 30121 is used to identify road edges in point cloud data using a preset density clustering algorithm to obtain point cloud road edge data.

[0093] The first extraction subunit is used to extract key points from the point cloud road edge data to obtain a set of 30122 point cloud road edge key points;

[0094] The second extraction subunit 30123 is used to extract key points from a two-dimensional image using a preset lane detection algorithm to obtain a set of key points for the lane line in the point cloud.

[0095] Optionally, the first identification subunit 30121 is specifically used for:

[0096] Point cloud data is classified into clusters with different densities using a pre-defined density clustering algorithm, resulting in at least one cluster. The bounding box size and point cloud entropy corresponding to each cluster are obtained, and the bounding box size and point cloud entropy corresponding to each cluster are matched using a pre-defined verification library to obtain the identification data corresponding to each cluster. The target cluster is extracted from the at least one cluster using the identification data corresponding to each cluster to obtain point cloud road edge data.

[0097] Optionally, the second extraction subunit 30123 is specifically used for:

[0098] The two-dimensional image is converted into a grayscale image using a preset grayscale conversion function. The grayscale image is then identified using a preset lane detection algorithm, which includes template matching, position detection, and edge detection algorithms to obtain a set of candidate lane line feature regions. The lane detection algorithm includes template matching, position detection, edge detection, and non-maximum suppression. The set of candidate lane line feature regions is then filtered using the non-maximum suppression algorithm to obtain a set of target lane line feature regions. Key points are extracted from the set of target lane line feature regions to obtain a set of point cloud lane line key points.

[0099] Optionally, the processing unit 3014 includes:

[0100] Acquisition subunit 30141 is used to acquire camera images of the region corresponding to the point cloud data;

[0101] The second recognition subunit 30142 is used to perform lane line recognition on the camera image through a preset lane line detection model to obtain a set of camera lane lines.

[0102] The third extraction subunit 30143 is used to extract key points from the camera lane line set to obtain the camera lane line key point set.

[0103] The fourth extraction subunit 30144 is used to extract key points from the camera image using a preset deep learning algorithm to obtain a set of key points for the camera traffic light location.

[0104] Optionally, the second identification subunit 30142 is specifically used for:

[0105] Lane lines are detected in camera images using a pre-set lane line detection model, resulting in lane line semantic segmentation and pixel feature information. Based on the lane line semantic segmentation and pixel feature information, lane line pixels in the camera images are clustered to obtain clustering results. The camera lane line set is obtained through the clustering results.

[0106] Optionally, the fourth extraction subunit 30144 is specifically used for:

[0107] The camera image is identified by a pre-set deep learning algorithm to obtain at least one traffic light detection box; the at least one traffic light detection box is filtered to obtain a set of target detection boxes; the center point of each target detection box in the set of target detection boxes is obtained, and the center point of each target detection box is determined as the key point of the traffic light location to obtain a set of key points of the traffic light location.

[0108] Optionally, the filtering module 302 is specifically used for:

[0109] The camera lane line keypoint set and the point cloud lane line keypoint set are compared to determine whether there are overlapping keypoints. Overlapping keypoints are used to indicate camera lane line keypoints in the camera lane line keypoint set that coincide with point cloud lane line keypoints in the point cloud lane line keypoint set. If overlapping keypoints exist, they are deleted from the camera lane line keypoint set to obtain the filtered camera lane line keypoint set.

[0110] In this embodiment of the invention, point cloud road edge key point set, point cloud lane line key point set, camera lane line key point set, and camera traffic light location key point set are obtained through point cloud data, two-dimensional images, and camera images. The camera lane line key point set is then filtered to obtain a filtered camera lane line key point set. The filtered camera lane line key point set and traffic light location key point set are used to mark key points on the marked two-dimensional image to obtain a target auxiliary annotation map, which improves the accuracy of the target auxiliary annotation map.

[0111] above Figure 3 and Figure 4The auxiliary annotation map generation device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The auxiliary annotation map generation device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0112] Figure 5 This is a schematic diagram of an auxiliary annotation map generation device 500 provided in an embodiment of the present invention. The auxiliary annotation map generation device 500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the auxiliary annotation map generation device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the auxiliary annotation map generation device 500.

[0113] The auxiliary annotation drawing generation device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 5 The illustrated auxiliary annotation drawing generation device structure does not constitute a limitation on the auxiliary annotation drawing generation device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0114] The present invention also provides an apparatus for generating auxiliary annotation maps. The computer apparatus includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor performs the steps of the auxiliary annotation map generation method described in the above embodiments. The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the auxiliary annotation map generation method.

[0115] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating auxiliary annotation maps, characterized in that, The method for generating the auxiliary annotation map includes: Acquire point cloud lane line key point set, camera lane line key point set, camera traffic light location key point set, and marked 2D image; The camera lane line key point set is filtered by the point cloud lane line key point set to obtain the filtered camera lane line key point set. The key points of the marked two-dimensional image are marked by the filtered set of key points of camera lane lines and key points of traffic light positions to obtain a target auxiliary annotation map. The acquisition of the point cloud lane line key point set, the camera lane line key point set, the camera traffic light location key point set, and the marked 2D image includes: acquiring point cloud data; converting the point cloud data to obtain a 2D image; classifying the point cloud data into clusters with different densities using a preset density clustering algorithm to obtain at least one cluster; acquiring the bounding box model size and point cloud entropy corresponding to each cluster, and matching the bounding box model size and point cloud entropy corresponding to each cluster using a preset verification library to obtain recognition data corresponding to each cluster; and using the recognition data corresponding to each cluster to obtain the at least one... The target cluster is extracted from the cluster to obtain point cloud road edge data; key points are extracted from the point cloud road edge data to obtain a point cloud road edge key point set; key points are extracted from the two-dimensional image using a preset lane detection algorithm to obtain a point cloud lane line key point set; key points are marked on the two-dimensional image using the point cloud lane line key point set and the point cloud road edge key point set to obtain a marked two-dimensional image; camera images of the region corresponding to the point cloud data are obtained, and key points are extracted from the camera images to obtain a camera lane line key point set and a camera traffic light location key point set.

2. The method for generating auxiliary annotation maps according to claim 1, characterized in that, The step involves extracting key points from the two-dimensional image using a pre-set lane detection algorithm to obtain a point cloud set of lane line key points, including: The two-dimensional image is converted into a grayscale image using a preset grayscale conversion function; The grayscale image is identified by the template matching algorithm, position detection algorithm and edge detection algorithm in the preset lane detection algorithm to obtain a set of candidate lane line feature regions. The lane detection algorithm includes template matching algorithm, position detection algorithm, edge detection algorithm and nonmaximum suppression algorithm. The set of candidate lane line feature regions is filtered by the nonmaximum suppression algorithm to obtain the set of target lane line feature regions. Key points are extracted from the set of feature regions of the target lane line to obtain a set of key points for the lane line in point cloud.

3. The method for generating auxiliary annotation maps according to claim 1, characterized in that, The step involves acquiring camera images of the region corresponding to the point cloud data, extracting key points from the camera images to obtain a set of key points for camera lane lines and a set of key points for camera traffic light locations, including: Obtain camera images of the regions corresponding to the point cloud data; The camera images are used to identify lane lines using a pre-set lane line detection model to obtain a set of camera lane lines. Key points are extracted from the camera lane line set to obtain the camera lane line key point set; The camera image is used to extract key points using a pre-set deep learning algorithm to obtain a set of key points for the camera's traffic light location.

4. The method for generating auxiliary annotation maps according to claim 3, characterized in that, The step of identifying camera lane lines using a pre-set lane line detection model to obtain a camera lane line set includes: The camera image is used to detect lane lines using a pre-set lane line detection model to obtain lane line semantic segmentation results and pixel feature information. Based on the lane line semantic segmentation result and the feature information of the pixel, the lane line pixels in the camera image are clustered to obtain the clustering result; The clustering results yield a set of camera lane lines.

5. The method for generating auxiliary annotation maps according to claim 3, characterized in that, The step involves extracting key points from the camera image using a pre-set deep learning algorithm to obtain a set of key points for the camera's traffic light location, including: The camera image is identified using a pre-set deep learning algorithm to obtain at least one traffic light detection box; The at least one traffic light detection box is filtered to obtain a set of target detection boxes; Obtain the center point of each target detection box in the target detection box set, and determine the center point of each target detection box as the key point of the traffic light location to obtain the set of key points of the traffic light location.

6. The method for generating auxiliary annotation maps according to claim 1, characterized in that, The step of filtering the camera lane line key point set using the point cloud lane line key point set to obtain the filtered camera lane line key points includes: The camera lane line key point set and the point cloud lane line key point set are compared to determine whether there are overlapping key points. The overlapping key points are used to indicate the camera lane line key points in the camera lane line key point set that overlap with the point cloud lane line key points in the point cloud lane line key point set. If any overlapping key points exist, the overlapping key points in the camera lane line key point set are deleted to obtain the filtered camera lane line key point set.

7. An apparatus for generating auxiliary annotation diagrams, characterized in that, The apparatus for generating the auxiliary annotation map includes: The acquisition module is used to acquire a set of key points for lane lines in point cloud, a set of key points for lane lines in camera, a set of key points for traffic light positions in camera, and a marked 2D image. The filtering module is used to filter the camera lane line key point set through the point cloud lane line key point set to obtain the filtered camera lane line key point set. The marking module is used to mark key points on the marked two-dimensional image using the filtered set of key points for camera lane lines and the set of key points for traffic light positions, so as to obtain a target auxiliary annotation map. The acquisition module is specifically used for: acquiring point cloud data; converting the point cloud data to obtain a two-dimensional image; classifying the point cloud data into clusters with different densities using a preset density clustering algorithm to obtain at least one cluster; acquiring the bounding box size and point cloud entropy corresponding to each cluster, and matching the bounding box size and point cloud entropy corresponding to each cluster using a preset verification library to obtain recognition data corresponding to each cluster; extracting the target cluster from the at least one cluster using the recognition data corresponding to each cluster to obtain point cloud road edge data; extracting key points from the point cloud road edge data to obtain a point cloud road edge key point set; extracting key points from the two-dimensional image using a preset lane detection algorithm to obtain a point cloud lane line key point set; marking key points in the two-dimensional image using the point cloud lane line key point set and the point cloud road edge key point set to obtain a marked two-dimensional image; acquiring camera images of the region corresponding to the point cloud data, extracting key points from the camera images to obtain a camera lane line key point set and a camera traffic light position key point set.

8. A device for generating auxiliary annotation maps, characterized in that, The device for generating the auxiliary annotation map includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the auxiliary annotation map generation device to perform the auxiliary annotation map generation method as described in any one of claims 1-6.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the method for generating auxiliary annotation maps as described in any one of claims 1-6.

Images