A curb detection method and device, a terminal device and a storage medium
By combining an improved neural network model with point cloud data, the problem of illumination interference in curb detection was solved, achieving higher detection accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGSHA XINGSHEN INTELLIGENT TECH CO LTD
- Filing Date
- 2022-06-28
- Publication Date
- 2026-07-07
AI Technical Summary
Existing pure image processing solutions are easily affected by factors such as lighting in curb detection, resulting in low detection accuracy.
An improved neural network model combined with point cloud data is used for road edge detection. By modifying the lane line detection model, road image data with marked road edges is used for training, and the anchor point threshold is increased to output a set of candidate road edge points. The point cloud data is used to assist in localization, and non-maximum suppression and line segment matching are performed.
It improves the accuracy of curb detection, reduces the impact of interference such as lighting, and achieves more accurate curb detection.
Smart Images
Figure CN117372975B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of environmental sensing technology, and in particular to a curb detection method, device, terminal equipment, and storage medium. Background Technology
[0002] The curb, or road boundary, defines the passable area for vehicles. Currently, a common curb detection method involves installing cameras around the vehicle to capture multiple road images, detecting road edge lines from these images, transforming these edge lines to the vehicle's coordinate system, and finally performing edge matching based on a parallax-based view. However, this method is a pure image processing solution without the aid of other sensors. If the captured road images are affected by lighting or other interference, the accuracy of curb detection will be low. Summary of the Invention
[0003] In view of this, embodiments of this application provide a curb detection method, apparatus, terminal device, and storage medium, which can improve the accuracy of curb detection.
[0004] A first aspect of this application provides a curb detection method, including:
[0005] Acquire the target image and point cloud data of the road to be tested;
[0006] The target image is input into a trained curb detection model for processing, and a set of candidate curb points is output. The curb detection model is obtained by modifying a neural network model used to realize lane line detection. It is trained using road image data with marked curbs as samples, and the threshold used to distinguish anchor points as positive or negative samples is greater than a specified threshold.
[0007] Based on the candidate curb point set and the point cloud data, the curb detection result of the road to be tested is obtained.
[0008] In this embodiment, considering that lane line detection and curb detection are similar in that both involve detecting line segments, the existing neural network model used for lane line detection, i.e., the lane line detection model, has been partially improved to make it suitable for curb detection. Specifically, the training samples of the lane line detection model can be replaced with road images of marked lane lines and road images of marked curbs. Furthermore, the threshold used by the lane line detection model to distinguish anchor points as positive or negative samples is increased to a certain extent. After this improvement, the lane line detection model becomes a curb detection model. When it is necessary to detect the curb of the road to be tested, the image of the road to be tested is input into the curb detection model for processing, and a set of candidate curb points is output. Then, the final curb detection result is obtained by combining the point cloud data of the road to be tested. This embodiment uses a deep learning-based curb detection model, which outperforms general pure image processing schemes. Moreover, by combining point cloud data for auxiliary localization, the influence of interference such as lighting on curb detection can be effectively reduced, thereby improving the accuracy of curb detection.
[0009] In one implementation of this application, the candidate curb point set includes the coordinate points of at least one candidate curb line segment; the step of detecting the curb detection result of the road to be tested based on the candidate curb point set and the point cloud data may include:
[0010] Using a non-maximum suppression method, the coordinates of the target roadside segment are selected from the coordinates of the at least one candidate roadside segment.
[0011] Connect the coordinate points of the target roadside segment with a line to obtain the target roadside segment;
[0012] The point cloud data is projected onto the target image;
[0013] Based on the point cloud of the area where the target roadside segment is located, which is obtained by projecting the point cloud data in the target image, the roadside marker points of the road to be tested are detected.
[0014] Furthermore, after connecting the coordinate points of the target roadside segment to obtain the target roadside segment, the process may further include:
[0015] The target roadside segment is subjected to contour thickening processing.
[0016] In one implementation of this application, the step of detecting the roadside marker points of the road to be tested based on the point cloud of the area where the target roadside segment is located, obtained by projecting the point cloud data from the target image, may include:
[0017] Detect whether there are target point clouds in the point cloud data whose height difference is greater than a set threshold;
[0018] If the target point cloud exists in the point cloud data, the roadside marker point of the road to be tested is detected based on the target point cloud.
[0019] If the target point cloud is not present in the point cloud data, the roadside marker point of the road to be tested is detected based on the point cloud of the area where the target roadside segment is located in the target image.
[0020] Furthermore, the target image comprises N consecutive frames, and the point cloud data comprises the point cloud corresponding to each of the N frames;
[0021] The step of projecting the point cloud data onto the target image can specifically be as follows:
[0022] For each of the N frames, the point cloud corresponding to that frame is projected onto that frame;
[0023] After obtaining the target roadside line segment of each of the N frames of images, it may further include:
[0024] Perform line segment matching processing on the target roadside line segments of all two adjacent frames contained in the N frames of images;
[0025] The step of detecting the roadside markers of the road to be tested based on the target point cloud may include:
[0026] If the line segment matching results of the target road edge line segments of all two adjacent frames are all matches, then the road edge marker points of each frame of the N frames are detected according to the target point cloud of each frame of the N frames.
[0027] The road edge markers of the road to be tested are determined based on the road edge markers of each of the N frames of images.
[0028] The step of detecting the roadside marker points of the road to be tested based on the point cloud of the area where the target roadside line segment is located in the target image may include:
[0029] If the line segment matching results of the target road edge line segments in all two adjacent frames are all matches, then the road edge marker points of each frame in the N frames are detected based on the point cloud of the region where the target road edge line segments of each frame are located.
[0030] The roadside markers of the road to be tested are determined based on the roadside markers of each of the N frames of images.
[0031] Furthermore, let the first image and the second image be any two adjacent images contained in the N frames of images. The process of performing line segment matching on the target roadside line segments of all adjacent images contained in the N frames of images may include:
[0032] The overlap between the contours of each target roadside line segment contained in the first image and the contours of each target roadside line segment contained in the second image is calculated.
[0033] Each target roadside segment contained in the first image is used as the target of the first part of the bipartite graph, and each target roadside segment contained in the second image is used as the target of the second part of the bipartite graph.
[0034] The KM algorithm is applied to the bipartite graph, using the respective overlap degrees as weights between the targets of the first part and the second part.
[0035] If, after processing with the KM algorithm, each target in the second part is successfully matched with a target in the first part, then the line segment matching result of the target roadside line segment in the first image and the second image is determined to be a match.
[0036] Furthermore, after performing line segment matching processing on the target roadside line segments of all adjacent two-frame images contained in the N-frame images, the process may further include:
[0037] If there are multiple consecutive images in the N frames where the target roadside line segment has a non-matching result, then the roadside detection result will be deleted.
[0038] A second aspect of this application provides a curb detection device, comprising:
[0039] The data acquisition module is used to acquire target images and point cloud data of the road under test;
[0040] The curb detection model processing module is used to input the target image into the trained curb detection model for processing and output a set of candidate curb points; wherein, the curb detection model is obtained by modifying the neural network model used to realize lane line detection, and is trained using road image data with marked curbs as samples, and the threshold used to divide anchor points into positive samples or negative samples is greater than a specified threshold.
[0041] The curb detection module is used to detect the curb detection results of the road to be tested based on the candidate curb point set and the point cloud data.
[0042] A third aspect of this application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the curb detection method provided in the first aspect of this application.
[0043] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the curb detection method provided in the first aspect of this application.
[0044] The fifth aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to perform the curb detection method as described in the first aspect of this application.
[0045] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description
[0046] Figure 1 This is a flowchart of a curb detection method provided in an embodiment of this application;
[0047] Figure 2 This is a schematic diagram of the lane detection results output using the LaneATT model;
[0048] Figure 3 yes Figure 1 A flowchart of a specific implementation of step 103;
[0049] Figure 4 This is a schematic diagram showing the effect of obtaining the curb marker points of the road under test using the curb detection method provided in the embodiments of this application;
[0050] Figure 5 This is a structural frame diagram of a curb detection device provided in an embodiment of this application;
[0051] Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of this application. Detailed Implementation
[0052] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.
[0053] This application proposes a curb detection method applicable to environmental perception during vehicle driving. For example, during vehicle operation, this curb detection method can accurately identify road edges and perceive passable areas in real time. For more specific technical implementation details of this application's embodiments, please refer to the method embodiments described below.
[0054] It should be understood that the execution subject of the various method embodiments of this application can be various types of terminal devices or servers, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of this application do not impose any restrictions on the specific type of terminal device and server.
[0055] Please see Figure 1 This illustration shows a curb detection method provided in an embodiment of the present application, comprising:
[0056] 101. Acquire the target image and point cloud data of the road to be tested;
[0057] First, acquire images (represented as target images) and point cloud data of the road to be tested. The road to be tested can be any type of road where curb detection is required. The point cloud data can be either LiDAR point cloud or image point cloud. In practice, for scenarios where a vehicle is driving on the road to be tested, LiDAR and a camera can be installed on the vehicle body. The camera acquires target images of the road ahead, and the LiDAR acquires point cloud data of the road ahead.
[0058] 102. Input the target image into the trained curb detection model for processing, and output a set of candidate curb points;
[0059] Considering the similarity between lane detection and curb detection tasks—both involve detecting line segments—this application improves existing lane detection models (i.e., various neural network models used for lane detection) to make them suitable for curb detection. Existing technologies have proposed many different lane detection models, such as LaneNet and LaneATT. Taking LaneATT as an example, its basic principle is: a feature extraction network is used to extract features from the input image, generating a feature map; then, the feature map is aggregated to extract features from each anchor point; the anchor point features are combined with a set of global image features generated by an attention mechanism module. By combining local and global features, information from other lanes can be more easily used when lane lines are occluded or there are no visible lane markings; finally, the fused features are passed to a fully connected layer to predict the final output lane. The LaneATT model ultimately outputs a set of discrete coordinate points representing the lane line position. Connecting these discrete coordinate points yields various line segments representing the lane lines, such as... Figure 2 As shown.
[0060] The embodiments of this application mainly improve the existing lane detection model in the following two aspects:
[0061] (1) Replace the road image with the lane line marked with the road line with the road image with the road edge marked with the road line ...
[0062] (2) Increase the threshold used by the lane detection model to classify anchor points as positive or negative samples to a certain extent, making it greater than a specified threshold. This specified threshold can be understood as the corresponding threshold used by the lane detection model when performing lane detection. Most lane lines are straight lines, with curves only frequently appearing at the end of the image. For road edges, both straight lines and curves are common. Therefore, in the selection of anchor points, it is necessary to modify the selection threshold for positive and negative samples so that the extracted positive and negative samples can satisfy the edge of the curve. Traditional lane detection models calculate the distance between anchor points and ground truth during training. When the distance is less than the threshold τ_p, the anchor point is treated as a positive sample; when the distance is greater than the threshold τ_n, the anchor point is treated as a negative sample; otherwise, the sample is ignored. To achieve road edge detection, the two thresholds τ_p and τ_n can be increased accordingly.
[0063] The improved lane detection model becomes a curb detection model. When the target image is input into the curb detection model for processing, it will output a discrete set of curb coordinate points, which is represented by a set of candidate curb points.
[0064] 103. Based on the candidate curb point set and the point cloud data, the curb detection result of the road to be tested is obtained.
[0065] To improve the accuracy of curb detection, this application embodiment combines point cloud data corresponding to the target image to assist in locating the curb position, thereby obtaining the curb detection result of the road to be tested. The curb detection result can be a series of curb marker points, or a curb marker curve obtained by curve fitting of the curb marker points.
[0066] In one implementation of the embodiments of this application, such as Figure 3 As shown, the candidate roadside point set includes the coordinate points of at least one candidate roadside line segment; step 103 may include:
[0067] 1031. Using the non-maximum suppression method, select the coordinate points of the target roadside segment from the coordinate points of the at least one candidate roadside segment;
[0068] 1032. Connect the coordinate points of the target roadside segment with a line to obtain the target roadside segment;
[0069] 1033. Project the point cloud data onto the target image;
[0070] 1034. Based on the point cloud of the area where the target roadside segment is located, obtained by projecting the point cloud data in the target image, the roadside marker points of the road to be tested are detected.
[0071] For step 1031 above, each road edge in the target image, after being detected by the lane line detection model, typically yields one or more discrete coordinate points of candidate road edge segments. For example, assuming the target image has a left and a right road edge, after detection by the lane line detection model, the left road edge will have one or more discrete coordinate points of candidate road edge segments detected, and the right road edge will also have one or more discrete coordinate points of candidate road edge segments detected. For each road edge, non-maximum suppression can be used to select the optimal one from one or more candidate road edge segments, denoted as the target road edge segment. That is, each road edge in the target image will have a corresponding target road edge segment detected. In the initial results of object detection, the same object may correspond to multiple detection boxes, which often overlap. In this case, non-maximum suppression (NMS) can be used to select the optimal detection box that is closest to the true target box from these detection boxes. Specifically, each candidate roadside segment can be enclosed by a detection box, resulting in multiple overlapping detection boxes. This allows for non-maximum suppression to be performed, from which the optimal detection box is selected. The candidate roadside segment enclosed by this optimal detection box is the target roadside segment.
[0072] For step 1032 above, since the lane detection model outputs discrete coordinate points, in order to facilitate line segment matching in subsequent steps, the coordinate points of the target roadside line segment can be connected into a line to generate a complete target roadside line segment. Under normal circumstances, the connected target roadside line segment is relatively thin, which is inconvenient for line segment matching. Therefore, after connecting the lines, the outline of the target roadside line segment can be thickened. The outline thickening of the line segment can also be called the line segment expansion process. The processed target roadside line segment occupies a larger pixel range, which is beneficial for performing line segment matching.
[0073] For steps 1033-1034 above, to assist in localization, point cloud data corresponding to the target image can be projected onto the target image. Then, based on the point cloud of the area where the target roadside segment is located, projected from the point cloud data in the target image, the roadside marker points of the road to be tested can be detected, thereby obtaining the roadside detection result. In one implementation of this application embodiment, the step of detecting the roadside marker points of the road to be tested based on the point cloud of the area where the target roadside segment is located, projected from the point cloud data in the target image, may include:
[0074] (1) Detect whether there are target point clouds in the point cloud data whose height difference is greater than a set threshold;
[0075] (2) If the target point cloud exists in the point cloud data, the roadside marker point of the road to be tested is detected based on the target point cloud;
[0076] (3) If the target point cloud does not exist in the point cloud data, the roadside marker point of the road to be tested is detected based on the point cloud of the area where the target roadside line segment is located in the target image.
[0077] In real-world scenarios, curbs can be categorized into two types: those with a significant height difference between the curb and the road surface, such as sections with curbs; and those without a significant height difference, such as sections where a concrete surface meets a grassy area. Different detection methods can be used for these two types of curbs: For the type with a significant height difference, the point cloud data will contain a portion of points with a height difference exceeding a set threshold, represented as target point clouds. By detecting these target point clouds, curb markers can be obtained. For the type without a significant height difference, the point cloud data will not contain points with a significant height difference. In this case, the point cloud within the region where the target curb segment is projected onto the target image can be found, and this portion of the point cloud can be detected to obtain curb markers. It can be seen that for the type with a significant height difference, the second detection method can also be used, namely, detecting the point cloud within the region where the target curb segment is projected onto the target image to obtain curb markers.
[0078] Furthermore, the target image comprises N consecutive frames, and the point cloud data comprises the point cloud corresponding to each of the N frames; the projection of the point cloud data onto the target image can specifically be:
[0079] For each of the N frames, the point cloud corresponding to that frame is projected onto that frame.
[0080] Assuming the N frames are image 1 to image N, and their corresponding point clouds are point cloud 1 to point cloud N, then point cloud 1 is projected onto image 1, point cloud 2 is projected onto image 2, and so on, with point cloud N projected onto image N. In other words, the point cloud data projected onto the images are from the same time point.
[0081] The above steps will yield the target roadside line segments for each of the N frames. Next, line segment tracking and matching can be performed, that is, line segment matching processing can be applied to the target roadside line segments of all adjacent pairs of images contained in the N frames. For example, target roadside line segment matching is performed between image 1 and image 2, between image 2 and image 3, and so on, between image N-1 and image N.
[0082] In one embodiment of this application, the first image and the second image are any two adjacent frames contained in the N frames. The process of matching the target roadside line segments using the first image and the second image may include:
[0083] (1) Obtain the first starting coordinate point and the first ending coordinate point of the target roadside line segment in the first image, and obtain the second starting coordinate point and the second ending coordinate point of the target roadside line segment in the second image;
[0084] (2) Select the target starting coordinate point with the larger coordinate value in the first direction from the first starting coordinate point and the second starting coordinate point;
[0085] (3) Select the target end coordinate point with the smaller coordinate value in the first direction from the first end coordinate point and the second end coordinate point;
[0086] (4) Within the range defined by the target start coordinate point and the target end coordinate point, calculate the difference between the second direction coordinate value of each coordinate point in the target roadside line segment of the first image and the second direction coordinate value of the corresponding coordinate point in the target roadside line segment of the second image; wherein the second direction and the first direction are perpendicular to each other.
[0087] (5) If the sum of each of the differences is less than a set threshold, then the line segment matching result of the target roadside line segment of the first image and the second image is determined to be a match.
[0088] The first and second directions are two mutually perpendicular directions in the coordinate system. For example, the first direction could be the Y-axis, and the second direction could be the X-axis. Assuming the starting and ending coordinates of the target roadside line segment in the first image are A1 = (x1, y1) and A1 = (x2, y2) respectively, and the starting and ending coordinates of the target roadside line segment in the second image are B1 = (x3, y3) and B2 = (x4, y4) respectively, then the starting coordinate point with the largest Y-coordinate is selected as the target starting coordinate point, and the ending coordinate point with the smallest Y-coordinate is selected as the target ending coordinate point. For example, if y1 is greater than y3 and y2 is greater than y4, then the target starting coordinate point is A1 of the target roadside line segment in the first image, and the target ending coordinate point is B2 of the target roadside line segment in the second image. Within the Y-coordinate range defined by the target's starting and ending coordinate points, both target roadside segments have coordinate points, allowing for line segment matching. Specifically, the difference between the X-coordinate of each coordinate point of the target roadside segment in the first image and the X-coordinate of the corresponding coordinate point (with the same Y-coordinate) of the target roadside segment in the second image can be calculated. If the sum of these differences is less than a set threshold, it indicates that the target roadside segments of the first and second images belong to the same roadside, and the corresponding line segment matching result is a match. Otherwise, it indicates that the target roadside segments of the first and second images do not belong to the same roadside, and the corresponding line segment matching result is a mismatch.
[0089] In another embodiment of this application, the process of matching the target roadside line segment between the first image and the second image may include:
[0090] (1) Iterate through and calculate the overlap between the contour of each target roadside line segment contained in the first image and the contour of each target roadside line segment contained in the second image.
[0091] (2) Each target roadside segment contained in the first image is taken as the target of the first part of the bipartite graph, and each target roadside segment contained in the second image is taken as the target of the second part of the bipartite graph.
[0092] (3) Using each degree of overlap as a weight between the target of the first part and the target of the second part, the KM algorithm is applied to the bipartite graph.
[0093] (4) If each target in the second part is successfully matched with the target in the first part after the KM algorithm is executed, then the line segment matching result of the target roadside line segment in the first image and the second image is determined to be a match.
[0094] Assuming the first image contains three target roadside line segments, denoted as a1, a2, and a3, and the second image contains three target roadside line segments, denoted as b1, b2, and b3, then the overlap between the contours of a1 and b1, a1 and b2, a1 and b3, a2 and b1, and so on, is calculated, resulting in nine overlap values. A bipartite graph is a special type of graph that can be divided into two parts, where the points within each part are not connected. A bipartite graph can be understood as all the object detection boxes in two consecutive frames of a video. Specifically, a1, a2, and a3 are taken as targets in the first part of the bipartite graph, and b1, b2, and b3 are taken as targets in the second part. The nine overlap values obtained above are used as weights between the targets in the two parts. For example, the overlap Iou1 between the contours of a1 and b1 can be used as the weight between targets a1 and b1, and the overlap Iou2 between the contours of a1 and b2 can be used as the weight between targets a1 and b2, and so on. Next, the KM algorithm can be used to process the bipartite graph to solve the optimal matching problem of weighted bipartite graphs. The specific operation principle of the KM algorithm can be found in existing technology. After processing by the KM algorithm, a target in the second part of the bipartite graph may or may not be matched with a target in the first part. If all targets in the second part of the bipartite graph are successfully matched with targets in the first part, it means that each target road segment in the second image is successfully matched with a target road segment in the first image. At this time, the line segment matching result of the target road segments in the first and second images can be determined as a match. Otherwise, if at least one target roadside segment in the second image cannot be matched with a target roadside segment in the first image, then the matching result of the target roadside segments in the first and second images can be determined as a mismatch. After completing the target roadside segment tracking and matching of N frames of images, the roadside marker points of the road to be tested obtained from the target point cloud detection described above can include:
[0095] (1) If the line segment matching results of the target road edge line segments of all two adjacent frames are all matched, then the road edge marker points of each frame of the N frames are detected according to the target point cloud of each frame of the N frames.
[0096] (2) Determine the road edge markers of the road to be tested based on the road edge markers of each of the N frames of images.
[0097] If the line segment matching results for the target roadside segments in images 1-N are all matches, it means that the target roadside segments in images 1-N all belong to the same roadside. For target point clouds with height differences: Detect target point clouds with height differences in point cloud 1, calculate the average X-coordinate of each row of points in the region where the target point cloud is located, and these average X-coordinates are considered as the roadside markers detected in image 1; detect target point clouds with height differences in point cloud 2, calculate the average X-coordinate of each row of points in the region where the target point cloud is located, and these average X-coordinates are considered as the roadside markers detected in image 2, and so on, until the roadside markers detected in image N are obtained. Finally, the average X-coordinate of each corresponding roadside marker in images 1-N can be calculated as the final output of the roadside markers for the tested road.
[0098] The aforementioned method of detecting roadside markers for the road under test based on the point cloud of the area where the target roadside line segment is located in the target image can include:
[0099] (1) If the line segment matching results of the target road edge line segments of all two adjacent frames are all matched, then the road edge marker points of each frame of the N frames are detected according to the point cloud of the area where the target road edge line segments of each frame of the N frames are located.
[0100] (2) Determine the road edge markers of the road to be tested based on the road edge markers of each of the N frames of images.
[0101] If the line segment matching results for the target roadside segments in Images 1 through N are all matches, it means that the target roadside segments in Images 1 through N all belong to the same roadside. For the case where there is no target point cloud: detect the point cloud of the region where the target roadside segment is located in Image 1, calculate the average X-coordinate of each row of points in that region, and treat these average X-coordinates as the roadside markers detected in Image 1; detect the point cloud of the region where the target roadside segment is located in Image 2, calculate the average X-coordinate of each row of points in that region, and treat these average X-coordinates as the roadside markers detected in Image 2; and so on, until the roadside markers detected in Image N are obtained. Finally, the average X-coordinate of each corresponding roadside marker in Images 1 through N can be calculated as the final output of the roadside markers for the tested road.
[0102] like Figure 4 The image shown is a schematic diagram illustrating the effect of obtaining the curb marker points of the road under test using the curb detection method provided in this application embodiment. Figure 4 The image only shows the left side curb markers, and the three white dashed lines represent the curb markers detected in the three frames at different times.
[0103] Furthermore, after performing line segment matching processing on the target roadside line segments of all adjacent two-frame images contained in the N-frame images, the process may further include:
[0104] If there are multiple consecutive images in the N frames where the target roadside line segment has a non-matching result, then the roadside detection result will be deleted.
[0105] In practice, a vector can be created to store the roadside detection results. This vector can store roadside markers for 2N frames of images. Each time a new image frame is input, its roadside markers are stored in this vector. When the vector already contains roadside markers from N frames, the average coordinates of the current roadside markers are calculated (as the current roadside detection result). Each time a new image frame is input, the average coordinates of the current roadside markers are recalculated. If the vector already contains roadside markers from 2N frames, the roadside markers from the first frame are deleted from the vector according to a first-in, first-out (FIFO) principle to make room for the new image's roadside markers. It should be noted that the line segment matching results for the target roadside line segments in each frame stored in each vector are all matched. If a target roadside line segment in a certain frame does not match with the previous frame, a new vector is created to store the roadside marker data for that frame. If there are multiple consecutive frames where the target roadside line segments do not match, the created vector will be deleted, which means the corresponding roadside detection result will be deleted. For example, if the target roadside line segments of images 1-5 match, vector A stores the roadside marker point data of images 1-5. If the target roadside line segments of image 6 do not match those of image 5, a new vector B will be created to store the roadside marker point data of image 6. If images 7-10 do not match those of image 5, the created vector A can be deleted.
[0106] In this embodiment, considering that lane line detection and curb detection are similar in that both involve detecting line segments, the existing neural network model used for lane line detection, i.e., the lane line detection model, has been partially improved to make it suitable for curb detection. Specifically, the training samples of the lane line detection model can be replaced with road images of marked lane lines and road images of marked curbs. Furthermore, the threshold used by the lane line detection model to distinguish anchor points as positive or negative samples is increased to a certain extent. After this improvement, the lane line detection model becomes a curb detection model. When it is necessary to detect the curb of the road to be tested, the image of the road to be tested is input into the curb detection model for processing, and a set of candidate curb points is output. Then, the final curb detection result is obtained by combining the point cloud data of the road to be tested. This embodiment uses a deep learning-based curb detection model, which outperforms general pure image processing schemes. Moreover, by combining point cloud data for auxiliary localization, the influence of interference such as lighting on curb detection can be effectively reduced, thereby improving the accuracy of curb detection.
[0107] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0108] The above mainly describes a curb detection method; the following will describe a curb detection device.
[0109] Please see Figure 5 One embodiment of a curb detection device in this application includes:
[0110] Data acquisition module 501 is used to acquire target images and point cloud data of the road to be tested;
[0111] The curb detection model processing module 502 is used to input the target image into the trained curb detection model for processing and output a set of candidate curb points; wherein, the curb detection model is obtained by modifying the neural network model used to realize lane line detection, and is trained using road image data with marked curbs as samples, and the threshold used to divide anchor points into positive samples or negative samples is greater than a specified threshold.
[0112] The curb detection module 503 is used to detect the curb detection result of the road to be tested based on the candidate curb point set and the point cloud data.
[0113] In one implementation of this application, the candidate curb point set includes the coordinate points of at least one candidate curb line segment, and the curb detection module may include:
[0114] The target roadside segment selection unit is used to select the coordinate points of the target roadside segment from the coordinate points of the at least one candidate roadside segment using a non-maximum suppression method.
[0115] A point-to-line unit is used to connect the coordinate points of the target roadside line segment into a line to obtain the target roadside line segment;
[0116] A point cloud projection unit is used to project the point cloud data onto the target image;
[0117] The curb marker detection unit is used to detect the curb markers of the road to be tested based on the point cloud of the area where the target curb line segment is located, which is obtained by projecting the point cloud data in the target image.
[0118] Furthermore, the curb detection module may also include:
[0119] The line segment thickening unit is used to perform contour thickening processing on the target roadside line segment.
[0120] Furthermore, the curb marker detection unit may include:
[0121] The target point cloud detection unit is used to detect whether there are target point clouds in the point cloud data whose height difference is greater than a set threshold.
[0122] The first road edge detection unit is used to detect the road edge marker point of the road to be tested based on the target point cloud if the target point cloud exists in the point cloud data.
[0123] The second road edge detection unit is used to detect the road edge marker point of the road to be tested based on the point cloud of the area where the target road edge line segment is located in the target image if the target point cloud does not exist in the point cloud data.
[0124] Furthermore, the target image comprises N consecutive frames, and the point cloud data comprises the point cloud corresponding to each of the N frames;
[0125] The point cloud projection unit can be specifically used to: project the point cloud corresponding to each of the N frames of images onto that frame of image;
[0126] The curb detection module may further include:
[0127] A line segment matching unit is used to perform line segment matching processing on the target roadside line segments of all two adjacent frames contained in the N frame images;
[0128] The first edge detection unit may include:
[0129] The first road edge marker detection subunit is used to detect the road edge markers of each of the N frames of images based on the target point cloud of each frame of the N frames of images if the line segment matching results of the target road edge line segments of all two adjacent frames of images are all matched.
[0130] The first road edge marker point determination subunit is used to determine the road edge marker points of the road to be tested based on the road edge marker points of each frame in the N frames of images.
[0131] The second edge detection unit may include:
[0132] The second road edge marker detection subunit is used to detect the road edge markers of each of the N frames of images based on the point cloud of the area where the target road edge line segment of each frame of the N frames of images is located, if the line segment matching results of the target road edge line segments of all two adjacent frames of images are all matched.
[0133] The second road edge marker determination subunit is used to determine the road edge markers of the road to be tested based on the road edge markers of each frame in the N frames of images.
[0134] Furthermore, if the first image and the second image are any two adjacent images contained in the N frames, the line segment matching unit may include:
[0135] The overlap calculation subunit is used to iterate and calculate the overlap between the contour of each target roadside line segment contained in the first image and the contour of each target roadside line segment contained in the second image.
[0136] The bipartite graph target construction subunit is used to take each target roadside segment contained in the first image as the target of the first part of the bipartite graph, and take each target roadside segment contained in the second image as the target of the second part of the bipartite graph.
[0137] The KM algorithm processing subunit is used to perform KM algorithm processing on the bipartite graph, using each of the overlap degrees as the weight between the target of the first part and the target of the second part.
[0138] The line segment matching result determination subunit is used to determine the line segment matching result of the target roadside line segments of the first image and the second image as a match if each target in the second part is successfully matched with the target in the first part after the KM algorithm processing is executed.
[0139] Furthermore, the curb detection module may also include:
[0140] The curb detection result deletion unit is used to delete the curb detection result if there are no matching results for the target curb line segments in the N frames of images for more than M consecutive frames.
[0141] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figure 1 This represents any kind of curb detection method.
[0142] This application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute the implementation of... Figure 1 This represents any kind of curb detection method.
[0143] Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of this application. For example... Figure 6 As shown, the terminal device 6 in this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the embodiments of the various curb detection methods described above, for example... Figure 1 Steps 101 to 103 are shown. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 The functions of modules 501 to 503 are shown.
[0144] The computer program 62 can be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 62 in the terminal device 6.
[0145] The processor 60 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0146] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 6. Furthermore, the memory 61 can include both internal and external storage units of the terminal device 6. The memory 61 is used to store the computer program and other programs and data required by the terminal device. The memory 61 can also be used to temporarily store data that has been output or will be output.
[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0149] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0150] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0151] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.
[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.
[0153] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.
[0155] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A curb detection method, characterized in that, include: Acquire the target image and point cloud data of the road to be tested; The target image is input into a trained curb detection model for processing, and a set of candidate curb points is output. The curb detection model is obtained by modifying a neural network model used to realize lane line detection. It is trained using road image data with marked curbs as samples, and the threshold used to distinguish anchor points as positive or negative samples is greater than a specified threshold. The set of candidate curb points includes the coordinate points of at least one candidate curb line segment. Using a non-maximum suppression method, the coordinates of the target roadside segment are selected from the coordinates of the at least one candidate roadside segment. Connect the coordinate points of the target roadside segment with a line to obtain the target roadside segment; The point cloud data is projected onto the target image; Detect whether there are target point clouds in the point cloud data whose height difference is greater than a set threshold; If the target point cloud exists in the point cloud data, the roadside marker point of the road to be tested is detected based on the target point cloud. If the target point cloud is not present in the point cloud data, then the roadside marker point of the road to be tested is detected based on the point cloud of the area where the target roadside segment is located, which is obtained by projecting the point cloud data from the point cloud data into the target image. The target image includes N consecutive frames, and the point cloud data includes the point cloud corresponding to each of the N frames. The step of projecting the point cloud data onto the target image specifically involves: For each of the N frames, the point cloud corresponding to that frame is projected onto that frame; After obtaining the target road edge segment of each of the N frames of images, the process further includes: Perform line segment matching processing on the target roadside line segments of all two adjacent frames contained in the N frames of images; The step of detecting the curb marker points of the road to be tested based on the target point cloud includes: If the line segment matching results of the target road edge line segments of all two adjacent frames are all matches, then the road edge marker points of each frame of the N frames are detected according to the target point cloud of each frame of the N frames. The road edge markers of the road to be tested are determined based on the road edge markers of each of the N frames of images. The step of detecting the roadside marker points of the road to be tested based on the point cloud of the area where the target roadside segment is located, obtained by projecting the point cloud data from the target image, includes: If the line segment matching results of the target road edge line segments in all two adjacent frames are all matches, then the road edge marker points of each frame in the N frames are detected based on the point cloud of the region where the target road edge line segments of each frame are located. The roadside markers of the road to be tested are determined based on the roadside markers of each of the N frames of images.
2. The method as described in claim 1, characterized in that, After connecting the coordinate points of the target roadside segment to obtain the target roadside segment, the method further includes: The target roadside segment is subjected to contour thickening processing.
3. The method as described in claim 1, characterized in that, Let the first image and the second image be any two adjacent images contained in the N frames of images. The step of performing line segment matching processing on the target roadside line segments of all two adjacent frames contained in the N frames of images includes: The overlap between the contours of each target roadside line segment contained in the first image and the contours of each target roadside line segment contained in the second image is calculated. Each target roadside segment contained in the first image is used as the target of the first part of the bipartite graph, and each target roadside segment contained in the second image is used as the target of the second part of the bipartite graph. The KM algorithm is applied to the bipartite graph, using the respective overlap degrees as weights between the targets of the first part and the second part. If, after processing with the KM algorithm, each target in the second part is successfully matched with a target in the first part, then the line segment matching result of the target roadside line segment in the first image and the second image is determined to be a match.
4. The method as described in claim 1 or 3, characterized in that, After performing line segment matching processing on the target roadside line segments of all adjacent two-frame images contained in the N-frame images, the process further includes: If there are multiple consecutive images in the N frames where the target roadside line segment has a non-matching result, then the roadside detection result will be deleted.
5. A curb detection device, characterized in that, include: The data acquisition module is used to acquire target images and point cloud data of the road under test; The curb detection model processing module is used to input the target image into a trained curb detection model for processing and output a set of candidate curb points. The curb detection model is obtained by modifying a neural network model used to realize lane line detection. It is trained using road image data with marked curbs as samples, and the threshold used to distinguish anchor points as positive or negative samples is greater than a specified threshold. The set of candidate curb points includes the coordinate points of at least one candidate curb line segment. The target roadside segment selection unit is used to select the coordinate points of the target roadside segment from the coordinate points of the at least one candidate roadside segment using a non-maximum suppression method. A point-to-line unit is used to connect the coordinate points of the target roadside line segment into a line to obtain the target roadside line segment; A point cloud projection unit is used to project the point cloud data onto the target image; The target point cloud detection unit is used to detect whether there are target point clouds in the point cloud data whose height difference is greater than a set threshold. The first road edge detection unit is used to detect the road edge marker point of the road to be tested based on the target point cloud if the target point cloud exists in the point cloud data. The second road edge detection unit is used to detect the road edge marker point of the road to be tested based on the point cloud of the area where the target road edge line segment is located, which is obtained by projecting the point cloud data from the point cloud data, if the target point cloud does not exist in the point cloud data. The target image includes N consecutive frames, and the point cloud data includes the point cloud corresponding to each of the N frames. The point cloud projection unit is specifically used to: project the point cloud corresponding to each of the N frames of images onto that frame of image; The device further includes: A line segment matching unit is used to perform line segment matching processing on the target roadside line segments of all two adjacent frames contained in the N frame images; The first edge detection unit includes: The first road edge marker detection subunit is used to detect the road edge markers of each of the N frames of images based on the target point cloud of each frame of the N frames of images if the line segment matching results of the target road edge line segments of all two adjacent frames of images are all matched. The first road edge marker point determination subunit is used to determine the road edge marker points of the road to be tested based on the road edge marker points of each frame in the N frames of images. The second edge detection unit includes: The second road edge marker detection subunit is used to detect the road edge markers of each of the N frames of images based on the point cloud of the area where the target road edge line segment of each frame of the N frames of images is located, if the line segment matching results of the target road edge line segments of all two adjacent frames of images are all matched. The second road edge marker determination subunit is used to determine the road edge markers of the road to be tested based on the road edge markers of each frame in the N frames of images.
6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the curb detection method as described in any one of claims 1 to 4.
7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the curb detection method as described in any one of claims 1 to 4.