A road target detection method, device, terminal equipment and storage medium

Abstract

The application relates to the technical field of environmental perception, and discloses a road target detection method and device, a terminal device and a storage medium. The method comprises the following steps: acquiring a target image of a to-be-detected road; inputting the target image into a trained multitask network for processing; outputting a road target detection result of the target image through a first detection head of the multitask network, and outputting a road edge detection result of the target image through a second detection head of the multitask network; wherein the multitask network is a neural network model trained by using road images with labeled road targets and road edges as a training set. The method adopts a deep learning multitask model, uses a multitask network to fuse road edge detection and road target detection, can realize one input and two outputs, and two types of target detection can share one network inference layer, so that different detection results can be output by setting different detection heads, thereby reducing the consumption of system resources to a certain extent.

Description

Technical Field

[0001] This application relates to the field of environmental perception technology, and in particular to a method, apparatus, terminal device and storage medium for road target detection. Background Technology

[0002] Currently, when garbage trucks operate on roads, they typically need to detect road edges and road targets such as garbage. There are two different types of targets to be detected: road edges and road targets. Existing technologies generally construct two different neural network models, one for detecting road edges and the other for detecting road targets. For example, existing technologies might construct a CenterNet-based target detection model and a LaneATT-based road edge detection model, then use the target detection model to detect road targets and the road edge detection model to detect road edges. However, running two neural network models simultaneously generates a huge computational load, resulting in high system resource consumption. Summary of the Invention

[0003] In view of this, embodiments of this application provide a road target detection method, apparatus, terminal device, and storage medium that can reduce the consumption of system resources.

[0004] The first aspect of this application provides a road target detection method, including:

[0005] Acquire the target image of the road to be tested;

[0006] The target image is input into a trained multi-task network for processing. The first detection head of the multi-task network outputs the road target detection result of the target image, and the second detection head of the multi-task network outputs the road edge detection result of the target image. The multi-task network is a neural network model trained using road images with labeled road targets and road edges as the training set.

[0007] In this embodiment, a target image of the road to be tested is first acquired. Then, the target image is input into a trained multi-task network for processing. This multi-task network has two detection heads: one outputs the road target detection result of the target image, and the other outputs the road edge detection result. This embodiment employs a deep learning multi-task model, using a multi-task network to fuse road edge detection and road target detection. This achieves one input and two outputs; both types of target detection can share a single network inference layer. Only different detection heads need to be set to output different detection results, thus reducing system resource consumption to a certain extent.

[0008] In one implementation of this application, the architecture of the multi-task network includes a feature extraction network and a feature pyramid network; the step of inputting the target image into the trained multi-task network for processing, outputting the road target detection result of the target image through the first detection head of the multi-task network, and outputting the road edge detection result of the target image through the second detection head of the multi-task network, may include:

[0009] The target image is input into the feature extraction network for processing to obtain a first feature image;

[0010] The first feature image is input into the feature pyramid network for processing to obtain the second feature image;

[0011] The second feature image is input into the first detection head for target detection processing to obtain the road target detection result;

[0012] The second feature image is input into the second detection head for curb detection processing to obtain the curb detection result.

[0013] In one implementation of this application, the target image comprises N consecutive frames, where N≥2. After inputting the target image into a trained multi-task network for processing, and outputting the road target detection result of the target image through the first detection head of the multi-task network, and outputting the road edge detection result of the target image through the second detection head of the multi-task network, the process may further include:

[0014] Create a first target tracking queue and a second target tracking queue;

[0015] The road target detection result of the first frame of the N frames is stored in the first target tracking queue, and the road edge detection result of the first frame is stored in the second target tracking queue.

[0016] The road target detection result of the second frame image in the N frames is matched with the road target detection result of the first target tracking queue, and the road target detection result of the first target tracking queue is updated according to the matching result;

[0017] The edge detection results of the second frame image are matched with the edge detection results of the second target tracking queue, and the edge detection results of the second target tracking queue are updated according to the matching results.

[0018] Furthermore, the step of matching the road target detection result of the second frame image in the N frames with the road target detection result in the first target tracking queue, and updating the road target detection result in the first target tracking queue according to the matching result, may include:

[0019] The overlap between each first target detection box in the road target detection result of the second frame image and each second target detection box in the road target detection result of the first target tracking queue is calculated.

[0020] Each of the first target detection boxes is used as the target of the first part of the bipartite graph, and each of the second target detection boxes is used as the target of the second part of the bipartite graph;

[0021] Using the overlap as the weight between the target in the first part and the target in the second part, the KM algorithm matching process is performed on the bipartite graph to obtain the matching results of each first target detection box and each second target detection box;

[0022] For each first target detection box that is successfully matched, the second target detection box that matches the first target detection box in the road target detection results of the first target tracking queue is updated using the first target detection box.

[0023] For each first target detection box that fails to match, a corresponding new target detection box is created, and the new target detection box is stored in the road target detection result of the first target tracking queue.

[0024] Furthermore, the step of matching the road edge detection result of the second frame image with the road edge detection result of the second target tracking queue, and updating the road edge detection result of the second target tracking queue according to the matching result, may include:

[0025] Contour thickening is performed on each first road edge segment in the road edge detection result of the second frame image, and contour thickening is performed on each second road edge segment in the road edge detection result of the second target tracking queue.

[0026] Iterate through and calculate the overlap between the contour of each first path segment and the contour of each second path segment;

[0027] Each of the first path segments is used as the target of the first part of the bipartite graph, and each of the second path segments is used as the target of the second part of the bipartite graph.

[0028] Using the overlap as the weight between the target of the first part and the target of the second part, the KM algorithm matching process is performed on the bipartite graph to obtain the matching results of each first path segment and each second path segment.

[0029] For each first road edge segment that is successfully matched, the second road edge segment that matches the first road edge segment in the road edge detection result of the second target tracking queue is updated using the first road edge segment;

[0030] For each first roadside segment that fails to match, a corresponding new roadside segment is created, and the new roadside segment is stored in the roadside detection result of the second target tracking queue.

[0031] Furthermore, the step of matching the road edge detection result of the second frame image with the road edge detection result of the second target tracking queue, and updating the road edge detection result of the second target tracking queue according to the matching result, may include:

[0032] The overlap ratio between each first road edge segment in the road edge detection result of the second frame image and each second road edge segment in the road edge detection result of the second target tracking queue is calculated. The overlap ratio is equal to the ratio of the number of road edge points in which the difference between the x coordinates of the two corresponding road edge segments within the overlapping y coordinate range is less than a set threshold to the total number of road edge points in the two road edge segments.

[0033] Each of the first path segments is used as the target of the first part of the bipartite graph, and each of the second path segments is used as the target of the second part of the bipartite graph.

[0034] Using the respective overlap ratios as weights between the targets of the first part and the second part, the KM algorithm matching process is performed on the bipartite graph to obtain the matching results of each first path segment and each second path segment.

[0035] For each first road edge segment that is successfully matched, the second road edge segment that matches the first road edge segment in the road edge detection result of the second target tracking queue is updated using the first road edge segment;

[0036] For each first roadside segment that fails to match, a corresponding new roadside segment is created, and the new roadside segment is stored in the roadside detection result of the second target tracking queue.

[0037] In one implementation of this application, after inputting the target image into a trained multi-task network for processing, outputting the road target detection result of the target image through the first detection head of the multi-task network, and outputting the road edge detection result of the target image through the second detection head of the multi-task network, the process may further include:

[0038] Project the point cloud data of the road to be tested onto the target image;

[0039] Based on the road target detection results of the point cloud data and the target image, the position coordinates of each road target on the road to be tested are calculated.

[0040] Based on the point cloud data and the road edge detection results of the target image, the position coordinates of each road edge point of the road to be tested are calculated;

[0041] Output the position coordinates of each road target and the position coordinates of each roadside point.

[0042] A second aspect of this application provides a road target detection device, comprising:

[0043] The image acquisition module is used to acquire target images of the road to be tested.

[0044] The target detection module is used to input the target image into a trained multi-task network for processing, output the road target detection result of the target image through the first detection head of the multi-task network, and output the road edge detection result of the target image through the second detection head of the multi-task network; wherein, the multi-task network is a neural network model trained using road images with labeled road targets and road edges as a training set.

[0045] A third aspect of this application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the road target detection method provided in the first aspect of this application.

[0046] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the road target detection method provided in the first aspect of this application.

[0047] The fifth aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the road target detection method provided in the first aspect of this application.

[0048] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0049] Figure 1 This is a flowchart of a road target detection method provided in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the architecture of a multi-task network provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the operation process for implementing garbage and curb detection using a multi-task network, as provided in the embodiments of this application.

[0052] Figure 4 This is a structural framework diagram of a road target detection device provided in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0054] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0055] Existing technologies typically construct multiple different neural networks when detecting different targets such as road edges and road objects, which leads to high system resource consumption. To address this issue, embodiments of this application provide a road object detection method, apparatus, terminal device, and storage medium that can reduce system resource consumption. For more specific technical implementation details of the embodiments of this application, please refer to the method embodiments described below.

[0056] It should be understood that the execution subject of the various method embodiments of this application can be various types of terminal devices or servers, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of this application do not impose any restrictions on the specific type of terminal device and server.

[0057] Please see Figure 1 This illustration shows a road target detection method provided by an embodiment of this application, comprising:

[0058] 101. Obtain the target image of the road to be tested;

[0059] First, an image of the road to be tested is acquired, represented as a target image. The road to be tested can be any type of road that requires detection of curbs and certain road targets (such as garbage, obstacles, etc.). In practice, for scenarios where vehicles such as sweepers are traveling on the road to be tested, LiDAR and cameras can be installed on the vehicle body. The camera acquires an image of the road ahead as the target image, while the LiDAR can acquire point cloud data of the road ahead. The point cloud data is mainly used for subsequent localization of the image targets.

[0060] 102. Input the target image into the trained multi-task network for processing. Output the road target detection result of the target image through the first detection head of the multi-task network, and output the road edge detection result of the target image through the second detection head of the multi-task network.

[0061] After obtaining the target image, it can be input into a trained multi-task network for processing. This multi-task network has two detection heads, referred to as the first detection head and the second detection head. The first detection head outputs the road target detection results of the target image, and the second detection head outputs the road edge detection results of the target image. In practice, a large number of road images with labeled road targets and road edges can be used as a training set to train a neural network model to obtain this multi-task network.

[0062] In one implementation of this application, the architecture of the multi-task network includes a feature extraction network and a feature pyramid network; the step of inputting the target image into the trained multi-task network for processing, outputting the road target detection result of the target image through the first detection head of the multi-task network, and outputting the road edge detection result of the target image through the second detection head of the multi-task network, may include:

[0063] (1) The target image is input into the feature extraction network for processing to obtain a first feature image;

[0064] (2) Input the first feature image into the feature pyramid network for processing to obtain the second feature image;

[0065] (3) Input the second feature image into the first detection head for target detection processing to obtain the road target detection result;

[0066] (4) Input the second feature image into the second detection head for curb detection processing to obtain the curb detection result.

[0067] like Figure 2 The diagram shows an architecture of a multi-task network provided in an embodiment of this application. This multi-task network includes a feature extraction network (as the backbone network), a feature pyramid network (FPN, as the neck network), and two detection heads (head networks, as two detection branches). After the target image is input into this multi-task network, the feature extraction network first extracts image features to obtain a first feature image. Then, the first feature image is input into the feature pyramid network for processing, outputting a second feature image. The use of the feature pyramid network here introduces multi-scale features, improving the detection effect for small targets. Next, the second feature image is input into the two detection branches respectively. The first detection head performs target detection processing, outputting the corresponding road target detection results, and the second detection head performs curb detection processing, outputting the corresponding curb detection results. During target detection, various existing target detection algorithms can be used, such as the anchor-free CenterNet algorithm, which detects all road targets (e.g., garbage) on the road surface and classifies them into only one category, outputting the target detection box and category in the target image. When performing curb detection, various existing curb detection algorithms can be used. For example, the LaneATT algorithm based on line anchors can be used to regress the set of coordinate points of each curb on the preset line anchor (that is, the coordinates of each curb point). The LaneATT algorithm supports the recognition of the following two common curbs: 1) curbs with curbs; 2) curbs without curbs.

[0068] In one implementation of this application, the target image comprises N consecutive frames, where N≥2. After inputting the target image into a trained multi-task network for processing, and outputting the road target detection result of the target image through the first detection head of the multi-task network, and outputting the road edge detection result of the target image through the second detection head of the multi-task network, the process may further include:

[0069] (1) Create a first target tracking queue and a second target tracking queue;

[0070] (2) Store the road target detection result of the first frame image in the N frames image into the first target tracking queue, and store the road edge detection result of the first frame image into the second target tracking queue.

[0071] (3) Match the road target detection result of the second frame image in the N frames with the road target detection result of the first target tracking queue, and update the road target detection result of the first target tracking queue according to the matching result;

[0072] (4) Match the edge detection results of the second frame image with the edge detection results of the second target tracking queue, and update the edge detection results of the second target tracking queue according to the matching results.

[0073] When a target (including road targets and road edges) is detected in an image, it is necessary to ensure the consistency of each target in the tracking queue between consecutive frames. Therefore, a target matching operation between consecutive frames is required. Assuming the target image consists of N consecutive (N≥2) frames, two target tracking queues need to be created when performing the target matching operation between consecutive frames. The queue used to store the road target detection results is called the first target tracking queue, and the queue used to store the road edge detection results is called the second target tracking queue. First, the road target detection results of the first frame image are stored in the first target tracking queue, and the road edge detection results of the first frame image are stored in the second target tracking queue. Then, the road target detection results of the second frame image are matched with the road target detection results in the first target tracking queue (at this point, the road target detection results stored in the first target tracking queue are the same as those in the first frame image, i.e., target matching between consecutive frames is performed), and the road target detection results in the first target tracking queue are updated according to the matching results. Similarly, the road edge detection results of the second frame image are matched with the road edge detection results in the second target tracking queue (at this point, the road edge detection results stored in the second target tracking queue are the same as those in the first frame image, i.e., target matching between consecutive frames is performed), and the road edge detection results in the second target tracking queue are updated according to the matching results. Specifically, if the target match is successful, the corresponding target information can be updated; if the target match fails, a new target can be created and stored in the target tracking queue. The above describes the target matching between the second and first frames. The same method can then be used sequentially to match the targets between the third and second frames, the fourth and third frames, and so on, until all consecutive frames are matched. At this point, the final target detection result can be output through two target tracking queues. Furthermore, the output can be limited to targets that appear more than a set threshold, for example, only targets detected in five or more consecutive frames can be output. This reduces false detections and improves the accuracy of target detection.

[0074] Furthermore, the step of matching the road target detection result of the second frame image in the N frames with the road target detection result in the first target tracking queue, and updating the road target detection result in the first target tracking queue according to the matching result, may include:

[0075] (1) Iterate through and calculate the overlap between each first target detection box in the road target detection result of the second frame image and each second target detection box in the road target detection result of the first target tracking queue;

[0076] (2) Using each of the first target detection boxes as targets in the first part of the bipartite graph, and using each of the second target detection boxes as targets in the second part of the bipartite graph;

[0077] (3) Using the overlap as the weight between the target in the first part and the target in the second part, perform KM algorithm matching on the bipartite graph to obtain the matching results of each first target detection box and each second target detection box;

[0078] (4) For each first target detection box that is successfully matched, update the second target detection box that matches the first target detection box in the road target detection result of the first target tracking queue using the first target detection box;

[0079] (5) For each first target detection box that fails to match, create a corresponding new target detection box and store the new target detection box in the road target detection result of the first target tracking queue.

[0080] When matching road target detection results between the second frame and the first frame, the KM algorithm can be used for target association matching. First, the overlap between each target detection box (represented by a first target detection box) in the second frame's road target detection results and each target detection box (represented by a second target detection box) in the first target tracking queue's road target detection results is calculated. Then, each first target detection box is used as the target of the first part of the bipartite graph, and each second target detection box is used as the target of the second part of the bipartite graph. Next, using the overlap as the weight between the targets of the first and second parts, the KM algorithm is performed on the bipartite graph to obtain the matching results for each first and second target detection box. Afterward, based on these matching results, successfully matched and unmatched first target detection boxes can be determined. For each successfully matched first target detection box, the second target detection boxes matching that first target detection box in the first target tracking queue's road target detection results are updated using that first target detection box. For each unmatched first target detection box, a corresponding new target detection box is created and stored in the first target tracking queue's road target detection results.

[0081] For example, assuming the first object detection box includes a1, a2, and a3, and the second object detection box includes b1, b2, and b3, then the overlap between a1 and b1, a1 and b2, a1 and b3, a2 and b1, and so on, is calculated, resulting in nine overlap values. A bipartite graph is a special type of graph that can be divided into two parts, where the points in each part are not connected. Using a1, a2, and a3 as the targets of the first part of the bipartite graph, and b1, b2, and b3 as the targets of the second part, the nine overlap values ​​are used as weights between the targets in the two parts. For example, the overlap Iou1 between a1 and b1 can be used as the weight between targets a1 and b1, the overlap Iou2 between a1 and b2 can be used as the weight between targets a1 and b2, and so on. Next, the KM algorithm can be used to process this bipartite graph, thereby solving the optimal matching problem in a weighted bipartite graph. The specific operating principle of the KM algorithm can be found in existing technologies. After processing by the KM algorithm, the matching results of each first target detection box (a1, a2, and a3) and each second target detection box (b1, b2, and b3) can be obtained.

[0082] Optionally, the step of matching the road edge detection result of the second frame image with the road edge detection result of the second target tracking queue, and updating the road edge detection result of the second target tracking queue according to the matching result, may include:

[0083] (1) Perform contour thickening processing on each first road edge segment in the road edge detection result of the second frame image, and perform contour thickening processing on each second road edge segment in the road edge detection result of the second target tracking queue.

[0084] (2) Iterate through and calculate the overlap between the contour of each first road segment and the contour of each second road segment;

[0085] (3) Using each of the first road segments as the target of the first part of the bipartite graph, and using each of the second road segments as the target of the second part of the bipartite graph;

[0086] (4) Using the overlap as the weight between the target of the first part and the target of the second part, perform KM algorithm matching processing on the bipartite graph to obtain the matching results of each first path segment and each second path segment.

[0087] (5) For each first roadside segment that is successfully matched, update the second roadside segment in the roadside detection result of the second target tracking queue that matches the first roadside segment using the first roadside segment;

[0088] (6) For each first roadside segment that fails to match, create a corresponding new roadside segment and store the new roadside segment in the roadside detection result of the second target tracking queue.

[0089] When matching the road edge detection results of the second frame image and the first frame image, the KM algorithm can also be used for target association matching. Since the detected road edge segments may be too thin, it is first necessary to perform contour thickening, i.e., segment dilation, on each road edge segment (represented by the first road edge segment) in the road edge detection results of the second frame image and each road edge segment (represented by the second road edge segment) in the road edge detection results of the second target tracking queue. Then, the overlap between the contours of each first road edge segment and each second road edge segment is calculated. Next, each first road edge segment is used as the target of the first part of the bipartite graph, and each second road edge segment is used as the target of the second part of the bipartite graph, with each overlap value used as the target of the first part. The weights between the target and the second part of the target are used to perform KM algorithm matching on the bipartite graph, thereby obtaining the matching results of each first path segment and each second path segment. Then, based on the matching results, the first path segments that are successfully matched and the first path segments that are not matched can be determined. For each successfully matched first path segment, the second path segment that matches the first path segment in the path detection results of the second target tracking queue is updated using the first path segment. For each unmatched first path segment, a corresponding new path segment is created and the new path segment is stored in the path detection results of the second target tracking queue.

[0090] Optionally, the step of matching the road edge detection result of the second frame image with the road edge detection result of the second target tracking queue, and updating the road edge detection result of the second target tracking queue according to the matching result, may include:

[0091] (1) Iterate through and calculate the overlap ratio between each first road edge segment in the road edge detection result of the second frame image and each second road edge segment in the road edge detection result of the second target tracking queue; wherein, the overlap ratio is equal to the ratio of the number of road edge points in which the difference between the x coordinates of the two corresponding road edge segments in the overlapping y coordinate range is less than a set threshold to the total number of road edge points of the two road edge segments.

[0092] (2) Using each of the first road segments as the target of the first part of the bipartite graph, and using each of the second road segments as the target of the second part of the bipartite graph;

[0093] (3) Using the overlap ratios as the weights between the targets of the first part and the second part, perform KM algorithm matching on the bipartite graph to obtain the matching results of each first path segment and each second path segment.

[0094] (4) For each first roadside segment that is successfully matched, update the second roadside segment in the roadside detection result of the second target tracking queue that matches the first roadside segment using the first roadside segment;

[0095] (5) For each first roadside segment that fails to match, create a corresponding new roadside segment and store the new roadside segment in the roadside detection result of the second target tracking queue.

[0096] When matching the road edge detection results of the second frame image and the first frame image, in addition to the method of calculating the overlap degree by line segment dilation as described above, the method of calculating the overlap ratio can also be used. First, iterate through and calculate the overlap ratio between each first road edge line segment in the road edge detection results of the second frame image and each second road edge line segment in the road edge detection results of the second target tracking queue. This overlap ratio is equal to the ratio of the number of road edge points in the overlapping y-coordinate range where the difference in x-coordinates between the two corresponding road edge line segments is less than a set threshold to the total number of road edge points in the two road edge line segments. For example, assuming that the starting and ending coordinates of a certain first road edge line segment are A1 = (x1, y1) and A1 = (x2, y2) respectively, and the starting and ending coordinates of a certain second road edge line segment are B1 = (x3, y3) and B2 = (x4, y4) respectively, then find the starting coordinate point with the largest y-coordinate as the target starting coordinate point, and find the ending coordinate point with the smallest y-coordinate as the target ending coordinate point. Assuming y1 is greater than y3 and y2 is greater than y4, the target starting coordinate point is the starting coordinate point A1 of the target roadside line segment in the first image, and the target ending coordinate point is the ending coordinate point B2 of the target roadside line segment in the second image. Within the y-coordinate range defined by the target starting and ending coordinate points (i.e., the y-coordinate range where the two roadside line segments overlap), both roadside line segments have coordinate points, allowing for line segment matching. Specifically, the difference between the x-coordinate of each coordinate point in the first roadside line segment and the x-coordinate of the corresponding coordinate point (with the same y-coordinate) in the second roadside line segment can be calculated. Then, the number of roadside points whose x-coordinate difference is less than a set threshold can be counted. Dividing this count by the total number of roadside points in both roadside line segments yields the overlap ratio. Then, using each first path segment as the target of the first part of the bipartite graph and each second path segment as the target of the second part of the bipartite graph, and using the overlap ratio as the weight between the targets of the first and second parts, the KM algorithm matching process is performed on the bipartite graph to obtain the matching results of each first path segment and each second path segment. After that, based on the matching results, the successfully matched first path segments and the unmatched first path segments can be determined. For each successfully matched first path segment, the second path segments that match the first path segment in the path detection results of the second target tracking queue are updated using the first path segment. For each unmatched first path segment, a corresponding new path segment is created and the new path segment is stored in the path detection results of the second target tracking queue.

[0097] In one implementation of this application, after inputting the target image into a trained multi-task network for processing, outputting the road target detection result of the target image through the first detection head of the multi-task network, and outputting the road edge detection result of the target image through the second detection head of the multi-task network, the process may further include:

[0098] (1) Project the point cloud data of the road to be tested onto the target image;

[0099] (2) Based on the road target detection results of the point cloud data and the target image, calculate the position coordinates of each road target on the road to be tested;

[0100] (3) Based on the point cloud data and the road edge detection results of the target image, calculate the position coordinates of each road edge point of the road to be tested;

[0101] (4) Output the position coordinates of each road target and the position coordinates of each roadside point.

[0102] After using a multi-task network to output road target detection and curb detection results for the target image, the point cloud data of the road under test at the same time can be projected onto the target image. The projected point cloud data allows for the calculation of the position coordinates of image targets (including road targets and curbs) in a specified coordinate system (e.g., vehicle coordinate system), thus achieving target localization. Specifically, when calculating the position coordinates of a road target, a portion of the point cloud of the central region of the target detection bounding box can be obtained. The average radar coordinates of this portion of the point cloud are then used as the position coordinates of the road target. The curb detection results already contain the pixel coordinates of each curb point. By matching and converting the pixel coordinates to the radar coordinates, the position coordinates of each curb point in the specified coordinate system can be obtained. Furthermore, after obtaining the position coordinates of each curb point, a corresponding curb curve can be fitted.

[0103] Furthermore, the step of outputting the position coordinates of each road target and the position coordinates of each roadside point may include:

[0104] (1) Determine the roadside area of ​​the road to be measured based on the position coordinates of each roadside point;

[0105] (2) Delete the road targets that are outside the roadside area from the various road targets;

[0106] (3) Output the position coordinates of each road target and the position coordinates of each road edge point after deleting the road targets outside the road edge area.

[0107] In scenarios where sweepers and other vehicles are traveling on a test road, if the sweeper detects litter (road targets) outside the curb area, it may cause the vehicle to malfunction. To avoid this problem, the curb area of ​​the test road can be determined based on the calculated positions of various curb points. Then, all road targets outside the curb area are deleted. The final output consists of the position coordinates of the road targets within the curb area and the position coordinates of each curb point. In other words, by filtering out litter targets outside the curb area, the normal driving and operation of sweepers and other vehicles on the road can be ensured.

[0108] In this embodiment, a target image of the road to be tested is first acquired. Then, the target image is input into a trained multi-task network for processing. This multi-task network has two detection heads: one outputs the road target detection result of the target image, and the other outputs the road edge detection result. This embodiment employs a deep learning multi-task model, using a multi-task network to fuse road edge detection and road target detection. This achieves one input and two outputs; both types of target detection can share a single network inference layer. Only different detection heads need to be set to output different detection results, thus reducing system resource consumption to a certain extent.

[0109] To facilitate understanding of the road target detection method provided in the embodiments of this application, a practical application scenario is listed below.

[0110] like Figure 3 The diagram illustrates the workflow for garbage and curb detection using a multi-task network. First, the road image is input into the trained multi-task network, which then outputs garbage and curb detection results through different detection branches. Next, target matching and tracking are performed on both the garbage and curb detection results. Then, radar point cloud data from the same moment is projected onto the image, and the target coordinates in the vehicle coordinate system, including garbage and curb coordinates, are calculated. Finally, the garbage and curb coordinates are fused and filtered to remove garbage targets outside the curb area, thus outputting the final result.

[0111] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0112] The above mainly describes a road target detection method; the following will describe a road target detection device.

[0113] Please see Figure 4 One embodiment of a road target detection device in this application includes:

[0114] Image acquisition module 401 is used to acquire target images of the road to be tested;

[0115] The target detection module 402 is used to input the target image into a trained multi-task network for processing, output the road target detection result of the target image through the first detection head of the multi-task network, and output the road edge detection result of the target image through the second detection head of the multi-task network; wherein, the multi-task network is a neural network model trained using road images with labeled road targets and road edges as a training set.

[0116] In one implementation of this application, the architecture of the multi-task network includes a feature extraction network and a feature pyramid network, and the target detection module may include:

[0117] The first feature extraction unit is used to input the target image into the feature extraction network for processing to obtain a first feature image;

[0118] The second feature extraction unit is used to input the first feature image into the feature pyramid network for processing to obtain the second feature image;

[0119] The target detection processing unit is used to input the second feature image into the first detection head for target detection processing to obtain the road target detection result;

[0120] The curb detection processing unit is used to input the second feature image into the second detection head for curb detection processing to obtain the curb detection result.

[0121] In one implementation of this application, the target image comprises N consecutive frames, where N ≥ 2, and the road target detection device may further include:

[0122] The target tracking queue creation module is used to create the first target tracking queue and the second target tracking queue;

[0123] The tracking queue initialization module is used to store the road target detection result of the first frame image in the N frames of images into the first target tracking queue, and to store the road edge detection result of the first frame image into the second target tracking queue.

[0124] The road target matching module is used to match the road target detection result of the second frame image in the N frames with the road target detection result of the first target tracking queue, and update the road target detection result of the first target tracking queue according to the matching result;

[0125] The road edge matching module is used to match the road edge detection results of the second frame image with the road edge detection results of the second target tracking queue, and update the road edge detection results of the second target tracking queue according to the matching results.

[0126] Furthermore, the road target matching module may include:

[0127] The first overlap calculation unit is used to iterate and calculate the overlap between each first target detection box in the road target detection result of the second frame image and each second target detection box in the road target detection result of the first target tracking queue.

[0128] The first bipartite graph target determination unit is used to use each of the first target detection boxes as targets in the first part of the bipartite graph, and to use each of the second target detection boxes as targets in the second part of the bipartite graph.

[0129] The first KM matching unit is used to perform KM algorithm matching processing on the bipartite graph, using the overlap as the weight between the target in the first part and the target in the second part, to obtain the matching results of each first target detection box and each second target detection box.

[0130] The first tracking queue update unit is used to update the second target detection box that matches the first target detection box in the road target detection result of the first target tracking queue for each first target detection box that is successfully matched;

[0131] The second tracking queue update unit is used to create a corresponding new target detection box for each first target detection box that fails to match, and store the new target detection box in the road target detection result of the first target tracking queue.

[0132] Furthermore, the curb matching module may include:

[0133] The line segment dilation unit is used to perform contour thickening processing on each first road edge line segment in the road edge detection result of the second frame image, and to perform contour thickening processing on each second road edge line segment in the road edge detection result of the second target tracking queue.

[0134] The second overlap calculation unit is used to iterate and calculate the overlap between the contour of each first road segment and the contour of each second road segment.

[0135] The second bipartite graph target determination unit is used to use each of the first path segments as targets of the first part of the bipartite graph, and each of the second path segments as targets of the second part of the bipartite graph.

[0136] The second KM matching unit is used to perform KM algorithm matching processing on the bipartite graph, using the overlap as the weight between the target of the first part and the target of the second part, to obtain the matching results of each first path segment and each second path segment.

[0137] The third tracking queue update unit is used to update the second edge segment in the edge detection result of the second target tracking queue that matches the first edge segment for each first edge segment that is successfully matched.

[0138] The fourth tracking queue update unit is used to create a corresponding new road edge segment for each first road edge segment that fails to match, and to store the new road edge segment in the road edge detection result of the second target tracking queue.

[0139] Furthermore, the curb matching module may include:

[0140] The overlap ratio calculation unit is used to iterate and calculate the overlap ratio between each first road edge segment in the road edge detection result of the second frame image and each second road edge segment in the road edge detection result of the second target tracking queue; wherein, the overlap ratio is equal to the ratio of the number of road edge points in which the difference between the x coordinates of the corresponding two road edge segments within the overlapping y coordinate range is less than a set threshold to the total number of road edge points of the two road edge segments.

[0141] The third bipartite graph target determination unit is used to take each of the first road segments as targets of the first part of the bipartite graph, and to take each of the second road segments as targets of the second part of the bipartite graph.

[0142] The third KM matching unit is used to perform KM algorithm matching processing on the bipartite graph using the respective overlap ratios as weights between the targets of the first part and the second part, to obtain the matching results of each first path segment and each second path segment.

[0143] The fifth tracking queue update unit is used to update the second road edge segment in the road edge detection result of the second target tracking queue that matches the first road edge segment for each first road edge segment that is successfully matched;

[0144] The sixth tracking queue update unit is used to create a corresponding new road edge segment for each first road edge segment that fails to match, and store the new road edge segment in the road edge detection result of the second target tracking queue.

[0145] In one implementation of this application, the road target detection device may further include:

[0146] A point cloud projection module is used to project the point cloud data of the road to be tested onto the target image;

[0147] The road target location calculation module is used to calculate the location coordinates of each road target on the road to be tested based on the point cloud data and the road target detection results of the target image.

[0148] The curb position calculation module is used to calculate the position coordinates of each curb point of the road under test based on the curb detection results of the point cloud data and the target image.

[0149] The position coordinate output module is used to output the position coordinates of each road target and the position coordinates of each roadside point.

[0150] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figure 1 This represents any road target detection method.

[0151] This application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute the implementation of... Figure 1 This represents any road target detection method.

[0152] Figure 5 This is a schematic diagram of a terminal device provided in an embodiment of this application. For example... Figure 5 As shown, the terminal device 5 in this embodiment includes: a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, it implements the steps in the embodiments of the various road target detection methods described above, for example... Figure 1 Steps 101 to 102 are shown. Alternatively, when the processor 50 executes the computer program 52, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of modules 401 to 402 are shown.

[0153] The computer program 52 can be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 52 in the terminal device 5.

[0154] The processor 50 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0155] The memory 51 can be an internal storage unit of the terminal device 5, such as a hard disk or memory of the terminal device 5. The memory 51 can also be an external storage device of the terminal device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 5. Furthermore, the memory 51 can include both internal and external storage units of the terminal device 5. The memory 51 is used to store the computer program and other programs and data required by the terminal device. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0156] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0157] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0158] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0159] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0160] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0161] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0162] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0163] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0164] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A road target detection method, characterized in that, include: Acquire the target image of the road to be tested; The target image is input into a trained multi-task network for processing. The first detection head of the multi-task network outputs the road target detection result of the target image, and the second detection head of the multi-task network outputs the road edge detection result of the target image. The multi-task network is a neural network model trained using road images with labeled road targets and road edges as the training set. The target image comprises N consecutive frames, where N ≥ 2. After inputting the target image into a trained multi-task network for processing, and outputting the road target detection result of the target image through the first detection head of the multi-task network, and the road edge detection result of the target image through the second detection head of the multi-task network, the process further includes: Create a first target tracking queue and a second target tracking queue; The road target detection result of the first frame of the N frames is stored in the first target tracking queue, and the road edge detection result of the first frame is stored in the second target tracking queue. The road target detection result of the second frame image in the N frames is matched with the road target detection result of the first target tracking queue, and the road target detection result of the first target tracking queue is updated according to the matching result; The edge detection results of the second frame image are matched with the edge detection results of the second target tracking queue, and the edge detection results of the second target tracking queue are updated according to the matching results.

2. The method as described in claim 1, characterized in that, The architecture of the multi-task network includes a feature extraction network and a feature pyramid network; the process of inputting the target image into the trained multi-task network for processing, outputting the road target detection result of the target image through the first detection head of the multi-task network, and outputting the road edge detection result of the target image through the second detection head of the multi-task network includes: The target image is input into the feature extraction network for processing to obtain a first feature image; The first feature image is input into the feature pyramid network for processing to obtain the second feature image; The second feature image is input into the first detection head for target detection processing to obtain the road target detection result; The second feature image is input into the second detection head for curb detection processing to obtain the curb detection result.

3. The method as described in claim 1, characterized in that, The step of matching the road target detection result of the second frame image in the N frames with the road target detection result in the first target tracking queue, and updating the road target detection result in the first target tracking queue according to the matching result, includes: The overlap between each first target detection box in the road target detection result of the second frame image and each second target detection box in the road target detection result of the first target tracking queue is calculated. Each of the first target detection boxes is used as the target of the first part of the bipartite graph, and each of the second target detection boxes is used as the target of the second part of the bipartite graph; Using the overlap as the weight between the target in the first part and the target in the second part, the KM algorithm matching process is performed on the bipartite graph to obtain the matching results of each first target detection box and each second target detection box; For each first target detection box that is successfully matched, the second target detection box that matches the first target detection box in the road target detection results of the first target tracking queue is updated using the first target detection box. For each first target detection box that fails to match, a corresponding new target detection box is created, and the new target detection box is stored in the road target detection result of the first target tracking queue.

4. The method as described in claim 1, characterized in that, The step of matching the road edge detection results of the second frame image with the road edge detection results of the second target tracking queue, and updating the road edge detection results of the second target tracking queue according to the matching results, includes: Contour thickening is performed on each first road edge segment in the road edge detection result of the second frame image, and contour thickening is performed on each second road edge segment in the road edge detection result of the second target tracking queue. Iterate through and calculate the overlap between the contour of each first path segment and the contour of each second path segment; Each of the first path segments is used as the target of the first part of the bipartite graph, and each of the second path segments is used as the target of the second part of the bipartite graph. Using the overlap as the weight between the target of the first part and the target of the second part, the KM algorithm matching process is performed on the bipartite graph to obtain the matching results of each first path segment and each second path segment. For each first road edge segment that is successfully matched, the second road edge segment that matches the first road edge segment in the road edge detection result of the second target tracking queue is updated using the first road edge segment; For each first roadside segment that fails to match, a corresponding new roadside segment is created, and the new roadside segment is stored in the roadside detection result of the second target tracking queue.

5. The method as described in claim 1, characterized in that, The step of matching the road edge detection results of the second frame image with the road edge detection results of the second target tracking queue, and updating the road edge detection results of the second target tracking queue according to the matching results, includes: The overlap ratio between each first road edge segment in the road edge detection result of the second frame image and each second road edge segment in the road edge detection result of the second target tracking queue is calculated. The overlap ratio is equal to the ratio of the number of road edge points in which the difference between the x coordinates of the two corresponding road edge segments within the overlapping y coordinate range is less than a set threshold to the total number of road edge points in the two road edge segments. Each of the first path segments is used as the target of the first part of the bipartite graph, and each of the second path segments is used as the target of the second part of the bipartite graph. Using the respective overlap ratios as weights between the targets of the first part and the second part, the KM algorithm matching process is performed on the bipartite graph to obtain the matching results of each first path segment and each second path segment. For each first road edge segment that is successfully matched, the second road edge segment that matches the first road edge segment in the road edge detection result of the second target tracking queue is updated using the first road edge segment; For each first roadside segment that fails to match, a corresponding new roadside segment is created, and the new roadside segment is stored in the roadside detection result of the second target tracking queue.

6. The method according to any one of claims 1 to 5, characterized in that, After inputting the target image into a trained multi-task network for processing, outputting the road target detection result of the target image through the first detection head of the multi-task network, and outputting the road edge detection result of the target image through the second detection head of the multi-task network, the process further includes: Project the point cloud data of the road to be tested onto the target image; Based on the road target detection results of the point cloud data and the target image, the position coordinates of each road target on the road to be tested are calculated. Based on the point cloud data and the road edge detection results of the target image, the position coordinates of each road edge point of the road to be tested are calculated; Output the position coordinates of each road target and the position coordinates of each roadside point.

7. A road target detection device, characterized in that, include: The image acquisition module is used to acquire the target image of the road to be tested, wherein the target image includes N consecutive frames, where N≥2; The target detection module is used to input the target image into a trained multi-task network for processing, output the road target detection result of the target image through the first detection head of the multi-task network, and output the road edge detection result of the target image through the second detection head of the multi-task network; wherein, the multi-task network is a neural network model trained using road images with labeled road targets and road edges as a training set; The target tracking queue creation module is used to create the first target tracking queue and the second target tracking queue; The tracking queue initialization module is used to store the road target detection result of the first frame image in the N frames of images into the first target tracking queue, and to store the road edge detection result of the first frame image into the second target tracking queue. The road target matching module is used to match the road target detection result of the second frame image in the N frames with the road target detection result of the first target tracking queue, and update the road target detection result of the first target tracking queue according to the matching result; The road edge matching module is used to match the road edge detection results of the second frame image with the road edge detection results of the second target tracking queue, and update the road edge detection results of the second target tracking queue according to the matching results.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the road target detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the road target detection method as described in any one of claims 1 to 6.

Images