Traffic cone detection method, device, equipment and storage medium

By marking the intersection points between the traffic cone and the ground and using convolutional neural network for traffic cone line detection, the problems of inconsistent traffic cone detection results and low robustness in the prior art are solved, and a more stable traffic cone connection representation is achieved.

CN115050003BActive Publication Date: 2024-11-08GUANGZHOU WERIDE TECH LTD CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210602966.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-30
Publication Date
2024-11-08
Estimated Expiration
2042-05-30

AI Technical Summary

Technical Problem

In the prior art, when detecting traffic cones, especially in complex road conditions, it is difficult to obtain consistent results, and the connection results are relatively robust and fluctuations are easily generated.

Method used

By obtaining the road image to be marked, marking the intersection of the traffic cone and the ground to draw the traffic cone line, generating an ordered set of traffic cone lines, and using a convolutional neural network for training, a preset detection model is obtained, and traffic is performed on the images to be detected. Cone line detection improves the consistency and robustness of detection results.

Benefits of technology

It effectively improves the consistency and robustness of the traffic cone connection results, ensures the higher-order semantic consistency of the traffic cone connection relationship, reduces the fluctuations in the detection results, and improves the robustness problem in the prior art.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115050003B_ABST
    Figure CN115050003B_ABST
Patent Text Reader

Abstract

The present application discloses a traffic cone line detection method, device, equipment and storage medium. The method comprises: obtaining a road image to be annotated, wherein the road image to be annotated includes at least one group of traffic cones; taking the intersection points of each traffic cone in the road image to be annotated and the ground as annotation points, drawing traffic cone lines on the road image to be annotated based on the annotation points to obtain an annotated road image, and generating an ordered point set of traffic cone lines; processing the road image to be detected based on the annotated road image and the ordered point set of traffic cone lines to obtain a traffic cone line detection result, thereby improving the technical problem that the prior art connects associated traffic cones into groups based on preset rules, which has the problem of difficulty in obtaining consistent results, and the connection results are prone to fluctuations and low robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a traffic cone detection method, device, equipment and storage medium. Background Art

[0002] Due to temporary traffic problems, traffic cones are usually set up on the road, either individually or in groups to change the road traffic conditions, and traffic cones are an important basis for autonomous vehicles to make decisions. Existing technologies usually detect traffic cones as a unique type of obstacle, and then connect the associated traffic cones into groups based on preset rules according to the detected traffic cones, the direction and position of the main vehicle, and provide them to autonomous vehicles for decision-making. This method is difficult to obtain consistent results in the face of different traffic conditions, especially complex road conditions, and the connection results are prone to fluctuations and have low robustness. Summary of the invention

[0003] The present application provides a traffic cone line detection method, device, equipment and storage medium, which are used to improve the technical problems that the existing technology is difficult to obtain consistent results, the connection results are prone to fluctuations, and the robustness is low in the prior art of connecting related traffic cones into groups based on preset rules.

[0004] In view of this, the first aspect of the present application provides a traffic cone marking method, comprising:

[0005] Acquire a road image to be annotated, wherein the road image to be annotated includes at least one group of traffic cones;

[0006] Taking the intersection points of each traffic cone and the ground in the road image to be labeled as the labeling points, drawing traffic cone lines on the road image to be labeled based on the labeling points to obtain a labeled road image, and generating a traffic cone line ordered point set of the traffic cone line;

[0007] The road image to be detected is processed based on the labeled road image and the ordered point set of traffic cones to obtain a traffic cone detection result.

[0008] Optionally, the processing of the road image to be detected based on the labeled road image and the ordered point set of traffic cones to obtain a traffic cone detection result includes:

[0009] Calculating the target offset of each traffic cone line by using the labeled road image and the ordered point set of the traffic cone line;

[0010] Taking the annotated road image as input data, training a preset convolutional neural network with the traffic cone line category, the target offset and the traffic cone line length as learning targets to obtain a preset detection model, wherein the traffic cone line category and the traffic cone line length are obtained by annotation;

[0011] Traffic cone line detection is performed on the road image to be detected by using the preset detection model to obtain a traffic cone line detection result.

[0012] Optionally, the calculating the target offset of each traffic cone line by using the labeled road image and the ordered point set of the traffic cone line includes:

[0013] Taking a point that is half of the cumulative length of each traffic cone in the traffic cone ordered point set as a target point, and calculating the pixel coordinates of the target point projected into the traffic cone feature map according to the size of the annotated road image and the size of the traffic cone feature map extracted by the preset convolutional neural network;

[0014] Calculate the target point offset of each traffic cone line through the pixel coordinates of the target point of each traffic cone line in the traffic cone line feature map;

[0015] Calculate the offsets of the sampling points other than the target point in each traffic cone line in the traffic cone line ordered point set and the corresponding target point to obtain the sampling point offsets;

[0016] The target point offset and the sampling point offset of each traffic cone line are used as the target offset, or the sampling point offset of each traffic cone line is normalized, and the target point offset and the normalized sampling point offset of each traffic cone line are used as the target offset.

[0017] Optionally, performing traffic cone line detection on the road image to be detected by using the preset detection model to obtain a traffic cone line detection result includes:

[0018] Performing traffic cone detection on the road image to be detected by using the preset detection model, and outputting the traffic cone existence probability, traffic cone category probability, target offset prediction value and traffic cone length prediction value of each pixel point in the traffic cone feature map extracted by the preset detection model;

[0019] Extracting target pixel points whose existence probability of the traffic cone is higher than an existence probability threshold from the traffic cone feature map output by the preset detection model;

[0020] Determine the traffic cone category of the target pixel point according to the traffic cone category probability of the target pixel point, and extract traffic cone instances whose traffic cone category probability is higher than a category probability threshold from the target pixel point;

[0021] Obtaining the length of the traffic cone line instance by using the traffic cone line length prediction value;

[0022] The pixel coordinates of each point in the traffic cone line instance in the image to be detected are obtained according to the target offset prediction value to obtain a traffic cone line sequence.

[0023] Optionally, the method further includes:

[0024] Calculating the distance between each traffic cone line in the traffic cone line sequence;

[0025] When the distance between two traffic cones is less than a preset distance threshold, the traffic cone sequence with a smaller probability of the traffic cone category is eliminated to obtain a final traffic cone sequence.

[0026] Optionally, the processing of the road image to be detected based on the labeled road image and the ordered point set of traffic cones to obtain a traffic cone detection result includes:

[0027] The connecting line of the marked points in the ordered point set of the traffic cone line is extended to a segmented mark with a preset width by pixels to obtain a target label of the marked road image;

[0028] Taking the annotated road image as input data and the target label as a network learning target, a preset convolutional neural network is trained to obtain a preset segmentation model;

[0029] Performing traffic cone segmentation on the road image to be detected by using the preset segmentation model, and outputting the traffic cone segmentation result;

[0030] Clustering the traffic cone pixel points in the traffic cone segmentation result to obtain a number of traffic cone instances;

[0031] The traffic cone line pixels in each of the traffic cone line instances are fitted to obtain the detected traffic cone line.

[0032] Optionally, the processing of the road image to be detected based on the labeled road image and the ordered point set of traffic cones to obtain a traffic cone detection result may also include:

[0033] Determine whether the number of marked points in the ordered point set of the traffic cone line is equal to the number of preset sampling points;

[0034] If not, then sequentially connect two adjacent marked points in the ordered point set of the traffic cone line to obtain a line segment vector group;

[0035] Calculating the curvature radius of the arc corresponding to two adjacent line segment vectors in the line segment vector group;

[0036] The traffic cone line ordered point set is processed according to the curvature radius so that the final number of marked points of the traffic cone line ordered point set is equal to the preset number of sampling points.

[0037] Optionally, the processing of the ordered point set of the traffic cone line according to the curvature radius so that the final number of annotation points of the ordered point set of the traffic cone line is equal to the preset number of sampling points includes:

[0038] When the number of the marked points of the ordered point set of the traffic cone line is less than the preset number of sampling points, the first number and the second number are calculated according to the difference between the preset number of sampling points and the number of the marked points of the ordered point set of the traffic cone line and the number of line segment vectors;

[0039] Sampling a second number plus one sampling point for a first line segment vector of two adjacent line segment vectors corresponding to the first number of minimum curvature radii, and sampling the second number of sampling points for each of the remaining line segment vectors, so that the final number of marked points of the ordered point set of the traffic cone line is equal to the preset number of sampling points, and the final marked points include the original marked points and the newly added sampling points;

[0040] When the number of the marked points of the ordered point set of the traffic cone line is greater than the preset number of sampling points, calculating the difference between the number of the marked points of the ordered point set of the traffic cone line and the preset number of sampling points to obtain a third number;

[0041] The end point of the first line segment vector of two adjacent line segment vectors corresponding to the third number of maximum curvature radii is deleted, so that the final number of marked points of the ordered point set of the traffic cone line is equal to the preset number of sampling points.

[0042] A second aspect of the present application provides a traffic cone line detection device, comprising:

[0043] An acquisition unit, configured to acquire a road image to be annotated, wherein the road image to be annotated includes at least one group of traffic cones;

[0044] a labeling unit, configured to use the intersection points of each traffic cone and the ground in the road image to be labeled as labeling points, draw traffic cone lines on the road image to be labeled based on the labeling points to obtain a labeled road image, and generate a traffic cone line ordered point set of the traffic cone line;

[0045] The detection unit is used to process the road image to be detected based on the labeled road image and the ordered point set of traffic cones to obtain a traffic cone detection result.

[0046] A third aspect of the present application provides a traffic cone detection device, the device comprising a processor and a memory;

[0047] The memory is used to store program code and transmit the program code to the processor;

[0048] The processor is used to execute any one of the traffic cone line detection methods described in the first aspect according to the instructions in the program code.

[0049] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and when the program code is executed by a processor, the traffic cone line detection method described in any one of the first aspects is implemented.

[0050] It can be seen from the above technical solutions that this application has the following advantages:

[0051] The present application provides a traffic cone line marking method, comprising: obtaining a road image to be marked, wherein the road image to be marked includes at least one group of traffic cones; using the intersection points of each traffic cone in the road image to be marked and the ground as marking points, drawing traffic cone lines on the road image to be marked based on the marking points to obtain a marked road image, and generating a traffic cone line ordered point set of the traffic cone line; processing the road image to be detected based on the marked road image and the traffic cone line ordered point set to obtain a traffic cone line detection result.

[0052] In the present application, traffic cone lines are drawn with the intersection points of each traffic cone and the ground in the road image to be annotated as the annotation points, that is, the grouped traffic cones are annotated as an independent instance, which helps to ensure the high-order semantic consistency of the traffic cone connection relationship, thereby obtaining a specific representation of the traffic cone connection relationship; then, based on the annotated road image and the ordered point set of traffic cone lines obtained by annotation, traffic cone line detection is performed on the road image to be detected, and the grouped traffic cone lines can be detected as an instance as a whole, which is not prone to fluctuations, and can effectively improve the consistency, continuity and robustness of the traffic cone connection results, thereby improving the technical problems of the prior art that the associated traffic cones are connected into groups based on preset rules, it is difficult to obtain consistent results, the connection results are prone to fluctuations, and the robustness is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A schematic diagram of a flow chart of a traffic cone marking method provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a labeled road image provided in an embodiment of the present application;

[0056] Figure 3 Another schematic diagram of a labeled road image provided by an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the structure of a traffic cone marking device provided in an embodiment of the present application;

[0058] Figure 5 A structural schematic diagram of a traffic cone line marking device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The present application provides a traffic cone line detection method, device, equipment and storage medium, which are used to improve the technical problems that the existing technology connects related traffic cones into groups based on preset rules, but it is difficult to obtain consistent results, the connection results are prone to fluctuations, and the robustness is low.

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] For easier understanding, see Figure 1 , the embodiment of the present application provides a traffic cone line marking method, including:

[0062] Step 101: Obtain a road image to be annotated.

[0063] The road image to be annotated can be collected by a vehicle sensor or obtained from a public database, wherein the road image to be annotated includes at least one group of traffic cones, and the group of traffic cones can at least be connected to form a traffic cone line. Traffic cones include common triangular cones, metal rods, metal columns, etc.

[0064] Step 102: Take the intersection points of each traffic cone and the ground in the road image to be annotated as annotation points, draw traffic cone lines on the road image to be annotated based on the annotation points to obtain an annotated road image, and generate an ordered point set of traffic cone lines.

[0065] The intersection points of each traffic cone and the visible ground in the road image to be annotated are taken as annotation points, and each annotation point is connected into a line, so that a virtual traffic cone line is generated in the road image to be annotated, thereby obtaining the corresponding annotated road image. After the traffic cone line is annotated, one of the two endpoints of the traffic cone line is taken as the starting point and the other endpoint is taken as the end point, and the pixel coordinates of the annotation points corresponding to each traffic cone line are saved in order from the starting point to the end point, thereby generating the corresponding ordered point set of the traffic cone line. It can be understood that when annotating, the traffic cone line category and the traffic cone line length can also be annotated. The traffic cone line category can be annotated according to the category of the traffic cone. For example, the traffic cone line corresponding to the triangular cone can be annotated as category one, and the traffic cone line corresponding to the metal rod can be annotated as category two. In the embodiment of the present application, the intersection point of the traffic cone and the ground is used as the annotation point for connecting the line, so as to facilitate the use of the ground plane constraint to project the traffic cone line in the image into the physical coordinate system for use by the autonomous driving system.

[0066] You can refer to Figure 2 A labeled road image is provided. Figure 2 A set of traffic cones in includes 3 traffic cones, corresponding to 3 annotation points. Connect two adjacent annotation points in sequence to get a traffic cone line. When generating the ordered point set of the traffic cone line, take one endpoint of the traffic cone line as the starting point and the other endpoint as the end point. Save the pixel coordinates of each annotation point in the annotated road image in order from the starting point to the end point, so as to generate the corresponding ordered point set of the traffic cone line. Please refer to Figure 3 Another annotated road image provided by Figure 2 The traffic cone line obtained by connecting the marked points in the connection method in the figure has a bending degree of approximately 90°. This bending part causes a significant change in the direction of the two segments before and after. At this time, it can be marked as two traffic cone lines, as shown in the following figure. Figure 3 The two traffic cone lines shown are traffic cone line 1 and traffic cone line 2, and then a traffic cone line ordered point set is generated for each traffic cone line.

[0067] It should be noted that, in addition to directly performing the above-mentioned annotation process on the collected road image to be annotated, the collected road image to be annotated can also be subjected to inverse perspective transformation or other processing methods to transform the collected road image to be annotated into a bird's-eye view, and then perform the above-mentioned annotation process on the bird's-eye view, and the generated annotated road image is a bird's-eye view, and the ordered point set of traffic cones is a point set on the bird's-eye view. The annotation result on the road image to be annotated can also be directly transformed into the annotation result on the bird's-eye view, and then the subsequent processing process is performed on the bird's-eye view.

[0068] Furthermore, considering that different traffic cone line ordered point sets may have different numbers of annotation points, for example, a group of traffic cones includes 5 traffic cones, and the corresponding traffic cone ordered point set includes 5 annotation points, and another group of traffic cones includes 8 traffic cones, and the corresponding traffic cone ordered point set includes 8 annotation points. In order to facilitate subsequent processing, each traffic cone line ordered point set can be preprocessed to obtain a traffic cone line ordered point set with a uniform number of annotation points. A number of sampling points can be preconfigured according to actual conditions or experience to obtain a preset number of sampling points. When the number of annotation points in the traffic cone line ordered point set is different from the preset number of sampling points, the traffic cone line ordered point set can be preprocessed to obtain a traffic cone line ordered point set with a preset number of sampling points.

[0069] In one embodiment, when the number of labeled points in the ordered point set of traffic cones is less than the preset number of sampling points, f sampling points can be randomly interpolated in the ordered point set of traffic cones according to the absolute difference f between the number of labeled points and the preset number of sampling points, so that the final number of labeled points in the ordered point set of traffic cones is equal to the preset number of sampling points. When the number of labeled points in the ordered point set of traffic cones is greater than the preset number of sampling points, f labeled points can be randomly deleted in the ordered point set of traffic cones according to the absolute difference f between the number of labeled points and the preset number of sampling points, and the starting point and the end point in the ordered point set of traffic cones are generally retained, so that the final number of labeled points in the ordered point set of traffic cones is equal to the preset number of sampling points.

[0070] In another embodiment, the ordered point set of the traffic cone line can be interpolated or some points can be deleted by using the curvature radius to obtain an ordered point set with a preset number of sampling points with minimal precision loss. Specifically, the process of obtaining the ordered point set of the traffic cone line with a preset number of sampling points can be:

[0071] S1. Determine whether the number of marked points in the ordered point set of the traffic cone line is equal to the number of preset sampling points;

[0072] It is determined whether the number of marked points in each traffic cone line ordered point set is equal to the preset number of sampling points, and then it can be determined whether the traffic cone line ordered point set needs to be processed. When the number of marked points in the traffic cone line ordered point set is equal to the preset sampling point data, all marked points are retained.

[0073] S2. If not, then connect two adjacent marked points in the ordered point set of the traffic cone line in sequence to obtain a line segment vector group;

[0074] When the number of annotated points in the ordered point set of traffic cones is not equal to the preset sampling point data, the line segment formed by two adjacent annotated points in the ordered point set of traffic cones is vectorized. Specifically, considering that in the road image collected by the vehicle, the traffic cone line close to the vehicle (such as the lane line close to the bottom of the image) is clearer, and the reliability and uniformity of the annotation are better. Therefore, the embodiment of the present application preferably uses the annotated point closest to the lower edge of the image as the starting point of the ordered point set of traffic cones, takes the two adjacent annotated points as endpoints, and sequentially connects the two adjacent annotated points to generate a line segment vector group. Each line segment vector in the line segment vector group can be represented in sequence as L1, L2, ..., L n , where n is the number of line segment vectors. Assume there are three marked points A(x1,y1), B(x2,y2), and C(x3,y3), where point A is the starting point and point C is the end point. Connect two adjacent marked points in sequence, and the resulting line segment vector group includes line segment vectors and the line segment vector Line segment vector It can be expressed as L1(x2-x1,y2-y1) and line segment vector It can be expressed as L2(x3-x2,y3-y2).

[0075] S3, calculating the curvature radius of the arc corresponding to two adjacent line segment vectors in the line segment vector group;

[0076] After obtaining the line segment vector group, the angle between two adjacent line segment vectors in the line segment vector group is calculated. In one embodiment, the cosine value of the angle between two adjacent line segment vectors in the line segment vector group can be calculated based on the inner product and the modulus of the two adjacent line segment vectors in the line segment vector group, such as the line segment vector L n-1 With the line segment vector L n The cosine of the angle is cosθ n-1 =(L n-1 ·L n ) / (|L n-1 ||L n |); The angle between two adjacent line segment vectors is calculated by the cosine value of the angle between the two adjacent line segment vectors, such as the line segment vector L n-1 With the line segment vector L n The angle is θ n-1 =arccos((L n-1 ·L n ) / (|L n-1 ||L n |)), where n>2.

[0077] In another embodiment, the sine value of the angle between two adjacent line segment vectors can be calculated according to the cross product and the modulus of the two adjacent line segment vectors in the line segment vector group, such as the line segment vector L n-1 With the line segment vector Ln The sine of the angle is sinθ n-1 =|L n-1 ×L n | / (|L n-1 ||L n |); The angle between two adjacent line segment vectors is calculated by the sine value of the angle between the two adjacent line segment vectors, such as the line segment vector L n-1 With the line segment vector L n The angle θ n-1 for:

[0078]

[0079] In another embodiment, the cosine value of the angle between two adjacent line segment vectors can also be calculated based on the inner product and the module length of two adjacent line segment vectors in the line segment vector group, such as the line segment vector L n-1 With the line segment vector L n The cosine of the angle is cosθ n-1 =(L n-1 ·L n ) / (|L n-1 ||L n |); Calculate the sine of the angle between two adjacent line segment vectors based on the cross product and modulus of the two adjacent line segment vectors, such as the line segment vector L n-1 With the line segment vector L n The sine of the angle is sinθ n-1 =|L n-1 ×L n | / (|L n-1 ||L n |); Calculate the tangent value of the angle between two adjacent line segment vectors by using the cosine value and sine value of the angle between two adjacent line segment vectors, such as line segment vector L n-1 With the line segment vector L n The tangent value of the angle is tanθ n-1 = sinθ n-1 / cosθ n-1 ; The angle between two adjacent line segment vectors is calculated by the tangent value of the angle between the two adjacent line segment vectors, such as the line segment vector L n-1 With the line segment vector L n The angle is θ n-1 =arctan(sinθ n-1 / cosθ n-1 ). Of course, we can also directly calculate the cross product and inner product of two adjacent line segment vectors in the line segment vector group, and then use the formula θ n-1 =arctan(|L n-1 ×L n | / L n-1 ·L n) calculates the angle between two adjacent line segment vectors.

[0080] The angle calculated through the above process has a value range of [0,180°]. After the angle between two adjacent line segment vectors is calculated, the arc corresponding to the two adjacent line segment vectors is determined by taking the line segments corresponding to the two adjacent line segment vectors as the tangents of the arc start point and the arc end point. The curvature radius of the arc corresponding to the two adjacent line segment vectors can be calculated based on the module length and the angle between the two adjacent line segment vectors. Specifically, a smooth arc is constructed from the starting point of the first line segment vector of two adjacent line segment vectors to the end point of the second line segment vector. The line segments corresponding to the two adjacent line segment vectors are used as the tangent of the arc starting point and the tangent of the arc end point to determine the only arc corresponding to the two adjacent line segment vectors. It is known that the curvature radius r of the arc is equal to the ratio of the arc length to the center angle corresponding to the arc (that is, the angle θ between the two adjacent line segment vectors corresponding to the arc). In the embodiment of the present application, the sum of the module lengths of the two adjacent line segment vectors is used to approximate the arc length of the arc corresponding to the angle between the two adjacent line segment vectors, and then the curvature radius of the arc corresponding to the two adjacent line segment vectors is calculated, that is, the curvature radius of the arc corresponding to the two adjacent line segment vectors L n-1 With L n The corresponding arc has a curvature radius of r n-1 =(|L n-1 |+|L n |) / θ n-1 . After calculating the curvature radius of the arc corresponding to two adjacent line segment vectors in the line segment vector group, the corresponding relationship between each curvature radius and the two adjacent line segment vectors can be associated. Assuming that there are three line segment vectors L1, L2, and L3 in the line segment vector group, the adjacent line segment vectors L1 and L2 can determine the unique arc L1L2. According to the module length and angle of the line segment vectors L1 and L2, the curvature radius r1 of the corresponding arc L1L2 can be calculated, that is, there is a corresponding relationship between the curvature radius r1 and the arc L1L2, and there is a corresponding relationship between the arc L1L2 and the adjacent line segment vectors L1 and L2. Therefore, the curvature radius r1 can be associated with the adjacent line segment vectors L1 and L2, that is, the adjacent line segment vector corresponding to the curvature radius r1 are line segment vectors L1 and L2; similarly, adjacent line segment vectors L2 and L3 can determine the unique arc L2L3, and the curvature radius r2 of the corresponding arc L2L3 can be calculated according to the module length and the included angle of the line segment vectors L2 and L3, that is, there is a corresponding relationship between the curvature radius r2 and the arc L2L3, and there is a corresponding relationship between the arc L2L3 and the adjacent line segment vectors L2 and L3, so that the curvature radius r2 can be associated with the adjacent line segment vectors L2 and L3, that is, the adjacent line segment vectors corresponding to the curvature radius r2 are the line segment vectors L2 and L3.

[0081] S4. Process the traffic cone line according to the curvature radius so that the final number of annotation points of the traffic cone line is equal to the preset number of sampling points, thereby obtaining an ordered point set of the preset number of sampling points corresponding to the traffic cone line image.

[0082] When the number of labeled points in the ordered point set of the traffic cone line is less than the preset number of sampling points, the ordered point set of the traffic cone line is interpolated according to the curvature radius so that the final number of labeled points in the ordered point set of the traffic cone line is equal to the preset number of sampling points. The final labeled points include the original labeled points and the newly added sampling points.

[0083] Specifically, when the number of labeled points N of the ordered point set of the traffic cone line is less than the preset number of sampling points M, the first number m=(MN) / n and the second number k=(MN)%n can be calculated according to the difference MN between the preset number of sampling points and the number of labeled points of the ordered point set of the traffic cone line and the number of line segment vectors n, where m is the quotient of MN divided by n, k is the remainder of MN divided by n, and n=N-1; the first line segment vector of the two adjacent line segment vectors corresponding to the first number of minimum curvature radii is interpolated with the second number plus one sampling point, and the second number of sampling points is interpolated for each of the remaining line segment vectors (i.e., the uninterpolated line segment vectors), so that the final number of labeled points of the ordered point set of the traffic cone line is equal to the preset number of sampling points. All curvature radii can be sorted in ascending order (or descending order) according to size, and then the first line segment vector of the two adjacent line segment vectors corresponding to the first k minimum curvature radii after ascending order can be uniformly interpolated with m+1 points, and each line segment vector of the remaining nk line segment vectors can be uniformly interpolated with m points. The smaller the radius of curvature, the greater the curvature of the line segment corresponding to the radius of curvature. When the number of annotation points in the ordered point set of the traffic cone line is less than the preset number of sampling points, it means that more annotation points are needed. The embodiment of the present application, while retaining the original annotation points, preferentially interpolates in the curved part of the line segment to fill the gap with the preset number of sampling points, without losing accuracy for the original ordered point set.

[0084] When the number of annotated points of the ordered point set of the traffic cone line is greater than the number of preset sampling points, some of the annotated points of the ordered point set of the traffic cone line may be deleted according to the curvature radius, so that the final number of annotated points of the ordered point set of the traffic cone line is equal to the number of preset sampling points. Specifically, when the number of annotated points N of the ordered point set of the traffic cone line is greater than the number of preset sampling points M, the difference NM between the number of annotated points of the ordered point set of the traffic cone line and the number of preset sampling points is calculated to obtain a third number NM; the end point of the first line segment vector of two adjacent line segment vectors corresponding to the third number NM maximum curvature radii is deleted, so that the final number of annotated points of the ordered point set of the traffic cone line is equal to the number of preset sampling points. When the number of annotated points N in the ordered point set of the traffic cone line is greater than the preset number of sampling points M, it means that NM annotated points in the ordered point set of the traffic cone line need to be deleted. All curvature radii can be sorted in descending order (or ascending order) according to their size, and then the end point of the first line segment vector in the two adjacent line segment vectors corresponding to the first NM largest curvature radii after descending order is deleted. At this time, the starting point and end point of the ordered point set of the traffic cone line will always be retained, that is, the deleted annotated point is the middle point of the line segment. The larger the curvature radius, the closer the corresponding line segment is to the straight line. When the number of annotated points needs to be reduced, the middle point of the line segment closest to the straight line can be deleted first, so as to keep the original shape of the traffic cone line as much as possible with a smaller error; deleting some annotated points in the ordered point set of the traffic cone line, although there will be a loss of accuracy relative to the original annotated points, this part of the loss in the embodiment of the present application is generated in the part of the line segment closest to the straight line, and the impact on the accuracy of the point set is very limited.

[0085] After determining the number of sampling points, traffic cones of different shapes can be represented by a unified ordered point set through the above processing method, and the number of points is usually not too many. Compared with the above-mentioned indiscriminate interpolation or deletion of marked points in the ordered point set of traffic cones, the method of processing traffic cones based on the curvature radius can use a relatively small number of points to more accurately represent traffic cones of various positions and shapes, with less loss of accuracy relative to the original ordered point set.

[0086] In the embodiment of the present application, two adjacent marked points in the traffic cone line ordered point set are connected to generate a line segment vector group so as to calculate the curvature radius of the arc corresponding to two adjacent line segment vectors in each line segment vector group, and then the traffic cone line ordered point set can be processed by the curvature radius, so that the final number of marked points in the traffic cone line ordered point set is unified with the preset number of sampling points, thereby realizing the unified representation of ordered point sets of different marked road surfaces and better meeting the actual needs.

[0087] Step 103: Process the road image to be detected based on the labeled road image and the ordered point set of traffic cones to obtain a traffic cone detection result.

[0088] In one embodiment, a preset segmentation model may be first trained using an annotated road image and an ordered set of traffic cone points, and then the preset segmentation model may be used to perform traffic cone segmentation processing on the road image to be detected, and then the segmentation results may be clustered and fitted to obtain traffic cone detection results. The specific process may be:

[0089] A1. Extend the lines connecting the marked points in the ordered point set of the traffic cone line to the segmentation mark of the preset width through pixels to obtain the target label of the marked road image;

[0090] Traffic cone detection can be performed by image segmentation. Therefore, before training the preset convolutional neural network, the traffic cones are first expanded to a segmentation annotation of a certain width through pixels, that is, the connection line of the marked points in the ordered point set of the traffic cones obtained by the above annotation can be expanded to a segmentation annotation of a preset width through pixels, that is, the traffic cones are a connected area in the image, and the connected areas corresponding to each traffic cone in the labeled road image are marked as the target labels of the labeled road image.

[0091] A2. Using the labeled road image as input data and the target label as the network learning target, the preset convolutional neural network is trained to obtain a preset segmentation model;

[0092] The preset convolutional neural network is trained with the labeled road image as input data and the target label as the network learning target. During the training process, the preset convolutional neural network will extract features and semantically segment the input labeled road image to obtain the segmentation result, which is a pixel-level traffic cone line connected area representation; the loss value is calculated according to the segmentation result of the labeled road image in the training process and the corresponding target label, and the network parameters of the preset convolutional neural network are updated by the loss value until the number of training iterations reaches the maximum number of iterations, or the training error remains basically unchanged and the training error is lower than the preset error threshold, then the preset convolutional neural network is determined to converge, and a trained preset convolutional neural network model is obtained, and the trained preset convolutional neural network model is used as the preset segmentation model. Among them, the existing convolutional neural network can be used as the preset convolutional neural network for training, and the network structure of the preset convolutional neural network is not specifically limited here, and technicians in this field can choose according to actual conditions.

[0093] A3. Use the preset segmentation model to perform traffic cone segmentation on the road image to be detected, and output the traffic cone segmentation result;

[0094] The road image to be detected is input into the preset segmentation model for traffic cone segmentation to segment the traffic cone from the background and obtain the traffic cone segmentation result. The traffic cone segmentation result is a pixel-level traffic cone connected domain representation, and the pixel points in the traffic cone connected domain representation are the traffic cone pixel points.

[0095] A4. Clustering the traffic cone pixel points in the traffic cone segmentation result to obtain several traffic cone instances;

[0096] After obtaining the traffic cone segmentation result, the traffic cone pixels in the traffic cone connected domain representation are clustered. The clustering result is that the traffic cone pixels of the same category constitute a traffic cone instance. When there are multiple categories of traffic cones, there are multiple traffic cone instances. The specific clustering method is not specifically limited here, and those skilled in the art can choose according to the actual situation.

[0097] A5. Fit the traffic cone pixel points in each traffic cone instance to obtain the detected traffic cone.

[0098] After clustering to obtain traffic cone instances, the traffic cones in each traffic cone instance are fitted, which can be a polynomial fit, to obtain traffic cones, that is, traffic cone detection results. Among them, one traffic cone instance corresponds to one traffic cone. In order to achieve a high-precision detection effect, a larger output resolution is usually required. When the network structure of the preset segmentation model is relatively complex, the high output resolution will greatly increase the calculation amount of the preset segmentation model, which will lead to unsatisfactory real-time detection results. It is difficult to meet the needs of some systems with high real-time requirements.

[0099] In another embodiment, a preset detection model may be directly trained using an annotated road image and an ordered set of traffic cone points. The preset detection model is used to perform end-to-end traffic cone detection on the road image to be detected to obtain a traffic cone detection result. The specific process may be:

[0100] B1. Calculate the target offset of each traffic cone line through the annotated road image and the ordered point set of traffic cone lines;

[0101] In the embodiment of the present application, when training a preset convolutional neural network, a network learning target of the preset convolutional neural network is constructed with a traffic cone line instance as a unit, and the specific representation of the traffic cone line instance can be: a point that is half of the cumulative length of each traffic cone line in the traffic cone line ordered point set is used as the target point of the corresponding traffic cone line instance, and a traffic cone line category of each traffic cone line instance is generated according to actual needs. Among them, one traffic cone line corresponds to one traffic cone line instance.

[0102] In the embodiment of the present application, it is preferred to use the point (x) that is half the cumulative length of each traffic cone line in the ordered point set. c ,y c) is the target point, and the pixel coordinates of the target point projected into the traffic cone feature map are calculated according to the size of the annotated road image img_width*img_length and the size of the traffic cone feature map extracted by the preset convolutional neural network F_width*F_length; the target point offset of each traffic cone line is calculated by the pixel coordinates of the target point of each traffic cone line in the traffic cone feature map. Specifically, the sampling multiple is calculated according to the size of the annotated road image and the size of the traffic cone feature map extracted by the preset convolutional neural network, and the sampling multiple includes the horizontal coordinate sampling multiple S x =img_width / F_width and vertical coordinate sampling multiple S y =img_length / F_length; the coordinates of the target point passing through each traffic cone line (x c ,y c ) and the sampling multiple are used to calculate the target point offset of each traffic cone line. Specifically, the horizontal coordinate value of the target point of each traffic cone line is divided by the horizontal coordinate sampling multiple, and the vertical coordinate value is divided by the vertical coordinate sampling multiple to obtain the middle value of the two coordinates of the target point of each traffic cone line (x g ,y g ), thereby projecting the target point in the original annotated road image into the grid of the traffic cone feature map, that is, x g =x c / S x ,y g =y c / S y ; Round the middle values ​​of the two coordinates of the target point to get the two coordinate index values ​​of the target point (g x ,g y ), i.e. g x =floor(x g ), g y =floor(y g ), floor() is a rounding function; the difference between the middle value of the two coordinates of the target point of each traffic cone line and the corresponding two coordinate index values ​​is calculated to obtain the target point offset (c x ,c y ), that is, c x =x g -g x , c y =y g -g y , the two coordinate values ​​of the target point offset are between 0 and 1.

[0103] After calculating the offset of the target point of each traffic cone line, first, divide the horizontal coordinate value of each sampling point of each traffic cone line except the target point by the horizontal coordinate sampling multiple, and divide the vertical coordinate value by the vertical coordinate sampling multiple to obtain the coordinates (x i ,y i ), and then according to the pixel coordinates (x i ,y i ) and the pixel coordinates of the corresponding target point on the traffic cone line feature map to calculate the offset between the sampling point and the corresponding target point on the traffic cone line feature map, and obtain the sampling point offset ( xi ,O yi ), namely O xi =x i -x g , O yi =y i -y g .

[0104] In one embodiment, the target point offset and the sampling point offset of each traffic cone line can be directly used as the target offset. The sampling point offset of each traffic cone line is directly input into the preset convolutional neural network as the target offset for learning. The numerical range span is relatively large, making the learning difficulty of the preset convolutional neural network relatively large.

[0105] In another embodiment, the sampling point offsets of each traffic cone line can be normalized, and the target point offset and the normalized sampling point offset are used as the target offset. Considering that directly using the sampling point offsets of each traffic cone line as the network learning target has a large numerical range and is difficult to learn. Therefore, the sampling point offsets of each traffic cone line can be normalized.

[0106] The horizontal coordinate value of the sampling point offset of each traffic cone line can be normalized by the width img_width of the labeled road image, and the vertical coordinate value of the sampling point offset of each traffic cone line can be normalized by the height img_length of the labeled road image. The value range is compressed to [0,1]. However, this method will increase the relative error for points with smaller offset values ​​(i.e., shorter traffic cone lines).

[0107] Alternatively, the two coordinate values ​​of the sampling point offset of each traffic cone line can be normalized by the length of the traffic cone line, and the same distribution of targets can be obtained for traffic cone lines of different lengths, which can reduce the learning difficulty.

[0108] The length of the traffic cone line can be an absolute length or a relative length normalized by the width, height or diagonal length of the annotated road image.

[0109] B2. Using the annotated road image as input data, the preset convolutional neural network is trained with traffic cone type, target offset and traffic cone length as learning targets to obtain a preset detection model;

[0110] The preset convolutional neural network in the embodiment of the present application includes a feature extraction module and a prediction module. An existing feature extraction network (such as a residual network, etc.) can be used as the feature extraction module. The feature extraction module is used to extract the feature map of the input image, and the prediction module is used to perform traffic cone detection on the feature map extracted by the feature extraction module. The prediction module can be composed of multiple detection heads. The detection head can be connected to any feature map of the feature extraction module, or it can be connected to multiple feature maps of different scales and different features at the same time. The feature map finally connected by the detection head is the traffic cone feature map mentioned above. The detection head can specifically include a traffic cone existence detection head, a traffic cone category detection head, a target point offset detection head, a sampling point offset detection head, and a traffic cone length detection head. The traffic cone existence detection head is used to detect whether there is a traffic cone in the input traffic cone feature map, and output the probability of traffic cone existence for each pixel on the traffic cone feature map. If the traffic cone feature map is the same size as the input annotated road image, then a pixel in the traffic cone feature map corresponds to a pixel in the original input annotated road image. If the traffic cone feature map is S times the size of the original input annotated road image after downsampling, then a pixel in the traffic cone feature map corresponds to the S*S range of pixels in the original input annotated road image. The value of each pixel output by the traffic cone existence detection head contains the probability of traffic cone existence. The traffic cone category detection head is used to detect the category of traffic cones in the input traffic cone feature map, and the output is the probability of traffic cone category. When the traffic cone is used as a detection task alone, the probability of traffic cone category can be a binary classification or a multi-classification; traffic cones can also be detected together with other road markings (such as lane lines, road edge lines, etc.). In this case, the detected category is multi-classification. When each traffic cone instance is represented by a probability of multiple classifications, each pixel point contains multiple classifications of the maximum number of traffic cones output. When the maximum number of traffic cones is set to 1, one pixel point corresponds to one traffic cone instance, and one traffic cone instance corresponds to one traffic cone. The target point offset detection head is used to predict the offset of the target point of each traffic cone on the traffic cone feature map, and output the target point offset prediction value. The sampling point offset detection head is used to predict the offset of the sampling points other than the target point in the traffic cone and the target point on the traffic cone feature map, and output the sampling point offset prediction value. The traffic cone length detection head is used to predict the length of each traffic cone and output the traffic cone length prediction value, which can be an absolute length or a relative length normalized by the width, height or diagonal length of the image. It can be understood that the prediction module in the embodiment of the present application can also use a visual transformer to replace the detection head, and use the visual transformer to directly predict a specified number of traffic cone instances.

[0111] The embodiment of the present application uses an annotated road image as input data, and trains a preset convolutional neural network with traffic cone line category, target offset and traffic cone line length as learning objectives until the preset convolutional neural network converges to obtain a trained preset convolutional neural network model, and uses the trained preset convolutional neural network model as a preset detection model.

[0112] The preset detection model in the embodiment of the present application is a training target constructed with traffic cone instances as units. The two classification detection heads can respectively realize the existence prediction and category prediction of traffic cones. The offset detection head can realize the prediction of the position and shape of traffic cones. Traffic cones of various shapes can be predicted with high detection accuracy. Moreover, the embodiment of the present application uses the point of half of the cumulative length of the traffic cone as the target point of the traffic cone instance, and can handle any number of traffic cones. The existence prediction is consistent with the actual performance of the image, and the subsequent association and merging between the line segments are more flexible.

[0113] B3. Perform traffic cone detection on the road image to be detected through a preset detection model to obtain the traffic cone detection result.

[0114] Traffic cone detection is performed on the road image to be detected through a preset detection model, and the traffic cone existence probability, traffic cone category probability, target offset prediction value and traffic cone length prediction value of each pixel point in the traffic cone feature map extracted by the preset detection model are output; target pixel points whose traffic cone existence probability is higher than the existence probability threshold are extracted from the traffic cone feature map output by the preset detection model; the traffic cone category of the target pixel point is determined according to the traffic cone category probability of the target pixel point, and traffic cone instances whose traffic cone category probability is higher than the category probability threshold are extracted from the target pixel point; the length of the traffic cone instance is obtained through the traffic cone length prediction value; the pixel coordinates of each point in the traffic cone instance in the image to be detected are obtained according to the target offset prediction value, and a traffic cone sequence is obtained.

[0115] When parsing the detection results output by the preset detection model, first, according to the traffic cone existence probability, the target pixel points whose traffic cone existence probability is higher than the existence probability threshold are extracted from the traffic cone feature map output by the preset detection model. The target pixel points whose traffic cone existence probability is higher than the existence probability threshold in the traffic cone feature map are points belonging to the traffic cone; then, the traffic cone category of the target pixel point is determined according to the traffic cone category probability of the target pixel point, and the traffic cone instances whose traffic cone category probability is higher than the category probability threshold are extracted from the target pixel point. The traffic cone category to which the target pixel point belongs is determined according to the traffic cone category probability of each traffic cone instance in each target pixel point on the traffic cone feature map. When there are multiple traffic cone categories, the traffic cone category detection head predicts the traffic cone category probability of each labeled category instance under each traffic cone category, and the sum of the traffic cone category probabilities under each category is 1, among which the category with the largest traffic cone category probability value is the category to which the traffic cone instance belongs, and then the traffic cone instances whose traffic cone category probability is higher than the category probability threshold are extracted from the target pixel point. The traffic cone line length of each traffic cone line instance can be obtained through the traffic cone line length prediction value.

[0116] The target offset in the embodiment of the present application includes a target point offset and a sampling point offset. For each extracted traffic cone line instance, first, the pixel coordinates of the target point of the traffic cone line instance on the scale of the traffic cone line feature map are obtained according to the target point offset. Specifically, when training the preset convolutional neural network, the calculated target point offset is directly used as the network learning target, so the target point offset prediction value (c') output by the target point offset detection head is directly used. x ,c' y ) to perform reverse calculation to obtain the pixel coordinates (x' g ,y' g ), where x' g =c' x +g x , y' g =c' y +g y , g x is the horizontal index value of the target pixel point corresponding to each traffic cone instance on the predicted traffic cone feature map, g y It is the vertical index value of the target pixel point corresponding to each traffic cone line instance on the predicted traffic cone line feature map.

[0117] Then, according to the size of the road image to be detected and the size of the traffic cone feature map, the pixel coordinates of the target point of the traffic cone instance on the traffic cone feature map are converted to pixel coordinates on the road image to be detected. Specifically, the horizontal coordinate sampling multiple S can be determined by the size of the road image to be detected and the size of the traffic cone feature map. x and the vertical sampling multiple S y ; The pixel coordinates (x' g ,y' g ) in the horizontal coordinate value x' g Multiply by the horizontal axis sampling multiple S x , the pixel coordinates (x' g ,y' g ) in the vertical coordinate value y' g Multiply by the vertical coordinate sampling multiple S y , thus obtaining the pixel coordinates (x' c ,y' c ). It is understandable that the size of the annotated road image input into the preset convolutional neural network during training is consistent with the size of the road image to be detected input into the preset detection model during prediction.

[0118] Finally, the pixel coordinates of the sampling points other than the target point in the traffic cone instance on the road image to be detected are calculated based on the predicted sampling point offset and the pixel coordinates of the target point in the traffic cone instance on the traffic cone feature map scale to obtain the traffic cone sequence. Specifically, for each traffic cone instance, if the calculated sampling point offset is directly used as the network learning target when training the preset convolutional neural network, the sampling point offset prediction value (O') output by the sampling point offset detection head is directly calculated. xi ,O' yi ) and the pixel coordinates (x') of the corresponding target point on the scale of the traffic cone feature map g ,y' g ) are summed to obtain the pixel coordinates (x' i ,y' i ), that is, x' i =O' xi +x' g , y' i =O' yi +y' g If the normalized sampling point offset is used as the network learning target when training the preset convolutional neural network, the sampling point offset prediction value (O' xi,O' yi ) to perform the normalization inverse operation to obtain the sampling point offset prediction value before normalization, and then the sampling point offset prediction value before normalization and the pixel coordinate (x' g ,y' g ) are summed to obtain the pixel coordinates (x' i ,y' i ). Among them, if the width and height of the input image are used to normalize the sampling point offset during training, the sampling point offset prediction value is inversely normalized based on the width and height of the input image during prediction; if the traffic cone line length is used to normalize the sampling point offset during training, the traffic cone line length obtained based on the traffic cone line length prediction value is inversely normalized based on the sampling point offset prediction value during prediction.

[0119] The pixel coordinates (x' i ,y' i ) in the horizontal coordinate value x' i Multiply by the horizontal axis sampling multiple S x , the pixel coordinates (x' i ,y' i ) in the vertical coordinate value y' i Multiply by the vertical coordinate sampling multiple S y , thereby obtaining the pixel coordinates of the sampling points other than the target point in each traffic cone line instance on the input road image to be detected, thereby obtaining a traffic cone line sequence. When visualizing the traffic cone line sequence, the points in each traffic cone line sequence are connected in sequence to obtain the line representation corresponding to each traffic cone line sequence.

[0120] End-to-end traffic cone detection is performed on the road image to be detected through the preset detection model. The detection process is simple and the detection speed is fast. After obtaining the traffic cone detection result, the traffic cone detection result is simply parsed to quickly restore the instance-level traffic cone representation. There is no need for complex post-processing, which is time-saving and has high real-time performance.

[0121] Furthermore, after the traffic cone line sequence is obtained, it can be further processed to filter out low-quality traffic cone line sequences. Specifically, the traffic cone line sequences can be filtered based on the distance between two traffic cone lines.

[0122] Specifically, first, the distance between each traffic cone line in the traffic cone line sequence is calculated.

[0123] In one embodiment, the horizontal distance between each traffic cone line can be calculated according to the coordinates of each point in the traffic cone line sequence, and the horizontal distance between each traffic cone line is used as the distance between each traffic cone line. Assume that the traffic cone line sequence corresponding to traffic cone line a is The traffic cone line sequence corresponding to traffic cone line b is in, Traffic cone sequence I a The i-th sampling point in Traffic cone sequence I b The i-th sampling point in a e Traffic cone sequence I a The end point index of b e Traffic cone sequence I b The end index of a s Traffic cone sequence I a The starting index of b s Traffic cone sequence I b The starting point index of the traffic cone line a and the horizontal distance D between the traffic cone line b can be calculated using the following formula:

[0124]

[0125] In the formula, e = min(a e ,b e ), s = max(a s ,b s ), min() is the minimum function, and max() is the maximum function.

[0126] In another embodiment, the angle between each traffic cone line and the horizontal line can be fitted according to the traffic cone line sequence; and the distance weight coefficient between each traffic cone line is calculated by the angle between each traffic cone line and the horizontal line. Assume that the traffic cone line sequence I corresponding to the fitted traffic cone line a a The angle with the horizontal line is θ a , fitting the traffic cone line sequence I corresponding to the traffic cone line b b The angle with the horizontal line is θ b , the distance weight coefficient w between traffic cone line a and traffic cone line b a,b = sin(max(θ a ,θ b )) / cos(|θ a -θ b| / 2). The horizontal distances between the traffic cones are weighted by the distance weight coefficients between the traffic cones to obtain the distances between the traffic cones. It is understandable that the angles between the points in each traffic cone sequence and the horizontal line can also be fitted, and the distance weight coefficients of the two points in the two traffic cone sequences can be calculated by the angles between the two points and the horizontal line. The horizontal distances between the two points (i.e., the difference in the horizontal coordinates of the two points) are weighted by the distance weight coefficients of the two points to obtain the horizontal weighted distance. Finally, the horizontal weighted distances of all the points in the two traffic cone sequences are averaged to obtain the distance between the two traffic cone sequences.

[0127] In the embodiment of the present application, the angle between each traffic cone line sequence and the horizontal line is fitted, and then the distance weight coefficient is calculated by the angle between the two traffic cone line sequences and the horizontal line. The horizontal distance between the two traffic cone line sequences is weighted by the distance weight coefficient, so that the calculated distance between the traffic cone line sequences is independent of the angle between the two traffic cone line sequences, thereby avoiding the influence of the angle relationship on the distance measurement result between the traffic cone lines.

[0128] When the distance between two traffic cones is less than the preset distance threshold, the traffic cone sequence with a smaller probability of traffic cone category is removed to obtain the final traffic cone sequence, and the removed traffic cone sequence is regarded as a low-quality traffic cone sequence; when the distance between two traffic cones is greater than or equal to the preset distance threshold, the traffic cone sequences corresponding to the two traffic cones are retained to obtain the final traffic cone sequence. The preset distance threshold can be set specifically according to actual conditions and is not specifically limited here.

[0129] After obtaining the final traffic cone line sequence, when the final traffic cone line sequence needs to be visualized, the points in each final traffic cone line sequence are connected in order to obtain the line representation corresponding to each final traffic cone line sequence. In addition, in the actual application scenario of autonomous driving, it is necessary to convert the coordinates of each point in the final traffic cone line sequence from two dimensions to three dimensions, so as to obtain the traffic cone line sequence in the physical coordinate system. Converting the coordinates of each point in the final traffic cone line sequence from two dimensions to three dimensions belongs to the prior art and will not be repeated here.

[0130] In the embodiment of the present application, traffic cone lines are drawn with the intersection points of each traffic cone and the ground in the road image to be annotated as the annotation points, that is, the grouped traffic cones are annotated as an independent instance, which helps to ensure the high-order semantic consistency of the traffic cone connection relationship, thereby obtaining a specific representation of the traffic cone connection relationship; then, based on the annotated road image and the ordered point set of traffic cone lines obtained by annotation, traffic cone line detection is performed on the road image to be detected, and the grouped traffic cone lines can be detected as an instance as a whole, which is not prone to fluctuations, and can effectively improve the consistency, continuity and robustness of the traffic cone connection results, thereby improving the technical problems of the prior art that the associated traffic cones are connected into groups based on preset rules, it is difficult to obtain consistent results, the connection results are prone to fluctuations, and the robustness is low.

[0131] The above is an embodiment of a traffic cone line detection method provided by the present application, and the following is an embodiment of a traffic cone line detection device provided by the present application.

[0132] Please refer to Figure 4 , a traffic cone line detection device provided in an embodiment of the present application includes:

[0133] An acquisition unit, used for acquiring a road image to be annotated, wherein the road image to be annotated includes at least one group of traffic cones;

[0134] A labeling unit, used for taking the intersection points of each traffic cone and the ground in the road image to be labeled as labeling points, drawing traffic cone lines on the road image to be labeled based on the labeling points to obtain a labeled road image, and generating a traffic cone line ordered point set of the traffic cone lines;

[0135] The detection unit is used to process the road image to be detected based on the labeled road image and the ordered point set of traffic cones to obtain the traffic cone detection result.

[0136] As a further improvement, the detection unit is specifically used for:

[0137] Calculate the target offset of each traffic cone line through the annotated road image and the ordered point set of traffic cone lines;

[0138] Using the annotated road image as input data, the preset convolutional neural network is trained with traffic cone category, target offset and traffic cone length as learning objectives to obtain a preset detection model. The traffic cone category and traffic cone length are obtained through annotation.

[0139] Traffic cone line detection is performed on the road image to be detected through the preset detection model to obtain the traffic cone line detection result.

[0140] As a further improvement, the traffic cone line detection device in the embodiment of the present application further includes: a screening unit, which is used to:

[0141] Calculate the distance between each traffic cone line in the traffic cone line sequence;

[0142] When the distance between two traffic cones is less than a preset distance threshold, the traffic cone sequence with a smaller traffic cone category probability is eliminated to obtain the final traffic cone sequence.

[0143] As a further improvement, the detection unit is specifically used for:

[0144] The connecting line of the marked points in the ordered point set of the traffic cone line is extended to the segmentation mark of the preset width by pixels to obtain the target label of the marked road image;

[0145] The preset convolutional neural network is trained with the labeled road image as input data and the target label as the network learning target to obtain the preset segmentation model;

[0146] Use the preset segmentation model to perform traffic cone segmentation on the road image to be detected, and output the traffic cone segmentation result;

[0147] Cluster the traffic cone pixel points in the traffic cone segmentation result to obtain several traffic cone instances;

[0148] The traffic cone pixel points in each traffic cone instance are fitted to obtain the detected traffic cone.

[0149] As a further improvement, the method further comprises: a preprocessing unit, which is used to:

[0150] Determine whether the number of marked points in the ordered point set of the traffic cone line is equal to the number of preset sampling points;

[0151] If not, then connect the two adjacent marked points in the ordered point set of the traffic cone line in sequence to obtain a line segment vector group;

[0152] Calculate the curvature radius of the arc corresponding to two adjacent line segment vectors in the line segment vector group;

[0153] The ordered point set of the traffic cone line is processed according to the curvature radius so that the final number of labeled points of the ordered point set of the traffic cone line is equal to the preset number of sampling points.

[0154] In the embodiment of the present application, traffic cone lines are drawn with the intersection points of each traffic cone and the ground in the road image to be annotated as the annotation points, that is, the grouped traffic cones are annotated as an independent instance, which helps to ensure the high-order semantic consistency of the traffic cone connection relationship, thereby obtaining a specific representation of the traffic cone connection relationship; then, based on the annotated road image and the ordered point set of traffic cone lines obtained by annotation, traffic cone line detection is performed on the road image to be detected, and the grouped traffic cone lines can be detected as an instance as a whole, which is not prone to fluctuations, and can effectively improve the consistency, continuity and robustness of the traffic cone connection results, thereby improving the technical problems of the prior art that the associated traffic cones are connected into groups based on preset rules, it is difficult to obtain consistent results, the connection results are prone to fluctuations, and the robustness is low.

[0155] Please refer to Figure 5 , the embodiment of the present application also provides a traffic cone line detection device, the device includes a processor and a memory;

[0156] The memory is used to store the program code and transmit the program code to the processor;

[0157] The processor is used to execute the traffic cone line detection method in the aforementioned method embodiment according to the instructions in the program code.

[0158] The embodiment of the present application further provides a computer-readable storage medium, which is used to store program code. When the program code is executed by a processor, the traffic cone line detection method in the aforementioned method embodiment is implemented.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0160] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0161] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0162] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0165] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0166] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A traffic cone detection method, characterized in that: include: Acquire a road image to be annotated, wherein the road image to be annotated includes at least one group of traffic cones; Taking the intersection points of each traffic cone and the ground in the road image to be labeled as the labeling points, drawing traffic cone lines on the road image to be labeled based on the labeling points to obtain a labeled road image, and generating a traffic cone line ordered point set of the traffic cone line; The road image to be detected is processed based on the labeled road image and the ordered point set of traffic cones to obtain a traffic cone detection result, including: Calculating the target offset of each traffic cone line by using the labeled road image and the ordered point set of the traffic cone line; Taking the annotated road image as input data, training a preset convolutional neural network with the traffic cone line category, the target offset and the traffic cone line length as learning targets to obtain a preset detection model, wherein the traffic cone line category and the traffic cone line length are obtained by annotation; Performing traffic cone line detection on the road image to be detected by using the preset detection model to obtain a traffic cone line detection result; Or, a line connecting the marked points in the ordered point set of the traffic cone line is extended to a segmented mark with a preset width by pixels, so as to obtain a target label of the marked road image; Taking the annotated road image as input data and the target label as a network learning target, a preset convolutional neural network is trained to obtain a preset segmentation model; Performing traffic cone segmentation on the road image to be detected by using the preset segmentation model, and outputting the traffic cone segmentation result; Clustering the traffic cone pixel points in the traffic cone segmentation result to obtain a number of traffic cone instances; The traffic cone line pixels in each of the traffic cone line instances are fitted to obtain the detected traffic cone line.

2. The traffic cone detection method according to claim 1, characterized in that: The calculating the target offset of each traffic cone line by using the labeled road image and the ordered point set of the traffic cone line includes: Taking a point that is half of the cumulative length of each traffic cone in the traffic cone ordered point set as a target point, and calculating the pixel coordinates of the target point projected into the traffic cone feature map according to the size of the annotated road image and the size of the traffic cone feature map extracted by the preset convolutional neural network; Calculate the target point offset of each traffic cone line through the pixel coordinates of the target point of each traffic cone line in the traffic cone line feature map; Calculate the offsets of the sampling points other than the target point in each traffic cone line in the traffic cone line ordered point set and the corresponding target point to obtain the sampling point offsets; The target point offset and the sampling point offset of each traffic cone line are used as the target offset, or the sampling point offset of each traffic cone line is normalized, and the target point offset and the normalized sampling point offset of each traffic cone line are used as the target offset.

3. The traffic cone detection method according to claim 1, characterized in that: The method of performing traffic cone line detection on the road image to be detected by using the preset detection model to obtain the traffic cone line detection result includes: Performing traffic cone detection on the road image to be detected by using the preset detection model, and outputting the traffic cone existence probability, traffic cone category probability, target offset prediction value and traffic cone length prediction value of each pixel point in the traffic cone feature map extracted by the preset detection model; Extracting target pixel points whose existence probability of the traffic cone is higher than an existence probability threshold from the traffic cone feature map output by the preset detection model; Determine the traffic cone category of the target pixel point according to the traffic cone category probability of the target pixel point, and extract traffic cone instances whose traffic cone category probability is higher than a category probability threshold from the target pixel point; Obtaining the length of the traffic cone line instance by using the traffic cone line length prediction value; The pixel coordinates of each point in the traffic cone line instance in the image to be detected are obtained according to the target offset prediction value to obtain a traffic cone line sequence.

4. The traffic cone detection method according to claim 3, characterized in that: The method further comprises: Calculating the distance between each traffic cone line in the traffic cone line sequence; When the distance between two traffic cones is less than a preset distance threshold, the traffic cone sequence with a smaller probability of the traffic cone category is eliminated to obtain a final traffic cone sequence.

5. The traffic cone detection method according to claim 1, characterized in that: The method further includes: processing the road image to be detected based on the labeled road image and the ordered point set of traffic cones to obtain a traffic cone detection result; Determine whether the number of marked points in the ordered point set of the traffic cone line is equal to the number of preset sampling points; If not, then sequentially connect two adjacent marked points in the ordered point set of the traffic cone line to obtain a line segment vector group; Calculating the curvature radius of the arc corresponding to two adjacent line segment vectors in the line segment vector group; The traffic cone line ordered point set is processed according to the curvature radius so that the final number of marked points of the traffic cone line ordered point set is equal to the preset number of sampling points.

6. The traffic cone detection method according to claim 5, characterized in that: The processing of the ordered point set of the traffic cone line according to the curvature radius so that the final number of annotation points of the ordered point set of the traffic cone line is equal to the preset number of sampling points includes: When the number of the marked points of the ordered point set of the traffic cone line is less than the preset number of sampling points, the first number and the second number are calculated according to the difference between the preset number of sampling points and the number of the marked points of the ordered point set of the traffic cone line and the number of line segment vectors; Sampling a second number plus one sampling point for a first line segment vector of two adjacent line segment vectors corresponding to the first number of minimum curvature radii, and sampling the second number of sampling points for each of the remaining line segment vectors, so that the final number of marked points of the ordered point set of the traffic cone line is equal to the preset number of sampling points, and the final marked points include the original marked points and the newly added sampling points; When the number of the marked points of the ordered point set of the traffic cone line is greater than the preset number of sampling points, calculating the difference between the number of the marked points of the ordered point set of the traffic cone line and the preset number of sampling points to obtain a third number; The end point of the first line segment vector of two adjacent line segment vectors corresponding to the third number of maximum curvature radii is deleted, so that the final number of marked points of the ordered point set of the traffic cone line is equal to the preset number of sampling points.

7. A traffic cone detection device, characterized in that: include: An acquisition unit, configured to acquire a road image to be annotated, wherein the road image to be annotated includes at least one group of traffic cones; a labeling unit, configured to use the intersection points of each traffic cone and the ground in the road image to be labeled as labeling points, draw traffic cone lines on the road image to be labeled based on the labeling points to obtain a labeled road image, and generate a traffic cone line ordered point set of the traffic cone line; A detection unit, configured to process the road image to be detected based on the annotated road image and the ordered point set of traffic cones to obtain a traffic cone detection result; The detection unit is specifically used for: Calculating the target offset of each traffic cone line by using the labeled road image and the ordered point set of the traffic cone line; Taking the annotated road image as input data, training a preset convolutional neural network with the traffic cone line category, the target offset and the traffic cone line length as learning targets to obtain a preset detection model, wherein the traffic cone line category and the traffic cone line length are obtained by annotation; Performing traffic cone line detection on the road image to be detected by using the preset detection model to obtain a traffic cone line detection result; Or, a line connecting the marked points in the ordered point set of the traffic cone line is extended to a segmented mark with a preset width by pixels, so as to obtain a target label of the marked road image; Taking the annotated road image as input data and the target label as a network learning target, a preset convolutional neural network is trained to obtain a preset segmentation model; Performing traffic cone segmentation on the road image to be detected by using the preset segmentation model, and outputting the traffic cone segmentation result; Clustering the traffic cone pixel points in the traffic cone segmentation result to obtain a number of traffic cone instances; The traffic cone line pixels in each of the traffic cone line instances are fitted to obtain the detected traffic cone line.

8. A traffic cone detection device, characterized in that: The device comprises a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the traffic cone line detection method according to any one of claims 1-6 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and when the program codes are executed by a processor, the traffic cone line detection method according to any one of claims 1 to 6 is implemented.