A road marking detection method, device, equipment and storage medium

Abstract

The present application discloses a road surface marking detection method, device, equipment and storage medium. The method comprises: obtaining a first road surface marking detection result obtained by a first detection model according to a road surface marking feature map extracted from a road image to be detected; obtaining input data of a second detection model from the road surface marking feature map through the first road surface marking detection result; inputting the input data into the second detection model to perform road surface marking detection, and obtaining a second road surface marking detection result of the road image to be detected, thereby improving the technical problem of low detection accuracy in the prior art.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a road marking detection method, device, equipment and storage medium. Background Art

[0002] Road marking detection is an important task in the field of autonomous driving. Most existing road marking detection methods are implemented using a single-stage deep learning method. This method has a significant impact on the prediction accuracy of road marking detection due to the insufficient receptive field of the fully convolutional neural network. In addition, this method requires global classification to distinguish effective instances, which is difficult. For linear road markings, it is difficult to accurately distinguish multiple parts of the line, resulting in low detection accuracy. Summary of the invention

[0003] The present application provides a road marking detection method, device, equipment and storage medium, which are used to improve the technical problem of low detection accuracy in the prior art.

[0004] In view of this, the first aspect of the present application provides a road surface marking detection method, comprising:

[0005] Obtaining a first road surface marking detection result obtained by a first detection model according to a road surface marking feature map extracted from a road image to be detected;

[0006] Obtaining input data of a second detection model from the road surface marking feature map according to the first road surface marking detection result;

[0007] The input data is input into the second detection model to perform road surface marking detection to obtain a second road surface marking detection result.

[0008] Optionally, the acquiring input data of a second detection model from the road surface marking feature map using the first road surface marking detection result includes:

[0009] parsing the first road surface marking detection result to obtain a first road surface marking sequence;

[0010] rounding the point set coordinates of each road surface marking in the first road surface marking sequence to obtain a grid index of each point in the road surface marking feature map;

[0011] Corresponding feature vectors are extracted from the road surface marking feature map according to the grid index and feature vector concatenation is performed to obtain input data of the second detection model.

[0012] Optionally, the first road surface marking detection result includes a road surface marking existence probability, a road surface marking category probability and a target offset prediction value of each pixel point in the road surface marking feature map;

[0013] The step of parsing the first road surface marking detection result to obtain a first road surface marking sequence includes:

[0014] Extracting target pixel points whose existence probability of the road surface marking is higher than an existence probability threshold from the road surface marking feature map;

[0015] Determine the road marking category of the target pixel point according to the road marking category probability of the target pixel point, and extract road marking instances whose road marking category probability is higher than a category probability threshold from the target pixel point;

[0016] According to the target offset prediction value, the position coordinates of each point in the road surface marking instance in the road surface marking feature map are obtained to obtain a first road surface marking sequence.

[0017] Optionally, the target offset prediction value includes a target point offset prediction value and a sampling point offset prediction value;

[0018] The step of acquiring the position coordinates of each point in the pavement marking instance in the pavement marking feature map according to the target offset prediction value to obtain a first pavement marking sequence comprises:

[0019] Acquire the position coordinates of the target point of the pavement marking instance in the pavement marking feature map according to the target point offset;

[0020] The position coordinates of the sampling points other than the target point in the pavement marking instance in the pavement marking feature map are calculated according to the sampling point offset prediction values ​​and the position coordinates of the target point of the pavement marking instance in the pavement marking feature map to obtain a first pavement marking sequence.

[0021] Optionally, the method further includes:

[0022] parsing the second road surface marking detection result to obtain a second road surface marking sequence;

[0023] calculating the distance between each pavement marking in the second pavement marking sequence;

[0024] When the distance between two road surface markings is less than a preset distance threshold, the road surface markings with a smaller road surface marking category probability in the second road surface marking sequence are removed to obtain a final road surface marking sequence.

[0025] Optionally, calculating the distance between each road surface marking in the second road surface marking sequence includes:

[0026] Calculating the horizontal distance between each road surface marking according to the position coordinates of each point in the second road surface marking sequence;

[0027] Taking the horizontal distance between each road surface marking as the distance between each road surface marking;

[0028] or, fitting the angle between each road surface marking and the horizontal line according to the second road surface marking sequence, and calculating the distance weight coefficient between each road surface marking according to the angle between each road surface marking and the horizontal line;

[0029] The horizontal distances between the road surface markings are weighted by the distance weight coefficients between the road surface markings to obtain the distances between the road surface markings.

[0030] A second aspect of the present application provides a road surface marking detection device, comprising:

[0031] A first acquisition unit is used to acquire a first road surface marking detection result obtained by a first detection model according to a road surface marking feature map extracted from a road image to be detected;

[0032] a second acquisition unit, configured to acquire input data of a second detection model from the road surface marking feature map according to the first road surface marking detection result;

[0033] The detection unit is used to input the input data into the second detection model to perform road surface marking detection to obtain a second road surface marking detection result.

[0034] Optionally, the second acquiring unit is specifically configured to:

[0035] parsing the first road surface marking detection result to obtain a first road surface marking sequence;

[0036] rounding the point set coordinates of each road surface marking in the first road surface marking sequence to obtain a grid index of each point in the road surface marking feature map;

[0037] Corresponding feature vectors are extracted from the road surface marking feature map according to the grid index and feature vector concatenation is performed to obtain input data of the second detection model.

[0038] A third aspect of the present application provides a road surface marking detection device, the device comprising a processor and a memory;

[0039] The memory is used to store program code and transmit the program code to the processor;

[0040] The processor is used to execute any one of the road surface marking detection methods of the first aspect according to the instructions in the program code.

[0041] A fourth aspect of the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, and when the program code is executed by a processor, it implements any road surface marking detection method described in the first aspect.

[0042] It can be seen from the above technical solutions that this application has the following advantages:

[0043] The present application provides a road surface marking detection method, comprising: obtaining a first road surface marking detection result obtained by a first detection model according to a road surface marking feature map extracted from a road image to be detected; obtaining input data of a second detection model from the road surface marking feature map through the first road surface marking detection result; inputting the input data into the second detection model to perform road surface marking detection, and obtaining a second road surface marking detection result of the road image to be detected.

[0044] In the present application, after obtaining the first road surface marking detection result of the first detection model, the first road surface marking detection result is used as a candidate road surface marking instance, and feature extraction is performed on the road surface marking feature map extracted from the first detection model through the first road surface marking detection result to obtain input data for the second detection model, and then the second detection model is used to fine-tune the detection result of each candidate road surface marking instance to obtain a second road surface marking detection result with higher accuracy, thereby improving the technical problem of low detection accuracy in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 A schematic flow chart of a road surface marking detection method provided in an embodiment of the present application;

[0047] Figure 2 Another schematic flow chart of a road surface marking detection method provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of the structure of a road surface marking detection device provided in an embodiment of the present application;

[0049] Figure 4 A schematic structural diagram of a road surface marking detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The present application provides a road marking detection method, device, equipment and storage medium, which are used to improve the technical problem of low detection accuracy in the prior art.

[0051] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] For easier understanding, see Figure 1 , the present application embodiment provides a road surface marking detection method, comprising:

[0053] Step 101: Obtain a first road surface marking detection result obtained by a first detection model based on a road surface marking feature map extracted from a road image to be detected.

[0054] The present application embodiment takes into account that the existing road surface marking detection methods are all based on a single-stage detection method for road surface marking detection, which has the technical problem of low detection accuracy. In order to improve this problem, the present application embodiment further processes the detection results obtained by the single-stage detection method to obtain a more accurate road surface marking detection result.

[0055] The road image can be collected by the vehicle-side sensor. Considering that the collected original road image is generally large, the original road image can be cropped, scaled, etc. to obtain a road image of a preset size; then it can be further normalized, and the normalized road image is used as the road image to be detected. The acquired road image to be detected is input into the first detection model, and the road surface marking feature map of the road image to be detected is extracted by the first detection model. The road surface marking detection is performed according to the road surface marking feature map by the first detection model to obtain the first road surface marking detection result. Among them, the first detection model is a trained convolutional neural network model. The road surface marking in the embodiment of the present application is a linear target, which can be a lane line, a road edge, other road marking lines or a rod-shaped object. The first detection model can be configured using an existing detection model configuration method, for example, trained by an existing lane line detection model and a road surface marking line detection model training method. The first detection model can also be configured using the configuration method in the embodiment of the present application.

[0056] Specifically, the configuration process of the first detection model in the present application may be:

[0057] A1. Obtain a road surface marking image and an ordered point set of a preset number of sampling points corresponding to the road surface marking image;

[0058] The road surface marking image includes at least one road surface marking, and a large number of road surface marking images and corresponding road surface marking ordered point sets can be obtained from a public database, and each road surface marking image is an image of a preset size. Considering that when training a model, the preset number of sampling points set according to actual conditions may be different from the number of annotated points in the road surface marking ordered point set pre-annotated in the database, when the number of annotated points is different from the preset number of sampling points, the annotated road surface marking ordered point set can be preprocessed to obtain an ordered point set with the preset number of sampling points.

[0059] In one embodiment, when the number of annotated points in the pavement marking ordered point set is less than the preset number of sampling points, f sampling points can be randomly interpolated in the pavement marking ordered point set according to the absolute difference f between the number of annotated points and the preset number of sampling points, so that the final number of annotated points in the pavement marking ordered point set is equal to the preset number of sampling points. When the number of annotated points in the pavement marking ordered point set is greater than the preset number of sampling points, f annotated points can be randomly deleted in the pavement marking ordered point set according to the absolute difference f between the number of annotated points and the preset number of sampling points, so that the final number of annotated points in the pavement marking ordered point set is equal to the preset number of sampling points. It should be noted that when deleting the original annotated points, the starting point and the end point in the pavement marking ordered point set are generally retained.

[0060] In another embodiment, the ordered point set may be interpolated or some points may be deleted by using the curvature radius to obtain an ordered point set with a preset number of sampling points with minimal precision loss. Specifically, the process of obtaining an ordered point set with a preset number of sampling points corresponding to the road surface marking image may be:

[0061] A10, obtaining an ordered point set of annotated road surface markings corresponding to the road surface marking image, and sequentially connecting two adjacent annotated points on the road surface markings according to the ordered point set of road surface markings to obtain a line segment vector group;

[0062] After obtaining the annotated road marking ordered point set, vectorize the line segments formed by two adjacent annotation points in the road marking ordered point set. Specifically, considering that in the road image collected by the vehicle, the road markings close to the vehicle (such as the lane lines near the bottom of the image) are clearer than the road markings far away from the vehicle, the reliability and uniformity of the annotations are better. Therefore, the embodiment of the present application preferably uses the annotation point closest to the lower edge of the image as the starting point of the road marking ordered point set, takes the two adjacent annotation points as endpoints, and sequentially connects the two adjacent annotation points to generate a line segment vector group. Each line segment vector in the line segment vector group can be represented in sequence as L1, L2, ..., L n, where n is the number of line segment vectors. Assume there are three marked points A(x1,y1), B(x2,y2), and C(x3,y3), where point A is the starting point and point C is the end point. Connect two adjacent marked points in sequence, and the resulting line segment vector group includes line segment vectors and the line segment vector Line segment vector It can be expressed as L1(x2-x1,y2-y1), the line segment vector It can be expressed as L2(x3-x2,y3-y2).

[0063] A11. Calculate the curvature radius of the arc corresponding to two adjacent line segment vectors in the line segment vector group;

[0064] After obtaining the line segment vector group, the angle between two adjacent line segment vectors in the line segment vector group is calculated. In one embodiment, the cosine value of the angle between two adjacent line segment vectors in the line segment vector group can be calculated based on the inner product and the modulus of the two adjacent line segment vectors in the line segment vector group, such as the line segment vector L n-1 With the line segment vector L n The cosine of the angle is cosθ n-1 =(L n-1 ·L n ) / (|L n-1 ||L n |); The angle between two adjacent line segment vectors is calculated by the cosine value of the angle between the two adjacent line segment vectors, such as the line segment vector L n-1 With the line segment vector L n The angle is θ n-1 =arccos((L n-1 ·L n ) / (|L n-1 ||L n |)), where n>2.

[0065] In another embodiment, the sine value of the angle between two adjacent line segment vectors can be calculated according to the cross product and the modulus of the two adjacent line segment vectors in the line segment vector group, such as the line segment vector L n-1 With the line segment vector L n The sine of the angle is sinθ n-1 =|L n-1 ×L n | / (|L n-1 ||L n |); The angle between two adjacent line segment vectors is calculated by the sine value of the angle between the two adjacent line segment vectors, such as the line segment vector L n-1 With the line segment vector L n The angle θ n-1 for:

[0066]

[0067] In another embodiment, the cosine value of the angle between two adjacent line segment vectors can also be calculated based on the inner product and the modulus of two adjacent line segment vectors in the line segment vector group, such as the line segment vector L n-1 With the line segment vector L n The cosine of the angle is cosθ n-1 =(L n-1 ·L n ) / (|L n-1 ||L n |); Calculate the sine of the angle between two adjacent line segment vectors based on the cross product and modulus of the two adjacent line segment vectors, such as the line segment vector L n-1 With the line segment vector L n The sine of the angle is sinθ n-1 =|L n-1 ×L n | / (|L n-1 ||L n |); Calculate the tangent value of the angle between two adjacent line segment vectors by using the cosine value and sine value of the angle between two adjacent line segment vectors, such as line segment vector L n-1 With the line segment vector L n The tangent value of the angle is tanθ n-1 = sinθ n-1 / cosθ n-1 ; The angle between two adjacent line segment vectors is calculated by the tangent value of the angle between the two adjacent line segment vectors, such as the line segment vector L n-1 With the line segment vector L n The angle is θ n-1 =arctan(sinθ n-1 / cosθ n-1 ). Of course, we can also directly calculate the cross product and inner product of two adjacent line segment vectors in the line segment vector group, and then use the formula θ n-1 =arctan(|L n-1 ×L n | / L n-1 ·L n ) calculates the angle between two adjacent line segment vectors.

[0068] The angle calculated through the above process has a value range of [0,180°]. After the angle between two adjacent line segment vectors is calculated, the arc corresponding to the two adjacent line segment vectors is determined by taking the line segments corresponding to the two adjacent line segment vectors as the tangents of the arc start point and the arc end point. The curvature radius of the arc corresponding to the two adjacent line segment vectors can be calculated based on the module length and the angle between the two adjacent line segment vectors. Specifically, a smooth arc is constructed from the starting point of the previous line segment vector of two adjacent line segment vectors to the end point of the next line segment vector. The line segments corresponding to the two adjacent line segment vectors are used as the tangent of the arc starting point and the tangent of the arc end point to determine the only arc corresponding to the two adjacent line segment vectors. It is known that the curvature radius r of the arc is equal to the ratio of the arc length of the arc to the center angle corresponding to the arc (that is, the angle θ between the two adjacent line segment vectors corresponding to the arc). In the embodiment of the present application, the sum of the module lengths of the two adjacent line segment vectors is used to approximate the arc length of the arc corresponding to the two adjacent line segment vectors, and then the curvature radius of the arc corresponding to the two adjacent line segment vectors is calculated, that is, the curvature radius of the arc corresponding to the two adjacent line segment vectors L n-1 With L n The corresponding arc has a curvature radius of r n-1 =(|L n-1 |+|L n ) / θ n-1 . After calculating the curvature radius of the arc corresponding to two adjacent line segment vectors in the line segment vector group, the corresponding relationship between each curvature radius and the two adjacent line segment vectors can be associated. Assuming that there are three line segment vectors L1, L2, and L3 in the line segment vector group, the adjacent line segment vectors L1 and L2 can determine the unique arc L1L2. According to the module length and angle of the line segment vectors L1 and L2, the curvature radius r1 of the corresponding arc L1L2 can be calculated, that is, there is a corresponding relationship between the curvature radius r1 and the arc L1L2, and there is a corresponding relationship between the arc L1L2 and the adjacent line segment vectors L1 and L2. Therefore, the curvature radius r1 can be associated with the adjacent line segment vectors L1 and L2, that is, the adjacent line segment vector corresponding to the curvature radius r1 are line segment vectors L1 and L2; similarly, adjacent line segment vectors L2 and L3 can determine the unique arc L2L3, and the curvature radius r2 of the corresponding arc L2L3 can be calculated according to the module length and the included angle of the line segment vectors L2 and L3, that is, there is a corresponding relationship between the curvature radius r2 and the arc L2L3, and there is a corresponding relationship between the arc L2L3 and the adjacent line segment vectors L2 and L3, so that the curvature radius r2 can be associated with the adjacent line segment vectors L2 and L3, that is, the adjacent line segment vectors corresponding to the curvature radius r2 are the line segment vectors L2 and L3.

[0069] A12. When the number of marking points of the road surface marking is not equal to the preset number of sampling points, the road surface marking is processed according to the curvature radius so that the final number of marking points of the road surface marking is equal to the preset number of sampling points, thereby obtaining an ordered point set of the preset number of sampling points corresponding to the road surface marking image.

[0070] When the number of annotated points in the pavement marking ordered point set is equal to the number of preset sampling points, all the annotated points in the pavement marking ordered point set are retained, wherein the number of preset sampling points can be data configured based on actual experience or experimental conditions. When the number of annotated points in the pavement marking ordered point set is not equal to the number of preset sampling points actually required, the pavement marking ordered point set can be processed according to the curvature radius so that the final number of annotated points in the pavement marking ordered point set is equal to the number of preset sampling points.

[0071] When the number of annotation points of the road surface marking ordered point set is less than the preset number of sampling points, the road surface marking ordered point set is interpolated according to the curvature radius so that the final number of annotation points of the road surface marking ordered point set is equal to the preset number of sampling points, and the final annotation points include the original annotation points and the newly added sampling points.

[0072] Specifically, when the number of annotated points N of the road surface marking ordered point set is less than the preset number of sampling points M, the first number m=(MN) / n and the second number k=(MN)%n can be calculated according to the difference MN between the preset number of sampling points and the number of annotated points of the road surface marking ordered point set and the number of line segment vectors n, where m is the quotient of MN divided by n, k is the remainder of MN divided by n, and n=N-1; the first line segment vector of the two adjacent line segment vectors corresponding to the first number of minimum curvature radii is interpolated by the second number plus one sampling point, and the second number of sampling points is sampled for each line segment vector in the remaining line segment vectors (i.e., the uninterpolated line segment vectors), so that the final number of annotated points of the road surface marking ordered point set is equal to the preset number of sampling points. All curvature radii can be sorted in ascending order (or descending order) according to size, and then the first line segment vector of the two adjacent line segment vectors corresponding to the first k minimum curvature radii after ascending order can be uniformly interpolated by m+1 points, and each line segment vector in the remaining nk line segment vectors can be uniformly interpolated by m points. The smaller the radius of curvature, the greater the curvature of the line segment corresponding to the radius of curvature. When the number of annotation points in the ordered point set of road markings is less than the preset number of sampling points, it means that more annotation points are needed. The embodiment of the present application, while retaining the original annotation points, preferentially interpolates in the curved part of the line segment to fill the gap with the preset number of sampling points, without losing accuracy to the original ordered point set.

[0073] When the number of annotated points in the road surface marking ordered point set is greater than the preset number of sampling points, some of the annotated points in the road surface marking ordered point set may be deleted according to the curvature radius, so that the final number of annotated points in the road surface marking ordered point set is equal to the preset number of sampling points. Specifically, when the number of annotated points N in the road surface marking ordered point set is greater than the preset number of sampling points M, the difference NM between the number of annotated points in the road surface marking ordered point set and the preset number of sampling points is calculated to obtain a third number NM; and the end point of the line segment of the first line segment vector in two adjacent line segment vectors corresponding to the third number NM of maximum curvature radii is deleted, so that the final number of annotated points in the road surface marking ordered point set is equal to the preset number of sampling points. When the number of annotated points N in the road surface marking ordered point set is greater than the preset number of sampling points M, it means that NM annotated points in the road surface marking ordered point set need to be deleted. All curvature radii can be sorted in descending order (or ascending order) according to their size, and then the end point of the first line segment vector in the two adjacent line segment vectors corresponding to the first NM largest curvature radii after descending order is deleted. At this time, the starting point and end point of the road surface marking ordered point set will always be retained, that is, the deleted annotated point is the middle point of the line segment. The larger the curvature radius, the closer the corresponding line segment is to a straight line. When the number of annotated points needs to be reduced, the middle point of the line segment closest to the straight line can be deleted first, so as to keep the original shape of the road surface marking as much as possible with a smaller error; deleting some annotated points in the road surface marking ordered point set, although there will be a loss of accuracy relative to the original annotated points, this part of the loss in the embodiment of the present application is generated in the line segment closest to the straight line, and the impact on the accuracy of the point set is very limited.

[0074] After determining the number of sampling points, road markings of different shapes can be represented by a unified ordered point set through the above processing method, and the number of points is usually not too large. Compared with the above method of interpolating or deleting the ordered point set of road markings in an indiscriminate manner, the method of processing road markings based on the curvature radius can use a relatively small number of points to more accurately represent road markings of various positions and shapes, with less loss of accuracy relative to the original ordered point set.

[0075] In an embodiment of the present application, two adjacent annotation points in a road surface marking ordered point set are connected to generate a line segment vector group so as to calculate the curvature radius of the arc corresponding to two adjacent line segment vectors in each line segment vector group, and then the road surface marking ordered point set can be processed by the curvature radius, so that the final number of annotation points in the road surface marking ordered point set is consistent with the preset number of sampling points, thereby achieving a unified representation of road surface ordered point sets with different markings and better meeting actual needs.

[0076] A2, calculating the target offset and the length of each road marking through the road marking image and the ordered point set;

[0077] In the embodiment of the present application, when training the preset convolutional neural network, the network learning target of the preset convolutional neural network is constructed with the road surface marking instance as a unit, and the road surface marking instance can be specifically represented as: taking the point of half the cumulative length of each road surface marking in the ordered point set as the target point corresponding to the road surface marking instance, and generating the road surface marking category of each road surface marking instance according to actual needs. Among them, one road surface marking corresponds to one road surface marking instance.

[0078] In the embodiment of the present application, it is preferred to use the point (x) that is half the cumulative length of each road surface marking in the ordered point set. c ,y c ) is the target point, and the sampling multiple is calculated according to the size of the road marking image img_width*img_length and the size of the road marking feature map output by the preset convolutional neural network F_width*F_length. The sampling multiple includes the horizontal axis sampling multiple S x =img_width / F_width and vertical coordinate sampling multiple S y =img_length / F_length; the position coordinates (x c ,y c ) and the sampling multiple to calculate the target point offset of each road surface mark. Specifically, the two coordinate values ​​of the target point of each road surface mark are divided by the sampling multiple to obtain the middle value of the two coordinates of the target point of each road surface mark (x g ,y g ), thereby projecting the target point in the original road marking image into the grid of the road marking feature map, that is, x g =x c / S x ,y g =y c / S y ; Round the middle values ​​of the two coordinates of the target point to get the two coordinate index values ​​of the target point (g x ,g y ), i.e. g x =floor(x g ), g y =floor(y g ), floor() is a rounding function; the difference between the middle values ​​of the two coordinates of the target point of each road surface mark and the corresponding two coordinate index values ​​is calculated to obtain the target point offset (c x ,c y ), that is, c x =x g -g x , c y =y g -g y, the two offset values ​​in the target point offset are in the range of 0 to 1.

[0079] After calculating the target point offset of each road marking, first, the horizontal coordinate value of each road marking sampling point other than the target point is divided by the horizontal coordinate sampling multiple, and the vertical coordinate value of each road marking sampling point other than the target point is divided by the vertical coordinate sampling multiple to obtain the position coordinates (x i ,y i ), and then according to the position coordinates (x i ,y i ) and the position coordinates of the corresponding target point on the road marking feature map, that is, the middle of the coordinates of the target point (x g ,y g ) calculates the offset between the sampling points other than the target point and the corresponding target point on the road surface marking feature map, thereby obtaining the sampling point offset ( xi ,O yi ), namely O xi =x i -x g , O yi =y i -y g .

[0080] In one embodiment, the target point offset and sampling point offset of each road surface mark can be directly used as the target offset. The sampling point offset of each road surface mark is directly input into the preset convolutional neural network as the target offset for learning. The numerical range span is relatively large, making the learning difficulty of the preset convolutional neural network relatively large.

[0081] In another embodiment, considering that directly taking the target point offset and sampling point offset of each road surface mark as the network learning target has a large value range and is difficult to learn, the sampling point offset of each road surface mark can be normalized, and the target point offset and the normalized sampling point offset can be used as the target offset.

[0082] Specifically, the horizontal coordinate value of the sampling point offset of each road marking can be normalized by the width img_width of the road marking image, and the vertical coordinate value of the sampling point offset of each road marking can be normalized by the height img_length of the road marking image. The coordinate value range of each point is compressed to [0,1]. However, this method will increase the relative error for points with smaller offset values ​​(i.e., shorter road markings).

[0083] Alternatively, the two coordinate values ​​of the sampling point offset of each road surface marking can be normalized by the road surface marking length, and the same distribution targets can be obtained for road surface markings of different lengths, which can reduce the learning difficulty.

[0084] The road marking length may be an absolute length or a relative length normalized by the width, height or diagonal length of the road marking image.

[0085] A3. Using the road marking image as input data, and using the road marking category, target offset, and road marking length as learning objectives, a preset convolutional neural network is trained to obtain a first detection model.

[0086] The preset convolutional neural network in the embodiment of the present application includes a feature extraction module and a prediction module. The existing feature extraction network (such as a residual network, etc.) can be used as the feature extraction module. The feature extraction module is used to extract the feature map of the input road surface marking image, and the prediction module is used to perform road surface marking detection on the feature map extracted by the feature extraction module. Among them, the prediction module can be composed of multiple detection heads. The detection head can be connected to any feature map of the feature extraction module, and can also be connected to multiple different scales and different feature maps at the same time. The feature map finally connected by the detection head is the road surface marking feature map mentioned above. The detection head can specifically include a road surface marking existence detection head, a road surface marking category detection head, a target point offset detection head, a sampling point offset detection head and a road surface marking length detection head. The road surface marking existence detection head is used to detect whether there is a road surface marking in the input road surface marking feature map, and output the road surface marking existence probability of each pixel point on the road surface marking feature map. If the road marking feature map is the same size as the input road marking image, then one pixel in the road marking feature map corresponds to one pixel in the original input road marking image. If the road marking feature map is S times the size of the original input road marking image after downsampling, then one pixel in the road marking feature map corresponds to the S*S range of pixels in the original image. The value of each pixel output by the road marking existence detection head finally contains the road marking existence probability. The road marking category detection head is used to detect the category of the road marking in the input road marking feature map, and the output is the road marking category probability. Each road marking instance is represented by the probability of multiple classifications. Each pixel contains multiple classifications of the maximum number of road markings output. When the maximum number of road markings is set to 1, one pixel corresponds to one road marking instance, and one road marking instance corresponds to one road marking. The target point offset detection head is used to predict the offset of the target point of each road marking on the road marking feature map and output the target point offset prediction value. The sampling point offset detection head is used to predict the offset between the sampling point and the target point in the road marking feature map except the target point, and output the sampling point offset prediction value. The road marking length detection head is used to predict the length of each road marking and output the road marking length prediction value, which can be an absolute length or a relative length normalized by the width, height or diagonal length of the image. It can be understood that the prediction module in the embodiment of the present application can also use a visual transformer network to replace the detection head, and use the visual transformer to directly predict a specified number of road marking instances.

[0087] The embodiment of the present application uses a road marking image as input data, and trains a preset convolutional neural network with road marking category, target offset and road marking length as learning objectives until the preset convolutional neural network converges to obtain a trained preset convolutional neural network model, and uses the trained preset convolutional neural network model as the first detection model.

[0088] In the prior art, when detecting road markings such as lane lines, the coordinate regression of uniformly sampled points along the image height is usually used to detect lane lines. This method is basically based on the premise that the vehicle is basically parallel to the lane when traveling straight, and cannot handle lane lines that are at a large angle to the vehicle, nor can it be extended to road markings of various shapes such as road edges. The prior art also uses the equation parameters of the direct regression lane line to detect lane lines. Due to the error of the true value itself fitting, it is impossible to obtain a high-precision detection result. The first detection model in the embodiment of the present application is constructed with the training target as the unit of the road marking instance. The two classification detection heads can respectively realize the prediction of the existence and category of the road marking, and the offset detection head can realize the prediction of the position and shape of the road marking. It can predict road markings of various shapes with high detection accuracy. In addition, the embodiment of the present application uses the point of half of the cumulative length of the road marking as the target point of the road marking instance, which can handle any number of road markings, and the existence prediction is consistent with the actual performance of the image, which is more flexible in the subsequent association and merging between the line segments.

[0089] Step 102: Obtain input data of a second detection model from a road surface marking feature map using the first road surface marking detection result.

[0090] In the embodiment of the present application, the input data of the second detection model is a feature vector extracted from the road surface marking feature map extracted from the first detection model according to the first road surface marking detection result. The process of obtaining the input data of the second detection model through the first road surface marking detection result can be:

[0091] B1. parsing the first road surface marking detection result to obtain a first road surface marking sequence;

[0092] Specifically, the road marking detection result in the embodiment of the present application includes the road marking existence probability, road marking category probability, target offset prediction value, and road marking length prediction value of each pixel point in the road marking feature map output by the first detection model, wherein the target offset prediction value includes the target point offset prediction value and the sampling point offset prediction value.

[0093] When parsing the road marking detection results, first, the target pixel points whose road marking existence probability is higher than the existence probability threshold are extracted from the road marking feature map. The target pixel points whose road marking existence probability is higher than the existence probability threshold in the road marking feature map are points belonging to the road marking; then, the road marking category of the target pixel point is determined according to the road marking category probability of the target pixel point, and the road marking instances whose road marking category probability is higher than the category probability threshold are extracted from the target pixel point. The road marking category to which the target pixel point belongs is determined according to the road marking category probability of each road marking instance in each target pixel point on the road marking feature map. When there are multiple road marking categories, the road marking category detection head predicts the road marking category probability of each marking category instance under each road marking category, and the sum of the road marking category probabilities under each category is 1, among which the category with the largest road marking category probability value is the category to which the road marking instance belongs, and then the road marking instances whose road marking category probability is higher than the category probability threshold are extracted from the target pixel point.

[0094] For each road marking instance extracted, the position coordinates of each point in the road marking instance in the road marking feature map are obtained according to the target offset prediction value to obtain the first road marking sequence. First, the position coordinates of the target point of the road marking instance in the road marking feature map are obtained according to the target point offset. Specifically, when training the preset convolutional neural network, the calculated target point offset is directly used as the network learning target. Therefore, the target point offset prediction value (c') output by the target point offset detection head is directly used. x ,c' y ) to perform reverse calculation to obtain the position coordinates (x' g ,y' g ), where x' g =c' x +g x , y' g =c' y +g y , g x is the lateral index value of the target pixel point corresponding to each road marking instance on the predicted road marking feature map, g y It is the longitudinal index value of the target pixel point corresponding to each road marking instance on the predicted road marking feature map.

[0095] Then, the position coordinates of the sampling points other than the target point in the pavement marking feature map are calculated based on the sampling point offset prediction value and the position coordinates of the target point in the pavement marking instance in the pavement marking feature map, so as to obtain the first pavement marking sequence. Specifically, for each pavement marking instance, if the calculated sampling point offset is directly used as the network learning target when training the preset convolutional neural network, the sampling point offset prediction value (O') output by the sampling point offset detection head is directly calculated. xi ,O' yi ) and the position coordinates (x' g ,y' g ) are summed to obtain the position coordinates (x' i ,y' i ), that is, x' i =O' xi +x' g , y' i =O' yi +y' g If the normalized sampling point offset is used as the network learning target when training the preset convolutional neural network, the sampling point offset prediction value (O' xi ,O' yi ) to perform the normalization inverse operation to obtain the sampling point offset prediction value before normalization, and then the sampling point offset prediction value before normalization and the position coordinate (x' g ,y' g ) are summed to obtain the position coordinates (x' i ,y' i ), the position coordinates of each point in each road marking instance in the road marking feature map are obtained, that is, the first road marking sequence. If the width and height of the input road marking image are used to normalize the sampling point offset during training, the sampling point offset prediction value is subjected to a normalized inverse operation based on the width and height of the input road image to be detected during prediction; if the road marking length is used to normalize the sampling point offset during training, the road marking length obtained based on the road marking length prediction value is subjected to a normalized inverse operation based on the sampling point offset prediction value during prediction.

[0096] B2. rounding the point set coordinates of each road surface marking in the first road surface marking sequence to obtain a grid index of each point in the road surface marking feature map;

[0097] Each road surface marking sequence in the first road surface marking sequence corresponds to a candidate road surface marking, except that each candidate road surface marking is represented by a point set. The horizontal coordinate value and the vertical coordinate value in the point set coordinates of each candidate road surface marking in the first road surface marking sequence are rounded to obtain the grid index of each point in the road surface marking feature map extracted by the first detection model.

[0098] B3, extracting corresponding feature vectors from the road surface marking feature map according to the grid index and concatenating the feature vectors to obtain input data for the second detection model;

[0099] For each sequence in the first road surface marking sequence, the feature vector corresponding to the grid is extracted from the road surface marking feature map according to the grid index to obtain the feature vector corresponding to each point. Even if multiple points are mapped to the same grid, it is only necessary to extract the feature vector corresponding to the grid from the road surface marking feature map according to the grid index as the feature vector corresponding to each point; then, the feature vectors corresponding to each point are concatenated according to the order of each point in each sequence, which can be as follows: Figure 2 The two-dimensional splicing shown can also splice the feature vectors corresponding to each point into a one-dimensional vector. Figure 2 Another schematic diagram of the road surface marking detection method provided in an embodiment of the present application.

[0100] When the first detection model in the embodiment of the present application is configured using the configuration method of steps A1-A3 above, each road surface marking sequence obtained by the first detection model has the same number of points. Therefore, the feature dimensions of the feature representation of the road surface marking instance after extracting the corresponding feature vectors on the road surface marked feature map of the same scale and then splicing them are always consistent, and there is no need to perform additional unification operations on the feature dimensions of the spliced ​​feature vectors of each road surface marking instance, and the spliced ​​feature vectors corresponding to each candidate road surface marking can be directly used as the input data of the second detection model. When the number of points of each road surface marking obtained by configuring the first detection model using other configuration methods is different, it is necessary to first perform a unification operation on the feature dimensions of the spliced ​​feature vectors corresponding to each candidate road surface marking, so that the dimensions of the spliced ​​feature vectors of each candidate road surface marking are the same, and then use the unified spliced ​​feature vectors as the input data of the second detection model.

[0101] It should be noted that a new branch may be derived after the road marking feature map of the feature extraction module of the first detection model, and operations such as calculating indexes, extracting feature vectors, and concatenating feature vectors may be performed through the new branch to obtain a concatenated feature vector.

[0102] Step 103: Input the input data into the second detection model to perform road surface marking detection to obtain a second road surface marking detection result.

[0103] Input data corresponding to each candidate road surface marking is input into the second detection model for road surface marking detection, and the second detection model directly performs road surface marking detection based on the input data. The second detection model in the embodiment of the present application is composed of a prediction module, and the prediction module is used to perform road surface marking detection based on the input data, wherein the prediction module can be a multi-layer perceptron (MLP) or a visual transformer network (visual transformer).

[0104] In one embodiment, the second detection model can be used to detect whether each candidate road surface marking is a road surface marking, the specific type of road surface marking, the offset of the target point, and the offset of other points except the target point. When parsing the second road surface marking detection result, it is necessary to parse based on the position coordinates of each point in the first road surface marking sequence to obtain the second road surface marking sequence corresponding to the second road surface marking detection result. It can be understood that at this time, when training the second detection model, the concatenated feature vector is used as input data, and the road surface marking category, the position coordinates of each point in the first road surface marking sequence, and the offset of the position coordinates of each point in the corresponding road surface marking ordered point set actually marked and projected into the road surface marking feature map are used as training targets.

[0105] Accordingly, when parsing the second road surface marking detection result, the final category of each candidate road surface marking can be determined according to the category distribution in the second road surface marking detection result, and the target point offset of each candidate road surface marking in the second road surface marking detection result and the offset of other points except the target point are added to the position coordinates of each corresponding point in the corresponding candidate road surface marking to obtain a refined road surface marking instance, and the position coordinates of each point in the refined road surface marking instance are the refined position coordinates.

[0106] Since the position coordinates of each point in the fine-tuned road surface marking instance are the position coordinates in the road surface marking feature map extracted by the first detection model, it is necessary to convert the position coordinates of each point in the fine-tuned road surface marking instance into the position coordinates in the road image to be detected. The position coordinates of the target point in the fine-tuned road surface marking instance in the road surface marking feature map can be converted into the position coordinates in the road image to be detected according to the size of the road image to be detected and the size of the road surface marking feature map.

[0107] Specifically, first, the horizontal coordinate sampling multiple S can be determined by the size of the road image to be detected and the size of the road surface marking feature map. x and the vertical sampling multiple S y ; Multiply the abscissa value of the position coordinate of the target point in the pavement marking feature map in each fine-tuned pavement marking instance by the abscissa sampling multiple S x, the ordinate value of the position coordinate of the target point in the pavement marking feature map in each fine-tuned pavement marking instance multiplied by the ordinate sampling multiple S y , thereby obtaining the position coordinates of the target points of each finely tuned road surface marking instance in the road image to be detected. It can be understood that the size of the road surface marking image input into the preset convolutional neural network during training is consistent with the size of the road image to be detected input into the first detection model during prediction.

[0108] Then, the abscissa values ​​of the position coordinates of the points other than the target point in each fine-tuned pavement marking instance in the pavement marking feature map are multiplied by the abscissa sampling multiple S x , multiply the ordinate value of the position coordinate of other points in the road marking feature map by the ordinate sampling multiple S y , thereby obtaining the position coordinates of other points in each fine-tuned road surface marking instance except the target point in the input road image to be detected, and thus obtaining the position coordinates of each point in each fine-tuned road surface marking instance in the road image to be detected, that is, the second road surface marking sequence. When visualizing the second road surface marking sequence, the points in the second road surface marking sequence corresponding to each road surface marking are sequentially connected to obtain the line representation corresponding to each second road surface marking sequence.

[0109] In another embodiment, the second detection model can be used to directly predict the point set coordinates of the road surface marking instance, without using the position coordinates of each point in the first road surface marking sequence to analyze the second road surface marking detection result. It can be understood that when the second detection model is trained, the concatenated feature vector is used as input data, and the road surface marking category and the position coordinates of each point in the road surface marking ordered point set projected into the road surface marking feature map are used as training targets.

[0110] When parsing the second road surface marking detection result, the category of the road surface marking instance can be determined according to the road surface marking category probability in the second road surface marking detection result; since the point set coordinates of the road surface marking instance predicted in the second road surface marking detection result are position coordinates on the input data scale, they need to be converted into position coordinates on the road image to be detected, and the abscissa value of each point in the point set coordinates of the road surface marking instance predicted in the second road surface marking detection result can be directly multiplied by the abscissa sampling multiple S x , multiply the ordinate value by the ordinate sampling multiple S y , thereby obtaining the position coordinates of each point in each fine-tuned road surface marking instance in the road image to be detected, that is, obtaining the second road surface marking sequence.

[0111] Furthermore, after the second pavement marking sequence is obtained, it can be screened to remove the second pavement marking sequence with low quality.

[0112] Specifically, the distance between each road surface marking in the second road surface marking sequence can be calculated; when the distance between two road surface markings is less than a preset distance threshold, the road surface markings with a smaller road surface marking category probability in the second road surface marking sequence are eliminated to obtain a final road surface marking sequence.

[0113] In one embodiment, the horizontal distance between each road marking is calculated according to the coordinates of each point in the second road marking sequence, and the horizontal distance between each road marking is used as the distance between each road marking. Assume that the road marking sequence corresponding to the road marking a is The road marking sequence corresponding to the road marking b is in, The second road marking sequence I a The i-th sampling point in For road marking sequence I b The i-th sampling point in a e For road marking sequence I a The end point index of b e For road marking sequence I b The end point index, a s For road marking sequence I a The starting index of b s For road marking sequence I b The horizontal distance D between the road marking a and the road marking b can be calculated using the following formula:

[0114]

[0115] In the formula, e = min(a e ,b e ), s = max(a s ,b s ), min() is the minimum function, and max() is the maximum function.

[0116] In another embodiment, the angle between each road surface mark and the horizontal line is fitted according to the second road surface mark sequence; and the distance weight coefficient between each road surface mark is calculated by the angle between each road surface mark and the horizontal line. Assume that the road surface mark sequence I corresponding to the fitted road surface mark a a The angle with the horizontal line is θ a , fitting the road marking sequence I corresponding to the road marking b b The angle with the horizontal line is θ b , the distance weight coefficient w between road marking a and road marking b a,b = sin(max(θ a ,θ b )) / cos(|θ a -θ b| / 2). The horizontal distances between the road markings are weighted by the distance weight coefficients between the road markings to obtain the distances between the road markings.

[0117] It is understandable that the angle between each point in each road surface marking sequence and the horizontal line can also be fitted, and the distance weight coefficients of two points in the two road surface marking sequences can be calculated by the angles between the two points and the horizontal line. The horizontal distance between the two points (i.e., the difference in the horizontal coordinates of the two points) is weighted by the distance weight coefficients of the two points to obtain the horizontal weighted distance. Finally, the horizontal weighted distances of all points in the two road surface marking sequences are averaged to obtain the distance between the two road surface marking sequences.

[0118] In the embodiment of the present application, the angle between each second road surface marking sequence and the horizontal line is fitted, and then the distance weight coefficient is calculated by the angle between the two second road surface marking sequences and the horizontal line. The horizontal distance between the two second road surface marking sequences is weighted by the distance weight coefficient, so that the calculated distance between the road surface marking sequences is independent of the angle between the two road surface marking sequences, thereby avoiding the influence of the angle relationship on the distance measurement result between the road surface markings.

[0119] When the distance between two second road markings is less than a preset distance threshold, the second road marking sequence corresponding to the road marking with a lower road marking category probability is removed, and the removed second road marking sequence is regarded as a low-quality road marking sequence; when the distance between two second road markings is greater than or equal to the preset distance threshold, the second road marking sequences corresponding to the two road markings are retained, thereby obtaining a final road marking sequence. The preset distance threshold can be specifically set according to actual conditions and is not specifically limited here.

[0120] After obtaining the final road surface marking sequence, when the final road surface marking sequence needs to be visualized, the points in each final road surface marking sequence are connected in order to obtain the line representation corresponding to each final road surface marking sequence. In addition, in the actual application scenario of autonomous driving, it is necessary to convert the coordinates of each point in the final road surface marking sequence from two dimensions to three dimensions, so as to obtain the road surface marking sequence in the physical coordinate system. Converting the coordinates of each point in the final road surface marking sequence from two dimensions to three dimensions belongs to the prior art and will not be described in detail here.

[0121] In the embodiment of the present application, after obtaining the first road surface marking detection result of the first detection model, the first road surface marking detection result is used as a candidate road surface marking instance, and feature extraction is performed on the road surface marking feature map extracted from the first detection model through the first road surface marking detection result to obtain input data for the second detection model, and then the second detection model is used to fine-tune the detection result of each candidate road surface marking instance to obtain a second road surface marking detection result with higher accuracy, thereby improving the technical problem of low detection accuracy in the prior art.

[0122] Furthermore, the embodiment of the present application calculates the distance between each road surface marking sequence to filter out the low-quality second road surface marking sequence to obtain the final road surface marking sequence, thereby further improving the accuracy of the road surface marking detection result.

[0123] The above is an embodiment of a road surface marking detection method provided by the present application, and the following is an embodiment of a road surface marking detection device provided by the present application.

[0124] Please refer to Figure 3 , a road surface marking detection device provided in an embodiment of the present application includes:

[0125] A first acquisition unit is used to acquire a first road surface marking detection result obtained by a first detection model according to a road surface marking feature map extracted from a road image to be detected;

[0126] A second acquisition unit, configured to acquire input data of a second detection model from a road surface marking feature map according to the first road surface marking detection result;

[0127] The detection unit is used to input the input data into the second detection model to perform road surface marking detection to obtain a second road surface marking detection result.

[0128] As a further improvement, the second acquisition unit is specifically used for:

[0129] parsing the first road surface marking detection result to obtain a first road surface marking sequence;

[0130] Rounding the point set coordinates of each road surface mark in the first road surface mark sequence to obtain a grid index of each point in the road surface mark feature map;

[0131] The corresponding feature vector is extracted from the road surface marking feature map according to the grid index and the feature vectors are concatenated to obtain input data of the second detection model.

[0132] As a further improvement, the road surface marking detection device in the embodiment of the present application further includes:

[0133] a parsing unit, configured to parse the second road surface marking detection result to obtain a second road surface marking sequence;

[0134] The screening unit is used to calculate the distance between each road surface marking in the second road surface marking sequence, and when the distance between two road surface markings is less than a preset distance threshold, remove the road surface markings with a smaller road surface marking category probability in the second road surface marking sequence to obtain a final road surface marking sequence.

[0135] As a further improvement, the screening unit is specifically used for:

[0136] Calculating the horizontal distance between each road surface marking according to the position coordinates of each point in the second road surface marking sequence;

[0137] The horizontal distance between each road surface marking is taken as the distance between each road surface marking;

[0138] or, fitting the angle between each road surface marking and the horizontal line according to the second road surface marking sequence, and calculating the distance weight coefficient between each road surface marking according to the angle between each road surface marking and the horizontal line;

[0139] The horizontal distance between each road surface marking is weighted by the distance weight coefficient between each road surface marking to obtain the distance between each road surface marking;

[0140] When the distance between two road surface markings is less than a preset distance threshold, road surface markings with a smaller road surface marking category probability in the second road surface marking sequence are removed to obtain a final road surface marking sequence.

[0141] In the embodiment of the present application, after obtaining the first road surface marking detection result of the first detection model, the first road surface marking detection result is used as a candidate road surface marking instance, and feature extraction is performed on the road surface marking feature map extracted from the first detection model through the first road surface marking detection result to obtain input data for the second detection model, and then the second detection model is used to fine-tune the detection result of each candidate road surface marking instance to obtain a second road surface marking detection result with higher accuracy, thereby improving the technical problem of low detection accuracy in the prior art.

[0142] Furthermore, the embodiment of the present application calculates the distance between each road surface marking sequence to filter out the low-quality second road surface marking sequence to obtain the final road surface marking sequence, thereby further improving the accuracy of the road surface marking detection result.

[0143] Please refer to Figure 4 , an embodiment of the present application also provides a road surface marking detection device, the device includes a processor and a memory;

[0144] The memory is used to store the program code and transmit the program code to the processor;

[0145] The processor is used to execute the road surface marking detection method in the aforementioned method embodiment according to the instructions in the program code.

[0146] The embodiment of the present application further provides a computer-readable storage medium, which is used to store program codes. When the program codes are executed by a processor, the road surface marking detection method in the aforementioned method embodiment is implemented.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0148] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0149] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0150] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0154] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A road marking detection method, characterized in that: include: Obtaining a first road surface marking detection result obtained by a first detection model according to a road surface marking feature map extracted from a road image to be detected; The first road surface marking detection result includes a road surface marking existence probability, a road surface marking category probability and a target offset prediction value of each pixel point in the road surface marking feature map; Acquiring input data of a second detection model from the road surface marking feature map according to the first road surface marking detection result, comprising: Extracting target pixel points whose existence probability of the road surface marking is higher than an existence probability threshold from the road surface marking feature map; Determine the road marking category of the target pixel point according to the road marking category probability of the target pixel point, and extract road marking instances whose road marking category probability is higher than a category probability threshold from the target pixel point; Acquire the position coordinates of each point in the pavement marking instance in the pavement marking feature map according to the target offset prediction value to obtain a first pavement marking sequence; rounding the point set coordinates of each road surface marking in the first road surface marking sequence to obtain a grid index of each point in the road surface marking feature map; Extracting corresponding feature vectors from the road surface marking feature map according to the grid index and performing feature vector concatenation to obtain input data for the second detection model; The input data is input into the second detection model to perform road surface marking detection to obtain a second road surface marking detection result.

2. The road surface marking detection method according to claim 1, characterized in that: The target offset prediction value includes a target point offset prediction value and a sampling point offset prediction value; The step of acquiring the position coordinates of each point in the pavement marking instance in the pavement marking feature map according to the target offset prediction value to obtain a first pavement marking sequence comprises: Acquire the position coordinates of the target point of the pavement marking instance in the pavement marking feature map according to the target point offset; The position coordinates of the sampling points other than the target point in the pavement marking instance in the pavement marking feature map are calculated according to the sampling point offset prediction values ​​and the position coordinates of the target point of the pavement marking instance in the pavement marking feature map to obtain a first pavement marking sequence.

3. The road surface marking detection method according to claim 1, characterized in that: The method further comprises: parsing the second road surface marking detection result to obtain a second road surface marking sequence; calculating the distance between each pavement marking in the second pavement marking sequence; When the distance between two road surface markings is less than a preset distance threshold, the road surface markings with a smaller road surface marking category probability in the second road surface marking sequence are removed to obtain a final road surface marking sequence.

4. The road surface marking detection method according to claim 3, characterized in that: The calculating the distance between each road surface marking in the second road surface marking sequence comprises: Calculating the horizontal distance between each road surface marking according to the position coordinates of each point in the second road surface marking sequence; Taking the horizontal distance between each road surface marking as the distance between each road surface marking; or, fitting the angle between each road surface marking and the horizontal line according to the second road surface marking sequence, and calculating the distance weight coefficient between each road surface marking according to the angle between each road surface marking and the horizontal line; The horizontal distances between the road surface markings are weighted by the distance weight coefficients between the road surface markings to obtain the distances between the road surface markings.

5. A road marking detection device, characterized in that: include: A first acquisition unit is used to acquire a first road surface marking detection result obtained by a first detection model according to a road surface marking feature map extracted from a road image to be detected; The first road surface marking detection result includes a road surface marking existence probability, a road surface marking category probability and a target offset prediction value of each pixel point in the road surface marking feature map; a second acquisition unit, configured to acquire input data of a second detection model from the road surface marking feature map according to the first road surface marking detection result, and specifically to extract target pixel points whose probability of existence of the road surface marking is higher than a threshold value of existence probability from the road surface marking feature map; Determine the road marking category of the target pixel point according to the road marking category probability of the target pixel point, and extract road marking instances whose road marking category probability is higher than a category probability threshold from the target pixel point; Acquire the position coordinates of each point in the pavement marking instance in the pavement marking feature map according to the target offset prediction value to obtain a first pavement marking sequence; rounding the point set coordinates of each road surface marking in the first road surface marking sequence to obtain a grid index of each point in the road surface marking feature map; Extracting corresponding feature vectors from the road surface marking feature map according to the grid index and performing feature vector concatenation to obtain input data for the second detection model; The detection unit is used to input the input data into the second detection model to perform road surface marking detection to obtain a second road surface marking detection result.

6. A road marking detection device, characterized in that: The device comprises a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the road surface marking detection method according to any one of claims 1 to 4 according to the instructions in the program code.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and when the program codes are executed by a processor, the road surface marking detection method according to any one of claims 1 to 4 is implemented.

Images