Deep learning-based traffic sign pole tilt angle automatic detection method

Abstract

The application discloses a kind of traffic signpost inclination angle automatic detection methods based on deep learning, comprising: image acquisition;Extract traffic sign from image;Calculate the inclination angle of traffic signpost in extracted traffic sign;The specific method for calculating the inclination angle of traffic signpost in extracted traffic sign is: the picture containing traffic sign is scaled and handled;Extract feature map from the picture after processing;Use several prediction lines to fit traffic signpost on the extracted feature map and calculate the inclination angle of traffic signpost according to the fitting result.The traffic signpost inclination angle automatic detection method based on deep learning of the application uses automatic identification mode instead of manual detection, greatly saves labor cost, and has high detection efficiency.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation, specifically relating to an automatic detection method for the tilt angle of traffic sign poles based on deep learning. Background Technology

[0002] In the realm of transportation, traffic signs play a vital role as legally mandated directional markers. However, after installation, natural disasters, traffic accidents, road repairs, and other factors can cause traffic signs to tilt or even collapse, disrupting traffic flow and creating safety hazards. Therefore, it is essential to develop a convenient and rapid method for assessing the proper functioning of traffic signs.

[0003] The traditional method for dealing with tilted traffic signs mainly involves citizens reporting the issue to the government or media through hotlines, and then the government forwarding the complaint to the traffic police for reinforcement. This method is time-consuming, and citizens may not be very willing to report such incidents. Summary of the Invention

[0004] This invention provides a deep learning-based method for automatically detecting the tilt angle of traffic sign poles to solve the aforementioned technical problems. Specifically, the technical solution is as follows:

[0005] A deep learning-based method for automatically detecting the tilt angle of traffic sign poles includes the following steps:

[0006] Image acquisition;

[0007] Extract traffic signs from images;

[0008] Calculate the tilt angle of the traffic sign poles in the extracted traffic signs;

[0009] The specific method for calculating the tilt angle of the traffic sign pole in the extracted traffic signs is as follows:

[0010] The extracted images containing traffic signs are scaled up.

[0011] Extract feature maps from the processed images;

[0012] Several prediction lines are used to fit the traffic sign pole on the extracted feature map, and the tilt angle of the traffic sign pole is calculated based on the fitting results.

[0013] Furthermore, the specific method for extracting traffic signs from images is as follows:

[0014] The acquired images are scaled and filled.

[0015] The DarkNet network is used to extract feature maps of different sizes at various depths of the processed image.

[0016] Traffic signs are identified on feature maps of different sizes by using multiple pre-defined bounding boxes with different aspect ratios and areas, which are set through clustering.

[0017] Furthermore, in the process of recognizing traffic signs on feature maps of different sizes using multiple pre-defined bounding boxes with different aspect ratios and areas based on clustering,

[0018] The traffic sign is identified by selecting the frame with the largest intersection-union ratio from these recognition frames.

[0019] Furthermore, feature maps of different sizes at three different depths are extracted from the processed image using the DarkNet network.

[0020] Furthermore, the specific method for extracting feature maps from the processed image is as follows:

[0021] Feature maps are extracted from the processed images using residual networks.

[0022] Furthermore, the specific method for fitting traffic sign poles onto the extracted feature map using several prediction lines and calculating the tilt angle of the traffic sign poles based on the fitting results is as follows:

[0023] Several local information is extracted by using prediction lines with different starting points and angles on the extracted feature map;

[0024] An attention mechanism is used to exchange local information corresponding to all prediction lines to obtain global information;

[0025] Determine whether the predicted line is a traffic sign pole by using local and global information;

[0026] The tilt angle of the traffic sign is calculated based on the predicted line that is identified as the traffic sign pole.

[0027] Furthermore, in the process of extracting local information using prediction lines with different starting points and angles on the extracted feature map,

[0028] Divide the image vertically into equal intervals, and denote the ordinate of the dividing lines as . Where N p To determine the number of lines, let the image height be h, then y i =i·(h / N p -1);

[0029] The horizontal coordinates represented by the vertical division are:

[0030]

[0031] Where (x, y) are the coordinates on the feature map, (x originy origin ) represents the coordinates of the traffic sign extracted from the image, δ back The image is reduced by a factor of θ through feature extraction, where θ represents the angle between the prediction line and the horizontal axis, and the prediction line passes through the set of points where it intersects these dividing lines. express.

[0032] Furthermore, the specific method for exchanging local information corresponding to all prediction lines using an attention mechanism to obtain global information is as follows:

[0033] The obtained local information is input into a fully connected attention network. After the information content is balanced by softmax, it is multiplied with the local information from other prediction lines and finally summed to generate a global information vector a. global The formula is as follows:

[0034]

[0035]

[0036] Where i is the label of this prediction line, j is the label of other prediction lines, and a local L represents local information of a single prediction line. att This represents a fully connected attention network.

[0037] Furthermore, the specific method for determining whether a predicted line is a traffic sign pole using local and global information is as follows:

[0038] Will be by all and A matrix of local and global information is assembled and input into a classification model. The classification model outputs the probability of whether each predicted line is a traffic sign pole.

[0039] Will be by all and The matrix, which combines local and global information, is input into the regression model. The regression model outputs the horizontal offset and length l of each point on the prediction line. For prediction lines that are judged to be traffic sign poles in the classification model, the final result is output through nonmaximum suppression.

[0040] Furthermore, the specific method for acquiring images is as follows:

[0041] The camera was mounted on the drone to capture images.

[0042] The advantage of this invention lies in the fact that the deep learning-based automatic detection method for the tilt angle of traffic sign poles provides an automatic identification method to replace manual detection, which greatly saves labor costs and has high detection efficiency. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of an automatic detection method for the tilt angle of traffic sign poles based on deep learning, according to the present invention.

[0045] Figure 2 This is a schematic diagram of the detection model;

[0046] Figure 3 It is a schematic diagram of the specific model used to calculate the tilt angle of the traffic sign pole in the extracted traffic sign. Detailed Implementation

[0047] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0048] like Figure 1 The diagram illustrates an automatic traffic sign pole tilt angle detection method based on deep learning, comprising the following steps: S1: Acquire an image. S2: Extract the traffic sign from the image. S3: Calculate the tilt angle of the traffic sign pole within the extracted traffic sign.

[0049] For step S1: Acquire images.

[0050] In this application, the specific method for acquiring images is as follows: a camera is mounted on a drone to capture images.

[0051] Understandably, images can also be captured by mounting the camera on various vehicles such as cars, or by manually using a tripod. The camera must be kept level during image capture. The captured images are then input into a system such as... Figure 2 In the model shown, steps S2 and S3 are further performed. Figure 3 A schematic diagram of a specific model for calculating the tilt angle of the traffic sign pole in the extracted traffic signs.

[0052] For step S2: Extract traffic signs from the image.

[0053] Specifically, the method for extracting traffic signs from images is as follows:

[0054] The acquired images are scaled and padded to ensure that the input images are of consistent size.

[0055] The DarkNet network extracts feature maps of different sizes at various depths from the processed image. Specifically, the DarkNet network extracts feature maps of different sizes at three different depths from the processed image. In this specific embodiment, for a 608*608 image, it is downsampled to 76*76, 38*38, and 19*19 pixels, and then detected. During detection, the information from each layer complements each other and is fused to enhance the recognition capability.

[0056] Traffic signs are identified on feature maps of different sizes using multiple sets of pre-defined bounding boxes with different aspect ratios and areas, set through clustering. Preferably, during the process of identifying traffic signs on feature maps of different sizes using multiple sets of pre-defined bounding boxes with different aspect ratios and areas, the bounding box with the largest intersection-union ratio is selected as the final frame for identifying the traffic sign, thereby identifying the traffic sign.

[0057] In this specific embodiment, the number of recognition boxes is set to 9, and the dimensions of the recognition boxes are set to 10*13, 16*30, 33*23, 30*61, 62*45, 59*119, 116*90, 156*198, and 373*326. These dimensions can be automatically learned during model training. Recognition of traffic sign targets is achieved by adjusting the coordinates and dimensions of the above recognition boxes. The box with the largest intersection-union ratio (IUU) is selected from all adjusted recognition boxes as the final frame for recognizing the traffic sign.

[0058] For step S3: Calculate the tilt angle of the traffic sign poles in the extracted traffic signs.

[0059] In this application, the specific method for calculating the tilt angle of the traffic sign pole in the extracted traffic sign is as follows:

[0060] The extracted images containing traffic signs are scaled to ensure they are the same size as the input images.

[0061] Feature maps are extracted from the processed images. In this application, feature maps are extracted from the processed images using a residual network.

[0062] Several prediction lines are used to fit the traffic sign pole on the extracted feature map, and the tilt angle of the traffic sign pole is calculated based on the fitting results.

[0063] Specifically, the method for fitting traffic sign poles onto the extracted feature map using several prediction lines and calculating the tilt angle of the traffic sign poles based on the fitting results is as follows:

[0064] Several local information is extracted from the extracted feature map using prediction lines with different starting points and angles. The starting points of the prediction lines are located on the left, bottom, and right sides of the image. The number of starting points and the density of prediction lines at the bottom are greater than those on the left and right sides. In this description, the positive x-axis is defined as horizontal to the right, and a prediction line can be uniquely determined by its starting point and the angle at which it intersects the positive x-axis. In this specific embodiment, there are 6 angles and 72 points where the prediction line starts on the left and right sides of the image. There are 15 angles and 128 points where the starting point is at the bottom of the image. The angles are evenly distributed between 0° and 180°, and the points are evenly distributed along the image edges. In practice, the angle distribution can be adjusted by clustering the dataset.

[0065] To simplify the calculations and reduce computational complexity, both the prediction line and the fitted line are represented as point sets. Specifically, the image is divided vertically at equal intervals, and the ordinate of the dividing line is denoted as... Where N p To determine the number of lines, let the image height be h, then y i =i·(h / N p -1).

[0066] The horizontal coordinates represented by the vertical division are:

[0067]

[0068] Where (x, y) are the coordinates on the feature map, (x origin y origin ) represents the coordinates of the traffic sign extracted from the image, δ back The image is reduced by a factor of θ through feature extraction, where θ represents the angle between the prediction line and the horizontal axis, and the prediction line passes through the set of points where it intersects these dividing lines. The line can be approximated by fitting a linear function to all points in the point set.

[0069] An attention mechanism is employed to exchange local information corresponding to all predicted lines to obtain global information. In practice, due to the presence of traffic signs at the top, a traffic sign pole will not cross the entire image. Therefore, this application uses a start index s and an end index e to define the length of a predicted line. The start index s is determined by the y-coordinate of the starting point of the predicted line. origin The ending index e is determined by the starting index and the length of the prediction line, and the formula is e = s + [l] - 1.

[0070] Meanwhile, the acquired images may suffer from occlusion issues, resulting in incomplete display of traffic sign poles. The solution proposed in this application is to acquire information about all other predicted lines through an attention mechanism based on the prediction of one predicted line, thus expanding the scope of information from a single line to the entire image.

[0071] Specifically, the obtained local information is input into a fully connected attention network. After the information content is balanced by softmax, it is multiplied by the local information from other prediction lines and finally summed to generate a global information vector a. global The formula is as follows:

[0072]

[0073]

[0074] Where i is the label of this prediction line, j is the label of other prediction lines, and a local L represents local information of a single prediction line. att This represents a fully connected attention network.

[0075] This application uses both local and global information to determine whether a predicted line is a traffic sign pole. A classification model is used to identify whether a predicted line is a traffic sign pole, and a regression model is used to detect the location of the traffic sign pole. (The last sentence appears to be incomplete and possibly refers to a separate process.) and A matrix composed of concatenated local and global information is input into a classification model, which outputs the probability of whether it is a traffic sign pole.

[0076] The regression model outputs the coordinate differences and length *l* between points on the 72 predicted lines and their predicted values, representing the number of valid coordinate differences. Each predicted line is assigned a binary label in the model, indicating whether it is a traffic sign pole. For all generated predicted lines, their distances *D* from the true values ​​are calculated using the following formula:

[0077]

[0078] Where L represents the prediction line, l represents the true value, X represents the x-coordinate of a point on the prediction line, and x represents the x-coordinate of a point on the true value. max The formula represents the common starting index of two lines. e min The common ending index of two lines is represented by the formula:

[0079] In this specific embodiment, t pos Set to 15, t negThe hyperparameter N is set to 20. For a predicted line, if its distance to the true value is smaller than all other lines, and this distance is less than 15, it is assigned a positive label. If its distance to all true values ​​is greater than 20, it is assigned a negative label. The loss value is calculated by summing these two types of predicted lines; all other predicted lines are not included in the loss calculation. To speed up processing while maintaining model accuracy, the number of times each predicted line is assigned a positive label is recorded during model training. When using the model, only lines assigned a positive label more than the set hyperparameter N are retained. a The prediction line. In this specific embodiment, N a Set to 1000.

[0080] The loss function formula for the overall model is as follows:

[0081]

[0082] For the classification part, p i This represents the classification probability obtained by the model after predicting the i-th prediction line. Indicates the truth value. The Focal Loss function is used, with λ used to balance the loss weights. For the regression part, r i This represents the predicted value of line t obtained through model prediction, and is a vector. The first dimension of the vector represents the length, i.e., the number of points actually used to represent the traffic sign poles. The remaining dimensions are the x-coordinates of the point set. This represents the true value that is closest to prediction line i. The SmoothL1 loss function is used.

[0083] The tilt angle of the traffic sign is calculated based on the predicted line that indicates it is a traffic sign pole. Finally, using the output of whether it is a traffic sign pole and the point set information, the optimal point set that is judged as a positive example is selected after non-maximum suppression and fitted with a linear function to obtain the recognition line of the traffic sign pole on the image. The tilt angle of the traffic sign pole can be obtained by calculating the angle between the recognition line and the horizontal line.

[0084] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A deep learning-based automatic detection method for a traffic signpost pole inclination angle, characterized in that, The method comprises the following steps: Collecting images; Extracting traffic signs from the images; Calculating the inclination angle of the traffic sign poles in the extracted traffic signs; The specific method for calculating the inclination angle of the traffic sign poles in the extracted traffic signs is: Scaling the extracted pictures containing traffic signs; Extracting feature maps from the processed pictures; Fitting traffic sign poles on the extracted feature maps using a plurality of prediction lines and calculating the inclination angle of the traffic sign poles according to the fitting results; The specific method for fitting traffic sign poles on the extracted feature maps using a plurality of prediction lines and calculating the inclination angle of the traffic sign poles according to the fitting results is: Extracting a plurality of local information on the extracted feature maps using a plurality of prediction lines with different starting points and different angles; Exchanging the local information corresponding to all prediction lines to obtain global information by using an attention mechanism; Judging whether the prediction lines are traffic sign poles by using the local information and the global information; Calculating the inclination angle of the traffic sign poles according to the prediction lines judged as traffic sign poles.

2. The deep learning-based automatic detection method for the inclination angle of traffic sign poles according to claim 1, wherein the specific method for extracting traffic signs from the images is: Performing scaling and padding processing on the collected images; Extracting different sizes of feature maps of the processed images at a plurality of different depths by using a DarkNet network; Identifying traffic signs on the extracted feature maps of different sizes by using a plurality of groups of recognition boxes with different aspect ratios and different areas set in advance by clustering.

3. The deep learning-based automatic detection method for the inclination angle of traffic sign poles according to claim 2, wherein in the process of identifying traffic signs on the extracted feature maps of different sizes by using a plurality of groups of recognition boxes with different aspect ratios and different areas set in advance by clustering, the recognition box with the largest intersection over union is selected from the recognition boxes as the final recognition box for identifying traffic signs, so as to identify traffic signs.

4. The deep learning-based automatic detection method for the inclination angle of traffic sign poles according to claim 2, wherein different sizes of feature maps of the processed images at three different depths are extracted by using a DarkNet network.

5. The deep learning-based automatic detection method for the inclination angle of traffic sign poles according to claim 1, wherein the specific method for extracting feature maps from the processed pictures is: Extracting feature maps from the processed pictures by using a residual network.

6. The deep learning-based automatic detection method for the inclination angle of traffic sign poles according to claim 1, wherein in the process of extracting a plurality of local information on the extracted feature maps using a plurality of prediction lines with different starting points and different angles, the horizontal direction coordinates are divided vertically as follows:

7. The deep learning-based automatic detection method for the inclination angle of traffic sign poles according to claim 6, wherein the specific method for exchanging the local information corresponding to all prediction lines to obtain global information by using an attention mechanism is:

8. The deep learning-based automatic detection method for the inclination angle of traffic sign poles according to claim 7, wherein ​ ​ Divide the image by equal distance in vertical direction, record the vertical coordinate of the division line as Where N p is the number of division lines, and let the image height be h, then y i = i·(h / N p -1); ​ where (x, y) is the coordinate on the feature map, (x origin ,y origin ) is the coordinate on the traffic sign extracted from the image, δ back represents the reduction multiple of the image through feature extraction, and θ represents the angle of the intersection between the prediction line and the horizontal axis. The horizontal coordinate of the point set through which the prediction line passes the intersection with the division lines is represented. ​ ​ The obtained local information is input into a fully connected attention network, balanced by a softmax function, multiplied with the local information on other prediction lines, and finally added to generate a global information vector a global The formula is as follows: where i is the label of the current prediction line, j is the label of other prediction lines, a local Local information of a single prediction line, L att represents a fully connected attention network. ​ The specific method for judging whether the prediction line is a traffic sign post through local information and global information is: will be made by all with the matrix input classification model splicing the aggregated local and global information, and the classification model outputs the probability of whether each predicted line is a traffic sign pole; will be made by all with The matrix input regression model spliced with the aggregated local and global information outputs the horizontal offset and length l of each point on the predicted line. For the predicted line determined as a traffic sign pole in the classification model, the final result is output through non-maximum suppression. 9.The deep learning-based traffic sign post inclination angle automatic detection method according to claim 1, characterized in that, The specific method for collecting images is: The camera is arranged on the unmanned aerial vehicle to capture images.

Images