A multi-scale parallel feature fusion landmark detection method based on YOLO

Abstract

The present invention relates to a multi-scale parallel feature fusion road sign detection method based on YOLO, and belongs to the field of traffic informatization. The method comprises: obtaining an image to be detected; creating a deep convolutional neural network including a feature extraction network, a multi-scale parallel feature fusion module, and a target positioning and classification; using a trained road sign detection neural network model to locate and classify traffic signs on the image to be detected, and annotating the image, and outputting the classified and annotated image. The present invention adds a parallel structure and a three-level feature fusion on the basis of YOLO v4‑tiny, so that its detection capability of traffic signs is improved. The provided traffic sign detection method can realize automatic extraction of target features completely relying on a deep neural network model, realize automatic detection of traffic signs, and has the advantages of simple detection process, fast detection speed, and high accuracy.

Description

Technical Field

[0001] The invention belongs to the field of traffic informatization and relates to a YOLO-based multi-scale parallel feature fusion road sign detection method. Background Art

[0002] The field of Intelligent Transportation System (ITS) has gradually developed and become a hot topic of research. Traffic Sign Recognition (TSR) is an important part of ITS. TSR mainly includes two technologies: traffic sign detection and traffic sign recognition.

[0003] Object detection, as one of the basic problems in the field of machine vision, can be widely used in various fields. The purpose of object detection is to locate and identify single or multiple targets in the image, and return its spatial position coordinates and the classification results of the target. The application in the field of unmanned driving is perception technology, which can collect the environmental status of the vehicle through the on-board camera and make corresponding countermeasures. The detection of traffic signs occupies a relatively important position in environmental perception technology, because rules are above all else, and the role of traffic signs in the traffic field is to standardize the behavior of vehicles, so the importance of traffic sign detection is self-evident.

[0004] In recent decades, many researchers in the field of object detection have proposed a lot of methods, but the current detection algorithms cannot handle detection under complex scene conditions well.

[0005] Although traffic signs are relatively eye-catching in color and simple in shape, the complex environment in which they are located poses certain challenges to their detection and recognition. The following problems are waiting to be solved:

[0006] (1) Traffic signs are located in an environment that is not only covered by trees, but may also be blocked by pedestrians or buildings. In addition, rainy and foggy days may make traffic signs unclear. Under these conditions, the image contrast and other feature information collected are severely attenuated.

[0007] (2) The images for traffic sign detection are collected by the vehicle-mounted camera. However, due to the influence of weather and light, the collected images are severely distorted and cannot be easily detected and identified.

[0008] (3) Due to the diversity of weather temperature and environmental humidity, traffic signs will inevitably fade and become damaged, resulting in the collection of incomplete signs.

[0009] (4) The detection speed of two-stage algorithms such as Faster-RCNN cannot meet the real-time requirements due to the complexity of the algorithms. Although the single-stage algorithm has a faster detection speed, it has a poor detection effect on small targets. In addition, most traffic sign targets only occupy 1% of the image. At this time, it is difficult to strike a balance between speed and accuracy. Therefore, proposing a real-time small target detection algorithm based on complex environmental conditions is a major problem at present.

[0010] (5) Since it is mainly used in vehicle systems, a lightweight network is required. However, the detection performance of lightweight networks is slightly insufficient, so it is necessary to propose a lightweight network with excellent detection performance.

[0011] (6) Most traffic signs are compact. In this case, the algorithm cannot separate the positions of multiple signs well, and dense small target detection and positioning is still a major problem. Summary of the invention

[0012] In view of this, the object of the present invention is to provide a multi-scale parallel feature fusion landmark detection method based on YOLO.

[0013] In order to achieve the above object, the present invention provides the following technical solutions:

[0014] A multi-scale parallel feature fusion landmark detection method based on YOLO, the method comprising the following steps:

[0015] S1: Obtain the image to be detected and annotate it, and divide it into training set and test set;

[0016] S2: Create a deep convolutional neural network consisting of a feature extraction backbone network, a multi-scale feature fusion module, and a landmark detection module;

[0017] S3: Use the created training and test sets to train and test the deep convolutional neural network model;

[0018] S4: Use the trained landmark detection neural network model to locate the target in the image to be detected for landmark detection and classification.

[0019] Optionally, the S1 is specifically:

[0020] S11: using a driving recorder or a vehicle-mounted camera to collect an image in front of the vehicle, photographing and recording a traffic sign in the traffic scene, and obtaining a vehicle-mounted image containing the traffic sign;

[0021] S12: Screening the images obtained in step S11 to select traffic sign images under different scenes, weather conditions, angles and occlusion conditions;

[0022] S13: Name the filtered images according to certain specifications, manually annotate them using LabelImg, and generate an XML label file corresponding to the target position of the image;

[0023] S14: performing data enhancement operations on the obtained image, which may include inversion, contrast enhancement, cropping, mirroring, etc.;

[0024] S15: Divide the training set and the test set;

[0025] S16: Use the K-means clustering method to cluster the prior box sizes for the images in the training set, cluster out 6 prior box sizes, and save them as the yolo_anchors.txt file.

[0026] Optionally, the specific operation of S2 is as follows:

[0027] S21: Determine the feature extraction backbone network structure of the deep neural network, which is the CSPDarkNet53-tiny structure of the backbone network of the YOLOv4-Tiny network model. The backbone network consists of multiple convolutional layers and three residual blocks with a CSP structure. Two residual edges are added between the residual blocks to ensure that shallow features can be used when extracting deep-level feature information. After passing through the feature extraction backbone network, shallow, middle and deep feature maps are obtained. The detection performance of small objects in the shallow feature map is better. The features of the image to be detected are extracted through the backbone network.

[0028] S22: Based on the CSP_ResBlock module structure, the amount of calculation and parameters in forward and backward propagation are reduced, and the performance of gradients is enhanced. There are two outputs in each ResBlock module, one for storage and the other for input to the output of a deeper network;

[0029] S23: The outputs of the three branches of the backbone network in S21 are recorded as branch 1, branch 2 and branch 3 respectively. From a macroscopic point of view, the three branches are an FPN structure. The following is a detailed explanation of the three branch structures;

[0030] S24: Branch 3 of the network is the output after two downsampling convolution operations and three Res_Block modules from the original input. The output is convolved twice and then upsampled once before being merged with branch 2.

[0031] S25: Branch 2 of the network is the output after two downsampling convolution operations and two Res_Block modules are calculated from the original input. The output is concatenated with the feature map after upsampling in branch 3 and the number of channels is adjusted. After the convolution operation, it is divided into two branches, denoted as branch 21 and branch 22. Branch 21 is saved after upsampling, and branch 22 is input into YoloHead for positioning and classification.

[0032] S26: Branch 1 of the network is the output after two downsampling convolution operations and a Res_Block module calculation from the original input, followed by a parallel module, which contains two parallel branches, both of which use dilated convolution instead of ordinary convolution operations, and are only used in network structures with large shallow feature maps. If dilated convolution is used, it is guaranteed that the range covered by the convolution kernel is further expanded under the condition that the size of the convolution kernel remains unchanged; different receptive fields are obtained by dilated convolution, thereby obtaining features of different scales; dilated convolution has a parameter, the dilation rate r, which determines the interval between adjacent pixels involved in the convolution calculation; for a convolution kernel of size k×k, its actual receptive field size is:

[0033] k result =k+(k-1)×(r-1) (1)

[0034] The dilation rates of r=2 and r=3 are selected for convolution operation, and the two paths are concatenated and feature fused through a 1×1 convolution kernel.

[0035] The output after fusion is concatenated with the output after upsampling of branch 2, and the number of channels is adjusted. Finally, the concatenated output is input to YoloHead for positioning and classification.

[0036] S27: The network loss function consists of three parts: position loss, confidence loss and classification loss;

[0037] The position loss function uses CIoU, the function is as follows:

[0038]

[0039] where v and α are as follows:

[0040]

[0041]

[0042] d represents the Euclidean distance between two center points, c represents the diagonal distance of the closure, α is a parameter used for trade-offs, and v is a parameter for measuring the consistency of aspect ratio;

[0043] The confidence loss function uses cross entropy and is divided into two parts: obj and noobj, which represent target and non-target losses respectively, as follows:

[0044]

[0045] The classification loss function uses cross entropy as follows:

[0046]

[0047] Optionally, the specific operation of S3 is as follows:

[0048] Configure the network environment, select the Windows operating system and the Pytorch deep learning framework for training; set the network model parameters, set the appropriate BatchSize, input image size, prior box size, number of recognized types and type names, number of iterations, and initial learning rate according to the GPU memory size; use the training set to train the deep convolutional neural network, continuously optimize the parameters of the neural network through the back-propagation algorithm, reduce the value of the loss function, optimize the network, and achieve end-to-end training;

[0049] Load the test set data and input it into the trained network to evaluate the network model by calculating its precision, recall, F value, AP, and mAP value.

[0050] Optionally, the specific operation of S4 is as follows:

[0051] The trained road sign detection neural network model is used to locate and classify the image to be detected, and the confidence threshold is judged; the bounding box coordinate position, confidence and category probability of all prediction boxes that may contain traffic signs are obtained, and then the redundant detection boxes are removed through non-maximum suppression (NMS) to generate the final detection result, and the output is an image that draws the target location and its classification result.

[0052] The beneficial effect of the present invention is that: for the existing mainstream target detection framework, the network structure of the present invention is simple and has fewer parameters to train, so the network structure is relatively lightweight, can be run on lower-end terminals such as mobile devices, and can be applied in autonomous driving technology with good development prospects. Since traffic signs account for a small proportion of the image, it is more difficult to detect them. The present invention adds a parallel structure to the output of the shallow network to set different proportions of dilated convolutions, the purpose of which is to increase the receptive field so that it can detect small targets more effectively, thereby improving the effect of traffic sign detection.

[0053] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0055] Figure 1 This is a flow chart of a lightweight road sign detection method based on multi-scale parallel feature fusion according to the present invention;

[0056] Figure 2 This is a diagram of the backbone network structure of the present invention;

[0057] Figure 3 It is a structural diagram of the CSP_ResBlock module in the backbone network of the present invention;

[0058] Figure 4 This is a network structure diagram of the present invention, including a parallel module and a multi-scale fusion module. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0060] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0061] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0062] like Figure 1 As shown, a lightweight road sign detection method based on multi-scale parallel feature fusion includes the following steps:

[0063] S1. Create different weather conditions and collect images of different scenes including traffic signs on urban roads to ensure the adequacy and practicality of the data before processing the images.

[0064] The specific method of step S1 is as follows:

[0065] S11. Use a vehicle-mounted camera or a driving recorder to capture images of the scene in front of the vehicle to obtain a vehicle-mounted image containing a traffic sign.

[0066] S12. Screening and processing the images collected in step S11 to select traffic sign images in different scenes, weather conditions, angles, and occlusion conditions.

[0067] S13, unify the naming specifications of the filtered images, use LabelImg to label them, and generate XML format files with the same naming, which record the target position information and category information marked in the image.

[0068] S14. Perform data enhancement processing on the obtained images to obtain more images and annotation files based on the original images. Generally, data enhancement is performed through operations such as rotation, cropping, contrast enhancement, mirroring, and adding weather noise.

[0069] S15. Divide the data set obtained after step S14 into a training set, a validation set, and a test set in a ratio of 7:1:2.

[0070] S16. Use the K-means clustering method for the images in the training set to cluster 6 prior box sizes and save them as the yolo_anchors.txt file.

[0071] S2. Embed a parallel module in the YOLO v4-tiny framework, adjust the output branch of the backbone network, and perform multi-scale parallel feature fusion to obtain the network model of the present invention.

[0072] The specific operations of step S2 are as follows:

[0073] S21, determine the feature extraction backbone network structure of the deep neural network, which is the backbone network CSPDarkNet53-tiny structure of the YOLOv4-Tiny network model. The backbone network is mainly composed of Figure 2 As shown in , it is composed of multiple convolutional layers and three residual blocks with CSP structure. The present invention further modifies it by adding two residual edges between the residual blocks to ensure that shallow features can be used when extracting deep feature information. After passing through the feature extraction backbone network, shallow, middle and deep feature maps can be obtained. The detection performance of small objects in the shallow feature map is better because its feature information does not disappear due to multiple downsampling, but there is also the problem of insufficient feature information. The deep feature map has richer semantic information, but due to multiple downsampling operations, some feature information is lost, so it has a certain impact on the detection performance. The above is the feature extraction backbone network structure, and this network is used to extract the features of the image to be detected.

[0074] S22, Figure 3 It is a CSP_ResBlock module structure, which uses the CSP idea, mainly to reduce the amount of calculation and parameters during forward and backward propagation, and can also enhance the performance of gradients. In each ResBlock module, there are two outputs, one for storage and the other for input to the output of a deeper network.

[0075] S23, Figure 4 This is the overall network structure diagram of the present invention. The outputs of the three branches of the backbone network in S21 are respectively recorded as branch 1, branch 2, and branch 3. From a macro perspective, the three branches are an FPN structure. The three branch structures are explained in detail below.

[0076] S24, branch 3 of the network is the output after the original input undergoes two downsampling convolution operations and three Res_Block modules calculate. The output is convolved twice and then upsampled once before being merged with branch 2.

[0077] S25, branch 2 of the network is the output after two downsampling convolution operations and two Res_Block modules are calculated from the original input. The output is concatenated with the feature map after upsampling of branch 3 and the number of channels is adjusted. After the convolution operation, it is divided into two branches, recorded as branch 21 and branch 22. Branch 21 is saved after the upsampling operation, and branch 22 is input into YoloHead for positioning and classification.

[0078] S26, branch 1 of the network is the output after two downsampling convolution operations and a Res_Block module calculation from the original input, followed by a parallel module, including two parallel branches, both of which use hole convolution instead of ordinary convolution operations, and are only used in network structures with larger shallow feature maps, because for the commonly used 3×3 convolution kernel, the range that can be covered is limited to 9 pixel positions, and the strong correlation characteristics between pixels cannot be effectively utilized. If a hole convolution is used, it can be guaranteed that the range that the convolution kernel can cover can be further expanded under the condition that the size of the convolution kernel remains unchanged, thereby expanding the receptive field and better utilizing the correlation characteristics between pixels. Therefore, this parallel structure of the present invention uses hole convolution to obtain different receptive fields, thereby obtaining features of different scales. The hole convolution has a parameter, the hole rate r, which determines the interval between adjacent pixels involved in the convolution calculation. For a convolution kernel of size k×k, the actual receptive field size is:

[0079] k result =k+(k-1)×(r-1) (1)

[0080] The present invention selects to use the void ratio of r=2 and r=3 for convolution operation, and finally performs feature fusion through a 1×1 convolution kernel after splicing the two branches.

[0081] The output after fusion is concatenated with the output after upsampling of branch 2, and the number of channels is adjusted. Finally, it is input to YoloHead for positioning and classification.

[0082] S27, the loss function of the network, the loss function mainly consists of three parts, namely position loss, confidence loss, and classification loss.

[0083] The position loss function uses CIoU, the function is as follows:

[0084]

[0085] where v and α are as follows:

[0086]

[0087]

[0088] d represents the Euclidean distance between two center points, c represents the diagonal distance of the closure, α is a parameter used for trade-offs, and v is a parameter for measuring the consistency of the aspect ratio.

[0089] The confidence loss function uses cross entropy and is divided into two parts: obj and noobj, which represent target and non-target losses respectively, as follows:

[0090]

[0091] The classification loss function uses cross entropy as follows:

[0092]

[0093] Furthermore, the step S3 specifically includes:

[0094] According to the above steps, configure the network environment, select the Windows operating system and the Pytorch deep learning framework for training. Set the network model parameters, set the appropriate BatchSize, input image size, prior box size, number of recognized types and type name class_txt, number of iterations, initial learning rate and learning strategy according to the GPU memory size. Use the training set to train the deep convolutional neural network, continuously optimize the parameters of the neural network through the back propagation algorithm, reduce the value of the loss function, optimize the network, and achieve end-to-end training. And you need to use a graphics card type that supports cuda acceleration.

[0095] Load the test set data and input it into the trained network to evaluate the network model by calculating its precision, recall, F value, AP, and mAP value.

[0096] Furthermore, the step S4 specifically includes:

[0097] According to the above steps, the trained road sign detection neural network model is used to locate and classify the image to be detected, and the bounding box coordinates, confidence, and category probability of all prediction boxes that may contain traffic signs are obtained through the judgment of the confidence threshold. Then, the redundant detection boxes are removed through NMS (non-maximum suppression) to generate the final detection result, and the output is an image that draws the target location and its classification result.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-scale parallel feature fusion landmark detection method based on YOLO, characterized by: The method comprises the following steps: S1: Obtain the image to be detected and annotate it, and divide it into training set and test set; S2: Create a deep convolutional neural network consisting of a feature extraction backbone network, a multi-scale feature fusion module, and a landmark detection module; S3: Use the created training and test sets to train and test the deep convolutional neural network model; S4: Use the trained landmark detection neural network model to locate the target in the image to be detected, so as to detect and classify the landmarks; The S1 is specifically: S11: using a driving recorder or a vehicle-mounted camera to collect an image in front of the vehicle, photographing and recording a traffic sign in the traffic scene, and obtaining a vehicle-mounted image containing the traffic sign; S12: Screening the images obtained in step S11 to select traffic sign images under different scenes, weather conditions, angles and occlusion conditions; S13: Name the filtered images according to certain specifications, manually annotate them using LabelImg, and generate an XML label file corresponding to the target position of the image; S14: performing data enhancement operations on the obtained image, including inversion, contrast enhancement, cropping and mirroring operations; S15: Divide the training set and the test set; S16: Use the K-means clustering method to cluster the prior box sizes for the images in the training set, cluster out 6 prior box sizes, and save them as the yolo_anchors.txt file; The specific operation of S2 is as follows: S21: Determine the feature extraction backbone network structure of the deep neural network, which is the CSPDarkNet53-tiny structure of the backbone network of the YOLOv4-Tiny network model. The backbone network consists of multiple convolutional layers and three residual blocks with a CSP structure. Two residual edges are added between the residual blocks to ensure that shallow features can be used when extracting deep-level feature information. After passing through the feature extraction backbone network, shallow, middle and deep feature maps are obtained. The detection performance of small objects in the shallow feature map is better. The features of the image to be detected are extracted through the backbone network. S22: Based on the CSP_ResBlock module structure, the amount of calculation and parameters in forward and backward propagation are reduced, and the performance of gradients is enhanced. There are two outputs in each ResBlock module, one for storage and the other for input to the output of a deeper network; S23: The outputs of the three branches of the backbone network in S21 are recorded as branch 1, branch 2 and branch 3 respectively. From a macroscopic point of view, the three branches are an FPN structure. The following is a detailed explanation of the three branch structures; S24: Branch 3 of the network is the output after two downsampling convolution operations and three Res_Block modules from the original input. The output is convolved twice and then upsampled once before being merged with branch 2. S25: Branch 2 of the network is the output after two downsampling convolution operations and two Res_Block modules are calculated from the original input. The output is concatenated with the feature map after upsampling in branch 3 and the number of channels is adjusted. After the convolution operation, it is divided into two branches, denoted as branch 21 and branch 22. Branch 21 is saved after upsampling, and branch 22 is input into YoloHead for positioning and classification. S26: Branch 1 of the network is the output after two downsampling convolution operations and a Res_Block module calculation from the original input, followed by a parallel module, which contains two parallel branches, both of which use dilated convolution instead of ordinary convolution operations, and are only used in network structures with large shallow feature maps. If dilated convolution is used, it is guaranteed that the range covered by the convolution kernel is further expanded under the condition that the size of the convolution kernel remains unchanged; different receptive fields are obtained by dilated convolution, thereby obtaining features of different scales; dilated convolution has a parameter, the dilation rate r, which determines the interval between adjacent pixels involved in the convolution calculation; for a convolution kernel of size k×k, its actual receptive field size is: k result =k+(k-1)×(r-1)(1) The dilation rates of r=2 and r=3 are selected for convolution operation, and the two paths are concatenated and feature fused through a 1×1 convolution kernel. The output after fusion is concatenated with the output after upsampling of branch 2, and the number of channels is adjusted. Finally, the concatenated output is input to YoloHead for positioning and classification. S27: The network loss function consists of three parts: position loss, confidence loss and classification loss; The position loss function uses LIoU, and the function is as follows: where v and α are as follows: d represents the Euclidean distance between two center points, c represents the diagonal distance of the closure, α is a parameter used for trade-offs, and v is a parameter for measuring the consistency of aspect ratio; The confidence loss function uses cross entropy and is divided into two parts: obj and noobj, which represent target and non-target losses respectively, as follows: The classification loss function uses cross entropy as follows:

2. A YOLO-based multi-scale parallel feature fusion landmark detection method according to claim 1, characterized in that: The specific operation of S3 is as follows: Configure the network environment, select the Windows operating system and the Pytorch deep learning framework for training; set the network model parameters, set the appropriate BatchSize, input image size, prior box size, number of recognized types and type names, number of iterations, and initial learning rate according to the GPU memory size; use the training set to train the deep convolutional neural network, continuously optimize the parameters of the neural network through the back-propagation algorithm, reduce the value of the loss function, optimize the network, and achieve end-to-end training; Load the test set data and input it into the trained network to evaluate the network model by calculating its precision, recall, F value, AP, and mAP value.

3. A YOLO-based multi-scale parallel feature fusion landmark detection method according to claim 2, characterized in that: The specific operation of S4 is as follows: The trained road sign detection neural network model is used to locate and classify the image to be detected, and the confidence threshold is judged; the bounding box coordinate position, confidence and category probability of all prediction boxes that may contain traffic signs are obtained, and then the redundant detection boxes are removed through non-maximum suppression (NMS) to generate the final detection result, and the output is an image that draws the target location and its classification result.

Images