Traffic cone recognition method and related device

Abstract

The application discloses a traffic cone recognition method and related equipment, the method comprises the following steps: acquiring image data collected by a vehicle, inputting the image data into a pre-trained cone detection classification model and a cone classification model respectively; based on the cone detection classification model, calculating the cone position and the first cone type according to the image data, and based on the cone classification model, calculating the second cone type according to the image data; fusing the first cone type and the second cone type to obtain a target cone type; according to the cone position and the target cone type, controlling the vehicle to automatically plan a driving route. In order to improve the overall recognition accuracy of the model, the algorithm adopts a two-stage model design method, namely detection + classification. The first-stage network performs position regression, the second-stage network classifies the results of the first-stage regression, and finally the results of the two are fused to obtain high-precision detection results. Through the model, the traffic cone recognition accuracy can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a traffic cone recognition method, system, vehicle, and computer-readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence, machine learning and big data, autonomous driving technology has been greatly improved. Currently, many car manufacturers, technology companies and startups have launched autonomous vehicles and begun road testing.

[0003] Autonomous driving technology needs to ensure the safety and reliability of vehicles in various complex environments. Through the development of artificial intelligence, cars will better understand the driver's needs and be able to better identify road environments, such as obstacles like traffic cones.

[0004] Currently, common methods for identifying traffic cones in automated driving processes include image differentiation, preset thresholds, ternary processing, and Hough transform. However, these methods are not accurate enough in scenarios with many types and colors of traffic cones, and are greatly affected by the external environment. For example, different weather conditions and overlapping scenarios can reduce the recognition rate of traffic cones. In addition, the data collected often varies greatly, which can also lead to low accuracy in traffic cone recognition.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this application is to provide a traffic cone recognition method, system, vehicle, and computer-readable storage medium, aiming to solve the problem of low accuracy in traffic cone recognition during autonomous driving in the prior art.

[0007] The first aspect of this application provides a traffic cone recognition method, comprising the following steps: acquiring image data collected by a vehicle; inputting the image data into a pre-trained traffic cone detection and classification model and a traffic cone classification model; calculating the traffic cone position and a first traffic cone type based on the image data using the traffic cone detection and classification model, and calculating a second traffic cone type based on the image data using the traffic cone classification model; fusing the first traffic cone type and the second traffic cone type to obtain a target traffic cone type; and controlling the vehicle to automatically plan a driving route based on the traffic cone position and the target traffic cone type.

[0008] Based on the above technical means, the embodiments of this application can process the image data collected by the vehicle based on two trained models, output the position and type of the road cone in the image data, and control the vehicle to automatically plan the driving route based on the position and type. The model algorithm adopts a two-stage model design method, namely detection + classification. The first-stage network performs position regression, the second-stage network classifies the result of the first-stage regression, and finally fuses the results of the two to obtain a high-precision detection result. Through the model, the accuracy of traffic road cone recognition can be effectively improved.

[0009] Optionally, in one embodiment of this application, the training process of the traffic cone detection classification model specifically includes acquiring images in a driving scene, marking the positions and categories of various traffic cones in the images to obtain first training data; after image enhancement of the first training data, inputting it into a one-stage detection network for training to obtain network output values, calculating a loss function based on the network output values ​​and the labels in the first training data, and adjusting the model parameters based on the loss function to obtain the traffic cone detection classification model.

[0010] Based on the above technical means, the embodiments of this application can provide a training process for a one-stage network, namely a road cone detection and classification model. Image enhancement is performed on the acquired image data, which can greatly enhance the robustness of the model. The enhanced data is input into the detection network for training, and various loss functions are calculated based on the network output value to continuously optimize the road cone detection and classification model. This allows the trained model to have high accuracy and obtain accurate road cone locations and first road cone types.

[0011] Optionally, in one embodiment of this application, the loss function includes a detection loss function and a classification loss function; the calculation process of the detection loss function is as follows: normalize the network output value to obtain the predicted probability, and then input the probability into the formula. The detection loss function is calculated in the following steps; the calculation process for the classification loss function is as follows: the network output value is normalized to obtain the predicted probability, and then the probability is input into the formula. The classification loss function is calculated in the middle; where, Indicates the probability of prediction. This represents the true label value of the i-th training data in the first training data. Indicates the actual number of tags. The number of training data points is indicated; the detection loss function is used to calculate the loss of the detection boxes in the network output value, and the classification loss function is used to calculate the loss of the classification names in the network output value.

[0012] Based on the above technical means, this application embodiment calculates the loss of the detection box and the loss of the classification name in the network output value separately, which facilitates subsequent data calculation and fusion; and when calculating the loss, the Sigmoid function is used for normalization processing. The output range of the Sigmoid function is 0 to 1. Since the output value in this application is limited to 0 to 1, the Sigmoid function is very suitable as the probability output of the network, which can improve the accuracy and efficiency of model training.

[0013] Optionally, in one embodiment of this application, the step of image-enhancing the first training data and then inputting it into a one-stage detection network for training to obtain network output values ​​specifically includes: image-enhancing the first training data to obtain enhanced first training data; inputting the enhanced first training data into the backbone network of the one-stage detection network for image feature extraction and fusion to obtain a target feature map; and inputting the target feature map into an intermediate layer for downsampling and upsampling to obtain network output values.

[0014] According to the above technical means, in the training model, the first training data is subjected to image enhancement processing to obtain enhanced first training data. The enhanced first training data is then input into the backbone network for image feature extraction and fusion. Feature maps output by multiple different pooling layers are fused. The target feature map is then input into the intermediate layer for downsampling and upsampling. While semantic information is transmitted from high dimension to low dimension, semantic information is also transmitted from low dimension to high dimension again, thereby enhancing the semantic expression and localization capabilities at multiple scales.

[0015] Optionally, in one embodiment of this application, the backbone network includes a CBS module, a BottleNeck_1 module, a BottleNeck_2 module, a C3_X_xn module, and an SPP module; the CBS module is used to extract image features to obtain a first feature map; the BottleNeck_1 module and the BottleNeck_2 module are composed of several CBS modules and are used to fuse the first feature map to obtain a second feature map; the C3_X_xn module is used to perform memory fusion on the second feature map obtained by the BottleNeck_1 module and the BottleNeck_2 module to obtain a third feature map; the SPP module is used to input the third feature map into multiple different pooling layers for dimensionality reduction and fuse the dimensionality reduction results to obtain a target feature map.

[0016] Based on the above technical means, the specific constituent modules of the backbone network and the function of each module are given in the embodiments of this application. There are connections between each module. Through the above modules, feature extraction and fusion of image data can be realized, and then input into the pooling layer for dimensionality reduction processing, which simplifies the network complexity, reduces the amount of computation, reduces memory consumption, and fuses the feature maps output by multiple different pooling layers.

[0017] Optionally, in one embodiment of this application, the training process of the traffic cone classification model specifically includes: acquiring images of a driving scene, selecting several regions in the images, and labeling the category of each traffic cone in the several regions to obtain second training data; performing data augmentation on the second training data to obtain augmented second training data; inputting the augmented second training data into a two-stage classification network for training, and adjusting the model parameters according to the focus loss function to obtain the traffic cone classification model; wherein, the calculation process of the focus loss function is as follows: ,in, For probability scores, To adjust the proportion of easily separable sample weights.

[0018] Based on the above technical means, this application embodiment provides a training process for a two-stage network, namely a road cone classification model. The original image is cropped and used as training data. Data augmentation is performed on the training data to increase the diversity of samples. FocalLoss is used as a penalty term to reduce the impact of sparse classes on model training, improve classification accuracy, enable the model to focus more on training difficult samples, and improve the model's classification accuracy.

[0019] Optionally, in one embodiment of this application, the two-stage classification network is a simplified Mobilenet-v3 classification network, which reduces the number of convolutional layers to a preset number to reduce the parameters of the simplified Mobilenet-v3 classification network.

[0020] Based on the above technical means, this application embodiment uses a simplified version of Mobilenet-v3 as a two-stage network. Since the training difficulty of classification models is usually relatively simple, the number of convolutional layers in the network is reduced to a preset number to reduce the network parameters. Therefore, a simplified Mobilenet-v3 is used for training with the assistance of data augmentation and FocalLoss to improve the accuracy of the classification model.

[0021] A second aspect of this application provides a traffic cone recognition system, comprising: a data acquisition module for acquiring image data collected by a current vehicle and inputting the image data into a pre-trained traffic cone detection and classification model and a traffic cone classification model; a model calculation module for calculating the position of a traffic cone and a first traffic cone type in the image data based on the traffic cone detection and classification model, and calculating a second traffic cone type in the image data based on the traffic cone classification model; a result fusion module for fusing the first traffic cone type and the second traffic cone type to obtain a target traffic cone type; and a driving planning module for controlling the vehicle to automatically plan a driving route based on the traffic cone position and the target traffic cone type.

[0022] Optionally, in one embodiment of this application, the data acquisition module includes: a camera unit for acquiring image data collected by the current vehicle; and a data input unit for inputting the image data into a pre-trained traffic cone detection and classification model and a traffic cone classification model, respectively.

[0023] Optionally, in one embodiment of this application, the model calculation module includes: a road cone position and first road cone type calculation unit, used to calculate the road cone position and first road cone type in the image data based on the road cone detection classification model; and a second road cone type calculation unit, used to calculate the second road cone type in the image data based on the road cone classification model.

[0024] Optionally, in one embodiment of this application, the result fusion module includes: a result fusion unit, used to fuse the first traffic cone type and the second traffic cone type to obtain the target traffic cone type.

[0025] Optionally, in one embodiment of this application, the planning driving module includes: a planning driving unit, used to control the vehicle to automatically plan a driving route based on the location of the traffic cone and the type of the target traffic cone.

[0026] A third aspect of this application provides a vehicle, the vehicle including: a memory, a processor, and a traffic cone recognition program stored in the memory and executable on the processor, wherein when the traffic cone recognition program is executed by the processor, it implements the steps of the traffic cone recognition method as described in the above embodiments.

[0027] A fourth aspect of this application provides a computer-readable storage medium storing a traffic cone recognition program, which, when executed by a processor, implements the steps of the traffic cone recognition method as described in the above embodiments.

[0028] The beneficial effects of this application are:

[0029] (1) The embodiments of this application can process the image data collected by the vehicle based on two trained models, output the position and type of the road cone in the image data, and control the vehicle to automatically plan the driving route based on the position and type. The model algorithm adopts a two-stage model design method, namely detection + classification. The first-stage network performs position regression, the second-stage network classifies the result of the first-stage regression, and finally the results of the two are fused to obtain a high-precision detection result. Through the model, the accuracy of traffic road cone recognition can be effectively improved.

[0030] (2) The embodiments of this application provide a training process for a one-stage network, namely a road cone detection and classification model. The acquired image data is enhanced to greatly improve the robustness of the model. The enhanced data is input into the detection network for training, and various loss functions are calculated based on the network output value to continuously optimize the road cone detection and classification model, so that the model after training can have high accuracy and obtain accurate road cone positions and first road cone types.

[0031] (3) The embodiments of this application provide a training process for a two-stage network, namely a road cone classification model. The original image is cropped and used as training data. Data augmentation is performed on the training data to increase the diversity of samples. Focalloss is used as a penalty term to reduce the impact of sparse classes on model training, improve classification accuracy, enable the model to focus more on training difficult samples, and improve the classification accuracy of the model.

[0032] (4) In this embodiment of the application, the detection and classification results are finally fused as the final output. The purpose is to ensure high recall and high accuracy of traffic cones in real-time vehicle detection tasks, so that the model output results can be applied in the autonomous driving route planning process, effectively improve the traffic cone recognition rate, improve the vehicle's safety performance in different environments, and improve the driver's driving experience.

[0033] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of a preferred embodiment of the traffic cone recognition method of this application;

[0036] Figure 2 This is a schematic diagram of the operation of the traffic cone detection and classification model and the traffic cone classification model in the whole vehicle in the traffic cone recognition method of this application;

[0037] Figure 3 This is a flowchart illustrating the calculation process of the traffic cone position and the target traffic cone type in the traffic cone identification method of this application;

[0038] Figure 4 This is a schematic diagram of the entire model training process in the traffic cone recognition method of this application;

[0039] Figure 5 This is a schematic diagram of the calculation process of the CBS module in the traffic cone recognition method of this application;

[0040] Figure 6 This is a schematic diagram of the calculation process of the BottleNeck_1 module in the traffic cone recognition method of this application;

[0041] Figure 7 This is a schematic diagram of the calculation process of the BottleNeck_2 module in the traffic cone recognition method of this application;

[0042] Figure 8 This is a schematic diagram of the calculation process of the C3_X_xn module in the traffic cone recognition method of this application;

[0043] Figure 9 This is a schematic diagram of the calculation process of the SPP module in the traffic cone recognition method of this application;

[0044] Figure 10 This is a schematic diagram of the calculation process of the FPN module in the traffic cone recognition method of this application;

[0045] Figure 11 This is a schematic diagram of the calculation process of the PAN module in the traffic cone recognition method of this application;

[0046] Figure 12 This is an example diagram of the model output results in the traffic cone recognition method of this application;

[0047] Figure 13 This is a schematic diagram of a preferred embodiment of the traffic cone recognition method of this application;

[0048] Figure 14 This is a structural schematic diagram of a preferred embodiment of the vehicle described in this application.

[0049] Among them, 10 Traffic cone recognition system; 100 Data acquisition module, 200 Model calculation module, 300 The results fusion module and 400 Driving planning module; 501 Memory, 502 Processor and 503 Communication interface. Detailed Implementation

[0050] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0051] The following describes a traffic cone recognition method and related equipment according to embodiments of this application with reference to the accompanying drawings. Addressing the problem of low accuracy in traffic cone recognition during autonomous driving mentioned in the background art, a traffic cone recognition method and related equipment are provided. In this method: image data collected by the vehicle is acquired; the image data is input into a pre-trained traffic cone detection and classification model and a traffic cone classification model, respectively; based on the traffic cone detection and classification model, the traffic cone position and a first traffic cone type are calculated from the image data; based on the traffic cone classification model, a second traffic cone type is calculated from the image data; the first traffic cone type and the second traffic cone type are fused to obtain a target traffic cone type; based on the traffic cone position and the target traffic cone type, the vehicle is controlled to automatically plan a driving route. To improve the overall recognition accuracy of the model, the algorithm adopts a two-stage model design: detection + classification. A first-stage network performs position regression, a second-stage network classifies the regression results from the first stage, and finally, the results of both are fused to obtain a high-precision detection result. This model effectively improves the accuracy of traffic cone recognition.

[0052] Specifically, Figure 1 This is a flowchart illustrating a traffic cone recognition method provided in an embodiment of this application.

[0053] like Figure 1 As shown, the traffic cone recognition method includes the following steps:

[0054] In step S101, image data collected by the vehicle is acquired, and the image data is input into the pre-trained road cone detection and classification model and road cone classification model, respectively.

[0055] It is understandable that, such as Figure 2 As shown, images of the area around the vehicle are collected by the vehicle's onboard camera to obtain image data, and the image data is then input into a pre-trained traffic cone detection and classification model and a traffic cone classification model, respectively.

[0056] Both the traffic cone detection and classification model and the traffic cone classification model are pre-trained models. The traffic cone detection and classification model is mainly responsible for detecting the location of the traffic cone in the image and calculating the first traffic cone type of the traffic cone in the location box. The traffic cone classification model is responsible for calculating the second traffic cone type.

[0057] The image data can include different types of traffic cone data, as well as common road isolation tools such as traffic cones, traffic posts, and guardrails.

[0058] In step S102, the location of the traffic cone and the first type of traffic cone are calculated based on the image data according to the traffic cone detection and classification model, and the second type of traffic cone is calculated based on the image data according to the traffic cone classification model.

[0059] like Figure 3 As shown, a flowchart illustrating the calculation process of the traffic cone position and the target traffic cone type is presented.

[0060] Understandably, the entire model training process is as follows: Figure 4 As shown, the basic steps for training a model include data preparation, model selection, model training, model evaluation, and model tuning. Before training the model, it is necessary to prepare the data for training. The quality and quantity of data have a significant impact on the training effect of the model. Data preparation includes steps such as data collection, data cleaning, data preprocessing, and data partitioning. Data collection can be achieved through methods such as web scraping and database queries. Data cleaning can remove duplicate data, handle missing values ​​and outliers, etc. Data preprocessing includes operations such as feature selection, feature transformation, and feature normalization. Finally, the data is divided into training and test sets for model training and evaluation. These steps are interconnected and require repeated iterations until a satisfactory model is obtained. Finally, the obtained model is deployed and used.

[0061] The traffic cone detection and classification model in this application is now being discussed. Figure 4 The key steps in the process will be explained.

[0062] Images of driving scenes are collected, and the locations and categories of various traffic cones are marked in the images to obtain the first training data. After image augmentation of the first training data, it is input into a one-stage detection network for training to obtain the network output value. The loss function is calculated based on the network output value and the labels in the first training data. The model parameters are adjusted according to the loss function to obtain the traffic cone detection and classification model.

[0063] The process of acquiring images of driving scenarios and marking the locations and categories of various traffic cones in the images to obtain the first training data is as follows: A data acquisition vehicle (a type of professional tool used to collect and record data such as geographic location, images, and sound; data acquisition vehicles are usually equipped with a variety of sensors and detection devices, including GPS, cameras, laser ranging systems, radar, sonar, meteorological instruments, etc., which can measure and analyze various environmental conditions) is used to acquire images of driving scenarios containing traffic cones, traffic posts, and guardrails, and 2D bounding boxes are used to mark the specific locations and categories in the images to provide the first training data for the detection and classification model.

[0064] The first-stage detection network used is YOLOX-Tiny (YOLOX is an open-source deep learning framework, and YOLOX-Tiny is a project under YOLOX).

[0065] When performing image augmentation on the first training data, image augmentation methods such as Mosaic and Mixup (without limitation) can be used to improve the robustness of the model. The Mosaic method stitches images together using random scaling, cropping, and arrangement, and it performs well in detecting small objects. The Mixup method first selects an image, fills its sides, and scales it to a certain size, denoted as image a. Then, it randomly selects another image, fills its top and bottom, and scales it to the size of image a, denoted as image b. Finally, it sets a fusion coefficient and weightedly fuses images a and b to obtain a new image.

[0066] As can be seen, the embodiments of this application can provide a training process for a one-stage network, namely a road cone detection and classification model. Image enhancement is performed on the acquired image data, which can greatly enhance the robustness of the model. The enhanced data is input into the detection network for training, and various loss functions are calculated based on the network's output value. The road cone detection and classification model is continuously optimized so that the trained model can have high accuracy and obtain accurate road cone locations and first road cone types.

[0067] Further, a loss function is calculated based on the network output value and the labels in the first training data; the loss function includes the BCEWithLogitsLoss loss function (binary cross-entropy loss function) and the BCELoss classification loss function (classification cross-entropy loss function); the calculation process of the detection loss function is as follows: the network output value is normalized to obtain the predicted probability, and then the probability is input into the formula. The detection loss function is calculated in the middle.

[0068] It is understandable that calculating BCEWithLogitsLoss is equivalent to calculating Sigmoid+BCELoss, that is, first normalizing the network output value to the range [0-1], and then using BCELoss to calculate the loss. The reason is that the output range of the Sigmoid function is 0 to 1. Since the output value is limited to 0 to 1, it normalizes the output of each neuron. Since the probability range is 0 to 1, the Sigmoid function is very suitable as the probability output of the network.

[0069] The calculation process of the classification loss function is as follows: normalize the network output value to obtain the predicted probability, and then input the probability into the formula. The classification loss function is calculated in the middle.

[0070] It should be noted that the formula Belongs to formula In the special case of N=2, if the probability of belonging to the first class is y, then the probability of belonging to the second class is naturally (1-y).

[0071] in, Indicates the probability of prediction. This represents the true label value of the i-th training data in the first training data. Indicates the actual number of tags. This indicates the number of training data points.

[0072] The detection loss function is used to calculate the loss of the detection boxes in the network output value, and the classification loss function is used to calculate the loss of the classification names in the network output value.

[0073] Furthermore, embodiments of this application provide a calculation process for the network output value, including: performing image enhancement on the first training data to obtain enhanced first training data; inputting the enhanced first training data into the backbone network of a one-stage detection network for image feature extraction and fusion to obtain a target feature map; and inputting the target feature map into an intermediate layer for downsampling and upsampling to obtain the network output value.

[0074] The backbone is the main component of the model, usually a convolutional neural network (CNN) or a residual neural network (ResNet). The backbone is responsible for extracting features from the input image for subsequent processing and analysis. The backbone usually has many layers and many parameters, which can extract high-level feature representations of the image.

[0075] The backbone network described in this application includes a CBS module, a BottleNeck_1 module, a BottleNeck_2 module, a C3_X_xn module, and an SPP module. The CBS module is used to extract image features to obtain a first feature map. The BottleNeck_1 module and the BottleNeck_2 module are composed of several CBS modules and are used to fuse the first feature map to obtain a second feature map. The C3_X_xn module is used to perform memory fusion on the second feature map obtained by the BottleNeck_1 module and the BottleNeck_2 module to obtain a third feature map. The SPP module is used to input the third feature map into multiple different pooling layers for dimensionality reduction and fuse the dimensionality reduction results to obtain a target feature map.

[0076] Specifically, the composition and function of each module in the backbone network are described below, such as... Figure 5 As shown, the CBS module includes a 3*3 convolutional layer (each convolutional layer in a convolutional neural network consists of several convolutional units, and the parameters of each convolutional unit are optimized through backpropagation. The purpose of convolution is to extract different features from the input. The first convolutional layer may only extract some low-level features such as edges, lines, and corners, while more layers of the network can iteratively extract more complex features from low-level features), a batch normalization module (BN), and an activation function (a function added to an artificial neural network to help the network learn complex patterns in the data. Similar to neuron-based models in the human brain, the activation function ultimately determines what to fire to the next neuron) SiLU. The CBS module is used to extract image features from the enhanced first training data to obtain the first feature map.

[0077] like Figure 6 and Figure 7 As shown, the BottleNeck_1 module and the BottleNeck_2 module are composed of several CBS modules, which are used to perform feature fusion on the first feature map extracted by the CBS module (feature fusion refers to merging the features extracted by different neural networks or different branches in the same neural network to improve the performance of the model) to obtain the second feature map.

[0078] like Figure 8 As shown, the C3_X_xn module consists of three CBS modules and one BottleNeck_X_xn, where BottleNeck_X_xn refers to either the BottleNeck_1 module or the BottleNeck_2 module. The C3_X_xn module is used to perform memory fusion on the second feature map obtained by the BottleNeck_1 module and the BottleNeck_2 module.

[0079] like Figure 9 As shown, the SPP module consists of multiple pooling layers (also known as sampling layers, which follow the convolutional layers and are composed of multiple feature surfaces, each of which corresponds to a feature surface in the previous layer, without changing the number of feature surfaces). It is used to input the third feature map into multiple different pooling layers for dimensionality reduction and to fuse the dimensionality reduction results to obtain the target feature map. This can solve the problem of needing to fix the size of the input image in the network, which leads to unnecessary accuracy loss.

[0080] The specific constituent modules of the backbone network and the function of each module are given in the embodiments of this application. There are connections between each module. Through the above modules, feature extraction and fusion of image data can be realized, and then input into the pooling layer for dimensionality reduction processing, which simplifies the network complexity, reduces the amount of computation, reduces memory consumption, and fuses the feature maps output by multiple different pooling layers.

[0081] like Figure 10 and Figure 11 As shown, after obtaining the target feature map, the target feature map is input into the intermediate layer Neck for downsampling and upsampling to obtain the network output value.

[0082] Understandably, the intermediate layer Neck in this application adopts an FPN+PAN structure. FPN is used to transmit semantic information from high dimension to low dimension; PAN is used to transmit semantic information from low dimension to high dimension again. Deep feature maps carry stronger semantic features and weaker localization information, while shallow feature maps carry stronger positional information and weaker semantic features. FPN transmits the semantic features from deep layer to shallow layer, thereby enhancing semantic expression at multiple scales, while PAN, on the contrary, transmits the localization information from shallow layer to deep layer, enhancing localization capability at multiple scales.

[0083] As can be seen, in this embodiment, the intermediate layer Neck adopts an FPN+PAN structure, which inputs the target feature map into the intermediate layer for downsampling and upsampling. While the high dimension transmits semantic information to the low dimension, the low dimension also transmits semantic information to the high dimension again, thereby enhancing the semantic expression and localization capabilities at multiple scales.

[0084] Furthermore, such as Figure 3 and Figure 4As shown, the training process of the traffic cone classification model in this embodiment is as follows: Images of a driving scene are acquired; several regions are selected within the images; and the category of each traffic cone is labeled within each region to obtain second training data; the second training data is augmented to obtain augmented second training data; the augmented second training data is input into a two-stage classification network for training, and the model parameters are adjusted according to the FocalLoss focus loss function to obtain the traffic cone classification model; wherein, the calculation process of the FocalLoss focus loss function is as follows: ,in, For probability scores, To adjust the proportion of easily separable sample weights.

[0085] Understandably, when When =0, Equal to the standard cross-entropy function; when When >0, because >=0, therefore The loss is less than or equal to the standard cross-entropy loss; the higher the probability of an easily separable sample, the closer it tends to 0, while the lower the probability of a difficult-to-separate sample, the closer it tends to 1; therefore, using... The focus loss function adjusts the model parameters, which is equivalent to attenuating the weights of easily classified samples and highlighting the weights of difficult samples. This allows the model to focus more on training the difficult samples and improve the model's classification accuracy.

[0086] It should be noted that the two-stage classification network is a simplified version of the Mobilenet-v3 classification network (a lightweight deep learning network structure). The simplified Mobilenet-v3 classification network reduces the number of convolutional layers to a preset number to reduce the parameters of the simplified Mobilenet-v3 classification network.

[0087] As can be seen, this application uses a simplified version of Mobilenet-v3 as a two-stage network. Since the training difficulty of classification models is usually relatively simple, the number of convolutional layers in the network is reduced to a preset number to reduce the network parameters. Therefore, a simplified Mobilenet-v3 is used, supplemented by data augmentation, and Focal loss is used as a penalty term to reduce the impact of sparse classes on model training, thereby improving classification accuracy and the precision of the classification model.

[0088] In step S103, the first traffic cone type and the second traffic cone type are fused to obtain the target traffic cone type.

[0089] To improve the overall recognition accuracy of the model, this application adopts a two-stage model design approach, namely detection + classification. The first-stage network mainly performs location regression, the second-stage network classifies the image data, and finally the results of the two are combined to obtain a high-precision detection result.

[0090] The following describes the entire process of obtaining the location of the traffic cone and the type of the target traffic cone in the traffic cone identification method of this application, as follows: Figure 3 As shown:

[0091] Step S1: Input the image data collected by the vehicle;

[0092] Step S2: Input the image data into the road cone detection and classification model for calculation;

[0093] Step S3: Obtain the location of the road cone based on the road cone detection and classification model;

[0094] Step S4: Obtain the first traffic cone type based on the traffic cone detection and classification model;

[0095] Step S5: Perform original image cropping on the image data;

[0096] Step S6: Input the image after cropping the original image into the road cone classification model for calculation;

[0097] Step S7: Obtain the second traffic cone type based on the traffic cone classification model;

[0098] Step S8: Merge the first traffic cone type and the second traffic cone type to obtain the target traffic cone type;

[0099] Step S9: Output the location of the traffic cone and the type of the target traffic cone.

[0100] In step S104, the vehicle is controlled to automatically plan a driving route based on the location of the traffic cone and the type of the target traffic cone.

[0101] It is understandable that, such as Figure 12 As shown, the final image output from the model will contain the position of the traffic cone detection box and the category name of the traffic cone.

[0102] like Figure 2 As shown, the vehicle terminal will control the vehicle to automatically plan a safe and reasonable driving route based on the position of the traffic cone detection box and the traffic cone category name contained in the output image, so as to avoid obstacles or stop moving forward.

[0103] In summary, the embodiments of this application can process the image data collected by the vehicle based on two trained models, output the position and type of the traffic cone in the image data, and control the vehicle to automatically plan the driving route based on the position and type. The model algorithm adopts a two-stage model design, namely detection + classification. The first-stage network performs position regression, the second-stage network classifies the result of the first-stage regression, and finally the results of the two are fused to obtain a high-precision detection result. Through the model, the accuracy of traffic cone recognition can be effectively improved.

[0104] Next, the traffic cone recognition system proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0105] Figure 13 This is a block diagram of a traffic cone recognition system according to an embodiment of this application.

[0106] like Figure 13 As shown, the traffic cone recognition system 10 includes: a data acquisition module 100, a model calculation module 200, a result fusion module 300, and a driving planning module 400.

[0107] Specifically, the data acquisition module is used to acquire image data collected by the current vehicle and input the image data into a pre-trained traffic cone detection and classification model and a traffic cone classification model, respectively; the model calculation module is used to calculate the traffic cone position and the first traffic cone type in the image data based on the traffic cone detection and classification model, and to calculate the second traffic cone type in the image data based on the traffic cone classification model; the result fusion module is used to fuse the first traffic cone type and the second traffic cone type to obtain the target traffic cone type; and the driving planning module is used to control the vehicle to automatically plan a driving route according to the traffic cone position and the target traffic cone type.

[0108] Optionally, in one embodiment of this application, the data acquisition module includes a camera unit and a data input unit.

[0109] The camera unit is used to acquire image data collected by the current vehicle.

[0110] The data input unit is used to input the image data into the pre-trained traffic cone detection and classification model and the traffic cone classification model, respectively.

[0111] Optionally, in one embodiment of this application, the model calculation module includes: a traffic cone position and first traffic cone type calculation unit and a second traffic cone type calculation unit.

[0112] The road cone position and first road cone type calculation unit is used to calculate the road cone position and first road cone type in the image data based on the road cone detection and classification model.

[0113] The second road cone type calculation unit is used to calculate the second road cone type in the image data based on the road cone classification model.

[0114] Optionally, in one embodiment of this application, the result fusion module includes: a result fusion unit.

[0115] The result fusion unit is used to fuse the first traffic cone type and the second traffic cone type to obtain the target traffic cone type.

[0116] Optionally, in one embodiment of this application, the driving planning module includes: a driving planning unit.

[0117] The planning and driving unit is used to control the vehicle to automatically plan a driving route based on the location of the traffic cone and the type of the target traffic cone.

[0118] It should be noted that the foregoing explanation of the traffic cone recognition method embodiment also applies to the traffic cone recognition system of this embodiment, and will not be repeated here.

[0119] According to the traffic cone recognition system proposed in this application, the image data collected by the vehicle is processed based on two trained models, and the position and type of the traffic cone in the image data are output. Based on the position and type, the vehicle is controlled to automatically plan the driving route. The model algorithm adopts a two-stage model design, namely detection + classification. The first-stage network performs position regression, the second-stage network classifies the result of the first-stage regression, and finally the results of the two are fused to obtain a high-precision detection result. Through the model, the accuracy of traffic cone recognition can be effectively improved.

[0120] This solves the problem of low accuracy in traffic cone recognition during autonomous driving in related technologies.

[0121] Figure 14 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include:

[0122] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0123] When the processor 502 executes the program, it implements the traffic cone recognition method provided in the above embodiments.

[0124] Furthermore, the vehicle also includes:

[0125] Communication interface 503 is used for communication between memory 501 and processor 502.

[0126] The memory 501 is used to store computer programs that can run on the processor 502.

[0127] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory. volatile memory), for example, at least one disk storage.

[0128] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EIS) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 14 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0129] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0130] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of this application.

[0131] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the traffic cone recognition method described above.

[0132] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0133] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0134] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable storage medium could be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0136] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0137] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0138] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0139] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0140] It should be understood that the application of this application is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A traffic cone recognition method, characterized by, The traffic cone recognition method comprises: Obtaining image data collected by a vehicle, and inputting the image data into a pre-trained cone detection classification model and a cone classification model respectively; Calculating a cone position and a first cone type based on the cone detection classification model according to the image data, and calculating a second cone type based on the cone classification model according to the image data; Fusing the first cone type and the second cone type to obtain a target cone type; Controlling the vehicle to automatically plan a driving route according to the cone position and the target cone type; The backbone network of the cone detection classification model comprises a CBS module, a BottleNeck_1 module, a BottleNeck_2 module, a C3_X_xn module and a SPP module; The CBS module is used to extract image features to obtain a first feature map; The BottleNeck_1 module and the BottleNeck_2 module are composed of a plurality of CBS modules, and are used to fuse the first feature map to obtain a second feature map; The C3_X_xn module is used to perform memory fusion on the second feature map obtained by the BottleNeck_1 module and the BottleNeck_2 module to obtain a third feature map; The SPP module is used to input the third feature map into a plurality of different pooling layers for dimension reduction, and fuse the dimension reduction results to obtain a target feature map; The cone classification model is a simplified mobilenet-v3 classification network, the simplified mobilenet-v3 classification network reduces the number of convolution layers to a preset number to reduce the parameters of the simplified mobilenet-v3 classification network, adopts a two-stage model design method of detection + classification, and improves the traffic cone recognition accuracy.

2. The traffic cone recognition method of claim 1, wherein, The training process of the cone detection classification model comprises: Collecting images in a driving scene, labeling the positions and categories of various cones in the images to obtain first training data; After image enhancement of the first training data, inputting the first training data into a one-stage detection network for training to obtain network output values, calculating a loss function according to the network output values and labels in the first training data, adjusting model parameters according to the loss function to obtain the cone detection classification model.

3. The traffic cone recognition method of claim 2, wherein, The loss function comprises a detection loss function and a classification loss function; The calculation process of the detection loss function is: normalizing the network output value to obtain a predicted probability, and inputting the probability into formula to calculate the detection loss function. The calculation process of the classification loss function is represented as: normalizing the network output value to obtain a predicted probability, and inputting the probability into formula to calculate the classification loss function. wherein, denotes a predicted probability, denotes a true label value of the i-th training data in the first training data, denotes a number of true labels, denotes a number of training data; The detection loss function is used to calculate the loss of the detection frame in the network output value, and the classification loss function is used to calculate the loss of the classification name in the network output value.

4. The traffic cone recognition method of claim 2, wherein, The first training data is image enhanced to obtain enhanced first training data; The enhanced first training data is input into the backbone network in the one-stage detection network for image feature extraction and fusion to obtain a target feature map; The target feature map is input into an intermediate layer for down-sampling and up-sampling to obtain network output values. ​ 5. The traffic cone recognition method of claim 1, wherein, The training process of the road cone classification model specifically comprises: Collecting images in a driving scene, framing a plurality of regions in the images, and labeling the classes of each road cone in the plurality of regions to obtain second training data; Performing data enhancement on the second training data to obtain enhanced second training data; Inputting the enhanced second training data into a two-stage classification network for training, and adjusting model parameters according to a focal loss function to obtain the road cone classification model; The calculation process of the focal point loss function is as follows: wherein, is a probability score, is a proportion for adjusting the weight of the easy-to-classify sample.

6. A traffic cone recognition system characterized by, The traffic road cone identification system is applied to the traffic road cone identification method of any one of claims 1-5, and the traffic road cone identification system comprises: A data acquisition module configured to acquire image data collected by a current vehicle, and input the image data into a pre-trained road cone detection classification model and a road cone classification model; A model operation module configured to calculate road cone positions and a first road cone type in the image data based on the road cone detection classification model, and calculate a second road cone type in the image data based on the road cone classification model; A result fusion module configured to fuse the first road cone type and the second road cone type to obtain a target road cone type; A driving planning module configured to control the vehicle to automatically plan a driving route according to the road cone positions and the target road cone type.

7. A vehicle characterized by comprising: The vehicle comprises a memory, a processor, and a traffic road cone identification program stored on the memory and executable on the processor, and the traffic road cone identification program, when executed by the processor, implements the steps of the traffic road cone identification method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a traffic road cone identification program, and the traffic road cone identification program, when executed by a processor, implements the steps of the traffic road cone identification method of any one of claims 1-5.

Images