Garbage identification method applied to unmanned vehicle

Abstract

The invention discloses a garbage identification method applied to an unmanned vehicle. The method belongs to the technical field of visual identification, image processing and unmanned vehicles, and comprises the following operation steps: collecting, preprocessing and labeling a junk data set; constructing a YOLOv8 neural network basic model, and introducing a dynamic snakelike convolution module to replace a common convolution module; training the optimized and improved YOLOv8 model by using the marked junk data set, and establishing a junk recognition model; based on the trained garbage recognition algorithm pt model, converting the garbage recognition algorithm pt model into an engine model available for TensorRT; an engine model under a TensorRT framework is adopted to accelerate garbage recognition reasoning, and the GPU preheating time is greatly saved; deploying the whole garbage identification model to the unmanned vehicle, and testing the actual combat effect; according to the method, the problems of poor training effect, low convergence speed and the like of a common model are solved, and the problems of low accuracy, low reasoning speed, poor actual effect and the like when garbage in life is identified and classified by using the convolutional neural network in the unmanned vehicle are also solved.

Description

Technical Field

[0001] The present invention belongs to the technical fields of visual recognition, image processing and unmanned vehicle, and relates to a garbage recognition method applied to an unmanned vehicle, thereby improving the automation level of garbage recognition and cleaning, reducing manual intervention, and promoting resource recovery and environmental protection. Background Art

[0002] The current society is facing an increasingly serious problem of garbage disposal, which has promoted the continuous development and application of garbage sorting technology. Traditional garbage sorting methods usually rely on manual identification and classification, which have problems such as low recognition accuracy and low efficiency. At the same time, they also face challenges such as insufficient human resources and high costs. With the continuous advancement of artificial intelligence and deep learning technologies, image recognition technology has been widely used in the field of garbage sorting. Neural networks such as GoogLeNet, VGGNet, ResNet, and YOLO have continuously improved the accuracy of image recognition. Using these technologies, garbage can be automatically classified and identified, which relatively improves the accuracy and efficiency of garbage sorting.

[0003] However, in real scenarios, unknown distribution deviations and the relative movement of unmanned vehicles may affect the recognition results. Therefore, when neural networks are directly used to identify and classify garbage displayed on the unmanned vehicle platform, the accuracy and efficiency are still relatively low. Therefore, it is necessary to develop a garbage recognition method suitable for unmanned vehicles, combining image recognition technology, deep learning algorithms and the characteristics of the unmanned vehicle itself, to achieve accurate and rapid recognition and classification of various types of garbage. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a high-precision and high-efficiency garbage identification method for unmanned vehicles.

[0005] Technical solution: The garbage identification method applied to an unmanned vehicle described in the present invention has the following operating steps:

[0006] Step (1): Collect garbage image datasets, and preprocess and label the datasets;

[0007] Step (2): Build the YOLOv8 neural network basic model and introduce dynamic snake convolution to improve the model;

[0008] Step (3): Introduce the scale factor ratio into the loss function to control the scale of the auxiliary border for calculating the loss function value;

[0009] Step (4): Use Lion optimizer to replace the Adam optimizer of the original model to reduce training parameters and improve the model convergence speed;

[0010] Step (5): Use the data set in step (1) to train the model in step (4) to obtain a garbage recognition model, and evaluate the training effect of the garbage recognition model based on relevant indicators;

[0011] Step (6): Use the TensorRT framework to accelerate the garbage recognition model reasoning;

[0012] Step (7): Deploy the entire garbage recognition model to the unmanned vehicle platform to realize real-time garbage recognition.

[0013] Furthermore, in step (1), the device for collecting the garbage image dataset is specifically:

[0014] The equipment used includes high-definition cameras and scanners to collect images of major garbage in life and divide them into five categories: waste batteries, paper boxes, glass, plastic, and metal;

[0015] A total of 5,000 garbage data sets were collected and randomly divided into training set, validation set, and test set in a ratio of 8:1:1. They were then labeled using labelimg software and divided into five categories: waste batteries, paper boxes, glass, plastic, and metal.

[0016] The dataset is enhanced using the Mixup technique, which mixes two images with adjacent labels to create synthetic images, thereby enhancing the generalization ability of the model by introducing label noise and visual variability.

[0017] Furthermore, the use of Mixup technology to enhance the data set is specifically as follows:

[0018] Assume P x1 is a sample of P, P y1 Is the P x1 The label corresponding to the sample; P x2 is another sample of P, P y2 Is the P x2 The labels corresponding to the samples, the data enhancement calculation steps based on the mixup principle are:

[0019] λ=Beta(α,β) (1)

[0020] mixed-P x =λ*P x1 +(1-λ)*P x2 (2)

[0021] mixed-P y =λ*P y1 +(1-λ)*P y2 (3)

[0022] Where Beta(α,β) refers to the Beta distribution function, mixed-P x is the mixed P sample, mixed-P y is the label corresponding to the mixed P sample, where the values ​​of α and β are determined by testing the minimum change in information loss of the synthesized image under different values.

[0023] Furthermore, in step (2), the construction of the YOLOv8 neural network basic model is specifically as follows:

[0024] The network structure of YOLOv8 consists of the following three parts: Backbone, Neck, and Head;

[0025] The Backbone part is responsible for feature extraction, using a series of convolution and deconvolution layers, and using residual connections and bottleneck structures to reduce the size of the network and improve performance. This part uses the C2f module as the basic building block. The Backbone part also includes improved technologies such as Depthwise Separable Convolution and DilatedConvolution to enhance the ability of feature extraction.

[0026] The Neck part is responsible for multi-scale feature fusion, which enhances the feature representation capability by fusing feature maps from different stages of Backbone.

[0027] Specifically, the Neck part of YOLOv8 includes the following components:

[0028] SPPF module: used for pooling operations of different scales, splicing feature maps of different scales together to improve the detection capability of objects of different sizes;

[0029] PAA module: used to intelligently allocate anchor boxes to optimize the selection of positive and negative samples and improve the training effect of the model;

[0030] PAN module: It includes two PAN modules, which are used for path aggregation of features at different levels, and enhance the expressiveness of feature maps through bottom-up and top-down paths;

[0031] The Head part is responsible for the final target detection and classification tasks, including a detection head and a classification head:

[0032] Detection head: Contains a series of convolutional and deconvolutional layers to generate detection results; these layers are responsible for predicting the bounding box regression value of each anchor box and the confidence of the object's existence;

[0033] Classification head: Global average pooling is used to classify each feature map, and the probability distribution of each category is output by reducing the dimension of the feature map; the design of the classification head enables YOLOv8 to effectively handle multi-category classification tasks.

[0034] Furthermore, in step (2), the introduction of dynamic snake convolution to improve the model specifically includes:

[0035] 1. Construct a dynamic snake convolution module with the following features:

[0036] (1) Snake-shaped core shape

[0037] The shape of the snake-shaped dynamic convolution kernel is similar to a snake path; its purpose is to capture curves and non-straight structures in the image; this kernel shape can adapt to complex edges and textures in the image;

[0038] (2) Dynamically adjust weights

[0039] The weights of the snake dynamic convolution are dynamically generated according to the input data. Usually a generative network is used to generate adaptive weights according to the current input. These weights are applied during the convolution operation, so that the convolution kernel has different characteristics at different positions.

[0040] (3) Multi-scale feature extraction

[0041] Snake dynamic convolution captures multi-scale features in images through convolution kernels of different scales; by combining features of different scales, it can fully describe the structural information in the image;

[0042] 2. Introduce the dynamic snake convolution module into the C2f module of the YOLOv8 backbone network. The specific steps are as follows:

[0043] The C2f module includes a Bottleneck bottleneck block, which has two convolution blocks. The second convolution block CBS in the Bottleneck bottleneck block in the C2f module is replaced with a dynamic snake convolution module to obtain a Bottleneck_DSC module. Then, the Bottleneck bottleneck block of the C2f module in the YOLOv8 network is replaced with a Bottleneck_DSC module to finally form C2f_DySnakeConv.

[0044] Furthermore, in step (3), the introduction of the scale factor ratio to calculate the loss function value specifically includes:

[0045] The basic model of YOLOv8 calculates the loss value by the intersection-over-union ratio; the scale factor ratio is introduced in the IoU. Compared with the ordinary IoU loss, when the ratio is less than 1, the auxiliary border size is smaller than the actual border, and its effective range of regression is smaller than the IoU loss, but its gradient absolute value is greater than the gradient obtained by the IoU loss, which can accelerate the convergence of high IoU samples; on the contrary, when the ratio is greater than 1, the larger-scale auxiliary border expands the range of regression, which is beneficial for the regression of low Iou.

[0046] Furthermore, in step (4), replacing the Adam optimizer of the original model with the Lion optimizer specifically includes:

[0047] In the Python terminal, enter: pip install lion-pytorch, introduce the Lion optimizer into the trainer.py file in YOLOv8, and import the corresponding package: from lion_pytorch import Lion, and finally set the YOLOv8 training parameters: optimizer = true.

[0048] Furthermore, in step (5), the step of training the model in step (4) using the data set in step (1) and evaluating the model training effect according to relevant indicators specifically includes:

[0049] Select yolov8s.pt as the initial weight model, write the garbage dataset path marked in step (1) into the custom dataset yaml file, and change the 8th layer C2f in the yolov8s.yaml file to C2f_DySnakeConv, and load the dataset yaml file, yolov8s.yaml file, and yolov8.pt file paths through train.py, and set the number of training rounds: epochs = 100, the number of pictures to be obtained at one time: batch = 4, and the training parameters are retained when the training is interrupted: resume = True, run the train.py file, and obtain the best.pt file with the best training effect;

[0050] The training effect of the model is evaluated by calculating indicators. In all the following expressions, TP represents the number of samples that are actually true and predicted to be true, TN represents the number of samples that are actually false and predicted to be false, FP represents the number of samples that are actually false and predicted to be true, and FN represents the number of samples that are actually true and predicted to be false. The indicators are as follows:

[0051] (1) Accuracy is a measure of the overall ability of a classifier to correctly classify samples. It is calculated by dividing the number of samples correctly classified by the classifier by the total number of samples. The calculation formula for accuracy is as follows:

[0052]

[0053] (2) Precision: Precision refers to the proportion of samples that are actually positive among the samples predicted by the classifier to be positive, which can reflect the prediction accuracy of the classifier on a certain category:

[0054]

[0055] (3) Recall rate refers to the proportion of all positive samples predicted as positive by the classifier, reflecting the ability of the classifier to identify all positive samples. The calculation formula of recall rate is as follows:

[0056]

[0057] (4) The F1 score is the harmonic mean of the precision and recall, which comprehensively considers the precision and recall of the classifier and can comprehensively evaluate the performance of the classifier. The calculation formula of the F1 score is as follows:

[0058]

[0059] (5) Confusion matrix: It is a special evaluation indicator for classification problems, which compares the deviation between the classifier prediction results and the true labels.

[0060] Furthermore, in step (6), the use of the TensorRT framework to accelerate the inference of the garbage identification model specifically includes:

[0061] Convert the trained best.pt to best.wts format, and then convert it to best.engine file format through the TensorRT framework. Run the yolov8_det_trt.py file, load the image path to be inferred, myplugins.dll file, and best.engine model to achieve inference.

[0062] 1. TensorRT reconstructs and optimizes the network structure, which is reflected in the following aspects:

[0063] (1) TensorRT eliminates useless output layers in the network by analyzing the network model to reduce calculations;

[0064] (2) Vertical integration of the network structure, that is, integrating the conv, BN, and Relu layers of the mainstream neural network into one layer;

[0065] (3) For the horizontal combination of networks, horizontal combination means fusing together layers that have the same tensor as input and perform the same operation;

[0066] (4) For the concat layer, the input of the contact layer is sent to the following operation, without having to perform concat separately and then calculate the input, which is equivalent to reducing the transmission throughput once;

[0067] 2. The TensorRT framework accelerates the reasoning process as follows:

[0068] Preparation before inference: Process the input in the CPU and use the common module in the TensorRT inference SDK to allocate input and output GPU memory;

[0069] Perform inference: copy the CPU input to the GPU; perform inference in the GPU and put the model output into the GPU memory;

[0070] Inference post-processing: copy the output from the GPU memory to the CPU; perform other post-processing in the CPU.

[0071] Furthermore, in step (7), the entire garbage identification model is deployed on the unmanned vehicle platform to achieve real-time garbage identification, specifically including:

[0072] Deploy the YOLOv8 runtime environment on the unmanned vehicle platform, and open the camera by loading the OpenCV library in the yolov8_det_trt.py file to implement YOLOv8 real-time detection;

[0073] 1. Deploy the YOLOv8 operating environment on the unmanned vehicle platform. The specific steps are as follows:

[0074] Install Ubuntu 18.04 system on the host computer of the unmanned vehicle, install the corresponding graphics card driver according to the graphics card configuration of 4060, install Anaconda version 23.5.2, create a Python virtual environment through conda activate, and install CUDA version 11.5 and Pytorch version 1.8.5;

[0075] 2. Run the improved YOLOv8 model for real-time detection. The process is as follows:

[0076] Transplant the yolov8_det_trt.py file debugged in the above computer segment to the host computer of the unmanned vehicle, adjust the position of the camera on the unmanned vehicle to make its shooting range wider, and finally start the unmanned driving. By running the Pythonyolov8_det_trt.py command in the terminal, YOLOv8 real-time garbage detection is realized.

[0077] Beneficial effects: The characteristics of the present invention are: 1. The present invention introduces dynamic snake convolution and replaces the C2f layer in the YOLOv8 structure with C2f_DySnakeConv, so that the model can more effectively capture the characteristics of the target by adaptively adjusting the shape and size of the convolution kernel, and can enhance the target detection model's perception of targets of different scales, shapes and postures, thereby improving the accuracy and robustness of target detection, while filtering out appearance differences due to deformation, occlusion, motion blur, etc., and improving the generalization ability of the model; 2. The present invention introduces a scale factor ratio in the loss function, and replaces the traditional Adam optimizer with the Lion optimizer, thereby reducing training parameters, improving the model convergence speed, and making the model more robust and generalized; 3. The TensorRT framework is used to finally convert the best.pt file format into the best.engine model, thereby improving the reasoning speed and efficiency of the deep learning model, and by optimizing the calculation graph and network structure of the model, reducing redundant calculations, merging levels, streamlining precision, and other methods, thereby improving the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a YOLOv8 network framework diagram in the present invention;

[0079] Figure 2 It is a diagram of the improved YOLOv8 network framework structure in the present invention;

[0080] Figure 3 Schematic diagram of the specific improvement of Bottleneck in the C2f layer of the present invention;

[0081] Figure 4 It is a detailed framework structure diagram of the C2f layer in the present invention;

[0082] Figure 5 is a schematic diagram of the improved C2f_DSC layer of the present invention;

[0083] Figure 6 It is a diagram of the model format conversion process in the present invention. DETAILED DESCRIPTION

[0084] In order to more clearly illustrate the technical solution of the present invention, the technical solution of the present invention is further described in detail below with reference to the accompanying drawings;

[0085] As shown in the figure, the garbage identification method applied to an unmanned vehicle according to the present invention has the following operating steps:

[0086] Step (1): Collect garbage image datasets, and preprocess and label the datasets;

[0087] Step (2): Build the YOLOv8 neural network basic model and introduce dynamic snake convolution to improve the model;

[0088] Step (3): Introduce the scale factor ratio into the loss function to control the scale of the auxiliary border for calculating the loss function value;

[0089] Step (4): Use Lion optimizer to replace the Adam optimizer of the original model to reduce training parameters and improve the model convergence speed;

[0090] Step (5): Use the data set in step (1) to train the model in step (4) to obtain a garbage recognition model;

[0091] Step (6): Use the TensorRT framework to accelerate the garbage recognition model reasoning;

[0092] Step (7): Deploy the entire garbage recognition model to the unmanned vehicle platform and test and evaluate the model.

[0093] Furthermore, in step (1), the device for collecting the garbage image dataset is specifically:

[0094] The equipment used includes high-definition cameras and scanners to collect images of major garbage in life and divide them into five categories: waste batteries, paper boxes, glass, plastic, and metal;

[0095] A total of 5,000 garbage data sets were collected and randomly divided into training set, validation set, and test set in a ratio of 8:1:1. They were then labeled using labelimg software and divided into five categories: waste batteries, paper boxes, glass, plastic, and metal.

[0096] The dataset is enhanced using the Mixup technique, which mixes two images with adjacent labels to create synthetic images, thereby enhancing the generalization ability of the model by introducing label noise and visual variability.

[0097] Furthermore, the use of Mixup technology to enhance the data set is specifically as follows:

[0098] Assume P x1 is a sample of P, P y1 Is the P x1 The label corresponding to the sample; P x2 is another sample of P, P y2 Is the P x2 The labels corresponding to the samples, the data enhancement calculation steps based on the mixup principle are:

[0099] λ=Beta(α,β) (1)

[0100] mixed-P x =λ*P x1 +(1-λ)*P x2 (2)

[0101] mixed-P y =λ*P y1 +(1-λ)*P y2 (3)

[0102] Where Beta(α,β) refers to the Beta distribution function, mixed-P x is the mixed P sample, mixed-P y is the label corresponding to the mixed P sample, where the values ​​of α and β are determined by testing the minimum change in information loss of the synthesized image under different values.

[0103] Furthermore, in step (2), the construction of the YOLOv8 neural network basic model is specifically as follows:

[0104] The network structure of YOLOv8 is mainly composed of the following three parts: Backbone (backbone network), Neck (neck), and Head (detection head);

[0105] The Backbone part is responsible for feature extraction, using a series of convolution and deconvolution layers, and using residual connections and bottleneck structures to reduce the size of the network and improve performance. This part uses the C2f module as the basic building block. Compared with the C3 module of YOLOv5, the C2f module has fewer parameters and better feature extraction capabilities. The Backbone part also includes some common improvement techniques, such as Depthwise Separable Convolution and Dilated Convolution, to further enhance the ability of feature extraction.

[0106] The Neck part is responsible for multi-scale feature fusion, which enhances the feature representation capability by fusing feature maps from different stages of Backbone.

[0107] Specifically, the Neck part of YOLOv8 includes the following components:

[0108] SPPF module (Spatial Pyramid Pooling Fast): used for pooling operations of different scales, splicing feature maps of different scales together to improve the detection ability of objects of different sizes;

[0109] PAA module (Probabilistic Anchor Assignment): used to intelligently assign anchor boxes to optimize the selection of positive and negative samples and improve the training effect of the model;

[0110] PAN module (Path Aggregation Network): includes two PAN modules, which are used for path aggregation of features at different levels, and enhance the expressiveness of feature maps through bottom-up and top-down paths;

[0111] The Head part is responsible for the final target detection and classification tasks, including a detection head and a classification head:

[0112] Detection head: Contains a series of convolutional and deconvolutional layers to generate detection results. These layers are responsible for predicting the bounding box regression value of each anchor box and the confidence of the object's existence;

[0113] Classification head: Global Average Pooling is used to classify each feature map, and the probability distribution of each category is output by reducing the dimension of the feature map. The design of the classification head enables YOLOv8 to effectively handle multi-category classification tasks.

[0114] Furthermore, in step (2), the improvement of the YOLOv8 basic model by introducing dynamic snake convolution specifically includes:

[0115] 1. Construct a dynamic snake convolution module with the following features:

[0116] (1) Snake-shaped core shape

[0117] The shape of the snake-shaped dynamic convolution kernel is not a fixed rectangle or square, but is similar to a snake path. The purpose of this design is to be able to more flexibly capture the curves and non-straight structures in the image. This kernel shape can better adapt to the complex edges and textures in the image;

[0118] (2) Dynamically adjust weights

[0119] The weights of the snake dynamic convolution are not fixed, but dynamically generated according to the input data. Usually a generative network (such as a small convolutional network or an attention mechanism) is used to generate adaptive weights according to the current input. These weights are applied during the convolution operation, so that the convolution kernel has different characteristics at different positions;

[0120] (3) Multi-scale feature extraction

[0121] Snake dynamic convolution can capture multi-scale features in images through convolution kernels of different scales. By combining features of different scales, the structural information in the image can be more comprehensively described;

[0122] 2. Introduce the dynamic snake convolution module (Dynamic SnakeConvolution) in the C2f module of the YOLOv8 backbone network. The specific steps are as follows:

[0123] The C2f module includes a Bottleneck bottleneck block, which has two convolution blocks (CBS). The second convolution block CBS in the Bottleneck bottleneck block in the C2f module is replaced with a dynamic snake convolution module (DynamicSnake Convolution) to obtain a Bottleneck_DSC module, and then the Bottleneck bottleneck block of the C2f module in the YOLOv8 network is replaced with a Bottleneck_DSC module to finally form C2f_DySnakeConv.

[0124] Furthermore, in step (3), the introduction of the scale factor ratio to calculate the loss function value specifically includes:

[0125] The basic model of YOLOv8 calculates the loss value by the intersection over union (IoU); the scale factor ratio is introduced in the IoU. Compared with the ordinary IoU loss, when the ratio is less than 1, the auxiliary border size is smaller than the actual border, and its effective range of regression is smaller than the IoU loss, but its absolute value of the gradient is greater than the gradient obtained by the IoU loss, which can accelerate the convergence of high IoU samples; on the contrary, when the ratio is greater than 1, the larger scale auxiliary border expands the effective range of regression, which is beneficial for the regression of low Iou.

[0126] Furthermore, in step (4), replacing the Adam optimizer of the original model with the Lion optimizer specifically includes:

[0127] In the Python terminal, enter: pip install lion-pytorch, introduce the Lion optimizer into the trainer.py file in YOLOv8, and import the corresponding package: from lion_pytorch import Lion, and finally set the YOLOv8 training parameters: optimizer = true.

[0128] Furthermore, in step (5), the step of training the model in step (4) using the data set in step (1) specifically includes:

[0129] Select yolov8s.pt as the initial weight model, write the garbage dataset path marked in step (1) into the custom dataset yaml file, and change the 8th layer C2f in the yolov8s.yaml file to C2f_DySnakeConv, and load the dataset yaml file, yolov8s.yaml file, and yolov8.pt file paths through train.py, and set the number of training rounds: epochs = 100, the number of images to be obtained at one time: batch = 4, and the training parameters are retained when the training is interrupted: resume = True, run the train.py file, and obtain the best.pt file with the best training effect.

[0130] Furthermore, in step (6), the use of the TensorRT framework to accelerate the inference of the garbage identification model specifically includes:

[0131] Convert the trained best.pt to best.wts format, and then convert it to best.engine file format through the TensorRT framework. Run the yolov8_det_trt.py file, load the image path to be inferred, myplugins.dll file, and best.engine model to achieve inference.

[0132] 1. TensorRT reconstructs and optimizes the network structure, which is reflected in the following aspects:

[0133] (1) TensorRT eliminates useless output layers in the network by analyzing the network model to reduce calculations;

[0134] (2) Vertical integration of the network structure, that is, integrating the conv, BN, and Relu layers of the current mainstream neural network into one layer;

[0135] (3) For the horizontal combination of networks, horizontal combination means fusing together layers that have the same tensor as input and perform the same operation;

[0136] (4) For the concat layer, the input of the contact layer is directly sent to the following operation, without having to perform concat separately and then input calculation, which is equivalent to reducing the transmission throughput once;

[0137] 2. The TensorRT framework accelerates the reasoning process as follows:

[0138] Preparation before inference: Process input in the CPU (such as reading data and standardization); use the common module in the TensorRT inference SDK to allocate input and output GPU memory;

[0139] Perform inference: copy the CPU input to the GPU; perform inference in the GPU and put the model output into the GPU memory;

[0140] Inference post-processing: copy the output from the GPU memory to the CPU; perform other post-processing in the CPU.

[0141] Furthermore, in step (7), the entire garbage identification model is deployed on the unmanned vehicle platform to achieve real-time garbage identification, which specifically includes:

[0142] Deploy the YOLOv8 runtime environment on the unmanned vehicle platform, and open the camera by loading the OpenCV library in the yolov8_det_trt.py file to implement YOLOv8 real-time detection;

[0143] Deploy the YOLOv8 operating environment on the unmanned vehicle platform. The specific steps are as follows:

[0144] 1. Install Ubuntu 18.04 system in the host computer of the unmanned vehicle, install the corresponding graphics card driver according to the graphics card configuration of 4060, install Anaconda version 23.5.2, create a Python virtual environment through conda activate, and install CUDA version 11.5 and Pytorch version 1.8.5;

[0145] 2. Run the improved YOLOv8 model for real-time detection. The process is as follows:

[0146] Transplant the yolov8_det_trt.py file debugged in the above computer segment to the host computer of the unmanned vehicle, adjust the position of the camera on the unmanned vehicle to make its shooting range wider, and finally start the unmanned driving. By running the Pythonyolov8_det_trt.py command in the terminal, YOLOv8 real-time garbage detection is realized.

Claims

1. A garbage identification method applied to an unmanned vehicle, characterized in that: The operation steps are as follows: Step (1): Collect garbage image datasets, and preprocess and label the datasets; Step (2): Build the YOLOv8 neural network basic model and introduce dynamic snake convolution to improve the model; Step (3): Introduce the scale factor ratio into the loss function to control the scale of the auxiliary border for calculating the loss function value; Step (4): Use Lion optimizer to replace the Adam optimizer of the original model to reduce training parameters and improve the model convergence speed; Step (5): Use the data set in step (1) to train the model in step (4) to obtain a garbage recognition model, and evaluate the training effect of the garbage recognition model based on relevant indicators; Step (6): Use the TensorRT framework to accelerate the garbage recognition model reasoning; Step (7): Deploy the entire garbage recognition model to the unmanned vehicle platform to realize real-time garbage recognition.

2. According to claim 1, a garbage identification method applied to an unmanned vehicle is characterized in that: In step (1), the device for collecting the garbage image dataset is specifically: The equipment used includes high-definition cameras and scanners to collect images of major garbage in life and divide them into five categories: waste batteries, paper boxes, glass, plastic, and metal; A total of 5,000 garbage data sets were collected and randomly divided into training set, validation set, and test set in a ratio of 8:1:

1. They were then labeled using labelimg software and divided into five categories: waste batteries, paper boxes, glass, plastic, and metal. The dataset is enhanced using the Mixup technique, which mixes two images with adjacent labels to create synthetic images, thereby enhancing the generalization ability of the model by introducing label noise and visual variability.

3. According to claim 2, a garbage identification method applied to an unmanned vehicle is characterized in that: The use of Mixup technology to enhance the data set is specifically as follows: Assume P x1 is a sample of P, P y1 Is the P x1 The label corresponding to the sample; P x2 is another sample of P, P y2 Is the P x2 The labels corresponding to the samples, the data enhancement calculation steps based on the mixup principle are: λ=Beta(α,β) (1) mixed-P x =λ*P x1 +(1-λ)*P x2 (2) mixed-P y =λ*P y1 +(1-λ)*P y2 (3) Where Beta(α,β) refers to the Beta distribution function, mixed-P x is the mixed P sample, mixed-P y is the label corresponding to the mixed P sample, where the values ​​of α and β are determined by testing the minimum change in information loss of the synthesized image under different values.

4. According to claim 1, a garbage identification method applied to an unmanned vehicle is characterized in that: In step (2), the specific steps of constructing the YOLOv8 neural network basic model are: The network structure of YOLOv8 consists of the following three parts: Backbone, Neck, and Head; The Backbone part is responsible for feature extraction, using a series of convolution and deconvolution layers, and using residual connections and bottleneck structures to reduce the size of the network and improve performance. This part uses the C2f module as the basic building block. The Backbone part also includes improved technologies such as Depthwise Separable Convolution and DilatedConvolution to enhance the ability of feature extraction. The Neck part is responsible for multi-scale feature fusion, which enhances the feature representation capability by fusing feature maps from different stages of Backbone. Specifically, the Neck part of YOLOv8 includes the following components: SPPF module: used for pooling operations of different scales, splicing feature maps of different scales together to improve the detection capability of objects of different sizes; PAA module: used to intelligently allocate anchor boxes to optimize the selection of positive and negative samples and improve the training effect of the model; PAN module: It includes two PAN modules, which are used for path aggregation of features at different levels, and enhance the expressiveness of feature maps through bottom-up and top-down paths; The Head part is responsible for the final target detection and classification tasks, including a detection head and a classification head: Detection head: Contains a series of convolutional and deconvolutional layers to generate detection results; these layers are responsible for predicting the bounding box regression value of each anchor box and the confidence of the object's existence; Classification head: Global average pooling is used to classify each feature map, and the probability distribution of each category is output by reducing the dimension of the feature map; the design of the classification head enables YOLOv8 to effectively handle multi-category classification tasks.

5. According to claim 1, a garbage identification method applied to an unmanned vehicle is characterized in that: In step (2), the introduction of dynamic snake convolution to improve the model specifically includes:

1. Construct a dynamic snake convolution module with the following features: (1) Snake-shaped core shape The shape of the snake-shaped dynamic convolution kernel is similar to a snake path; its purpose is to capture curves and non-straight structures in the image; this kernel shape can adapt to complex edges and textures in the image; (2) Dynamically adjust weights The weights of the snake dynamic convolution are dynamically generated according to the input data. Usually a generative network is used to generate adaptive weights according to the current input. These weights are applied during the convolution operation, so that the convolution kernel has different characteristics at different positions. (3) Multi-scale feature extraction Snake dynamic convolution captures multi-scale features in images through convolution kernels of different scales; by combining features of different scales, it can fully describe the structural information in the image; 2. Introduce the dynamic snake convolution module into the C2f module of the YOLOv8 backbone network. The specific steps are as follows: The C2f module includes a Bottleneck bottleneck block, which has two convolution blocks. The second convolution block CBS in the Bottleneck bottleneck block in the C2f module is replaced with a dynamic snake convolution module to obtain a Bottleneck_DSC module. Then, the Bottleneck bottleneck block of the C2f module in the YOLOv8 network is replaced with a Bottleneck_DSC module to finally form C2f_DySnakeConv.

6. The garbage identification method for unmanned vehicles according to claim 1 is characterized in that ; In step (3), the introduction of the scale factor ratio to calculate the loss function value specifically includes: The basic model of YOLOv8 calculates the loss value by the intersection-over-union ratio; the scale factor ratio is introduced in the IoU. Compared with the ordinary IoU loss, when the ratio is less than 1, the auxiliary border size is smaller than the actual border, and its effective range of regression is smaller than the IoU loss, but its gradient absolute value is greater than the gradient obtained by the IoU loss, which can accelerate the convergence of high IoU samples; on the contrary, when the ratio is greater than 1, the larger-scale auxiliary border expands the range of regression, which is beneficial for the regression of low Iou.

7. The garbage identification method applied to an unmanned vehicle according to claim 1 is characterized in that: In step (4), the step of replacing the Adam optimizer of the original model with the Lion optimizer specifically includes: In the Python terminal, enter: pip install lion-pytorch, introduce the Lion optimizer into the trainer.py file in YOLOv8, and import the corresponding package: from lion_pytorch import Lion, and finally set the YOLOv8 training parameters: optimizer = true.

8. The garbage identification method applied to an unmanned vehicle according to claim 1 is characterized in that: In step (5), the step of training the model in step (4) using the data set in step (1) specifically includes: Select yolov8s.pt as the initial weight model, write the garbage dataset path marked in step (1) into the custom dataset yaml file, and change the 8th layer C2f in the yolov8s.yaml file to C2f_DySnakeConv, and load the dataset yaml file, yolov8s.yaml file, and yolov8.pt file paths through train.py, and set the number of training rounds: epochs = 100, the number of pictures to be obtained at one time: batch = 4, and the training parameters are retained when the training is interrupted: resume = True, run the train.py file, and obtain the best.pt file with the best training effect; The training effect of the model is evaluated by calculating indicators. In all the following expressions, TP represents the number of samples that are actually true and predicted to be true, TN represents the number of samples that are actually false and predicted to be false, FP represents the number of samples that are actually false and predicted to be true, and FN represents the number of samples that are actually true and predicted to be false. The indicators are as follows: (1) Accuracy is a measure of the overall ability of a classifier to correctly classify samples. It is calculated by dividing the number of samples correctly classified by the classifier by the total number of samples. The calculation formula for accuracy is as follows: (2) Precision: Precision refers to the proportion of samples that are actually positive among the samples predicted by the classifier to be positive, which can reflect the prediction accuracy of the classifier on a certain category: (3) Recall rate refers to the proportion of all positive samples predicted as positive by the classifier, reflecting the ability of the classifier to identify all positive samples. The calculation formula of recall rate is as follows: (4) The F1 score is the harmonic mean of the precision and recall, which comprehensively considers the precision and recall of the classifier and can comprehensively evaluate the performance of the classifier. The calculation formula of the F1 score is as follows: (5) Confusion matrix: It is a special evaluation indicator for classification problems, which compares the deviation between the classifier prediction results and the true labels.

9. The garbage identification method applied to an unmanned vehicle according to claim 1 is characterized in that: In step (6), the use of the TensorRT framework to accelerate the garbage identification model reasoning specifically includes: Convert the trained best.pt to best.wts format, and then convert it to best.engine file format through the TensorRT framework. Run the yolov8_det_trt.py file, load the image path to be inferred, myplugins.dll file, and best.engine model to achieve inference.

1. TensorRT reconstructs and optimizes the network structure, which is reflected in the following aspects: (1) TensorRT eliminates useless output layers in the network by analyzing the network model to reduce calculations; (2) Vertical integration of the network structure, that is, integrating the conv, BN, and Relu layers of the mainstream neural network into one layer; (3) For the horizontal combination of networks, horizontal combination means fusing together layers that have the same tensor as input and perform the same operation; (4) For the concat layer, the input of the contact layer is sent to the following operation, without having to perform concat separately and then calculate the input, which is equivalent to reducing the transmission throughput once; 2. The TensorRT framework accelerates the reasoning process as follows: Preparation before inference: Process the input in the CPU and use the common module in the TensorRT inference SDK to allocate input and output GPU memory; Perform inference: copy the CPU input to the GPU; perform inference in the GPU and put the model output into the GPU memory; Inference post-processing: copy the output from the GPU memory to the CPU; perform other post-processing in the CPU.

10. The garbage identification method applied to an unmanned vehicle according to claim 1 is characterized in that: In step (7), the entire garbage identification model is deployed on the unmanned vehicle platform to achieve real-time garbage identification, specifically including: Deploy the YOLOv8 runtime environment on the unmanned vehicle platform, and open the camera by loading the OpenCV library in the yolov8_det_trt.py file to implement YOLOv8 real-time detection; 1. Deploy the YOLOv8 operating environment on the unmanned vehicle platform. The specific steps are as follows: Install Ubuntu 18.04 system on the host computer of the unmanned vehicle, install the corresponding graphics card driver according to the graphics card configuration of 4060, install Anaconda version 23.5.2, create a Python virtual environment through conda activate, and install CUDA version 11.5 and Pytorch version 1.8.5; 2. Run the improved YOLOv8 model for real-time detection. The process is as follows: Transplant the yolov8_det_trt.py file debugged in the above computer segment to the host computer of the unmanned vehicle, adjust the position of the camera on the unmanned vehicle, expand its shooting range, and finally start the unmanned driving. By running the Python yolov8_det_trt.py command in the terminal, YOLOv8 real-time garbage detection is realized.

Images