Efficient road garbage identification method and device

Abstract

The invention provides an efficient road garbage recognition method and device, and the method comprises the steps: obtaining a data set containing a garbage image, and marking a garbage region in the image and the type information of the garbage region; a semantic segmentation model is trained, the semantic segmentation model takes a backbone network of MobileNet-MH as a feature extraction module, and outputs are connected to a feature fusion module which combines context information of different scales, extracts attention under different scales and fuses features in a multi-scale addition mode; carrying out model quantification and pruning on the trained semantic segmentation model; and inputting an actual image into the optimized semantic segmentation model to obtain an actual junk image, and performing post-processing. For a small target segmentation task, replacing an IoU measurement in the Turkey loss with a normalized Gaussian Wasserstein distance, and reconstructing a new Turkey loss training semantic segmentation model; for the whole garbage segmentation task, loss measurement is carried out through IoU loss. And the precision of the garbage segmentation task and the performance of small target segmentation are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an efficient road garbage recognition method and device. Background Art

[0002] Road litter identification has significant social significance and is a current research hotspot. Deep learning technology, particularly convolutional neural networks (CNNs), has become a mainstream approach for litter identification. Using deep learning models, efficient feature extraction and classification of litter images can be achieved, improving the accuracy and robustness of litter identification.

[0003] Existing garbage recognition algorithms primarily focus on garbage classification and target detection. Garbage classification is primarily used in smart trash cans and mobile applications, but lacks the ability to locate garbage. Road debris is characterized by small size and irregular shapes, and often overlaps and occludes each other, resulting in highly complex features. Therefore, garbage recognition methods based on target detection cannot accurately provide location information for road debris.

[0004] In addition, road garbage identification algorithms are mainly used on vehicle-mounted equipment such as street sweepers and sanitation vehicles. The computing power of the vehicle-mounted platform is low, and the real-time and accuracy of garbage identification must also be taken into account. Summary of the Invention

[0005] In order to solve the technical problems mentioned in the background technology, the present invention proposes an efficient road garbage identification method and device to solve the above technical problems.

[0006] According to one aspect of the present invention, an efficient road garbage identification method is proposed, comprising:

[0007] S1. Obtain a dataset containing garbage images, and mark the garbage areas and their category information in the images;

[0008] S2. Train a semantic segmentation model using the dataset. The semantic segmentation model uses the MobileNet-MH backbone network as a feature extraction module. The output of the feature extraction module is connected to a feature fusion module. The feature fusion module combines contextual information at different scales and extracts attention at different scales to fuse features in a multi-scale addition manner. For the overall garbage segmentation task, the loss is measured using the IoU loss. For the small object garbage segmentation task, the semantic segmentation model replaces the IoU metric in the Turkey loss with the normalized Gaussian Wasserstein distance and reconstructs a new Turkey loss for training.

[0009] S3. Optimizing the trained semantic segmentation model, wherein the optimization includes model quantization and pruning;

[0010] S4, inputting the actual image into the optimized semantic segmentation model to obtain the actual garbage image;

[0011] S5. Post-processing the actual garbage image.

[0012] Through the method of the present application, the semantic segmentation-based solution can effectively identify the category of garbage and assist the sweeper in adjusting the size and suction force of the suction cup; it can also segment the area of ​​road garbage, adapt to various shapes of garbage, and accurately provide garbage location information; it can meet the garbage detection needs in terms of computing power and accuracy. The semantic segmentation model used in the method of the present application has a computing power of 35.474GMac and a required parameter volume of 37.658M. In the garbage segmentation task, its mIOU performance index reached 84.91. After network structure optimization and quantization operations, it can be deployed on vehicle-mounted equipment to meet speed and accuracy requirements.

[0013] In some specific embodiments, step S2 includes:

[0014] S21, based on the polygonal area of ​​the garbage truth value marked by the semantic segmentation model, using a preset threshold γ to filter out small garbage targets in the garbage truth value;

[0015] S22. Generate an overall mask image M for all marked garbage. The pixel value in the garbage area of ​​the overall mask image M is 1, and the pixel value in other areas is 0. For small-target garbage, model each small-target garbage area as a two-dimensional Gaussian distribution pseudo mask image M. s , the two-dimensional Gaussian distribution pseudo mask image M s The pixel weight decreases from the garbage center to the boundary, and the pixel weight is set in the range of [0,1].

[0016] S24: Obtain an image to be tested, input the image to be tested into the semantic segmentation model for prediction, and obtain a probability map P for segmenting the entire garbage area. b , and the probability map P for segmenting small target garbage s ;

[0017] S25, based on the probability map P s and the pseudo mask image M s , replace the IoU metric in Turkey loss with the normalized Gaussian Wasserstein distance W(P s ,M s ), reconstruct a new Turkey loss L to train the semantic segmentation model, and the new Turkey loss L is expressed as:

[0018]

[0019] Where α and β represent hyperparameters, α+β=1, and the range of α and β is 0 to 1;

[0020] S26, for the overall mask image M, using the probability map P b , the difference between the model output and the true label is measured based on the cross entropy loss function, Lovasz loss and boundary loss. Due to the large fluctuation range of road garbage size, there are a large number of small-target garbage, but there is also a lot of large-target garbage stacked on each other. In order to better optimize the segmentation of small targets while also maintaining the accuracy of the large-target segmentation task, when training the garbage segmentation model, the present invention proposes the idea of ​​strengthening small-target segmentation. For the small-target segmentation task, the normalized Gaussian Wasserstein distance is added as the loss measure of small-target garbage; for the overall garbage segmentation task, the original IOU loss is still maintained for measurement. The IoU measurement method has great differences in sensitivity to objects of different scales. For small objects (6x6 pixels), a small position deviation (4 pixels) will cause the IoU to drop significantly; while for objects of normal size (36x36 pixels), under the same position deviation, the IoU has no obvious change. Therefore, for small objects, IoU as a loss measure cannot provide sufficient supervision information. Compared to the IoU metric, the normalized Gaussian-Wasserstein distance is scale-invariant and can measure the similarity between non-overlapping or mutually contained regions, making it a better measure of the distance between small objects. Furthermore, because the boundaries of road trash are fuzzy, and the normalized Gaussian-Wasserstein distance emphasizes the center of trash rather than its boundaries, a boundary loss must be added to strengthen the trash boundaries.

[0021] In some specific embodiments, the normalized Gaussian Wasserstein distance in step S25 is expressed as:

[0022]

[0023] in, is the Frobenius norm, m1 represents the Gaussian center of the predicted garbage, m2 represents the Gaussian center of the true garbage, P s To predict the Gaussian distribution of garbage, M s is the Gaussian distribution of the true garbage dataset, and C is the average absolute size of small objects in the garbage dataset. Setting C to the average absolute size of small objects in the garbage dataset yields the best performance.

[0024] In some specific embodiments, the feature extraction module in step S2 includes a 3*3 convolution with a stride of 2 connected to a HardSwish activation layer, several layers of inverse bottleneck layers, and a 1*1 convolution layer with a stride of 1 connected to a HardSwish activation layer. The MobileNet-MH network structure is obtained by searching an architecture search method (NAS) based on reinforcement learning. It can have good performance on a variety of hardware platforms while maintaining lightweight. It adopts a deep separable convolution and a lightweight network structure, making the model very lightweight and computationally efficient. It also introduces a multi-head structure and adds multiple branches to the middle layer of the network. Each branch is responsible for learning different feature representations. This application captures feature information of different scales and semantic levels through this multi-head structure, and ensures the computing power requirements for use on vehicle-mounted equipment.

[0025] In some specific embodiments, the feature fusion module in step S2 includes a first scale extraction module, a second scale extraction module, and a third scale extraction module connected in sequence. The first scale extraction module, the second scale extraction module, and the third scale extraction module each include a 3*3 convolution module and at least one ConvNeXt block connected in sequence. The 3*3 convolution module includes a 3*3 convolution layer, a batch normalization layer, and a ReLU activation layer connected in sequence. The outputs of the first scale extraction module, the second scale extraction module, and the third scale extraction module are connected to fuse features corresponding to different scales; wherein the outputs of the first scale extraction module and the second scale extraction module are both resized to the same size by a resizing module. This feature fusion module combines the advantages of convolutional attention and the multi-layer structure of UNet, incorporating contextual information at different scales and extracting attention at different scales, fully considering information from details to the global picture. In addition, this structure abandons the original UNet method of upsampling layer by layer and then skipping connections to fuse features, and directly adopts a three-scale addition method, effectively reducing computing power and adapting to the computing power requirements of vehicle-mounted equipment.

[0026] In some specific embodiments, the number of ConvNeXt blocks in the first scale extraction module is 18.

[0027] In some specific embodiments, the outputs of the first scale extraction module, the second scale extraction module, and the third scale extraction module, and the outputs after the features corresponding to the different scales are respectively connected to corresponding loss functions. Auxiliary loss is introduced to enhance the main segmentation task. Auxiliary loss can promote rapid convergence of the network. By introducing multiple loss functions at different layers or branches of the network, supervised learning of features at different levels or branches can be performed during training, thereby learning effective feature representations more quickly.

[0028] In some specific embodiments, the post-processing in step S5 includes removing small noise areas and connecting adjacent junk image areas, so as to obtain a more accurate junk image recognition result.

[0029] According to a second aspect of the present invention, an efficient road garbage identification device is provided, the device comprising:

[0030] a data set acquisition unit configured to acquire a data set containing junk images, and mark junk areas in the images and their category information;

[0031] A model training unit is configured to train a semantic segmentation model using the dataset. The semantic segmentation model uses the MobileNet-MH backbone network as a feature extraction module, and connects the output of the feature extraction module to a feature fusion module. The feature fusion module combines contextual information at different scales and extracts attention at different scales to fuse features in a multi-scale addition manner. For the overall garbage segmentation task, the loss is measured using the Intersection over Union (IoU) loss. For the small object garbage segmentation task, the semantic segmentation model replaces the IoU metric in the Turkey loss with the normalized Gaussian Wasserstein distance and reconstructs a new Turkey loss for training.

[0032] A model optimization unit, configured to optimize the trained semantic segmentation model, wherein the optimization includes model quantization and pruning;

[0033] An image classification unit is configured to input an actual image into the optimized semantic segmentation model to obtain an actual garbage image;

[0034] The image processing unit is configured to perform post-processing on the actual garbage image.

[0035] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, any of the above methods is implemented.

[0036] The present invention proposes an efficient road garbage identification method and device, the technical effects of which are:

[0037] (1) The semantic segmentation-based solution can effectively identify the category of garbage and assist the sweeper in adjusting the size and suction force of the suction cup.

[0038] (2) It can segment the area of ​​road garbage, adapt to the various shapes of garbage, and accurately provide garbage location information.

[0039] (3) Using the MobileNet-MH backbone as the feature extraction module, combined with a feature fusion module with a three-scale fusion structure, the proposed semantic segmentation model achieves both computing power and accuracy that meet the requirements of garbage detection. The model achieves a computing power of 35.474 GMac and requires 37.658M parameters. In the garbage segmentation task, its mean intersection over union (mIOU) performance index reaches 84.91.

[0040] (4) For the road garbage segmentation task, this paper proposes the idea of ​​strengthening small target segmentation. For this task, the small target area is extracted separately, and the normalized Gaussian Wasserstein distance is added as the loss metric for small target garbage, which is then combined with the Turkey loss. For the overall garbage segmentation task, the original IOU loss is still used as the metric. This method effectively improves the accuracy of garbage segmentation and the performance of small target segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many of the intended advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. Other features, objects, and advantages of the present application will become more apparent upon reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0042] Figure 1 This is a flow chart of an efficient road garbage identification method according to an embodiment of the present application;

[0043] Figure 2 This is a schematic diagram of the structure of a feature extraction module according to an embodiment of the present application;

[0044] Figure 3 This is a schematic diagram of the structure of a feature fusion module according to an embodiment of the present application;

[0045] Figure 4 This is a schematic diagram of the feature fusion module structure connected to the auxiliary loss function in one embodiment of the present application;

[0046] Figure 5 This is a schematic diagram of an efficient road garbage identification device according to an embodiment of the present application;

[0047] Figure 6 It is a structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] It should be noted that Figure 1 This is a flow chart of an efficient road garbage identification method according to an embodiment of the present application; Figure 1 As shown, according to an embodiment of the present application, an efficient road garbage identification method includes:

[0051] S1. Obtain a dataset containing garbage images, and mark the garbage areas and their category information in the images;

[0052] S2. Train a semantic segmentation model using the dataset. The semantic segmentation model uses the MobileNet-MH backbone network as a feature extraction module. The output of the feature extraction module is connected to a feature fusion module. The feature fusion module combines contextual information at different scales and extracts attention at different scales to fuse features in a multi-scale addition manner. For the overall garbage segmentation task, the loss is measured using the Intersection over Union (IoU) loss. For the small object garbage segmentation task, the semantic segmentation model replaces the IoU metric in the Turkey loss with the normalized Gaussian Wasserstein distance and reconstructs a new Turkey loss for training.

[0053] Specifically, during the training process, the model parameters are continuously adjusted through the back-propagation algorithm to minimize the loss function, so that the model can accurately learn the garbage areas in the image.

[0054] Specifically, Figure 2 This is a schematic diagram of the feature extraction module structure of an embodiment of the present application. Figure 2As shown, the feature extraction module in step S2 includes a 3*3 convolution with a stride of 2 connected to a HardSwish activation layer, several layers of inverse bottleneck layers, and a 1*1 convolution layer with a stride of 1 connected to a HardSwish activation layer, which are connected in sequence. The MobileNet-MH network structure is obtained by searching an architecture search method (NAS) based on reinforcement learning. It can have good performance on a variety of hardware platforms while maintaining lightweight. It adopts a deep separable convolution and a lightweight network structure, making the model very lightweight and computationally efficient. It also introduces a multi-head structure and adds multiple branches to the middle layer of the network. Each branch is responsible for learning different feature representations. This application captures feature information of different scales and semantic levels through this multi-head structure, and ensures the computing power requirements for use on vehicle-mounted equipment.

[0055] Further, Figure 3 This is a schematic diagram of the feature fusion module structure of an embodiment of the present application. Figure 3 As shown, the feature fusion module in step S2 includes a first scale extraction module, a second scale extraction module, and a third scale extraction module connected in sequence. The first scale extraction module, the second scale extraction module, and the third scale extraction module each include a 3*3 convolutional module and at least one ConvNeXt block connected in sequence. The 3*3 convolutional module includes a 3*3 convolutional layer, a batch normalization layer, and a ReLU activation layer connected in sequence. The outputs of the first scale extraction module, the second scale extraction module, and the third scale extraction module are connected to fuse features corresponding to different scales. The outputs of the first scale extraction module and the second scale extraction module are resized to the same size by a resizing module. The number of ConvNeXt blocks in the first scale extraction module is 18. This feature fusion module combines the advantages of convolutional attention and the multi-layer structure of UNet, incorporating contextual information at different scales and extracting attention at different scales, fully considering information from details to the global picture. In addition, this structure abandons the original UNet method of layer-by-layer upsampling and skip connection to fuse features, and directly adopts a three-scale summation method, effectively reducing computing power and adapting to the computing power requirements of vehicle-mounted equipment.

[0056] Specifically, step S2 includes:

[0057] S21, based on the polygonal area of ​​the garbage truth value marked by the semantic segmentation model, using a preset threshold γ to filter out small garbage targets in the garbage truth value;

[0058] S22. Generate an overall mask image M for all marked garbage. The pixel value in the garbage area of ​​the overall mask image M is 1, and the pixel value in other areas is 0. For small-target garbage, model each small-target garbage area as a two-dimensional Gaussian distribution pseudo mask image M. s , the two-dimensional Gaussian distribution pseudo mask image M s The pixel weight decreases from the garbage center to the boundary, and the pixel weight is set in the range of [0,1].

[0059] S24: Obtain an image to be tested, input the image to be tested into the semantic segmentation model for prediction, and obtain a probability map P for segmenting the entire garbage area. b , and the probability map P for segmenting small target garbage s ;

[0060] S25, based on the probability map P s and the pseudo mask image M s , replace the IoU metric in Turkey loss with the normalized Gaussian Wasserstein distance W(P s ,M s ), reconstruct a new Turkey loss L to train the semantic segmentation model, and the new Turkey loss L is expressed as:

[0061]

[0062] Where α and β represent hyperparameters, α+β=1, and the range of α and β is 0 to 1;

[0063] S26, for the overall mask image M, using the probability map P b, the difference between the model output and the true label is measured based on the cross entropy loss function, Lovasz loss and boundary loss. Due to the large fluctuation range of road garbage size, there are a large number of small-target garbage, but there is also a lot of large-target garbage stacked on each other. In order to better optimize the segmentation of small targets while also maintaining the accuracy of the large-target segmentation task, when training the garbage segmentation model, the present invention proposes the idea of ​​strengthening small-target segmentation. For the small-target segmentation task, the normalized Gaussian Wasserstein distance is added as the loss metric for small-target garbage; for the overall garbage segmentation task, the original IOU loss is still maintained for measurement. The IoU metric has great differences in sensitivity to objects of different scales. For micro-water objects (6x6 pixels), a small position deviation (4 pixels) will cause the IoU to drop significantly; while for objects of normal size (36x36 pixels), under the same position deviation, the IoU has no obvious change. Therefore, for small objects, IoU as a loss metric cannot provide sufficient supervision information. Compared to the IoU metric, the normalized Gaussian-Wasserstein distance is scale-invariant and can measure the similarity between non-overlapping or mutually contained regions, making it a better measure of the distance between small objects. Furthermore, because the boundaries of road trash are fuzzy, and the normalized Gaussian-Wasserstein distance emphasizes the center of trash rather than its boundaries, a boundary loss must be added to strengthen the trash boundaries.

[0064] Furthermore, the normalized Gauss-Wasserstein distance in step S25 is expressed as:

[0065] For two two-dimensional Gaussian distributions P s =N(m1,∑1) and M s =(m2,∑2),P s and M s The second-order Wasserstein distance between is defined as:

[0066]

[0067] can be simplified to:

[0068]

[0069] However, Is a distance metric and cannot be used directly as a similarity metric (i.e., a value between 0 and 1 is used as IoU). Here it is exponentially normalized to obtain a new metric of normalized Gaussian Wasserstein distance:

[0070]

[0071] in, is the Frobenius norm, m1 represents the Gaussian center of the predicted garbage, m2 represents the Gaussian center of the true garbage, P s To predict the Gaussian distribution of garbage, M s is the Gaussian distribution of the true garbage dataset, and C is the average absolute size of small objects in the garbage dataset. Setting C to the average absolute size of small objects in the garbage dataset yields the best performance.

[0072] In addition, auxiliary loss is introduced to enhance the main segmentation task. Auxiliary loss can promote the rapid convergence of the network. By introducing multiple loss functions at different layers or branches of the network, the features of different layers or branches can be supervised during the training process, so that effective feature representation can be learned faster. Figure 4 This is a schematic diagram of the feature fusion module structure accessing the auxiliary loss function in an embodiment of the present application, as shown in FIG. Figure 4 As shown, the outputs of the first scale extraction module, the second scale extraction module and the third scale extraction module, and the outputs after the features corresponding to the different scales are connected to the corresponding loss functions respectively.

[0073] S3. Optimizing the trained semantic segmentation model, wherein the optimization includes model quantization and pruning;

[0074] S4, inputting the actual image into the optimized semantic segmentation model to obtain the actual garbage image;

[0075] S5. Post-processing the actual garbage image.

[0076] Specifically, the post-processing in step S5 includes removing small noise areas and connecting adjacent junk image areas, thereby obtaining a more accurate junk image recognition result.

[0077] Specifically, the garbage in the above method includes images of leaves and plastic bags.

[0078] Figure 5 FIG. 1 is a schematic diagram of an efficient road garbage identification device according to an embodiment of the present application. Figure 5As shown, the device includes: the device includes a data set acquisition unit 501, which is configured to acquire a data set containing garbage images and mark the garbage areas and their category information in the images; a model training unit 502, which is configured to use the data set to train a semantic segmentation model, wherein the semantic segmentation model uses the MobileNet-MH backbone network as a feature extraction module, and connects the output of the feature extraction module to a feature fusion module, wherein the feature fusion module combines context information at different scales and extracts attention at different scales, and fuses features in a multi-scale addition manner; wherein, for the overall garbage segmentation task, the loss measurement is performed by the IoU loss; for the small target garbage segmentation task, the semantic segmentation model replaces the IoU measurement in the Turkey loss with the normalized Gaussian Wasserstein distance, and reconstructs a new Turkey loss for training; a model optimization unit 503, which is configured to optimize the trained semantic segmentation model, wherein the optimization includes model quantization and pruning; an image classification unit 504, which is configured to input an actual image into the optimized semantic segmentation model to obtain an actual garbage image; and an image processing unit 505, which is configured to post-process the actual garbage image.

[0079] The technical effects achieved by the above method are:

[0080] (1) The semantic segmentation-based solution can effectively identify the category of garbage and assist the sweeper in adjusting the size and suction force of the suction cup.

[0081] (2) It can segment the area of ​​road garbage, adapt to the various shapes of garbage, and accurately provide garbage location information.

[0082] (3) Using the MobileNet-MH backbone as the feature extraction module, combined with a feature fusion module with a three-scale fusion structure, the proposed semantic segmentation model achieves both computing power and accuracy that meet the requirements of garbage detection. The model achieves a computing power of 35.474 GMac and requires 37.658M parameters. In the garbage segmentation task, its mean intersection over union (mIOU) performance index reaches 84.91.

[0083] (4) For the road garbage segmentation task, this paper proposes the idea of ​​strengthening small target segmentation. For this task, the small target area is extracted separately, and the normalized Gaussian Wasserstein distance is added as the loss metric for small target garbage, which is then combined with the Turkey loss. For the overall garbage segmentation task, the original IoU loss is still used as the metric. This method effectively improves the accuracy of garbage segmentation and the performance of small target segmentation.

[0084] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a data set containing junk images, marks the junk areas in the images and their category information; uses the data set to train a semantic segmentation model, wherein the semantic segmentation model uses the MobileNet-MH backbone network as a feature extraction module, and connects the output of the feature extraction module to a feature fusion module. The feature fusion module combines contextual information at different scales and extracts attention at different scales, and fuses features in a multi-scale addition manner; wherein, for the overall junk segmentation task, the loss measurement is performed using the IoU loss; for the small target junk segmentation task, the semantic segmentation model replaces the IoU measurement in the Turkey loss with the normalized Gaussian Wasserstein distance, and reconstructs a new Turkey loss for training; optimizes the trained semantic segmentation model, and the optimization includes model quantization and pruning; inputs the actual image into the optimized semantic segmentation model to obtain the actual junk image; and post-processes the actual junk image.

[0085] Reference below Figure 6 , which shows a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 6 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0086] like Figure 6 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 into the random access memory (RAM) 603. Various programs and data required for system operation are also stored in the RAM 603. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0087] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including a liquid crystal display (LCD) and speakers; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0088] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0089] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0090] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0091] The modules described in the embodiments of the present application may be implemented by software or hardware.

[0092] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. An efficient road garbage identification method, characterized in that: include: S1. Obtain a data set containing garbage images, and mark the garbage areas in the images and their category information; S2. The semantic segmentation model is trained using the data set. The semantic segmentation model uses the backbone network of MobileNet-MH as a feature extraction module, and the output of the feature extraction module is connected to a feature fusion module. The feature fusion module combines context information of different scales and extracts attention at different scales to fuse features in a multi-scale addition manner. For the overall garbage segmentation task, the loss is measured by the IoU loss. For the small target garbage segmentation task, the semantic segmentation model replaces the IoU metric in the Turkey loss with the normalized Gaussian Wasserstein distance, and reconstructs a new Turkey loss for training. S3, optimizing the trained semantic segmentation model, wherein the optimization includes model quantization and pruning; S4, inputting the actual image into the optimized semantic segmentation model to obtain the actual garbage image; S5. Post-processing the actual garbage image.

2. An efficient road garbage identification method according to claim 1, characterized in that: Step S2 includes: S21, according to the polygonal area of ​​the garbage truth value marked by the semantic segmentation model, using a preset threshold γ to filter out small target garbage in the garbage truth value; S22, for all the marked garbage, generate an overall mask image M, in which the pixel value in the garbage area in the overall mask image M is 1, and the pixel value in other areas is 0; for small target garbage, model each small target garbage area as a two-dimensional Gaussian distribution pseudo mask image M s , the two-dimensional Gaussian distribution pseudo mask image M s The pixel weight decreases from the garbage center to the boundary, and the pixel weight is set in the range of [0,1]; S24, obtaining an image to be tested, inputting the image to be tested into the semantic segmentation model for prediction, and obtaining a probability map P for segmenting the entire garbage area. b , and the probability map P for segmenting small target garbage s ; S25, based on the probability map P s and the pseudo mask image M s , replace the IoU metric in Turkey loss with the normalized Gaussian Wasserstein distance W(P s ,M s ), reconstruct a new Turkey loss L to train the semantic segmentation model, and the new Turkey loss L is expressed as: Where α and β represent hyperparameters, α+β=1, and the range of α and β is 0 to 1; S26, for the overall mask image M, using the probability map P b , based on the cross entropy loss function, Lovasz loss and boundary loss to measure the difference between the model output and the true label.

3. An efficient road garbage identification method according to claim 2, characterized in that: The normalized Gauss-Wasserstein distance in step S25 is expressed as: in, is the Frobenius norm, m1 represents the Gaussian center of the predicted garbage, m2 represents the Gaussian center of the true garbage, P s To predict the Gaussian distribution of garbage, M s is the Gaussian distribution of the true garbage, and C is the average absolute size of small objects in the garbage dataset.

4. The efficient road garbage identification method according to claim 1, characterized in that: The feature extraction module in step S2 includes a 3*3 convolution with a step size of 2 connected to a HardSwish activation layer, several layers of inverse bottleneck layers, and a 1*1 convolution layer with a step size of 1 connected to a HardSwish activation layer.

5. The efficient road garbage identification method according to claim 1, characterized in that: The feature fusion module in step S2 includes a first scale extraction module, a second scale extraction module and a third scale extraction module connected in sequence, wherein the first scale extraction module, the second scale extraction module and the third scale extraction module each include a 3*3 convolution module and at least one ConvNeXt block connected in sequence, the 3*3 convolution module includes a 3*3 convolution layer, a batch normalization layer and a ReLU activation layer connected in sequence, and the outputs of the first scale extraction module, the second scale extraction module and the third scale extraction module are connected to fuse features corresponding to different scales; wherein the outputs of the first scale extraction module and the second scale extraction module are both adjusted to the same size through a size adjustment module.

6. An efficient road garbage identification method according to claim 5, characterized in that: The number of ConvNeXt blocks in the first scale extraction module is 18.

7. The efficient road garbage identification method according to claim 5, characterized in that: The outputs of the first scale extraction module, the second scale extraction module and the third scale extraction module, and the outputs after the features corresponding to the different scales are connected to the corresponding loss functions respectively.

8. The efficient road garbage identification method according to claim 1, characterized in that: The post-processing in step S5 includes removing small noise areas and connecting adjacent garbage image areas.

9. An efficient road garbage identification device, characterized in that: The device comprises: A data set acquisition unit is configured to acquire a data set containing junk images, and mark junk areas in the images and their category information; A model training unit is configured to train a semantic segmentation model using the data set, wherein the semantic segmentation model uses a MobileNet-MH backbone network as a feature extraction module, and connects the output of the feature extraction module to a feature fusion module, wherein the feature fusion module combines context information at different scales, extracts attention at different scales, and fuses features in a multi-scale addition manner; wherein, for the overall garbage segmentation task, the loss is measured by the IoU loss; for the small target garbage segmentation task, the semantic segmentation model replaces the IoU metric in the Turkey loss with the normalized Gaussian Wasserstein distance, and reconstructs a new Turkey loss for training; A model optimization unit, configured to optimize the trained semantic segmentation model, wherein the optimization includes model quantization and pruning; An image classification unit, configured to input the actual image into the optimized semantic segmentation model to obtain an actual garbage image; The image processing unit is configured to perform post-processing on the actual garbage image.

10. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 8 is implemented.

Images