Multi-task intelligent sorting device for garbage based on deep learning and mobile mechanical claw

Abstract

The application provides a multi-task garbage intelligent sorting device based on deep learning and a mobile mechanical gripper, and relates to the technical field of garbage classification. The device comprises a sorting box body provided with a garbage throwing port at the top end, and a garbage sorting equipment, a camera, a development board and a controller arranged in the box body; the camera detects garbage images thrown through the garbage throwing port in real time and transmits the garbage images to the development board; the development board identifies the garbage images through a visual algorithm and gives corresponding garbage categories and coordinate positions, and transmits the garbage categories and coordinate positions to the controller; and the controller controls the garbage sorting equipment to throw the garbage into corresponding garbage cans below the box body according to the received garbage coordinate values. The device combines software and hardware, adds target positioning, cooperates with the mechanical gripper, and solves the problems of classification identification and delivery of various types of garbage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of garbage classification, and in particular to a multi-task garbage intelligent sorting device based on deep learning and a mobile mechanical claw. BACKGROUND

[0002] Nowadays, environmental problems gradually appear, and environmental pollution gradually intensifies, attracting global attention. Improper garbage disposal is one of the main factors leading to environmental pollution, and orderly garbage classification has become the primary task to solve such pollution. In recent years, China is gradually promoting garbage classification in various parts of the country, which has shown good results. There are 46 major cities conducting preliminary experiments, which has promoted the positive development of garbage classification.

[0003] Garbage classification can help us maximize the use of garbage resources, reduce the total amount of garbage, improve our living environment, reduce the pollution of groundwater by garbage, and reduce the land area occupied by garbage. Many garbage contains substances that are difficult to naturally decompose, which are very harmful to the land, and garbage classification can identify and separately handle these recyclable or difficult-to-degrade substances, thereby reducing the amount of garbage by at least 60% or more.

[0004] Therefore, as the first link in the garbage disposal process, garbage classification is particularly important, and intelligent garbage classification has become a research hotspot. Traditional intelligent garbage classification devices can mostly meet the requirements of garbage being put in one by one, and then completing autonomous identification and classified storage of various garbage through computer vision detection technology. For the case where multiple different types of garbage are mixed together and put into a garbage can, the design of the traditional intelligent garbage can cannot perform intelligent classification, delivery and storage, which affects the overall sorting efficiency. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a multi-task garbage intelligent sorting device based on deep learning and a mobile mechanical claw to solve the problems of the prior art.

[0006] To solve the above technical problems, the technical solution adopted by the present application is: a multi-task garbage intelligent sorting device based on deep learning and a mobile mechanical claw, comprising a sorting box body provided with a garbage throwing port at the top end, and a garbage sorting device, a camera, a development board and a controller arranged inside the box body; the camera detects the garbage image thrown through the garbage throwing port in real time and transmits it to the development board, the development board identifies the corresponding garbage category and coordinate position through a visual algorithm and transmits them to the controller; the controller controls the garbage sorting device to throw into the corresponding garbage can below the box body according to the received garbage coordinate value.

[0007] Preferably, the device further comprises a detection platform and a tray arranged in the sorting box body; the detection platform is located at the lower part of the garbage throwing port and is composed of a middle-opening acrylic plate, the two ends of the acrylic plate are fixed on the optical shaft respectively, the optical shaft is controlled to rotate through the first steering engine, and the opening and closing of the detection platform are realized; the tray is arranged below the detection platform and the rotation angle is controlled through the second steering engine; the first steering engine and the second steering engine control the rotation angle through the controller.

[0008] Preferably, the garbage sorting equipment comprises a two-dimensional sliding table, a mechanical gripper and a stepping motor; when the STM32 controller receives the coordinate value of the current garbage, the garbage sorting equipment uses an acceleration and deceleration algorithm to drive the stepping motor to control the two-dimensional sliding table and the mechanical gripper to move to the coordinate position of the garbage, starts the mechanical gripper to grab, returns to the central position of the detection platform after grabbing the garbage, and controls the first steering engine that controls the opening and closing of the detection platform to rotate 90 degrees, so that the detection platform is opened, the garbage falls into the tray controlled by the second steering engine below, the second steering engine controls the tray to rotate to the position corresponding to the garbage can, and the accurate classification of the garbage is realized.

[0009] Preferably, the device further comprises a display screen arranged at the top of the sorting box body, which is used for displaying the classification and delivery results of various types of garbage in real time.

[0010] Preferably, the development board deploys a visual algorithm to calculate the object category and specific coordinate value of the recognized garbage, and feeds back the coordinate value to the controller, which controls the stepping motor to realize the movement of the two-dimensional sliding table, and then positions the mechanical gripper to the garbage at the specific coordinate value, so as to realize the grabbing of the garbage.

[0011] Preferably, the specific method for the development board to calculate the specific coordinate value of the recognized garbage is:

[0012] Step 1, constructing a garbage classification detection model based on YOLOv5 and a normalized attention module;

[0013] YOLOv5 network model + normalized attention module are used as the main framework of the garbage classification detection model, the YOLOv5 network model is divided into four parts, which are backbone feature extraction network, feature fusion network, DBL convolution layer and loss function; in the YOLOv5 network, a normalized attention module is embedded at the end of each network block, so that the network is more lightweight and the detection efficiency is improved;

[0014] Step 2, counting common types of household garbage and constructing a training data set to train the garbage classification detection model;

[0015] A plurality of different types of garbage pictures are collected in a detection environment and a non-detection environment respectively, and are labeled to construct a training data set; in order to realize accurate classification of multiple types of garbage, a multi-classification model is trained to realize accurate classification of multiple types of garbage;

[0016] Step 3, using the trained garbage classification detection model to identify and classify the garbage put into the detection platform, and calculating the coordinate position of the garbage put into the detection platform and marking the frame;

[0017] Step 4, using the target detection characteristics of the YOLO model, returning the relative coordinates of the target in the camera detection range, measuring the distance between the camera and the detection platform to obtain depth information, and then mapping the target coordinates obtained by the camera to the coordinates of the detection target in the world coordinate system through vector transformation, and converting the coordinates of the detection target in the world coordinate system to the target coordinates of the two-dimensional sliding table through proportional operation, to realize accurate grabbing of the target garbage.

[0018] Preferably, the development board sends the identified garbage category and coordinate position to the controller through a serial port protocol, the controller drives the stepper motor to move to the coordinate position of the corresponding garbage after receiving the corresponding information using a speed-up and speed-down algorithm, the controller drives the steering engine to move the mechanical gripper, and the target garbage is grabbed, and after the garbage is grabbed, the detection platform returns to the middle position, the garbage falls, and the detection platform is opened at the same time, so that the garbage falls into the tray controlled by the steering engine, thereby realizing the garbage being put into the designated garbage can.

[0019] Preferably, the device further comprises a diffuse reflection photoelectric switch to detect whether the garbage can is full, the diffuse reflection photoelectric switch is controlled by the controller, the photoelectric switch is installed at a fixed height of each garbage can, when the garbage in the garbage can reaches the height, a voice broadcast module is started to give a full load prompt, and a full load display of the corresponding garbage can is given on the display screen, thereby realizing the full load judgment of the garbage can.

[0020] The technical scheme has the beneficial effects that the multi-task garbage intelligent sorting device based on deep learning and a mobile mechanical gripper provided by the application,

[0021] (1) The improved YOLOv5+NAM attention module is used, the network is more lightweight, the detection efficiency and anti-interference ability are further improved, and the robustness and accuracy of detection are higher;

[0022] (2) The electric control algorithm is improved, and the speed-up and speed-down algorithm is used to drive the stepper motor, so that the garbage after being clamped is more stably put;

[0023] (3) The software and hardware are combined, and target positioning is added, which cooperates with the mechanical gripper to solve the problems of classification and identification and delivery of multiple types of garbage. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A structural schematic diagram of a multi-task garbage intelligent sorting device based on deep learning and a mobile mechanical claw is provided for an embodiment of the present application.

[0025] Figure 2 A flowchart of intelligent garbage sorting by a multi-task garbage intelligent sorting device based on deep learning and a mobile mechanical claw is provided for an embodiment of the present application.

[0026] Figure 3 A diffuse reflection photoelectric switch detection schematic diagram is provided for an embodiment of the present application, wherein (a) is a diffuse reflection switch installation schematic diagram, and (b) is a full load detection schematic diagram.

[0027] Figure 4 An improved YOLOV5+NAM model structure diagram is provided for an embodiment of the present application.

[0028] Figure 5 A schematic diagram of a CBAM channel attention sub-module is provided for an embodiment of the present application.

[0029] Figure 6 A schematic diagram of a SAM spatial attention sub-module is provided for an embodiment of the present application.

[0030] Figure 7 An improved partial YOLOV5 network model structure schematic diagram is provided for an embodiment of the present application.

[0031] Figure 8 A target automatic labeling result diagram on an original image is provided for an embodiment of the present application.

[0032] Figure 9 A schematic diagram of deducing relative position coordinates is provided for an embodiment of the present application.

[0033] In the figure: 1, camera; 2, highlight display screen; 3, mechanical claw; 4, second steering engine; 5, first steering engine; 6, garbage can; 7, acrylic plate; 8, stepping motor; 9, aluminum profile; 10, two-dimensional sliding table; 11, detection platform. DETAILED DESCRIPTION

[0034] The specific embodiments of the present application will be further described in detail below in combination with the drawings and examples. The following examples are used to illustrate the present application, but are not used to limit the scope of the present application.

[0035] In this embodiment, a multi-task garbage intelligent sorting device based on deep learning and a mobile mechanical claw is as follows: Figure 1 , 2As shown, including the sorting box body provided with a garbage throwing port at the top end, and a detection platform 11, a tray, a first steering engine 5, a second steering engine 4, a garbage sorting device, a camera 1, a jetson nano development board and an STM32 controller arranged inside the box body; the camera 1 detects the image of the garbage thrown through the garbage throwing port in real time and transmits it to the jetson nano development board, the jetson nano development board identifies the coordinate position of the corresponding garbage through a visual algorithm and sends the identified garbage category and coordinate position to the STM32 controller through a serial port protocol; the detection platform 11 is located below the garbage throwing port and is composed of a middle opening and closing acrylic plate 7, the two ends of the acrylic plate 7 are fixed on the optical shaft, the optical shaft is controlled to rotate through the first steering engine 5 to realize the opening and closing of the detection platform 11; the tray is arranged below the detection platform 11 and the rotation angle is controlled through the second steering engine 4; the first steering engine 5 and the second steering engine 4 control the rotation angle through the controller.

[0036] The garbage sorting device includes a two-dimensional sliding table 10, a flexible mechanical gripper 3 and a stepping motor 8; when the STM32 controller receives the coordinate value of the current garbage, it uses an acceleration and deceleration algorithm to drive the stepping motor 8 to control the two-dimensional sliding table 10 and the mechanical gripper 3 to move to the coordinate position of the garbage, start the mechanical gripper 3 to grab, grab the garbage and return to the central position of the detection platform 11, at this time, the first steering engine 5 controlling the opening and closing of the detection platform 11 is rotated by 90 degrees, the detection platform 11 is opened (only one side of the acrylic plate is opened), the garbage falls into the tray controlled by the second steering engine 4 below, the second steering engine 4 controls the tray to rotate to the position corresponding to the garbage can, realizing accurate classification of the garbage.

[0037] Meanwhile, the device also includes a high-brightness display screen 2 arranged at the top of the sorting box body and a diffuse reflection photoelectric switch arranged on the inner wall of each garbage can, as shown in Figure 3 The display screen 2 is used to display the classification and delivery results of each type of garbage in real time. The diffuse reflection photoelectric switch is controlled by the STM32 single-chip microcomputer to detect whether the garbage can is full. In this embodiment, the photoelectric switch is installed at a position 80% high of the four garbage cans, when the height of the garbage in the garbage can reaches 80%, a voice broadcast module is started to give a full load prompt, and a corresponding garbage can full display is given on the display screen 2, so as to judge whether there is a full garbage can.

[0038] In this embodiment, the sorting box body adopts 4040 aluminum profile 9 as the frame, the link between the aluminum profiles 9 is fastened by M5 bolts and boat-shaped nuts, the device is packaged with black acrylic plates around and on the top, a high-brightness display screen 2 is placed on the top of the box body for displaying the real-time classification and delivery results of various types of garbage, a 200*200mm garbage throwing port is designed on the top of the box body for the delivery of multi-task garbage. A two-dimensional sliding table 10 mechanism is designed inside the box body to control the movement of the mechanical gripper 3. When multiple garbage is thrown into the detection platform 11 through the throwing port, the camera 1 on the top of the box body detects the garbage, the jetson nano development board accurately identifies the type of garbage through the visual algorithm and gives the coordinate position of the corresponding garbage. When the STM32 controller receives the coordinate value of the current garbage, it uses the acceleration and deceleration algorithm to drive the stepper motor 8 to control the two-dimensional sliding table 10 and the mechanical gripper 3 to move to the coordinate position of the garbage, and then starts the mechanical gripper 3 to grab the garbage and returns to the central position of the detection platform 10. At this time, the first steering wheel 5 that controls the opening and closing of the detection platform 10 rotates 90 degrees, the detection platform 11 opens, and the garbage falls into the tray controlled by the steering wheel below, thereby falling into the corresponding garbage can. If the camera 1 detects that there is still garbage on the platform 11, the mechanical gripper 3 returns to the coordinate position of the next garbage, grabs it and throws it into the corresponding garbage can, until the garbage on the detection platform is sorted, thereby realizing the intelligent classification and delivery of multi-task garbage. In this embodiment, four sub-garbage cans 6 with a size of 200mm*200mm*350mm are designed at the bottom of the box body, which are respectively used to store recyclable garbage, kitchen garbage, hazardous garbage and other garbage. A diffuse reflection photoelectric switch is installed at a height of 80% of the four garbage cans to realize the full load detection of the garbage in the can. When the garbage can is full, the corresponding voice prompt and information such as the full load of the corresponding garbage can are displayed on the screen.

[0039] In this embodiment, the jetson nano development board deploys a visual algorithm to calculate the object category and specific coordinate value of the recognized garbage, and feeds back the coordinate value to the STM32 controller. The controller controls the stepper motor to realize the movement of the two-dimensional sliding table, and then positions the mechanical gripper to the garbage at the specific coordinate value to realize the grabbing of the garbage.

[0040] The specific method for the jetson nano development board to calculate the specific coordinate value of the recognized garbage is:

[0041] Step 1, build a garbage classification detection model based on YOLOv5 and normalization-based attention module (NAM);

[0042] The YOLOv5 network model + normalization-based attention module NAM is used as the main framework of the garbage classification detection model, as shown inFigure 4 As shown in the figure, the YOLOv5 network model is divided into 4 parts, which are the backbone feature extraction network, the feature fusion network, the DBL convolution layer and the loss function; in the YOLOv5 network, an NAM module is embedded at the end of each network block, making the network more lightweight and improving the detection efficiency;

[0043] NAM is a high-efficiency and lightweight attention module, which does not need additional calculations and parameters such as full connection and convolution, but directly uses the scaling factor of batch normalization (BN) to calculate the attention weight, and further suppresses insignificant features by adding a regularization term. NAM adopts channel attention module (CBAM) and spatial attention module (SAM), and redesigns the channel attention and spatial attention submodules; in CBAM, BN is used to measure the importance of pixels, i.e. pixel normalization, as shown in formula (1), the scaling factor in BN reflects the size of the change of each channel, which also represents the importance of the channel, the scaling factor is the variance in BN, the greater the variance, the more drastic the change of the channel, the richer the information contained in the channel, and the greater the importance, while those channels with little change have single information and small importance.

[0044]

[0045] In the formula, μ B and σ B are the mean and standard deviation of the small batch B respectively; γ and β are the trainable affine transformation parameters (scale and displacement).

[0046] The CBAM channel attention submodule is as shown in formula (2) and Figure 5 , where M c represents the output feature, sigmoid represents the activation function, γ is the scale factor of each channel, and the weight is W γ = γ i / ∑ j＝0 γ j , F1 represents the input feature. If the same normalization method is used for each pixel in space, the weight of spatial attention can be obtained, i.e. pixel normalization;

[0047] The SAM spatial attention submodule is as shown in formula (3) and Figure 6 , where M s represents the output feature, sigmoid represents the activation function, ρ is the scale factor, and the weight is W ρ = ρ i / ∑ j＝0 ρ j , F2 represents the input feature,

[0048] M c =sigmoid(W γ (BN(F1))) (2)

[0049] M s =sigmoid(W ρ (BN s (F2)) (3)

[0050] The loss function of the NAM attention module adds a regularization term to suppress unimportant features, and the loss function is shown in formula (4).

[0051] Loss=∑ (x,y) l(f(x,W),y)+p∑g(γ)+p∑g(ρ) (4)

[0052] In the above formula, x is the input; y is the output; W represents the network weight; f(·) is the input feature, l(·) is the loss function; g(·) is the l1 norm penalty function; p is a penalty parameter, used to balance g(γ) and g(ρ).

[0053] The improved YOLOv5 network model as shown in Figure 7 is obtained by combining YOLOv5 and NAM modules as a garbage classification detection model.

[0054] Step 2, count the common household garbage types and build a training data set to train the garbage classification detection model.

[0055] In this embodiment, 4300 garbage pictures of different types are collected in the detection environment and the non-detection environment, and are labeled to build a training data set; in order to realize accurate classification of multiple types of garbage, a multi-classification model is trained in this embodiment, which can realize accurate classification of multiple types of garbage, for example, plastic bottles, paper cups, and easy-open cans in common garbage can be recognized.

[0056] Step 3, use the trained garbage classification detection model to identify and classify the garbage put into the detection platform, and in the detection process, common garbage is classified into four categories of recyclable garbage, kitchen garbage, hazardous garbage, and other garbage, and the coordinates of the garbage put into the detection platform are calculated and are framed, as shown in Figure 8 .

[0057] Step 4, using the target detection characteristics of the YOLO model, the relative coordinates of the target in the camera detection range are returned, the depth information is obtained by measuring the distance between the camera and the detection platform, and the target coordinates obtained by the camera are mapped to the coordinates of the detection target in the world coordinate system through vector transformation. Since the two-dimensional sliding table does not have the influence of three-dimensional parameters on the grabbing, the coordinates of the detection target in the world coordinate system are converted to the target coordinates of the two-dimensional sliding table through simple proportional operation, as shown in Figure 9 Through the above algorithm, accurate grabbing of the target garbage can be realized.

[0058] The present embodiment takes the upper left corner of the detection platform as the origin of the coordinate system, finds the detect.py file in the algorithm code, opens it and finds plot_one_box, and the code is as follows:

[0059]

[0060]

[0061] Then, ctr+mouse click, enter general.py, and automatically locate to plot_one_box function, modify the function as

[0062]

[0063] The target coordinate information can be output, and the coordinate position of the garbage to be grabbed is obtained.

[0064] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the present application.

Claims

1. A multi-task intelligent garbage sorting device based on deep learning and mobile mechanical claws, characterized in that: The application relates to a sorting box device, which comprises a sorting box body provided with a top end garbage throwing opening, a garbage sorting device, a camera, a development board and a controller arranged in the box body; the camera detects garbage images thrown through the garbage throwing opening in real time and transmits the garbage images to the development board; the development board identifies the corresponding garbage category and coordinate position through a visual algorithm and transmits the garbage category and coordinate position to the controller; and the controller controls the garbage sorting device to throw the garbage into a corresponding garbage can below the box body according to the received garbage coordinate value. The device further comprises a detection platform and a tray arranged in the sorting box body; the detection platform is located below the garbage throwing opening and is composed of middle-split acrylic plates; the two ends of the acrylic plates are fixed on optical shafts; the optical shafts are controlled to rotate through a first steering engine, so that the detection platform is opened and closed; the tray is arranged below the detection platform and is controlled to rotate through a second steering engine; and the first steering engine and the second steering engine are controlled to rotate through the controller. The garbage sorting device comprises a two-dimensional sliding table, a mechanical gripper and a stepping motor; when the STM32 controller receives the coordinate value of the current garbage, the stepping motor is driven to control the two-dimensional sliding table and the mechanical gripper to move to the coordinate position of the garbage through a speed-up and speed-down algorithm; the mechanical gripper is started to grab the garbage; the garbage is returned to the central position of the detection platform; the first steering engine for opening and closing the detection platform is controlled to rotate by 90 degrees; the detection platform is opened; the garbage falls into the tray controlled by the second steering engine; the tray is controlled to rotate to the position of a corresponding garbage can, so that the garbage is accurately classified. The development board deploys a visual algorithm to calculate the object category and specific coordinate value of the identified garbage and feeds back the coordinate value to the controller; the controller controls the stepping motor to realize the movement of the two-dimensional sliding table and then positions the mechanical gripper to the garbage at the specific coordinate value, so that the garbage is grabbed. The specific method for the development board to calculate the specific coordinate value of the identified garbage is as follows: Step 1: constructing a garbage classification detection model based on YOLOv5 and a normalized attention module; A YOLOv5 network model + a normalized attention module is used as the main framework of the garbage classification detection model; the YOLOv5 network model is divided into four parts, namely a backbone feature extraction network, a feature fusion network, a DBL convolution layer and a loss function; a normalized attention module is embedded at the end of each network block in the YOLOv5 network, so that the network is more lightweight and the detection efficiency is improved; Step 2: counting common household garbage types and constructing a training data set to train the garbage classification detection model; A plurality of garbage pictures of different types are collected in a detection environment and a non-detection environment respectively, and the garbage pictures are labeled to construct a training data set; in order to realize accurate classification of multiple types of garbage, a multi-classification model is trained to realize accurate classification of multiple types of garbage; Step 3: using the trained garbage classification detection model to identify and classify the garbage thrown onto the detection platform, calculating the coordinate position of the garbage thrown onto the detection platform and marking the coordinate position. Step 4, using the target detection characteristics of the YOLO model, the relative coordinates of the target in the detection range of the camera are returned, the depth information is obtained by measuring the distance between the camera and the detection platform, and then the target coordinates obtained by the camera are mapped to the coordinates of the detection target in the world coordinate system through vector transformation, the coordinates of the detection target in the world coordinate system are converted into the target coordinates of the two-dimensional sliding table through proportional operation, and the precise grabbing of the target garbage is realized.

2. The multi-task intelligent garbage sorting device based on deep learning and mobile mechanical claws according to claim 1, characterized in that: The device also includes a display screen arranged at the top of the sorting box body, for real-time display of the classification and delivery results of various types of garbage.

3. The multi-task intelligent garbage sorting device based on deep learning and mobile mechanical claws according to claim 2, characterized in that: The development board sends the recognized garbage category and coordinate position to the controller through a serial port protocol, the controller drives the stepping motor to move to the corresponding garbage coordinate position using a speed control algorithm after receiving the corresponding information, the controller drives the steering engine to move the mechanical gripper, the target garbage is grabbed, and the garbage is dropped after returning to the middle position of the detection platform, the detection platform is opened, and the garbage falls into the tray controlled by the steering engine, so that the garbage is put into the designated garbage can.

4. The multi-task intelligent garbage sorting device based on deep learning and mobile mechanical claws according to claim 3, characterized in that: The device also includes a diffuse reflection photoelectric switch to detect whether the garbage can is full, the diffuse reflection photoelectric switch is controlled by the controller, the photoelectric switch is installed at a fixed height of each garbage can, when the garbage in the garbage can reaches the height, a voice full load prompt is given, and the corresponding garbage can full load display is given on the display screen, so that the full load of the garbage can is judged.

Images