Recyclable garbage identification and sorting system based on RTMDet algorithm and performance identification method thereof

Abstract

The invention discloses a recoverable garbage identifying and sorting system based on an RTMDet algorithm and a method for identifying performance of the recoverable garbage identifying and sorting system. Belongs to the technical field of robots, machine vision and artificial intelligence. Sorting out the data and making label data; expanding training data, and designing a data enhancement algorithm to improve the algorithm image recognition capability; identifying the category of the recyclable object by using a rotation detection algorithm; calculating the similarity between adjacent frames; and detected object information is sent to the industrial robot, and the clamping jaw is guided to grab the object. The problem of uneven light distribution in an actual scene is solved by using a random cutting algorithm, a brightness adjustment algorithm and a histogram equalization algorithm, recognition and detection of recyclable garbage are achieved, information such as the rotation angle and the pose of a detected object is obtained and sent to the industrial robot, and grabbing of the recyclable object is completed.

Description

Technical Field

[0001] The present invention belongs to the field of robots, and relates to a recyclable garbage identification and sorting system based on an RTMDet algorithm and a method for improving its identification performance, and in particular to a method capable of improving the identification performance of recyclable garbage by an artificial intelligence method in a complex light environment; specifically, a recyclable garbage identification and sorting system based on an adaptive image brightness equalization algorithm and an RTMDet algorithm. Background Art

[0002] Sorting recyclable waste is of great significance, mainly in terms of resource reuse and environmental protection. Through sorting, waste items can be converted into new materials and new products, reducing dependence on primary resources, reducing resource mining and production costs, while reducing landfill and incineration, and reducing pollution to soil, water sources and the atmosphere. In addition, sorting and recycling can also save energy, such as recycling plastic bottles, plastic bags, paper cups and other plastic and paper products. By recycling and using recyclable waste, the public's environmental awareness can be enhanced and the sustainable development of society can be promoted.

[0003] The core of automated garbage sorting technology lies in the recognition algorithm. The recyclable garbage recognition algorithm currently used has problems in detection accuracy, adaptability to complex environments, processing speed, data dependence, etc. Since the environment of garbage sorting is relatively harsh, automated equipment should be used as much as possible to reduce manual labor. This unmanned working scenario puts higher requirements on the automated sorting system. Solving these problems requires improving the robustness of the algorithm, reducing data dependence, and enhancing the adaptability of the algorithm.

[0004] In the existing recyclable waste identification methods, there is a lack of algorithms optimized for industrial practical scenarios. First, the development of deep learning methods has greatly promoted the improvement of image recognition accuracy. However, there are also problems such as low adaptability and slow processing speed. Secondly, in practical scenarios, light conditions may be affected by various factors, resulting in reduced robustness of recyclable waste identification algorithms. Therefore, at this stage, there is an urgent need for a recyclable waste identification system that can be applied to industrial scenarios. The system improves the recognition accuracy of the algorithm by using deep learning methods, and optimizes computing resources and computing efficiency, reduces model parameters, and improves computing speed. At the same time, the applicability of the algorithm to weak light conditions is increased, thereby building a more practical, fast, and robust recyclable waste identification system. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to propose a recyclable garbage identification and sorting system based on the RTMDet algorithm and a method for identifying its performance, so as to improve the efficiency and accuracy of garbage sorting.

[0006] The technical solution of the present invention is: the present invention provides a recyclable garbage identification and sorting system based on a deep learning method (a recyclable garbage identification and sorting system based on an RTMDet algorithm and a method for identifying performance thereof) that can be used stably in scenes with insufficient and unbalanced lighting conditions;

[0007] The system uses a random cropping algorithm, a brightness adjustment algorithm, and a histogram equalization algorithm to improve the problem of uneven light distribution in actual scenes and the problem of misidentification caused by the different sizes of objects to be sorted, realizes correction of strong and weak light areas, and improves the algorithm's adaptability to camera field of view changes; by establishing an object rotation detection algorithm based on a deep learning method, it realizes the identification and detection of recyclable garbage, and obtains the rotation angle, posture and other information of the detected object, which is sent to the industrial robot, driving the robot's gripper to change direction and depth, and completes the grasping of recyclable objects.

[0008] The recyclable garbage identification and sorting system based on the RTMDet algorithm of the present invention comprises a belt line, an industrial robot is arranged on one side of the belt line, and a gripper is arranged on the industrial robot;

[0009] An industrial camera is arranged on the other side of the belt line, and a light shield is arranged on the industrial camera; an industrial computer is also included, one end of the industrial computer is connected to the belt line through a wire, and the other end of the industrial computer is connected to the industrial camera through a wire.

[0010] The operation steps are implemented through the following technical solutions:

[0011] Step (1): construct a camera and robot sorting device for identifying and detecting recyclable waste;

[0012] Step (2): Manually sort out a small batch of data for training data collection and annotation to produce labeled data;

[0013] Step (3): Expand the training data to enhance the algorithm's ability to recognize images under different lighting intensity conditions and at different focal lengths.

[0014] Step (4): Use a histogram equalization method to equalize the image histogram collected by the camera to balance the image lighting conditions;

[0015] Step (5): Use a rotation detection algorithm based on deep learning to identify the category of the recyclable object and calculate the rotation angle of the object;

[0016] Step (6): Calculate the similarity between adjacent frames, remove similar frames, and complete the deduplication operation;

[0017] Step (7): Establish network communication with the robot, send the detected object information to the robot, and guide the robot to grasp the object.

[0018] Furthermore, in step (1), the automated line capable of batch processing of recyclable waste includes the following devices: a camera device (an industrial camera for photographing materials), a light shield (a light shield for shielding stray light), a light source, a belt conveyor (a belt line for conveying materials), an industrial robot, and a control machine for realizing identification and control (for running a deep learning algorithm for recyclable waste detection);

[0019] Among them, industrial robots include multi-axis robots, parallel robots and other types.

[0020] Furthermore, in step (2), the data collection and labeling process includes the following steps:

[0021] (2.1) Collect recyclable garbage data using the garbage sorting device constructed in step (1);

[0022] (2.2) Select images that can be used for deep learning model training based on the types of garbage, keeping the number of types to be detected similar;

[0023] (2.3) Collect images with different density of garbage, including images of garbage accumulation and occlusion;

[0024] (2.4) Use the labeling tool X-AnyLabeling software for manual labeling. Select the object to be detected according to the robot's gripping rotation angle and mark the category.

[0025] Furthermore, in step (3), the data set expansion method includes operations such as adjusting image brightness and cropping images, and using generated random numbers to control image brightness and the proportion of cropped images.

[0026] Furthermore, in step (4), the histogram equalization algorithm refers to a method for adjusting image brightness and contrast before using a deep learning algorithm to detect an object, and the method comprises the following steps:

[0027] (4.1) Count the grayscale distribution of the image and calculate the number of pixels contained in each grayscale;

[0028] (4.2) Normalize the image grayscale using the maximum and minimum values;

[0029] (4.3) Use the cumulative distribution function to calculate the new grayscale value of the image;

[0030] (4.4) Inverse normalization is used to calculate the new grayscale value of the image.

[0031] Furthermore, in step (5), the deep learning algorithm for recyclable waste identification and detection mainly includes the following steps:

[0032] (5.1) The deep learning algorithm takes the image obtained after the histogram equalization operation as input, which contains three RGB channel images;

[0033] (5.2) Extract image color, shape, and contextual semantic features through convolutional networks;

[0034] (5.3) Calculate the output image category, the location frame of the object to be detected, the rotation angle of the location frame, and other information;

[0035] (5.4) Add the detected object information to the queue.

[0036] Furthermore, in step (6), the method for calculating the similarity of adjacent frames is to calculate the cosine similarity of the detected object between two frames, which mainly includes the following steps:

[0037] (6.1) Set the detection area in the image;

[0038] (6.2) Calculate the cosine similarity of the detected objects in the current frame and the previous frame;

[0039] (6.3) Set a similarity threshold, and those above the threshold will be kicked out;

[0040] (6.4) Calculate the time that similar objects are in the area to be detected;

[0041] (6.5) and filter out similar objects with time intervals below the interval.

[0042] Furthermore, in step (7), the method for industrial robot communication is Socket communication, with the industrial computer as the client, and establishing a connection with the industrial robot through the TCP / IP port;

[0043] Using a multi-threaded approach, the network communication of the industrial robot is continuously monitored, and the location and rotation angle of the detected object are sent to the industrial robot. After the industrial robot establishes a connection and receives the Socket socket, the industrial robot posture is adjusted according to the location and angle to achieve grasping.

[0044] The beneficial effects of the present invention are: 1. The present invention fully considers the actual use scenarios of kitchen waste sorting, adopts a solution combining cameras and robots, realizes automatic identification and sorting of recyclable waste, effectively replaces manual operation, and avoids the risk of manual intervention in harsh environments; through this technical means, the garbage disposal process is safer and more efficient; 2. The current garbage sorting system based on deep learning has high hardware requirements, and usually requires an industrial computer equipped with a high-performance graphics processor to meet the inference computing needs; the present invention innovatively uses a rotation detection algorithm with a smaller number of parameters, which not only reduces the dependence on hardware performance, but also reduces the demand for graphics processors; the present invention through The similarity detection of adjacent frames realizes the deduplication function and avoids complex object tracking detection, thereby further reducing the hardware burden and improving the overall operating efficiency of the system; 3. The present invention is specially optimized for complex application environments, fully considering factors such as lighting conditions and inconsistent garbage feeding heights in the usage scenarios, and designs a camera device that is resistant to environmental interference; at the same time, the system is equipped with a deep learning algorithm that can adapt to a variety of lighting conditions and focal lengths, so that the sorting equipment can maintain efficient and accurate working performance in different environments; this design greatly improves the practicability and robustness of the garbage sorting automation production line, and provides important support for the technological development in this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the device structure of the present invention;

[0046] Figure 2 It is a software flow chart of the present invention;

[0047] Figure 3 is a network communication flow chart of the present invention;

[0048] Figure 4 is a schematic diagram of the visual detection area of ​​the present invention;

[0049] In the figure, 1 is the belt line; 2 is the industrial camera; 3 is the light shield; 4 is the industrial computer; 5 is the industrial robot; and 6 is the gripper. DETAILED DESCRIPTION

[0050] The specific technical solutions of the present invention are further described in detail below in conjunction with specific examples.

[0051] As shown in the figure, the present invention comprises a belt line 1, an industrial robot 5 is arranged on one side of the belt line 1, and a gripper 6 is arranged on the industrial robot 5;

[0052] An industrial camera 2 is arranged on the other side of the belt line 1, and a light shield 3 is arranged on the industrial camera 2; an industrial computer 4 is also included, one end of the industrial computer 4 is connected to the belt line 1 through a wire, and the other end is connected to the industrial camera 4 through a wire.

[0053] It is achieved through the following technical solutions:

[0054] Step 1: The object to be detected is viewed from the outside of industrial camera 2 ( Figure 1 Right side) enters the belt transmission line (belt line 1) and starts to enter the process to be tested;

[0055] Step 2: When the object reaches the shooting range of industrial camera 2, industrial camera 2 starts to collect images (such as Figure 4 shown);

[0056] Step 3: In the algorithm training phase, the data set is collected by continuously delivering new materials and placing materials in a loop, which requires differences in object form, posture, and shape;

[0057] Step 4: According to the actual scene requirements and the selected deep learning algorithm, collect enough images to train the algorithm and avoid overfitting problems, and manually annotate the collected images, including object location information, object coverage box, rotation angle and other information;

[0058] Step 5: Expand the data set of the labeled data; the present invention designs a data set expansion method that can improve the applicable scenarios of the algorithm. The specific steps are as follows:

[0059] Step 5-1: Randomly crop the image to simulate the object scene captured by the camera at different focal lengths;

[0060] First, a random number between 0 and 1 is generated and used as the cropping ratio. According to the cropping ratio, the cropping boundary is limited and a random point within the image boundary is selected to complete random cropping. According to the cropped image, the recognition frame, object position, and object rotation angle information are recalculated.

[0061] Step5-2: Randomly generate images with different brightness and contrast;

[0062] First, generate a random number between 0 and 5 and calculate it using the following formula:

[0063]

[0064] In the formula, I and I max are the original value and the maximum value of the image, N r It represents a random number between 0 and 5;

[0065] Step5-3: Randomly combine Step5-1 and Step5-2 and apply them to the data set to expand the data set;

[0066] Realize the recognition, classification, positioning and screening of objects Figure 1 The three steps of input data preprocessing method, deep learning algorithm and deduplication algorithm are combined Figure 1 For detailed explanation:

[0067] Step 6: After the camera obtains the RGB image, it is used for the calculation of the deep learning algorithm after adjusting the image size and resolution. The specific steps include the following:

[0068] Step 6-1: Add the brightness adjustment step in formula (1), where the random number N r Replaced with manual adjustment parameters to adjust image brightness in scenes with complex lighting intensity;

[0069] Step 6-2: Add a histogram equalization operation to the algorithm to balance the image brightness under different lighting scenes and achieve adaptation to different light intensities, as shown in equations (2) to (3):

[0070]

[0071]

[0072] Where, formula (2) is the operation of normalizing the image, obtaining I∈[0,1], and recording the grayscale in the image as r∈[0,1], p r is the cumulative distribution function; after calculation of formula (3), the new grayscale value of the grayscale is obtained, which is recorded as s k ∈[0,1]; after this operation, the brightness and contrast of the image tend to be consistent;

[0073] Step 6-3: In constructing a deep learning algorithm for recyclable object recognition, positioning, classification, and rotation angle detection, the present invention mainly uses the RTMDet algorithm. This is because the applicability of the gripper 6 of the industrial robot 5 is taken into consideration, so the rotation angle of the object is added to control the rotation direction of the gripper 6 when grasping;

[0074] When using deep learning algorithms such as YOLO and Faster RCNN, you can add a derivation module for predicting the rotation angle and a function for calculating the rotation angle of the object recognition box.

[0075] Step 6-4: Use the expanded data set in Step 5 to train the algorithm, retain the optimal parameters of the algorithm, and use it for offline use;

[0076] Step 7: Set the graspable area of ​​the object to adapt to the grasping range of the gripper 6 of the industrial robot 5 (such as Figure 4 As shown in the figure), duplicate objects are filtered out through a deduplication algorithm to avoid repeated grabbing. The specific steps of the deduplication algorithm are as follows:

[0077] Step 7-1: According to the size and grasping height of the gripper 6 of the industrial robot 5, the mapping width range of the graspable area of ​​the object under the camera is defined ( Figure 4 The grasping judgment area in the image is defined, and the height of the grasping judgment area is defined within the safety area captured by the set industrial camera 2 (the motion range of the object involved in the deduplication calculation);

[0078] Step 7-2: After the object enters the grasping judgment area, the present invention simplifies the deduplication operation and uses a cosine similarity detection algorithm with lower computing power requirements to determine whether the objects are consistent, as shown in formula (4):

[0079]

[0080] In the formula, A and B represent the object areas of the previous and next two frames respectively; similar objects are screened out by similarity threshold;

[0081] Step 7-3: Calculate the duration of similar objects in the grasping judgment area and set a threshold. When the duration exceeds the threshold, it is judged as a new object.

[0082] Step 7-4: Calculate whether the object is beyond the graspable area;

[0083] Step 7-5: Place the deduplicated object information into the sending queue, waiting for the network communication between the industrial computer 4 and the industrial robot 5;

[0084] Step 8: Figure 1 As shown in the grasping judgment and industrial robot 5 communication part, the industrial computer 4 and the industrial robot 5 use Socket communication (the industrial computer 4 is the client and the industrial robot 5 is the host); in order to ensure that the communication between the industrial computer 4 and the industrial robot 5 remains stable, a network monitoring judgment algorithm is set at the industrial computer 4, such as Figure 3 As shown in the figure, in the client, the communication information of the host is constantly monitored, and the sending channel of the Socket is tested to see if it exists. If the sending channel is disconnected, the communication parameters are updated immediately to ensure the stability of the client software.

[0085] Step 9: Recyclable items will be grabbed by the industrial robot 5 and placed in a classified manner for subsequent processing steps; non-recyclable objects will be transported to the next process by the conveyor belt.

Claims

1. The recyclable waste identification and sorting system based on RTMDet algorithm is characterized by: It includes a belt line, an industrial robot is arranged on one side of the belt line, and a gripper is arranged on the industrial robot; An industrial camera is arranged on the other side of the belt line, and a light shield is arranged on the industrial camera.

2. The recyclable garbage identification and sorting system based on the RTMDet algorithm according to claim 1 is characterized in that: It also includes an industrial computer, one end of which is connected to the belt line through a wire, and the other end of which is connected to the industrial camera through a wire.

3. The method for identifying performance of a recyclable garbage identification and sorting system based on the RTMDet algorithm according to any one of claims 1 to 2, characterized in that: The operation steps are as follows: Step (1): construct a camera and robot sorting device for identifying and detecting recyclable waste and performing batch processing; Step (2): Manually sort out a small batch of data for training data collection and annotation to produce labeled data; Step (3): Expand the training data, design and use image enhancement algorithms to improve the image recognition capabilities under different lighting intensity conditions and at different focal lengths of industrial cameras; Step (4): Use the histogram equalization method to equalize the image histogram collected by the industrial camera to balance the image lighting conditions; Step (5): Use a rotation detection algorithm based on deep learning to identify the category of the recyclable object and calculate the rotation angle of the object; Step (6): Calculate the similarity between adjacent frames, remove similar frames, and complete the deduplication operation; Step (7): Establish network communication with the robot, send the detected object information to the industrial robot, and guide the industrial robot to grasp the object through the gripper.

4. The method for identifying performance of a recyclable garbage identification and sorting system based on the RTMDet algorithm according to claim 3, characterized in that: In step (1), the automated line for batch processing of recyclable waste includes: an industrial camera, a light shield, a light source, a belt line, an industrial robot, and an industrial computer for realizing identification and control; Wherein, the industrial robots include multi-axis robots and parallel robots.

5. The method for identifying performance of a recyclable garbage identification and sorting system based on the RTMDet algorithm according to claim 3, characterized in that: In step (2), the steps of data collection and labeling are as follows: (2.1): Use the garbage sorting device constructed in step (1) to collect recyclable garbage data; (2.2): Select images that can be used for deep learning model training based on the types of garbage, keeping the number of types to be detected similar; (2.3): Collect images of different levels of garbage density, including images of garbage accumulation and occlusion; (2.4): Use the labeling tool X-AnyLabeling software for manual labeling, select the object to be detected according to the gripping rotation angle of the industrial robot, and mark the category.

6. The method for identifying performance of a recyclable garbage identification and sorting system based on the RTMDet algorithm according to claim 3 is characterized in that: In step (3), the data set expansion method includes operations such as adjusting image brightness and cropping images, and using generated random numbers to control image brightness and the proportion of cropped images.

7. The method for identifying performance of a recyclable garbage identification and sorting system based on the RTMDet algorithm according to claim 3 is characterized in that: In step (4), the histogram equalization method refers to a method of adjusting the brightness and contrast of an image before using a deep learning algorithm to detect an object. The specific operation steps are as follows: (4.1): Count the grayscale distribution of the image and calculate the number of pixels contained in each grayscale; (4.2): Use the maximum and minimum values ​​to normalize the image grayscale; (4.3): Use the cumulative distribution function to calculate the new grayscale value of the image; (4.4): Inverse normalization calculates the new grayscale value of the image.

8. The method for identifying performance of a recyclable garbage identification and sorting system based on the RTMDet algorithm according to claim 3, characterized in that: In step (5), the specific operation steps of the deep learning algorithm for recyclable waste identification and detection are as follows: (5.1): The deep learning algorithm takes the image obtained after the histogram equalization operation as input, which contains three RGB channel images; (5.2): Extract image color, shape, and contextual semantic features through convolutional networks; (5.3): Calculate the output image category, the positioning frame of the object to be detected, and the rotation angle of the positioning frame; (5.4): Add the detected object information to the queue.

9. The method for identifying performance of a recyclable garbage identification and sorting system based on the RTMDet algorithm according to claim 3, characterized in that: In step (6), the method for calculating the similarity of adjacent frames is to calculate the cosine similarity of the detected object between two frames, and the specific operation steps are as follows: (6.1): Set the detection area in the image; (6.1): Calculate the cosine similarity between the detected objects in the current frame and the previous frame; (6.3): Set a similarity threshold, and users above the threshold will be kicked out; (6.4): Calculate the time that similar objects are in the area to be detected; (6.5): Filter out similar objects with time intervals below the interval.

10. The method for identifying performance of a recyclable garbage identification and sorting system based on the RTMDet algorithm according to claim 3, characterized in that: In step (7), the method used for robot communication is Socket communication, which uses an industrial computer as a client and establishes a connection with the industrial robot through a TCP / IP port; It adopts a multi-threaded approach to continuously monitor the network communication of the industrial robot, and sends the location and rotation angle of the detected object to the industrial robot. After the industrial robot establishes a connection and receives the Socket socket, it adjusts the posture of the industrial robot according to the location and angle to achieve grasping.

Images