Intelligent garbage sorting system and method based on visual grabbing of mechanical arm

Abstract

The invention discloses an intelligent garbage sorting system and method based on mechanical arm visual grabbing, the system comprises an upper computer, a depth camera and a mechanical arm, and a visual perception module, a target recognition module, a grabbing prediction module and a planning control module are integrated in the upper computer; the visual perception module is used for controlling the depth camera to obtain junk RGB-D information in a scene; the target recognition module receives the RGB information, performs garbage recognition by adopting a multi-target detection recognition algorithm, recognizes garbage types and position information in a scene, and obtains a local depth image according to the garbage position information and depth information; the grabbing prediction module is used for carrying out grabbing pose prediction by adopting a grabbing pose prediction algorithm according to the local depth image to obtain an optimal grabbing pose; the planning control module completes coordinate transformation according to the optimal grabbing posture and controls the mechanical arm to execute grabbing and sorting tasks according to the grabbing posture after coordinate transformation and the garbage category; the accuracy and efficiency of garbage classification are improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent garbage sorting, and in particular relates to an intelligent garbage sorting system and method based on mechanical arm visual grasping. Background Art

[0002] At present, the problem of urban domestic waste is becoming more and more serious. The rapid advancement of urbanization and the improvement of residents' living standards have led to an increase in the amount of urban domestic waste generated year by year. This situation has put forward higher requirements for the collection, classification and treatment of waste. It is urgent to improve management and treatment capabilities to cope with the increasing amount of waste and the resulting environmental pressure.

[0003] The existing automated sorting facilities generally have problems with under-detection and missed picking when dealing with small and medium-sized target garbage. On the one hand, due to the wide variety of garbage, different shapes, and frequent occlusion and stacking, traditional target detection networks find it difficult to accurately identify and locate small and medium-sized target garbage. For example, small-volume garbage such as waste batteries and discarded medicines are easily misjudged as background or ignored. On the other hand, the existing grasping prediction network lacks accuracy when dealing with small and medium-sized targets, making it difficult to generate reliable grasping postures, resulting in the robot arm being unable to accurately grasp the target garbage. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide an intelligent garbage sorting system based on robotic arm visual grasping to solve the problems of low efficiency, inaccurate classification and difficulty in adapting to garbage of multiple sizes in existing garbage sorting methods.

[0005] Another object of the present invention is to provide an intelligent garbage sorting method based on robotic arm visual grasping.

[0006] Technical solution: The present invention discloses an intelligent garbage sorting system based on mechanical arm visual grasping, comprising a host computer, a depth camera and a mechanical arm. The host computer is internally integrated with a visual perception module, a target recognition module, a grasping prediction module and a planning control module. The visual perception module is used to control the depth camera to obtain RGB-D information of garbage in the scene; the target recognition module receives the RGB information and adopts a multi-target detection and recognition algorithm to identify the garbage, identify the garbage category and location information in the scene, and obtain a local depth image according to the garbage location information and depth information; the grasping prediction module adopts a grasping posture prediction algorithm to predict the grasping posture according to the local depth image, and obtains the optimal grasping posture; the planning control module completes the coordinate conversion according to the optimal grasping posture, and controls the mechanical arm to perform grasping and sorting tasks according to the grasping posture and garbage category after the coordinate transformation;

[0007] Among them, in the multi-object detection and recognition algorithm, garbage recognition is performed based on the improved YOLOV8 network. The improved YOLOV8 network replaces the second-to-last and third C2f modules in the backbone network of the YOLOV8 network with deformable convolutional DCN modules; in the grasping pose prediction algorithm, the improved GGCNN network is used for grasping pose prediction. The improved GGCNN network introduces an RFB multi-scale receptive field between the Conv and TConv of the GGCNN network.

[0008] Optionally, the robotic arm and the depth camera form a motion control part. The installation method is that the eye is outside the robotic arm. The upper computer communicates with the depth camera through a serial port and generates the control trajectory and its commands for the robotic arm based on the captured images.

[0009] Optionally, the visual perception module controls the depth camera to read the RGB-D information of the garbage in the scene at a preset time interval or after receiving a trigger signal.

[0010] Optionally, the target recognition module preprocesses the read RGB information, including image enhancement and normalization operations, and then inputs it into the improved YOLOV8 network for garbage recognition.

[0011] Optionally, when the planning and control module performs coordinate transformation, it converts the optimal grasping pose information from the image coordinate system to the camera coordinate system, and then to the robotic arm coordinate system to ensure that the robotic arm can accurately reach the grasping position.

[0012] Optionally, the robotic arm is a six-degree-of-freedom robotic arm, which can flexibly adjust its posture to complete garbage grasping and sorting tasks at different positions and angles.

[0013] The present invention also provides a garbage classification and sorting method for sampling a garbage intelligent sorting system based on robotic arm vision grasping, including the following steps:

[0014] (1) The visual perception module controls the depth camera to detect the area to be classified, obtains the RGB information and depth information of the garbage in the area, and the visual perception module registers the RGB information and depth information;

[0015] (2) The target recognition module obtains the RGB information published by the visual perception module, uses a multi-object detection and recognition algorithm to perform multi-object recognition on the RGB information, obtains the category information and position information of the object to be grasped, and obtains the local depth image corresponding to the garbage position information according to the garbage position information and depth information;

[0016] Among them, the multi-object detection and recognition algorithm uses an improved YOLOV8 network for garbage recognition. The improved YOLOV8 network replaces the penultimate and the third C2f modules in the backbone network of the YOLOV8 network with deformable convolutional DCN modules;

[0017] (3) The grasping prediction module obtains the local depth image, and uses the grasping pose prediction algorithm to predict the grasping pose of the local depth image at the position to be grasped, and obtains the optimal grasping pose;

[0018] Among them, the grasping prediction module uses an improved GGCNN network for grasping pose prediction. The improved GGCNN network introduces an RFB multi-scale receptive field between the Conv and TConv of the GGCNN network;

[0019] (4) The planning and control module obtains the optimal grasping pose, and performs the conversion of the image coordinates → camera coordinates → robotic arm base coordinates of the optimal grasping pose, uses the robot inverse kinematics to solve the joint motion process, and then controls the robotic arm and gripper according to the category information to complete the grasping and sorting tasks.

[0020] Further, the visual perception module controls the depth camera to read the RGB-D information of the garbage in the scene at a preset time interval or after receiving a trigger signal.

[0021] Further, the target recognition module preprocesses the read RGB information, including image enhancement and normalization operations, and then inputs it into the improved YOLOV8 network for target recognition.

[0022] The present invention also provides an electronic device, including a memory, a processor, and a computer program / instructions stored on the memory and executable on the processor. When the computer program / instructions are executed by the processor, the steps of the garbage sorting method are implemented.

[0023] Beneficial effects: Compared with the prior art, the remarkable technical effects of the present invention are as follows: The deformable convolution and multi-scale receptive fields are utilized to improve the detection ability of the target detection network and the grasping prediction network for small targets; the deformable convolution can adaptively adjust the sampling positions of the convolution kernels, better capturing the feature information of small targets; the improved YOLOV8 network has a 4% increase in the accuracy rate in the self-built garbage classification dataset compared with the traditional YOLOV8 network, and the accuracy rate for small target detection is significantly increased by 10%; the multi-scale receptive fields can process targets of different scales simultaneously, improving the network's detection ability for garbage of different sizes; the improved GGCNN network has significantly improved its detection ability for objects of multiple sizes, reducing the phenomena of missing detection and false negative detection, and can meet the requirements of garbage classification for different body types; through this innovative design, it is expected to solve the problem of missing detection and false negative detection of medium and small targets in current automated sorting facilities, improve the accuracy and efficiency of garbage classification, and provide strong support for the efficient treatment of garbage and the maximum recycling of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is the software structure schematic diagram of the present invention;

[0025] Figure 2 is the hardware structure schematic diagram of the present invention;

[0026] Figure 3 is the schematic diagram of the improved YOLOV8 network structure;

[0027] Figure 4 is the schematic diagram of the improved GGCNN network structure;

[0028] Figure 5 is the grasping pose generation diagram, where (a) is the optimal grasping pose of the garbage in the upright pose (0° rotation angle), (b) is the optimal grasping pose of the garbage in the tilted pose (45° tilt angle), and (c) is the optimal grasping pose of the garbage in the inverted pose (90° rotation angle);

[0029] Figure 6 is the effect diagram of garbage recognition in a multi-target scenario, where (a) is the target detection result when five garbage are present, (b) is the target detection result when three garbage are present, (c) is the target detection result when two garbage are present, and (d) is the target detection result when a single garbage is present;

[0030] Figure 7 is the whole-process diagram of garbage classification, where (a) is the algorithm for target detection and generating the grasping anchor box of a certain target; (b) is the robotic arm grasping the target; (c) is the robotic arm successfully grasping and sorting, and (d) is the robotic arm completing the sorting. DETAILED DESCRIPTION OF THE INVENTION

[0031] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] A system relying on computer vision is studied for the identification, classification, and localization of garbage, and combined with intelligent hardware such as a robotic arm to achieve the grasping and sorting of garbage in complex stacking scenarios, which can not only significantly reduce the labor intensity of manual sorting but also improve the classification accuracy and pipeline efficiency.

[0033] The present invention conducts automated sorting of garbage in response to the complex and diverse types of garbage and the classification requirements of multiple sizes and shapes in the garbage classification scenario. As Figure 1 shown, the intelligent garbage sorting system based on robotic arm vision grasping of the present invention includes a host computer, a depth camera, and a robotic arm. The depth camera is installed in a suitable position, and the calibration method is eye - in - hand - out - of - arm, which is used to collect RGB - D information of the garbage scene and transmit the data to the host computer. The depth camera can simultaneously obtain the color image (RGB information) and depth image (depth information) of the garbage, providing comprehensive data support for subsequent identification and grasping; the host computer internally integrates a visual perception module, a target recognition module, a grasping prediction module, and a planning and control module. The host computer serves as the control core of the system, running each module for data processing and decision - making, and the modules work together to achieve automated identification of garbage; the robotic arm is connected to the host computer, receives the control instructions of the host computer, and completes the grasping and sorting actions of the garbage.

[0034] The robotic arm and the depth camera form the motion control part, and the installation method is eye - in - hand - out - of - arm. The host computer communicates with the depth camera through a serial port, and generates the control trajectory and its commands of the robotic arm based on the captured pictures.

[0035] As Figure 2 shown, the visual perception module is responsible for interacting with the depth camera, used to control the depth camera to obtain the RGB - D information of the garbage in the scene, including the RGB image and depth image of the garbage in the scene, and register the depth image with the RGB image; the target recognition module receives the RGB information to identify the types of garbage, identifies the types of garbage in the scene, and transmits the detection results to the grasping prediction module; the grasping prediction module predicts the grasping pose according to the detection results identified by the target recognition module, and uploads the obtained grasping pose information to the planning and control module; the planning and control module completes coordinate transformation and controls the robotic arm to execute the grasping and sorting tasks. Through data transmission and instruction interaction between the modules, a complete software control process is formed.

[0036] The specific functions of each module are realized as follows:

[0037] Visual perception module: It can control the depth camera to collect data and read the RGB-D information of the garbage in the scene according to the preset time interval or after receiving a specific trigger signal. In actual operation, the time interval can be reasonably set according to the speed and efficiency requirements of garbage disposal. For example, for scenes with large garbage flow, the time interval can be appropriately shortened to ensure that new garbage information is obtained in time. The RGB-D information collected by the depth camera contains the color image (RGB information) and depth image (depth information) of the garbage. This information will be uploaded to the host computer for subsequent processing. The visual perception module aligns the RGB information and the depth information. The specific alignment is to match the RGB information with the depth information one by one, which is completed by the program provided by the depth camera manufacturer.

[0038] Target recognition module: used to read RGB information and use multi-target detection and recognition algorithm to identify the target, identify the garbage category and location information in the scene, and obtain the local depth image corresponding to the garbage location information based on the garbage location information and depth information; in order to improve the accuracy and adaptability of garbage recognition, the improved YOLOV8 network is used in the multi-target detection and recognition algorithm for garbage recognition. Figure 3 The figure shows the schematic diagram of the improved YOLOV8 network structure. In order to adapt to the problem of irregular shapes and multiple sizes of garbage in garbage classification scenarios, the penultimate and third C2f modules in the backbone network of the YOLOV8 network are replaced with deformable convolution DCN modules (i.e. C2F_DCN) to enhance the network's detection effect for small and medium-sized garbage (such as used batteries, expired medicines, etc.). The deformable convolution DCN module can adaptively adjust the sampling position of the convolution kernel by learning the offset, so as to better capture the characteristics of garbage of different sizes and shapes and improve the recognition accuracy.

[0039] In deformable convolution, it achieves the deformation effect of the convolution kernel by adjusting the sampling points. n To expand p n , p n represents a fixed offset relative to p0 (e.g. (-1,0) means a shift of 1 pixel to the left), {Δp n |n=1,2,…,N} is the offset generated by the input feature map and another convolution layer, N is the number of elements in the sampling network R, which causes the sampling position of the convolution kernel on the input feature map to shift. The following formula shows the calculation steps of deformable convolution:

[0040]

[0041] Among them, y(p0) is the output value of the next layer of the network, w(p n ) is the convolution kernel at position p n The weight parameter at x(p0+pn +Δp n ) is the pixel value of the input feature map at the position (p0 + p n +Δp n ). p0 is the current position coordinate (center point) on the output feature map, and R is the sampling network of the convolution kernel.

[0042] The deformable convolution DCN learns an offset for each point on the convolution kernel, enabling the convolution kernel to deform freely. These offsets are learned through backpropagation, allowing the convolution kernel to adapt to the spatial variations of the input features. For example, when dealing with rotations, scalings, or other non-rigid transformations, the deformable convolution can capture target features more accurately.

[0043] Data preprocessing:

[0044] Before inputting the RGB information into the improved YOLOV8 network, the target recognition module performs preprocessing on it. Specifically, it includes image enhancement operations such as adjusting brightness, contrast, saturation, etc., to improve the clarity and feature expressiveness of the image; it also performs normalization operations to normalize the pixel values of the image to a specific range, facilitating the training and recognition of the network. The preprocessed image will be input into the improved YOLOV8 network for target recognition, identifying the garbage categories and location information in the scene, and obtaining a local depth image based on the location information and depth information, and transmitting the local depth image to the grasping prediction module.

[0045] Grasping prediction module: Reads the local depth image perceived by the target recognition module, uses the grasping pose prediction algorithm to predict the grasping pose, obtains the optimal grasping pose, and uploads the obtained optimal grasping pose information to the planning and control module. The present invention uses the GGCNN network for grasping recognition. To adapt to the problems of irregular shapes and multiple sizes of garbage in the garbage classification scenario, an RFB multi-scale receptive field is introduced into the GGCNN network to obtain an improved GGCNN network. As Figure 4 shown, it is a schematic diagram of the improved GGCNN network structure. In this embodiment, the improved GGCNN network is used for grasping recognition. The RFB multi-scale receptive field can simulate the multi-scale perception mechanism of the human visual system, enabling the network to capture features of different scales simultaneously, thereby improving the grasping pose prediction ability for garbage of multiple sizes. When performing grasping pose prediction, the grasping prediction module will combine the type, size, and shape information of the garbage, and comprehensively consider various factors to improve the accuracy of grasping pose prediction.

[0046] The input layer of the RFB module receives the feature map from the previous layer as input. Parallel branches: It contains multiple parallel convolutional branches, and each branch uses convolutional kernels of different scales and dilated convolutions. The first branch uses a smaller convolutional kernel (such as 1x1) and a smaller dilation rate to extract detailed features; the subsequent branches use larger convolutional kernels (such as 3x3, 5x5) and larger dilation rates to obtain large-scale feature information. The concatenation layer concatenates the output feature maps of each branch in the channel dimension to obtain a feature map containing features of different scales. The fusion layer performs feature fusion on the concatenated feature map through a 1x1 convolutional layer to reduce the number of channels and enhance the feature representation ability at the same time. Output layer: Outputs the fused multi-scale feature map as the input of the next layer.

[0047] Grasping pose prediction:

[0048] The grasping prediction module reads the local depth image perceived by the target recognition module and performs grasping pose prediction. By analyzing the features of the garbage and model calculation, it determines the optimal grasping position and pose, and uploads the obtained grasping pose information to the planning and control module. Figure 5 Figures (a) to (c) show the generated grasping pose diagrams, visually presenting the predicted grasping poses.

[0049] Planning and control module: After receiving the optimal grasping pose information, the planning and control module first performs coordinate transformation. Since the grasping pose information is obtained in the camera coordinate system, while the movement of the robotic arm is based on the robotic arm coordinate system, it is necessary to transform the grasping pose information from the camera coordinate system to the robotic arm coordinate system to ensure that the robotic arm can accurately reach the grasping position. Through precise coordinate transformation, it ensures that the robotic arm can accurately reach the grasping position.

[0050] Then, the planning and control module plans the movement trajectory of the robotic arm according to the transformed coordinate information. The robotic arm is a multi-degree-of-freedom robotic arm that can flexibly adjust its pose to complete garbage grasping and sorting tasks at different positions and angles. The planning and control module manipulates the robotic arm to move along the planned trajectory and completes the garbage grasping and sorting actions according to the garbage category obtained by the target recognition module.

[0051] In order to achieve the working coordination between different sensing devices, before the system works, it is necessary to calibrate relevant parameters, including camera parameter calibration, calibration of the relationship between the robotic arm and the camera, etc.

[0052] The garbage sorting method based on deep learning includes the following steps:

[0053] (1) The visual perception module controls the depth camera to detect the area to be classified and obtains the RGB information and depth information of the garbage in the area. The visual perception module aligns the RGB information and the depth information. The specific content of the alignment is to match the RGB image data with the depth information one by one, which is completed by the program provided by the depth camera manufacturer.

[0054] (2) The target recognition module obtains the RGB information released by the visual perception module, uses the multi-target detection and recognition algorithm to perform multi-target recognition on the RGB information, obtains the category information and location information of the object to be captured, and obtains the local depth image corresponding to the garbage location information based on the garbage location information and depth information;

[0055] Among them, the multi-target detection and recognition algorithm uses an improved YOLOV8 network for garbage identification. The improved YOLOV8 network replaces the penultimate and third C2f modules with deformable convolution DCN modules in the backbone network of the YOLOV8 network.

[0056] (3) The grasping prediction module obtains the local depth image and uses the grasping posture prediction algorithm to predict the grasping posture of the local depth image of the grasping position to obtain the optimal grasping posture;

[0057] Among them, the grasping prediction module uses an improved GGCNN network to predict grasping posture. The improved GGCNN network introduces the RFB multi-scale receptive field between the Conv and TConv of the GGCNN network.

[0058] (4) The planning control module obtains the optimal grasping posture and performs the conversion from image coordinates of the optimal grasping posture to camera coordinates to robot arm base coordinates. The robot inverse kinematics is used to solve the joint motion process, and then the robot arm and gripper are controlled according to the category information to complete the grasping and sorting tasks.

[0059] System operation effect:

[0060] Garbage identification in multi-target scenarios: Figure 6 (a) to (d) show the effect diagram of garbage identification in a multi-target scene. It can be seen from the figure that the system of the present invention can accurately identify multiple different types of garbage in the scene, proving the effectiveness and accuracy of the improved YOLOV8 network in garbage identification.

[0061] The whole process of garbage classification: Figure 7 Figures (a) to (d) are the entire process of garbage classification, which clearly show the entire process from garbage information collection, identification, grasping posture prediction to the completion of grasping and sorting by the robotic arm. Through the collaborative work of various modules, the system realizes the automatic classification and sorting of garbage, improving the efficiency and accuracy of garbage classification.

Claims

1. An intelligent garbage sorting system based on robotic arm vision grasping, characterized in that, It includes a host computer, a depth camera, and a robotic arm. The host computer is internally integrated with a visual perception module, a target recognition module, a grasping prediction module, and a planning and control module. The visual perception module is used to control the depth camera to obtain the RGB-D information of the garbage in the scene; the target recognition module receives the RGB information and uses a multi-target detection and recognition algorithm to identify the garbage, identifies the garbage category and location information in the scene, and obtains a local depth image based on the garbage location information and depth information; the grasping prediction module uses a grasping pose prediction algorithm based on the local depth image to predict the grasping pose and obtain the optimal grasping pose; the planning and control module completes coordinate transformation according to the optimal grasping pose, and controls the robotic arm to perform grasping and sorting tasks according to the grasping pose and garbage category after coordinate transformation. Among them, in the multi-target detection and recognition algorithm, garbage recognition is performed based on the improved YOLOV8 network. The improved YOLOV8 network replaces the penultimate and third C2f modules in the backbone network of the YOLOV8 network with deformable convolutional DCN modules; in the grasping pose prediction algorithm, the improved GGCNN network is used to predict the grasping pose. The improved GGCNN network introduces an RFB multi-scale receptive field between the Conv and TConv of the GGCNN network.

2. The garbage intelligent sorting system based on robotic arm vision grasping according to claim 1, characterized in that, The robotic arm and the depth camera form the motion control part. The installation method is that the eye is outside the robotic arm. The host computer communicates with the depth camera through a serial port and generates the control trajectory and its commands of the robotic arm based on the captured pictures.

3. The intelligent garbage sorting system based on robotic arm vision grasping according to claim 1, wherein The visual perception module controls the depth camera to read the RGB-D information of the garbage in the scene at a preset time interval or after receiving a trigger signal.

4. The intelligent garbage sorting system based on robotic arm vision grasping according to claim 1, wherein The target recognition module preprocesses the read RGB information, including image enhancement and normalization operations, and then inputs it into the improved YOLOV8 network for garbage recognition.

5. The intelligent garbage sorting system based on robotic arm visual grasping according to claim 1, characterized in that, When the planning and control module performs coordinate transformation, it converts the optimal grasping pose information from the image coordinate system to the camera coordinate system, and then to the robotic arm coordinate system to ensure that the robotic arm can accurately reach the grasping position.

6. The garbage intelligent sorting system based on robotic arm vision grasping according to claim 1, characterized in that, The robotic arm is a six-degree-of-freedom robotic arm that can flexibly adjust its posture to complete garbage grasping and sorting tasks at different positions and angles.

7. A garbage classification and sorting method for sampling the garbage intelligent sorting system based on robotic arm vision grasping according to any one of claims 1 to 6, characterized in that, It includes the following steps: (1) The visual perception module controls the depth camera to detect the area to be classified, obtains the RGB information and depth information of the garbage in the area, and the visual perception module registers the RGB information and depth information. (2) The target recognition module obtains the RGB information published by the visual perception module, uses a multi-target detection and recognition algorithm to perform multi-target recognition on the RGB information, obtains the category information and location information of the object to be grasped, and obtains a local depth image corresponding to the garbage location information based on the garbage location information and depth information. Among them, the multi-target detection and recognition algorithm uses the improved YOLOV8 network for garbage recognition. The improved YOLOV8 network replaces the penultimate and third C2f modules in the backbone network of the YOLOV8 network with deformable convolutional DCN modules. (3) The grasping prediction module obtains the local depth image, and uses the grasping pose prediction algorithm to predict the grasping pose of the local depth image at the position to be grasped, and obtains the optimal grasping pose; Among them, the grasping prediction module uses the improved GGCNN network to predict the grasping pose. The improved GGCNN network introduces the RFB multi-scale receptive field between the Conv and TConv of the GGCNN network; (4) The planning and control module obtains the optimal grasping pose, and performs the conversion of the image coordinates → camera coordinates → manipulator base coordinates of the optimal grasping pose, uses the inverse kinematics of the robot to solve the joint motion process, and then controls the manipulator and gripper according to the category information to complete the grasping and sorting tasks.

8. The garbage classification and sorting method according to claim 7, characterized in that, The visual perception module controls the depth camera to read the RGB-D information of the garbage in the scene at a preset time interval or after receiving a trigger signal.

9. The waste sorting method according to claim 7, characterized in that, The target recognition module preprocesses the read RGB information, including image enhancement and normalization operations, and then inputs it into the improved YOLOV8 network for target recognition.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program / instructions stored on the memory and executable on the processor, wherein the computer program / instructions, when executed by the processor, implement the steps of the garbage sorting method according to any one of claims 7-9.

Images