Intelligent garbage sorting method and system based on multi-robot cooperative work

Abstract

The invention relates to an intelligent garbage sorting method and system based on multi-robot cooperative work, and the method comprises the following steps: S1, completing garbage image collection through a visual sensor, and obtaining multi-sensor data through a laser radar and an infrared spectrometer; garbage is classified, and accurate coordinates are obtained; s3, the classification result and the accurate coordinates are transmitted to a robot scheduling module, and the robot scheduling module adopts an auction algorithm to carry out task allocation; s4, a dynamic environment map is built by combining a visual sensor with a laser radar, and each robot senses the environment according to the real-time states of the conveyor belt and other robots; and S5, the robot senses environment information based on the task distribution result and the real-time state, adjusts a grabbing route, grabs the garbage and sorts the garbage into the storage boxes of the corresponding types. The intelligent garbage sorting efficiency and accuracy can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of robot cooperative control, and particularly to an intelligent garbage sorting method and system based on multi-robot cooperative work. Background Art

[0002] In the garbage collection and sorting industry, with the acceleration of urbanization and the improvement of environmental protection requirements, garbage classification has become a global challenge. However, in the current system, most garbage sorting operations are still mainly manual. Although automated systems have been promoted in some fields, traditional automated sorting systems mainly adopt the form of fixed robotic arms combined with conveyor belts, and are equipped with basic vision systems to identify and grab garbage. The deficiencies of such systems in large-scale garbage sorting scenarios are as follows: Single sensor dependence: Many sorting systems rely only on visual detection for garbage recognition, and it is difficult to accurately sort in scenarios with light changes, a large number of occlusions, and similar garbage materials (such as a mixture of plastics and glass). Insufficient sorting accuracy: Lack of high-precision sensors such as spectral analysis, making it difficult to accurately classify industrial garbage or complex mixed garbage. Lack of dynamic adjustment ability: Traditional systems can usually only handle static or weakly dynamic environments (such as garbage on a stable conveyor belt), and the sorting failure rate is relatively high when facing position changes or dynamically conveyed garbage. Summary of the Invention

[0003] In order to solve the above problems, the purpose of the present invention is to provide an intelligent garbage sorting method and system based on multi-robot cooperative work, which can significantly improve the efficiency and accuracy of intelligent garbage sorting.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions: An intelligent garbage sorting method based on multi-robot cooperative work, comprising the following steps: S1: Garbage is conveyed to the camera detection area by a conveyor belt, and garbage image acquisition is completed through a vision sensor. Multi-sensor data is obtained through a lidar and an infrared spectrometer, and image features and multi-sensor features are extracted: S2: Combining the image features and multi-sensor features, based on a multi-level classification model, the garbage is classified and accurate coordinates are obtained; S3: The classification result and accurate coordinates are transmitted to the robot scheduling module, and the robot scheduling module uses an auction algorithm for task allocation; S4: Using a vision sensor combined with a lidar, a dynamic environment map is built, and each robot perceives the environment according to the real-time states of the conveyor belt and other robots; S5: Based on the task allocation result and the real-time state perception environment information, the robot adjusts the grasping route, grabs the garbage and sorts it into the corresponding type of storage box.

[0005] Further, S1 is specifically as follows: Use a high-speed industrial RGB camera to capture the image data of the garbage. The image data is used to extract the image features F of the garbage through a convolutional neural network (CNN). v ; The lidar is installed above the detection area to scan the surface of the garbage with multi-beam laser to obtain the point cloud feature F l ; The infrared spectrometer collects the absorption or reflection spectrum of the infrared light on the surface of the garbage to generate a spectral curve. In the spectral curve, the absorption peak or reflection characteristics of each material are different. According to the Lambert-Beer law, the spectral feature F is extracted s ; Align the data collected by each sensor in the world coordinate system and perform timestamp synchronization processing.

[0006] Further, the multi-level classification model includes a lightweight fast classification sub-model, a fine classification sub-model, and a precise localization sub-model, which are specifically as follows: The lightweight fast classification sub-model is based on the image feature F v , and uses a lightweight algorithm to quickly identify the large categories of the garbage to complete real-time rough classification; and provides a preliminary bounding box for the location of the garbage area as the input for further fine classification and coordinate optimization; The fine classification sub-model combines the point cloud feature F l and the spectral feature F s for further fine classification using high-precision classification; The precise localization sub-model combines the rough classification area of the first layer with the fine classification result of the second layer, and determines the category and three-dimensional precise coordinates of the garbage through the three-dimensional shape and infrared spectral material information provided by the lidar point cloud data.

[0007] Further, use the lightweight algorithm YOLO Tiny to quickly identify the large categories of the garbage, specifically as follows: Input the extracted image feature F v into the fully connected layer, and combine the classification weight W v and the bias b v to calculate the classification score:

[0008] where is the classification score of category c; Use the Softmax function to calculate the probability distribution of each category:

[0009] where P(c) is the confidence of category c; C is the total number of garbage categories;S k The classification score for category k; Classification result , that is, the most likely major category label (such as metal, plastic, etc.).

[0010] While classifying, YOLO Tiny will detect the location of the garbage and generate corresponding bounding boxes, predicting the parameters of the detection box B: ; Among them, (x1, y1) are the coordinates of the upper left corner of the bounding box; (x2, y2) are the coordinates of the lower right corner of the bounding box; Adjust the bounding box using the regression parameters predicted by YOLO Tiny, and the center point position ( x c , y c ) is predicted as: ; Among them, (x0, y0) is the default position of the candidate box; (t x , t y ) are the translation offsets output by the network; (w a , h a ) are the default width and height of the candidate box; is the activation function; The width and height of the bounding box ( w , h ) are predicted as:

[0011] Among them, (t w , t h ) are the scaling ratios of the bounding box.

[0012] Convert to the full bounding box form: ; The model finally outputs: the bounding box of each detection target and the corresponding category prediction and confidence.

[0013] Furthermore, for the uncertain major categories, the fine classification sub-model combines the point cloud feature F l and the spectral feature F s , and uses the high-precision model ResNet-RCNN to extract deep features for further fine classification, as follows: Crop the garbage area image from the bounding box B output by the lightweight model, and use ResNet-50 as the backbone network to extract deep image features as new image features : ; Among them, is the garbage area image cropped according to the bounding box B; and combined with the point cloud feature F l , spectral feature F s to form a combined feature vector: ; Among them, α1, α2, α3 are weight factors; Input the combined feature vector F into ResNet-RCNN: ; Among them, is the final class prediction of the garbage; W is the classifier weight matrix; b is the bias vector, and Softmax is the activation function.

[0014] Furthermore, for the precise localization sub-model, combine the rough classification area of the first layer with the fine classification result of the second layer, and determine the category and three-dimensional precise coordinates of the garbage through the three-dimensional shape and infrared spectral material information provided by the lidar point cloud data, specifically as follows: According to the center point (x c , y c ) corresponding to the lightweight model bounding box B, extract the target point area in the point cloud data, and obtain the relevant point cloud P region through the neighborhood detection algorithm: ; Calculate the mean value of its three-dimensional coordinates for the point cloud data (x c , y c ) belonging to the neighborhood Δ(x i , y i , z i ) to obtain the center point coordinates (X c , Y c , Z c ) of the target garbage object: ; Verify the material information in the corresponding point cloud area through spectral analysis. If the spectral material result is inconsistent with the fine classification, discard the current localization result; if it is consistent, determine the final garbage center.

[0015] Furthermore, S3 is specifically: Divide the recycling plant into M working areas, and the g-th working area is equipped with N k robots. Then the robot group R g in the g-th working area is expressed as: ; Among them, Indicates the th robot in the gth working area; Output the classification results and precise coordinates of each piece of garbage according to the multi-level classification model:

[0016] Among them, is the category of the jth piece of garbage; is the three-dimensional coordinate of the jth piece of garbage in the working area; is the task priority (set according to the garbage category or the processing time limit); is the output of the multi-level classification model; N is the number of garbage; Adopt a distributed auction algorithm to solve the task allocation problem. The robot group R in each working area k independently executes the auction algorithm and dynamically allocates tasks to the robots within its group. The goal is to maximize benefits and minimize costs: In the initial state, assume that all tasks and robots are in an unallocated state, and initialize the revenue to 0; Each task is bid by all idle robots, and the robot submits a bid : ; Among them, represents the cost for the robot to execute the task ; Record the current ; Task is allocated to the robot with the highest bid : ; The allocated task is removed from the task set; the robot with the allocated task updates to the target point and starts to execute; if there are unallocated tasks, repeat the above steps for the remaining task set until all tasks are allocated or all robots are busy. After the robot completes the task, it becomes available again and participates in the next task auction.

[0017] Furthermore, the dynamic environment map is constructed as follows: Define a two-dimensional coordinate system with a fixed grid resolution, and the state of all position points is represented by the occupancy probability :

[0018] Among them, Q is all the grids in the current environment map; each grid The occupancy probability is dynamically updated based on the lidar and vision sensors; N q is the total number of grids; The probability of updating the visual and lidar observations for each frame is: ; in, is the current observation z t Lower grid Probability of being occupied; is the sensor observation model likelihood; is the prior occupancy probability; is the normalization factor for all observations; Simplify the update using log-probability:

[0019] Combined with each frame data update, maintain real-time changing environment map M t , the map is divided into: static obstacle area; dynamic obstacle area.

[0020] Furthermore, the robot adjusts the grabbing route based on the task allocation results and real-time state perception of environmental information, grabs the garbage and sorts it into the corresponding type of storage box, as follows: From the environment map M t Extract the radius R around the robot p The local map of the robot; from the current position of the robot (x R ,y R ,z R ) Plan to the grab point (x) of the target garbage j ,y j ,z j ) path, the path needs to avoid obstacles and minimize the path cost; For each grid p on the path a , define the path cost as C(p a ): C(p a )=Ctravel(p a )+Cobstacle(p a )+Cdynamic(p a ); Among them, Ctravel(p a )=dist(p a-1 ,p a ) is the path length cost; Cobstacle(p a ) is the obstacle cost; Cobstacle(p a )=Wobs As the occupancy probability cost increases, W obs is the obstacle cost weight: Cdynamic(p a ) is the cost of dynamic objects, and an additional distance buffer is given according to the dynamic state of the lidar and visual perception of dynamic garbage or robot movement; The robot moves from the starting point (x R , y R ) to the target point (x j , y j ) The path cost is the sum of each grid p a The path cost C(p a ); Based on the environmental map M t , use the DWA dynamic window algorithm to search for the optimal path; Dynamically update the path at any time when the following events occur: the sample garbage position moves; other robots or dynamic obstacles enter the path; the sensor senses new map changes.

[0021] A garbage intelligent sorting system based on multi-robot collaborative work, including a processor, a memory, and a computer program stored on the memory. When the processor executes the computer program, it specifically executes the steps in a garbage intelligent sorting method based on multi-robot collaborative work as described above.

[0022] The present invention has the following beneficial effects: 1. The present invention combines multi-modal data of visual sensors, lidar, and infrared spectrometers to obtain the visual features, three-dimensional geometric features, and material features of garbage, providing more complete garbage feature information and greatly improving the classification accuracy; 2. The present invention adopts a strategy of combining lightweight fast classification and fine classification, quickly completing the preliminary classification, improving the real-time performance of the system, and performing fine classification on uncertain samples to ensure the classification accuracy, significantly improving the classification accuracy of complex garbage; 3. The present invention uses an auction algorithm for dynamic task allocation to achieve the optimal allocation of robot resources and improve the overall operation efficiency of the system; and constructs a dynamic environment map based on visual sensors and lidar to real-time sense the conveyor belt state and the positions of other robots, ensuring the stable operation of the system in a dynamic environment. Brief Description of the Drawings

[0023] Figure 1 It is a flowchart of the method of the present invention. Detailed Embodiments

[0024] The following further describes the present invention in detail with reference to the drawings and specific embodiments: ReferenceFigure 1 , in this embodiment, an intelligent garbage sorting method based on multi-robot collaborative work is provided, including the following steps: S1: The garbage is conveyed to the camera detection area by a conveyor belt, the garbage image is collected through a vision sensor, multi-sensor data is obtained through a lidar and an infrared spectrometer, and image features and multi-sensor features are extracted: S2: Combining the image features and multi-sensor features, based on a multi-level classification model, the garbage is classified and precise coordinates are obtained; S3: The classification result and precise coordinates are transmitted to the robot scheduling module, and the robot scheduling module uses the auction algorithm for task allocation; S4: Using the vision sensor combined with the lidar, a dynamic environment map is built, and each robot perceives the environment according to the real-time states of the conveyor belt and other robots; S5: Based on the task allocation result and the real-time state perception environment information, the robot adjusts the grasping route, grasps the garbage and sorts it into the storage bins of the corresponding type.

[0025] In this embodiment, S1 is specifically as follows: Use a high-speed industrial RGB camera to capture the image data of the garbage, and the image data extracts the image feature F of the garbage through a convolutional neural network (CNN) v (texture, color, shape, etc.); The lidar (LiDAR) is installed above the detection area, and the surface of the garbage is scanned with multi-beam lasers to obtain the point cloud feature F l ; The infrared spectrometer (IR) collects the absorption or reflection spectrum of the infrared light on the surface of the garbage to generate a spectral curve; in the spectral curve, the absorption peak or reflection characteristics of each material (such as metal, plastic, organic matter) are different, and according to the Lambert-Beer law, the spectral feature F is extracted s ; Align the data collected by each sensor with the world coordinate system and perform timestamp synchronization processing.

[0026] In this embodiment, the multi-level classification model includes a lightweight fast classification sub-model, a fine classification sub-model, and a precise positioning sub-model, specifically as follows: The lightweight fast classification sub-model, based on the image feature F v , uses a lightweight algorithm to quickly identify the large categories of garbage (such as metal, plastic, paper, organic matter, glass, etc.), completes real-time rough classification; and provides a preliminary bounding box for the location of the garbage area as the input for further fine classification and coordinate optimization; The fine classification sub-model, for the uncertain large categories, combines the point cloud feature F l and the spectral feature F s, further sub-classify using high-precision classification (such as distinguishing different plastic components, stainless steel from iron materials, etc.); Precision positioning sub-model, combines the rough classification area of the first layer with the fine classification results of the second layer, and determines the category and three-dimensional precise coordinates of the garbage through the three-dimensional shape and infrared spectrum material information provided by the lidar point cloud data.

[0027] In this embodiment, the lightweight algorithm YOLO Tiny is used to quickly identify the major categories of garbage, specifically as follows: Input the extracted image feature F v into the fully connected layer, combine with the classification weight W v and the bias b v , calculate the classification score:

[0028] Among them, is the classification score for category c; Use the Softmax function to calculate the probability distribution of each category:

[0029] Among them, P(c) is the confidence (probability) of category c (such as metal, plastic); C is the total number of garbage categories (such as: metal = 1, plastic = 2, organic matter = 3,...); S k is the classification score for category k; Classification result , that is, the most likely major category label (such as metal, plastic, etc.).

[0030] While classifying, YOLO Tiny will detect the location of the garbage and generate the corresponding bounding box, predicting the parameters of the detection box B: ; Among them, (x1,y1) is the upper left corner coordinate of the bounding box; (x2,y2) is the lower right corner coordinate of the bounding box; Use the regression parameters predicted by YOLO Tiny to adjust the bounding box, and predict the center point position ( x c , y c ) prediction: ; Among them, (x0, y0) is the default position of the candidate box; (t x , t y ) is the translation offset output by the network; (w a , h a ) are the default width and height of the candidate box; is the activation function; Bounding box width and height ( w , h ) prediction:

[0031] where (t w , t h ) is the scaling ratio of the bounding box.

[0032] Convert to the full bounding box form: ; The model finally outputs: the bounding box of each detected object , the corresponding class prediction and confidence.

[0033] In this embodiment, the fine classification sub-model, for the uncertain large categories, combines the point cloud feature F l and the spectral feature F s , uses the high-precision model ResNet-RCNN to extract deep features for further fine classification, as follows: The garbage area image cropped from the bounding box B output by the lightweight model is used to extract deep image features as new image features using ResNet-50 as the backbone network : ; where is the garbage area image cropped according to the bounding box B; and combines with the point cloud feature F l , the spectral feature F s to form a joint feature vector: ; where α1, α2, α3 are weight factors; Input the joint feature vector F into ResNet-RCNN: ; where is the final class prediction of the garbage; W is the classifier weight matrix; b is the bias vector, and Softmax is the activation function.

[0034] In this embodiment, the precise localization sub-model combines the rough classification area of the first layer with the fine classification result of the second layer, and determines the category and three-dimensional precise coordinates of the garbage through the three-dimensional shape and infrared spectral material information provided by the lidar point cloud data, as follows: According to the center point (x c , y c) Extract the target point region from the point cloud data, and obtain the relevant point cloud P through the neighborhood detection algorithm region : ; For the point cloud data (x c , y c ) belonging to the neighborhood Δ(x i , y i , z i ), calculate the mean value of its three-dimensional coordinates to obtain the center point coordinates (X c , Y c , Z c ) of the target garbage object: ; Verify the material information in the corresponding point cloud region through spectral analysis. If the spectral material result is inconsistent with the fine classification (such as the type should be metal, but the spectrum matches plastic), then discard the current positioning result; if it is consistent, determine the final garbage center.

[0035] In this embodiment, S3 is specifically: Divide the recycling plant into M working areas, and the g-th working area is equipped with N k robots. Then the robot group R g in the g-th working area is expressed as: ; Among them, represents the -th robot in the g-th working area; According to the multi-level classification model, output the classification result and accurate coordinates of each garbage:

[0036] Among them, is the category of the j-th garbage; is the three-dimensional coordinates of the j-th garbage in the working area; is the task priority (set according to the garbage category or the processing time limit); is the output of the multi-level classification model; N is the number of garbage; Use the distributed auction algorithm to solve the task allocation problem. The robot group R k in each working area independently executes the auction algorithm, and dynamically allocates tasks to the robots in its group. The goal is to maximize the benefit (such as the sum of task priorities) and minimize the cost (such as the total time or the moving distance): In the initial state, assume that all tasks and robots are in an unallocated state, and initialize the revenue to 0; Each task is bid by all idle robots, and the robot Submit a quotation : ; Among them, represents the cost of the robot executing the task ; Record the current ; task and allocate the task to the robot with the highest bid : ; The allocated task is removed from the task set; the robot with the allocated task updates to the target point and starts to execute; if there are unallocated tasks, repeat the above steps for the remaining task set until all tasks are allocated or all robots are busy. After the robot completes the task, it becomes available again and participates in the next task auction.

[0037] In this embodiment, the dynamic environment map is built as follows: Define a two-dimensional coordinate system with a fixed grid resolution, and the state of all position points is represented by the occupancy probability :

[0038] Among them, Q is all the grids in the current environment map; the occupancy probability of each grid is dynamically updated based on the lidar and vision sensors; N q is the total number of grids; Update the occupancy probability with each frame of vision and lidar observations: ; Among them, is the probability that the grid t is occupied under the current observation z ; is the sensor observation model likelihood; is the prior occupancy probability; is the normalization factor for all observations; Use logarithmic probability to simplify the update:

[0039] Combine the update of each frame of data to maintain the real-time changing environment map M t , and divide the map into: static obstacle areas (such as walls, fixed racks); dynamic obstacle areas (such as moving garbage, robots).

[0040] In this embodiment, the robot adjusts the grabbing route based on the task allocation result and the real-time state perception environment information, grabs the garbage and sorts it into the corresponding type of storage box, as follows: From the environment map M t Extract the radius around the robot as R p The local map of the robot; from the current position of the robot (x R ,y R ,z R ) Plan to the grab point (x) of the target garbage j ,y j ,z j ) path, the path needs to avoid obstacles and minimize the path cost; For each grid p on the path a , define the path cost as C(p a ): C(p a )=Ctravel(p a )+Cobstacle(p a )+Cdynamic(p a ); Among them, Ctravel(p a )=dist(p a-1 ,p a ) is the path length cost; Cobstacle(p a ) is the obstacle cost; Cobstacle(p a )=W obs As the occupation probability cost increases, W obs is the obstacle cost weight: Cdynamic(p a ) is the dynamic object cost, which gives an additional distance buffer according to the dynamic garbage or robot motion state perceived by the lidar and vision; The robot starts from the starting point (x R ,y R ) to the target point (x j ,y j ) has a path cost of 0 for each grid p a Path cost C(p a ) Based on the environment map M t , use DWA dynamic window algorithm to search for the optimal path; The path is dynamically updated at any time when the following events occur: the location of the sample garbage moves (such as garbage offset on the conveyor belt); other robots or dynamic obstacles enter the path; sensors perceive new map changes.

[0041] An intelligent garbage sorting system based on multi-robot collaborative work, including a processor, a memory, and a computer program stored on the memory. When the processor executes the computer program, it specifically executes the steps in the above-mentioned intelligent garbage sorting method based on multi-robot collaborative work.

[0042] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0043] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0044] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0045] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0046] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention in any other form. Any person skilled in the relevant art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical content of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A garbage intelligent sorting method based on multi-robot collaborative work, characterized in that, Including the following steps: S1: The garbage is conveyed to the camera detection area by a conveyor belt. The garbage image is collected through a vision sensor, and multi-sensor data is obtained through a lidar and an infrared spectrometer, and image features and multi-sensor features are extracted: S2: Combining the image features and multi-sensor features, based on a multi-level classification model, the garbage is classified and the precise coordinates are obtained; S3: Transmit the classification results and precise coordinates to the robot scheduling module, and the robot scheduling module uses the auction algorithm for task allocation; S4: Using the vision sensor combined with the lidar, a dynamic environment map is built, and each robot perceives the environment according to the real-time states of the conveyor belt and other robots; S5: Based on the task allocation results and the real-time state perception environment information, the robot adjusts the grasping route, grasps the garbage and sorts it into the storage bins of the corresponding types.

2. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 1, wherein, The specific content of S1 is as follows: The image data of the garbage is captured using a high-speed industrial RGB camera, and the image features F of the garbage are extracted from the image data through a convolutional neural network v ; The lidar is installed above the detection area to scan the surface of the garbage with multi-beam lasers to obtain the point cloud feature F l ; The infrared spectrometer collects the absorption or reflection spectrum of the infrared light on the surface of the garbage and generates a spectral curve; in the spectral curve, the absorption peak or reflection characteristics of each material are different. According to the Lambert-Beer law, the spectral feature Fs is extracted; Align the data collected by each sensor in the world coordinate system and perform timestamp synchronization processing.

3. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 2, wherein, The multi-level classification model includes a lightweight fast classification sub-model, a fine classification sub-model, and a precise localization sub-model, which are specifically as follows: The lightweight and fast classification sub-model is based on the image feature F v , and uses a lightweight algorithm to quickly identify the major categories of garbage and complete real-time coarse classification; and provides a preliminary bounding box for the location of the garbage as the input for further fine classification and coordinate optimization; The fine classification sub-model combines the point cloud feature F for the uncertain large categories l and the spectral feature F s and uses high-precision classification for further fine classification; The precise localization sub-model combines the rough classification area of the first layer with the fine classification result of the second layer, and determines the category and three-dimensional precise coordinates of the garbage through the three-dimensional shape and infrared spectral material information provided by the lidar point cloud data.

4. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 3, wherein Use the lightweight algorithm YOLO Tiny to quickly identify the large categories of garbage, specifically as follows: The extracted image features F v are input into the fully connected layer, combined with the classification weights W v and the bias b v , and the classification score is calculated: ； Among them, is the classification score for category c; Use the Softmax function to calculate the probability distribution of each category: ； Among them, P(c) is the confidence of class c; C is the total number of garbage categories; S k is the classification score of class k; Classification result , i.e., the most likely major class label; While classifying, YOLO Tiny will detect the location of the garbage and generate the corresponding bounding box, and predict the parameters of the detection box B: ； Among them, (x1, y1) is the upper left coordinate of the bounding box; (x2, y2) is the lower right coordinate of the bounding box; Adjust the bounding box using the regression parameters predicted by YOLO Tiny, and predict the center point position ( x c , y c ): ； Among them, (x0, y0) is the default position of the candidate box; (t x , t y ) is the translation offset output by the network; (w a , h a ) are the default width and height of the candidate box; is the activation function; Bounding box width and height ( w , h ) prediction: ； Among them, (t w , t h ) is the scaling ratio of the bounding box; Convert it to the complete bounding box form: ； Final output of the model: bounding boxes for each detected object and the corresponding class predictions and confidences.

5. A garbage intelligent sorting method based on multi-robot collaborative work according to claim 4, characterized in that, For the uncertain major categories, the fine-grained classification sub-model combines the point cloud feature F l and the spectral feature F s , and uses the high-precision model ResNet-RCNN to extract deep features for further fine classification as follows: The garbage area image cropped from the bounding box B output by the lightweight model uses ResNet-50 as the backbone network to extract deep image features as new image features : ； Among them, is the garbage area image cropped according to the bounding box B; and combine with the point cloud feature F l , the spectral feature F s to form a combined feature vector: ； Among them, α1, α2, and α3 are weight factors; Input the joint feature vector F into ResNet-RCNN: ； Among them, is the final category prediction of the garbage; W is the classifier weight matrix; b is the bias vector, and Softmax is the activation function.

6. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 5, characterized in that, The precise localization sub-model combines the rough classification area of the first layer with the fine classification result of the second layer, and determines the category and three-dimensional precise coordinates of the garbage through the three-dimensional shape and infrared spectral material information provided by the lidar point cloud data, specifically as follows: According to the center point (x c , y c ) corresponding to the lightweight model bounding box B, extract the target point region in the point cloud data, and obtain the relevant point cloud P region : ； For the point cloud data (x c , y c ) belonging to the neighborhood Δ(x i , y i , z i ), calculate the mean value of its three-dimensional coordinates to obtain the center point coordinates (X c , Y c , Z c ) of the target garbage object: ； The material information in the corresponding point cloud area is verified through spectral analysis. If the spectral material result is inconsistent with the fine classification, the current localization result is discarded; if they are consistent, the final garbage center is determined.

7. A garbage intelligent sorting method based on multi-robot collaborative work according to claim 1, characterized in that, The specific content of S3 is as follows: The recycling plant is divided into M working areas, and the g-th working area is equipped with N k robots. Then the robot group R g in the g-th working area is expressed as: ； Among them, represents the th robot in the gth working area; Output the classification results and precise coordinates of each garbage according to the multi-level classification model: ； Among them, is the category of the j-th piece of garbage; is the three-dimensional coordinate of the j-th piece of garbage in the working area; is the task priority; is the output of the multi-level classification model; N is the number of pieces of garbage; Use a distributed auction algorithm to solve the task allocation problem. The robot swarm R in each work area k independently executes the auction algorithm to dynamically allocate tasks to the robots within the group, with the goal of maximizing benefits and minimizing costs: In the initial state, assume that all tasks and robots are in the unallocated state at the beginning, and the initial revenue is 0; Each task is auctioned by all idle robots, and the robots submit bids : ； Among them, represents the robot executing tasks cost; The current record; Task Allocate to the robot with the highest bid : ； Assigned task Removed from the task set; the robot assigned the task updates to the target point and starts execution; if there are unassigned tasks, repeat the above steps for the remaining task set until all tasks are assigned or all robots are busy. After the robot completes the task, it becomes available again and participates in the next task auction.

8. A garbage intelligent sorting method based on multi-robot collaborative work according to claim 1, characterized in that, The construction of the dynamic environment map is specifically as follows: Define a two-dimensional coordinate system with a fixed grid resolution, and the state of all position points is represented by the occupancy probability as follows: ； where Q is all the grids in the current environmental map; each grid occupancy probability is dynamically updated based on lidar and vision sensors; N q is the total number of grids; Update the occupancy probability for each frame of vision and lidar observations: ； wherein, is the probability that the grid is occupied under the current observation z t ; is the likelihood of the sensor observation model; is the prior occupancy probability; ; is the normalization factor for all observations; Use logarithmic probability to simplify the update: ； Maintain the real-time changing environmental map M by combining the update of each frame of data t , and divide the map into: a static obstacle area; a dynamic obstacle area.

9. A method for intelligent garbage sorting based on multi-robot collaborative work according to claim 1, characterized in that, Based on the task assignment result and real-time status, the robot perceives the environmental information, adjusts the grasping route, grasps the garbage and sorts it into the storage bins of corresponding types, specifically as follows: From the environment map M t Extract the radius around the robot as R p The local map of the robot; from the current position of the robot (x R ,y R ,z R ) Plan to the grab point (x) of the target garbage j ,y j ,z j ) path, the path needs to avoid obstacles and minimize the path cost; For each grid p on the path a , define the path cost as C(p a ): C(p a ) = Ctravel(p a ) + Cobstacle(p a ) + Cdynamic(p a ); Among them, Ctravel(p a ) = dist(p a-1 , p a ) is the path length cost; Cobstacle(p a ) is the obstacle cost; Cobstacle(p a ) = W obs is the increase with the occupancy probability cost, and W obs is the obstacle cost weight: Cdynamic(p a ) is the dynamic object cost, which gives an additional distance buffer according to the dynamic garbage or robot motion state detected by lidar and vision perception. The robot moves from the starting point (x R , y R ) to the target point (x j , y j ), and the path cost is the sum of the path cost C(p a ) for each grid p a ); Based on the environmental map M t , use the DWA (Dynamic Window Approach) algorithm to search for the optimal path; Dynamically update the path at any time when the following events occur: the position of the sample garbage moves; other robots or dynamic obstacles enter the path; the sensor perceives new map changes.

10. An intelligent garbage sorting system based on multi-robot collaborative work, characterized in that, It includes a processor, a memory, and a computer program stored on the memory. When the processor executes the computer program, it specifically executes the steps in a garbage intelligent sorting method based on multi-robot collaborative work according to any one of claims 1-9.

Images