A method and system for intelligent garbage sorting based on multi-robot collaborative work

Abstract

The present invention relates to a multi-robot collaborative intelligent garbage sorting method and system, comprising the following steps: S1: garbage image acquisition using visual sensors, and multi-sensor data acquisition using laser radar and infrared spectrometers; S2: combining image features and multi-sensor features, based on a multi-level classification model, classifying garbage and obtaining precise coordinates; S3: transmitting the classification results and precise coordinates to a robot scheduling module, which uses an auction algorithm to assign tasks; S4: using visual sensors combined with laser radar to build a dynamic environmental map, each robot perceives the environment based on the real-time status of conveyor belts and other robots; S5: based on the task assignment results and real-time state perception of the environment, the robots adjust their grasping routes, grasp garbage, and sort it into corresponding storage bins. This invention can significantly improve the efficiency and accuracy of intelligent garbage sorting.

Description

Technical Field

[0001] The present invention relates to the field of robot collaborative control, and in particular to an intelligent garbage sorting method and system based on the collaborative work of multiple robots. Background Art

[0002] In the waste recycling and sorting industry, with accelerating urbanization and rising environmental protection requirements, waste sorting has become a global challenge. However, in the current system, most waste sorting operations are still largely manual. Although automated systems have been introduced in some areas, traditional automated sorting systems primarily utilize fixed robotic arms combined with conveyor belts, equipped with basic vision systems to identify and grasp waste. The shortcomings of these systems in large-scale waste sorting scenarios are as follows: Reliance on a single sensor: Many sorting systems rely solely on visual detection for waste identification, making it difficult to accurately sort waste in situations with fluctuating lighting, significant occlusion, or similar waste materials (such as mixed plastic and glass). Inadequate sorting accuracy: The lack of high-precision sensors such as spectral analysis makes it difficult to accurately classify industrial waste or complex mixed waste. Lack of dynamic adjustment capabilities: Traditional systems can generally only handle static or slightly dynamic environments (such as waste on a stable conveyor belt), resulting in a high sorting failure rate when faced with positional fluctuations or dynamic waste transport. Summary of the Invention

[0003] In order to solve the above problems, the purpose of the present invention is to provide a garbage intelligent sorting method and system based on the collaborative work of multiple robots, which can significantly improve the efficiency and accuracy of garbage intelligent sorting.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for intelligent garbage sorting based on multi-robot collaborative work, comprising the following steps:

[0006] S1: Garbage is transported to the camera detection area via a conveyor belt. The visual sensor collects garbage images, and the laser radar and infrared spectrometer obtain multi-sensor data, and extracts image features and multi-sensor features:

[0007] S2: Combining image features and multi-sensor features, based on a multi-level classification model, classify the garbage and obtain the precise coordinates;

[0008] S3: The classification results and precise coordinates are transmitted to the robot scheduling module, which uses an auction algorithm to allocate tasks;

[0009] S4: Using visual sensors combined with LiDAR to build a dynamic environment map, each robot perceives the environment based on the real-time status of the conveyor belt and other robots;

[0010] S5: Based on the task assignment results and real-time state perception of environmental information, the robot adjusts the grasping route, grabs the garbage and sorts it into the corresponding type of storage box.

[0011] Furthermore, S1 is specifically:

[0012] A high-speed industrial RGB camera is used to capture image data of garbage, and the image data is passed through a convolutional neural network (CNN) to extract the image features F of the garbage. v ;

[0013] The laser radar is installed above the detection area and scans the garbage surface with multi-beam laser to obtain point cloud features F l ;

[0014] Infrared spectrometer collects the absorption or reflection spectrum of infrared light on the garbage surface and generates a spectrum curve. In the spectrum curve, each material has different absorption peaks or reflection characteristics. According to the Lambert-Beer law, the spectrum feature F is extracted. s ;

[0015] The data collected by each sensor is aligned with the world coordinate system and time stamp synchronization is performed.

[0016] Furthermore, the multi-level classification model includes a lightweight fast classification sub-model, a fine classification sub-model, and a precise positioning sub-model, as follows:

[0017] The lightweight fast classification sub-model is based on the image feature F v , using lightweight algorithms to quickly identify large categories of garbage and complete real-time coarse classification; and provide preliminary bounding boxes of the garbage area location as input for further fine classification and coordinate optimization;

[0018] The fine classification sub-model is used to combine the point cloud features F for the uncertain large categories. l and spectral characteristics F s , use high-precision classification for further sub-classification;

[0019] The precise positioning sub-model combines the coarse classification area of ​​the first layer with the fine classification results of the second layer, and determines the category and precise three-dimensional coordinates of the garbage through the three-dimensional shape and infrared spectrum material information provided by the lidar point cloud data.

[0020] Furthermore, we use the lightweight algorithm YOLO Tiny to quickly identify the major categories of garbage, as follows:

[0021] The extracted image features F v Input the fully connected layer, combined with the classification weight W v and bias b v , calculate the classification score:

[0022]

[0023] in, is the classification score of category c;

[0024] Use the Softmax function to calculate the probability distribution of each category:

[0025]

[0026] Where P(c) is the confidence level of category c; C is the total number of garbage categories; S k is the classification score of category k;

[0027] Classification results , that is, the most likely category label (such as metal, plastic, etc.).

[0028] While classifying, YOLO Tiny detects the location of the garbage, generates a corresponding bounding box, and predicts the parameters of the detection box B:

[0029] ;

[0030] Among them, (x1, y1) is the coordinate of the upper left corner of the bounding box; (x2, y2) is the coordinate of the lower right corner of the bounding box;

[0031] Use the regression parameters predicted by YOLO Tiny to adjust the bounding box and the center point position ( x c , y c )predict:

[0032] ;

[0033] Among them, (x0, y0) is the default position of the candidate box; (t x , t y ) is the translation offset of the network output; (w a , h a ) are the default width and height of the candidate box; is the activation function;

[0034] Bounding box width and height ( w , h )predict:

[0035]

[0036] Among them, (t w , t h ) is the scaling factor of the bounding box.

[0037] Convert to full bounding box form:

[0038] ;

[0039] The final output of the model: the bounding box of each detected target , corresponding category predictions and confidence levels.

[0040] Furthermore, the fine classification sub-model combines the point cloud features F for the uncertain large categories l and spectral characteristics F s , use the high-precision model ResNet-RCNN to extract deep features and further refine the classification, as follows:

[0041] The bounding box B output from the lightweight model is used to crop the garbage area image and use ResNet-50 as the backbone network to extract deep image features as new image features. :

[0042] ;

[0043] in, is the garbage area image cropped according to the bounding box B;

[0044] And with the point cloud feature F l , spectral characteristics F s Combine to form a joint eigenvector:

[0045] ;

[0046] Among them, α1, α2, and α3 are weight factors;

[0047] Input the joint feature vector F into ResNet-RCNN:

[0048] ;

[0049] in, is the final category prediction of garbage; W is the classifier weight matrix; b is the bias vector, and Softmax is the activation function.

[0050] Furthermore, the precise positioning sub-model combines the coarse classification areas of the first layer with the fine classification results of the second layer, and uses the three-dimensional shape and infrared spectral material information provided by the lidar point cloud data to determine the category and precise three-dimensional coordinates of the garbage, as follows:

[0051] According to the center point (x c ,y c ), extract the target point area in the point cloud data, and obtain the relevant point cloud P through the neighborhood detection algorithm region:

[0052] ;

[0053] For the neighborhood Δ(x c ,y c ) point cloud data (x i ,y i ,z i ) calculates the mean of its three-dimensional coordinates and obtains the center coordinates of the target garbage object (X c ,Y c ,Z c ):

[0054] ;

[0055] The material information in the corresponding point cloud area is verified through spectral analysis. If the spectral material result is inconsistent with the fine classification, the current positioning result is discarded; if they are consistent, the final garbage center is determined.

[0056] Furthermore, S3 is specifically:

[0057] The recycling plant is divided into M operating areas, and the g-th operating area is equipped with N k robots, then the robot group R in the g-th work area g Expressed as:

[0058] ;

[0059] in, Indicates the g-th working area A robot;

[0060] Output the classification results and precise coordinates of each garbage item based on the multi-level classification model:

[0061]

[0062] in, is the category of the j-th garbage; is the three-dimensional coordinate of the jth garbage in the working area; Task priority (set according to garbage category or processing time limit); is the output of the multi-class classification model; N is the number of garbage;

[0063] A distributed auction algorithm is used to solve the task allocation problem. The robot group R in each work area k Independently execute the auction algorithm to dynamically assign tasks to robots within its group, with the goal of maximizing benefits and minimizing costs:

[0064] Initial state: all tasks and robots are initially unassigned, and the initial payoff is 0;

[0065] Each task Auctioned by all idle robots, robots Submit a Quote :

[0066] ;

[0067] in, Represents a robot Execute the task cost;

[0068] The current Record;

[0069] Task Assigned to the bot with the highest bid :

[0070] ;

[0071] Assigned tasks Remove from the task set; the robot assigned the task updates to the target point and starts execution; if there are unassigned tasks, repeat the above steps for the remaining task sets until all tasks are assigned or all robots are busy. After the robot completes the task, it becomes available again and participates in the next task auction.

[0072] Furthermore, the dynamic environment map is built as follows:

[0073] Define a two-dimensional coordinate system with a fixed grid resolution, and the status of all position points is expressed by occupancy probability express:

[0074]

[0075] Among them, Q is all the grids in the current environment map; each grid The occupancy probability is dynamically updated based on the lidar and vision sensors; N q is the total number of grids;

[0076] The updated occupancy probability of vision and lidar observations for each frame is:

[0077] ;

[0078] in, is the current observation z t Lower grid Probability of being occupied; is the sensor observation model likelihood; is the prior occupancy probability; is the normalization factor for all observations;

[0079] Use log-probability to simplify the update:

[0080]

[0081] Combined with each frame data update, maintain real-time changing environment map M t , the map is divided into: static obstacle area; dynamic obstacle area.

[0082] Furthermore, the robot adjusts its grasping route based on the task assignment results and real-time state perception of the environment, grabs the garbage, and sorts it into the corresponding storage bins, as follows:

[0083] From the environment map M t Extract the radius R around the robot p The local map of the robot; from the current position of the robot (x R ,y R ,z R ) Plan to grab the target garbage point (x j ,y j ,z j ) path, the path needs to avoid obstacles and minimize the path cost;

[0084] For each grid p on the path a , define the path cost as C(p a ):

[0085] C(p a )=Ctravel(p a )+Cobstacle(p a )+Cdynamic(p a );

[0086] Among them, Ctravel(p a )=dist(p a-1 ,p a ) is the path length cost; Cobstacle(p a ) is the obstacle cost; Cobstacle(p a )=W obs As the occupation probability cost increases, W obs is the obstacle cost weight: Cdynamic(p a ) is the dynamic object cost, which provides an additional distance buffer based on the dynamic garbage or robot motion state perceived by the lidar and visual perception;

[0087] The robot starts from the starting point (x R ,y R ) to the target point (x j ,y j ) path cost for each grid p a Path cost C(p a )

[0088] Based on the environment map M t , use DWA dynamic window algorithm to search for the optimal path;

[0089] The path is dynamically updated at any time when the following events occur: the sample garbage location moves; other robots or dynamic obstacles enter the path; sensors perceive new map changes.

[0090] A garbage intelligent sorting system based on the collaborative work of multiple robots includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps of the above-mentioned garbage intelligent sorting method based on the collaborative work of multiple robots.

[0091] The present invention has the following beneficial effects:

[0092] 1. This invention combines multimodal data from visual sensors, lidar, and infrared spectrometers to obtain the visual, three-dimensional, and material characteristics of garbage, providing more complete garbage feature information and significantly improving classification accuracy.

[0093] 2. The present invention adopts a strategy that combines lightweight fast classification with fine classification to quickly complete preliminary classification, improve the real-time performance of the system, and finely classify uncertain samples to ensure classification accuracy, significantly improving the classification accuracy of complex garbage;

[0094] 3. The present invention adopts an auction algorithm for dynamic task allocation to achieve the optimal configuration of robot resources and improve the overall operating efficiency of the system; and builds a dynamic environment map based on visual sensors and lidar to perceive the conveyor belt status and the position of other robots in real time, ensuring the stable operation of the system in a dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0096] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0097] refer to Figure 1In this embodiment, a method for intelligent garbage sorting based on multi-robot collaboration is provided, comprising the following steps:

[0098] S1: Garbage is transported to the camera detection area via a conveyor belt. The visual sensor collects garbage images, and the laser radar and infrared spectrometer obtain multi-sensor data, and extracts image features and multi-sensor features:

[0099] S2: Combining image features and multi-sensor features, based on a multi-level classification model, classify the garbage and obtain the precise coordinates;

[0100] S3: The classification results and precise coordinates are transmitted to the robot scheduling module, which uses an auction algorithm to allocate tasks;

[0101] S4: Using visual sensors combined with LiDAR to build a dynamic environment map, each robot perceives the environment based on the real-time status of the conveyor belt and other robots;

[0102] S5: Based on the task assignment results and real-time state perception of environmental information, the robot adjusts the grasping route, grabs the garbage and sorts it into the corresponding type of storage box.

[0103] In this embodiment, S1 is specifically:

[0104] A high-speed industrial RGB camera is used to capture image data of garbage, and the image data is passed through a convolutional neural network (CNN) to extract the image features F of the garbage. v (texture, color, shape, etc.);

[0105] The LiDAR is installed above the detection area and scans the garbage surface with multi-beam lasers to obtain point cloud features F l ;

[0106] Infrared spectrometer (IR) collects the absorption or reflection spectrum of infrared light on the garbage surface and generates a spectrum curve. In the spectrum curve, each material (such as metal, plastic, organic matter) has different absorption peaks or reflection characteristics. According to the Lambert-Beer law, the spectrum feature F is extracted. s ;

[0107] The data collected by each sensor is aligned with the world coordinate system and time stamp synchronization is performed.

[0108] In this embodiment, the multi-level classification model includes a lightweight fast classification sub-model, a fine classification sub-model, and a precise positioning sub-model, as follows:

[0109] The lightweight fast classification sub-model is based on the image feature F v, using lightweight algorithms to quickly identify large categories of garbage (such as metal, plastic, paper, organic matter, glass, etc.), completing real-time coarse classification; and providing preliminary bounding boxes of the garbage area location as input for further fine classification and coordinate optimization;

[0110] The fine classification sub-model is used to combine the point cloud features F for the uncertain large categories. l and spectral characteristics F s , using high-precision classification for further sub-classification (such as distinguishing different plastic components, stainless steel and iron, etc.);

[0111] The precise positioning sub-model combines the coarse classification area of ​​the first layer with the fine classification results of the second layer, and determines the category and precise three-dimensional coordinates of the garbage through the three-dimensional shape and infrared spectrum material information provided by the lidar point cloud data.

[0112] In this embodiment, the lightweight algorithm YOLO Tiny is used to quickly identify the major categories of garbage, as follows:

[0113] The extracted image features F v Input the fully connected layer, combined with the classification weight W v and bias b v , calculate the classification score:

[0114]

[0115] in, is the classification score of category c;

[0116] Use the Softmax function to calculate the probability distribution of each category:

[0117]

[0118] Where P(c) is the confidence (probability) of category c (e.g., metal, plastic); C is the total number of waste categories (e.g., metal = 1, plastic = 2, organic matter = 3, ...); S k is the classification score of category k;

[0119] Classification results , that is, the most likely category label (such as metal, plastic, etc.).

[0120] While classifying, YOLO Tiny detects the location of the garbage, generates a corresponding bounding box, and predicts the parameters of the detection box B:

[0121] ;

[0122] Among them, (x1, y1) is the coordinate of the upper left corner of the bounding box; (x2, y2) is the coordinate of the lower right corner of the bounding box;

[0123] Use the regression parameters predicted by YOLO Tiny to adjust the bounding box and the center point position ( x c , y c )predict:

[0124] ;

[0125] Among them, (x0, y0) is the default position of the candidate box; (t x , t y ) is the translation offset of the network output; (w a , h a ) are the default width and height of the candidate box; is the activation function;

[0126] Bounding box width and height ( w , h )predict:

[0127]

[0128] Among them, (t w , t h ) is the scaling factor of the bounding box.

[0129] Convert to full bounding box form:

[0130] ;

[0131] The final output of the model: the bounding box of each detected target , corresponding category predictions and confidence levels.

[0132] In this embodiment, the fine classification sub-model is used to combine the point cloud features F l and spectral characteristics F s , use the high-precision model ResNet-RCNN to extract deep features and further refine the classification, as follows:

[0133] The bounding box B output from the lightweight model is used to crop the garbage area image and use ResNet-50 as the backbone network to extract deep image features as new image features. :

[0134] ;

[0135] in, is the garbage area image cropped according to the bounding box B;

[0136] And with the point cloud feature F l , spectral characteristics F s Combine to form a joint eigenvector:

[0137] ;

[0138] Among them, α1, α2, and α3 are weight factors;

[0139] Input the joint feature vector F into ResNet-RCNN:

[0140] ;

[0141] in, is the final category prediction of garbage; W is the classifier weight matrix; b is the bias vector, and Softmax is the activation function.

[0142] In this embodiment, the precise positioning sub-model combines the coarse classification areas of the first layer with the fine classification results of the second layer, and uses the three-dimensional shape and infrared spectral material information provided by the lidar point cloud data to determine the category and precise three-dimensional coordinates of the garbage. The details are as follows:

[0143] According to the center point (x c ,y c ), extract the target point area in the point cloud data, and obtain the relevant point cloud P through the neighborhood detection algorithm region :

[0144] ;

[0145] For the neighborhood Δ(x c ,y c ) point cloud data (x i ,y i ,z i ) calculates the mean of its three-dimensional coordinates and obtains the center coordinates of the target garbage object (X c ,Y c ,Z c ):

[0146] ;

[0147] The material information in the corresponding point cloud area is verified through spectral analysis. If the spectral material result is inconsistent with the fine classification (for example, the type should be metal, but the spectrum matches plastic), the current positioning result is discarded; if they are consistent, the final garbage center is determined.

[0148] In this embodiment, S3 is specifically:

[0149] The recycling plant is divided into M operating areas, and the g-th operating area is equipped with N k robots, then the robot group R in the g-th work area g Expressed as:

[0150] ;

[0151] in, Indicates the g-th working area A robot;

[0152] Output the classification results and precise coordinates of each garbage item based on the multi-level classification model:

[0153]

[0154] in, is the category of the j-th garbage; is the three-dimensional coordinate of the jth garbage in the working area; Task priority (set according to garbage category or processing time limit); is the output of the multi-class classification model; N is the number of garbage;

[0155] A distributed auction algorithm is used to solve the task allocation problem. The robot group R in each work area k Independently execute the auction algorithm to dynamically assign tasks to robots within its group, with the goal of maximizing benefits (such as the sum of task priorities) and minimizing costs (such as total time or travel distance):

[0156] Initial state: all tasks and robots are initially unassigned, and the initial payoff is 0;

[0157] Each task Bid by all idle robots, robots Submit a Quote :

[0158] ;

[0159] in, Represents a robot Execute the task cost;

[0160] The current Record;

[0161] Task Assigned to the bot with the highest bid :

[0162] ;

[0163] Assigned tasks Remove from the task set; the robot assigned the task updates to the target point and starts execution; if there are unassigned tasks, repeat the above steps for the remaining task sets until all tasks are assigned or all robots are busy. After the robot completes the task, it becomes available again and participates in the next task auction.

[0164] In this embodiment, the dynamic environment map is constructed as follows:

[0165] Define a two-dimensional coordinate system with a fixed grid resolution, and the status of all position points is expressed by occupancy probability express:

[0166]

[0167] Among them, Q is all the grids in the current environment map; each grid The occupancy probability is dynamically updated based on the lidar and vision sensors; N q is the total number of grids;

[0168] The updated occupancy probability of vision and lidar observations for each frame is:

[0169] ;

[0170] in, is the current observation z t lower grid Probability of being occupied; is the sensor observation model likelihood; is the prior occupancy probability; is the normalization factor for all observations;

[0171] Use log-probability to simplify the update:

[0172]

[0173] Combined with each frame data update, maintain real-time changing environment map M t , the map is divided into: static obstacle areas (such as walls, fixed frames); dynamic obstacle areas (such as moving garbage, robots).

[0174] In this embodiment, the robot adjusts its grasping route based on the task assignment results and real-time state perception of the environment, grabs the garbage, and sorts it into the corresponding type of storage bin, as follows:

[0175] From the environment map M t Extract the radius R around the robot p The local map of the robot; from the current position of the robot (xR ,y R ,z R ) Plan to grab the target garbage point (x j ,y j ,z j ) path, the path needs to avoid obstacles and minimize the path cost;

[0176] For each grid p on the path a , define the path cost as C(p a ):

[0177] C(p a )=Ctravel(p a )+Cobstacle(p a )+Cdynamic(p a );

[0178] Among them, Ctravel(p a )=dist(p a-1 ,p a ) is the path length cost; Cobstacle(p a ) is the obstacle cost; Cobstacle(p a )=W obs As the occupation probability cost increases, W obs is the obstacle cost weight: Cdynamic(p a ) is the dynamic object cost, which provides an additional distance buffer based on the dynamic garbage or robot motion state perceived by the lidar and visual perception;

[0179] The robot starts from the starting point (x R ,y R ) to the target point (x j ,y j ) path cost for each grid p a Path cost C(p a )

[0180] Based on the environment map M t , use DWA dynamic window algorithm to search for the optimal path;

[0181] The path is dynamically updated at any time when the following events occur: the position of the sample garbage moves (such as garbage offset on the conveyor belt); other robots or dynamic obstacles enter the path; sensors perceive new map changes.

[0182] A garbage intelligent sorting system based on the collaborative work of multiple robots includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps of the above-mentioned garbage intelligent sorting method based on the collaborative work of multiple robots.

[0183] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0187] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A method for intelligent garbage sorting based on multi-robot collaborative work, characterized in that: The following steps are involved: S1: Garbage is transported to the camera detection area via a conveyor belt. The visual sensor collects garbage images, and the laser radar and infrared spectrometer obtain multi-sensor data, and extracts image features and multi-sensor features: S2: Combining image features and multi-sensor features, based on a multi-level classification model, classify the garbage and obtain the precise coordinates; S3: The classification results and precise coordinates are transmitted to the robot scheduling module, which uses an auction algorithm to allocate tasks; S4: Using visual sensors combined with LiDAR to build a dynamic environment map, each robot perceives the environment based on the real-time status of the conveyor belt and other robots; S5: Based on the task assignment results and real-time state perception of the environment, the robot adjusts the grasping route, grasps the garbage and sorts it into the corresponding type of storage box; The S1 is specifically: A high-speed industrial RGB camera is used to capture image data of garbage, and the image data is passed through a convolutional neural network to extract the image features F of the garbage. v ; The laser radar is installed above the detection area and scans the garbage surface with multi-beam laser to obtain point cloud features F l ; Infrared spectrometer collects the absorption or reflection spectrum of infrared light from the garbage surface and generates a spectral curve. In the spectral curve, each material has different absorption peaks or reflection characteristics. According to the Lambert-Beer law, the spectral feature Fs is extracted. The data collected by each sensor is aligned with the world coordinate system and time stamped and synchronized; The multi-level classification model includes a lightweight fast classification sub-model, a fine classification sub-model and a precise positioning sub-model, as follows: The lightweight fast classification sub-model is based on the image feature F v , using lightweight algorithms to quickly identify large categories of garbage and complete real-time coarse classification; and provide preliminary bounding boxes of the garbage area location as input for further fine classification and coordinate optimization; The fine classification sub-model is used to combine the point cloud features F for the uncertain large categories. l and spectral characteristics F s , use high-precision classification for further sub-classification; The precise positioning sub-model combines the coarse classification area of ​​the first layer with the fine classification results of the second layer, and determines the category and precise three-dimensional coordinates of the garbage through the three-dimensional shape and infrared spectrum material information provided by the lidar point cloud data.

2. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 1 is characterized in that: Use the lightweight algorithm YOLO Tiny to quickly identify large categories of garbage, as follows: The extracted image features F v Input the fully connected layer, combined with the classification weight W v and bias b v , calculate the classification score: ； in, is the classification score of category c; Use the Softmax function to calculate the probability distribution of each category: ； Where P(c) is the confidence level of category c; C is the total number of garbage categories; S k is the classification score of category k; Classification results , that is, the most likely category label; While classifying, YOLO Tiny detects the location of the garbage, generates a corresponding bounding box, and predicts the parameters of the detection box B: ； Among them, (x1, y1) is the coordinate of the upper left corner of the bounding box; (x2, y2) is the coordinate of the lower right corner of the bounding box; Use the regression parameters predicted by YOLO Tiny to adjust the bounding box and the center point position ( x c , y c )predict: ； Among them, (x0, y0) is the default position of the candidate box; (t x , t y ) is the translation offset of the network output; (w a , h a ) are the default width and height of the candidate box; is the activation function; Bounding box width and height ( w , h )predict: ； Among them, (t w , t h ) is the scaling factor of the bounding box; Convert to full bounding box form: ； The final output of the model: the bounding box of each detected target , corresponding category predictions and confidence levels.

3. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 2 is characterized in that: The fine classification sub-model is used to combine the point cloud features F for the uncertain large categories. l and spectral characteristics F s , use the high-precision model ResNet-RCNN to extract deep features and further refine the classification, as follows: The bounding box B output from the lightweight model is used to crop the garbage area image and use ResNet-50 as the backbone network to extract deep image features as new image features. : ； in, is the garbage area image cropped according to the bounding box B; And with the point cloud feature F l , spectral characteristics F s Combine to form a joint eigenvector: ； Among them, α1, α2, and α3 are weight factors; Input the joint feature vector F into ResNet-RCNN: ； in, is the final category prediction of garbage; W is the classifier weight matrix; b is the bias vector, and Softmax is the activation function.

4. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 3 is characterized in that: The precise positioning sub-model combines the coarse classification areas of the first layer with the fine classification results of the second layer, and uses the three-dimensional shape and infrared spectral material information provided by the lidar point cloud data to determine the category and precise three-dimensional coordinates of the garbage. The details are as follows: According to the center point (x c ,y c ), extract the target point area in the point cloud data, and obtain the relevant point cloud P through the neighborhood detection algorithm region : ； For the neighborhood Δ(x c ,y c ) point cloud data (x i ,y i ,z i ) calculates the mean of its three-dimensional coordinates and obtains the center coordinates of the target garbage object (X c ,Y c ,Z c ): ； The material information in the corresponding point cloud area is verified through spectral analysis. If the spectral material result is inconsistent with the fine classification, the current positioning result is discarded; if they are consistent, the final garbage center is determined.

5. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 1 is characterized in that: The S3 is specifically: The recycling plant is divided into M operating areas, and the g-th operating area is equipped with N k robots, then the robot group R in the g-th work area g Expressed as: ； in, Indicates the g-th working area A robot; Output the classification results and precise coordinates of each garbage item based on the multi-level classification model: ； in, is the category of the j-th garbage; is the three-dimensional coordinate of the jth garbage in the working area; For task priority; is the output of the multi-class classification model; N is the number of garbage; A distributed auction algorithm is used to solve the task allocation problem. The robot group R in each work area k Independently execute the auction algorithm to dynamically assign tasks to robots within the group, with the goal of maximizing benefits and minimizing costs: Initial state: all tasks and robots are initially unassigned, and the initial payoff is 0; Each task Auctioned by all idle robots, robots Submit a Quote : ； in, Represents a robot Execute the task cost; The current Record; Task Assigned to the bot with the highest bid : ； Assigned tasks Remove from the task set; the robot assigned the task updates to the target point and starts execution; if there are unassigned tasks, repeat the above steps for the remaining task sets until all tasks are assigned or all robots are busy. After the robot completes the task, it becomes available again and participates in the next task auction.

6. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 1 is characterized in that: Dynamic environment map construction, as follows: Define a two-dimensional coordinate system with a fixed grid resolution, and the status of all position points is expressed by occupancy probability express: ； Among them, Q is all the grids in the current environment map; each grid The occupancy probability is dynamically updated based on the lidar and vision sensors; N q is the total number of grids; The updated occupancy probability of vision and lidar observations for each frame is: ； in, is the current observation z t lower grid Probability of being occupied; is the sensor observation model likelihood; is the prior occupancy probability; is the normalization factor for all observations; Use log-probability to simplify the update: ； Combined with each frame data update, maintain real-time changing environment map M t , the map is divided into: static obstacle area; dynamic obstacle area.

7. The intelligent garbage sorting method based on multi-robot collaborative work according to claim 1 is characterized in that: The robot adjusts its grabbing route based on the task assignment results and real-time state perception of the environment, grabs the garbage and sorts it into storage bins of the corresponding type, as follows: From the environment map M t Extract the radius R around the robot p The local map of the robot; from the current position of the robot (x R ,y R ,z R ) Plan to grab the target garbage point (x j ,y j ,z j ) path, the path needs to avoid obstacles and minimize the path cost; For each grid p on the path a , define the path cost as C(p a ): C(p a )=Ctravel(p a )+Cobstacle(p a )+Cdynamic(p a ); Among them, Ctravel(p a )=dist(p a-1 ,p a ) is the path length cost; Cobstacle(p a ) is the obstacle cost; Cobstacle(p a )=W obs As the occupation probability cost increases, W obs is the obstacle cost weight: Cdynamic(p a ) is the dynamic object cost, which provides an additional distance buffer based on the dynamic garbage or robot motion state perceived by the lidar and visual perception; The robot starts from the starting point (x R ,y R ) to the target point (x j ,y j ) path cost for each grid p a Path cost C(p a ) Based on the environment map M t , use DWA dynamic window algorithm to search for the optimal path; The path is dynamically updated at any time when the following events occur: the sample garbage location moves; other robots or dynamic obstacles enter the path; sensors perceive new map changes.

8. An intelligent garbage sorting system based on multi-robot collaborative work, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory. When the processor executes the computer program, the method specifically performs the steps of the intelligent garbage sorting method based on multi-robot collaborative work as described in any one of claims 1 to 7.

Images