Solid waste automatic sorting method and device using machine vision

Abstract

The present invention relates to the field of image recognition, and in particular to a method and device for automatic sorting of solid waste using machine vision. The method comprises the following steps: acquiring monitoring images of a plurality of orientations of the area to be sorted; performing spatial geometric perspective deviation correction on the monitoring images of the plurality of orientations of the area to be sorted, and performing spatial panoramic image optimization to construct a panoramic image of the sorting area; performing frame-by-frame solid waste visual positioning scanning on the panoramic image of the sorting area, and performing three-dimensional morphological structure analysis to generate a three-dimensional morphological structure representation of each solid waste; making a solid type classification decision based on the three-dimensional morphological structure representation of each solid waste, and performing bounding box coding mapping to obtain a coding mapping bounding box of each solid waste; performing device action sequence simulation on the coding mapping bounding box of each solid waste, and constructing a fine-tuning action sequence. The present invention realizes efficient solid waste identification and improves sorting efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a method and device for automatically sorting solid waste using machine vision. Background Art

[0002] With the increasing severity of global environmental problems, the effective management and recycling of solid waste has become an important issue for the sustainable development of societies in all countries. Especially in the context of accelerated industrialization and urbanization, the amount of solid waste generated is increasing. How to efficiently and accurately classify and treat solid waste has become an important means to achieve resource recovery and reduce environmental pollution. Traditional solid waste sorting methods usually rely on manual operations. Although preliminary sorting work can be completed in some cases, there are problems such as high labor intensity, low sorting efficiency, and easy errors. Especially in large-scale production or processing scenarios, manual sorting cannot meet the needs of efficient, accurate and continuous operations.

[0003] With the continuous development of machine vision technology, intelligent and automated solid waste sorting systems have gradually become a hot topic of research and application. Machine vision technology can simulate the human eye's ability to recognize objects. Through image acquisition, processing and analysis, it can identify the type, shape, color and other characteristic information of solid waste, thereby realizing automatic classification of waste. Compared with traditional manual sorting methods, the solid waste sorting system based on machine vision has the advantages of high efficiency, accuracy and automation. It can maintain a high level of work efficiency in a high-speed and continuous production environment, reduce labor costs and reduce error rates. However, solid waste is of various types and shapes, and the surface of the waste is often contaminated, damaged or obstructed. This makes waste sorting based on machine vision face many challenges. There is often inaccurate object recognition, resulting in low solid waste sorting efficiency.

[0004] Therefore, researching an efficient, intelligent and accurate method for automatic sorting of solid waste based on machine vision has become an important technical need in the current fields of environmental protection and resource recycling. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention proposes a method and device for automatic sorting of solid waste using machine vision to solve at least one of the above technical problems.

[0006] To achieve the above object, the present invention provides a method for automatically sorting solid waste using machine vision, comprising the following steps:

[0007] Step S1: Acquire monitoring images of the area to be sorted in multiple directions; perform spatial geometric perspective deviation correction on the monitoring images of the area to be sorted in multiple directions, and perform spatial panoramic image optimization to construct a panoramic image of the sorting area;

[0008] Step S2: Performing a frame-by-frame visual positioning scan of the solid waste on the panoramic image of the sorting area and performing a three-dimensional morphological structure analysis to generate a three-dimensional morphological structure representation of each solid waste;

[0009] Step S3: performing solid type classification decision based on the three-dimensional morphological structure representation of each solid waste, and performing bounding box coding mapping to obtain a coding mapping bounding box for each solid waste;

[0010] Step S4: simulate the equipment action sequence for the coded mapping bounding box of each solid waste, and then fine-tune the action parameters to construct a fine-tuned action sequence;

[0011] Step S5: Performing instant automatic sorting based on the fine-tuning action sequence and collecting sorting images at multiple time points; performing temporal sorting efficiency trend evolution on the sorting images at multiple time points to generate sorting efficiency trend evolution features;

[0012] Step S6: Dynamic sorting path planning optimization and adaptive reinforcement transfer learning are performed on the sorting efficiency trend evolution characteristics to build an adaptive intelligent sorting model to perform automatic solid waste sorting operations.

[0013] The present invention improves image quality by correcting spatial geometric viewing angle deviation and optimizing panoramic images, making subsequent processing easier.

[0014] Constructing a panoramic image of the sorting area helps provide an overall perspective, laying the foundation for subsequent solid waste identification and sorting. Through frame-by-frame visual positioning scanning of solid waste and three-dimensional morphological structure analysis, the morphological characteristics of each solid waste are accurately obtained, and the generation of three-dimensional morphological structure representation helps in subsequent classification and sorting decisions, improving the accuracy and efficiency of automatic sorting. Solid type classification based on three-dimensional morphological structure representation accurately identifies different types of solid waste. Bounding box coding mapping provides an accurate mark for the location of each solid waste, providing a basis for subsequent action simulation and sorting. Equipment action sequence simulation and fine-tuning optimize the sorting process The actions and parameters in the sorting process can improve the accuracy of automatic sorting. Building a fine-tuning action sequence can help optimize the operating procedures of robots or equipment, improve sorting efficiency and accuracy, and analyze the temporal evolution characteristics of sorting images at multiple time points to help evaluate the changing trend of sorting efficiency, timely discover problems and improvement directions, and generate sorting efficiency trend evolution characteristics to help optimize sorting strategies and improve overall operational efficiency. Dynamic sorting path planning optimization and adaptive reinforcement transfer learning can improve the intelligence level and adaptability of the automatic sorting system. Building an adaptive intelligent sorting model can help achieve efficient automatic sorting of solid waste, improve production efficiency and reduce labor costs.

[0015] Preferably, step S1 includes the following steps:

[0016] Step S11: Acquire monitoring images of the area to be sorted in multiple directions based on a high-resolution panoramic camera;

[0017] Step S12: Optimizing the image contrast of monitoring images of the areas to be sorted in multiple directions to construct a multi-directional contrast-balanced monitoring image;

[0018] Step S13: performing image feature point recognition on the multi-directional contrast balance monitoring image and marking multiple image feature points;

[0019] Step S14: performing multi-directional registration on multiple image feature points to obtain multi-angle matching data for each feature point;

[0020] Step S15: performing spatial geometric perspective deviation correction based on the multi-angle matching data of each feature point, thereby obtaining a spatial geometric perspective deviation corrected image;

[0021] Step S16: performing spatial panoramic image optimization on the spatial geometric perspective deviation-corrected image to construct a panoramic image of the sorting area.

[0022] The present invention improves the clarity and detail capture of images by using a high-resolution panoramic camera to acquire images, which helps to accurately identify solid waste. The monitoring images in multiple directions provide comprehensive coverage, which helps to fully understand the situation in the sorting area. The image contrast is optimized to enhance the details and contrast in the image, which helps to improve the accuracy of solid waste recognition. The multi-directional contrast balanced monitoring images help to unify the visual effects of images in different directions and provide consistent input for subsequent processing. The identification and marking of image feature points helps to determine the key positions and features in the image and provide basic data for subsequent matching and correction. The marking of image feature points helps the algorithm to better understand the image content and improve the accuracy and efficiency of subsequent processing. Precisely matching feature points and obtaining matching data of feature points at different angles helps to accurately correct the spatial geometric perspective. Multi-angle matching data provides rich information and more accurately restores the position and form of objects in space. Through spatial geometric perspective deviation correction, the perspective deformation and distortion in the image are eliminated, and the accuracy and visualization of the image are improved. The corrected image is helpful for subsequent processing and analysis, providing a more accurate data basis for automatic sorting. Constructing a panoramic image of the sorting area provides an overall perspective and scene, which helps to fully understand the situation in the sorting area and provide basic data for subsequent automatic sorting of solid waste. The optimization of spatial panoramic images improves image quality and provides better visual support for subsequent analysis and operations.

[0023] Preferably, the specific steps of step S12 are:

[0024] Performing image region division on monitoring images of the areas to be sorted in multiple directions to obtain multiple image regions;

[0025] Calculating original grayscale pixel values ​​for multiple image regions to generate original grayscale pixel values ​​for each region;

[0026] Perform global average calculation on the original grayscale pixel values ​​of each region to obtain the average grayscale pixel value of the image;

[0027] Perform pixel linear stretching enhancement processing on the average grayscale pixel value of the image to generate enhanced grayscale pixel value;

[0028] Stretching pixel values ​​of multiple image regions according to enhanced grayscale pixel values ​​to generate a brightness-enhanced monitoring image;

[0029] Perform grayscale pixel frequency distribution analysis based on the original grayscale pixel value of each area to generate image grayscale pixel frequency distribution data;

[0030] Performing cumulative distribution function calculation on the grayscale pixel frequency distribution data of the image to extract the cumulative distribution function of the image;

[0031] Performing pixel contrast optimization processing on the cumulative distribution function of the image to obtain a contrast optimized cumulative distribution function;

[0032] The contrast optimization cumulative distribution function is used to perform contrast equalization on the brightness enhanced monitoring image, thereby constructing a multi-directional contrast balanced monitoring image.

[0033] The present invention helps to decompose the image into smaller processable units by dividing the image area, thereby improving the accuracy and efficiency of subsequent processing. The division of each image area can make the processing more targeted and help to adopt personalized processing methods for different parts. Calculating the original grayscale pixel value helps to understand the grayscale distribution of each area and provide basic data for subsequent processing. The calculation of these values ​​helps to evaluate the brightness distribution of the image and provide a basis for subsequent enhancement and optimization. Calculating the average grayscale pixel value of the image helps to understand the overall brightness level and provide a benchmark for subsequent enhancement processing. This step helps to adjust the brightness balance of the image and improve the visual quality and analyzability of the image. The pixel linear stretching enhancement processing expands the grayscale range of the image and enhances the contrast and visual effect of the image. This processing method It helps to highlight the details and features in the image and improve the visualization of the image. Analyzing the grayscale pixel frequency distribution helps to understand the distribution of each grayscale level in the image and provide guidance for subsequent processing. By analyzing the pixel frequency, the contrast and brightness of the image can be better adjusted to improve the quality and analyzability of the image. Calculating the cumulative distribution function helps to describe the overall contrast of the image and provide an important basis for subsequent contrast optimization. The calculation of the cumulative distribution function helps to understand the overall grayscale distribution of the image and provide an effective adjustment direction for subsequent processing. Contrast optimization processing improves the contrast of the image, enhances the details and features in the image, and improves the visualization of the image. The optimized contrast helps to highlight the important information in the image and provides better visual support for subsequent processing and analysis.

[0034] Preferably, the specific steps of step S2 are:

[0035] Step S21: performing a frame-by-frame visual positioning scan of solid waste on the panoramic image of the sorting area, and marking each solid waste positioning data;

[0036] Step S22: performing solid waste boundary contour recognition based on each solid waste location data, and extracting the boundary contour line of each solid waste;

[0037] Step S23: performing fine-grained segmentation on the panoramic image of the sorting area based on the boundary contour lines of each solid waste to generate an image boundary box of each solid waste;

[0038] Step S24: performing object geometric analysis on the image bounding box of each solid waste to generate solid waste geometric features;

[0039] Step S25: performing object texture detail recognition on the image bounding box of each solid waste to obtain solid waste texture detail data;

[0040] Step S26: performing a three-dimensional morphological structure analysis on the solid waste texture detail data and the solid waste geometric morphological features, thereby generating a three-dimensional morphological structure representation of each solid waste.

[0041] The present invention helps to accurately locate each object by scanning and marking the position of solid waste frame by frame, providing accurate data for subsequent processing, and can effectively identify the location information of solid waste, providing key data support for subsequent steps. Identifying the boundary contours of solid waste helps to accurately extract the shape and edge information of the object, preparing for subsequent analysis. The identification of boundary contours can help to accurately locate the shape and size of solid waste, providing an important basis for subsequent processing. Generating the boundary box of solid waste through fine-grained segmentation helps to accurately frame the position range of each object, providing data support for subsequent analysis. The image boundary box of each solid waste provides the accurate position and boundary information of the object in the image, providing a basis for subsequent processing. ,Object geometric morphology analysis can extract the shape characteristics of solid waste, such as aspect ratio, area, etc., providing an important basis for subsequent classification and sorting. Through geometric morphology feature analysis, we can have a deeper understanding of the morphological characteristics of solid waste, which is helpful for further processing and classification. Identifying the texture details of solid waste helps to distinguish different types of objects and improve the accuracy of classification. The acquisition of texture detail data helps to distinguish objects with similar shapes but different textures, providing more information for automatic sorting. Through three-dimensional morphological structure analysis, the geometric morphology and texture characteristics of solid waste are comprehensively considered to provide a more comprehensive object description, which helps to have a deeper understanding of the characteristics of solid waste and provide more accurate identification and classification basis for the automatic sorting system.

[0042] Preferably, the specific steps of step S3 are:

[0043] Step S31: performing solid type classification decision based on the three-dimensional morphological structure representation of each solid waste to obtain type data of each solid waste;

[0044] Step S32: performing sorting sequence analysis on the type data of each solid waste to generate sorting sequence data for each solid waste;

[0045] Step S33: performing sorting sequence coding on the sorting sequence data of each solid waste to obtain a solid waste sequence code;

[0046] Step S34: performing bounding box code mapping on the image bounding box of each solid waste based on the solid waste sequential code to obtain a code mapping bounding box of each solid waste.

[0047] The present invention makes classification decisions based on the three-dimensional morphological structure representation of solid waste, which helps to accurately distinguish different types of solid waste, realize automatic classification of solid waste, and provide correct type data for subsequent sorting and processing. Analyzing the sorting order of solid waste helps to determine the priority and order of each object in the processing process. By analyzing the sorting order, the sorting efficiency is improved, ensuring that the solid waste is correctly processed and classified. Coding the sorting order helps the system accurately record the processing order of each solid waste, facilitating subsequent tracking and management. Sorting order coding helps to improve the traceability and management efficiency of the automatic sorting system. Bounding box coding mapping based on solid waste sequence coding helps to associate each solid waste with its processing order, improve the accuracy and efficiency of solid waste processing, and ensure that each object is sorted and processed in the correct order.

[0048] Preferably, the specific steps of step S4 are:

[0049] Step S41: performing equipment action sequence simulation on the code mapping boundary box of each solid waste to obtain action sequence simulation data of the automatic sorting equipment;

[0050] Step S42: performing dynamic sorting movement tracking on the action sequence simulation data of the automatic sorting equipment to extract the dynamic sorting movement trajectory;

[0051] Step S43: Analyze the solid waste position change of the dynamic sorting movement trajectory to generate solid waste position change data;

[0052] Step S44: Calculating the equipment sorting accuracy based on the solid waste location change data to obtain the equipment sorting accuracy;

[0053] Step S45: identifying the action sequence deviation according to the equipment sorting accuracy to obtain the action sequence deviation value;

[0054] Step S46: performing deviation compensation calculation on the action sequence deviation value to generate action sequence deviation compensation data;

[0055] Step S47: fine-tuning the action parameters of the action sequence simulation data of the automatic sorting equipment based on the action sequence deviation compensation data to construct a fine-tuning action sequence.

[0056] The present invention generates action sequence simulation data of automatic sorting equipment by simulating the equipment action sequence of the coded mapping boundary box of solid waste, which helps to determine the operation order and method of the equipment in the processing flow. By dynamically sorting and moving the equipment action sequence simulation data, the movement trajectory of the solid waste during the processing process is extracted, which helps to track and monitor the position changes of the solid waste. The dynamic sorting movement trajectory is analyzed to generate solid waste position change data, which helps to understand the position changes of the solid waste during the processing process and provide data support for subsequent accuracy calculations. By calculating the solid waste position change data, the accuracy of equipment sorting is obtained, which helps to evaluate the performance and accuracy of the automatic sorting equipment. Action sequence deviation is identified based on the equipment sorting accuracy to determine the deviation in the equipment operation, providing a basis for further optimizing the equipment action. The action sequence deviation value is calculated and compensated, which helps to reduce errors in the equipment action and improve the accuracy and efficiency of sorting. By fine-tuning the equipment action sequence simulation data based on the deviation compensation data, a more accurate action sequence is constructed, thereby improving the performance and accuracy of the automatic sorting equipment.

[0057] Preferably, the specific steps of step S5 are:

[0058] Step S51: performing instant automatic sorting based on the fine-tuning action sequence and collecting sorting images at multiple time points;

[0059] Step S52: performing sorting status analysis on the multi-time point sorting images to extract the sorting status parameters of the automation equipment;

[0060] Step S53: performing a quantitative evaluation of the sorting efficiency of the automated equipment sorting state parameters to generate a quantitative evaluation value of the sorting efficiency;

[0061] Step S54: performing time-series sorting efficiency trend evolution on the quantitative evaluation value of the sorting efficiency to generate a sorting efficiency trend evolution feature.

[0062] The present invention achieves real-time monitoring and feedback by performing instant automatic sorting based on fine-tuning action sequences and collecting sorting images at multiple time points. This helps verify the effectiveness of action sequence fine-tuning and provides real-time sorting image data for subsequent data analysis. The multi-time point sorting images are analyzed to extract sorting status parameters of the automated equipment, such as sorting speed and accuracy. This helps understand the performance of the equipment in actual operation and provides data support for subsequent evaluations. By quantitatively evaluating the sorting status parameters of the automated equipment, the sorting efficiency of the equipment can be accurately assessed. This helps identify existing problems and propose improvement suggestions to improve sorting efficiency. A time series analysis of the quantitative evaluation values ​​of the sorting efficiency is performed to generate trend evolution characteristics of the sorting efficiency. This helps understand the changing trends of the equipment's sorting efficiency and provides a reference for long-term performance monitoring and optimization.

[0063] Preferably, the specific steps of step S6 are:

[0064] Step S61: performing dynamic sorting path planning optimization on the sorting efficiency trend evolution characteristics to obtain a dynamic sorting optimization path;

[0065] Step S62: Optimizing the entire sorting process of the fine-tuning action sequence based on the dynamic sorting optimization path to generate a global optimized action sequence;

[0066] Step S63: Adaptive reinforcement transfer learning is performed on the global optimization action sequence to build an adaptive intelligent sorting model to perform automatic solid waste sorting operations.

[0067] The present invention helps to adjust the sorting path according to the real-time situation and improve efficiency by analyzing the evolution characteristics of the sorting efficiency trend and performing dynamic path planning optimization. A more efficient solid waste sorting process is achieved through dynamic path planning optimization, which reduces time and resource waste. Full-process optimization of the fine-tuning action sequence based on the dynamic sorting optimization path helps to improve the efficiency and accuracy of the entire sorting process. The optimized action sequence can ensure that solid waste is accurately and quickly classified and processed, and improve the overall performance of the automatic sorting system. Through adaptive reinforcement transfer learning, an adaptive intelligent sorting model is constructed to continuously optimize system performance and adapt to the ever-changing environment and needs. Transfer learning helps the model learn and improve sorting strategies in different environments, thereby improving the accuracy and efficiency of automatic sorting of solid waste.

[0068] In this specification, a device for automatically sorting solid waste using machine vision is provided, which is used to perform the method for automatically sorting solid waste using machine vision as described above, comprising:

[0069] The geometric perspective correction module is used to obtain monitoring images of the area to be sorted in multiple directions; perform spatial geometric perspective deviation correction on the monitoring images of the area to be sorted in multiple directions, and perform spatial panoramic image optimization to construct a panoramic image of the sorting area;

[0070] The 3D morphological structure module is used to perform frame-by-frame visual positioning scanning of solid waste in the panoramic image of the sorting area and perform 3D morphological structure analysis to generate a 3D morphological structure representation of each solid waste;

[0071] A code mapping module is used to make solid type classification decisions based on the three-dimensional morphological structure representation of each solid waste, and perform bounding box code mapping to obtain a code mapping bounding box for each solid waste;

[0072] The action parameter fine-tuning module is used to simulate the equipment action sequence for the coded mapping boundary box of each solid waste, and then fine-tune the action parameters to construct a fine-tuned action sequence;

[0073] The sorting efficiency trend module is used to perform instant automatic sorting based on fine-tuning action sequences and collect sorting images at multiple time points. It also analyzes the temporal sorting efficiency trend evolution of the multi-time point sorting images and generates sorting efficiency trend evolution features.

[0074] The sorting path optimization module is used to perform dynamic sorting path planning optimization and adaptive reinforcement transfer learning based on the sorting efficiency trend evolution characteristics, thereby building an adaptive intelligent sorting model to perform automatic solid waste sorting operations.

[0075] The present invention obtains monitoring images of the area to be sorted in multiple directions and performs geometric perspective deviation correction, which helps to eliminate perspective distortion in the image and improve sorting accuracy. It constructs a panoramic image of the sorting area to provide a more comprehensive field of view for subsequent solid waste identification and sorting, and increases the global perception ability of the system. Through frame-by-frame solid waste visual positioning scanning and three-dimensional morphological structure analysis, the morphological characteristics of each solid waste are accurately obtained, providing important data support for subsequent classification and sorting. The generation of three-dimensional morphological structure representation helps the system to accurately identify and classify solid waste, improve sorting accuracy and efficiency, and make classification decisions and bounding box coding mapping based on the three-dimensional morphological structure representation of solid waste to achieve accurate solid type classification and position marking. The coding mapping bounding box provides accurate Accurate location information provides an important basis for subsequent processing and sorting. Through equipment action sequence simulation and action parameter fine-tuning, the solid waste processing action is optimized to improve the accuracy and efficiency of sorting. The construction of fine-tuning action sequence helps to ensure that each solid waste is correctly processed and reduce processing errors and losses. Analysis of the evolution characteristics of time-series sorting efficiency trends helps the system understand the changing rules of sorting efficiency, optimize sorting strategies and improve overall efficiency. Through instant automatic sorting and analysis of efficiency trends, the system adjusts strategies in real time to improve sorting efficiency and accuracy. Dynamic sorting path planning optimization and adaptive reinforcement transfer learning help to build intelligent sorting models and improve system adaptability and performance. The adaptive intelligent sorting model adjusts the sorting path according to real-time conditions to improve the efficiency and accuracy of automatic sorting of solid waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a schematic flow chart of the steps of a method for automatically sorting solid waste using machine vision according to the present invention;

[0077] Figure 2 Detailed implementation flow chart of step S1;

[0078] Figure 3 Detailed implementation flow chart of step S2;

[0079] Figure 4Schematic diagram of the detailed implementation steps of step S3. DETAILED DESCRIPTION

[0080] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0081] This application provides a method and apparatus for automatically sorting solid waste using machine vision. The execution entities of this method and apparatus include, but are not limited to, the following: mechanical equipment, data processing platforms, cloud server nodes, network upload devices, etc., which can be considered general computing nodes of this application. The data processing platform includes, but is not limited to, at least one of an audio and image management system, an information management system, and a cloud data management system.

[0082] See also Figures 1 to 4 The present invention provides a method for automatically sorting solid waste using machine vision, and the method comprises the following steps:

[0083] Step S1: Acquire monitoring images of the area to be sorted in multiple directions; perform spatial geometric perspective deviation correction on the monitoring images of the area to be sorted in multiple directions, and perform spatial panoramic image optimization to construct a panoramic image of the sorting area;

[0084] Step S2: Performing a frame-by-frame visual positioning scan of the solid waste on the panoramic image of the sorting area and performing a three-dimensional morphological structure analysis to generate a three-dimensional morphological structure representation of each solid waste;

[0085] Step S3: performing solid type classification decision based on the three-dimensional morphological structure representation of each solid waste, and performing bounding box coding mapping to obtain a coding mapping bounding box for each solid waste;

[0086] Step S4: simulate the equipment action sequence for the coded mapping bounding box of each solid waste, and then fine-tune the action parameters to construct a fine-tuned action sequence;

[0087] Step S5: Performing instant automatic sorting based on the fine-tuning action sequence and collecting sorting images at multiple time points; performing temporal sorting efficiency trend evolution on the sorting images at multiple time points to generate sorting efficiency trend evolution features;

[0088] Step S6: Dynamic sorting path planning optimization and adaptive reinforcement transfer learning are performed on the sorting efficiency trend evolution characteristics to build an adaptive intelligent sorting model to perform automatic solid waste sorting operations.

[0089] The present invention improves image quality by correcting spatial geometric viewing angle deviation and optimizing panoramic images, making subsequent processing easier.

[0090] Constructing a panoramic image of the sorting area helps provide an overall perspective, laying the foundation for subsequent solid waste identification and sorting. Through frame-by-frame visual positioning scanning of solid waste and three-dimensional morphological structure analysis, the morphological characteristics of each solid waste are accurately obtained, and the generation of three-dimensional morphological structure representation helps in subsequent classification and sorting decisions, improving the accuracy and efficiency of automatic sorting. Solid type classification based on three-dimensional morphological structure representation accurately identifies different types of solid waste. Bounding box coding mapping provides an accurate mark for the location of each solid waste, providing a basis for subsequent action simulation and sorting. Equipment action sequence simulation and fine-tuning optimize the sorting process The actions and parameters in the sorting process can improve the accuracy of automatic sorting. Building a fine-tuning action sequence can help optimize the operating procedures of robots or equipment, improve sorting efficiency and accuracy, and analyze the temporal evolution characteristics of sorting images at multiple time points to help evaluate the changing trend of sorting efficiency, timely discover problems and improvement directions, and generate sorting efficiency trend evolution characteristics to help optimize sorting strategies and improve overall operational efficiency. Dynamic sorting path planning optimization and adaptive reinforcement transfer learning can improve the intelligence level and adaptability of the automatic sorting system. Building an adaptive intelligent sorting model can help achieve efficient automatic sorting of solid waste, improve production efficiency and reduce labor costs.

[0091] In the embodiment of the present invention, see Figure 1 , is a flowchart of the steps of a method for automatically sorting solid waste using machine vision according to the present invention. In this example, the steps of the method for automatically sorting solid waste using machine vision include:

[0092] Step S1: Acquire monitoring images of the area to be sorted in multiple directions; perform spatial geometric perspective deviation correction on the monitoring images of the area to be sorted in multiple directions, and perform spatial panoramic image optimization to construct a panoramic image of the sorting area;

[0093] In this embodiment, multiple high-resolution cameras are configured to ensure coverage of all directions of the sorting area (such as the front, side, and top), ensure camera stability, and avoid image quality degradation due to vibration. The cameras are started to capture images at predetermined time intervals to ensure that clear images can be obtained under different lighting and environmental conditions. An automated script is used to control the operation of each camera to ensure synchronization of image acquisition time. The captured images are stored locally or in cloud storage, and image files are ensured to have a good naming convention for subsequent processing (for example, using timestamps and camera numbers). A calibration board (such as a checkerboard) is used for camera calibration to obtain the intrinsic parameters (focal length, optical center position, distortion coefficient) and extrinsic parameters (position and orientation) of each camera. A camera calibration algorithm (such as Zhang's calibration method) is applied to calculate the geometric parameters of each camera. The image distortion correction function in a computer vision library such as OpenCV is used to perform geometric correction on the images captured by each camera to ensure that the shape and size of objects in the image are consistent with the actual objects. Eliminate deviations caused by lens distortion, identify common feature points in each image through feature matching algorithms (such as SIFT and ORB), calculate the transformation matrix between perspectives, align images from different perspectives to ensure they are spatially aligned to form a common reference frame, use image stitching algorithms (such as the RANSAC algorithm combined with image registration) to stitch the corrected images to generate a preliminary panoramic image, process overlapping areas that appear during the stitching process, use weighted averaging and other methods to smooth the stitching gaps and improve image quality, post-process the generated panoramic image, apply image enhancement techniques (such as contrast adjustment and brightness balance) to improve image visibility, use denoising algorithms (such as Gaussian filtering and median filtering) to improve image clarity, ensure that details in the sorting area are clearly visible, save the optimized panoramic image to the database, ensure it has a good format (such as JPEG, PNG) and resolution for subsequent analysis and use, and add relevant metadata (such as acquisition time and camera position) to the panoramic image for subsequent retrieval and management.

[0094] Step S2: Performing a frame-by-frame visual positioning scan of the solid waste on the panoramic image of the sorting area and performing a three-dimensional morphological structure analysis to generate a three-dimensional morphological structure representation of each solid waste;

[0095] In this embodiment, the panoramic image is preprocessed, including denoising (using Gaussian filtering or median filtering) and contrast enhancement (for example, using histogram equalization) to improve the visibility of solid waste, converting the image into a grayscale image to reduce computational complexity and facilitate subsequent feature extraction, using an edge detection algorithm (such as Canny edge detection) to extract edge information in the image to identify the outline of solid waste, applying morphological operations (such as dilation and erosion) to further clarify the edges and eliminate small noise points, and using an object detection algorithm (such as YOLO, SSD) to detect solid waste in the image, identify the position and bounding box of each object, and post-process the detection results to filter out low-confidence detection results to ensure that only accurate object positioning information is retained. The position information of each solid waste (such as bounding box coordinates, object ID, etc.) is recorded in a data structure for subsequent analysis and processing. If the camera used supports depth perception (such as a stereo camera or a structured light camera), the position of each solid waste is obtained. , and the depth information of the object is used to construct a depth map. If depth perception is not supported, monocular vision is used in combination with the known size of the object to estimate the depth (such as through the perspective principle and camera intrinsic parameter inversion). A 3D reconstruction algorithm (such as voxel method, point cloud method or multi-view stereo vision) is used to combine the two-dimensional boundary contour and depth information of the solid waste to generate a 3D model of each object. The reconstructed 3D model is smoothed to eliminate the noise and irregularities generated during the reconstruction process. The geometric features of each 3D model (such as volume, surface area, shape complexity, etc.) are calculated, and these features are recorded as the morphological structure representation of the object. Morphological analysis methods (such as principal component analysis) are used to further analyze the morphological features of the object to extract the main structural features. The 3D model of each solid waste and its morphological features are saved in the database to ensure that the data structure is clear (such as including object ID, shape features, size information, etc.). A report or visualization interface is generated to display the 3D morphological structure of each solid waste to facilitate subsequent analysis and decision-making.

[0096] Step S3: performing solid type classification decision based on the three-dimensional morphological structure representation of each solid waste, and performing bounding box coding mapping to obtain a coding mapping bounding box for each solid waste;

[0097] In this embodiment, key morphological features such as volume, surface area, aspect ratio, shape complexity, etc. are extracted from the three-dimensional model of each solid waste. Features meaningful for classification decisions are selected, including geometric features, color histograms, and texture features. The extracted feature data are standardized to eliminate the influence of different feature dimensions and ensure the effectiveness of the classification algorithm. Dimensionality reduction is performed using methods such as principal component analysis (PCA) to reduce the complexity of the feature space while retaining the main information. Appropriate classification algorithms such as support vector machines (SVMs), random forests, and deep learning (such as convolutional neural networks) are selected based on the size of the dataset and the complexity of the features. If a well-labeled training dataset is available, a classification model is constructed using supervised learning. If not, unsupervised learning methods (such as K-Means clustering) are considered for preliminary classification.

[0098] Use the extracted feature dataset to train the selected classification model. For supervised learning, the dataset needs to be divided into training and test sets to ensure the accuracy and generalization ability of the training. Optimize the model parameters through cross-validation and other methods to ensure the best performance of the classification model. Input the three-dimensional morphological structural features of each solid waste into the trained classification model to classify the solid type. Record the classification results of each solid waste for subsequent processing. Design a coding system to ensure that each solid type has a unique code. For example, use alphanumeric combinations to represent different waste types (such as "P1" for plastic, "M1" for metal, etc.). Based on the previous visual positioning results, generate a corresponding bounding box for each solid waste. The coordinates of the bounding box should be based on the position and size of the object in the panoramic image. Use the calculated bounding box coordinates to generate the corresponding The bounding box graphics are used to ensure that they accurately surround the target object. The classification results of each solid waste are combined with its bounding box information to form a coded mapping bounding box. The unique code, bounding box coordinates and object type of each solid waste are recorded. The coded mapping results are stored in the database, and a suitable data structure (such as object ID, code, bounding box coordinates) is designed for subsequent retrieval and use. The coded mapping bounding box results of each solid waste are output to a standard format (such as JSON, XML or CSV) to facilitate subsequent data processing and analysis. A visual report of the sorting results is generated to display the classification of each solid waste and its corresponding bounding box. The performance of the classification model is verified, and the classification accuracy, recall rate and F1 score are calculated to evaluate the reliability of the model. According to the verification results, the model is adjusted and optimized to ensure the accuracy of the classification effect.

[0099] Step S4: simulate the equipment action sequence for the coded mapping bounding box of each solid waste, and then fine-tune the action parameters to construct a fine-tuned action sequence;

[0100] In this example, ensure that the bounding box information of each solid waste is correctly recorded, including the coordinates of the bounding box, object type and unique code. According to the function and operation manual of the equipment, design a preliminary action sequence, including key actions such as grasping, lifting, moving and placing. Each action should have clear parameters, such as starting position, target position, action speed and grasping force, etc. Use simulation software (such as Gazebo, MATLAB) to simulate the action sequence. Build a virtual sorting environment using Simulink and Unity to ensure that it can simulate the operation of real equipment, import coded mapping bounding box data to ensure that the model can accurately reflect the actual position and status of the object, execute the designed action sequence in the simulation environment, observe the impact of each action on solid waste, and record key data during the simulation, including action execution time, success rate, and any problems that arise (such as grasping failure or misoperation). According to the results of the simulation run, set fine-tuning goals, for example, improving the grasping success rate, reducing the action execution time, or improving sorting efficiency. Identify the types of parameters that need to be adjusted, such as grasping angle, grasping force, moving speed, etc. Apply parameter optimization algorithms (such as grid search, random search, or genetic algorithm) to systematically adjust the action parameters, select a series of parameter combinations, and gradually test their effects in the simulation environment to find the optimal parameter configuration. Run the fine-tuned action sequence, observe and Record the execution effect of each action, compare the data before and after fine-tuning, analyze whether the effect of fine-tuning has achieved the expected goal, further adjust the parameters according to the test results, optimize the action sequence, form a cyclic iterative optimization process, record the results of each fine-tuning, ensure that the performance of different parameter configurations can be tracked, integrate the action sequence that has undergone multiple fine-tuning into a final version, ensure that the execution order and parameter settings of each action are reasonable, ensure that the fine-tuned action sequence can be efficiently executed on the real device, record the final fine-tuned action sequence in a document, including a detailed description of each action and its parameter settings, store the fine-tuned action sequence in the control system of the automatic sorting equipment, ensure that it can be called in actual operation, test run on the actual equipment, verify the effectiveness of the fine-tuned action sequence, observe its performance in the actual environment, and make necessary final adjustments based on the feedback from actual operation to ensure that the operation of the equipment achieves the best results.

[0101] Step S5: Performing instant automatic sorting based on the fine-tuning action sequence and collecting sorting images at multiple time points; performing temporal sorting efficiency trend evolution on the sorting images at multiple time points to generate sorting efficiency trend evolution features;

[0102] In this embodiment, it is ensured that the automatic sorting equipment has loaded the fine-tuned action sequence, and a pre-operation test is performed to confirm that the equipment status is normal. Sensors (such as lidar and cameras) are configured to ensure real-time monitoring of the sorting area. The sorting operation is started in the control system. The automated equipment performs the sorting task according to the fine-tuned action sequence. The equipment begins to receive solid waste and classifies and processes it according to the coding mapping information. The image acquisition device is triggered at key nodes in the sorting process (such as grabbing, moving and placing) to record the sorting status at each time point. The image acquisition frequency is set to ensure that images at multiple time points can be captured during the sorting process (for example, one image is captured every 1 second). The collected multi-time point sorting images are preprocessed, including noise removal and contrast enhancement, to improve the accuracy of subsequent analysis. The images are converted into grayscale images or binary images to simplify the analysis process. Computer vision technology (such as image segmentation and feature extraction) is used to extract sorting status information from each image, including the type, quantity and location of each solid waste. , record the sorting success rate, failure rate and other related indicators at each time point, calculate the sorting efficiency indicators based on the extracted sorting status information, such as: the number of objects sorted per hour (Throughput), correct sorting rate (number of correctly sorted objects / total number of sorted objects), average sorting time (average sorting time for each object), organize the calculated sorting efficiency indicators into time series data, ensure consistent data format and accurate timestamps, create a data framework, record the sorting efficiency indicators at each time point for subsequent analysis, apply time series analysis methods (such as moving average, exponential smoothing method) to perform trend analysis on sorting efficiency data, identify patterns of efficiency changes, generate efficiency trend graphs, display the dynamic characteristics of sorting efficiency over time, extract sorting efficiency trend evolution characteristics, including efficiency fluctuation amplitude, upward / downward trends and key time nodes, generate sorting efficiency analysis reports, including data visualization (such as line charts, bar charts, etc.) and key analysis results, so that management can formulate improvement strategies.

[0103] Step S6: Dynamic sorting path planning optimization and adaptive reinforcement transfer learning are performed on the sorting efficiency trend evolution characteristics to build an adaptive intelligent sorting model to perform automatic solid waste sorting operations.

[0104] In this embodiment, the sorting efficiency trend evolution feature data generated in the previous steps is collected and organized, including information such as efficiency fluctuations, trend changes, and time nodes. Statistical analysis methods (such as regression analysis and correlation analysis) are used to conduct in-depth analysis of the sorting efficiency features, identify key factors affecting sorting efficiency, generate a feature report, and clarify which features have a significant impact on improving sorting efficiency. An appropriate path planning algorithm (such as the A* algorithm, Dijkstra algorithm, or RRT) is selected to construct a dynamic sorting path model. The sorting area is divided into multiple grids or regions to ensure that the path planning can cover all objects to be sorted. The path planning model is adjusted in real time based on the sorting efficiency trend characteristics to ensure that the optimal path is selected under different workloads and material types. A dynamic optimization algorithm (such as a genetic algorithm or particle swarm optimization) is applied to optimize the path in real time to ensure flexibility and efficiency in path selection. The optimized sorting path is verified in a simulation environment to ensure that it can be effectively executed in actual operations. Feedback data after the path execution is collected to further adjust and optimize the path planning. The framework of the adaptive intelligent sorting model is determined, and an appropriate reinforcement learning algorithm (such as Deep Q-Learning, Proximal Optimization, etc.) is selected. The adaptive intelligent sorting model is trained in a simulated environment. The strategy is optimized through a continuous trial-and-error process, and the sorting results are recorded. The model parameters are adjusted using the feedback mechanism of the reinforcement learning algorithm to improve sorting efficiency and accuracy. The model is evaluated in an actual sorting environment, and its performance in real tasks is monitored to ensure that it can adapt to different solid waste types and working conditions. The model's performance indicators, such as sorting accuracy, speed, and efficiency, are calculated to evaluate its effectiveness. The trained adaptive intelligent sorting model is deployed in the control system of the automatic sorting equipment to ensure that it can perform sorting tasks in real time. Data from the real environment is regularly collected, and the model is updated and retrained online to ensure that the model can continuously adapt to new sorting conditions.

[0105] In this embodiment, refer to Figure 2 , is a flowchart of the detailed implementation steps of step S1. In this embodiment, the detailed implementation steps of step S1 include:

[0106] Step S11: Acquire monitoring images of the area to be sorted in multiple directions based on a high-resolution panoramic camera;

[0107] Step S12: Optimizing the image contrast of monitoring images of the areas to be sorted in multiple directions to construct a multi-directional contrast-balanced monitoring image;

[0108] Step S13: performing image feature point recognition on the multi-directional contrast balance monitoring image and marking multiple image feature points;

[0109] Step S14: performing multi-directional registration on multiple image feature points to obtain multi-angle matching data for each feature point;

[0110] Step S15: performing spatial geometric perspective deviation correction based on the multi-angle matching data of each feature point, thereby obtaining a spatial geometric perspective deviation corrected image;

[0111] Step S16: performing spatial panoramic image optimization on the spatial geometric perspective deviation-corrected image to construct a panoramic image of the sorting area.

[0112] In this embodiment, an on-site assessment is conducted within the sorting area to determine the key locations to be monitored. These locations should cover the entire sorting area to ensure that there are no blind spots. High-resolution panoramic cameras are installed at multiple predetermined locations to ensure that the camera fields of view overlap for subsequent image processing. The camera height and angle are ensured to capture the optimal view. Camera parameters (such as resolution and frame rate) are configured to obtain clear images. After the setup is completed, the monitoring system is started, and images of the sorting area captured by each camera are periodically saved. The monitoring images of the sorting area obtained from each camera are loaded using an image processing tool for subsequent contrast optimization. , select a suitable contrast optimization algorithm according to the characteristics of the image, such as histogram equalization or adaptive histogram equalization. These methods enhance the contrast of the image, making the details clearer and more discernible. Apply the selected contrast optimization algorithm to each monitoring image to generate a monitoring image with multi-directional contrast balance. This process significantly improves the details and visualization effects in the image. Select a suitable feature point detection algorithm, such as SIFT, SURF or ORB, to identify the key feature points in the image. These feature points should have good repeatability and stability. Apply the selected feature point detection algorithm to each contrast-optimized monitoring image to identify and mark multiple image feature points. These feature points will become the basis for the subsequent matching process. Select a suitable feature matching algorithm, such as FLANN or BFMatcher. These algorithms can effectively find the correspondence between feature points between different images, match the feature points in each pair of images, and generate multi-angle matching data for each feature point. The purpose of this step is to find the corresponding feature points of the same object under different perspectives, providing a basis for subsequent geometric correction. Based on the matched feature points, calculate the geometric transformation between different perspectives, including translation and rotation. Use the RANSAC algorithm to effectively remove mismatches in the matching, thereby improving the accuracy of the calculation. Apply the calculated geometric transformation to each image to correct its perspective deviation and generate a spatial geometric perspective deviation-corrected image. This step ensures that images from different perspectives can be aligned in the same coordinate system. Use a panoramic stitching algorithm to stitch multiple corrected images into a complete panoramic image. This process requires processing the overlapping areas between images to ensure smoothness and seamless connection of the stitching. Post-processing is performed on the stitched panoramic image, including color correction and image smoothing, to improve the overall visual effect and readability of the information. The optimized panoramic image of the sorting area is saved to the specified file path to ensure the safe storage of the image data for subsequent analysis and use.

[0113] In this embodiment, the specific steps of step S12 are:

[0114] Performing image region division on monitoring images of the areas to be sorted in multiple directions to obtain multiple image regions;

[0115] Calculating original grayscale pixel values ​​for multiple image regions to generate original grayscale pixel values ​​for each region;

[0116] Perform global average calculation on the original grayscale pixel values ​​of each region to obtain the average grayscale pixel value of the image;

[0117] Perform pixel linear stretching enhancement processing on the average grayscale pixel value of the image to generate enhanced grayscale pixel value;

[0118] Stretching pixel values ​​of multiple image regions according to enhanced grayscale pixel values ​​to generate a brightness-enhanced monitoring image;

[0119] Perform grayscale pixel frequency distribution analysis based on the original grayscale pixel value of each area to generate image grayscale pixel frequency distribution data;

[0120] Performing cumulative distribution function calculation on the grayscale pixel frequency distribution data of the image to extract the cumulative distribution function of the image;

[0121] Performing pixel contrast optimization processing on the cumulative distribution function of the image to obtain a contrast optimized cumulative distribution function;

[0122] The contrast optimization cumulative distribution function is used to perform contrast equalization on the brightness enhanced monitoring image, thereby constructing a multi-directional contrast balanced monitoring image.

[0123] In this embodiment, according to the characteristics of the area to be sorted, an appropriate image segmentation strategy is selected, such as uniform division or division based on specific features (such as color or texture), to divide the entire image into multiple small areas, ensuring that each area contains the object to be sorted, and this is achieved by setting a grid or using a region growing method. For each divided image area, its original grayscale pixel value is calculated, which is usually achieved by converting the RGB image into a grayscale image, summing the grayscale values ​​of each area and dividing it by the total number of pixels in the area to obtain the average grayscale pixel value of the area, and storing the average grayscale value of each area in a list or data table for subsequent use. A linear stretching method is used, usually by determining the minimum and maximum grayscale values ​​of the image and linearly mapping them to a new range (such as 0 to 255), applying a linear stretching formula to the grayscale values ​​of each area to generate enhanced grayscale pixel values. This process ensures that the contrast of the image is improved, and assigning a brightness value to each area of ​​the original image based on the enhanced grayscale pixel value to generate a monitoring image with enhanced brightness. All brightness-enhanced areas are synthesized into a complete monitoring image for subsequent analysis. The original grayscale pixel values ​​of each area are counted, and the frequency of occurrence of each grayscale value is calculated to obtain the grayscale pixel frequency distribution data of each area. The grayscale frequency distribution data is stored in a data structure for subsequent cumulative distribution calculation. The grayscale pixel frequency distribution is cumulatively calculated to generate a cumulative distribution function (CDF) for each grayscale value, which represents the proportion of pixels less than or equal to the grayscale value. The calculated cumulative distribution function data is stored in an appropriate data structure for subsequent processing. A suitable contrast optimization algorithm is selected, which is usually a contrast adjustment based on the cumulative distribution function. The optimized cumulative distribution function is used to process the brightness-enhanced monitoring image to improve the overall contrast of the image. Histogram equalization is performed on the brightness-enhanced monitoring image based on the contrast-optimized cumulative distribution function to generate a contrast-balanced image. The generated multi-directional contrast-balanced monitoring image is saved to a storage device for subsequent use and analysis.

[0124] In this embodiment, refer to Figure 3 , is a flowchart of the detailed implementation steps of step S2. In this embodiment, the detailed implementation steps of step S2 include:

[0125] Step S21: performing a frame-by-frame visual positioning scan of solid waste on the panoramic image of the sorting area, and marking each solid waste positioning data;

[0126] Step S22: performing solid waste boundary contour recognition based on each solid waste location data, and extracting the boundary contour line of each solid waste;

[0127] Step S23: performing fine-grained segmentation on the panoramic image of the sorting area based on the boundary contour lines of each solid waste to generate an image boundary box of each solid waste;

[0128] Step S24: performing object geometric analysis on the image bounding box of each solid waste to generate solid waste geometric features;

[0129] Step S25: performing object texture detail recognition on the image bounding box of each solid waste to obtain solid waste texture detail data;

[0130] Step S26: performing a three-dimensional morphological structure analysis on the solid waste texture detail data and the solid waste geometric morphological features, thereby generating a three-dimensional morphological structure representation of each solid waste.

[0131] In this embodiment, a suitable target detection algorithm, such as YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector), is selected for real-time detection of solid waste in images. Solid waste is detected for each frame of the image, and the detected location and category of the solid waste are marked as positioning data. The coordinates and size of each solid waste are recorded. An edge detection algorithm (such as Canny edge detection) is used to identify the boundary contour of the solid waste. Edge detection is applied to each solid waste positioning image region to extract its boundary contour line and generate boundary data. A watershed algorithm or a deep learning-based segmentation network (such as Mask R-CNN) is used for fine-grained image segmentation to ensure that the boundary of each solid waste is clearly visible. Based on the extracted boundary contour, the image bounding box of each solid waste is generated, and the coordinate information of the bounding box is recorded. Geometric morphological analysis methods, such as shape descriptors (such as Hu moments), are used to calculate the geometric features of each solid waste, including area, perimeter, aspect ratio, etc. The bounding box of each solid waste is geometrically analyzed to extract its morphological features and record these feature data. Texture analysis algorithms, such as gray-level co-occurrence matrix (GLCM) or LBP (local binary pattern) are used to analyze the texture features of solid waste. Texture features are extracted from the bounding box of each solid waste to generate texture detail data of solid waste. Three-dimensional reconstruction technology, such as structured light or stereo vision methods, is used to combine geometric morphology and texture data to generate a three-dimensional model of solid waste. According to the extracted geometric morphological features and texture detail data, a three-dimensional morphological structure representation of each solid waste is constructed to form a visual three-dimensional model.

[0132] In this embodiment, refer to Figure 4 , is a flowchart of the detailed implementation steps of step S3. In this embodiment, the detailed implementation steps of step S3 include:

[0133] Step S31: performing solid type classification decision based on the three-dimensional morphological structure representation of each solid waste to obtain type data of each solid waste;

[0134] Step S32: performing sorting sequence analysis on the type data of each solid waste to generate sorting sequence data for each solid waste;

[0135] Step S33: performing sorting sequence coding on the sorting sequence data of each solid waste to obtain a solid waste sequence code;

[0136] Step S34: performing bounding box code mapping on the image bounding box of each solid waste based on the solid waste sequential code to obtain a code mapping bounding box of each solid waste.

[0137] In this embodiment, based on the three-dimensional morphological structure data of solid waste, a suitable machine learning or deep learning classification algorithm is selected, such as support vector machine (SVM), random forest, or convolutional neural network (CNN) to collect and annotate three-dimensional morphological data containing different types of solid waste to form a training set and a test set to facilitate model training and verification. The training data is used to train the selected classification model and optimize the model parameters to improve the classification accuracy. The three-dimensional morphological structure data of each solid waste is input into the trained model to perform type classification and generate type data for each solid waste. According to the type and characteristics of the solid waste, a corresponding sorting strategy is formulated, taking into account factors such as priority, processing time, and category, and the type data of each solid waste is analyzed to generate The sorting sequence data of each solid waste is usually sorted by priority, and the rules for sorting sequence coding are determined, such as using simple digital codes (such as 1, 2, 3) or more complex coding methods (such as a combination of letters and numbers). According to the formulated coding scheme, the sorting sequence data of each solid waste is encoded to generate a sequential code of the solid waste, and the encoded sorting sequence data is saved to the database to ensure the persistence and accessibility of the data. The image bounding box data of each solid waste is collected, including the coordinates and size information of the bounding box. According to the sequential code of the solid waste, the image bounding box of each solid waste is coded and mapped to generate a coded mapping bounding box, and the generated coded mapping bounding box data is recorded in the database to facilitate subsequent sorting operations and management.

[0138] In this embodiment, step S4 includes the following steps:

[0139] Step S41: performing equipment action sequence simulation on the code mapping boundary box of each solid waste to obtain action sequence simulation data of the automatic sorting equipment;

[0140] Step S42: performing dynamic sorting movement tracking on the action sequence simulation data of the automatic sorting equipment to extract the dynamic sorting movement trajectory;

[0141] Step S43: Analyze the solid waste position change of the dynamic sorting movement trajectory to generate solid waste position change data;

[0142] Step S44: Calculating the equipment sorting accuracy based on the solid waste location change data to obtain the equipment sorting accuracy;

[0143] Step S45: identifying the action sequence deviation according to the equipment sorting accuracy to obtain the action sequence deviation value;

[0144] Step S46: performing deviation compensation calculation on the action sequence deviation value to generate action sequence deviation compensation data;

[0145] Step S47: fine-tuning the action parameters of the action sequence simulation data of the automatic sorting equipment based on the action sequence deviation compensation data to construct a fine-tuning action sequence.

[0146] In this embodiment, coded information of solid waste, including type, size, and weight, is collected. Sensors (e.g., laser scanners) are used to obtain bounding box data of the waste to form the basis of a 3D model. Computer vision tools (e.g., OpenCV, Blender) are used to convert the bounding box data into a 3D model. The model is optimized to ensure that it can run effectively in simulation software. According to the operating manual of the automatic sorting equipment, an action sequence including basic actions such as grasping, moving, rotating, and placing is designed. Specific parameters, such as grasping height and placing angle, are set for each action. The 3D model is then simulated in a simulation environment (e.g., Gazebo, MATLAB). Input the designed action sequence into Simulink, run the simulation, observe the execution of each action and its impact on the location of solid waste, record the time delay, action success rate and error type of each step, deploy sensors (such as lidar and cameras) on the sorting equipment, capture the equipment's motion data in real time, set the data collection frequency to ensure the real-time and accuracy of the data, use data processing algorithms such as Kalman filters to smooth and filter sensor data to reduce noise, extract the motion trajectory based on timestamp and coordinate data, including the starting point, end point and path, and use visualization tools (such as Matplotlib and Plotly) to plot the extracted trajectory in three-dimensional space to ensure that the visualization results can clearly show the equipment's motion path and the sorting of solid waste. According to the sorting situation, the dynamic sorting movement trajectory is compared with the initial position of the solid waste frame by frame to identify the displacement of each piece of waste, and the movement of each solid waste object is marked for subsequent analysis. The displacement vector of each solid waste object, including the moving distance (Euclidean distance) and direction (angle), is calculated, and the difference between the final position and the expected position of each object is recorded. The position change data is stored in the database, and a suitable data table structure is designed, including fields such as object ID, initial position, end position, displacement, etc., to ensure the scalability of data storage for subsequent large-scale data analysis, and to clarify the calculation standards for sorting accuracy, such as the correct sorting rate (the number of correctly sorted wastes / the total number of sorted wastes) and the missorting rate (the number of missorted wastes / the total number of sorted wastes).

[0147] Use statistical analysis tools (such as Python's Pandas library) to analyze solid waste location change data, calculate sorting accuracy indicators, and generate statistical reports containing charts and key data. Generate detailed sorting accuracy reports with data visualization (such as bar charts and pie charts) and analysis results to help management understand equipment performance. Based on the sorting accuracy calculation results, set standards for deviation identification. For example, if the success rate of an action falls below a threshold, it is marked as a deviation. Compare the expected action sequence with the actual action sequence to gradually identify deviations. Record the execution time, location, and status information of each action to determine the source of the deviation and record the identified action sequence deviation values ​​in a database. Design appropriate data structures (such as action ID, deviation type, deviation value, etc.) and design compensation algorithms, including PID controllers or machine learning models, to adjust action parameters based on the deviation value. Set compensation rules. For example, when the deviation exceeds a certain range, automatically adjust the gripping force and angle. Dynamically adjust action parameters based on real-time data feedback to ensure that each operation corrects the previous deviation. Conduct real-time monitoring to quickly respond to different sorting situations. Generate records containing compensation data and ensure that the data format is standardized for subsequent use.

[0148] Apply the compensation data to the control system of the equipment to ensure that it is effective in actual operation. Based on the compensation data, fine-tune the action sequence and update the time, speed, gripping angle and other parameters of each action to ensure that each parameter adjustment has a clear basis and record the adjustment process and results. Test the fine-tuned action sequence again through simulation software to observe its performance under different conditions. Conduct multiple tests to ensure that the new sequence is stable and efficient in actual operation. Update the fine-tuned action sequence data to the control system of the automatic sorting equipment to ensure that the equipment can use the new strategy in actual work. Develop a monitoring plan and regularly evaluate the performance of the new sequence to ensure continuous optimization and improvement.

[0149] In this embodiment, step S5 includes the following steps:

[0150] Step S51: performing instant automatic sorting based on the fine-tuning action sequence and collecting sorting images at multiple time points;

[0151] Step S52: performing sorting status analysis on the multi-time point sorting images to extract the sorting status parameters of the automation equipment;

[0152] Step S53: performing a quantitative evaluation of the sorting efficiency of the automated equipment sorting state parameters to generate a quantitative evaluation value of the sorting efficiency;

[0153] Step S54: performing time-series sorting efficiency trend evolution on the quantitative evaluation value of the sorting efficiency to generate a sorting efficiency trend evolution feature.

[0154] In this embodiment, it is ensured that the fine-tuned action sequence has been loaded into the automatic sorting equipment, and preliminary tests are conducted to verify that the system is operating normally. An image acquisition device (such as a high-resolution camera) is configured to ensure that it covers the sorting area without blind spots. The automatic sorting program is started in the control system, and real-time sorting operations are performed according to the fine-tuned action sequence. The equipment begins to receive solid waste materials and classifies them according to the preset sorting logic. The image acquisition device is triggered at each key time point of sorting (such as when the material approaches the sorting point, when the material sorting is completed, etc.) to record images of the sorting process. The image acquisition frequency is set, for example, one image per second, to ensure that dynamic changes can be captured. The collected images are stored in the cloud or local database, and the image file naming standards are ensured to facilitate subsequent retrieval and analysis. The collected multi-time point sorting images are preprocessed, including noise removal, contrast enhancement, brightness adjustment, etc., to improve the accuracy of subsequent analysis.

[0155] Use image processing tools (such as OpenCV) to convert the image format into a format suitable for analysis. Apply computer vision algorithms (such as object recognition and edge detection) to extract sorting status parameters from the image, such as the type and number of sorted objects, the execution status of the sorting action (success, failure), and the operating status of the equipment (such as operating speed and gripping force). Use deep learning models (such as YOLO and Faster R-CNN) for real-time object detection to ensure high accuracy of status extraction. The extracted sorting status parameters (such as object type, sorting status, and timestamp) are recorded in a database for subsequent analysis. Quantitative indicators of sorting efficiency are defined, such as the number of objects sorted per hour (throughput), the correct sorting rate (number of correctly sorted objects / total number of sorted objects), and the sorting time (average sorting time per object). Use analytical tools (such as Python's Pandas and NumPy) to perform statistical analysis on the sorting status parameters, calculate sorting efficiency indicators, and generate a report containing detailed data and charts for each indicator. Based on the calculation results, a quantitative evaluation value for sorting efficiency is generated, forming a standardized evaluation report to facilitate subsequent comparison and optimization.

[0156] The quantitative evaluation values ​​of sorting efficiency are organized in chronological order to form a time series data set to facilitate the analysis of trend changes. This ensures consistent data format and accurate timestamps for subsequent analysis. Time series analysis methods (such as moving average and exponential smoothing) are used to perform trend analysis on sorting efficiency data, identify potential patterns and cyclical changes in efficiency, and identify factors affecting sorting efficiency. Characteristic data on the evolution of sorting efficiency trends is generated, including the time points when efficiency increases or decreases, the amplitude and frequency of efficiency fluctuations, and visualization tools (such as Matplotlib and Seaborn) are used to draw sorting efficiency trend charts to demonstrate the dynamic characteristics of efficiency changes.

[0157] In this embodiment, the specific steps of step S6 are:

[0158] Step S61: performing dynamic sorting path planning optimization on the sorting efficiency trend evolution characteristics to obtain a dynamic sorting optimization path;

[0159] Step S62: Optimizing the entire sorting process of the fine-tuning action sequence based on the dynamic sorting optimization path to generate a global optimized action sequence;

[0160] Step S63: Adaptive reinforcement transfer learning is performed on the global optimization action sequence to build an adaptive intelligent sorting model to perform automatic solid waste sorting operations.

[0161] In this embodiment, sorting efficiency trend evolution characteristic data is collected, including sorting efficiency indicators and sorting path information in different time periods. Key influencing factors, such as object type, sorting location, and equipment status, are identified. A path planning algorithm (such as the A* algorithm, Dijkstra algorithm, or RRT algorithm) is used to construct a preliminary sorting path model. The sorting area is divided into multiple areas, and the starting and target locations of the objects are taken into consideration to ensure that the path is as short and efficient as possible. The path planning model is adjusted in real time based on the sorting efficiency trend characteristics. For example, if the sorting efficiency of a certain path decreases, the system will automatically select another more optimal path. Optimization techniques (such as genetic algorithms and particle swarm optimization) are used to iteratively optimize the dynamic path to ensure the adaptability of path selection under different conditions. A dynamic sorting optimized path is generated and recorded in a database for subsequent reference and application. The optimized path is displayed using visualization tools to ensure that the operator can intuitively understand it.

[0162] Clarify the goals of full-process optimization, such as improving overall sorting efficiency, reducing energy consumption, and shortening sorting time. Determine the action sequences that need to be optimized, including grasping, moving, placing, etc. Based on the dynamic sorting optimization path, analyze and fine-tune the execution order and timing of each action in the action sequence. Apply scheduling algorithms (such as greedy algorithms and dynamic programming) to globally optimize the action sequence to ensure the coordination and coherence between each action, generate a globally optimized action sequence, record the parameters of each action (such as execution time, force, position, etc.), and ensure the reasonable connection between actions. Store the globally optimized action sequence in the control system for call in actual operation, determine the framework of the adaptive intelligent sorting model, select a suitable reinforcement learning algorithm (such as Deep Q-Learning, Proximal Policy Optimization), and design the state space, action space, and reward mechanism so that the model can learn how to make the best decision in the sorting task.

[0163] The global optimization action sequence is used as training data, and the model is pre-trained using transfer learning methods (such as fine-tuning). The training model adapts to the new environment and quickly adjusts the strategy to deal with unprecedented sorting situations. The model is trained in a simulated environment, and the strategy is optimized through a continuous trial-and-error process. The sorting effect of each time is recorded, and the model performance is evaluated regularly. The validation set is used to test the model's performance in real sorting scenarios to ensure its high accuracy and efficiency. The trained adaptive intelligent sorting model is deployed to the control system of the automatic sorting equipment for on-site testing. The performance of the model in actual sorting operations is monitored in real time to ensure that it can automatically adjust its strategy to adapt to different sorting tasks.

[0164] In this embodiment, a device for automatically sorting solid waste using machine vision is provided, which is used to perform the method for automatically sorting solid waste using machine vision, including:

[0165] The geometric perspective correction module is used to obtain monitoring images of the area to be sorted in multiple directions; perform spatial geometric perspective deviation correction on the monitoring images of the area to be sorted in multiple directions, and perform spatial panoramic image optimization to construct a panoramic image of the sorting area;

[0166] The 3D morphological structure module is used to perform frame-by-frame visual positioning scanning of solid waste in the panoramic image of the sorting area and perform 3D morphological structure analysis to generate a 3D morphological structure representation of each solid waste;

[0167] A code mapping module is used to make solid type classification decisions based on the three-dimensional morphological structure representation of each solid waste, and perform bounding box code mapping to obtain a code mapping bounding box for each solid waste;

[0168] The action parameter fine-tuning module is used to simulate the equipment action sequence for the coded mapping boundary box of each solid waste, and then fine-tune the action parameters to construct a fine-tuned action sequence;

[0169] The sorting efficiency trend module is used to perform instant automatic sorting based on fine-tuning action sequences and collect sorting images at multiple time points. It also analyzes the temporal sorting efficiency trend evolution of the multi-time point sorting images and generates sorting efficiency trend evolution features.

[0170] The sorting path optimization module is used to perform dynamic sorting path planning optimization and adaptive reinforcement transfer learning based on the sorting efficiency trend evolution characteristics, thereby building an adaptive intelligent sorting model to perform automatic solid waste sorting operations.

[0171] The present invention obtains monitoring images of the area to be sorted in multiple directions and performs geometric perspective deviation correction, which helps to eliminate perspective distortion in the image and improve sorting accuracy. It constructs a panoramic image of the sorting area to provide a more comprehensive field of view for subsequent solid waste identification and sorting, and increases the global perception ability of the system. Through frame-by-frame solid waste visual positioning scanning and three-dimensional morphological structure analysis, the morphological characteristics of each solid waste are accurately obtained, providing important data support for subsequent classification and sorting. The generation of three-dimensional morphological structure representation helps the system to accurately identify and classify solid waste, improve sorting accuracy and efficiency, and make classification decisions and bounding box coding mapping based on the three-dimensional morphological structure representation of solid waste to achieve accurate solid type classification and position marking. The coding mapping bounding box provides accurate Accurate location information provides an important basis for subsequent processing and sorting. Through equipment action sequence simulation and action parameter fine-tuning, the solid waste processing action is optimized to improve the accuracy and efficiency of sorting. The construction of fine-tuning action sequence helps to ensure that each solid waste is correctly processed and reduce processing errors and losses. Analysis of the evolution characteristics of time-series sorting efficiency trends helps the system understand the changing rules of sorting efficiency, optimize sorting strategies and improve overall efficiency. Through instant automatic sorting and analysis of efficiency trends, the system adjusts strategies in real time to improve sorting efficiency and accuracy. Dynamic sorting path planning optimization and adaptive reinforcement transfer learning help to build intelligent sorting models and improve system adaptability and performance. The adaptive intelligent sorting model adjusts the sorting path according to real-time conditions to improve the efficiency and accuracy of automatic sorting of solid waste.

[0172] The present invention is therefore intended to be illustrative and non-restrictive in all respects, with the scope of the invention being defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the application documents are intended to be embraced therein.

[0173] The foregoing description is intended only to provide specific embodiments of the present invention, which are intended to enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for automatic sorting of solid waste using machine vision, characterized in that: The following steps are involved: Step S1: Acquire monitoring images of the area to be sorted in multiple directions; Correct the spatial geometric perspective deviation of the monitoring images of the sorting area in multiple directions, optimize the spatial panoramic image, and construct a panoramic image of the sorting area; Step S2: performing visual positioning scanning of solid waste frame by frame on the panoramic image of the sorting area, and performing three-dimensional morphological structure analysis, thereby generating a three-dimensional morphological structure representation of each solid waste; Step S3: making a solid type classification decision based on the three-dimensional morphological structure representation of each solid waste, and performing bounding box coding mapping to obtain a coding mapping bounding box of each solid waste; Step S4: simulate the equipment action sequence for each coded mapping boundary box of solid waste, and then fine-tune the action parameters to construct a fine-tuned action sequence; Step S5: Performing instant automatic sorting based on the fine-tuning action sequence, and collecting sorting images at multiple time points; Perform temporal sorting efficiency trend evolution on multi-time point sorting images to generate sorting efficiency trend evolution features; Step S6: Dynamic sorting path planning optimization and adaptive reinforcement transfer learning are performed on the sorting efficiency trend evolution characteristics, thereby building an adaptive intelligent sorting model to perform automatic solid waste sorting operations; The specific steps of step S4 are: Step S41: performing equipment action sequence simulation on the code mapping boundary box of each solid waste to obtain action sequence simulation data of the automatic sorting equipment; Step S42: Dynamically sorting movement tracking is performed on the action sequence simulation data of the automatic sorting device to extract the dynamic sorting movement trajectory; Step S43: Analyze the solid waste position change of the dynamic sorting movement trajectory to generate solid waste position change data; Step S44: Calculating the equipment sorting accuracy for the solid waste position change data to obtain the equipment sorting accuracy; Step S45: identifying the action sequence deviation according to the device sorting accuracy to obtain the action sequence deviation value; Step S46: performing deviation compensation calculation on the action sequence deviation value to generate action sequence deviation compensation data; Step S47: fine-tuning the action parameters of the action sequence simulation data of the automatic sorting device based on the action sequence deviation compensation data to construct a fine-tuning action sequence; The specific steps of step S5 are: Step S51: performing instant automatic sorting based on the fine-tuning action sequence, and collecting sorting images at multiple time points; Step S52: performing sorting status analysis on the multi-time point sorting images to extract the sorting status parameters of the automation equipment; Step S53: performing a quantitative evaluation of the sorting efficiency of the automated equipment sorting state parameters to generate a quantitative evaluation value of the sorting efficiency; Step S54: performing time-series sorting efficiency trend evolution on the quantitative evaluation value of the sorting efficiency to generate sorting efficiency trend evolution characteristics; the sorting efficiency trend evolution characteristics specifically include: time nodes when the efficiency increases or decreases, and the amplitude and frequency of efficiency fluctuations.

2. The method for automatic sorting of solid waste using machine vision according to claim 1, characterized in that: The specific steps of step S1 are: Step S11: Acquire monitoring images of the area to be sorted in multiple directions based on a high-resolution panoramic camera; Step S12: Optimizing the image contrast of the monitoring images of the to-be-sorted areas in multiple directions to construct a multi-directional contrast-balanced monitoring image; Step S13: performing image feature point recognition on the multi-directional contrast equalization monitoring image, and marking a plurality of image feature points; Step S14: performing multi-directional registration on multiple image feature points, thereby obtaining multi-angle matching data for each feature point; Step S15: performing spatial geometric perspective deviation correction based on the multi-angle matching data of each feature point, thereby obtaining a spatial geometric perspective deviation corrected image; Step S16: performing spatial panoramic image optimization on the spatial geometric perspective deviation correction image to construct a panoramic image of the sorting area.

3. The method for automatic sorting of solid waste using machine vision according to claim 2, characterized in that: The specific steps of step S12 are: Performing image region division on monitoring images of the to-be-sorted areas in multiple directions to obtain multiple image regions; Calculating original grayscale pixel values ​​of multiple image regions to generate original grayscale pixel values ​​of each region; Perform global average calculation on the original grayscale pixel value of each area to obtain the average grayscale pixel value of the image; Perform pixel linear stretching enhancement processing on the average grayscale pixel value of the image to generate an enhanced grayscale pixel value; Stretching pixels in a plurality of image regions according to enhanced grayscale pixel values ​​to generate a brightness enhanced monitoring image; Perform grayscale pixel frequency distribution analysis based on the original grayscale pixel value of each region to generate image grayscale pixel frequency distribution data; Performing cumulative distribution function calculation on the image grayscale pixel frequency distribution data to extract the cumulative distribution function of the image; Perform pixel contrast optimization processing on the cumulative distribution function of the image to obtain a contrast optimized cumulative distribution function; The contrast optimization cumulative distribution function is used to perform contrast equalization on the brightness enhanced monitoring image, thereby constructing a multi-directional contrast balanced monitoring image.

4. The method for automatic sorting of solid waste using machine vision according to claim 1, characterized in that: The specific steps of step S2 are: Step S21: performing a solid waste visual positioning scan frame by frame on the panoramic image of the sorting area, and marking each solid waste positioning data; Step S22: identifying the boundary contour of the solid waste according to each solid waste location data, and extracting the boundary contour line of each solid waste; Step S23: fine-grained segmentation of the panoramic image of the sorting area is performed based on the boundary contour line of each solid waste to generate an image boundary frame of each solid waste; Step S24: performing object geometry analysis on each solid waste image boundary frame to generate solid waste geometry features; Step S25: performing object texture detail recognition on each solid waste image boundary frame to obtain solid waste texture detail data; Step S26: Performing a three-dimensional morphological structure analysis on the solid waste texture detail data and the solid waste geometric morphological features, thereby generating a three-dimensional morphological structure representation of each solid waste.

5. The method for automatic sorting of solid waste using machine vision according to claim 1, characterized in that: The specific steps of step S3 are: Step S31: making solid type classification decisions based on the three-dimensional morphological structure representation of each solid waste, and obtaining type data of each solid waste; Step S32: performing sorting sequence analysis on the type data of each solid waste to generate sorting sequence data of each solid waste; Step S33: performing sorting sequence coding on the sorting sequence data of each solid waste to obtain a solid waste sequence code; Step S34: performing bounding box coding mapping on the image bounding box of each solid waste based on the solid waste sequential coding to obtain a coded mapping bounding box of each solid waste.

6. The method for automatic sorting of solid waste using machine vision according to claim 1, characterized in that: The specific steps of step S6 are: Step S61: performing dynamic sorting path planning optimization on the sorting efficiency trend evolution characteristics to obtain a dynamic sorting optimization path; Step S62: optimizing the whole sorting process of the fine-tuning action sequence based on the dynamic sorting optimization path to generate a global optimization action sequence; Step S63: Perform adaptive reinforcement transfer learning on the global optimization action sequence to build an adaptive intelligent sorting model to perform automatic sorting operations for solid waste.

7. An automatic solid waste sorting device using machine vision, characterized in that: The method for automatically sorting solid waste using machine vision as claimed in claim 1 comprises: The geometric perspective correction module is used to obtain monitoring images of the area to be sorted in multiple directions; perform spatial geometric perspective deviation correction on the monitoring images of the area to be sorted in multiple directions, and perform spatial panoramic image optimization to construct a panoramic image of the sorting area; The three-dimensional morphological structure module is used to perform visual positioning scanning of solid waste frame by frame in the panoramic image of the sorting area and perform three-dimensional morphological structure analysis to generate a three-dimensional morphological structure representation of each solid waste; A coding mapping module is used to make solid type classification decisions based on the three-dimensional morphological structure representation of each solid waste, and perform bounding box coding mapping to obtain a coding mapping bounding box for each solid waste; The action parameter fine-tuning module is used to simulate the equipment action sequence for each solid waste coding mapping boundary box, and then fine-tune the action parameters to construct a fine-tuning action sequence; The sorting efficiency trend module is used to perform instant automatic sorting based on fine-tuning action sequences and collect sorting images at multiple time points; it performs temporal sorting efficiency trend evolution on the sorting images at multiple time points and generates sorting efficiency trend evolution features; The sorting path optimization module is used to perform dynamic sorting path planning optimization and adaptive reinforcement transfer learning on the sorting efficiency trend evolution characteristics, so as to build an adaptive intelligent sorting model to perform automatic solid waste sorting operations.

Images