Garbage sorting system and sorting method based on vision

Abstract

The invention provides a garbage sorting system and sorting method based on vision, and relates to the technical field of garbage sorting systems. According to the method, a reasonable initial global segmentation threshold is set based on the image with the highest definition and the largest capture range, dynamic adjustment is carried out according to reflection characteristics, secondary image information and local contrast characteristics are fully utilized, a segmentation result is further optimized through an adjustment strategy of setting a local segmentation threshold, and the segmentation accuracy is improved. And finally, the global and local segmentation results are fused to obtain accurate and complete reflective area information, so that the garbage sorting efficiency and accuracy are remarkably improved, efficient and reliable technical support is provided for garbage classification and resource recovery, and the garbage sorting efficiency and the garbage sorting accuracy are improved conveniently according to the reflective characteristics and local contrast characteristics of different garbage. And the segmentation threshold value is dynamically adjusted so as to adapt to garbage identification requirements of different types and different states.

Description

Technical Field

[0001] The present invention relates to the technical field of garbage sorting systems, and in particular to a garbage sorting system and a sorting method based on vision. Background Art

[0002] In the current field of garbage sorting, with the enhancement of environmental awareness and the increase in demand for resource recycling, automated and intelligent garbage sorting technology has become a research hotspot. Among them, the existence of reflective areas has an important impact on the accuracy and efficiency of garbage sorting. The reflective areas often contain important characteristic information of garbage, such as material, color, etc. This information is crucial for subsequent classification and resource recycling.

[0003] Traditional garbage sorting systems and methods often have many shortcomings when dealing with reflective areas. For example, segmentation methods based on fixed thresholds are difficult to adapt to changes in different lighting conditions and reflective properties of different materials, resulting in inaccurate segmentation results; and segmentation methods based on complex algorithms can improve segmentation accuracy to a certain extent, but are often accompanied by problems such as large amount of calculation and slow processing speed, making it difficult to meet real-time requirements. How to achieve accurate recognition and segmentation of reflective areas and improve the accuracy and efficiency of garbage sorting has become a technical problem that needs to be solved in the current garbage sorting field. Therefore, it is necessary to provide a vision-based garbage sorting system and sorting method to solve the above technical problems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a vision-based garbage sorting system and sorting method for solving the problems of poor applicability, large amount of calculation, slow processing speed and inaccurate segmentation results of the existing garbage sorting system based on fixed threshold segmentation method.

[0005] The present invention provides a garbage sorting method based on vision, and the sorting method comprises the following steps: By setting up a three-dimensional visual information collection device and a three-dimensional laser scanning device in a triangular orientation, the three-dimensional image information, spatial position information and three-dimensional data of the garbage are obtained; Based on the obtained three-dimensional data of the garbage, the three-dimensional image information of the garbage is optimized to obtain the optimized three-dimensional image information; Select a single-azimuth image from the optimized three-azimuth image information as the main image information, capture the reflective features of garbage at different scales, and dynamically adjust the global segmentation threshold between the reflective area and the non-reflective area; Select the remaining two-position image information from the optimized three-position image information as the secondary image information, and dynamically adjust the local segmentation threshold of the reflective area and the non-reflective area according to the local contrast characteristics of the image; The reflective area and the non-reflective area are segmented by adjusting the global segmentation threshold and the local segmentation threshold, and the reflective features are extracted; Match the extracted reflective features with the feature database to identify the type of garbage; According to the identified garbage type, the spatial location information of the garbage that needs to be sorted is fed back to the control center to generate a control signal, which is then sent to the robotic arm to perform garbage grabbing operations.

[0006] Preferably, the method of selecting a single orientation image information from the optimized three orientation image information as the main image information, capturing the reflective features of garbage at different scales, and dynamically adjusting the global segmentation threshold of the reflective area and the non-reflective area, specifically comprises the following steps: From the three-dimensional image information of the optimized garbage, select an image with the highest definition and the largest capture range as the main image and perform preprocessing; Based on the historical reflectivity detection data of garbage and the gray level of the main image, an initial global segmentation threshold is set; Setting a threshold adjustment strategy based on the reflective characteristics of the reflective area, specifically, if the brightness and area of ​​the reflective area exceed the preset adjustment threshold, then increasing the segmentation threshold, and if the brightness and area of ​​the reflective area do not exceed the preset adjustment threshold, then decreasing the segmentation threshold; Use computer vision technology combined with an initial global segmentation threshold to preliminarily segment the reflective areas in the image; The reflective features of garbage in the reflective area of ​​the preliminary segmented image are extracted, and the global segmentation threshold of the reflective area and the non-reflective area is dynamically adjusted through the set threshold adjustment strategy.

[0007] Preferably, the remaining two-position image information is selected from the optimized three-position image information as the secondary image information, and the local segmentation threshold of the reflective area and the non-reflective area is dynamically adjusted according to the local contrast characteristics of the image, and the specific steps are: Based on the complementarity of the main image information, the remaining two-position image information is selected from the optimized three-position image information as the secondary image information, and preprocessed; Setting the adjustment strategy of the local segmentation threshold, specifically including increasing the local segmentation threshold if the brightness difference between the garbage and the background exceeds the preset difference threshold; and decreasing the local segmentation threshold if the brightness difference between the garbage and the background does not exceed the preset difference threshold; Using image processing technology, extracting local contrast features of garbage from the pre-processed secondary image information, wherein the extracted local contrast features of garbage specifically include but are not limited to the sharpness of the edge of garbage and the brightness difference between garbage and background; Analyze the changes of local contrast features in the secondary image information, including the intensity, distribution and change trend of the features, and obtain analysis results; According to the obtained analysis results, the local segmentation thresholds of the reflective area and the non-reflective area are dynamically adjusted by setting the adjustment strategy of the local segmentation threshold.

[0008] Preferably, the reflective area and the non-reflective area are segmented by adjusting the global segmentation threshold and the local segmentation threshold, and the reflective feature is extracted, and the specific steps are: According to the dynamically adjusted global segmentation threshold of the reflective area and the non-reflective area, the main image information is globally segmented into the reflective area and the non-reflective area to obtain a global segmentation result; According to the dynamically adjusted local segmentation thresholds of the reflective area and the non-reflective area, segmentation adjustment is performed on the local area of ​​the main image information to obtain a local segmentation adjustment result; The global segmentation result and the local segmentation adjustment result are merged to obtain the final segmentation result of the reflective area and the non-reflective area; In the final segmentation result of the reflective area and the non-reflective area, the reflective features of the garbage are extracted through the final reflective area, including but not limited to the reflection intensity, the reflective area and the shape of the reflective area.

[0009] Preferably, the extracted reflective features are matched with a feature database to identify the type of garbage, and the specific steps are: The garbage reflective features extracted from the final reflective area are sorted to form a feature vector; The feature vector is matched with the garbage type feature in the database to determine whether the matching degree exceeds the preset garbage type feature matching degree threshold, and the features exceeding the preset garbage type feature matching degree threshold are identified as the corresponding garbage type to obtain the garbage type identification result.

[0010] Preferably, according to the identified garbage types, the spatial position information of the garbage to be sorted is fed back to the control center to generate a control signal, and sent to the mechanical arm to perform the garbage grabbing operation, and the specific steps are: Receive the identification result of the garbage type and determine its spatial location information, wherein the spatial location information specifically includes the coordinates, direction and size of the garbage in three-dimensional space; The spatial position information is converted from the image coordinate system to the working coordinate system of the robot arm, and a target grasping position signal is generated and sent to the robot arm through a communication connection; The robotic arm receives the target grabbing position signal and moves according to the existing programmed motion instructions to grab and sort the target garbage.

[0011] The specific steps of optimizing the three-dimensional image information of the garbage based on the obtained three-dimensional data of the garbage are as follows: Preprocess the three-dimensional data of garbage, including denoising, filtering and registration; The preprocessed three-dimensional data is mapped to the three-dimensional image information through coordinate transformation for registration. Specifically, each point X, Y, Z in the preprocessed three-dimensional data is mapped to the pixel u, v on the image plane. The coordinate transformation expression is as follows: , where P is the projection matrix; Depth information is extracted from the preprocessed three-dimensional data, and the depth information is superimposed as an additional channel of the three-dimensional image information to obtain the three-dimensional image information of the optimized garbage.

[0012] A garbage sorting system based on vision, the sorting system comprising: The visual information collection module is used to obtain three-dimensional image information, spatial position information and three-dimensional data of garbage by setting a three-dimensional visual information collection device and a three-dimensional laser scanning device in a triangular orientation; An image optimization module, used for optimizing the three-dimensional image information of the garbage based on the obtained three-dimensional data of the garbage, and obtaining the optimized three-dimensional image information; A global segmentation threshold adjustment module is used to select a single-direction image information from the three-direction image information as the main image information, capture the reflective features of garbage at different scales, and dynamically adjust the global segmentation threshold of the reflective area and the non-reflective area; A local segmentation threshold adjustment module is used to select the remaining two-position image information from the optimized three-position image information as the secondary image information, and dynamically adjust the local segmentation threshold of the reflective area and the non-reflective area according to the local contrast characteristics of the image; A segmentation and extraction module, used to segment the reflective area and the non-reflective area through the adjusted global segmentation threshold and the local segmentation threshold, and extract reflective features; A category matching module is used to match the extracted reflective features with the feature database to identify the type of garbage; The information feedback module is used to feed back the spatial location information of the garbage to be sorted to the control center according to the identified garbage type, generate a control signal, and send it to the robotic arm to perform garbage grabbing operations.

[0013] Compared with the related art, the vision-based garbage sorting system and sorting method provided by the present invention have the following beneficial effects: The present invention sets a reasonable initial global segmentation threshold based on the image with the highest clarity and the largest capture range, and dynamically adjusts it according to the reflective features. At the same time, it makes full use of the secondary image information and local contrast features, and further optimizes the segmentation result by setting the adjustment strategy of the local segmentation threshold. Finally, the global and local segmentation results are integrated to obtain accurate and complete reflective area information, thereby significantly improving the efficiency and accuracy of garbage sorting, providing efficient and reliable technical support for garbage classification and resource recycling, and facilitating the dynamic adjustment of the segmentation threshold according to the reflective characteristics and local contrast characteristics of different garbage, so as to adapt to the needs of garbage identification of different types and states. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart of a visual-based garbage sorting method provided by an embodiment of the present invention; Figure 2 A system block diagram of a vision-based garbage sorting system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes. Example 1

[0016] like Figure 1 As shown, a garbage sorting method based on vision includes the following steps: S100, obtaining three-dimensional image information, spatial position information and three-dimensional data of garbage by setting a three-dimensional visual information collection device and a three-dimensional laser scanning device in a triangular orientation; Specifically, a three-directional visual information collection device is set up in a triangular orientation, including high-resolution cameras, and the cameras are placed in three different orientations of the garbage treatment area to form a triangular viewing angle coverage, so as to ensure that comprehensive information about the garbage is captured from multiple angles through its layout to reduce blind spots; when the garbage is thrown into the treatment area, the cameras in the three orientations will be started at the same time to start collecting image information of the garbage, including visual features such as the shape, color, and texture of the garbage. Secondly, the spatial position information of the garbage is collected by arranging sensors installed near the cameras. The sensors measure parameters such as the distance and angle between the garbage and the camera, thereby determining the specific position of the garbage in three-dimensional space, and collecting three-dimensional data of the garbage through three-dimensional laser scanning equipment.

[0017] S200, optimizing the three-dimensional image information of the garbage based on the obtained three-dimensional data of the garbage, and obtaining optimized three-dimensional image information; S201, preprocessing the three-dimensional data of garbage, specifically including denoising, filtering and registration; S202, mapping the preprocessed three-dimensional data to the three-dimensional image information by coordinate transformation for registration, specifically mapping each point X, Y, Z in the preprocessed three-dimensional data to pixels u, v on the image plane, wherein the coordinate transformation expression is specifically: , where P is the projection matrix; S203, extracting depth information from the preprocessed three-dimensional data, and superimposing the depth information as an additional channel of the three-dimensional image information to obtain optimized three-dimensional image information. The optimization method can make the image segmentation effect of the subsequent steps more accurate and refined, help reduce the situation of mis-segmentation and missed segmentation, and improve the quality and efficiency of image segmentation.

[0018] S300, selecting a single orientation image information from the optimized three orientation image information as the main image information, capturing the reflection characteristics of garbage at different scales, and dynamically adjusting the global segmentation threshold of the reflection area and the non-reflection area; In the specific implementation process, the specific implementation steps of step S300 are: S301, selecting an image with the highest definition and the largest capture range from the three-dimensional image information of the optimized garbage as a main image, and performing preprocessing; Specifically, from the three-dimensional image information of the optimized garbage, an image with the highest definition and the largest capture range is selected as the main image, and existing image processing tools are used to perform pre-processing operations such as denoising and contrast enhancement on the main image.

[0019] S302, setting an initial global segmentation threshold based on historical reflectivity detection data of garbage and the grayscale of the main image; Specifically, a large amount of reflective detection data on different types of garbage is collected, including reflective intensity information under different lighting conditions, different materials, and different angles. In this embodiment, based on the fact that metal garbage usually has a higher reflective intensity, while paper garbage has a weaker reflective intensity, the reflective characteristics of different types of garbage are determined; the grayscale of the current main image is calculated through the existing image processing algorithm, and an initial global segmentation threshold is set based on the historical reflective detection data and the calculated grayscale of the current main image.

[0020] In this embodiment, a batch of garbage to be sorted includes metal bottles, plastic bottles and cardboard. According to the historical reflectivity detection data, the reflectivity of the metal bottles is high, while the reflectivity of the cardboard is weak. By calculating the grayscale, it is found that the overall brightness of the image is moderate. Therefore, an initial global segmentation threshold is set, specifically: the threshold is slightly higher than the average grayscale value of the cardboard, but lower than the average grayscale value of the metal bottles.

[0021] S303, setting a threshold adjustment strategy based on the reflective characteristics of the reflective area, specifically including increasing the segmentation threshold if the reflective area is bright and has a large area (i.e., the brightness and area of ​​the reflective area exceed the preset threshold), and decreasing the segmentation threshold if the reflective area is dark and has a small area (i.e., the brightness and area of ​​the reflective area do not exceed the preset threshold); Specifically, in this embodiment, the brightness threshold is adjusted: if the average brightness of the reflective area 200 is judged to exceed the preset brightness threshold 150, the segmentation threshold is gradually increased; on the contrary, if the average brightness of the reflective area is judged to be lower than the preset threshold, the segmentation threshold is lowered; the area threshold is adjusted: if the area of ​​the reflective area (5000 pixels) is judged to account for a large proportion of the total image area (exceeding 10%), the segmentation threshold is increased; on the contrary, if the area of ​​the reflective area is judged to account for a small proportion, the segmentation threshold is lowered.

[0022] S304, using computer vision technology combined with an initial global segmentation threshold to preliminarily segment the reflective area in the image; Specifically, according to the set initial global segmentation threshold, the threshold segmentation and edge detection methods of the image segmentation algorithm in the existing computer vision technology are applied, and the image is preliminarily segmented according to the global segmentation threshold. In this embodiment, for a main image containing a metal can, a plastic bottle and a cardboard, according to the set initial global segmentation threshold (grayscale value is 150), the pixels in the image are divided into reflective areas (grayscale value is greater than 150) and non-reflective areas (grayscale value is less than 150). Through preliminary segmentation, the reflective area of ​​the metal can and the non-reflective areas of the cardboard and plastic bottle can be roughly distinguished.

[0023] S305, extracting the reflective features of the garbage in the reflective area of ​​the preliminary segmented image, and dynamically adjusting the global segmentation threshold of the reflective area and the non-reflective area through a set threshold adjustment strategy; Specifically, based on the preliminary segmentation, the reflective features of the garbage in the reflective area are extracted, including the brightness, area, shape and color of the reflective area; and the threshold adjustment strategy is set according to the reflective features of the reflective area. If the reflective area is bright and has a large area (that is, the brightness and area of ​​the reflective area exceed the preset threshold), the segmentation threshold is increased; if the reflective area is dark and has a small area (that is, the brightness and area of ​​the reflective area do not exceed the preset threshold), the segmentation threshold is lowered, and the global segmentation threshold of the reflective area and the non-reflective area is adjusted in stages; In this embodiment, the brightness threshold is adjusted: if the average brightness of the reflective area 200 is judged to exceed the preset brightness threshold 150, the segmentation threshold is gradually increased; on the contrary, if the average brightness of the reflective area is judged to be lower than the preset threshold, the segmentation threshold is lowered; the area threshold is adjusted: if the area of ​​the reflective area (5000 pixels) accounts for a large proportion of the total image area (exceeding 10%), the segmentation threshold is increased; on the contrary, if the area of ​​the reflective area is judged to account for a small proportion, the segmentation threshold is lowered for dynamic adjustment.

[0024] S400, selecting the remaining two-position image information from the three-position image information as the secondary image information, and dynamically adjusting the local segmentation threshold of the reflective area and the non-reflective area according to the local contrast characteristics of the image; In the specific implementation process, the specific steps of step S400 are: S401, selecting the remaining two-position image information from the three-position image information as secondary image information based on the complementarity of the primary image information, and performing preprocessing; Specifically, based on the complementarity of the main image information, images in two other directions are selected from the image information of the left and right sides in different directions as secondary image information to supplement the missing or unclear parts of the main image information, and existing image processing technology is used to perform pre-processing operations such as denoising and contrast enhancement on the secondary image to improve image quality.

[0025] S402, setting an adjustment strategy for the local segmentation threshold, specifically including increasing the local segmentation threshold if the local contrast feature is strong (i.e., the brightness difference between the garbage and the background exceeds a preset difference threshold); and decreasing the local segmentation threshold if the local contrast feature is weak (i.e., the brightness difference between the garbage and the background does not exceed a preset difference threshold); Specifically, in this embodiment, in a secondary image, i.e., the side image, the brightness difference between the metal object in the garbage and the background is 60 units, which exceeds the difference threshold, so the local segmentation threshold is increased, specifically by 10-15 values, to more accurately segment the area of ​​the metal object, while the brightness difference between other positions of the metal object and the background is only 30 units, which does not exceed the difference threshold, so the local segmentation threshold is lowered, specifically by 10-15 values, to reduce the possibility of mis-segmentation.

[0026] S403, using image processing technology to extract local contrast features of the garbage from the pre-processed secondary image information, wherein the extracted local contrast features of the garbage specifically include but are not limited to the sharpness of the edge of the garbage and the brightness difference between the garbage and the background; Specifically, edge detection using image processing technology is used to extract local contrast features of garbage from the pre-processed secondary image information, including but not limited to the sharpness of the garbage edge and the brightness difference between the garbage and the background.

[0027] S404, analyzing the change of the local contrast feature in the secondary image information, specifically including the intensity, distribution and change trend of the feature, to obtain an analysis result; Specifically, a histogram analysis method is used to identify the distribution change of brightness differences in a histogram, determine the change trend of local contrast features, and obtain an analysis result.

[0028] S405, dynamically adjusting the local segmentation thresholds of the reflective area and the non-reflective area by setting an adjustment strategy for the local segmentation threshold according to the obtained analysis results; Specifically, in this embodiment, if the edge sharpness value of the metal object is high and the brightness difference is obvious, the local segmentation threshold of the area will be increased (from 130 to 140), while the top of the metal can is highly reflective and the edge sharpness value is reduced, so the local segmentation threshold of the area will be reduced (from 110 to 100).

[0029] S500, segmenting the reflective area and the non-reflective area by using the adjusted global segmentation threshold and the local segmentation threshold, and extracting reflective features; In the specific implementation process, the specific steps of step S500 are: S501, globally segmenting the main image information into reflective areas and non-reflective areas according to the dynamically adjusted global segmentation threshold of the reflective areas and non-reflective areas, to obtain a global segmentation result; Specifically, according to the dynamically adjusted global segmentation threshold of the reflective area and the non-reflective area, the main image information is globally segmented into the reflective area and the non-reflective area by using an edge segmentation algorithm in the image processing technology.

[0030] S502, performing segmentation adjustment on the local area of ​​the main image information according to the dynamically adjusted local segmentation thresholds of the reflective area and the non-reflective area to obtain a local segmentation adjustment result; Specifically, in this embodiment, since the edge sharpness value of the metal object is relatively high and the brightness difference is obvious, the local segmentation threshold of the area will be increased (from 130 to 140), and since the top of the metal can is highly reflective and the edge sharpness value is reduced, the local segmentation threshold of the area will be reduced (from 110 to 100), and the edge segmentation algorithm in the image processing technology is used to segment and adjust the local area of ​​the main image information.

[0031] S503, fusing the global segmentation result and the local segmentation adjustment result to obtain the final segmentation result of the reflective area and the non-reflective area; Specifically, the global segmentation result and the local segmentation adjustment result are fused. Specifically, during the fusion process, each pixel in the global segmentation result and the local segmentation adjustment result are compared one by one. If a pixel is classified as a reflective area or a non-reflective area in both results, the classification result is retained; if a pixel is classified differently in the two results, it is selected and integrated based on the priority given to the local segmentation adjustment result.

[0032] S504: In the final segmentation result of the reflective area and the non-reflective area, the reflective characteristics of the garbage are extracted through the final reflective area, including but not limited to the reflective intensity, the reflective area and the shape of the reflective area.

[0033] S600, matching the extracted reflective features with a feature database to identify the type of garbage; In the specific implementation process, the specific steps of step S600 are: S601, sorting out the garbage reflective features extracted from the final reflective area to form a feature vector; Specifically, the reflective features extracted from the final reflective area are collected and organized as follows: Reflection intensity: 0.8 (range 0-1, indicating the brightness of the reflection) Reflection color: RGB (255, 200, 100) (indicates a yellowish color) Reflection distribution mode: Uniform distribution (meaning that the reflections are evenly distributed over the entire image).

[0034] S602, matching the feature vector with the garbage type feature in the database, determining whether the matching degree exceeds a preset garbage type feature matching degree threshold, identifying the feature exceeding the preset garbage type feature matching degree threshold as the corresponding garbage type, and obtaining a garbage type identification result; Specifically, in this embodiment, the preset garbage type feature matching degree threshold is 0.8. The feature vector is matched with the garbage type feature in the database to determine whether the matching degree exceeds the preset garbage type feature matching degree threshold. The features that exceed the preset garbage type feature matching degree threshold are identified as the corresponding garbage types.

[0035] For example, for plastic bottles: [0.75, RGB (255, 220, 150), partial reflection], for glass bottles: [0.9, RGB (200, 200, 200), uniform reflection], the cosine similarity between the feature vector and the feature vector of the plastic bottle is 0.85, and the cosine similarity between the feature vector and the feature vector of the glass bottle is 0.70. According to the preset garbage type feature matching threshold of 0.8, it can be obtained that the matching degree between the feature vector and the plastic bottle exceeds the threshold, and therefore the garbage is identified as a plastic bottle.

[0036] S700: According to the identified garbage types, the spatial location information of the garbage to be sorted is fed back to the control center to generate a control signal, and then sent to the robot arm to perform garbage grabbing operations; In the specific implementation process, the specific steps of step S700 are: S701, receiving the identification result of the garbage type and determining its spatial location information, wherein the spatial location information specifically includes the coordinates, direction and size of the garbage in three-dimensional space; S702, converting the spatial position information from the image coordinate system into the working coordinate system of the robot arm, and generating a target grasping position signal and sending it to the robot arm through a communication connection; S703, the robot arm receives the target grabbing position signal, and moves according to the existing programmed motion instructions to grab and sort the target garbage; Specifically, the robot arm receives the target grasping position signal sent from step S702, which contains the precise position information of the garbage in the robot arm's working coordinate system. The robot arm starts to perform the grasping operation based on this information and its built-in programmed motion instructions. Example 2

[0037] like Figure 2 As shown, a vision-based garbage sorting system applied to a vision-based garbage sorting method specifically includes: The visual information collection module is used to obtain three-dimensional image information, spatial position information and three-dimensional data of garbage by setting a three-dimensional visual information collection device and a three-dimensional laser scanning device in a triangular orientation; An image optimization module, used for optimizing the three-dimensional image information of the garbage based on the obtained three-dimensional data of the garbage, and obtaining the optimized three-dimensional image information; A global segmentation threshold adjustment module is used to select a single-direction image information from the three-direction image information as the main image information, capture the reflective features of garbage at different scales, and dynamically adjust the global segmentation threshold of the reflective area and the non-reflective area; A local segmentation threshold adjustment module is used to select the remaining two-position image information from the optimized three-position image information as the secondary image information, and dynamically adjust the local segmentation threshold of the reflective area and the non-reflective area according to the local contrast characteristics of the image; A segmentation and extraction module, used to segment the reflective area and the non-reflective area through the adjusted global segmentation threshold and the local segmentation threshold, and extract reflective features; A category matching module is used to match the extracted reflective features with the feature database to identify the type of garbage; The information feedback module is used to feed back the spatial location information of the garbage to be sorted to the control center according to the identified garbage type, generate a control signal, and send it to the robotic arm to perform garbage grabbing operations.

[0038] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0039] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0040] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A garbage sorting method based on vision, characterized in that: The sorting method comprises the following steps: By setting up a three-dimensional visual information collection device and a three-dimensional laser scanning device in a triangular orientation, the three-dimensional image information, spatial position information and three-dimensional data of the garbage are obtained; Based on the obtained three-dimensional data of the garbage, the three-dimensional image information of the garbage is optimized to obtain the optimized three-dimensional image information; Select a single-azimuth image from the optimized three-azimuth image information as the main image information, capture the reflective features of garbage at different scales, and dynamically adjust the global segmentation threshold between the reflective area and the non-reflective area; Select the remaining two-position image information from the optimized three-position image information as the secondary image information, and dynamically adjust the local segmentation threshold of the reflective area and the non-reflective area according to the local contrast characteristics of the image; The reflective area and the non-reflective area are segmented by adjusting the global segmentation threshold and the local segmentation threshold, and the reflective features are extracted; Match the extracted reflective features with the feature database to identify the type of garbage; According to the identified garbage type, the spatial location information of the garbage that needs to be sorted is fed back to the control center to generate a control signal, which is then sent to the robotic arm to perform garbage grabbing operations.

2. A visual-based garbage sorting method according to claim 1, characterized in that: The specific steps of selecting a single orientation image information from the optimized three orientation image information as the main image information, capturing the garbage reflective features of different scales, and dynamically adjusting the global segmentation threshold of the reflective area and the non-reflective area are as follows: From the three-dimensional image information of the optimized garbage, select an image with the highest definition and the largest capture range as the main image and perform preprocessing; Based on the historical reflectivity detection data of garbage and the gray level of the main image, an initial global segmentation threshold is set; Setting a threshold adjustment strategy based on the reflective characteristics of the reflective area, specifically, if the brightness and area of ​​the reflective area exceed the preset adjustment threshold, then increasing the segmentation threshold, and if the brightness and area of ​​the reflective area do not exceed the preset adjustment threshold, then decreasing the segmentation threshold; Use computer vision technology combined with an initial global segmentation threshold to preliminarily segment the reflective areas in the image; The reflective features of garbage in the reflective area of ​​the preliminary segmented image are extracted, and the global segmentation threshold of the reflective area and the non-reflective area is dynamically adjusted through the set threshold adjustment strategy.

3. A visual-based garbage sorting method according to claim 1, characterized in that: The specific steps of selecting the remaining two-position image information from the optimized three-position image information as the secondary image information and dynamically adjusting the local segmentation threshold of the reflective area and the non-reflective area according to the local contrast characteristics of the image are as follows: Based on the complementarity of the main image information, the remaining two-position image information is selected from the optimized three-position image information as the secondary image information, and preprocessed; Setting the adjustment strategy of the local segmentation threshold, specifically including increasing the local segmentation threshold if the brightness difference between the garbage and the background exceeds the preset difference threshold; and decreasing the local segmentation threshold if the brightness difference between the garbage and the background does not exceed the preset difference threshold; Using image processing technology, extracting local contrast features of garbage from the pre-processed secondary image information, wherein the extracted local contrast features of garbage specifically include but are not limited to the sharpness of the edge of garbage and the brightness difference between garbage and background; Analyze the changes of local contrast features in the secondary image information, including the intensity, distribution and change trend of the features, and obtain analysis results; According to the obtained analysis results, the local segmentation thresholds of the reflective area and the non-reflective area are dynamically adjusted by setting the adjustment strategy of the local segmentation threshold.

4. A visual-based garbage sorting method according to claim 1, characterized in that: The reflective area and the non-reflective area are segmented by adjusting the global segmentation threshold and the local segmentation threshold, and the reflective features are extracted. The specific steps are as follows: According to the dynamically adjusted global segmentation threshold of the reflective area and the non-reflective area, the main image information is globally segmented into the reflective area and the non-reflective area to obtain a global segmentation result; According to the dynamically adjusted local segmentation thresholds of the reflective area and the non-reflective area, segmentation adjustment is performed on the local area of ​​the main image information to obtain a local segmentation adjustment result; The global segmentation result and the local segmentation adjustment result are merged to obtain the final segmentation result of the reflective area and the non-reflective area; In the final segmentation result of the reflective area and the non-reflective area, the reflective features of the garbage are extracted through the final reflective area, including but not limited to the reflection intensity, the reflective area and the shape of the reflective area.

5. The method for garbage sorting based on vision according to claim 1, characterized in that: The extracted reflective features are matched with the feature database to identify the type of garbage, and the specific steps are as follows: The garbage reflective features extracted from the final reflective area are sorted to form a feature vector; The feature vector is matched with the garbage type feature in the database to determine whether the matching degree exceeds the preset garbage type feature matching degree threshold, and the features exceeding the preset garbage type feature matching degree threshold are identified as the corresponding garbage type to obtain the garbage type identification result.

6. The method for garbage sorting based on vision according to claim 1, characterized in that: According to the identified garbage types, the spatial location information of the garbage to be sorted is fed back to the control center to generate a control signal, and then sent to the robotic arm to perform garbage grabbing operations. The specific steps are as follows: Receive the identification result of the garbage type and determine its spatial location information, wherein the spatial location information specifically includes the coordinates, direction and size of the garbage in three-dimensional space; The spatial position information is converted from the image coordinate system to the working coordinate system of the robot arm, and a target grasping position signal is generated and sent to the robot arm through a communication connection; The robotic arm receives the target grabbing position signal and moves according to the existing programmed motion instructions to grab and sort the target garbage.

7. The method for garbage sorting based on vision according to claim 1, characterized in that: The specific steps of optimizing the three-dimensional image information of the garbage based on the obtained three-dimensional data of the garbage are as follows: Preprocess the three-dimensional data of garbage, including denoising, filtering and registration; The preprocessed three-dimensional data is mapped to the three-dimensional image information through coordinate transformation for registration. Specifically, each point X, Y, Z in the preprocessed three-dimensional data is mapped to the pixel u, v on the image plane. The coordinate transformation expression is as follows: ， Where P is the projection matrix; Depth information is extracted from the preprocessed three-dimensional data, and the depth information is superimposed as an additional channel of the three-dimensional image information to obtain the three-dimensional image information of the optimized garbage.

8. A vision-based garbage sorting system, applied to any one of the vision-based garbage sorting methods as claimed in claims 1-7, characterized in that: The sorting system comprises: The visual information collection module is used to obtain three-dimensional image information, spatial position information and three-dimensional data of garbage by setting a three-dimensional visual information collection device and a three-dimensional laser scanning device in a triangular orientation; An image optimization module, used for optimizing the three-dimensional image information of the garbage based on the obtained three-dimensional data of the garbage, and obtaining the optimized three-dimensional image information; A global segmentation threshold adjustment module is used to select a single-direction image information from the three-direction image information as the main image information, capture the reflective features of garbage at different scales, and dynamically adjust the global segmentation threshold of the reflective area and the non-reflective area; A local segmentation threshold adjustment module is used to select the remaining two-position image information from the optimized three-position image information as the secondary image information, and dynamically adjust the local segmentation threshold of the reflective area and the non-reflective area according to the local contrast characteristics of the image; A segmentation and extraction module, used to segment the reflective area and the non-reflective area through the adjusted global segmentation threshold and the local segmentation threshold, and extract reflective features; A category matching module is used to match the extracted reflective features with the feature database to identify the type of garbage; The information feedback module is used to feed back the spatial location information of the garbage to be sorted to the control center according to the identified garbage type, generate a control signal, and send it to the robotic arm to perform garbage grabbing operations.

Images