A method for detecting and classifying a material

Abstract

The present invention provides a method for detecting and classifying materials. The method includes: at a first moment, obtaining the first marker point coordinates of at least one vertex on any diagonal of the rectangular positioning frame corresponding to each material; at a second moment, obtaining the second marker point coordinates of two vertices on the diagonal of the rectangular positioning frame corresponding to each material, and calculating the marked range of the rectangular positioning frame corresponding to each material at the second moment; obtaining the moving distance Lm of the conveyor belt on a straight line between the first moment and the second moment; determining whether there is a target material whose value of the first marker point coordinates + Lm falls within the marked range. If so, defining the target material and the material corresponding to the second marker point coordinates as the same material.

Description

Technical Field

[0001] The present invention relates to the technical field of material detection and classification, and in particular to a method for detecting and classifying the same material. Background Art

[0002] With the implementation of the garbage classification policy, people's demand for domestic waste sorting systems is increasing day by day. In the garbage sorting system, due to the irregularity of the shape and size of the materials and the disorder of the feeding, in order to ensure that no materials are missed during detection, when the camera takes pictures, the adjacent two frames of pictures need to overlap more than 20% of the field of view. However, this operation will cause some materials to be detected repeatedly, and then the material information will be sent to the motion control system repeatedly, resulting in multiple grabs by the motion control system, leading to empty grabs of the system and a decline in system performance.

[0003] At present, the main classification algorithms of deep learning include Resnet, VGG, and Efficientnet, etc., and the main detection algorithms include Fast-rcnn, SSD, YOLO, and Efficientdet, etc. Although these algorithms perform well on large-scale data sets.

[0004] However, deep learning algorithms rely on high-performance hardware and have a slow post-processing speed of the model, which cannot meet the requirements of real-time and practicality. In view of this, it is necessary to provide a method for detecting and classifying materials to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for detecting and classifying the same material.

[0006] To achieve one of the above-mentioned invention purposes, an embodiment of the present invention provides a method for detecting and classifying materials, the method includes: at a first moment, obtaining the first marker point coordinates of at least one vertex on any diagonal of the rectangular positioning frame corresponding to each material;

[0007] At a second moment, obtaining the second marker point coordinates of two vertices on the diagonal of the rectangular positioning frame corresponding to each material, and calculating the marked range of the rectangular positioning frame corresponding to each material at the second moment;

[0008] Obtaining the moving distance Lm of the conveyor belt on the straight line between the first moment and the second moment;

[0009] Judging whether there is a target material whose value of the first marker point coordinate + Lm falls within the marked range. If so, defining the target material and the material corresponding to the second marker point coordinate as the same material.

[0010] As a further improvement of an embodiment of the present invention, the method for obtaining the coordinates of the first marked point further includes: at the first moment, obtaining the marked point coordinates of two vertices on any diagonal of the rectangular positioning frame corresponding to each material as the coordinates of the first marked point.

[0011] As a further improvement of an embodiment of the present invention, a method for detecting and classifying materials further includes: at the first moment, assigning an identity number to each of the materials. If it is determined that the target material and the material corresponding to the second marked point coordinates are the same material, the identity numbers are kept consistent; if they are not the same material, a new identity number is assigned to the material.

[0012] As a further improvement of an embodiment of the present invention, a method for detecting and classifying materials further includes: respectively obtaining the 2D and 3D bird's-eye view image data of each material at the first moment and the second moment;

[0013] Registering the 2D bird's-eye view image data with the 3D bird's-eye view image data to obtain the RGB-D image data of each material;

[0014] Loading the RGB-D image data into a deep learning image algorithm for feature extraction to obtain the rectangular positioning frame, grasping point and category information of each material.

[0015] As a further improvement of an embodiment of the present invention, the method for processing RGB-D image data further includes: preprocessing the RGB-D image data: removing distortion and scaling the size.

[0016] As a further improvement of an embodiment of the present invention, where the RGB-D image data is loaded into a deep learning image algorithm for feature extraction to obtain the rectangular positioning frame, grasping point and category information of each material, specifically including:

[0017] Extracting three-layer outputs from top to bottom and downsampling twice to obtain five-layer feature output layers;

[0018] Performing feature fusion in a way of constructing a feature pyramid;

[0019] Obtaining the first rectangular positioning frame and category information.

[0020] As a further improvement of an embodiment of the present invention, where the method further includes: post-processing the first rectangular positioning frame and category information:

[0021] Judging the recyclable probability of the material according to the first rectangular positioning frame and category information of the material;

[0022] Performing threshold suppression according to the recyclable probability to remove the rectangular positioning frames with low probability;

[0023] Load the non-maximum suppression algorithm (NMS) to remove duplicates from the first rectangular positioning frames and category information.

[0024] Based on the de-duplicated rectangular positioning frames and category information, suppress the materials corresponding to the rectangular positioning frames with low recyclable probabilities, and retain the rectangular positioning frames and category information with high recyclable probabilities.

[0025] Obtain the second rectangular positioning frames and category information.

[0026] As a further improvement of an embodiment of the present invention, the method further includes: finally processing the 2D and 3D bird's-eye view image data, the second rectangular positioning frames and category information of each material:

[0027] Extract the rectangular positioning frames of individual materials and put them into an array of the same size as the original image.

[0028] Convert the array into a grayscale image, perform filtering, binarization, and morphological closing operations on it.

[0029] Calculate the center of the largest inscribed circle of the binarized image, and this center is the grasping point of the material.

[0030] Obtain the final material grasping points, rectangular positioning frames and category information.

[0031] As a further improvement of an embodiment of the present invention, for a method of detecting and classifying materials, if the radius of the largest inscribed circle of the material image is smaller than the radius of the suction cup of the suction cup type spider manipulator, then this material is not grasped.

[0032] To achieve one of the above-mentioned invention purposes, an embodiment of the present invention provides a device for detecting and classifying materials, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps of the above-mentioned method for detecting and classifying materials.

[0033] To achieve one of the above-mentioned invention purposes, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program, and when the computer program runs, it causes the device where the computer-readable storage medium is located to execute the steps of the above-mentioned method for detecting and classifying materials.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows: Through a lightweight same-material detection algorithm, the materials on the conveyor belt are identified and judged, avoiding repeated detection of materials and empty grasping by the system, accelerating the system processing speed, improving the efficiency of detecting and classifying the same materials, and greatly enhancing the system performance. Description of the Drawings

[0035] Figure 1It is a schematic diagram of the steps of the same material detection and classification method provided by an embodiment of the present invention;

[0036] Figure 2 It is according to Figure 1 The effect diagram after processing according to the steps shown;

[0037] Figure 3 It is a schematic diagram of the steps for positioning the material;

[0038] Figure 4 It is on the basis of Figure 3 A schematic diagram of the steps for performing feature extraction operations on the target material;

[0039] Figure 5 It is on the basis of Figure 4 A schematic diagram of the steps for performing post-processing operations on the target material information;

[0040] Figure 6 It is on the basis of Figure 5 A schematic diagram of the steps for performing final processing operations on the target material information;

[0041] Figure 7 It is according to Figure 2 — Figure 6 The effect diagram after processing according to the steps shown. Specific embodiments

[0042] The present invention will be described in detail below in conjunction with the specific embodiments shown in the drawings. However, these embodiments do not limit the present invention, and any structural, method, or functional transformation made by those of ordinary skill in the art according to these embodiments is included in the protection scope of the present invention.

[0043] Please refer to Figure 1 As shown, in the first embodiment of the present invention, a material detection and classification method is provided, and the method includes:

[0044] S101. At the first moment, obtain the first marker point coordinates of at least one vertex on any diagonal of the rectangular positioning frame corresponding to each material;

[0045] S102. At the second moment, obtain the second marker point coordinates of two vertices on the diagonal of the rectangular positioning frame corresponding to each material, and calculate the marked range of the rectangular positioning frame corresponding to each material at the second moment;

[0046] S103. Obtain the moving distance Lm of the conveyor belt on the straight line between the first moment and the second moment;

[0047] S104. Determine whether there is a target material whose value of the first marker point coordinates + Lm falls within the marked range:

[0048] If so, define the material corresponding to the target material and the coordinates of the second marked point as the same material.

[0049] In a specific embodiment of the present invention, for step S1, the marked point coordinates of two vertices on any diagonal of the rectangular positioning frame corresponding to each material can also be obtained as the first marked point coordinates.

[0050] In a specific embodiment of the present invention, the target material is the garbage to be detected, such as wooden boards, stones, plastics, cardboard, etc.

[0051] Further, for step S104, an XY rectangular coordinate system is established with the upper left corner of the original image as the coordinate origin, the positive direction of the X-axis extends to the right from the coordinate origin, and the positive direction of the Y-axis extends downward from the coordinate origin. Among them, the x-axis is the horizontal axis and the y-axis is the vertical axis. At the first moment, let the j-th object in the i-th frame be S(i,j); at the second moment, let the j-th object in the i-th frame be S(i + 1,j);

[0052] In a specific embodiment of the present invention, for step S101, at the first moment, the first marked point coordinates are denoted as (X1(i,j), Y1(i,j)) and (Xr(i,j), Yr(i,j)); for step S102, at the second moment, the second marked point coordinates are denoted as (X1(i + 1,j), Y1(i + 1,j))

[0053] and (Xr(i + 1,j), Yr(i + 1,j));

[0054] Further, determine whether there is a target material in which the value of the first marked point coordinate + Lm falls within the marked range;

[0055] As an embodiment of the present invention, if the conveyor belt moves linearly along the defined X-axis and the above marked points satisfy the following relationship, then define the material corresponding to the target material and the coordinates of the second marked point as the same material. In the following formula, R is the conversion ratio factor between the world coordinate and the image pixel coordinate:

[0056] X1(i + 1,j) = (X1(i,j) + R × Lm

[0057] Y1(i + 1,j) = Y1(i,j)

[0058] Xr(i + 1,j) = Xr(i,j) + R × Lm

[0059] Yr(i + 1,j) = Yr(i,j);

[0060] As an embodiment of the present invention, if the conveyor belt moves linearly along the defined Y-axis and the above-mentioned marked points satisfy the following relationship, it is defined that the target material and the material corresponding to the coordinates of the second marked point are the same material. In the following formula, R is the conversion scale factor between the world coordinates and the image pixel coordinates:

[0061] X1(i + 1, j) = (X1(i, j)

[0062] Y1(i + 1, j) = Y1(i, j) + R × Lm

[0063] Xr(i + 1, j) = Xr(i, j)

[0064] Yr(i + 1, j) = Yr(i, j) + R × Lm.

[0065] Further, for step S4, after the judgment is completed, the method steps further include:

[0066] If it is determined that the target material and the material corresponding to the coordinates of the second marked point are the same material, the identity numbers are kept consistent;

[0067] If they are not the same material, a new identity number is assigned to this material,

[0068] Preferably, in the specific embodiments of the present invention, the implementation effects are as Figure 2 shown.

[0069] Further, in step S101, the specific steps of obtaining the rectangular positioning frame corresponding to each material for material positioning are as follows. Please refer to Figure 3 shown:

[0070] S1011. Obtain the 2D and 3D bird's-eye view image data of each material at the first moment and the second moment respectively;

[0071] S1012. Register the 2D bird's-eye view image data with the 3D bird's-eye view image data to obtain the RGB-D image data of each material;

[0072] S1013. Load the RGB-D image data into the deep learning image algorithm for feature extraction to obtain the rectangular positioning frame, grasping point and category information of each material.

[0073] Specifically, for step S1011, the 2D bird's-eye view image data is obtained by a line array camera to obtain RGB picture data, and the 3D bird's-eye view image data is obtained by a 3D camera to obtain point cloud data;

[0074] Specifically, the line array camera works in the encoder trigger mode. When the conveyor belt moves a distance of Lm, the camera takes a picture once. The field of view range of the camera along the belt line direction is Lc, and the constraint condition Lm ≤ 0.7 × Lc needs to be satisfied;

[0075] It is understandable that the above 2D image data includes the color information of the image, and the 3D point cloud data, i.e., three-dimensional information, includes three-dimensional shape, three-dimensional contour, position and other information;

[0076] Specifically, by combining the RGB image and the point cloud data, the RGB-D information of the garbage to be classified is obtained;

[0077] It is understandable that the RGB-D image data contains the color information, three-dimensional position information, three-dimensional contour information, etc. of the garbage; in the present invention, by performing deep learning on the RGB-D image data to identify materials and obtain the target materials, due to the addition of height information in the model, the recognition accuracy and recognition rate when materials are adhered or overlapped can be effectively improved;

[0078] Specifically, before loading the RGB-D image data into the deep learning image algorithm, the RGB-D image data needs to be preprocessed, including operations such as distortion removal and size scaling.

[0079] Furthermore, for step S1013, after obtaining the target materials, feature extraction needs to be performed on the target materials to obtain the rectangular positioning frame, grasping points and category information of each material, and feature extraction operations also need to be performed on the target materials. Please refer to Figure 4 as shown, specifically including:

[0080] S1021: Use a backbone network with lightweight design as the feature extraction network, extract three layers of outputs from top to bottom and downsample twice, with a total of five layers as the feature output layer (Feature Maps);

[0081] S1022: Adopt the method of constructing a feature pyramid for feature fusion to make the features more robust and the overall system more stable;

[0082] S1023: Output the first rectangular positioning frame and category information.

[0083] Furthermore, for step S1023, after obtaining the first rectangular positioning frame and category information of the target materials, post-processing operations also need to be performed on the target material information. Please refer to Figure 5 as shown, specifically including:

[0084] S1031: According to the first rectangular positioning frame and category information of the material, judge the recyclable probability of the material;

[0085] S1032: Perform threshold suppression according to the recyclable probability to remove the rectangular positioning frames with low probability;

[0086] S1033. Load the non-maximum suppression algorithm (NMS) to remove duplicates from the first rectangular positioning boxes and class information;

[0087] S1034. According to the de-duplicated rectangular positioning boxes and class information, suppress the materials corresponding to the rectangular positioning boxes with low recyclable probabilities, and retain the rectangular positioning boxes and class information with high recyclable probabilities;

[0088] S1035. Output the second rectangular positioning boxes and class information.

[0089] Further, for step S1035, after obtaining the second rectangular positioning boxes and class information of the target materials, final processing operations need to be performed on the target material information. Please refer to Figure 6 as shown, specifically including:

[0090] S1041. Extract the second rectangular positioning box of a single material and put it into an array of the same size as the original image;

[0091] S1042. Perform grayscale conversion, filtering, binarization, and morphological closing operations on this array;

[0092] S1043. Take the largest inscribed circle corresponding to the outer contour of the target material, calculate the center of the largest inscribed circle of this binarized image, and this center is the grasping point of the material;

[0093] S1044. Obtain the final material grasping point, rectangular positioning box, and class information.

[0094] Further, in an embodiment of the present invention, the manipulator for grasping is a spider manipulator with a suction cup. The suction cup is made of soft material and can grasp small materials;

[0095] Preferably, in a specific embodiment of the present invention, the implementation effect is as Figure 7 shown.

[0096] Further, for step S1043, according to the largest inscribed circle of the target material, determine whether to grasp this material:

[0097] If the radius of this inscribed circle is greater than or equal to the radius of the suction cup of the suction cup type spider manipulator, control to grasp this material;

[0098] If the radius of this inscribed circle is less than the radius of the suction cup of the suction cup type spider manipulator, do not grasp this material.

[0099] Further, an embodiment of the present invention provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps in the above-mentioned material detection and classification method.

[0100] Further, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned method for detecting and classifying materials are implemented.

[0101] In summary, a method for detecting and classifying materials according to the present invention can accurately determine whether they are the same material.

[0102] It should be understood that although this specification is described according to embodiments, not every embodiment only contains an independent technical solution. This narrative way of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0103] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

Claims

1. A method for detecting and classifying materials, characterized in that, it includes the following steps: At a first moment, obtain the first marker point coordinates of at least one vertex on any diagonal of the rectangular positioning frame corresponding to each material; At a second moment, obtain the second marker point coordinates of two vertices on the diagonal of the rectangular positioning frame corresponding to each material, and calculate the marked range of the rectangular positioning frame corresponding to each material at the second moment; Obtain the moving distance Lm of the conveyor belt on a straight line between the first moment and the second moment; Determine whether there is a target material whose value of the first marker point coordinates + Lm falls within the marked range. If so, define the target material and the material corresponding to the second marker point coordinates as the same material; Wherein, at the first moment and the second moment respectively, obtain the 2D and 3D bird's-eye view image data of each material; Register the 2D bird's-eye view image data with the 3D bird's-eye view image data to obtain the RGB-D image data of each material; Load the RGB-D image data into a deep learning image algorithm for feature extraction to obtain the rectangular positioning frame and category information of each material.

2. The method for detecting and classifying materials according to claim 1, characterized in that, At the first moment, obtain the marker point coordinates of two vertices on any diagonal of the rectangular positioning frame corresponding to each material as the first marker point coordinates.

3. The method for detecting and classifying materials according to claim 1, characterized in that, The method further includes: Assign an identity number to each of the materials at the first moment. If it is determined that the target material and the material corresponding to the second marker point coordinates are the same material, keep the identity numbers consistent; if they are not the same material, assign a new identity number to the material.

4. The method for detecting and classifying materials according to claim 1, characterized in that, The method further includes: Load the RGB-D image data into a deep learning image algorithm for feature extraction to obtain the grasping point of each material.

5. The method for detecting and classifying materials according to claim 4, characterized in that, The method further includes: Preprocess the RGB-D image data: remove distortion and scale the size.

6. The method for detecting and classifying materials according to claim 4, characterized in that, Loading the RGB-D image data into a deep learning image algorithm for feature extraction to obtain the rectangular positioning frame, grasping point and category information of each material specifically includes: Extract three layers of outputs from top to bottom and downsample twice to obtain five-layer feature output layers; Adopt a method of constructing a feature pyramid for feature fusion; Obtain the first rectangular positioning frame and category information.

7. The method for detecting and classifying materials according to claim 6, characterized in that, The method further includes: Post-process the first rectangular positioning frame and category information: According to the first rectangular positioning frame and category information of the material, judge the recyclable probability of the material; According to the recyclable probability, perform threshold suppression to remove rectangular positioning frames with low probability; Load the non-maximum suppression algorithm (NMS) to remove duplicates from the first rectangular positioning frame and category information; Suppress the materials corresponding to the rectangular positioning frames with low recyclable probabilities according to the deduplicated rectangular positioning frames and category information, and retain the rectangular positioning frames and category information with high recyclable probabilities; Obtain the second rectangular positioning frame and category information.

8. The method for detecting and classifying materials according to claim 4, wherein, the method further includes: Finally process the 2D and 3D bird's-eye view image data, the second rectangular positioning frame and category information of each material: Extract the rectangular positioning frame of a single material and put it into an array of the same size as the original image; Perform grayscale conversion, filtering, binarization, and morphological closing operation on the array; Calculate the center of the largest inscribed circle of the binarized image, and this center is the grasping point of the material; Obtain the final material grasping point, rectangular positioning frame and category information.

9. The method for detecting and classifying materials according to claim 8, wherein, For the largest inscribed circle of the material image, if the radius of this inscribed circle is smaller than the radius of the suction cup of the suction cup type spider manipulator, then this material is not grasped.

10. A device for detecting and classifying materials, wherein, it includes: A memory and a processor, the memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps of the method for detecting and classifying materials according to any one of claims 1-9.

11. A computer storage medium, wherein, it stores a computer program, and when the computer program runs, it causes the device where the computer storage medium is located to execute the steps of the method for detecting and classifying materials according to any one of claims 1-9.

Images