Coal gangue recognition method based on depth camera image detection

Abstract

The application discloses a coal and gangue recognition method based on a depth camera image detection, and comprises the following steps: S1, acquiring a depth image and an RGB image of a certain conveying area by using a depth camera; S2, pre-processing the RGB image according to the depth image; S3, finding out the contours of the coal and the gangue in the conveying area in the pre-processed RGB image, and drawing the minimum circumscribed rectangle of the coal and the gangue; and S4, calculating the average gray value of a specified position in the minimum circumscribed rectangle, and judging that the object in the minimum circumscribed rectangle is the gangue if the average gray value exceeds a threshold value, and otherwise, the object is the coal. The coal and gangue recognition process is fast, the real-time performance is improved, the recognition accuracy is high, and the coal and gangue recognition effect is improved without increasing hardware facilities.

Description

Technical Field

[0001] The present invention belongs to the technical field of coal gangue identification, and particularly relates to a coal gangue identification method based on depth camera image detection. Background Art

[0002] Coal mining often produces large amounts of gangue. If not promptly disposed of, this gangue can affect coal quality. Therefore, sorting gangue from coal after mining can effectively improve coal quality. Therefore, accurate and effective gangue identification is a prerequisite before gangue sorting. Existing gangue recognition algorithms, based on RGB images, suffer from the following disadvantages: The presence of numerous interfering factors in the images directly impacts detection performance. For example, the high similarity and rich texture information between coal and gangue can easily lead to misidentification. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0004] To this end, the present invention proposes a coal gangue recognition method based on depth camera image detection, which has the advantages of fusing RGB images with depth images, while taking into account the coal gangue recognition accuracy and real-time requirements.

[0005] According to an embodiment of the present invention, a coal gangue identification method based on depth camera image detection includes: S1, using a depth camera to obtain a depth image and an RGB image of a certain conveying area; S2, preprocessing the RGB image according to the depth image; S3, finding the outlines of the coal and gangue in the conveying area in the preprocessed RGB image, and drawing the minimum circumscribed rectangle of the coal and gangue; S4, calculating the grayscale average value of a specified position within the minimum circumscribed rectangle. If the grayscale average value exceeds a threshold, it is determined that the object within the minimum circumscribed rectangle is gangue, otherwise it is coal.

[0006] The beneficial effect of the present invention is that the present invention uses a depth camera to shoot the coal and gangue in transportation, thereby simultaneously obtaining a depth image and an RGB image, and pre-processing the depth image and the RGB image in combination, so as to more accurately and quickly identify the contours of the coal and gangue, and draw the minimum circumscribed rectangle corresponding to the contour, and judge the type of the object within the minimum circumscribed rectangle according to the size of the grayscale average value of the specified position in the area. The coal and gangue recognition process is fast, the real-time performance is improved, and the accuracy rate is high during recognition. Without increasing hardware facilities, the effect of coal and gangue recognition is improved.

[0007] According to one embodiment of the present invention, the pre-processing operation in S2 includes a correction alignment process and a contrast enhancement process.

[0008] According to one embodiment of the present invention, the ground information of the depth image is integrated into the aligned RGB image in S2 to form an RGBD image. The RGBD image is copied, the background color of one of the RGBD images is converted to black, and then the convolution operation function filter2D and the Laplacian operator are used to improve the contrast.

[0009] According to one embodiment of the present invention, the Filter2D convolution operation formula is:

[0010] , recorded as formula (1); where, It is a single-channel floating-point matrix; is the input image; The kernel's reference point is (-1, -1) by default. central location; is the output image.

[0011] According to one embodiment of the present invention, the Laplace operator formula is: , recorded as formula (2); where, is the input image, is the output image.

[0012] According to one embodiment of the present invention, S3 includes: S31, taking the RGBD image after contrast enhancement as input, and using a function findContours of a contour finding algorithm to find the contour of the object in the RGBD image after contrast enhancement.

[0013] According to one embodiment of the present invention, S3 further includes: S32, setting an area threshold of the minimum enclosing rectangle to filter out invalid minimum enclosing rectangles; S33, determining whether two minimum enclosing rectangles are generated by the same object; if the two minimum enclosing rectangles are generated by the same object, filtering out the actual minimum enclosing rectangle corresponding to the object from the two minimum enclosing rectangles.

[0014] According to one embodiment of the present invention, in S32, the constraint formula used to filter out invalid minimum bounding rectangles is: , recorded as formula (3); where, is the minimum area threshold, is the maximum area threshold, Respectively represent The width and height of the smallest enclosing rectangle, is the total number of contours in the RGBD image.

[0015] According to one embodiment of the present invention, in S33, determining whether two minimum bounding rectangles are generated by the same object needs to satisfy the following two constraint formulas: , recorded as formula (4); where, Represent the width and height of the minimum enclosing rectangle, Respectively represent the x-coordinate and y-coordinate of the center point in the minimum circumscribed rectangle, is the distance threshold, Indicates the The minimum enclosing rectangle, Indicates the Minimum bounding rectangle; , recorded as formula (5); where, is the area difference threshold, Respectively represent The width and height of the smallest enclosing rectangle, Respectively represent The width and height of the minimum enclosing rectangle.

[0016] According to one embodiment of the present invention, in S4, a 5×5 pixel area is taken at the center of the actual minimum circumscribed rectangle, and the sum of the average grayscale values ​​of the three channels in the 5×5 pixel area is calculated. If it is less than 135, the object corresponding to the area is coal, otherwise it is gangue.

[0017] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments thereof, in which:

[0020] Figure 1 It is a flow chart of the coal gangue identification method based on depth camera image detection in the present invention;

[0021] Figure 2 is the RGB image acquired by the depth camera in the present invention;

[0022] Figure 3 is the depth image acquired by the depth camera in the present invention;

[0023] Figure 4 is the RGBD image obtained after preprocessing in the present invention;

[0024] Figure 5 The image obtained by drawing the minimum circumscribed rectangle of the coal and gangue in S3 of the present invention;

[0025] Figure 6 This is the effect diagram after gangue recognition in the RGBD image of the present invention;

[0026] Figure 7 The present invention is Figure 5 The effect after combining with the depth image. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0028] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0029] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0030] The following describes in detail a coal gangue identification method based on depth camera image detection according to an embodiment of the present invention with reference to the accompanying drawings.

[0031] like Figures 1 to 7As shown, the coal gangue identification method based on depth camera image detection according to an embodiment of the present invention includes: S1, using a depth camera to obtain a depth image and an RGB image of a certain conveying area; S2, preprocessing the RGB image according to the depth image; S3, finding the contours of the coal and gangue in the conveying area in the preprocessed RGB image, and drawing the minimum circumscribed rectangle of the coal and gangue; S4, calculating the grayscale average value of the specified position in the minimum circumscribed rectangle. If the grayscale average value exceeds a threshold, it is judged that the object in the minimum circumscribed rectangle is gangue, otherwise it is coal.

[0032] The present invention uses a depth camera to shoot the coal and gangue in transportation, thereby simultaneously obtaining a depth image and an RGB image, and preprocessing the depth image and the RGB image together, so as to more accurately and quickly identify the contours of the coal and gangue, and draw the minimum circumscribed rectangle corresponding to the contour. The type of object within the minimum circumscribed rectangle is judged according to the size of the grayscale average value of the specified position in the area. The coal and gangue recognition process is fast, the real-time performance is improved, and the recognition accuracy is high. Without increasing the hardware facilities, the coal and gangue recognition effect is improved.

[0033] According to one embodiment of the present invention, the pre-processing operation in S2 includes a correction alignment process and a contrast enhancement process.

[0034] Furthermore, the ground information of the depth image is integrated into the RGB image after alignment in S2 to form an RGBD image. The RGBD image is copied, and the background color of one of the RGBD images is converted to black. Then, the convolution operation function filter2D and the Laplacian operator are used to improve the contrast. In other words, from Figure 2 It can be seen that since the camera is far away from the edge of the conveying area, the RGB image directly taken by the depth camera is curved. Figure 4 In the image processing, the RGBD image is corrected and aligned, and the edge background becomes black. At the same time, the contrast between the coal and gangue and the surrounding environment is improved, and the edges of the coal and gangue are highlighted, so that the contour search will be more accurate and efficient. The depth camera uses the FM851-GI-E1 camera of Shanghai Tuyang Technology, which has a function TYMapDepthImageToColorCoordinate that maps the depth map to the color map coordinate. The correction and alignment operation is performed by directly calling this function.

[0035] The Filter2D convolution operation formula used is:

[0036] , recorded as formula (1); where, It is a single-channel floating-point matrix; is the input image; The kernel's reference point is (-1, -1) by default. central location; is the output image.

[0037] The Laplace operator formula used is: , recorded as formula (2); where, is the input image, is the output image.

[0038] According to one embodiment of the present invention, S3 includes the following steps: S31, using the contrast-enhanced RGBD image as input and using the findContours function of the contour search algorithm to find the contours of objects in the contrast-enhanced RGBD image. When using the findContours function to search for contours, the hyperparameter defining the contour retrieval mode is selected as RETR_TREE. This parameter is used to detect all contours, but only two hierarchical relationships are established for all contours: the outer periphery is the top layer. If the inner contour within the outer periphery also contains other contour information (the depth image has ground information, forming an inner contour), all contours within the inner periphery are classified as the top layer. The approximation method for defining contours is selected as CHAIN_APPROX_SIMPLE. This parameter only saves the inflection point information of the contours, saving all points at the contour inflection points in the contours vector, and not retaining information on the straight line segments between the inflection points.

[0039] Furthermore, the S3 also includes: S32, setting an area threshold of the minimum enclosing rectangle to filter out invalid minimum enclosing rectangles; S33, judging whether two minimum enclosing rectangles are generated by the same object. If the two minimum enclosing rectangles are generated by the same object, the actual minimum enclosing rectangle corresponding to the object is filtered out from the two minimum enclosing rectangles.

[0040] Preferably, in S32, the constraint formula used to filter out invalid minimum bounding rectangles is: , recorded as formula (3); where, is the minimum area threshold, is the maximum area threshold, Respectively represent The width and height of the smallest enclosing rectangle, is the total number of contours in the RGBD image. In the present invention, Take 300mm 2 , Take 240000mm 2 .

[0041] Preferably, in S33, determining whether two minimum bounding rectangles are generated by the same object needs to satisfy the following two constraint formulas: , recorded as formula (4); where, Represent the width and height of the minimum enclosing rectangle, Respectively represent the x-coordinate and y-coordinate of the center point in the minimum circumscribed rectangle, is the distance threshold, in the present invention, Take 50mm, Indicates the The minimum enclosing rectangle, Indicates the Minimum bounding rectangle; , recorded as formula (5); where, is the area difference threshold, in the present invention, Take 1500mm 2 , Respectively represent The width and height of the smallest enclosing rectangle, Respectively represent The width and height of the minimum enclosing rectangle.

[0042] That is to say, the minimum enclosing rectangle corresponding to the object needs to satisfy formula (3), formula (4) and formula (5) at the same time. Formula (3) mainly determines whether the area of ​​the minimum enclosing rectangle falls within 300-240000mm. 2 To eliminate invalid minimum enclosing rectangles; Formula (4) mainly determines whether two minimum enclosing rectangles correspond to the same object by whether the deviation of the position, width and height of the two minimum enclosing rectangles exceeds 50mm. It should be noted that the same parameters of the two minimum enclosing rectangles are compared, such as: The x coordinate of the first minimum bounding rectangle is the same as the The x coordinate of the minimum enclosing rectangle is subtracted from the distance threshold Comparison; Formula (5) mainly determines whether the difference in the area of ​​the two minimum circumscribed rectangles exceeds 1500mm 2 To determine whether two minimum enclosing rectangles correspond to the same object. Finally, when two minimum enclosing rectangles satisfy formulas (3), (4), and (5) at the same time, it means that these two minimum enclosing rectangles are generated by the same object in the RGBD image. Since an object has only one minimum enclosing rectangle by default, the minimum enclosing rectangle with the largest area is selected as the actual minimum enclosing rectangle of this object.

[0043] According to one embodiment of the present invention, in S4, a 5×5 pixel area is selected at the center of the actual minimum circumscribed rectangle. The sum of the grayscale averages of the three channels in the 5×5 pixel area is calculated. If the sum is less than 135, the object corresponding to the area is coal; otherwise, it is gangue. By comparing the sum of the grayscale averages of the three channels in the specified pixel area with the set value of 135, gangue can be quickly identified and filtered out from the coal.

[0044] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0045] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions, and alterations may be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. A coal gangue identification method based on depth camera image detection, characterized in that: include: S1. Use a depth camera to obtain a depth image and RGB image of a certain conveying area; S2. Preprocess the RGB image according to the depth image; S3, finding the outlines of the coal and gangue in the conveying area in the pre-processed RGB image, and drawing the minimum circumscribed rectangle of the coal and gangue; S4. Calculate the grayscale average value of the specified position within the minimum circumscribed rectangle. If the grayscale average value exceeds a threshold, the object within the minimum circumscribed rectangle is determined to be gangue; otherwise, it is coal. Set the area threshold of the minimum enclosing rectangle to filter out invalid minimum enclosing rectangles. The constraint formula used to filter out invalid minimum enclosing rectangles is: , recorded as formula (3); in, is the minimum area threshold, is the maximum area threshold, Respectively represent The width and height of the smallest enclosing rectangle, is the total number of contours in the RGBD image; Determine whether two minimum enclosing rectangles are generated by the same object. If the two minimum enclosing rectangles are generated by the same object, filter out the actual minimum enclosing rectangle corresponding to the object from the two minimum enclosing rectangles. Determining whether the two minimum enclosing rectangles are generated by the same object requires satisfying the following two constraint formulas: , recorded as formula (4); in, Represent the width and height of the minimum enclosing rectangle, Respectively represent the x-coordinate and y-coordinate of the center point in the minimum circumscribed rectangle, is the distance threshold, Indicates the The minimum enclosing rectangle, Expressed as Minimum bounding rectangle; , recorded as formula (5); in, is the area difference threshold, Respectively represent The width and height of the smallest enclosing rectangle, Respectively represent The width and height of the minimum enclosing rectangle.

2. The coal gangue identification method based on depth camera image detection according to claim 1, characterized in that: The pre-processing operations in S2 include alignment correction processing and contrast enhancement processing.

3. The coal gangue identification method based on depth camera image detection according to claim 2, characterized in that: The depth image information is fused into the rectified and aligned RGB image in S2 to form an RGBD image. The RGBD image is copied, and the background color of one of the RGBD images is converted to black. Then, the convolution operation function filter2D and the Laplacian operator are used to improve the contrast.

4. The coal gangue identification method based on depth camera image detection according to claim 3 is characterized in that: The Filter2D convolution operation formula is: ， Denoted as formula (1); in, It is a single-channel floating-point matrix; is the input image; , The kernel's reference point is (-1, -1) by default. central location; is the output image.

5. The coal gangue identification method based on depth camera image detection according to claim 3 is characterized in that: The Laplace operator formula is: ， Denoted as formula (2); in, is the input image, is the output image.

6. The method for identifying coal gangue based on depth camera image detection according to claim 3, characterized in that: The S3 includes: S31. Using the RGBD image after contrast enhancement as input, the contour of the object in the RGBD image after contrast enhancement is found using the function findContours of the contour finding algorithm.

7. The method for identifying coal gangue based on depth camera image detection according to claim 1, characterized in that: In S4, a 5×5 pixel area is taken at the center of the actual minimum circumscribed rectangle, and the sum of the average grayscale values ​​of the three channels in the 5×5 pixel area is calculated. If it is less than 135, the object corresponding to the area is coal, otherwise it is gangue.

Images