A coal block identification, detection and sorting system
The identification and sorting system based on machine vision and deep learning automatically identifies and sorts coal blocks and coal gangue, solving the problems of poor environment, high labor intensity and safety hazards in the coal preparation industry when manually sorting gangue, and realizing efficient and accurate automated sorting.
Patent Information
- Application Number
- CN201911216428.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-02
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-12-02
AI Technical Summary
In the coal preparation industry, manual sorting of gangue is characterized by poor working conditions, high labor intensity, difficulty in finding workers, and safety hazards. Existing technologies are insufficient for efficiently identifying and sorting coal blocks and gangue.
A machine vision and deep learning-based identification and sorting system is adopted. It utilizes an image acquisition device, a deep neural network, and a robotic arm sorting device to achieve automatic identification and sorting of coal blocks and coal gangue. The image acquisition device captures images of coal, the deep neural network is used for identification, and the robotic arm controller controls the robotic arm to sort the coal blocks and coal gangue into the corresponding bins.
It enables efficient and automatic identification and sorting of coal blocks and coal gangue, reducing manual intervention, improving identification accuracy, and saving time and labor costs.
Smart Images

Figure CN111266304B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of coal gangue detection and identification, and in particular to a coal block identification, detection and sorting system. Background Art
[0002] Coal is China's main energy and carbon resource, accounting for more than 70% of the primary energy structure for many consecutive years. It is estimated that by 2030, coal will account for 60% of primary energy consumption. Coal preparation is the source and foundation of clean coal technology. The "Action Plan for Air Pollution Prevention and Control" issued by the State Council regards increasing the proportion of coal washing as an important means to accelerate the adjustment of energy structure and increase the supply of clean energy. Manual selection of gangue is an important process link in the washing of raw coal. This position has a poor working environment, high labor intensity, difficulty in hiring, and major safety hazards. It has become one of the pain points that need to be urgently solved in the coal preparation industry. Summary of the invention
[0003] In order to solve the above problems, the present invention provides a clean coal and gangue identification and sorting system based on machine vision and deep learning.
[0004] The specific technical solutions are as follows:
[0005] A coal block identification, detection and sorting system comprises a conveyor belt, wherein the conveyor belt is provided with a speed control device, and the speed control device is connected to a management system; an image acquisition device and a manipulator sorting device are provided above the conveyor belt;
[0006] The image acquisition device includes a light source, a camera and a focusing device, the image acquisition device is connected to an image recognition module, and the image recognition module is connected to the management system;
[0007] The robot sorting device is connected to a robot controller, and the robot controller is connected to a management system;
[0008] The image acquisition device captures images of the raw coal and coal gangue transported on the conveyor belt. The image recognition module identifies coal blocks and coal gangue through a deep neural network. The management system sends an action signal to the robot controller based on the recognition result of the image recognition module. The robot controller controls the robot sorting device to transfer the raw coal and coal gangue to the raw coal bin and the gangue bin respectively.
[0009] Furthermore, the robot controller is also connected to a conveyor belt speed testing device.
[0010] Furthermore, the light source of the image acquisition device adopts a surface light source with uniform illumination.
[0011] Furthermore, the management system is also connected to a cloud data center.
[0012] Furthermore, the robot sorting device includes two robots.
[0013] Furthermore, the method for the image recognition module to identify coal blocks and coal gangue through a deep neural network includes the following steps:
[0014] Step 1) Establish and initialize a deep neural network model for gangue classification and recognition; establish a database, wherein the database has multiple calibrated gangue images; use open source data sets ImageNet, COCO, and VOC to pre-train a deep neural network;
[0015] Step 2) fine-tune the deep network using the calibrated images in the database to complete the training of the deep neural network;
[0016] Step 3) Collect the initial image through the image acquisition device; convert the initial image from the RGB color space to the HSV color space, then grayscale it, and obtain the image to be detected after performing closing and opening operation filtering.
[0017] Step 4) The image to be detected is input into the trained deep neural network, and the gangue recognition model outputs the recognition results of coal blocks and gangue.
[0018] Furthermore, the specific steps of pre-training the deep neural network in step 1) are: using the ImageNet dataset to perform classification training on the base network; removing the last classification layer in the base network, and connecting the recognition network to the 8x, 16x, and 32x downsampling layers in the middle layer of the neural network; and using the COCO and VOC datasets to train the entire network.
[0019] Furthermore, the method for converting the initial image from the RGB color space to the HSV color space specifically comprises the following steps:
[0020] Let the coordinates of the color parameters of the pixel point in the RGB normalized cube be (R, G, B), let the maximum value of the color parameters (R, G, B) be MAX, and the minimum value be MIN.
[0021] Step 612, let the coordinates of the color parameter of the pixel point in the HSV color space be (H, S, V).
[0022] Step 613, calculate the H value using Formula 1:
[0023]
[0024] Step 614, calculate the S value using Formula 2:
[0025]
[0026] Step 615, calculate the V value using Formula 3.
[0027] V=MAX
[0028] The beneficial result of the present invention is that the present invention completes the recognition training operation of coal blocks and coal gangue by collecting a large number of coal block and coal gangue training images, then obtains the coal block and coal gangue images to be identified from the high-speed camera, and finally inputs the collected images into the deep neural network model after training, and the deep neural network outputs the recognition results of the coal blocks and coal gangue to be identified. The above whole process does not require manual intervention, has a high recognition accuracy rate, and can achieve real-time recognition. It can save a lot of time and labor costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to clearly explain the technical solutions in the cases of the present invention, the drawings required for the description of the implementation cases are described below. The drawings described are structural solutions of the models used in the present invention, which are examples of part of the present invention, but not all examples. Those skilled in the art can also obtain other design solutions and drawings based on these drawings without creative work.
[0030] Figure 1 This is a schematic diagram of the block recognition, detection and sorting system structure.
[0031] Figure 2 This is the effect diagram of the robot;
[0032] Figure 3 Identify the effect of coal blocks;
[0033] Figure 4 is the base network structure of the neural network in the implementation method.
[0034] Figure 5 2 is a unit structure diagram of the base network in an implementation manner.
[0035] Figure 6 is the identification network structure of the neural network in the implementation method. DETAILED DESCRIPTION
[0036] System structure:
[0037] like Figure 1 A coal block identification, detection and sorting system shown includes a conveyor belt, characterized in that the conveyor belt is provided with a speed control device, and the speed control device is connected to a management system; an image acquisition device and a manipulator sorting device are provided above the conveyor belt;
[0038] The image acquisition device comprises a light source, a camera and a focusing device. The image acquisition device is connected to an image recognition module, and the image recognition module is connected to a management system.
[0039] The robot sorting device is connected to a robot controller, and the robot controller is connected to a management system; Figure 2 The robot sorting device shown includes two robots. A three-degree-of-freedom translational robot mechanism, in which a high-speed parallel mechanism is arranged in the XZ plane to drive the gripper to move up and down and left and right to realize the lateral pushing of the selected object. The motion range is X*Z=800mm*300mm. The lateral thrust of the end of the robot is 10kg. Each "pushing waste rock-returning" action cycle is 1 second. A linear motion mechanism is arranged in the Y direction. When the gripper pushes the selected object, it moves at the same speed as the belt to keep the gripper and the selected object relatively still in the Y direction. The motion range is 150mm. The XZ motion robot adopts a symmetrical parallel structure, which is equivalent to the movement of the two arms of the human body. Two sets of servo motor reducers are installed on the shoulders. Synchronous belts and parallel rod groups are installed on the two arms to constrain the direction of the gripper. The coordinated movement of the two servo motors' rotation angles and speeds realizes the movement of the end gripper along a predetermined trajectory. Synchronous belts and elastic connecting rods are used to reduce the impact of shock on the mechanism. The Y-direction movement is driven by a servo motor and a ball screw, and a double-column guide is adopted for linear bearings. The outer fingers of the gripper are driven to swing by a cylinder respectively.
[0040] The image acquisition device takes images of raw coal and coal gangue transported on the conveyor belt, and the image recognition module identifies coal blocks and coal gangue through a deep neural network. The management system sends an action signal to the manipulator controller according to the recognition result of the image recognition module, and the manipulator controller controls the manipulator sorting device to transfer the raw coal and coal gangue to the raw coal bin and the gangue bin respectively. The manipulator controller is also connected to a conveyor belt speed test device.
[0041] The light source of the image acquisition device adopts a surface light source with uniform illumination. Specifically, an LED lamp is used in combination with a diffusion plate. The biggest feature of the diffusion plate is that it causes interference to the light source, and the original messy design light distribution curve is adjusted through astigmatism to form a stable constant light system with a fixed angle of 160°-176°. Studies have shown that the larger the beam angle, the lower the illumination, and some will also produce light filtering. The light transmittance of the acrylic diffuser plate can reach 85%, which can make the light source evenly distributed on the screen without forming a residual image. The light astigmatism is not dazzling and does not penetrate the LED light source.
[0042] Furthermore, the management system is also connected to a cloud data center.
[0043] Gangue identification method:
[0044] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings, but the implementation and protection scope of the present invention are not limited thereto.
[0045] Step 100: Establish and initialize a deep neural network model for gangue classification and recognition.
[0046] Step 200: Establish a database, wherein the database has a plurality of calibrated coal gangue images.
[0047] Step 300. Use open source datasets ImageNet, COCO, and VOC to pre-train deep neural networks.
[0048] Step 400: Use the calibrated images in the database to fine-tune the deep network and complete the training of the deep neural network.
[0049] Step 500: Capture an image using a high-speed camera, and define the captured image as an initial image.
[0050] Step 600: Perform image preprocessing on the initial image acquired in step 500 to obtain an image to be detected.
[0051] Step 700: Input the image to be detected in step 600 into the deep neural network in step 400, and the gangue recognition model outputs the recognition results of coal blocks and gangue.
[0052] As a further improvement of the above technical solution, in step 300, step 400 and step 700, the structure of the deep neural network is as shown in the attached figure. Figure 4 As shown. It includes the base network and the recognition network.
[0053] As a further improvement of the above technical solution, in step 300, pre-training the deep neural network includes the following steps:
[0054] Step 310: Use the ImageNet dataset to perform classification training on the base network.
[0055] Step 320. Remove the last classification layer in the base network, and connect the recognition network to the 8-fold, 16-fold, and 32-fold downsampling layers in the middle layer of the neural network.
[0056] Step 330: Use COCO and VOC datasets to train the entire network.
[0057] As a further improvement of the above technical solution, step 600 includes the following steps:
[0058] Step 610: convert the image to be tested from the RGB color space to the HSV color space.
[0059] Step 620: grayscale the image to be tested.
[0060] Step 630, performing closing operation and opening operation filtering processing on the image to be tested.
[0061] As a further improvement of the above technical solution, in step 610, the following operations are performed on each pixel point of the image to be tested.
[0062] Step 611, let the coordinates of the color parameters of the pixel point in the RGB normalized cube be (R, G, B), let the maximum value of the color parameters (R, G, B) be MAX, and let the minimum value be MIN.
[0063] Step 612, let the coordinates of the color parameter of the pixel point in the HSV color space be (H, S, V).
[0064] Step 613, calculate the H value using Formula 1:
[0065]
[0066] Step 614, calculate the S value using Formula 2:
[0067]
[0068] Step 615, calculate the V value using Formula 3.
[0069] V=MAX (Formula 3)
[0070] The recognition result of step 700 includes the size and position of coal lumps and coal gangue. If there are coal lumps in the recognition result, the size level of the coal lumps and the distance from the calibration point are output. Figure 3 shown.
[0071] Deep Neural Networks:
[0072] Attached Figure 4 The base network shown:
[0073] In the base network structure, the output of each layer is imported into all the following layers. Different from the addition of the residual network, the unit structure of the present invention uses a concatenated structure, such as Figure 2 Such a structure can reduce network parameters and avoid the shortcomings that may occur in residual networks (such as some layers being selectively discarded, information blocking, etc.).
[0074] The calculation formula is:
[0075] x l =H([x0,x1,…,x l-1 ])
[0076] x l represents the l-layer FeatureMap, and H represents the convolution transformation.
[0077] Attached Figure 1 The transition layer in the image contains a bottleneck layer (i.e., a 1*1 convolutional layer) and a pooling layer. The 1*1 convolutional layer in the bottleneck layer is used to compress parameters. Each layer outputs k feature maps. In theory, each Dense Block outputs 4k feature maps, but in reality, the number is greater than this. The function of the convolutional layer is to compress the parameters of a Dense Block to 4k.
[0078] Attached Figure 5 The component function used is Batch Normalization+ReLU+3*3Conv.
[0079] As attached Figure 6 The recognition network structure shown is:
[0080] DBL is the basic component of the recognition network. It is convolution + BN + Leaky relu. BN and leaky relu are inseparable parts of the convolution layer (except the last convolution layer), and together they constitute the smallest component.
[0081] res_unit is a large component of the recognition network. It borrows the residual structure of ResNet. Using this structure can make the network structure deeper. Its basic component is DBL. concat is tensor concatenation. It is to concatenate the upsampling of the middle layer and a certain layer behind. The concatenation operation is different from the add operation of the residual layer. Concatenation will expand the dimension of the tensor, while add is just a direct addition without causing the change of the tensor dimension. Its input is the feature map of three different resolutions of the base network, corresponding to the attached layers in the base network. Figure 1 The outputs of Dense Block (2), Dense Block (3), and Dense Block (4) are used to detect and identify coal lumps and coal gangue in the input image at different scales. The feature map output by Dense Block (2) detects and identifies large-sized objects, the feature map output by Dense Block (3) detects and identifies medium-sized objects, and the feature map output by Dense Block (4) detects and identifies medium-sized objects. The input branch needs to be upsampled and spliced with the feature map of the previous resolution in order to increase the ability to recognize textures.
Claims
1. A coal block identification, detection and sorting system, comprising a conveyor belt, characterized in that: The conveyor belt is provided with a speed control device, which is connected to the management system; an image acquisition device and a manipulator sorting device are provided above the conveyor belt; The image acquisition device includes a light source, a camera and a focusing device, the image acquisition device is connected to an image recognition module, and the image recognition module is connected to the management system; The robot sorting device is connected to a robot controller, and the robot controller is connected to a management system; The image acquisition device captures images of raw coal and coal gangue transported on the conveyor belt. The image recognition module identifies coal blocks and coal gangue through a deep neural network. The management system sends an action signal to the manipulator controller based on the recognition result of the image recognition module. The manipulator controller controls the manipulator sorting device to transfer the raw coal and coal gangue to the raw coal bin and the gangue bin respectively. The manipulator controller is also connected to a conveyor belt speed testing device; The light source of the image acquisition device adopts a surface light source with uniform illumination; The management system is also connected to a cloud data center; The robot sorting device includes two robots; in, The method for the image recognition module to identify coal blocks and coal gangue through a deep neural network comprises the following steps: Step 1) Establish and initialize a deep neural network model for gangue classification and recognition; Establishing a database, wherein the database has a plurality of calibrated coal gangue images; Use open source datasets ImageNet, COCO, and VOC to pre-train deep neural networks respectively; The specific steps of pre-training the deep neural network in step 1) are: Use the ImageNet dataset to train the base network for classification; Remove the last classification layer in the base network, and connect the recognition network to the 8x, 16x, and 32x downsampling layers in the middle layer of the neural network respectively; use the COCO and VOC datasets to train the entire network; Step 2) fine-tune the deep network using the calibrated images in the database to complete the training of the deep neural network; Step 3) acquiring an initial image by an image acquisition device; Convert the original image from RGB color space to HSV color space, Then grayscale it. After performing closing operation and opening operation filtering, the image to be detected is obtained; Step 4) The image to be detected is input into the trained deep neural network, and the gangue recognition model outputs the recognition results of coal blocks and gangue.
2. A coal block identification, detection and sorting system as claimed in claim 1, characterized in that: The method for converting the initial image from the RGB color space into the HSV color space specifically comprises the following steps: Let the color parameters of the pixel point have coordinates (R, G, B) in the RGB normalized cube, let the maximum value of the color parameters (R, G, B) be MAX, and the minimum value be MIN; Let the color parameters of the pixel point have coordinates (H, S, V) in the HSV color space; Calculate H value: Calculate the S value: Calculate V value: V=MAX.
Citation Information
Patent Citations
Technology for visually identifying coal and coal gangue and conducting separation through mechanical arms
CN106216219A
Relevance score assignment for artificial neural network
CN107636693A
Material color recognition method used for industrial robot visual sorting
CN108890692A
Construction method of deep neural network model based on contact network image quality evaluation
CN110321944A
Coal briquette identifying, detecting and sorting system
CN212883587U
