Coal gangue grabbing method and system based on image recognition

Abstract

The present application provides a kind of coal gangue classification and capture method and system based on image recognition, it is related to coal gangue sorting technical field.Therein, coal gangue classification and capture method based on image recognition includes the following steps: construct coal gangue target detection model, for the identification and classification of two targets of coal and coal gangue;Coal gangue capture detection model is constructed, for the optimal capture posture of the coal gangue identified by the coal gangue target detection model is generated;Build coal gangue capture planning system, for according to the optimal capture posture control robot moves to the posture position under the robot coordinate system and executes the task of the coal gangue identified by capture.From the point of view, with reduce labor cost, reduce misjudgment loss, reduce environmental pollution, reduce governance cost, the identification efficiency of coal gangue is high, the identification accuracy is high, the capture efficiency of coal gangue is high, the capture accuracy is high, improve the beneficial technical effects such as the level of coal production automation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of coal gangue processing, and particularly relates to a coal gangue grabbing method and system based on image recognition. BACKGROUND

[0002] In the process of coal production, coal and coal gangue are inevitably mixed in raw coal. Since the coal gangue has a low calorific value, the mixing of coal gangue with coal will reduce the calorific value of the coal. Therefore, it is necessary to sort out the coal gangue from the coal to improve the quality of the coal and clean utilization.

[0003] Manual identification and sorting of coal gangue have problems of low efficiency, low accuracy, high labor cost and high labor intensity. SUMMARY

[0004] The embodiment of the present application aims to provide a coal gangue grabbing method and system based on image recognition to solve the problems of low efficiency, low accuracy, high labor cost and high labor intensity in manual identification and sorting of coal gangue in the prior art.

[0005] To solve the above technical problems, the embodiment of the present application provides the following technical scheme:

[0006] According to a first aspect of the present application, a coal gangue classification and grabbing method based on image recognition is provided, comprising the following steps: constructing a coal gangue target detection model for identification and classification of two targets of coal and coal gangue; constructing a coal gangue grabbing detection model for generating an optimal grabbing posture of the coal gangue identified by the coal gangue target detection model; and building a coal gangue grabbing planning system for controlling the robot to move to a posture position in the robot coordinate system according to the optimal grabbing posture to perform the task of grabbing the identified coal gangue.

[0007] Further, the construction of the coal gangue target detection model comprises the following steps: making an image recognition data set, which includes coal images, coal gangue images and corresponding label files; training a target recognition network using the image recognition data set, and improving the detection scale and loss function of the target recognition network to obtain the constructed coal gangue target detection model.

[0008] Further, the image recognition dataset is specifically made by the following steps: taking a plurality of coal images in a closed environment; taking a plurality of coal gangue images in the closed environment; taking a plurality of images of coal and coal gangue mixture in the actual generation process of raw coal; preprocessing the plurality of coal images, the plurality of coal gangue images and the plurality of images of coal and coal gangue mixture; labeling the plurality of coal images, the plurality of coal gangue images and the plurality of images of coal and coal gangue mixture respectively to obtain a plurality of one-to-one corresponding label files; the preprocessed plurality of coal images, the plurality of coal gangue images, the plurality of images of coal and coal gangue mixture and the plurality of one-to-one corresponding label files form the image recognition dataset; in the step of constructing the coal gangue target detection model, the YOLOv5s network is selected as the target recognition network; the detection scale of the target recognition network is improved by the following steps: simplifying the output end of the YOLOv5s network structure, deleting the 19*19 feature map branch in the output end, and retaining the 76*76 feature map and the 38*38 feature map as the output; the loss function of the target recognition network is improved by the following steps: selecting the CIoU loss function as the loss function of the model training.

[0009] Further, the coal gangue grabbing detection model is constructed by the following steps: making a simulation dataset, the simulation dataset including coal gangue simulation image data and corresponding label files; training the pose estimation network by using the coal gangue simulation dataset, and integrating attention mechanism and multi-scale residual block into the pose estimation network to obtain the constructed coal gangue grabbing detection model.

[0010] Further, the coal gangue simulation dataset is specifically made by the following steps: modeling according to the actual size of coal and coal gangue by using the SolidWorks software, and selecting the graphite attribute in each material attribute; generating the urdf model of coal gangue by using the sw2urdf plug-in in the SolidWorks software; adding the urdf model of coal gangue into the Gazebo model in the workspace of ROS; calling the depth camera to collect a plurality of coal gangue simulation image data, and saving the image size as a preset size; preprocessing the plurality of coal gangue simulation image data; labeling the plurality of coal gangue simulation image data to obtain a plurality of one-to-one corresponding label files; the preprocessed plurality of coal gangue simulation image data and the plurality of one-to-one corresponding label files form the coal gangue simulation dataset; in the step of constructing the coal gangue grabbing detection model, the GR-ConvNet network is selected as the pose estimation network; the attention mechanism is integrated into the pose estimation network by the following steps: embedding the attention module in the network structure of the GR-ConvNet, the attention module including a channel attention submodule and a spatial attention submodule; the multi-scale residual block is integrated into the pose estimation network by the following steps: embedding the multi-scale residual block in the network structure of the GR-ConvNet.

[0011] Further, the coal gangue classification and grabbing method based on image recognition further comprises the following steps: calibrating the coal gangue grabbing coordinate system, including a camera calibration step and a hand-eye calibration step, the camera calibration step comprising: establishing a mapping relationship between a pixel coordinate system, an image coordinate system, a camera coordinate system and a world coordinate system; using Zhang Zhengyou calibration method for calibration, in the calibration process, the corner point position data is used to calculate the internal parameter matrix and distortion coefficients k1, k2, p1, p2, k3 of the camera; the image is corrected by calculating the internal parameters and distortion coefficients of the camera; the hand-eye calibration step comprises: establishing the relative position and attitude relationship between the robot end effector and the vision sensor; evaluating the coal gangue grabbing detection model, using a rectangular measurement index as the standard for evaluating the candidate grabbing accuracy, when the accuracy evaluation result of the parameter grabbing appears the following two cases, it can be concluded that the grabbing configuration is correct: (1) the angle difference between the predicted grabbing box g p and the real grabbing box g t is within 30 degrees; (2) the J(g p , g t ) of the predicted grabbing box g p and the real grabbing box g t is greater than 0.25.

[0012] Wherein, g p is the predicted grabbing box, and g t is the real grabbing box.

[0013] Further, the coal gangue grabbing planning system comprises the following steps: building a coal gangue grabbing planning system based on the ROS framework and the Ubuntu environment; using the Gazebo simulation environment to build a robot grabbing simulation experiment platform, combining the motion planning module of the MoveIt software to realize the optimal attitude calculation and grabbing operation of the gangue; controlling the simulation robot to perform the grabbing action through the ros_control function.

[0014] Further, the coal gangue classification and grabbing method based on image recognition further comprises the following steps: obtaining the to-be-detected image; using the pre-constructed coal gangue target detection model to identify and classify the coal and coal gangue two targets in the to-be-detected image; the coal gangue target detection model feeds back the recognized coal gangue category information and transmits it to the coal gangue grabbing detection environment; using the pre-constructed coal gangue grabbing detection model to generate the optimal grabbing attitude of the recognized coal gangue; converting the optimal grabbing attitude into a grabbing attitude vector in the robot coordinate system through the coordinate system conversion method; sending instructions to the controller through the upper computer to control the robot to move to the attitude position and execute the grabbing task.

[0015] Further, generating the optimal grasping pose of the recognized coal gangue by using the pre-constructed coal gangue grasping detection model comprises the following steps: inputting the to-be-detected image I into the coal gangue grasping detection model; I = R n×h×w , h is height, w is width, and n is the number of channels; the coal gangue grasping detection model extracts features from the to-be-detected image I and generates a grasping success rate image, an angle image and a width image; the grasping success rate image, the angle image and the width image are pixel-level representation images of the grasped object with the same size as the input image; the coal gangue grasping detection model infers the optimal grasping pose from the grasping success rate image, the angle image and the width image, and outputs a pose vector G i of the robot gripper in the image coordinate system; G i = (x, y, Θ i , W i , Q); wherein x and y are the center position coordinates of the grasped object in the image coordinate system, in units of pixels; Θ i represents the direction or pose of the robot gripper, described by Euler angles or quaternions, the rotation angle being in units of radians in the camera reference coordinate system, and the value range of Θ i is [-π / 2, π / 2]; W i represents the opening width of the robot gripper, used to generate the distance between the robot fingers, in units of pixels; Q is a score for evaluating the grasping quality; the value range of Q is 0 to 1, and the closer Q is to 1, the higher the possibility of successful grasping; it is required that W i ≤ W max , and W max is the maximum width of the robot using the antipodal gripper; converting the optimal grasping pose into a grasping pose vector in the robot coordinate system by a coordinate system conversion method comprises the following steps: the grasping pose vector in the robot coordinate system is represented as G r , G r = (P, Θ r , W r , Q), wherein P = (x, y, z) is the center position of the tool, Θ r is the rotation of the tool around the Z axis, W r is the required width of the tool, and Q is the grasping quality score; the coordinate system conversion is realized by the following formula: G r = T rc (T ci (G i )); wherein T ci is a transformation matrix from the robot end effector coordinate system to the camera coordinate system, describing the position and direction of the camera in the robot coordinate system; T rc is a transformation matrix from the camera coordinate system to the world coordinate system, describing the position and direction of the object seen by the camera in the world coordinate system; G ris a grasp pose vector in a robot coordinate system; G i is a pose vector of a robot gripper in an image coordinate system.

[0016] According to another aspect of the present application, the present application provides an image recognition-based coal gangue classification and grasping system, comprising: a target detection module, the target detection module comprising a pre-constructed coal gangue target detection model, the coal gangue target detection model being used for identification and classification of two targets of coal and coal gangue; a pose detection module connected to the target detection module, the target detection module feeding back coal gangue category information identified to the pose detection module; the pose detection module comprising a pre-constructed coal gangue grasping detection model, the coal gangue grasping detection model being used for generating an optimal grasping pose of the identified coal gangue; a grasping execution module connected to the pose detection module, the pose detection module feeding back the optimal grasping pose to the grasping execution module, the grasping execution module converting the optimal grasping pose into a grasp pose vector in a robot coordinate system through a coordinate system conversion method; the grasping execution module comprising a robot, the robot being used for moving to a pose position and executing a grasping task.

[0017] Compared with the prior art, the image recognition-based coal gangue grasping method and system provided by the present application have the beneficial technical effects of reducing labor cost, reducing misjudgment loss, reducing environmental pollution, reducing treatment cost, high identification efficiency of coal gangue, high identification accuracy, high grasping efficiency of coal gangue, high grasping accuracy, and improving the automation level of coal production. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the exemplary embodiments of the present application will be more apparent from the following detailed description read in conjunction with the accompanying drawings, in which several embodiments of the present application are shown by way of example, and wherein the same reference numerals refer to the same or similar components throughout several instances. In the drawings:

[0019] Figure 1 A framework schematic diagram of the image recognition-based coal gangue grasping system is schematically shown; Figure 2 A flowchart of a model construction part of the image recognition-based coal gangue grasping method is schematically shown; Figure 3 A flowchart of a model application part of the image recognition-based coal gangue grasping method is schematically shown; Figure 4 A coordinate calibration process diagram of the image recognition-based coal gangue grasping method is schematically shown; Figure 5 A conversion flowchart between coordinate systems of the image recognition-based coal gangue grasping method is schematically shown; Figure 6 Images of calibration boards in different orientations are schematically shown; Figure 7 A hand-eye calibration schematic diagram is schematically shown; Figure 8A coal image is schematically shown; Figure 9 A coal gangue image is schematically shown; Figure 10 An image of a mixture of coal and coal gangue is schematically shown; Figure 11 A schematic diagram of the YOLOv5s network structure before improvement is schematically shown; Figure 12 The intermediate network and output end of the improved coal gangue grabbing detection model are schematically shown; Figure 13 The experimental results before and after the improvement of the YOLOv5s by the present application are schematically shown; Figure 14 The position, confidence and category information subscribed in target detection are schematically shown; Figure 15 A GR-ConvNet grabbing pose estimation network model structure diagram is shown; Figure 16 A structure diagram of the attention module CBAM is schematically shown; Figure 17 A structure diagram of the multi-scale residual block is schematically shown; Figure 18 The experimental results of the CNN, the GR-ConvNet before improvement, and the improved coal gangue grabbing detection model are compared; Figure 19 The experimental results of the GR-ConvNet before improvement and the improved coal gangue grabbing detection model are compared; Figure 20 A flowchart of the GR-ConvNet outputting the optimal grabbing grabbing pose is schematically shown;

[0020] Figure 21 The grabbing rectangle inferred by the grabbing quality, angle and width output by the GR-ConvNet is schematically shown; Figure 22 The position and pose information subscribed in grabbing detection is schematically shown; Figure 23 A framework diagram of the gangue simulation grabbing system is schematically shown; Figure 24 A structure diagram of the coal gangue classification grabbing system based on image recognition is schematically shown. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art. If not specifically stated, the technical means used in the examples are conventional means known to those skilled in the art.

[0022] It should be noted that the technical terms or scientific terms used in the present application should be understood as the general meaning understood by the skilled person in the field to which the present application belongs, unless otherwise specified. In the present application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. The terms "connected", "connected" and the like should be interpreted broadly, for example, it can be fixed connection, or detachable connection, or integral; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium. The terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0023] In combination Figures 1 to 24 As shown, the present application provides a coal gangue classification and grabbing method based on image recognition, comprising the following steps: constructing a coal gangue target detection model for identifying and classifying coal and coal gangue; constructing a coal gangue grabbing detection model for generating the optimal grabbing posture of the coal gangue identified by the coal gangue target detection model; and building a coal gangue grabbing planning system for controlling the robot to move to the posture position under the robot coordinate system according to the optimal grabbing posture to perform the task of grabbing the identified coal gangue. In this way, the present application can realize automatic identification and automatic grabbing of coal gangue, has the beneficial technical effects of reducing labor cost, reducing misjudgment loss, reducing environmental pollution, reducing governance cost, high identification efficiency of coal gangue, high identification accuracy, high grabbing efficiency of coal gangue, high grabbing accuracy, and improving the automation level of coal production.

[0024] Optionally, constructing the coal gangue target detection model comprises the following steps: preparing an image recognition data set, the image recognition data set comprising coal images, coal gangue images and corresponding label files; training the target recognition network using the image recognition data set, and improving the detection scale and loss function of the target recognition network to obtain the constructed coal gangue target detection model. In this way, the coal gangue target detection model obtained by training and improving optimization has higher identification efficiency and higher identification accuracy for coal and coal gangue.

[0025] In the specific embodiments of the present application, two categories of coal and gangue are set up, and each picture is labeled to indicate which category the object in the picture belongs to.

[0026] Optionally, the step of making the image recognition dataset specifically includes the following steps: taking a plurality of coal images in a closed environment; taking a plurality of gangue images in the closed environment; taking a plurality of images of coal and gangue mixture in the actual production process of raw coal; preprocessing the plurality of coal images, the plurality of gangue images, and the plurality of images of coal and gangue mixture; labeling the plurality of coal images, the plurality of gangue images, and the plurality of images of coal and gangue mixture respectively to obtain a plurality of one-to-one corresponding label files; and the preprocessed plurality of coal images, the plurality of gangue images, the plurality of images of coal and gangue mixture, and the plurality of one-to-one corresponding label files form the image recognition dataset.

[0027] The image recognition dataset can help improve the efficiency and safety of coal mine production, reduce the mixing of coal and gangue, and also help promote the application and development of machine learning technology in the industrial field. The image recognition dataset contains two categories: coal and gangue.

[0028] The closed environment refers to using light-blocking cloth to form a closed space, and then selecting LED as an auxiliary light source to be arranged in the closed space, so as to prevent natural light from interfering with the experiment.

[0029] In the specific embodiments of the present application, when collecting the gangue images, 3000 images are obtained through cooperation with the mining area, including 1200 coal images taken in a closed environment, 1200 gangue images, and 600 images of coal and gangue mixture taken in the actual production process. The coal images are as shown in Figure 8 The gangue images are as shown in Figure 9 The images of coal and gangue mixture are as shown in Figure 10The LabelImg software is used to label each picture, and after labeling, an xml file corresponding to each picture is generated, which saves the class and position information of the object in the picture. The labeled picture is converted into the Pascal VOC2007 data set format, which includes the JPEGImages folder of the training picture and the test picture. The xml file corresponding to the picture name in the JPEGImages folder is saved in the Annotations folder, which is used to store the labeling information. The ImageSets folder contains text information for training, testing and verification, which are stored in train.txt, test.txt and val.txt respectively. For the prepared 3000 picture data set, a random division is performed, in which the training set accounts for 60%, the test set accounts for 20%, and the verification set accounts for 20%. When training the model using the NVIDIA RTX3080 graphics card, 500 training rounds, 16 samples per batch, and an initial learning rate of 0.01 are set. In order to speed up the training process, PyTorch is used as a deep learning framework, and CUDA-11.1 is used to accelerate it.

[0030] Optionally, the plurality of coal images, the plurality of coal gangue images, and the plurality of images of the mixture of coal and coal gangue are preprocessed in one or more of the following ways: denoising, size unification, scaling, cropping, graying, histogram equalization, normalization, and data enhancement.

[0031] In the optional embodiments provided in the present application, the preprocessing of the plurality of coal images, the plurality of coal gangue images, and the plurality of images of the mixture of coal and coal gangue specifically includes the following steps: (1) image denoising: using median filtering and Gaussian filtering to reduce noise in the image. (2) image size unification: adjusting all input images to the same size, which can ensure that they have the same feature dimension. (3) image scaling: scaling the image by a certain proportion, usually scaling the image to a fixed size to meet the input requirements of the model. (4) image cropping: if there are irrelevant edges or backgrounds in the image, the image can be cropped to only keep the region of interest. (5) image graying: converting a color image to a grayscale image, which helps to reduce the dimension of the data and may be more suitable for certain tasks. (6) histogram equalization: enhancing the contrast and brightness of the image by adjusting the distribution of image pixels to make them more uniform. (7) normalization: normalizing the image to scale the pixel values to a fixed range (such as between 0 and 1) to better adapt to the input of the model. (8) data enhancement: expanding the data set by applying transformations such as rotation, translation, flipping, scaling, etc. to increase the diversity of the data and improve the generalization ability of the model.

[0032] Optionally, in the step of constructing the coal gangue target detection model, the YOLOv5s network is selected as the target recognition network. The YOLOv5s network has the advantages of high efficiency and high accuracy, and can realize fast and accurate target detection.

[0033] Optionally, the improvement of the detection scale of the target recognition network specifically includes the following steps: simplifying the output end of the YOLOv5s network structure, deleting the 19x19 feature map branch in the output end, and retaining the 76x76 feature map and the 38x38 feature map as the output. By deleting the 19x19 feature map branch, the model detection speed is further improved, and the consumption of computing resources is reduced, thereby achieving the purpose of model optimization.

[0034] Specifically, the network model of the YOLOv5s network is composed of an input end, a backbone network, a feature extraction network and an output end. The output end of the YOLOv5s network before improvement includes a 19x19 feature map, a 76x76 feature map and a 38x38 feature map. In the output feature map, different scale feature maps correspond to different size receptive fields, that is, different perception abilities for different regions of the input image. Through analysis of the network structure, it is found that the 19x19 feature map has the largest receptive field and is more suitable for detecting large objects. However, in the present application, coal gangue is mostly medium and small target, so taking the 76x76 and 38x38 feature maps as the output is more in line with the actual demand. Therefore, by deleting the 19x19 feature map branch, the model detection speed can be improved without affecting the recognition effect, the consumption of computing resources is reduced, and the purpose of model optimization is achieved. The YOLOv5s network structure before improvement is shown in Figure 11 , and the improved network structure is shown in Figure 12 .

[0035] Optionally, the improvement of the loss function of the target recognition network specifically includes the following steps: selecting the CIoU loss function as the loss function of the model training. The main advantage of the CIoU loss function is that it can comprehensively consider the distance between rectangular frames, the degree of overlap of rectangular frames and the aspect ratio of rectangular frames, so it can more accurately measure the regression effect of the target detection frame. Therefore, the present application selects the CIoU loss function as the loss function of the model training to improve the performance and stability of target detection.

[0036] Specifically, before the improvement, the YOLOv5s network adopts a GIoU loss function as the loss function for model training, but the GIoU loss function is prone to slow or unstable convergence when processing high-aspect-ratio target boxes. Therefore, the loss function is improved in the present application, and the GIoU loss function is replaced by a CIoU loss function. The CIoU loss function not only considers the aspect ratio and the difference between the overlapping parts of the detection boxes, but also introduces an angle measurement, and considers the difference in the aspect ratio of the target box, so that when processing high-aspect-ratio targets, the convergence speed is faster and more stable. In addition, the CIoU loss function can better handle small targets and overlapping targets. Coal and gangue in the data set are small targets, so the effect of using the CIoU loss function for processing is better.

[0037] The model before the loss function is improved has a large fluctuation in the fitting process, and the convergence speed is slow. The loss value of the model before the improvement is about 3.94%. The model after the loss function is improved has a small fluctuation in the fitting process, and the convergence speed is faster. The loss value of the improved model is about 3.37%. Compared with the previous one, the error of the improved model is reduced by 0.57%.

[0038] Recall rate, precision rate, accuracy rate and rate are used as evaluation indexes of the target recognition network, as shown in Figure 13 The performance of the improved coal and gangue target detection model is better than that before the improvement. The position, confidence and category information subscribed by the coal and gangue target detection model are as shown in Figure 14

[0039] Optionally, the construction of the coal and gangue grabbing detection model includes the following steps: preparing a simulation data set, the simulation data set including coal and gangue simulation image data and a corresponding label file; training the pose estimation network using the coal and gangue simulation data set, and integrating an attention mechanism and a multi-scale residual block into the pose estimation network to obtain a constructed coal and gangue grabbing detection model. In this way, the coal and gangue grabbing detection model obtained after training and improvement has higher grabbing efficiency and higher grabbing accuracy.

[0040] ​Optionally, the method for making the coal gangue simulation dataset specifically comprises the following steps: modeling according to the actual size of coal and coal gangue and selecting graphite properties in each material property by using SolidWorks software; generating a urdf model of the coal gangue by using a sw2urdf plug-in in the SolidWorks software; adding the urdf model of the coal gangue to a Gazebo model in a workspace of ROS; calling a depth camera to collect multiple coal gangue simulation image data, and saving the image size to be a preset size; preprocessing the multiple coal gangue simulation image data; labeling the multiple coal gangue simulation image data to obtain multiple one-to-one corresponding label files; and forming the coal gangue simulation dataset by the multiple processed coal gangue simulation image data and the multiple one-to-one corresponding label files.

[0041] In the specific embodiments of the present application, the pose estimation network is trained by using the Cornell dataset and the coal gangue simulation dataset. The data is randomly divided into a training set and a test set, wherein the training set accounts for 80%, and the test set accounts for 20%. The training is performed on an NVIDIA RTX3080 graphics card by using a PyTorch deep learning framework, the batch processing size is 8, the initial learning rate is 0.01, and the number of training rounds is 100. At the same time, Cuda-11.1 is used for training acceleration.

[0042] In the specific embodiments of the present application, the D435 depth camera is selected to collect images. 180 pieces of coal gangue simulation image data are collected by calling the D435 depth camera, and the image size is saved as 640*480. The collected coal gangue simulation image data is labeled by using the RoLabelImg labeling tool, the coal gangue is labeled as gangue, and a rectangular capture frame with a rotation angle is labeled. After labeling the rectangular frame, the angle is rotated according to the actual capture requirement. The label file is stored in xml format, and the file content includes the center point (cx, cy) of the rectangular frame, the height w, the width h and the rotation angle angle and other information.

[0043] Optionally, the multiple coal gangue simulation image data is preprocessed in one or more of the following ways: size unification, normalization and data enhancement.

[0044] In the optional embodiments provided in the present application, the preprocessing of the multiple coal gangue simulation image data specifically comprises the following steps: (1) image size unification: adjusting all input images to the same size, which can ensure that they have the same feature dimension; (2) normalization: normalizing the image to scale the pixel value to a fixed range (for example, between 0 and 1), so as to better adapt to the input of the model; (3) data enhancement: expanding the dataset by applying rotation, translation, flipping, scaling and other transformations to increase the diversity of the data and improve the generalization ability of the model.

[0045] Optionally, in the step of constructing the coal gangue grabbing detection model, a GR-ConvNet network is selected as the pose estimation network. Compared with other material detection networks, the GR-ConvNet network has better operation efficiency and better accuracy, and can meet the real-time coal gangue grabbing requirements of the industry. As shown in Figure 18 , the GR-ConvNet network used in the present application has higher pose detection accuracy and higher grabbing success rate compared with the CNN network.

[0046] Specifically, as shown in Figure 15 , the generative residual convolutional neural network model (GR-ConvNet network) uses multi-layer convolution and residual structure to extract features from input images and generate pixel-level representations of grabbed objects. The main components of the model include convolution layers, residual layers, and convolution transpose layers, etc. The convolution layer is used to learn the spatial and semantic features of the image, which can extract features of different scales and levels, gradually improve the abstract level of the features, and thus enhance the representation ability of the model; the residual layer can alleviate the problems of gradient disappearance and dimension error, thereby improving the accuracy and stability of the model; the convolution transpose layer restores the size of the image to the input size through upsampling operation, preserving the spatial features and interpretability of the image; through the synergistic effect of these parts, the model can accurately capture the details in the input image and generate accurate pixel-level representations, providing strong support for the robot grabbing task.

[0047] Optionally, the attention mechanism is integrated into the pose estimation network, which specifically includes the following steps: embedding an attention module in the network structure of the GR-ConvNet, as shown in Figure 16 , the attention module includes a channel attention submodule and a spatial attention submodule. By adding the CBAM module to the GR-ConvNet model, the attention mechanism is introduced, which helps the model to focus on relevant features and improves its representation learning ability.

[0048] Optionally, the multi-scale residual block is integrated into the pose estimation network, which specifically includes the following steps: embedding a multi-scale residual block in the network structure of the GR-ConvNet. Thus, the model has strong ability to process features of different scales, improving the representation ability of the model.

[0049] Specifically, as shown in Figure 17As shown, the multi-scale residual block is mainly divided into two components: fusion of multi-scale features and local residual learning. In the multi-scale feature fusion, different scale feature maps are extracted through multiple downsampling methods, and then the features are made more significant through upsampling and addition operations. Local residual learning divides the input feature map into multiple sub-blocks, then performs residual learning on each sub-block, and combines the local differences with the global features to improve the discriminability and expressiveness of the features. This network architecture has strong ability to process features of different scales, improves the representation ability of the model, and thus shows good results in image feature detection.

[0050] Optionally, in the network structure of the GR-ConvNet, the multi-scale residual block and the attention module are sequentially arranged, and the output of the multi-scale residual block is transmitted through the CBAM module.

[0051] In the specific embodiments of the present application, the attention module CBAM is defined as an independent class, which includes a channel attention submodule and a spatial attention submodule. Then, the forward method of the GR-ConvNet model is modified, and an instance of the CBAM class is added after the multi-scale residual block (Multi-Scale Residual Blocks). In the forward method of the model, the output of the multi-scale residual block is transmitted through the CBAM module to integrate the attention mechanism. Continue the forward propagation of the model, including other layers and outputs. The multi-scale residual block is defined as an independent class, including necessary convolution and batch normalization layers. The GR-ConvNet model is modified by adding an instance of the multi-scale residual block class in the architecture of the model. In the forward method of the model, the output of each multi-scale residual block is transmitted through the appropriate instance. Continue the forward propagation of the model, including other layers and outputs.

[0052] As Figure 19 shown, compared with the model before improvement, the accuracy can be improved by 2.1 percentage points by only integrating the attention mechanism; the accuracy can be improved by 1.4 percentage points by only using the multi-scale residual block; and the accuracy can be improved by 2.9 percentage points by integrating the attention mechanism and the multi-scale residual block. The experimental results show that the improvement scheme proposed in the present application can effectively improve the performance of the pose detection model.

[0053] The structure of the GR-ConvNet model is shown in Figure 20 , which shows the whole process of generating the optimal grasping pose of the GR-ConvNet model. As Figure 21 shown, the grasping quality, angle and width output by the GR-ConvNet model, and the inferred grasping rectangle are shown.

[0054] Optionally, as Figures 4 to 7As shown, the image recognition-based coal gangue classification and grabbing method further includes the following steps: calibrating the coal gangue grabbing coordinate system, including a camera calibration step and a hand-eye calibration step, the camera calibration step including: establishing the mapping relationship between the pixel coordinate system, the image coordinate system, the camera coordinate system, and the world coordinate system; using Zhang Zhengyou calibration method for calibration, in the calibration process, the internal parameter matrix and distortion coefficients k1, k2, p1, p2, k3 of the camera are calculated by using the corner position data; the internal parameters and distortion coefficients of the camera are calculated to correct the image and improve the quality and accuracy of the image; the hand-eye calibration step includes: establishing the relative position and attitude relationship between the robot end effector and the vision sensor. By calibrating the coal gangue grabbing coordinate system, the robot can accurately perform various tasks according to the visual information.

[0055] Using the traditional calibration method in Zhang Zhengyou calibration method, it is not necessary to understand the motion process of the object, only the same fixed camera is used to shoot black and white checkerboard images at different positions, such as Figure 6 As shown, the camera calibration can be completed, and this method has low cost, easy implementation, and high calibration accuracy. Specifically, the monocular camera is placed in a fixed position, and the attitude of the calibration board is adjusted one by one, and the images of the calibration board in multiple different attitudes are shot and saved to cover different angles and distances in the camera field of view. The shooting of 9 pictures is shown in Figure 6 As shown. Figure 4 The camera imaging model is shown, which describes the mapping process from the three-dimensional coordinates of the object to the two-dimensional coordinates of the image. Figure 5 The transformation process from the physical imaging plane to the pixel coordinate system, the image coordinate system, the camera coordinate system, and the world coordinate system is shown. Specifically, the imaging process of the camera is from the coordinates (X w ,Y w ) of a certain coal gangue P point in the known world coordinate system, to obtain the corresponding coordinates (X c ,Y c ) of P point in the camera coordinate system through rigid transformation, further to obtain the corresponding coordinates (X, Y) of P point in the image coordinate system through perspective projection, and finally to obtain the pixel coordinates (u, v) through discretization. As shown in Figure 7As shown, in order to carry out hand-eye calibration, a mechanical arm is used to grab the round dot calibration board, and the mechanical arm is controlled to move within the field of view of the camera to ensure that the entire calibration board is within the field of view of the camera. Then, the camera takes images of the calibration board one by one. In the hand-eye calibration system, four different coordinate systems are established: the camera coordinate system (CAM), the calibration board coordinate system (CAL), the robot base coordinate system (BASE), and the robot end coordinate system (TOOL). The coordinate transformation relationship is calculated to determine the conversion relationship of the camera coordinate system relative to the robot base coordinate system, so as to convert the target pose in two-dimensional space into a three-dimensional space pose that can be used by the robot arm, which helps the robot to complete the grasping task.

[0056] Optionally, the coal gangue grasping detection model is evaluated, and a rectangular metric index is used as a standard for evaluating candidate grasping accuracy. When the accuracy evaluation result of the parameter grasping appears the following two cases, it can be concluded that the grasping configuration is correct: (1) the angle difference between the predicted grasping box g p and the real grasping box g t is within 30 degrees; (2) the J(g p , g t ) of the predicted grasping box g p and the real grasping box g t is greater than 0.25;

[0057] wherein g p is the predicted grasping box, and g t is the real grasping box. The rectangular metric index is used as a standard for evaluating candidate grasping accuracy, and the success probability of grasping detection is measured by considering the complete grasping rectangle. If the parameters of the predicted grasping box and the real grasping box are within a certain interval, then the grasping detection is successful.

[0058] When evaluating the grasping detection output by the network, the previous evaluation index uses point measurement, which calculates the distance between the center points of the predicted grasping box and the real grasping box to determine whether the grasping is successful. If the distance is less than a certain threshold, the grasping is considered successful. However, this index does not consider the influence of the grasping angle or size, and the evaluation effect is not good.

[0059] Optionally, the coal gangue grasping planning system is built, including the following steps: building a coal gangue grasping planning system based on the ROS framework and the Ubuntu environment; for example Figure 23As shown, a robot grasping simulation platform was built using the Gazebo simulation environment. Combined with the motion planning module of MoveIt software, the optimal posture calculation and grasping operation for grasping gangue were realized. The ros_control function was used to control the simulated robot to execute the grasping action. In this way, the simulation experiment can verify whether the algorithm is executable. If the effect is not good, the algorithm can be further improved and trained, thereby improving the grasping accuracy and efficiency.

[0060] Optionally, the image recognition-based coal gangue classification and grasping method further includes the following steps: acquiring the image to be detected; using a pre-built coal gangue target detection model to identify and classify the two targets, coal and coal gangue, in the image to be detected; the coal gangue target detection model feeds back the identified coal gangue category information and transmits it to the coal gangue grasping and detection environment; using the pre-built coal gangue grasping and detection model to generate the optimal grasping posture for the identified coal gangue; converting the optimal grasping posture into a grasping posture vector in the robot coordinate system through a coordinate system transformation method; and sending instructions from the host computer to the controller to control the robot to move to the posture position and execute the grasping task.

[0061] Optionally, the robot's posture and position can be calculated by solving the robot's forward and inverse kinematic equations.

[0062] Optionally, generating the optimal grasping posture for the identified coal gangue using a pre-built coal gangue grasping and detection model includes the following steps: inputting the image to be detected I into the coal gangue grasping and detection model; I = R n×h×w h represents the height, w represents the width, and n represents the number of channels. The coal gangue grasping detection model extracts features from the image to be detected I and generates a grasping success rate image, an angle image, and a width image. The grasping success rate image, angle image, and width image are pixel-level representations of the grasped object of the same size as the input image. The coal gangue grasping detection model infers the optimal grasping posture from the grasping success rate image, angle image, and width image, and outputs the posture vector G of the robot gripper in the image coordinate system. i G i =(x,y,Θ) i W i ,Q); where x and y are the center coordinates of the object in the image coordinate system, in pixels; Θ i The orientation or posture of the robot's gripper is represented by Euler angles or quaternions, with rotation angles expressed in radians in the camera reference coordinate system, Θ. i The range of values ​​for W is [-π / 2, π / 2]; irepresents the opening width of the robot gripper for generating the distance between the robot fingers in pixels; Q is a score for evaluating the grasping quality; Q ranges from 0 to 1, the closer Q is to 1, the higher the possibility of successful grasping; it is required that W i ≤ W max , W max is the maximum width of the robot using the anti-point gripper; the optimal grasping pose is converted into a grasping pose vector in the robot coordinate system by a coordinate system conversion method, including the following steps: the grasping pose vector in the robot coordinate system is represented as G r , G r = (P, Θ r , W r , Q), wherein P = (x, y, z) is the center position of the tool, Θ r is the rotation of the tool around the Z axis, W r is the width required by the tool, and Q is the grasping quality score; the coordinate system conversion is realized by the following formula: G r = T rc (T ci (G i )); wherein T ci is represented as a transformation matrix from the robot end effector coordinate system to the camera coordinate system, which describes the position and direction of the camera in the robot coordinate system; T rc is represented as a transformation matrix from the camera coordinate system to the world coordinate system, which describes the position and direction of the object seen by the camera in the world coordinate system; G r is the grasping pose vector in the robot coordinate system; G i is the pose vector of the robot gripper in the image coordinate system.

[0063] According to another aspect of the present application, the present application provides an image recognition-based coal gangue classification and grabbing system, comprising: a target detection module, the target detection module comprising a pre-constructed coal gangue target detection model, the coal gangue target detection model being used for identification and classification of two targets of coal and coal gangue; a pose detection module connected with the target detection module, the target detection module feeding back coal gangue category information identified to the pose detection module; the pose detection module comprising a pre-constructed coal gangue grabbing detection model, the coal gangue grabbing detection model being used for generating an optimal grabbing pose of the identified coal gangue; a grabbing execution module connected with the pose detection module, the pose detection module feeding back the optimal grabbing pose to the grabbing execution module, the grabbing execution module converting the optimal grabbing pose into a grabbing pose vector in a robot coordinate system through a coordinate system conversion method; the grabbing execution module comprising a robot, the robot being used for moving to a pose position and executing a grabbing task. The image recognition-based coal gangue classification and grabbing system provided by the present application can realize automatic identification and grabbing sorting of coal gangue, has beneficial technical effects of reducing labor cost, reducing misjudgment loss, reducing environmental pollution, reducing treatment cost, high identification efficiency of coal gangue, high identification accuracy, high grabbing efficiency of coal gangue, high grabbing accuracy, and improving the automation level of coal production.

[0064] Optionally, the image recognition-based coal gangue classification and grabbing system further comprises a depth camera, which is used for acquiring a to-be-detected image in real time. In the specific embodiments provided by the present application, the depth camera adopts a D435 depth camera, the robot adopts a UR5 mechanical arm and a Robotiq2F-85 adaptive gripper. The D435 camera can provide high-resolution images and depth information, and the vision processor of the D435 camera can provide efficient data processing and transmission, so as to realize fast and accurate object detection, tracking and identification. The D435 depth camera can also be connected with a computer or other devices through a USB interface, so as to facilitate configuration and integration. The UR5 mechanical arm has advantages of flexibility, ease of use and low cost, and can realize high-precision and high-efficiency production operation. The UR5 mechanical arm is also combined with a ROS open source system, so as to control the mechanical arm. The Robotiq2F-85 adaptive gripper has characteristics of high efficiency, precision and reliability.

[0065] As shown in FIG. 1, Figure 24 In the specific embodiments of the image recognition-based coal gangue classification and grabbing system provided by the present application, the image recognition-based coal gangue classification and grabbing system comprises a computer, a controller, a mechanical arm, a gripper, a depth camera and a coal gangue collection box, and the pre-constructed coal gangue target detection model and the coal gangue grabbing detection model are stored in the computer.

[0066] The coal gangue grabbing system based on image recognition provided in the application also designs an actual grabbing experiment platform to simulate the recognition and grabbing environment of the coal gangue on the desktop. The image recognition technology is used to accurately recognize the coal gangue and obtain the shape, size, position and other information of the coal gangue, so as to provide accurate data basis for the grabbing operation. Finally, the best grabbing posture of the coal gangue is obtained through the grabbing detection network to realize effective and stable grabbing. Through the image recognition algorithm, the target detection can not only give the classification result of the coal and the coal gangue, but also provide the semantic information and positioning information of the objects contained in the picture. These information helps the robot to distinguish the target objects and the background, so as to realize more intelligent operation. The rectangular frame as a positioning means is especially suitable for some scenes, which can help the robot to quickly find the target grabbing object. Using target detection to find a smaller detection range helps to reduce the interference of information irrelevant to the target object and prepares for subsequent grabbing posture detection. With the continuous development of artificial intelligence technology, combined with computer vision technology, it is applied to the coal gangue grabbing detection, which not only saves human resources, but also greatly improves the detection efficiency and accuracy. The application integrates artificial intelligence and industry, making the raw coal sorting technology more automated and intelligent. The application provides a coal gangue grabbing detection method and system based on image recognition, which can effectively improve the speed and accuracy of recognizing and classifying coal and coal gangue, and the attention mechanism and multi-scale residual block are integrated into the pose estimation network to improve the accuracy of the grabbing coal gangue target posture. The application also carries out simulation environment experiment and real environment experiment. During the experiment, the mechanical arm first moves to the initial pose, and the gripper is in the open state; after the control end of the mechanical arm receives the grabbing pose information of the target object, the robot is driven to move above the target object, and the angle of the last joint is adjusted to the yaw angle of the corresponding grabbing pose; the mechanical arm moves downward until it reaches the grabbing pose; the gripper is closed to implement the grabbing action; the mechanical arm grabs the target object and lifts it upward; the mechanical arm transports the target object above the preset placement point; the mechanical arm moves downward to the object placement point; the gripper is opened to release the object, and one object grabbing action is completed.

[0067] As Figure 1As shown, the coal gangue grabbing system based on image recognition provided by the present application contains a research technical route of 3-part tasks. First, in the target detection part, the image preprocessing and the construction of the data set required by the network model are completed, and in the target detection model improvement stage, the basic YOLOv5 is optimized to improve the detection performance of the image recognition gangue; secondly, in the pose detection stage, the calibration work of the gangue coordinate system is carried out based on the ROS environment, and the basic grabbing detection network is optimized to improve the accuracy of the model's pose detection; finally, in the grabbing execution stage, the generated optimal pose is converted into the target attitude required by the mechanical arm through coordinate system conversion. Through the use of inverse kinematics, the execution of grabbing is realized in the simulation environment through ros_control. As shown in the figure, Figure 2 As shown, the system is composed of a vision system and a robot control mechanical arm system. The servo control based on visual feedback is a control method that obtains the position and attitude information of the object in the scene through a vision sensor to realize the control of the mechanical arm grabbing. The servo control based on visual feedback generally includes the following steps: first, the position and attitude information of the target object is obtained through image recognition using a camera or other sensors, and if the image recognition accuracy meets 90% or more, the execution continues; secondly, the position and attitude information of the target object is converted into the motion control command of the robot by applying the robot kinematics and control theory; next, the motion control command of the robot is transmitted to the robot controller to execute the motion control of the robot; finally, after the grabbing task is completed, the grabbing result of the robot is detected through sensors and other means, and if the grabbing fails and does not meet the grabbing index, the grabbing task is re-executed. As shown in the figure, Figure 3 As shown, the process of grabbing execution includes the following steps: first, the optimal gangue grabbing attitude is generated by using the grabbing pose detection network algorithm; secondly, the detected pose is converted into the attitude based on the robot coordinate system through the coordinate system conversion method; finally, the controller is instructed by the upper computer to move the robot to a specific attitude position to execute the grabbing task. As shown in the figure, Figure 4 As shown, the camera imaging model describes the mapping process from the three-dimensional coordinates of the object to the two-dimensional coordinates of the image. The conversion flow chart of coal gangue from the world coordinate system to the pixel coordinate system is as shown in the figure Figure 5 As shown, the images of the calibration board in different directions in the camera calibration experiment are as shown in the figure Figure 6 As shown, the target is to adjust its attitude of the calibration board, and the Zhang Zhengyou calibration method is used to perform the calibration process, the internal parameters and distortion coefficients of the camera are calculated to correct the image, and the quality and accuracy of the image are improved. The hand-eye schematic diagram collected is as shown in the figure Figure 7 As shown, the Eye-to-Hand hand-eye calibration can obtain the coordinate system transformation relationship between the camera and the robot arm, so as to convert the target attitude in the two-dimensional space into the three-dimensional space attitude that can be used by the robot arm, which is helpful for the robot to complete the grabbing task.Figure 16 As shown, the attention module (Convolutional Block Attention Module, CBAM) is composed of two sub-modules: channel attention module (CAM) and spatial attention module (SAM). These sub-modules can be used to introduce attention mechanisms in convolutional neural networks to improve their representation capabilities. Among them, CAM is used to weight process feature maps of different channels to better capture the relationship between different features; SAM is used to weight process the spatial position of feature maps to better capture the relationship between different spatial positions. The combination of CAM and SAM can achieve global weighting processing of feature maps, improving the effect of feature expression. The framework of the gangue simulation grasping system is as shown in Figure 23 As shown, in order to verify the performance of the grasping detection system under different scenarios, a random gangue scenario generation method is designed to simulate the diversity of gangue distribution in the real world. By testing the system under these randomly generated gangue scenarios, it is verified that the system has good grasping ability and stability under various conditions.

[0068] In an optional embodiment of the present application, a coal gangue grabbing detection method based on image recognition is proposed, which includes the following steps: S1: coal gangue grabbing system construction and coordinate calibration; specifically, in S1, the coal gangue grabbing system construction specifically includes: in the gangue grabbing system process, it is divided into three parts of target detection, pose detection and grabbing execution. The target detection task is to detect the position of the gangue by using the depth camera and the target detection algorithm, and to obtain its depth and color information. Through the target detection algorithm, the recognition and classification of the gangue target are realized. The grabbing detection task is to determine the optimal gangue grabbing pose through the grabbing pose detection network with the help of the information captured by the camera. The process of grabbing execution includes the following steps: first, use the grabbing pose detection network algorithm to generate the optimal gangue grabbing pose; second, convert the detected pose into the pose based on the robot coordinate system through the coordinate system conversion method; finally, send instructions to the controller through the upper computer to make the robot move to a specific pose position and execute the grabbing task. Specifically, in S1, the coordinate calibration of the system is divided into two parts: camera calibration and hand-eye calibration. The first part is camera calibration, which is to determine the mapping relationship between the physical imaging plane and the world coordinate system, including the transformation process of pixel coordinate system, image coordinate system and camera coordinate system. Start the calibration process, calculate the internal parameter matrix and distortion coefficients k1, k2, p1, p2, k3 of the camera using the corner position data. Generally, Zhang Zhengyou calibration method is used to calibrate the process, and the internal parameters and distortion coefficients of the camera are calculated to correct the image and improve the quality and accuracy of the image; the second part is hand-eye calibration, which is to establish the relative position and attitude relationship between the robot end effector (hand) and the vision sensor (eye). Through this calibration, the robot can accurately execute various tasks according to the visual information. S2: make an image recognition dataset, complete the training of two target recognition networks, select the YOLOv5s network with better performance, and improve the detection scale and loss function. Specifically, in S2, the application scenario of the image recognition dataset is coal and gangue recognition, which can be applied in coal mines and other places. This dataset can help improve the efficiency and safety of coal production, reduce the mixing of coal and gangue, and also contribute to the application and development of machine learning technology in the industrial field. The dataset contains two categories: coal and gangue. In this collection of gangue images, 3000 pictures were obtained through cooperation with the mining area, including 1200 coal pictures, 1200 gangue pictures, and 600 coal-gangue mixture pictures taken in the actual production process. In the output feature map, different scale feature maps correspond to different size receptive fields, i.e. different perception abilities for different regions of the input image. Through analysis of the network structure, it is found that the 19x19 feature map has the largest receptive field and is more suitable for detecting large objects.However, in practical applications, most of the gangue objects in the experimental dataset are small to medium-sized targets. Therefore, using 76×76 and 38×38 feature maps as outputs is more in line with actual needs. This application further improves the model detection speed and reduces the consumption of computational resources by simplifying the network structure and deleting the 19×19 feature map branch, thereby achieving the purpose of model optimization. The main advantage of the CIoU loss function is that it can comprehensively consider the distance between rectangles, the degree of overlap between rectangles, and the aspect ratio of the rectangles, thus more accurately measuring the regression effect of the target detection box. Therefore, this application selects the CIoU loss function as the main loss function for model training to improve the performance and stability of target detection. S3: Construct a grasping detection model, use SolidWorks software to create a coal and gangue simulation dataset for annotation and preprocessing, and train and optimize the pose detection network model on the dataset. Specifically, in S3, firstly, in SolidWorks software, models should be created according to the actual size of coal and gangue, and graphite properties should be selected for each material property. Secondly, the sw2urdf plugin in SolidWorks software was used to generate a URDF model of the gangue. Finally, this model was added to the Gazebo model in the ROS workspace, and a D435 depth camera was used to acquire 180 images, saving each image at a size of 640×480 pixels. After data acquisition, the dataset was labeled. The GR-ConvNet pose estimation network is a grasping point-based model that can directly predict the dense grasping pose of the object to be grasped using depth information and provide a grasping quality estimate for each grasping point. The robot grasping problem is defined as predicting the anti-joint grasping of an unknown object from n-channel images of the scene and performing it on the robot. The grasping pose of this model in the robot frame is represented as: G. r =(P,Θ r W r Q0, where P = (x, y, z) is the center position of the tool, Θ r It is the rotation of the tool around the Z-axis, W r Where I is the width required by the tool, and Q is the grasp quality score. The basic formal definition of grasp object detection is: its input I = R n×h×w Let G be an image with height h, width w, and number of channels n. The goal is to detect the pose vector G of the robot's gripper from this image. i Its G i Represented as: G i =(x,y,Θ) i W i ,Q); where G irepresents the pose vector of the robot gripper, containing information such as the position, orientation, size, and quality score of the graspable object. x and y are the center position coordinates of the graspable object in the image coordinate system, in pixels; Θ i represents the orientation or pose of the robot gripper, usually described by Euler angles or quaternions, with rotation angles typically in radians in the camera reference coordinate system; W i represents the opening width of the robot gripper, used to determine the distance between the robot fingers, in pixels; Q is the score used to evaluate the grasp quality. Based on this definition, the grasp detection algorithm needs to identify a graspable object from the input image I and output the corresponding G i vector to guide the robot to perform the grasping action. In the grasp pose estimation network algorithm, the grasp quality score Q, the rotation angle Θ i and the width W i of the graspable object are all key parameters for describing and evaluating the quality and suitability of the graspable object in the image. The grasp quality score Q represents the grasp quality at each point in the image, with a value range of 0 to 1, and when the value is close to 1, it indicates a higher probability of successful grasping. This value can be calculated by evaluating the relative position, orientation, and geometric features of the graspable object and the surrounding objects. The rotation angle Θ i of the graspable object represents the measurement of the inverse angle rotation required at each point for grasping. This rotation angle is an important parameter for determining the grasping direction and grasping pose, with a value range of [-π / 2, π / 2]. In the grasp detection algorithm, it is usually necessary to estimate the optimal rotation angle at each grasp point to ensure that the robot has sufficient stability and adaptability during grasping. The width W i of the graspable object represents the distance that the gripper needs to open during grasping, in pixels. It is usually necessary to estimate the optimal width based on the depth information of the grasp point to ensure that the robot can fully cover the graspable object and successfully grasp it. Parameter estimation and calculation in the grasp target pose detection algorithm are its core part, which requires knowledge and technology from multiple fields such as image processing, machine learning, computer vision, etc. W max is the maximum width of the anti-point gripper, that is, the maximum distance that the gripper used by the robot can open. When estimating the width of the graspable object, it is necessary to ensure that the estimated value does not exceed this maximum width limit to avoid the robot being unable to complete the grasping action. When performing grasping operations on the robot, it is necessary to convert the position, orientation, and width of the graspable object detected in the image space to the reference coordinate system of the robot. This conversion can be achieved through the following formula: G r = T rc (T ci (G i )) where G irepresents the pose vector of the robot gripper, including information such as the position, orientation, opening width and grasping mass of the gripper; G r is the grasping pose vector in the robot reference coordinate system; T ci represents the transformation matrix from the robot end effector coordinate system to the camera coordinate system, describing the position and orientation of the camera in the robot coordinate system; T rc represents the transformation matrix from the camera coordinate system to the object coordinate system, describing the position and orientation of the object seen by the camera in the world coordinate system. In the gangue grasping system, in order to generate a pixel-level representation of the grasped object, the model uses two elements to extract angle information and output different values, thereby combining the required angle. This representation helps the robot determine the direction and stability of the grasping point, thereby improving the success rate and efficiency of grasping. Among them, the model can output a pixel-level representation of the grasped object with the same size as the input image, which is used for the grasping operation of the robot. The grasping success rate, angle and width images generated by the grasping pose detection network are used to infer the grasping pose of the object. S4: Build a grasping planning system based on the ROS framework and Ubuntu environment, which can realize the grasping task of the robot arm. Through the forward and inverse kinematics equations of the robot arm, the pose and position information of the robot arm can be calculated, thereby realizing the motion process of the robot arm. Specifically, in S4, a grasping planning system based on the ROS framework and Ubuntu environment is built, which can realize the grasping task of the robot arm. A robot arm grasping experiment platform is built using the Gazebo simulation environment, and combined with the motion planning module of the MoveIt software, the optimal pose calculation and grasping operation of the gangue grasping are realized. Finally, the ros_control function is used to control the simulation robot arm to perform the grasping action. Among them, the pose result is mapped to the robot base coordinate, verifying the practicability and feasibility of this mapping method in the robot grasping planning task. In the experiment, gangue and coal are placed on the table for grasping experiments, and the results show that the robot grasping detection algorithm used in the present application has a high success rate, which can meet the requirements of the grasping task.

[0069] The beneficial effects of the present application at least include: in view of the problems of low recognition rate and low sorting accuracy in gangue image recognition and grasping pose detection in robot grasping. Through the research on robot vision and grasping pose detection, a gangue grasping system framework based on image recognition is designed, and a gangue grasping experiment platform is built. A target detection network model is designed, and in order to improve the detection speed and accuracy, two improvement strategies of optimizing network detection scale and loss function are proposed. The results show that the improved target detection algorithm of the present application improves the accuracy by 2.1% and the detection rate by 5.7FPS. The grasping detection pose estimation network model is established, and the attention mechanism and multi-scale residual block are integrated into the basic grasping pose detection network to improve the accuracy of pose detection. The results show that the improved grasping pose detection model of the present application can achieve an accuracy of 95.1%.

[0070] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for classifying and grasping coal gangue based on image recognition, characterized in that, Includes the following steps: A coal gangue target detection model was constructed for the identification and classification of two targets: coal and coal gangue. A coal gangue grasping and detection model is constructed to generate the optimal grasping posture of the coal gangue identified by the coal gangue target detection model; A coal gangue grasping planning system is built to control the robot to move to the posture position in the robot coordinate system according to the optimal grasping posture to perform the task of grasping and identifying coal gangue. The image recognition-based coal gangue classification and grasping method also includes the following steps: Acquire the image to be detected; The pre-built coal gangue target detection model is used to identify and classify coal and coal gangue targets in the image to be detected. The coal gangue target detection model feeds back the identified coal gangue category information and transmits it to the coal gangue grasping and detection environment; The optimal grasping posture for the identified coal gangue is generated using a pre-built coal gangue grasping and detection model. The optimal grasping posture is converted into a grasping posture vector in the robot coordinate system through a coordinate system transformation method. The host computer sends instructions to the controller to move the robot to the stated posture position and perform the grasping task. The process of generating the optimal grasping posture of the identified coal gangue using a pre-constructed coal gangue grasping and detection model includes the following steps: inputting the image to be detected I into the coal gangue grasping and detection model; , For height, For width, The number of channels is specified. The coal gangue grasping detection model extracts features from the image to be detected (I) and generates a grasping success rate image, an angle image, and a width image. The grasping success rate image, angle image, and width image are pixel-level representations of the grasped object of the same size as the input image. The coal gangue grasping detection model infers the optimal grasping posture from the grasping success rate image, angle image, and width image, and outputs the posture vector of the robot gripper in the image coordinate system. ; ;in, and These are the center coordinates of the captured object in the image coordinate system, in pixels. This represents the direction or orientation of the robot's gripper, described using Euler angles or quaternions. The rotation angle is expressed in radians in the camera's reference coordinate system. The range of values ​​is ; This represents the opening width of the robot's gripper, used to generate the distance between the robot's fingers, in pixels. It is a score used to evaluate the quality of the capture; The value range is from 0 to 1. The closer the value is to 1, the higher the probability of a successful capture; requirements , The maximum width of the reverse-point gripper used by the robot.

2. The coal gangue classification and grasping method based on image recognition according to claim 1, characterized in that, The construction of the coal gangue target detection model includes the following steps: An image recognition dataset is created, which includes coal images, coal gangue images, and their corresponding label files; The target recognition network was trained using an image recognition dataset, and the detection scale and loss function of the target recognition network were improved to obtain a well-constructed coal gangue target detection model.

3. The coal gangue classification and grasping method based on image recognition according to claim 2, characterized in that, The creation of the image recognition dataset specifically includes the following steps: taking multiple images of coal in a closed environment; taking multiple images of coal gangue in a closed environment; taking multiple images of a mixture of coal and coal gangue during the actual coal production process; preprocessing the multiple coal images, multiple coal gangue images, and multiple images of a mixture of coal and coal gangue; labeling the multiple coal images, multiple coal gangue images, and multiple images of a mixture of coal and coal gangue to obtain multiple one-to-one corresponding label files; the preprocessed multiple coal images, multiple coal gangue images, multiple images of a mixture of coal and coal gangue, and multiple one-to-one corresponding label files form the image recognition dataset. In the step of constructing the coal gangue target detection model, the YOLOv5s network is selected as the target recognition network; The improvement of the detection scale of the target recognition network specifically includes the following steps: simplifying the output of the YOLOv5s network structure, deleting the 19×19 feature map branch in the output, and retaining the 76×76 feature map and the 38×38 feature map as output; The improvement of the loss function of the target recognition network specifically includes the following steps: selecting the CIoU loss function as the loss function for model training.

4. The coal gangue classification and grasping method based on image recognition according to claim 1, characterized in that, The construction of the coal gangue grabbing and detection model includes the following steps: A simulation dataset is created, which includes simulated coal gangue image data and its corresponding label files; The pose estimation network was trained using a coal gangue simulation dataset, and an attention mechanism and multi-scale residual blocks were incorporated into the pose estimation network to obtain a well-constructed coal gangue grasping and detection model.

5. The coal gangue classification and grasping method based on image recognition according to claim 4, characterized in that, The specific steps for creating a coal gangue simulation dataset include: Modeling coal and gangue using SolidWorks software based on their actual dimensions, and selecting graphite as the material property in each attribute; using the sw2urdf plugin in SolidWorks to generate a URDF model of the gangue; adding the URDF model of the gangue to the Gazebo model in the ROS workspace; acquiring multiple simulation images of the gangue using a depth camera, saving the images at a preset size; preprocessing the multiple simulation images of the gangue; labeling the multiple simulation images of the gangue to obtain multiple one-to-one corresponding label files; and finally, the processed multiple simulation images of the gangue and the multiple one-to-one corresponding label files together form the coal gangue simulation dataset. In the step of constructing the coal gangue grabbing and detection model, the GR-ConvNet network is selected as the pose estimation network; The integration of an attention mechanism into the pose estimation network specifically includes the following steps: embedding an attention module into the GR-ConvNet network structure, wherein the attention module includes a channel attention submodule and a spatial attention submodule; The specific steps of incorporating multi-scale residual blocks into the pose estimation network include: embedding multi-scale residual blocks into the GR-ConvNet network structure.

6. The coal gangue classification and grasping method based on image recognition according to claim 1, characterized in that, The image recognition-based coal gangue classification and grasping method further includes the following steps: The coordinate system for coal gangue grasping is calibrated, including camera calibration and hand-eye calibration steps. The camera calibration step includes: establishing the mapping relationship between the pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system; using the Zhang Zhengyou calibration method for calibration; during the calibration process, calculating the camera's internal parameter matrix and distortion coefficients k1, k2, p1, p2, k3 using corner point position data; and correcting the image by calculating the camera's internal parameters and distortion coefficients. The hand-eye calibration step includes: establishing the relative position and attitude relationship between the robot's end effector and the vision sensor. The coal gangue grabbing and detection model was evaluated using a rectangular metric as the standard for assessing candidate grabbing accuracy. The grabbing configuration can be considered correct when the accuracy evaluation results for the parameters show the following two scenarios: (1) Predicting the capture box Compared to the real capture box The angle difference is within 30 degrees; (2) Predicting the capture box Compared to the real capture box of Greater than 0.25; ； in, It predicts the capture box. It is a real capture box.

7. The coal gangue classification and grasping method based on image recognition according to claim 1, characterized in that, The construction of the coal gangue grabbing and planning system includes the following steps: Build a coal gangue grabbing and planning system based on the ROS framework and Ubuntu environment; A robot grasping simulation experimental platform was built using the Gazebo simulation environment. Combined with the motion planning module of MoveIt software, the optimal posture calculation and grasping operation for grasping gangue were realized. The ros_control function was used to control the simulated robot to perform the grasping action.

8. The coal gangue classification and grasping method based on image recognition according to claim 1, characterized in that, The process of converting the optimal grasping posture into a grasping posture vector in the robot coordinate system using a coordinate system transformation method includes the following steps: The grasping posture vector in the robot coordinate system is represented as follows: , ,in, It is the center position of the tool. It is a tool around Rotation of the axis It is the width required by the tool. This refers to the quality score; coordinate system transformation is achieved using the following formula: ;in, It is represented as a transformation matrix from the robot end effector coordinate system to the camera coordinate system, describing the position and orientation of the camera in the robot coordinate system; It is represented as a transformation matrix from the camera coordinate system to the world coordinate system, describing the position and orientation of the object seen by the camera in the world coordinate system; This is the grasping posture vector in the robot coordinate system; This is the pose vector of the robot's gripper in the image coordinate system.

9. A system for a coal gangue classification and grasping method based on image recognition as described in claim 1, characterized in that, include: The target detection module includes a pre-built coal gangue target detection model, which is used to identify and classify two targets: coal and coal gangue. The pose detection module is connected to the target detection module. The target detection module feeds back the identified coal gangue category information to the pose detection module. The pose detection module includes a pre-built coal gangue grasping detection model, which is used to generate the optimal grasping posture for the identified coal gangue. The grasping execution module is connected to the pose detection module. The pose detection module feeds back the optimal grasping posture to the grasping execution module. The grasping execution module converts the optimal grasping posture into a grasping posture vector in the robot coordinate system through a coordinate system transformation method. The grasping execution module includes a robot, which is used to move to the posture position and perform the grasping task.

Images