A method for tracking and detecting gangue based on improved YOLOv5s and Deepsort

Abstract

This invention discloses a gangue tracking and detection method based on improved YOLOv5s and Deepsort, comprising the following steps: S1, acquiring a sample image dataset of coal and gangue mixtures; S2, constructing an improved YOLOv5s network model, training the improved YOLOv5s network model using the sample image dataset to obtain a gangue detection model; S3, using the gangue detection model to detect real-time coal and gangue images, outputting the location information and target bounding boxes of the gangue; S4, using the deepsort algorithm to perform real-time tracking and prediction of the gangue location information based on the gangue location information and target bounding boxes; S5, sending the gangue location information output by the deepsort algorithm to a gangue sorting robot to sort the gangue. This invention identifies the location information of the gangue and outputs the target bounding boxes through the gangue detection model, and predicts the movement trajectory of the gangue on the conveyor belt through the deepsort algorithm, which can avoid the same gangue being repeatedly detected, thus improving the accuracy of gangue sorting by the gangue sorting robot.

Description

Technical Field

[0001] This invention relates to the field of coal mining technology, and in particular to a method for tracking and detecting gangue based on improved YOLOv5S and Deepsort. Background Technology

[0002] Coal is a crucial basic energy source and strategic resource in my country. Due to my country's energy structure of being "rich in coal, poor in oil, and lacking in gas," coal will remain the mainstay of China's energy consumption for a considerable period. According to 2022 energy statistics, coal consumption accounted for approximately 56% of China's total energy consumption. As coal mining continues, environmental problems are becoming increasingly apparent, impacting the greening, intelligentization, and sustainable development of the coal industry. The quality of coal products is of great significance for reducing overcapacity in the coal industry, adjusting and upgrading the industrial structure, improving the environment, and building a resource-saving and environmentally friendly society. One of the key factors affecting coal quality is the presence or absence of coal gangue. Therefore, effectively separating gangue from coal can not only prevent damage to equipment during transportation caused by large pieces of gangue, such as belt tearing, misalignment, and blockage, but also reduce coal production and maintenance costs and subsequent transportation expenses.

[0003] Currently, the most commonly used methods for gangue selection include manual selection, wet separation, and X-ray identification. Manual selection is inefficient, prone to missing certain gangue, and the harsh working environment can negatively impact worker health, failing to meet the requirements of intelligent mining development. Wet separation easily leads to water waste. While X-ray identification offers advantages such as high efficiency and speed, its application is limited by radiation side effects.

[0004] In recent years, with the development of artificial intelligence and computers, image recognition technology has been gradually applied to the detection of coal gangue. However, existing image processing methods only sort gangue by its texture and do not track its position. When the recognition information works in conjunction with the gangue sorting robot, if there is a delay in information and misselection occurs, the conveyor belt may even be damaged due to changes in position information. Summary of the Invention

[0005] The technical problem this invention aims to solve is the inaccuracy of existing methods in sorting coal gangue. This invention provides a gangue tracking and detection method based on an improved YOLOv5s and Deepsort algorithm. The improved YOLOv5s model identifies the location information of the gangue, and the Deepsort algorithm tracks the location trajectory of the gangue. This trajectory is then sent to a gangue sorting robot for sorting, thereby improving the detection accuracy of gangue sorting.

[0006] The technical solution adopted by this invention to solve its technical problem is: a gangue tracking and detection method based on improved YOLOv5S and Deepsort, comprising the following steps:

[0007] Step S1: Obtain a dataset of sample images of a mixture of coal and gangue;

[0008] Step S2: Construct an improved YOLOv5s network model and train the improved YOLOv5s network model using the sample image dataset to obtain a gangue detection model.

[0009] Step S3: Use the gangue detection model to detect the real-time image of coal gangue and output the location information and target box of the gangue;

[0010] Step S4: Based on the location information of the gangue and the target box, the deepsort algorithm is used to perform real-time tracking and prediction of the location information of the gangue;

[0011] Step S5: Send the location information of the gangue output by the deepsort algorithm to the gangue sorting robot to sort out the gangue.

[0012] Furthermore, the construction of the improved YOLOv5s network model specifically includes:

[0013] A YOLOv5s base network was built, and the feature pyramid structure in the YOLOv5s base network was replaced with the OPA-FPN search module. The loss function was set to CIoU_Loss and the activation function was set to SiLu.

[0014] Furthermore, the search paths of the OPA-FPN search module include: top-down path, bottom-up path, residual connection path, and fusion split path.

[0015] Furthermore, the formula for the CIoU_Loss function is: in, In the formula, v represents the influence factor, wA and hA represent the length and height of the true bounding box, wB and hB represent the length and height of the predicted bounding box, c represents the length of the connection line of the minimum bounding rectangle, and ρ(A ctr B ctr ) represents the Euclidean distance between the center points of the ground truth bounding box and the predicted bounding box.

[0016] Furthermore, the formula for the SiLu function is: f(x) = x × sigmoid(x), where, x represents a variable.

[0017] Furthermore, in step S4, the deepsort algorithm is used to track and predict the location information of the gangue in real time, including:

[0018] Predict the location information of gangue in the next frame image using Kalman filtering;

[0019] The Hungarian algorithm is used to match the gangue in the current frame image with the gangue in the next frame image, and the movement trajectory of the gangue is output.

[0020] Furthermore, the movement trajectory of the gangue is sent to the gangue sorting robot for sorting.

[0021] Furthermore, the training process in step S2 includes:

[0022] The sample image dataset is divided into a training set, a validation set, and a test set;

[0023] The improved YOLOv5s network model is trained using the training set, validated using the validation set, and tested using the test set until the output of the improved YOLOv5s network model meets the accuracy requirements.

[0024] The improved YOLOv5S network model that meets the output accuracy requirements is set as the gangue detection model.

[0025] Furthermore, the ratio of the training set, validation set, and test set is 7:2:1.

[0026] The beneficial effects of this invention are that the gangue tracking and detection method based on the improved YOLOv5s and Deepsort can improve the speed and efficiency of model detection by improving the YOLOv5s network model. The gangue detection model identifies the location information of the gangue and outputs the target box. The location information and target box are synchronized to the Deepsort algorithm to predict the movement trajectory of the gangue on the conveyor belt. The movement trajectory is sent to the gangue sorting robot for sorting. This can avoid the same gangue being detected repeatedly, which is beneficial to improving the accuracy of gangue sorting by the gangue sorting robot. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] Figure 1 This is a flowchart of the gangue tracking and detection method of the present invention.

[0029] Figure 2 This is an architecture diagram of the YOLOv5S basic network of this invention.

[0030] Figure 3 This is a schematic diagram of the search path of the OPA-FPN search module of the present invention.

[0031] Figure 4 This is a schematic diagram of the recognition results of the present invention. Detailed Implementation

[0032] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0033] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, features defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0034] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0035] like Figures 1 to 4 As shown, the present invention provides a gangue tracking and detection method based on improved YOLOv5s and Deepsort, comprising the following steps: Step S1: Obtain a sample image dataset of coal and gangue mixtures. Step S2: Construct an improved YOLOv5s network model and train it using the sample image dataset to obtain a gangue detection model. Step S3: Detect real-time coal and gangue images using the gangue detection model, and output the location information and bounding boxes of the gangue. Step S4: Based on the location information and bounding boxes of the gangue, use the Deepsort algorithm to perform real-time tracking and prediction of the gangue's location information. Step S5: Send the gangue location information output by the Deepsort algorithm to a gangue sorting robot to sort the gangue.

[0036] In other words, this invention first uses a gangue detection model to identify the position information of gangue in the conveyor belt and the target bounding box for selecting gangue from real-time images of coal gangue. Then, the position information and target bounding box of gangue are synchronized to the deepsort algorithm. The deepsort algorithm predicts the position of gangue in the next moment based on the position information of gangue, forming a gangue movement trajectory. The gangue sorting robot then sorts out the gangue from the conveyor belt according to the gangue movement trajectory. Compared with the prior art, this method can realize the tracking of gangue position, avoid the same gangue being repeatedly detected in different positions, which would lead to the gangue sorting robot making decision errors, thereby improving the accuracy and precision of gangue sorting.

[0037] It should be noted that the sample image dataset consists of 2000 images of coal gangue captured by a high-speed industrial camera. Gangue and coal are then manually labeled in these sample images. Gangue and coal differ in color and texture, making them distinguishable to the human eye. After labeling the sample image dataset, it is divided into training, validation, and test sets, for example, in a 7:2:1 ratio. The improved YOLOv5s network model is then trained using the training set, validated using the validation set, and tested using the test set until its output meets the accuracy requirements. This improved YOLOv5s network model is then designated as the gangue detection model. In other words, when a real-time image of coal gangue is input into the trained gangue detection model, the model can use bounding boxes to identify the gangue in the real-time image and output its location information (e.g., its coordinates).

[0038] Specifically, in step S2, an improved YOLOv5s network model is constructed, including: building a YOLOv5s base network, replacing the feature pyramid structure in the YOLOv5s base network with an OPA-FPN search module, setting the loss function to CIoU_Loss, and setting the activation function to SiLu. As shown in the figure, the YOLOv5s base network includes an input, a backbone network, a Neck structure, and an output. The Neck structure in the YOLOv5s base network uses a feature pyramid structure (FPN+PAN), which, although simple and efficient, has low processing efficiency and weak adaptability. Therefore, in this invention, the feature pyramid structure is replaced with an OPA-FPN (One-shot Path Aggregation FPN) search module. The OPA-FPN search module can search for the best subnetwork from the supernetwork, thereby improving the computational processing speed. For example, the search paths of the OPA-FPN search module include: top-down path, bottom-up path, residual connection path, and fusion split path. Among them, the residual connection path and the fusion split path are the empty information path and the residual connection path, respectively. They do not participate in the calculation and are mainly used to reduce the model complexity and speed up the model fitting speed.

[0039] For example, the formula for the CIoU_Loss function is: in, In the formula, v represents the influence factor, wA and hA represent the length and height of the true bounding box, wB and hB represent the length and height of the predicted bounding box, c represents the length of the connection line of the minimum bounding rectangle, and ρ(A ctr B ctr The distance () represents the Euclidean distance between the center points of the ground truth bounding box and the predicted bounding box. The CIoU_Loss function of this invention adds an influence factor v, considering both the center distance and overlap rate of the ground truth bounding box and the predicted bounding box, which can more accurately reflect the loss during model training, thereby improving the training effect. It should be noted that both the ground truth bounding box and the predicted bounding box here refer to the target bounding box. The formula for the SiLu function is: f(x) = x × sigmoid(x), where... x represents a variable. This invention uses the SiLu function as the activation function, which makes the network smoother.

[0040] After format conversion, the trained gangue detection model can be deployed to the edge computing module. The edge computing module controls a high-speed industrial camera to acquire real-time video images of coal gangue on the conveyor belt and sends them to the built-in gangue detection model for detection and identification. In step S4, the real-time tracking and prediction of gangue position information using the deepsort algorithm includes: predicting the position information of gangue in the next frame image using Kalman filtering; matching gangue in the current frame image with gangue in the next frame image using the Hungarian algorithm, and outputting the movement trajectory of gangue. Then, the movement trajectory of gangue is sent to the gangue sorting robot for gangue sorting. Kalman filtering is an algorithm that uses the state equation of a linear system to optimally estimate the system state through system input and output observation data. The Hungarian algorithm finds the maximum number of matches by continuously searching for augmenting paths, that is, matching the target box with the tracking box (the tracking box predicted by Kalman filtering) to find the optimal match, thereby achieving tracking. In other words, the deepsort algorithm can be used to track the location of gangue in real time, preventing the same gangue from being identified repeatedly during movement, thereby improving the accuracy of gangue detection and sorting.

[0041] In summary, the gangue tracking and detection method based on improved YOLOv5s and Deepsort of this invention improves the speed and efficiency of gangue detection by modifying the YOLOv5s network model. The gangue detection model identifies the location information of the gangue and outputs the target box. The location information and target box are synchronized to the Deepsort algorithm to predict the movement trajectory of the gangue on the conveyor belt. The movement trajectory is then sent to the gangue sorting robot for sorting. This avoids the same gangue being detected repeatedly and helps to improve the accuracy of gangue sorting by the gangue sorting robot.

[0042] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined by the scope of the claims.

Claims

1. A method for tracking and detecting gangue based on improved YOLOv5s and Deepsort, characterized in that, Includes the following steps: Step S1: Obtain a dataset of sample images of a mixture of coal and gangue; Step S2: Construct an improved YOLOv5s network model and train the improved YOLOv5s network model using the sample image dataset to obtain a gangue detection model. Step S3: Use the gangue detection model to detect the real-time image of coal gangue and output the location information and target box of the gangue; Step S4: Based on the location information of the gangue and the target box, the deepsort algorithm is used to perform real-time tracking and prediction of the location information of the gangue; Step S5: Send the location information of the gangue output by the deepsort algorithm to the gangue sorting robot to sort out the gangue; The construction of the improved YOLOv5s network model specifically includes: A YOLOv5s base network is built, and the feature pyramid structure in the YOLOv5s base network is replaced by the OPA-FPN search module to search for the optimal subnetwork from the super network, thereby improving the computation speed. The loss function is set to CIoU_Loss function and the activation function is set to SiLu function. The search paths of the OPA-FPN search module include: top-down path, bottom-up path, residual connection path, and fusion split path; In step S4, the deepsort algorithm is used to track and predict the location information of the gangue in real time, including: Predict the location information of gangue in the next frame image using Kalman filtering; The Hungarian algorithm is used to match the gangue in the current frame image with the gangue in the next frame image, and output the movement trajectory of the gangue. The movement trajectory is sent to the gangue sorting robot to sort the gangue, so as to avoid the same gangue being detected repeatedly.

2. The gangue tracking and detection method as described in claim 1, characterized in that, The formula for the CIoU_Loss function is: ,in, , In the formula, Indicates the impact factor. and This represents the length and height of the actual bounding box. and This represents the length and height of the prediction box. This represents the length of the connection line of the minimum bounding rectangle. This represents the Euclidean distance between the center points of the ground truth bounding box and the predicted bounding box.

3. The gangue tracking and detection method as described in claim 1, characterized in that, The formula for the SiLu function is: ,in, , Represents a variable.

4. The gangue tracking and detection method as described in claim 1, characterized in that, The movement trajectory of the gangue is sent to the gangue sorting robot for sorting.

5. The gangue tracking and detection method as described in claim 1, characterized in that, The training process in step S2 includes: The sample image dataset is divided into a training set, a validation set, and a test set; The improved YOLOv5s network model is trained using the training set, validated using the validation set, and tested using the test set until the output of the improved YOLOv5s network model meets the accuracy requirements. The improved YOLOv5S network model that meets the output accuracy requirements is set as the gangue detection model.

6. The gangue tracking and detection method as described in claim 5, characterized in that, The ratio of the training set, validation set, and test set is 7:2:1.

Images