Artificial intelligence video identification belt load coal quantity monitoring method

Abstract

The invention belongs to the technical field of coal mine safety engineering and artificial intelligence, and discloses an artificial intelligence video identification belt load coal quantity monitoring method, and the specific technical scheme is as follows: collecting monitoring images of underground coal gangue and a coal flow belt conveyor; all macroscopic features are marked, and targets are abandoned for coal gangue and coal flow which are difficult to distinguish by naked eyes; a data set is divided by using K-fold cross, different models and parameters are evaluated and compared, and an optimal model and parameter combination is selected; decomposing the image into different sub-images by adopting a contrast-limited adaptive histogram equalization algorithm, and suppressing image noise; a YOLOv5 network post-processing structure is improved, a coal flow threshold judgment algorithm is introduced to carry out non-contact detection on the belt coal quantity load existing in a monitoring video, finally, a coal quantity judgment result is output, coal quantity load detection is achieved, the judged load state serves as the output of the algorithm, and a decision basis is provided for an intelligent control system of the belt conveyor.

Description

Technical Field

[0001] The invention belongs to the fields of coal mine safety engineering technology and artificial intelligence technology, and specifically relates to an artificial intelligence video recognition belt load coal quantity monitoring method. Background Art

[0002] After coal is mined underground, it must be transported to the ground through a large number of belt conveyors. Therefore, the operating conditions of the belt conveyors directly affect the production capacity and efficiency of the coal mine. However, in the current transportation process, there are problems such as no-load and light-load causing waste of electricity, large workload of manual monitoring, low accuracy, and high cost. Generally, there are some discontinuous situations in the coal mining process, that is, the amount of coal mined varies, and even no coal is mined in some time periods. Therefore, during the transportation process, the amount of coal transported by the belt conveyor is not uniform. If the belt conveyor keeps running at high speed, it will consume a lot of electricity. Therefore, it is very important to detect the amount of coal loaded on the belt conveyor.

[0003] At present, there are two methods for measuring the amount of coal transported by belt conveyors: one is contact type, that is, the measuring sensor is in direct contact with the coal; the other is non-contact type, that is, the coal is measured using other measuring devices. In the contact measurement method, the measuring sensor is easily hit and scratched by the coal, and the measuring sensor is easily damaged, resulting in inaccurate detection results. Therefore, the non-contact measurement method is more widely used.

[0004] At present, in coal mines, managers monitor coal conveyor belts through surveillance videos. In the daily management of coal conveyor belts, data on the coal flow on the conveyor belts is often needed. Traditional video surveillance requires employees to stay in front of the screen at all times. With the continuous increase in surveillance video time and the continuous expansion of the monitoring scope, coupled with the complex environment of coal mines, such as high dust, all-weather lighting of mining lamps, headlights, etc., the workload of employees will inevitably increase, which is very easy to cause visual fatigue and greatly reduce the accuracy of manual resolution. Summary of the invention

[0005] In order to solve the technical problems existing in the prior art, the present invention provides a method for monitoring the light, medium and heavy loads of belt coal based on improved YOLOv5, which realizes data set acquisition, data set annotation, data set splitting, image preprocessing, model training and tuning, and post-processing, and finally obtains the coal quantity monitoring results.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is: an artificial intelligence video recognition belt load coal quantity monitoring method, the specific steps are as follows:

[0007] Step 1: Dataset collection: Collect monitoring images of underground coal gangue and coal flow belt conveyors at different working time periods, different lighting conditions, and different coal loads on the belt conveyors;

[0008] Step 2: Dataset annotation: All features visible to the naked eye are annotated, and the targets of gangue and coal flow that are difficult to distinguish with the naked eye are discarded. Fine-grained annotation is performed according to the detection requirements;

[0009] Step 3: Dataset splitting: Use K-fold cross-validation to divide the dataset. Based on K-fold cross-validation, evaluate and compare different models and parameters to select the best model and parameter combination.

[0010] Step 4: Image preprocessing: Use the contrast-limited adaptive histogram equalization algorithm to decompose the image into different sub-images, enhance the sub-image contrast and suppress image noise;

[0011] Step 5. Model training, tuning and post-processing: Select the YOLOv5 model as the coal flow target detection model, set the learning rate, batch processing size, number of iterations, and activation function, introduce contrastive learning loss joint training, improve the YOLOv5 network post-processing structure, introduce a pixel area-based ratio algorithm to calculate the belt area covered by coal flow and coal gangue, and combine the actual belt coal load light load, medium load, and heavy load conditions. The coal flow threshold judgment algorithm is introduced to perform non-contact detection of the belt coal load in the monitoring video, and finally output the coal volume judgment result.

[0012] In step 2, the specific steps of marking are:

[0013] Step S21, marking the coal flow target blocked by the external working environment interference, and drawing the edge of the blocked part according to the edge shape in the unblocked scene;

[0014] Step S22, marking targets visible to human eyes, where the target pixel range is greater than 20*20;

[0015] Step S23, only mark the objects that can be distinguished by human eyes, and discard the objects that are difficult to distinguish by human eyes;

[0016] Step S24: Ensure that the frame coordinates are not on the image boundary, and discard target frames with extreme aspect ratios.

[0017] In step 4, the adaptive histogram equalization algorithm includes four parts: contrast stretching, histogram equalization, contrast limitation and adaptive histogram equalization, where:

[0018] Contrast stretching: Image contrast is defined as the difference between the brightest pixel and the darkest pixel in the image. The Michelson quantitative measurement formula is as follows:

[0019]

[0020] Among them, I max Indicates the brightest pixel, Imin Indicates the darkest pixel. When the grayscale of the whitest pixel and the blackest pixel of an image are both 128, the image contrast is the lowest, C = 0; when the grayscale of the whitest pixel of an image is 255 and the grayscale of the blackest pixel is 0, the image contrast is the highest, C = 1.0; when the grayscale value of the image is concentrated in the interval (120, 180), a piecewise linear function is used to map the grayscale value from (120, 180) to (0, 255) to achieve contrast stretching;

[0021] Histogram equalization: Use the histogram cumulative distribution function to stretch the original image to obtain a rectangular histogram;

[0022] Contrast limit: Set the histogram distribution threshold, and evenly distribute the distribution exceeding the set threshold to the probability density distribution to limit the increase of the cumulative histogram;

[0023] Adaptive histogram equalization: calculate the grayscale histogram and the corresponding cumulative histogram of each sub-region of the image; if the sub-image is located at the top corner of the image and there are only two adjacent sub-images, grayscale mapping is performed according to the transformation function of the sub-image; if the sub-image is located at the four sides of the image, that is, there are three adjacent sub-images, it is transformed according to the transformation function of the two adjacent sub-images and then linearly interpolated to obtain the grayscale equalization; if the sub-image has four adjacent sub-images, it is transformed according to the transformation function of the four adjacent sub-images and then bilinearly interpolated to obtain the grayscale equalization.

[0024] In step five, the specific processing steps are:

[0025] Step S51, InfoNCE loss function tuning: introduce contrastive learning loss to supervise network training, learn model parameters for learning feature representation by comparing the similarity between positive samples and negative samples, and improve the contrastive learning loss based on noise contrast estimation. Noise contrast estimation regards the probability estimation problem as a basic binary classification, that is, the classification object only contains objects and noise, and uses noise contrast to improve the estimation of the model. The specific formula is expressed as follows:

[0026]

[0027] Among them, x is the data sample, is the noise sample, P data (x) represents the distribution from real data, P model (x) represents the model prediction, P noise (x) is represented as a noise distribution; a contrastive learning loss is proposed, assuming that the feature selection reference vector q is a representation vector related to the task, and the key vector {k0, k1...k K} represents the information index of each position in the input sequence. The specific formula of InfoNCE loss is as follows:

[0028]

[0029] Among them, k0 is the sample associated with the query vector q, k1...k K is a sample that is unrelated to the query vector, τ is a temperature parameter used to adjust the smoothness of the similarity score and control the model's discrimination of negative samples. InfoNCE loss maximizes the similarity between positive samples and the query vector and minimizes the similarity between negative samples and the query vector.

[0030] Step S52, coal quantity monitoring post-processing: In the detection stage, after the model is inferred and the inference target is obtained, the target with a confidence level greater than 0.75 is first selected as the basis for judging the gangue and coal flow load, and the target coordinates are extracted according to the category, and the detection coordinates are mapped and transformed according to the original image pixel ratio;

[0031] If there is no overlapping relationship between the extracted targets, the pixel area of ​​the detected target is directly calculated based on the pixel value of the detected target conversion coordinates, and the target pixel area is included in the total target area; if there is an overlapping relationship between the targets, the area union is calculated, and the union pixel area is included in the total target pixel area; finally, based on the ratio of the total target pixel area to the belt pixel area in the image, the light load, medium load, and heavy load thresholds of the coal are defined and judged, and the results are output and printed to the original image.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention utilizes artificial intelligence video recognition technology for monitoring, which has the advantages of safety, strong stability, and high efficiency.

[0034] 2. The present invention is based on the mainstream YOLOv5 detection model, and combines the image preprocessing algorithm on it. It uses the contrast-constrained adaptive histogram equalization algorithm to process the image and extract significant features such as the edge contour and RGBA grayscale value of the coal flow. Then, the YOLOv5 deep neural network is trained by combining the InfoNCE loss function to detect the coal flow target. Finally, the inference post-processing judgment based on pixel-level proportion calculation is introduced to realize the coal load detection of the belt conveyor, and the judged load state is used as the output of the algorithm to provide a decision-making basis for the intelligent control system of the belt conveyor.

[0035] 3. The present invention uses artificial intelligence video to identify the light, medium and heavy load monitoring of the belt coal volume, which can avoid the control system decision loss caused by inaccurate monitoring of the contact sensor due to collision, aging and other reasons, and at the same time can reduce the daily manpower maintenance cost of the contact sensor and improve production efficiency. At the same time, artificial intelligence video recognition can avoid the risk of decreased manual monitoring resolution accuracy caused by the increase in monitoring video time, the expansion of monitoring range and the complex underground environment, improve the efficiency of underground operations, greatly reduce human and financial costs, and improve the safety of underground operations. Therefore, the social benefits obtained by the artificial intelligence video recognition method for monitoring the light, medium and heavy load of the belt coal volume are outstanding and worthy of promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of the belt coal quantity monitoring method based on improved YOLOv5.

[0037] Figure 2 Schematic diagram of K-fold cross splitting of the data set.

[0038] Figure 3 Schematic diagram of the histogram equalization and contrast limitation principles.

[0039] Figure 4 Flowchart for object detection inference.

[0040] Figure 5 Flowchart of post-detection processing.

[0041] Figure 6 This is a schematic diagram of the actual underground scene test results. Figure 6 (a) is a schematic diagram of the belt light load test results. Figure 6 (b) is a schematic diagram of the belt medium load test results. Figure 6 (c) is a schematic diagram of the belt heavy load test results. DETAILED DESCRIPTION

[0042] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] like Figure 1 As shown, an artificial intelligence video recognition belt load coal quantity monitoring method, the specific steps are as follows:

[0044] S1. Dataset collection: Download the camera history records containing the belt conveyor scene, and randomly extract frames from the historical video to save images for the complex lighting environment and low-probability emergencies underground. Filter and save images, including monitoring images collected at different working time periods, different lighting conditions, and different loads of coal on the belt conveyor, to ensure that the collected images can cover all possible disturbances to the belt work outside the belt working area, working conditions on the belt, and lighting conditions at different positions of the belt, to ensure the diversity and representativeness of the materials, so that subsequent deep learning can learn more features.

[0045] S2. Dataset annotation: Use labelme polygon annotation tool to annotate the coal flow targets on the belt, and try to fit the edge of the object as accurately as possible and follow the following annotation rules:

[0046] 1. Mark the coal flow targets that are blocked due to interference from the external working environment (the blocked range is less than half of the overall coal flow area), and the edges of the blocked parts are drawn according to the edge shapes in the unblocked scene.

[0047] 2. Mark small targets visible to the human eye, where the pixel range of small targets should be larger than 20*20.

[0048] 3. Only mark the targets that can be distinguished by the human eye and discard the targets that are difficult to distinguish by the human eye.

[0049] 4. Ensure that the frame coordinates are not on the image boundary to prevent out-of-bounds errors during data loading or data expansion, and discard target frames with extreme aspect ratios.

[0050] S3. Dataset splitting: Use K-fold cross validation to divide the collected data set into 5 subsets, randomly select a subset as the validation set, and the remaining partitions as the training set. Finally, get the results of 5 models, average the results of each model, get the average result of the model, and take the model with the best average result among the 5 models as the final model. The principle of data set splitting is shown in Figure 2 .

[0051] S4. Image preprocessing: In traditional image processing, image feature matching includes feature detection and extraction, feature description, and feature matching. When the underground belt is transporting coal, the camera captures the blurred image and there is strong light above the belt, which is not conducive to image feature extraction. Therefore, the CLAHE algorithm of the bilinear interpolation scheme is used to preprocess the image. The preprocessing process is shown in Figure 3 .

[0052] Strong illumination of the image will cause the shadow of the detection target to deepen, thereby obscuring the local texture and structure of the target object. The contrast-limited adaptive histogram equalization algorithm (CLAHE) is used to reduce the impact of illumination and blur on the image. The CLAHE algorithm includes four parts: contrast stretching, histogram equalization, contrast limitation and adaptive histogram equalization.

[0053] S41. Contrast stretching: Image contrast is defined as the difference between the brightest pixel and the darkest pixel in the image. The Michelson quantitative measurement formula is as follows:

[0054]

[0055] Among them, I max Indicates the brightest pixel, I min Indicates the darkest pixel. When the grayscale of the whitest pixel and the blackest pixel of an image are both 128, the image contrast is the lowest, C = 0; when the grayscale of the whitest pixel of an image is both 255 and the grayscale of the blackest pixel is 0, the image contrast is the highest, C = 1.0. When the grayscale value of the image is concentrated in the interval (120, 180), a piecewise linear function is used to map the grayscale value from (120, 180) to (0, 255) to achieve contrast stretching.

[0056] S42, histogram equalization: using the histogram cumulative distribution function to stretch the original image to obtain a rectangular histogram;

[0057] S43, Contrast limitation: For areas with concentrated grayscale, the histogram will be stretched to sparseness, which will increase the image contrast and become noise. Moreover, after the histogram is equalized, more detailed information will be lost in the area with concentrated grayscale. Therefore, the CLANE algorithm sets a histogram distribution threshold and evenly distributes the distribution exceeding the set threshold to the probability density distribution, thereby limiting the increase of the cumulative histogram. See the principle of histogram equalization and contrast limitation for details. Figure 4 .

[0058] S44, adaptive histogram equalization: If the original image is divided into multiple sub-images and the above transformation is performed, discontinuous transformation between regions will occur, which will affect image processing. Use bilinear interpolation to reduce the discontinuous changes between regions. For each sub-region of the image, first calculate its grayscale histogram and the corresponding cumulative histogram. If the sub-image is located at the top corner of the image, that is, there are only two adjacent sub-images, then grayscale mapping is performed according to the transformation function of the sub-image in which it is located; if the sub-image is located on the four sides of the image (excluding the top corner), that is, there are three adjacent sub-images, then it is transformed according to the transformation function of the two adjacent sub-images and then linear interpolation is performed to obtain it; if the sub-image has four adjacent sub-images, then it is transformed according to the transformation function of the four adjacent sub-images and then bilinear interpolation is performed to obtain it.

[0059] S5, YOLOv5 network training, tuning and post-processing:

[0060] S51, InfoNCE loss function tuning: The present invention introduces contrastive learning loss (InfoNCE Loss) to supervise network training. Contrastive learning loss is a loss function used for self-supervised learning. It learns model parameters by comparing the similarities between positive samples and negative samples, and is used for learning feature representation or characterization learning. Contrastive learning loss is improved based on noise contrast estimation (NCE). Noise contrast estimation regards the probability estimation problem as a basic binary classification, that is, the classification object only contains objects and noise, and uses noise contrast to improve the estimation of the model. The specific formula is as follows:

[0061]

[0062] Among them, x is the data sample, is the noise sample, P data (x) represents the distribution of real data, P model (x) represents the model prediction, P noise (x) is represented by the noise distribution, usually uniform distribution or Gaussian distribution.

[0063] In the case of multiple complex noises, the noise samples cannot be simply classified into one category. It is more reasonable to classify multiple noises into multiple categories. At this time, the model transforms from a simple binary classification problem to a multi-classification problem. Multi-classification means that negative samples are classified into multiple categories. Therefore, contrastive learning loss is proposed. It is assumed that the feature selection reference vector q is the representation vector related to the task, and the key vector {k0, k1...k K} represents the information index of each position in the input sequence, and the mathematical formula of InfoNCE loss is as follows:

[0064]

[0065] Among them, k0 is the sample associated with the query vector q, k1...k K is a sample that is irrelevant to the query vector, and τ is a temperature parameter used to adjust the smoothness of the similarity score and control the model's discrimination of negative samples. InfoNCE loss maximizes the similarity between positive samples and query vectors and minimizes the similarity between negative samples and query vectors, so that the training model can suppress the interference of irrelevant samples on the training model and better distinguish relevant samples from irrelevant samples.

[0066] S52, coal quantity monitoring post-processing: In the detection stage, after the model infers and obtains the inference target, first select the target with a confidence level greater than 0.75 as the basis for judging the gangue and coal flow load, so as to reduce the impact of repeated judgment targets on subsequent calculations. Extract the target coordinates according to the category, and map the detection coordinates according to the original image pixel ratio.

[0067] like Figure 5 As shown in the figure, if there is no overlapping relationship between the extracted targets, the pixel area of ​​the detected target is directly calculated based on the pixel value of the detected target conversion coordinates, and the target pixel area is included in the total target area; if there is an overlapping relationship between the targets, the area union is calculated, and the pixel area of ​​the union is included in the total target pixel area. Finally, according to the ratio of the total target pixel area to the belt pixel area in the image, the light load, medium load, and heavy load thresholds of coal are delineated and judged, and the results are output and printed to the original image. The test results are shown in the figure. Figure 6 shown.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. An artificial intelligence video recognition belt load coal quantity monitoring method, characterized in that: The specific steps are as follows: Step 1: Dataset collection: Collect monitoring images of underground coal gangue and coal flow belt conveyors at different working time periods, different lighting conditions, and different coal loads on the belt conveyors; Step 2: Dataset annotation: All features visible to the naked eye are annotated, and the targets of gangue and coal flow that are difficult to distinguish with the naked eye are discarded. Fine-grained annotation is performed according to the detection requirements; Step 3: Dataset splitting: Use K-fold cross-validation to divide the dataset. Based on K-fold cross-validation, evaluate and compare different models and parameters to select the best model and parameter combination. Step 4: Image preprocessing: Use the contrast-limited adaptive histogram equalization algorithm to decompose the image into different sub-images, enhance the sub-image contrast and suppress image noise; Step 5. Model training, tuning and post-processing: Select the YOLOv5 model as the coal flow target detection model, set the learning rate, batch processing size, number of iterations, and activation function, introduce contrastive learning loss joint training, improve the YOLOv5 network post-processing structure, introduce a pixel area-based ratio algorithm to calculate the belt area covered by coal flow and coal gangue, and combine the light, medium, and heavy load conditions of the belt coal. Introduce a coal flow threshold judgment algorithm to perform contactless detection of the belt coal load in the monitoring video and output the coal volume judgment result.

2. The method for monitoring the amount of coal loaded on a belt by using artificial intelligence video recognition according to claim 1 is characterized in that: In step 2, the specific marking steps are: Step S21, marking the obstructed coal flow target, and drawing the edge of the obstructed part according to the edge shape in the unobstructed scene; Step S22, marking targets visible to human eyes, where the target pixel range is greater than 20*20; Step S23, discarding targets that are difficult for human eyes to distinguish; Step S24: Ensure that the frame coordinates are not on the image boundary, and discard the target frame whose aspect ratio exceeds the set value.

3. The method for monitoring the amount of coal loaded on a belt by using artificial intelligence video recognition according to claim 2 is characterized in that: In step 4, the adaptive histogram equalization algorithm includes four parts: contrast stretching, histogram equalization, contrast limitation and adaptive histogram equalization, where: Contrast stretching: Image contrast is defined as the difference between the brightest pixel and the darkest pixel in the image. The Michelson quantitative measurement formula is as follows: Among them, I max Indicates the brightest pixel, I min Indicates the darkest pixel. When the grayscale of the whitest pixel and the blackest pixel of an image are both 128, the image contrast is the lowest, C = 0; when the grayscale of the whitest pixel of an image is 255 and the grayscale of the blackest pixel is 0, the image contrast is the highest, C = 1.0; when the grayscale value of the image is concentrated in the interval (120, 180), a piecewise linear function is used to map the grayscale value from (120, 180) to (0, 255) to achieve contrast stretching; Histogram equalization: Use the histogram cumulative distribution function to stretch the original image to obtain a rectangular histogram; Contrast limit: Set the histogram distribution threshold, and evenly distribute the distribution exceeding the set threshold to the probability density distribution to limit the increase of the cumulative histogram; Adaptive histogram equalization: calculate the grayscale histogram and the corresponding cumulative histogram of each sub-region of the image; if the sub-image is located at the top corner of the image and there are only two adjacent sub-images, grayscale mapping is performed according to the transformation function of the sub-image; if the sub-image is located at the four sides of the image, that is, there are three adjacent sub-images, it is transformed according to the transformation function of the two adjacent sub-images and then linearly interpolated to obtain the grayscale equalization; if the sub-image has four adjacent sub-images, it is transformed according to the transformation function of the four adjacent sub-images and then bilinearly interpolated to obtain the grayscale equalization.

4. The method for monitoring the amount of coal loaded on a belt by using artificial intelligence video recognition according to claim 3 is characterized in that: In step five, the specific processing steps are: Step S51, InfoNCE loss function tuning: introduce contrastive learning loss to supervise network training, learn model parameters for learning feature representation by comparing the similarity between positive samples and negative samples, and improve learning loss based on noise contrast estimation. Noise contrast estimation regards the probability estimation problem as a basic binary classification, that is, the classification object only contains objects and noise, and uses noise contrast to improve the estimation of the model. The specific expression is as follows: Among them, x is the data sample, is the noise sample, P data (x) represents the distribution of real data, P model (x) represents the model prediction, P noise (x) is represented as a noise distribution; a contrastive learning loss is proposed, assuming that the feature selection reference vector q is a representation vector related to the task, and the key vector {k0, k1...k K } represents the information index of each position in the input sequence. The specific formula of InfoNCE loss is as follows: Among them, k0 is the sample associated with the query vector q, k1...k K is a sample that is unrelated to the query vector, τ is a temperature parameter, and the InfoNCE loss maximizes the similarity between the positive sample and the query vector and minimizes the similarity between the negative sample and the query vector; Step S52, coal quantity monitoring post-processing: In the detection stage, after the model is inferred and the inference target is obtained, the target with a confidence level greater than 0.75 is first selected as the basis for judging the gangue and coal flow load, and the target coordinates are extracted according to the category, and the detection coordinates are mapped and transformed according to the original image pixel ratio; If there is no overlapping relationship between the extracted targets, the pixel area of ​​the detected target is directly calculated based on the pixel value of the detected target conversion coordinates, and the target pixel area is included in the total target area; if there is an overlapping relationship between the targets, the area union is calculated, and the union pixel area is included in the total target pixel area; finally, based on the ratio of the total target pixel area to the belt pixel area in the image, the light load, medium load, and heavy load thresholds of the coal are defined and judged, and the results are output and printed to the original image.

Images