Foreign matter detection method and system for belt conveyor, computer equipment and storage medium

Abstract

The foreign matter detection method for the belt conveyor comprises the following steps: acquiring a real-time video in a raw coal conveying process, and extracting a coal flow image from the real-time video; performing feature extraction on the coal flow image by using a pre-trained anomaly detection model to obtain a plurality of specific scale feature maps; performing structural similarity calculation on the specific scale feature map and a pre-extracted template feature map, and taking the template feature map with the highest similarity with the specific scale feature map as a basic feature map; and splicing all the specific scale feature maps and the basic feature map along the channel dimension, and decoding to obtain an abnormal region and a corresponding abnormal score. According to the method, the input coal flow image is processed to extract the feature map, and the feature map level structural similarity comparison with the template feature map extracted from the template image is carried out, so that the abnormal degree of the coal flow image can be quantitatively evaluated, and whether foreign matters exist in the coal flow or not can be judged.

Description

Technical Field

[0001] The invention relates to the field of belt conveyors, in particular to a foreign body detection method, system, computer equipment and storage medium for belt conveyors. Background Art

[0002] Belt conveyors, also known as rubber belt conveyors, are used to transport materials and are widely used in various industries such as mining, electronics, machinery, tobacco, injection molding, and food. In some production scenarios, due to the complex operating environment of belt conveyors, it is easy for foreign matter to be mixed into the transported materials. For example, in the process of transporting raw coal, the belt conveyors used in coal mines often mix some foreign matter in the raw coal. These foreign matter include gangue, metal parts, and wooden parts, such as steel wire, bolts, and sticks. Although these foreign matter are small in quantity, they can cause many hazards during the transportation and subsequent crushing of raw coal, such as damage to equipment. In order to avoid the hazards caused by these debris, it is a good way to identify the debris during the transportation of raw coal and remove it in time.

[0003] In order to quickly identify foreign objects in raw coal, it is an effective way to shoot the raw coal and build a coal flow anomaly detection model based on deep learning. In the prior art, such coal flow anomaly detection models mainly identify and classify abnormal features extracted from coal flow images to determine the corresponding foreign object types. This requires a large number of samples with clear foreign object types for training. In actual industrial environments, it is difficult to obtain samples that can meet both requirements at the same time. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the present invention provides a foreign matter detection method, system, computer equipment and storage medium for a belt conveyor. By processing the input coal flow image to extract a feature map, and performing a structural similarity comparison at the feature map level with the template feature map extracted from the template image, the abnormality of the coal flow image can be quantitatively evaluated, and whether there are foreign matters in the coal flow can be determined. This can provide a basis for subsequent foreign matter processing and ensure that the raw coal can be smoothly transported and processed.

[0005] In order to achieve the above object, the specific scheme adopted by the present invention is:

[0006] Beneficial effect: The foreign body detection method for a belt conveyor comprises the following steps: Obtain real-time video of the raw coal transportation process and extract coal flow images from it; The pre-trained anomaly detection model is used to extract features from the coal flow image to obtain multiple specific scale feature maps; Calculate the structural similarity between the specific scale feature map and the pre-extracted template feature map, and use the template feature map with the highest similarity to the specific scale feature map as the basic feature map; All the specific scale feature maps and basic feature maps are concatenated along the channel dimension and then decoded to obtain the abnormal area and the corresponding abnormal score.

[0007] As a further optimization of the above-mentioned foreign body detection method for belt conveyor: when the coal flow image is subjected to feature extraction using the pre-trained anomaly detection model, three specific scale feature maps are obtained. The three specific scale feature maps are: F 1 ∈i bs×64×64×64 F 2 ∈i bs×128×32×64 ; F 3 ∈i bs×256×16×16 Where bs is the number of coal flow images; When calculating the structural similarity between the specific scale feature map and the template feature map, the specific scale feature map and the template feature map have the same scale. The three template feature maps are: T 1 ∈i n×64×64×64 T 2 ∈i n×128×32×64 ; T 3 ∈i n×256×16×16 Where n is the number of template feature maps.

[0008] As a further optimization of the above-mentioned foreign body detection method for belt conveyors: The specific method of structural similarity calculation is: S l [i,j]=SSIM(F l [i],T l [j]); Among them, S l ∈i bs×n represents the structural similarity matrix, i is the index of the specific scale feature map and i=1,...,bs, j is the index of the template feature map and j=1,...,n, and F l [i] and T l The average value of [j], and F l [i] and T lThe variance of [j], c 1 is and c 2 is a constant, where c 1 =(K 1 D) 2 , c 2 =(K 2 D) 2 , K 1 and K 2 It is used to control c 1 is and c 2 is an adjustable parameter, and D is the dynamic range of pixel values.

[0009] As a further optimization of the above-mentioned foreign body detection method for belt conveyors: the specific method for splicing the corresponding specific scale feature map and the basic feature map along the channel dimension is: C 1,i =concatenate(F 1 [i],T 1 [M 1 [i]],axis=1) C 2,i =concatenate(F 2 [i],T 2 [M 3 [i]],axis=1); C 3,i =concatenate(F 3 [i],T 3 [M 3 [i]],axis=1) Among them, M l [i] = argmax j S l [i,j].

[0010] As a further optimization of the above-mentioned foreign body detection method for belt conveyors: The method of training anomaly detection models includes: Obtaining an operating video of the raw coal conveyor, and extracting video frames from the operating video to obtain a normal sample image; Generate noise constraint conditions based on the occupied area of ​​the raw coal conveyor in the normal sample image, and add simulated noise to part of the normal sample image based on the noise constraint conditions to obtain abnormal sample images; A sample set is constructed using normal sample images and abnormal sample images, and the original model is optimized and trained using the stochastic gradient descent method to obtain an anomaly detection model.

[0011] As a further optimization of the above-mentioned foreign body detection method for belt conveyors: the noise constraint condition is used to determine the noise-addable range in the normal sample image, and the specific method of adding simulated noise to part of the normal sample image based on the noise constraint condition includes: A two-dimensional noise image is generated using a noise generator, and the noise image is processed to obtain an abnormality mask; generating a mask image based on the anomaly mask; After enhancing the noise image, it is fused with the mask image to generate a noise foreground image; The noisy foreground image is added to the noisy region to obtain the abnormal sample image.

[0012] As a further optimization of the above-mentioned foreign object detection method for belt conveyor: in the process of training the original model, the cross entropy loss and the mean square error loss are used as the supervision loss, and the structural similarity index measurement loss is used as the self-supervision loss.

[0013] Foreign object detection system for belt conveyors, including: An image acquisition module is used to acquire real-time video during the raw coal transportation process and extract coal flow images from it; A feature extraction module is used to extract features from coal flow images using a pre-trained anomaly detection model to obtain multiple specific scale feature maps; A similarity calculation module is used to calculate the structural similarity between the specific scale feature map and the template feature map extracted in advance, and use the template feature map with the highest similarity to the specific scale feature map as the basic feature map; The decoding module is used to decode the spliced ​​image obtained by splicing all the specific scale feature maps and the basic feature maps, and output the abnormal area and the corresponding abnormal score.

[0014] Computer equipment, including: Memory for storing computer programs; A processor is used to execute the computer program to implement the above-mentioned foreign body detection method for a belt conveyor.

[0015] A computer-readable storage medium is used to store a computer program, and when the computer program is executed, the foreign body detection method for a belt conveyor is implemented.

[0016] The present invention processes the input coal flow image to extract a feature map, and compares the structural similarity at the feature map level with the template feature map extracted from the template image. It can quantitatively evaluate the abnormality of the coal flow image and determine whether there are foreign objects in the coal flow. This can provide a basis for subsequent foreign object processing and ensure that the raw coal can be smoothly transported and processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural schematic diagram of the anomaly detection model in the present invention; Figure 2 is an example graph of structural similarity calculation; Figure 3 It is a schematic diagram of the process of adding simulated noise to some normal sample images based on noise constraints; Figure 4 This is an example of an actual photo collected during the experiment; Figure 5 This is an example of the result image finally obtained by the present invention and several existing methods in the experiment; Figure 6 It is an example diagram of the detection results of foreign objects in coal flow under different lighting conditions of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Since belt conveyors have a wide range of application scenarios, for the sake of clear description, the present invention will be described in detail below taking only the belt conveyor used for transporting raw coal in coal mines as a processing target.

[0020] A foreign body detection method for a belt conveyor, including S1 to S4.

[0021] S1. Obtain real-time video during the raw coal conveying process and extract the coal flow image from it. The real-time video can be shot by a camera fixed above the raw coal conveyor, and the obtained video frame image is used as the coal flow image through frame analysis. These are conventional technologies and will not be described in detail here.

[0022] S2. Use the pre-trained anomaly detection model to extract features from the coal flow image and obtain multiple specific scale feature maps. Figure 1 As shown, in this embodiment, a ResNet18 encoder pre-trained on ImageNet is used to extract feature maps, which belongs to the prior art in the field and will not be described in detail here.

[0023] In this embodiment, features of the input image are extracted at three different scales. Specifically, when the coal flow image is subjected to feature extraction using the pre-trained anomaly detection model, three specific scale feature maps are obtained. The three specific scale feature maps are: F 1∈i bs×64×64×64 F 2 ∈i bs×128×32×64 ; F 3 ∈i bs×256×16×16 Where bs is the number of coal flow images.

[0024] When calculating the structural similarity between the specific scale feature map and the template feature map, the specific scale feature map and the template feature map have the same scale. The three template feature maps are: T 1 ∈i n×64×64×64 T 2 ∈i n×128×32×64 ; T 3 ∈i n×256×16×16 Where n is the number of template feature maps.

[0025] S3, calculating the structural similarity between the specific scale feature map and the template feature map extracted in advance, and taking the template feature map with the highest similarity to the specific scale feature map as the basic feature map. Figure 2 The specific method for calculating structural similarity is: S l [i,j]=SSIM(F l [i],T l [j]); Among them, S l ∈i bs×n represents the structural similarity matrix, i is the index of the specific scale feature map and i=1,...,bs, j is the index of the template feature map and j=1,...,n, and F l [i] and T l The average value of [j], and F l [i] and T l The variance of [j], c 1 is and c 2 is a constant used to stabilize the results of the structural similarity calculation, where c 1 =(K 1 D) 2 , c 2 =(K 2 D) 2 , K 1 and K 2 It is used to control c1 is and c 2 is an adjustable parameter, and D is the dynamic range of pixel values. In one embodiment of the present invention, K 1 =0.01, K 2 = 0.03. For each scale l, the specific scale feature map F l And the corresponding template feature map T l SSIM is calculated once, where l = 1, 2, 3. The higher the SSIM value, the higher the matching degree between the specific scale feature map and the template feature map. Among all the template feature maps, the basic feature map selected based on the SSIM value is M l [i] = argmax j S l [i,j]

[0026] S4. The corresponding specific scale feature map and the basic feature map are spliced ​​and decoded along the channel dimension to obtain the abnormal area and the corresponding abnormal score. The specific method for splicing all specific scale feature maps and the basic feature map along the channel dimension is: C 1,i =concatenate(F 1 [i],T 1 [M 1 [i]],axis=1) C 2,i =concatenate(F 2 [i],T 2 [M 3 [i]],axis=1); C 3,i =concatenate(F 3 [i],T 3 [M 3 [i]],axis=1) Among them, M l [i] = argmax j S l [i,j].

[0027] The present invention processes the input coal flow image to extract a feature map, and compares it with the template feature map extracted from the template image. It can quantitatively evaluate the abnormality of the coal flow image and determine whether there are foreign objects in the coal flow. This can provide a basis for subsequent foreign object processing and ensure that the raw coal can be smoothly transported and processed.

[0028] In the present invention, the anomaly detection model is set as a U-shaped network, which mainly includes an encoder, an anomaly assessment module and a decoder. The encoder part adopts the ResNet18 decoder, and its first three layers are used to extract the specific scale feature map and the template feature map. The decoder contains four UpConvBlock (and residual transposed convolution) modules, which include an upsampling layer, a convolution layer, a batch normalization layer and a ReLU activation function, wherein the upsampling layer adopts a bilinear interpolation upsampling method. The anomaly assessment module is used to realize the structural similarity calculation of the above-mentioned specific scale feature map and the template feature map.

[0029] Based on the above model structure, when the anomaly detection model is running, the input image is processed by the encoder to obtain three feature maps of different resolutions, that is, three feature maps of different scales. After that, the obtained specific scale feature map and template feature map are simultaneously input into the anomaly assessment module for structural similarity calculation to achieve feature map matching. After that, the most similar template feature map and specific scale feature map are spliced ​​along the channel dimension and transmitted to the decoder part through jump connection, which is further processed by the decoder to output the anomaly score and identify the abnormal area.

[0030] In order to ensure the accuracy of the detection results, the anomaly detection model needs to be fully trained. Specifically, the method for training the anomaly detection model includes the following steps.

[0031] First, the operation video of the raw coal conveyor is obtained, and the video frames are extracted from the operation video to obtain the normal sample image. The part of the coal flow in the operation video where there is no foreign matter can be manually screened out to obtain an accurate normal sample image.

[0032] Secondly, noise constraints are generated based on the occupied area of ​​the raw coal conveyor in the normal sample image, and simulated noise is added to some normal sample images based on the noise constraints to obtain abnormal sample images. Considering that the type, shape and position of foreign matter in raw coal are not fixed, it is difficult to obtain a sufficient number of abnormal samples as samples to train the model, and the number of abnormal samples is too small, which will lead to insufficient accuracy of the trained abnormal detection model. Therefore, the present invention generates abnormal sample images by simulating abnormalities, which can effectively ensure the number of abnormal sample images. Specifically, after obtaining the normal sample image, the noise constraints are first determined based on the shooting conditions of the normal sample image, and the noise constraints are used to determine the noise-addable range in the normal sample image. At present, raw coal is mainly transported by belt conveyors after mining. When the belt conveyor is photographed from above to obtain the normal sample image, the belt conveyor will also be photographed, and the foreign matter in the coal flow will only exist in the belt part of the belt conveyor, and there will be no foreign matter on the side support part of the belt conveyor. Therefore, the noise constraint condition can be used to determine the noise-addable range in the normal sample image, that is, determine the part of the normal sample image that belongs to the conveyor belt, and ignore the part of the side bracket, so as to avoid adding noise to the part of the side bracket when adding noise. By setting the noise constraint condition, the noise addition range can be reduced, thereby reducing the complexity of the subsequent process of adding noise to the normal sample image.

[0033] like Figure 3 As shown, the specific method of adding simulated noise to some normal sample images based on the noise constraint condition includes the following steps.

[0034] First, a noise generator is used to generate a two-dimensional noise image, and the noise image is processed to obtain an abnormal mask. In this embodiment, a Perlin noise generator is used, and the generated two-dimensional noise image is represented as I n , the Perlin ratio is set to 6, the minimum Perlin ratio is 0, and the Perlin noise threshold is 0.5. n The specific method for processing to obtain the exception mask is to n Perform random uniform sampling and apply a binarization threshold t to obtain the anomaly mask Mp.

[0035] Second, a mask image is generated based on the abnormal mask. The specific method is to transform the abnormal mask M p With the predefined mask area M I Perform element-by-element multiplication to obtain the mask image M, that is, M = M I e M P .

[0036] Third, the noise image is enhanced and fused with the mask image to generate a noise foreground image. The method for enhancing the noise image P is: randomly extracting three enhancement functions from a pre-set enhancement function set for processing, and the enhancement function set is {tone separation, sharpening, exposure, equalization, brightness change, color change, automatic contrast}. The enhancement method for each noise image may be different, so as to avoid the generated abnormal sample images being too similar and the training effect of the abnormal detection model being poor. Further, the mask image M is combined with the enhanced noise image I n Performing element-by-element multiplication, we can obtain I defined by M n The method for generating the noise foreground image is as follows: n ′=β(MeI n )+(1-β)(Me I), where β is the transparency factor used to balance the mask image M and the enhanced noise I n The mixing ratio between them is set to make the obtained noise foreground image and the subsequently obtained abnormal sample image more realistic. In this embodiment, the value range of β is set to [0.15, 1].

[0037] Fourth, the mask image M is inverted to obtain an inverted image M, and the inverted image M and the noise image I are subjected to n Perform element-by-element multiplication to obtain the basic image I′.

[0038] Fifth, add the noise foreground image to the noisy region to obtain the abnormal sample image. The specific method is: A =Me I+I n ′, where I A is an abnormal sample image.

[0039] Finally, a sample set is constructed using normal sample images and abnormal sample images, and the original model is optimized and trained using the stochastic gradient descent method to obtain an anomaly detection model.

[0040] On the other hand, in the prior art, anomaly detection methods based on deep learning generally include three categories: supervised learning, unsupervised learning, and semi-supervised learning. Among them, the anomaly detection algorithm based on supervised learning has the best performance, but it also has certain limitations, mainly reflected in the fact that model training requires a large amount of labeled data that is difficult to obtain in actual industrial scenarios, resulting in the inability to handle foreign objects of unknown categories in a timely manner when they appear; the anomaly detection algorithm based on unsupervised learning can build a model with only normal samples, and detect anomalies by analyzing the differences between normal samples and abnormal samples, so that anomalies of unknown categories can be identified, but there is still a problem of limited performance on data with complex backgrounds; the anomaly detection algorithm based on semi-supervised learning introduces simulated abnormal samples in the training process using only normal samples, which improves the performance of the model in various fields. However, the existing abnormal sample simulation strategy usually lacks an objective and quantitative standard to evaluate the degree of abnormality of the simulated samples, and the lack of an objective evaluation mechanism in the abnormal simulation strategy will lead to insufficient detection performance when identifying coal flow anomalies in complex mine environments.

[0041] In order to further ensure that the anomaly detection model at the training site can obtain sufficiently accurate detection results, the present invention uses the cross entropy loss and the mean square error loss as the supervision loss and the structural similarity index measurement loss as the self-supervision loss during the training of the original model.

[0042] Cross entropy loss is used to improve the model's ability in image-level anomaly detection. Cross entropy loss is mainly used to measure the difference between the predicted probability distribution and the actual probability distribution, effectively guiding the model to improve classification accuracy. The formula of the cross entropy loss function is as follows: Among them, p io and p i1 Respectively represent the probability that each pixel in the predicted output image of the i-th image belongs to the normal and abnormal categories. i = 1 means the image belongs to the abnormal category, and y i =0 means the image is classified as normal.

[0043] As a regularization mechanism, mean square error loss minimizes the squared difference between the predicted value and the actual value and provides strict penalties to refine the classification accuracy, thereby preventing overfitting and ensuring the generalization ability of the model in complex coal flow scenarios. Mean square error loss is combined with cross entropy loss to ensure that the prediction results are both correct and accurate, and enhance the classification accuracy of the model. Based on mean square error loss, the true value S of the abnormal area is compared with the model prediction value. The difference between

[0044] Structural similarity loss is used to improve the segmentation performance of the model, especially in depicting detailed features. Structural similarity loss ensures that the model can identify the presence of anomalies and accurately capture their boundaries and textures. By measuring the structural similarity between the predicted image and the actual image, the structural similarity loss encourages the model to maintain the integrity of the texture and structure and enhance the localization and segmentation of abnormal areas. The loss function of structural similarity loss is:

[0045] Combining the above three types of loss functions, the final objective function can be constructed as follows: L Sum =ω CE L CE +ω L2 L L2 +ω SSIM L SSIM ; where ω CE ,ω L2 , and ω SSIM L CE , L L2 , and L SSIM The weight hyperparameter of the loss function. Based on this objective function, the final optimization goal of the model during training is to make the objective function L Sum minimize.

[0046] In the process of training the model, the present invention can effectively enhance the robustness of the model when applied in a complex coal mine environment through a comprehensive training strategy including three loss functions, and can improve the performance of the model in image-level anomaly detection and the accuracy in pixel-level detection.

[0047] The effect of the detection method of the present invention is experimentally verified as follows.

[0048] First, we collected actual photos taken by an optical camera from a coal mine in Tangshan City, Hebei Province to form a dataset. Examples of actual photos are as follows: Figure 4 The dataset includes 1130 training images and 400 test images. An overview of the dataset is shown in Table 1.

[0049] Table 1 Statistics of the number of images in the dataset

[0050] This dataset covers a variety of typical and atypical foreign objects in coal flow, including coal gangue, steel wire, large pieces of coal, wooden boards, sticks, and bolts. The detailed categories and quantities of foreign objects in the test set are shown in Table 2.

[0051] Table 2 Statistics of categories and quantities of foreign objects in the dataset Exception category Sample size Coal Gangue 101 Steel Wire 51 Large coal 23 board 11 bolt 9 Great 5

[0052] Different from typical datasets that only contain two or three foreign object categories, in order to provide a more comprehensive coal flow foreign object detection scenario, the dataset collected in this experiment has uneven lighting and poor image quality. Foreign objects in the coal flow are randomly distributed, and the shapes and sizes of coal gangue vary significantly.

[0053] During the model training process, the stochastic gradient descent (SGD) method was used for optimization. The initial learning rate was set to 0.001, the weight decay was 0.0005, the momentum was 0.9, and the Nesterov acceleration was not used. The learning strategy during the training process was the cosine annealing algorithm with a warm-up phase, the batch size was set to 16, and the configuration ratio of normal samples to abnormal samples was 3:2. The training process included 1500 cycles, and all input images were resized to 256x256. In the multi-task supervised training strategy, the cross entropy loss (L CE ), L2 loss (L L2 ) and structural similarity loss (L SSIM ), their weights ω CE ,ω L2 and ω SSIM Set to 1, 0.4 and 0.2 respectively.

[0054] In this experiment, the semi-supervised and unsupervised anomaly detection methods commonly used in the prior art are concentrated as comparative examples, including the CutPaste method published in the literature "Li, Chun-Liang, et al." Cutpaste: Self-supervised learning for anomaly detection and localization." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021.", the PaDiM method published in the literature "Defard, Thomas, et al." Padim: a patch distribution modeling framework for anomaly detection and localization." International Conference on Pattern Recognition. Cham: Springer International Publishing, 2021.", and the PaDiM method published in the literature "Zavrtanik, Vitjan, Matej Kristan, and Danijel Sko aj."Draem-a discriminatively trained reconstruction embedding for surface anomaly detection."Proceedingsof the IEEE / CVF International Conference on Computer Vision.2021." The public DRAEM method and literature "Liu, Zhikang, et al."Simplenet: A simple network for image anomaly detection and localization."Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023." The public EfficientAD method. The evaluation indicators used are AUROC (area under the receiver operating characteristic curve) and FPS at the image level and pixel level. These indicators demonstrate the efficiency and accuracy of the method in the anomaly detection task. The experimental results of several methods are shown in Table 4.

[0055] Table 4 Comparison of experimental results of the present invention and several existing methods AUROC-image AUROC-pixel FPS CutPaste

[34] 0.5628 - 269.58 PaDiM

[43] 0.903 0.717 15.38 DRAEM

[44] 0.913 0.734 51.24 EfficientAD

[45] 0.927 0.749 216.91 Ours 0.952 0.693 30.30

[0056] The quantitative analysis results in Table 4 show that the present invention achieves the highest AUROC-image score of 0.952 on the dataset while maintaining a relatively high processing speed. The experimental results show that the present invention can maintain high recognition accuracy even under conditions of poor image quality and uneven illumination. However, due to the complexity of the real coal mine scene represented by the dataset, all models generally perform poorly on the AUROC-pixel indicator. Although the present invention is slightly inferior to EfficientAD and several other methods in pixel-level AUROC, in the actual application of coal flow foreign body detection, the ability to detect foreign bodies is often more critical than accurate location positioning. Therefore, despite not leading in pixel-level AUROC performance, the excellent ability of the present invention in image-level detection still effectively ensures the safety of coal flow transportation. In the actual coal mine conveyor belt scene, the conveyor belt speed is usually between 1m / s and 4m / s, and the model detection speed of 20FPS or higher is sufficient to meet the real-time monitoring needs. Therefore, although CutPaste and EfficientAD show extremely high detection speeds, these speeds exceed the actual requirements. The present invention meets the performance requirements of real-time coal flow monitoring while ensuring high detection accuracy.

[0057] Furthermore, the result images obtained by the present invention and several existing methods are as follows: Figure 5 As shown. Figure 5 It can be seen that although PaDiM and EfficientAD mark the location of foreign objects more accurately than the present invention, their predictions contain many redundant predictions that do not conform to the actual mask. These redundant predictions may distract the attention of supervisors and make it difficult to quickly and accurately identify abnormal areas. Although the model of the present invention does not completely accurately depict the outline of foreign objects, it does not produce too many incorrect predictions. This result effectively provides approximate information about foreign objects and provides reliable support for supervisors to respond quickly. This capability enables supervisors to be quickly notified when foreign objects in coal flows are accurately detected, thereby ensuring the safety of coal mine transportation. Therefore, although the pixel-level AUROC is slightly lower than other models, the overall performance of the model of the present invention shows its unique advantages in practical applications.

[0058] In order to evaluate the robustness of the present invention under different image acquisition conditions, the brightness, contrast, color distortion, global fog, compression distortion and other environments were adjusted in the test set of the dataset, where the brightness adjustment simulated the scene of 30% increase and 30% decrease in brightness. The robustness test results are shown in Table 5.

[0059] Table 5 Robustness test results of the method of the present invention under different conditions environment AUROC-image AUROC-pixel normal 0.952 0.693 Brightness +30% 0.884 0.653 Brightness - 30% 0.886 0.623 Low contrast 0.9304 0.6717 High contrast 0.8722 0.7005 Color Distortion 0.9234 0.6744 Global Fog 0.8425 0.7029 Compression artifacts 0.8425 0.6195

[0060] It can be seen from Table 5 that the detection performance of the anomaly detection model of the present invention has decreased under all test conditions. When the brightness is increased, the performance decreases slightly, while when the brightness is reduced, the performance decreases more significantly. Figure 6 The model's detection of foreign objects in coal flow under different lighting conditions is shown. When the brightness is increased, the model shows a certain degree of diffusion when identifying areas with foreign objects in the coal flow. This may be due to the contrast enhancement and noise increase brought by strong lighting, resulting in slight misjudgment of some local areas. On the contrary, when the brightness is reduced, the model not only weakens its ability to recognize abnormal areas, but also has slight positional shifts and recognition deviations. This may be due to the reduction in brightness, which reduces the visible details of the image, making it difficult for the model to accurately capture abnormal features.

[0061] Although performance degrades under different lighting conditions, especially in low light environments, the present invention is still able to determine the approximate location of foreign objects in the coal flow. Therefore, the present invention exhibits a certain robustness even under conditions of low resolution and uneven lighting. This robustness allows the model to continue to detect foreign objects in the coal flow under conditions of uneven lighting, which is critical to ensuring the safety and efficiency of coal transportation.

[0062] The present invention further provides a foreign matter detection system for a belt conveyor, comprising an image acquisition module, a feature extraction module, a similarity calculation module and a decoding module.

[0063] The image acquisition module is used to acquire the real-time video during the raw coal conveying process and extract the coal flow image from it. The image acquisition module can be an optical camera fixed above the raw coal conveyor.

[0064] The feature extraction module is used to extract features from the coal flow image using a pre-trained anomaly detection model to obtain multiple specific scale feature maps.

[0065] The similarity calculation module is used to calculate the structural similarity between the specific scale feature map and the template feature map extracted in advance, and use the template feature map with the highest similarity to the specific scale feature map as the basic feature map.

[0066] The decoding module is used to decode the spliced ​​image obtained by splicing all the specific scale feature maps and the basic feature maps, and output the abnormal area and the corresponding abnormal score.

[0067] The feature extraction module, similarity calculation module and decoding module mentioned above may adopt separate devices or integrated devices, and only need to have data processing capability.

[0068] The present invention also provides a computer device, including a memory and a processor.

[0069] Memory, used to store computer programs.

[0070] The processor is used to execute the computer program to implement the above-mentioned foreign body detection method for the belt conveyor.

[0071] The above-mentioned memory can be volatile storage or persistent storage. The program stored in the memory may include one or more modules, each of which may include a series of instruction operations in the computer device. Furthermore, the central processing unit may be configured to communicate with the memory and execute a series of instruction operations in the memory on the computer device. The computer device may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows ServerTM, Mac OSXTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0072] Furthermore, the present application also provides a computer-readable storage medium for storing a computer program, and when the computer program is executed, the above-mentioned foreign matter detection method for a belt conveyor is implemented.

[0073] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0074] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0076] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

[0077] In the above, only the application of the present invention to the belt conveyor in the coal mine is described in detail. For the belt conveyor applied to other scenarios, only the collected material image and the template image are different, and the rest of the methods are exactly the same. For example, the above-mentioned coal flow image can be replaced with a grain flow image, and the template image can be replaced with a normal grain flow image. Because the application scenarios of the belt conveyor are wide, they will not be described one by one here.

[0078] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A foreign body detection method for a belt conveyor, characterized in that: The steps include: Obtain real-time video of the raw coal transportation process and extract coal flow images from it; The pre-trained anomaly detection model is used to extract features from the coal flow image to obtain multiple specific scale feature maps; Calculate the structural similarity between the specific scale feature map and the pre-extracted template feature map, and use the template feature map with the highest similarity to the specific scale feature map as the basic feature map; The corresponding specific scale feature map and basic feature map are concatenated along the channel dimension and then decoded to obtain the abnormal area and the corresponding abnormal score.

2. The foreign body detection method for a belt conveyor according to claim 1, characterized in that: When the pre-trained anomaly detection model is used to extract features from the coal flow image, three specific scale feature maps are obtained. The three specific scale feature maps are: Where bs is the number of coal flow images; When calculating the structural similarity between the specific scale feature map and the template feature map, the specific scale feature map and the template feature map have the same scale. The three template feature maps are: Where n is the number of template feature maps.

3. The foreign body detection method for a belt conveyor according to claim 2, characterized in that: The specific method for calculating structural similarity is: S l [i,j]=SSIM(F l [i],T l [j]); in, represents the structural similarity matrix, i is the index of the specific scale feature map and i=1,...,bs, j is the index of the template feature map and j=1,...,n, and F l [i] and T l The average value of [j], and F l [i] and T l The variance of [j], c1 is and c2 are constants, where c1 = (K1D) 2 , c2=(K2D) 2 , K1 and K2 are adjustable parameters for controlling c1 and c2, and D is the dynamic range of pixel values.

4. The foreign body detection method for a belt conveyor according to claim 3, characterized in that: The specific method for concatenating all the specific scale feature maps and the basic feature maps along the channel dimension is: C 1,i =concatenate(F1[i],T1[M1[i]],axis=1) C 2,i =concatenate(F2[i],T2[M3[i]],axis=1); C 3,i =concatenate(F3[i],T3[M3[i]],axis=1) Among them, M l [i] = argmax j S l [i,j].

5. The foreign body detection method for a belt conveyor according to claim 1, characterized in that: Methods for training anomaly detection models include: Obtaining an operating video of the raw coal conveyor, and extracting video frames from the operating video to obtain a normal sample image; Generate noise constraint conditions based on the occupied area of ​​the raw coal conveyor in the normal sample image, and add simulated noise to part of the normal sample image based on the noise constraint conditions to obtain abnormal sample images; A sample set is constructed using normal sample images and abnormal sample images, and the original model is optimized and trained using the stochastic gradient descent method to obtain an anomaly detection model.

6. The foreign body detection method for a belt conveyor according to claim 5, characterized in that: The noise constraint condition is used to determine the range of noise that can be added in the normal sample image. The specific method of adding simulated noise to some normal sample images based on the noise constraint condition includes: A two-dimensional noise image is generated using a noise generator, and the noise image is processed to obtain an abnormality mask; generating a mask image based on the anomaly mask; After enhancing the noise image, it is fused with the mask image to generate a noise foreground image; The noisy foreground image is added to the noisy region to obtain the abnormal sample image.

7. The foreign body detection method for a belt conveyor according to claim 5, characterized in that: In the process of training the original model, the cross entropy loss and mean square error loss are used as supervision losses, and the structural similarity index measurement loss is used as self-supervision loss.

8. A foreign body detection system for a belt conveyor, characterized in that: include: An image acquisition module is used to acquire real-time video during the raw coal transportation process and extract coal flow images from it; A feature extraction module is used to extract features from coal flow images using a pre-trained anomaly detection model to obtain multiple specific scale feature maps; A similarity calculation module is used to calculate the structural similarity between the specific scale feature map and the template feature map extracted in advance, and use the template feature map with the highest similarity to the specific scale feature map as the basic feature map; The decoding module is used to decode the spliced ​​image obtained by splicing all the specific scale feature maps and the basic feature map, and output the abnormal area and the corresponding abnormal score.

9. Computer device, characterized in that include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the foreign body detection method for a belt conveyor as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed, implements the foreign body detection method for a belt conveyor as described in any one of claims 1 to 6.

Images