Crushing operation monitoring method and system based on image processing

Abstract

The application relates to the field of intelligent monitoring, and particularly discloses a broken operation monitoring method and system based on image processing, which extracts multi-scale implicit feature distribution information of an image of raw coal after a broken operation in a high-dimensional feature space by adopting an artificial intelligence monitoring technology based on machine learning after dust removal of the image, and represents a broken effect feature of the raw coal after the broken operation based on the difference between multi-scale raw coal broken image features. Further, whether there is a large piece of raw coal with a size different from that of a small piece of raw coal is detected based on a global difference feature distribution, and then intelligent monitoring of the raw coal broken effect is performed. In this way, the broken effect of the raw coal can be monitored in real time according to actual conditions, so that broken control is performed when the raw coal broken effect does not meet a predetermined standard, and the effect of the raw coal broken operation is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent monitoring, and more specifically, to a method and system for monitoring crushing operations based on image processing. Background Technology

[0002] Coal is an important energy source related to the national economy and people's livelihood. It is generally buried deep underground and needs to be dug out before it can be used. The raw coal dug out from the ground is mostly in the form of irregular lumps, which brings great inconvenience to subsequent transportation, storage and use. In order to overcome this problem, the raw coal needs to be crushed.

[0003] Existing raw coal crushers typically employ methods such as impact, shearing, and grinding to crush raw coal. However, raw coal varies in size, with some pieces being quite large. Even after entering the crusher, large and small pieces of coal may still exist, necessitating secondary crushing. Currently, the varying sizes of raw coal during mining and the inconsistent crushing results each time present challenges to operational control.

[0004] Therefore, an optimized crushing operation monitoring scheme is desired. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide an image processing-based method and system for monitoring coal crushing operations. This method employs machine learning-based artificial intelligence monitoring technology to extract multi-scale latent feature distribution information in a high-dimensional feature space from images after coal crushing operations, after dust removal. The crushing effect of the raw coal after the crushing operation is represented by the differential features between the multi-scale raw coal crushing image features. Furthermore, the presence of large raw coal pieces of different sizes than the small raw coal pieces is detected based on the global differential feature distribution, thereby enabling intelligent monitoring of the raw coal crushing effect. In this way, the crushing effect of the raw coal can be monitored in real time according to the actual situation, allowing for crushing control when the crushing effect does not meet predetermined standards, thus improving the efficiency of the raw coal crushing operation.

[0006] According to one aspect of this application, a method for weaving a high abrasion-resistant textile fabric is provided, comprising:

[0007] Acquire images of the crushing operation captured by a camera;

[0008] The image after the crushing operation is processed by a dust removal generator based on a generative adversarial network to obtain a new image after the crushing operation.

[0009] The generated fragmentation image is passed through a dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map. The first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate.

[0010] Calculate the difference feature map between the first scale feature map and the second scale feature map;

[0011] The feature matrices along the channel dimension of the differential feature map are expanded into feature vectors to obtain multiple local differential feature vectors.

[0012] The plurality of local difference feature vectors are passed through a converter-based context encoder to obtain a global difference feature vector; and

[0013] The global differential feature vector is passed through a classifier to obtain a classification result, which is used to indicate whether the raw coal crushing effect meets the predetermined standard.

[0014] In the above-mentioned image processing-based crushing operation monitoring method, the adversarial generative network includes a discriminator and a generator. The step of passing the post-crushing operation image through the adversarial generative network-based dust removal generator to obtain a generated post-crushing operation image includes: inputting the post-crushing operation image into the adversarial generative network-based dust removal generator so that the generator of the adversarial generative network generates the generated post-crushing operation image by deconvolution encoding.

[0015] In the above-mentioned image processing-based crushing operation monitoring method, the step of generating the image after the crushing operation and passing it through a dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map includes: using each layer of the first convolutional neural network to perform convolution processing on the input data with a first convolutional kernel during the forward propagation of the layer; performing convolution processing on the input data to obtain a convolutional feature map; performing pooling processing on the convolutional feature map based on a local feature matrix to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activation feature map; wherein, the last layer of the first convolutional neural network... The output is the first scale feature map, and the input of the first layer of the first convolutional neural network is the image generated after the crushing operation; and, each layer of the second convolutional neural network performs convolution processing on the input data with a second convolutional kernel during the forward propagation of the layer: convolution processing is performed on the input data to obtain a convolutional feature map; pooling processing based on the local feature matrix is ​​performed on the convolutional feature map to obtain a pooled feature map; and nonlinear activation is performed on the pooled feature map to obtain an activation feature map; wherein, the output of the last layer of the second convolutional neural network is the second scale feature map, and the input of the first layer of the second convolutional neural network is the image generated after the crushing operation.

[0016] In the above-mentioned image processing-based crushing operation monitoring method, the step of calculating the difference feature map between the first scale feature map and the second scale feature map includes: calculating the difference feature map between the first scale feature map and the second scale feature map using the following formula;

[0017] The formula is as follows:

[0018]

[0019] Where F1 represents the first scale feature map, F2 represents the second scale feature map, and F c This represents the difference feature map. This indicates differentiation based on position.

[0020] In the above-mentioned image processing-based crushing operation monitoring method, the step of expanding each feature matrix along the channel dimension of the differential feature map into feature vectors to obtain multiple local differential feature vectors includes: expanding each feature matrix along the channel dimension of the differential feature map into feature vectors along row vectors or column vectors to obtain the multiple local differential feature vectors.

[0021] In the above-mentioned image processing-based crushing operation monitoring method, obtaining a global differential feature vector by passing the multiple local differential feature vectors through a transducer-based context encoder includes: arranging the multiple local differential feature vectors in one dimension to obtain a global differential feature vector; calculating the product between the global differential feature vector and the transpose of each differential feature vector in the multiple local differential feature vectors to obtain multiple self-attention association matrices; standardizing each self-attention association matrix in the multiple self-attention association matrices to obtain multiple standardized self-attention association matrices; passing each standardized self-attention association matrix in the multiple standardized self-attention association matrices through a Softmax classification function to obtain multiple probability values; weighting each differential feature vector in the multiple local differential feature vectors using each probability value as a weight to obtain multiple context semantic differential feature vectors; and concatenating the multiple context semantic differential feature vectors to obtain the global differential feature vector.

[0022] In the above-mentioned image processing-based crushing operation monitoring method, the step of passing the global differential feature vector through a classifier to obtain a classification result includes: processing the global differential feature vector using the classifier with the following formula to obtain the classification result, wherein the formula is:

[0023] softmax{(M c B c )}, softmax{(M c B c )| X}, where M c B is the weight matrix of the fully connected layer. c X represents the bias vector of the fully connected layer, and X is the global difference feature vector.

[0024] The above-mentioned image processing-based crushing operation monitoring method further includes a training step: training the dual-stream detection network, the converter-based context encoder, and the classifier; wherein, the training step includes: acquiring training data, the training data including images after the crushing operation and classification results of whether the raw coal crushing effect meets a predetermined standard; passing the images after the crushing operation trained by the dust removal generator based on the generative adversarial network to obtain images after the crushing operation trained; passing the images after the crushing operation trained by the dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map trained, wherein the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilation rate. The method involves: constructing a dilated convolution kernel; calculating a training difference feature map between the training first-scale feature map and the training second-scale feature map; expanding each feature matrix along the channel dimension of the training difference feature map into feature vectors to obtain multiple training local difference feature vectors; passing the multiple training local difference feature vectors through the converter-based context encoder to obtain a training global difference feature vector; and passing the training global difference feature vectors through the classifier to obtain a classification loss function value; calculating a sequence-to-sequence response rule internalization learning loss function value based on the distance between the training first-scale feature map and the training second-scale feature map; and calculating a weighted sum of the classification loss function value and the sequence-to-sequence response rule internalization learning loss function value as the loss function value to train the two-stream detection network, the converter-based context encoder, and the classifier.

[0025] In the above-mentioned image processing-based method for monitoring crushing operations, the step of calculating the sequence-to-sequence response rule internalization learning loss function value based on the distance between the training first-scale feature map and the training second-scale feature map includes: calculating the sequence-to-sequence response rule internalization learning loss function value based on the distance between the training first-scale feature map and the training second-scale feature map using the following formula; wherein, the formula is:

[0026]

[0027]

[0028]

[0029] Where V1 and V2 are the feature vectors obtained after expanding the first-scale training feature map and the second-scale training feature map, respectively, and W1 and W2 are the weight matrices of the classifier for the feature vectors obtained after expanding the first-scale training feature map and the second-scale training feature map, respectively. ReLU(·) represents the ReLU activation function, and Sigmoid(·) represents the Sigmoid activation function. This represents matrix multiplication, and d(·,·) represents the Euclidean distance between the two vectors. This represents the value of the sequence-to-sequence response rule internalization learning loss function.

[0030] According to another aspect of this application, a crushing operation monitoring system based on image processing is provided, comprising:

[0031] The image acquisition module is used to acquire images of the crushing operation captured by the camera.

[0032] The dust removal module is used to generate a post-crushing image by passing the post-crushing image through a dust removal generator based on a generative adversarial network.

[0033] A multi-scale convolution module is used to pass the image generated after the fragmentation operation through a dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map, wherein the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate.

[0034] The difference module is used to calculate the difference feature map between the first scale feature map and the second scale feature map;

[0035] The feature vector generation module is used to expand each feature matrix along the channel dimension of the difference feature map into feature vectors to obtain multiple local difference feature vectors.

[0036] A context encoding module is used to pass the plurality of local difference feature vectors through a converter-based context encoder to obtain a global difference feature vector; and

[0037] The classification result generation module is used to pass the global differential feature vector through a classifier to obtain a classification result, which is used to indicate whether the raw coal crushing effect meets the predetermined standard.

[0038] According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing computer program instructions, which, when executed by the processor, cause the processor to perform the image processing-based crushing operation monitoring method as described above.

[0039] According to another aspect of this application, a computer-readable medium is provided having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the image processing-based crushing operation monitoring method as described above.

[0040] Compared with existing technologies, this application provides an image processing-based method and system for monitoring coal crushing operations. It employs machine learning-based artificial intelligence monitoring technology to extract multi-scale latent feature distribution information in a high-dimensional feature space from images after coal crushing operations, after dust removal. The crushing effect of the raw coal is represented by the differences between the multi-scale coal crushing image features. Furthermore, the presence of large pieces of raw coal of different sizes than the small pieces is detected based on the global difference feature distribution, thereby enabling intelligent monitoring of the coal crushing effect. This allows for real-time monitoring of the coal crushing effect based on actual conditions, enabling crushing control when the crushing effect does not meet predetermined standards, thus improving the efficiency of the coal crushing operation. Attached Figure Description

[0041] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0042] Figure 1 This is an application scenario diagram of the image processing-based crushing operation monitoring method according to an embodiment of this application;

[0043] Figure 2 This is a flowchart of a crushing operation monitoring method based on image processing according to an embodiment of this application;

[0044] Figure 3 This is a flowchart of the training phase in the image processing-based crushing operation monitoring method according to an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the architecture of the image processing-based crushing operation monitoring method according to an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the architecture of the training phase in the image processing-based crushing operation monitoring method according to an embodiment of this application;

[0047] Figure 6 This is a flowchart of the context encoding process in the image processing-based crushing operation monitoring method according to an embodiment of this application;

[0048] Figure 7 This is a block diagram of an image processing-based crushing operation monitoring system according to an embodiment of this application;

[0049] Figure 8 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0050] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0051] Scene Overview

[0052] As mentioned in the background technology, coal is an important energy source related to the national economy and people's livelihood. It is generally buried deep underground and needs to be dug out before it can be used. The raw coal dug out from the ground is mostly in the form of irregular lumps, which brings great inconvenience to subsequent transportation, storage and use. In order to overcome this problem, it is necessary to crush the dug-out raw coal.

[0053] Existing raw coal crushers typically employ methods such as impact, shearing, and grinding to crush raw coal. However, raw coal varies in size, with some pieces being quite large. Even after initial crushing, large and small pieces of coal may remain, necessitating secondary crushing. Currently, the varying sizes of raw coal during mining and the inconsistent crushing results each time present challenges to operational control. Therefore, an optimized crushing operation monitoring scheme is desired.

[0054] Currently, deep learning and neural networks have been widely applied in fields such as computer vision, natural language processing, and speech signal processing. Furthermore, deep learning and neural networks have demonstrated near-human or even surpassed human-level performance in areas such as image classification, object detection, semantic segmentation, and text translation.

[0055] In recent years, the development of deep learning and neural networks has provided new solutions and approaches for intelligent monitoring of crushing operations.

[0056] Accordingly, monitoring the crushing status of raw coal can be achieved by monitoring images of the raw coal after crushing operations. This involves setting up a remote control terminal at the central control center and using image data collected by cameras to monitor the entire system's transfer and crushing operations in real time. However, considering the large amount of information in the images captured by cameras after crushing operations, and the influence of the raw coal mining environment, the captured image data may be blurred due to interference from environmental dust, making it difficult to monitor the raw coal crushing effect.

[0057] Based on this, the technical solution of this application employs machine learning-based artificial intelligence monitoring technology. After dust removal processing, the images of raw coal crushing operations are used to extract multi-scale latent feature distribution information in a high-dimensional feature space. The crushing effect of the raw coal after the crushing operation is represented by the difference features between the multi-scale raw coal crushing image features. Furthermore, considering that some areas in the images may contain large pieces of raw coal, the presence of large pieces of raw coal of different sizes than the smaller pieces is detected based on the global difference feature distribution, thereby enabling intelligent monitoring of the raw coal crushing effect. In this way, the crushing effect of the raw coal can be monitored in real time according to the actual situation, allowing for crushing control when the crushing effect does not meet predetermined standards, thus improving the efficiency of the raw coal crushing operation.

[0058] Specifically, in the technical solution of this application, firstly, images of the crushing operation are captured using a camera. Then, considering that the presence of a large amount of dust and other small environmental particles in the coal mining site affects the clarity of the images, this interference from external environmental factors during feature extraction can blur the implicit features of the images, thereby reducing the accuracy of judging the coal crushing effect. Therefore, in the technical solution of this application, image clarity is enhanced before feature extraction using a dust removal generator based on a generative adversarial network (GAN). Specifically, the images of the crushing operation are input into the dust removal generator based on the GAN, which generates the generated images of the crushing operation through deconvolutional encoding. In particular, the GAN includes a discriminator and a generator. The generator generates the dust-removed image, and the discriminator calculates the difference between the dust-removed image and the real image, updating the network parameters of the generator using a gradient descent directional propagation algorithm to obtain a generator with dust removal functionality.

[0059] Then, a convolutional neural network model, which excels in extracting latent features from images, is used to perform feature mining on the generated post-crushing image. Specifically, considering that the information on the post-crushing size and uniformity distribution of the raw coal in the generated post-crushing image is crucial for detecting the crushing effect, the generated post-crushing image is processed through a dual-stream detection network comprising a first convolutional neural network model and a second convolutional neural network model to extract the local multi-scale latent feature distribution information, thereby obtaining a first-scale feature map and a second-scale feature map. It is worth noting that here, the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate.

[0060] Next, considering that in the hidden feature information of the image after raw coal crushing, if the crushing effect of the raw coal meets the preset standard, then there will be similar feature distribution information in the image feature information at different scales. Therefore, in the technical solution of this application, the difference feature map between the first scale feature map and the second scale feature map is further calculated, so as to represent the crushing effect feature of the raw coal after crushing operation based on the difference features between the features of the multi-scale raw coal crushing image.

[0061] Furthermore, considering that the image feature distribution after raw coal crushing may contain local features with large pieces of raw coal, which may not effectively detect the crushing effect of the raw coal during local feature mining and comparison, the feature matrices along the channel dimension of the differential feature map are further expanded into feature vectors along row or column vectors to obtain multiple local differential feature vectors. These multiple local differential feature vectors are then encoded using a converter-based context encoder to extract the correlation features of the multiple local raw coal crushing effect features in the channel dimension based on the global feature distribution, thereby obtaining a global differential feature vector.

[0062] Then, the global differential feature vector, which has global characteristics of raw coal crushing effect, is classified by a classifier to obtain a classification result indicating whether the raw coal crushing effect meets a predetermined standard, so as to carry out intelligent monitoring of the raw coal crushing effect.

[0063] Specifically, in the technical solution of this application, when calculating the difference feature map between the first scale feature map and the second scale feature map, since the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate, it is expected to improve the intrinsic response relationship between the feature distributions of the first scale feature map and the second scale feature map under feature extraction with dilated convolutional kernels of different dilation rates, thereby improving the calculation accuracy of the difference feature map between the first scale feature map and the second scale feature map.

[0064] Therefore, the applicant of this application regards the first scale feature map and the second scale feature map as a sequence of feature values, and calculates the sequence-to-sequence response rule internalization learning loss function, expressed as:

[0065]

[0066]

[0067]

[0068] V1 and V2 are the feature vectors obtained by expanding the first-scale feature map and the second-scale feature map, respectively, and W1 and W2 are the weight matrices of the classifier for V1 and V2, respectively.

[0069] In other words, by employing a squeeze-and-encouragement channel attention mechanism on the weight matrices of different sequences of feature vectors V1 and V2, the classifier achieves enhanced discriminative ability between the sequences of feature vectors. By training the network with this loss function, the causal relationship features between responsive sequences can be recovered with better discriminative power, allowing for internalized learning of the cause-and-effect response rules between the sequences of feature vectors V1 and V2. This enhances the intrinsic response relationship between feature vectors V1 and V2 as feature sequences, improves the computational accuracy of the difference feature map between the first-scale feature map and the second-scale feature map, and thus improves classification accuracy. This enables real-time monitoring of the raw coal crushing effect based on actual conditions, allowing for crushing control when the crushing effect does not meet predetermined standards, thereby improving the efficiency of the raw coal crushing operation.

[0070] Based on this, this application provides an image processing-based method for monitoring crushing operations, comprising: acquiring a post-crushing image captured by a camera; passing the post-crushing image through a dust removal generator based on a generative adversarial network to obtain a generated post-crushing image; passing the generated post-crushing image through a dual-stream detection network including a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map, wherein the first convolutional neural network model uses a first dilated convolutional kernel with a first porosity, and the second convolutional neural network model uses a second dilated convolutional kernel with a second porosity; calculating a difference feature map between the first-scale feature map and the second-scale feature map; expanding each feature matrix along the channel dimension of the difference feature map into feature vectors to obtain multiple local difference feature vectors; passing the multiple local difference feature vectors through a context encoder based on a converter to obtain a global difference feature vector; and passing the global difference feature vector through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the raw coal crushing effect meets a predetermined standard.

[0071] Figure 1 This is an application scenario diagram of the image processing-based crushing operation monitoring method according to an embodiment of this application. For example... Figure 1 As shown, in this application scenario, through a camera (e.g., such as...) Figure 1 As shown in C), an image of the crushing operation is acquired. This image is then input to a server deployed with an image-processing-based crushing operation monitoring algorithm (e.g., [example server]). Figure 1 In S), the server is able to process the input image using the image processing-based crushing operation monitoring algorithm to generate a classification result indicating whether the raw coal crushing effect meets a predetermined standard.

[0072] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0073] Exemplary methods

[0074] Figure 2 This is a flowchart of a crushing operation monitoring method based on image processing according to an embodiment of this application. Figure 2As shown, the image processing-based crushing operation monitoring method according to an embodiment of this application includes: S110, acquiring a post-crushing operation image captured by a camera; S120, passing the post-crushing operation image through a dust removal generator based on a generative adversarial network to obtain a generated post-crushing operation image; S130, passing the generated post-crushing operation image through a dual-stream detection network including a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map, wherein the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate. The network model uses a second dilated convolution kernel with a second dilation rate; S140, calculates a difference feature map between the first-scale feature map and the second-scale feature map; S150, expands each feature matrix along the channel dimension of the difference feature map into feature vectors to obtain multiple local difference feature vectors; S160, passes the multiple local difference feature vectors through a converter-based context encoder to obtain a global difference feature vector; and S170, passes the global difference feature vector through a classifier to obtain a classification result, the classification result being used to indicate whether the raw coal crushing effect meets a predetermined standard.

[0075] Figure 4 This is a schematic diagram of the architecture of a crushing operation monitoring method based on image processing according to an embodiment of this application. Figure 4 As shown, in this network architecture, during the inference phase, images of the crushing operation are first acquired via a camera; then, these images are processed by a dust removal generator based on a generative adversarial network to generate images of the crushing operation; next, these generated images are processed by a dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain first-scale feature maps and second-scale feature maps, wherein the first convolutional neural network model uses a first dilated convolutional kernel with a first porosity, and the second convolutional neural network model uses a second dilated convolutional kernel with a second porosity; then, a difference feature map is calculated between the first-scale feature map and the second-scale feature map; each feature matrix along the channel dimension of the difference feature map is expanded into feature vectors to obtain multiple local difference feature vectors; subsequently, these multiple local difference feature vectors are processed by a context encoder based on a converter to obtain global difference feature vectors; finally, these global difference feature vectors are processed by a classifier to obtain classification results, which are used to indicate whether the raw coal crushing effect meets a predetermined standard.

[0076] More specifically, in step S110, images of the crushing operation after the crushing process are acquired by a camera. Considering that the crushing status of raw coal can be monitored by observing images of the raw coal after the crushing operation, the technical solution of this application uses a remote control terminal set up in the control center to monitor the entire system's transfer and crushing operations in real time using image data acquired by the camera.

[0077] More specifically, in step S120, the image after the crushing operation is processed by a dust removal generator based on a generative adversarial network to generate a post-crushing operation image. Considering that the presence of a large amount of dust and other small environmental particles in the coal mining site affects the clarity of the post-crushing operation image, the implicit features of the image become blurred during feature extraction due to interference from external environmental factors, thus reducing the accuracy of judging the coal crushing effect. Therefore, in the technical solution of this application, image clarity is enhanced by a dust removal generator based on a generative adversarial network before feature extraction. Specifically, the image after the crushing operation is input into the dust removal generator based on a generative adversarial network, which generates the post-crushing operation image through deconvolutional encoding. Specifically, the adversarial generative network here includes a discriminator and a generator, wherein the generator is used to generate a deduplicated image, the discriminator is used to calculate the difference between the deduplicated image and the real image, and the network parameters of the generator are updated by the gradient descent directional propagation algorithm to obtain a generator with deduplicated function.

[0078] More specifically, in step S130, the generated post-crushing image is processed through a two-stream detection network comprising a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map. The first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate. A convolutional neural network model with excellent performance in extracting latent features of images is used for feature mining of the generated post-crushing image. In particular, considering that the information on the post-crushing size and uniformity distribution of the raw coal in the generated post-crushing image is crucial for detecting the crushing effect when judging the crushing effect, the generated post-crushing image is processed through a two-stream detection network comprising a first convolutional neural network model and a second convolutional neural network model to extract the local multi-scale latent feature distribution information in the generated post-crushing image, thereby obtaining the first-scale feature map and the second-scale feature map. It is worth mentioning that here, the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate. More specifically, the step of passing the image generated after the fragmentation operation through a dual-stream detection network containing the first and second convolutional neural network models to obtain a first-scale feature map and a second-scale feature map includes: performing convolution processing on the input data using the first convolutional kernel in the forward propagation of each layer of the first convolutional neural network; performing convolution processing on the input data to obtain a convolutional feature map; performing pooling processing on the convolutional feature map based on the local feature matrix to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activation feature map; wherein, the output of the last layer of the first convolutional neural network is the first-scale feature map. The second convolutional neural network (CNN) comprises a second scale feature map, wherein the input of the first layer of the first convolutional neural network is the image generated after the crushing operation; and, during the forward propagation of each layer of the second CNN, the input data is convolved using a second convolutional kernel: the input data is convolved to obtain a convolutional feature map; the convolutional feature map is pooled based on a local feature matrix to obtain a pooled feature map; and the pooled feature map is nonlinearly activated to obtain an activation feature map; wherein the output of the last layer of the second CNN is the second scale feature map, and the input of the first layer of the second CNN is the image generated after the crushing operation. The first and second CNNs comprise multiple cascaded neural network layers, each comprising a convolutional layer, a pooling layer, and an activation layer.

[0079] More specifically, in step S140, a difference feature map is calculated between the first-scale feature map and the second-scale feature map. Considering that in the hidden feature information of the image after raw coal crushing, if the crushing effect of the raw coal meets a preset standard, then similar feature distribution information will exist in the image feature information at different scales. Therefore, in the technical solution of this application, a difference feature map is further calculated between the first-scale feature map and the second-scale feature map to represent the crushing effect features of the raw coal after crushing based on the difference features between the multi-scale raw coal crushing image features. In a specific example of this application, the difference feature map between the first-scale feature map and the second-scale feature map is calculated using the following formula; wherein, the formula is:

[0080]

[0081] Where F1 represents the first scale feature map, F2 represents the second scale feature map, and F c This represents the difference feature map. This indicates differentiation based on position.

[0082] More specifically, in step S150, the feature matrices along the channel dimension of the differential feature map are expanded into feature vectors to obtain multiple local differential feature vectors. Considering that the image feature distribution after raw coal crushing may contain local features with large pieces of raw coal, this may not effectively detect the crushing effect of the raw coal during local feature mining and comparison. Therefore, in a specific example of this application, the feature matrices along the channel dimension of the differential feature map are further expanded into feature vectors along row or column vectors to obtain the multiple local differential feature vectors.

[0083] More specifically, in step S160, the plurality of local differential feature vectors are processed by a converter-based context encoder to obtain a global differential feature vector. That is, the plurality of local differential feature vectors are encoded by a converter-based context encoder to extract the correlation features of the plurality of local raw coal crushing effect features in the channel dimension based on the global feature distribution, thereby obtaining the global differential feature vector. More specifically, the step of obtaining a global differential feature vector by passing the plurality of local differential feature vectors through a transducer-based context encoder includes: arranging the plurality of local differential feature vectors in one dimension to obtain a global differential feature vector; calculating the product between the global differential feature vector and the transpose of each differential feature vector in the plurality of local differential feature vectors to obtain a plurality of self-attention association matrices; standardizing each self-attention association matrix in the plurality of self-attention association matrices to obtain a plurality of standardized self-attention association matrices; passing each standardized self-attention association matrix in the plurality of standardized self-attention association matrices through a Softmax classification function to obtain a plurality of probability values; weighting each differential feature vector in the plurality of local differential feature vectors using each probability value as a weight to obtain a plurality of context semantic differential feature vectors; and concatenating the plurality of context semantic differential feature vectors to obtain the global differential feature vector.

[0084] Figure 6 This is a flowchart illustrating the context encoding process in the image processing-based crushing operation monitoring method according to an embodiment of this application. Figure 6 As shown, the context encoding process includes: S310, arranging the plurality of local difference feature vectors in one dimension to obtain a global difference feature vector; S320, calculating the product between the global difference feature vector and the transpose of each difference feature vector in the plurality of local difference feature vectors to obtain a plurality of self-attention association matrices; S330, standardizing each self-attention association matrix in the plurality of self-attention association matrices to obtain a plurality of standardized self-attention association matrices; S340, applying a Softmax classification function to each standardized self-attention association matrix in the plurality of standardized self-attention association matrices to obtain a plurality of probability values; S350, weighting each difference feature vector in the plurality of local difference feature vectors using each probability value as a weight to obtain a plurality of context semantic difference feature vectors; and S360, concatenating the plurality of context semantic difference feature vectors to obtain the global difference feature vector.

[0085] More specifically, in step S170, the global differential feature vector is processed by a classifier to obtain a classification result, which is used to indicate whether the raw coal crushing effect meets a predetermined standard. It should be understood that the global differential feature vector, which has global characteristics of the raw coal crushing effect, is processed by a classifier to obtain a classification result indicating whether the raw coal crushing effect meets a predetermined standard, thereby enabling intelligent monitoring of the raw coal crushing effect. In a specific example of this application, the classifier includes multiple fully connected layers and a Softmax layer cascaded with the last fully connected layer of the multiple fully connected layers. In the classification process of the classifier, the global difference feature vector is encoded multiple times using multiple fully connected layers of the classifier to obtain an encoded classification feature vector. Then, the encoded classification feature vector is input into the Softmax layer of the classifier, that is, the Softmax classification function is used to classify the encoded classification feature vector to obtain a classification result representing whether the raw coal crushing effect meets a predetermined standard. More specifically, the step of passing the global difference feature vector through the classifier to obtain a classification result includes: processing the global difference feature vector using the classifier with the following formula to obtain a classification result, where the formula is:

[0086] softmax{(M c B c )}, softmax{(M c B c )|X}, where M c B is the weight matrix of the fully connected layer. c X represents the bias vector of the fully connected layer, and X is the global difference feature vector.

[0087] It should be understood that before utilizing the aforementioned neural network encoding, the dual-stream detection network, the converter-based context encoder, and the classifier need to be trained. That is, the image processing-based crushing operation monitoring method of this application includes a training module for training the dual-stream detection network, the converter-based context encoder, and the classifier.

[0088] Figure 3 This is a flowchart of the training phase in the image processing-based crushing operation monitoring method according to an embodiment of this application. Figure 3As shown, the image processing-based crushing operation monitoring method according to an embodiment of this application includes: S210, acquiring training data, the training data including training images after crushing operations, and classification results of whether the raw coal crushing effect meets a predetermined standard; S220, passing the training images after crushing operations through the dust removal generator based on an adversarial generative network to obtain training generated images after crushing operations; S230, passing the training generated images after crushing operations through the dual-stream detection network including a first convolutional neural network model and a second convolutional neural network model to obtain training first-scale feature maps and training second-scale feature maps, wherein the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate; S240, calculating the training first-scale feature maps and the training second-scale feature maps. S250: Training differential feature maps between scale feature maps; S260: Expanding each feature matrix along the channel dimension of the training differential feature maps into feature vectors to obtain multiple training local differential feature vectors; S270: Passing the multiple training local differential feature vectors through the converter-based context encoder to obtain training global differential feature vectors; and S280: Passing the training global differential feature vectors through the classifier to obtain classification loss function values; S290: Calculating sequence-to-sequence response rule internalization learning loss function values ​​based on the distance between the training first scale feature map and the training second scale feature map; and S201: Calculating the weighted sum of the classification loss function value and the sequence-to-sequence response rule internalization learning loss function value as the loss function value to train the two-stream detection network, the converter-based context encoder, and the classifier.

[0089] Figure 5 This is a schematic diagram of the architecture of the training phase in the image processing-based crushing operation monitoring method according to an embodiment of this application. Figure 5As shown, in this network architecture, during the training phase, training data is first acquired, including images after the crushing operation and classification results indicating whether the raw coal crushing effect meets predetermined standards. Then, the images after the crushing operation are passed through a dust removal generator based on an adversarial generative network to obtain images after the crushing operation. These images are then passed through a dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain training first-scale feature maps and training second-scale feature maps. The first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate. Finally, the training first-scale feature maps and the training second-scale feature maps are calculated. The training differential feature map is generated between the feature maps of different dimensions. The feature matrices along the channel dimension of the training differential feature map are expanded into feature vectors to obtain multiple training local differential feature vectors. Then, the multiple training local differential feature vectors are passed through the converter-based context encoder to obtain a training global differential feature vector. The training global differential feature vector is passed through the classifier to obtain a classification loss function value. Furthermore, the sequence-to-sequence response rule internalization learning loss function value is calculated based on the distance between the training first-scale feature map and the training second-scale feature map. The weighted sum of the classification loss function value and the sequence-to-sequence response rule internalization learning loss function value is used as the loss function value to train the two-stream detection network, the converter-based context encoder, and the classifier.

[0090] Specifically, in the technical solution of this application, when calculating the difference feature map between the first scale feature map and the second scale feature map, since the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate, it is expected to improve the intrinsic response relationship between the feature distributions of the first scale feature map and the second scale feature map under feature extraction with dilated convolutional kernels of different dilation rates, thereby improving the calculation accuracy of the difference feature map between the first scale feature map and the second scale feature map.

[0091] Therefore, the applicant of this application regards the first scale feature map and the second scale feature map as a sequence of feature values, and calculates the sequence-to-sequence response rule internalization learning loss function, expressed as:

[0092]

[0093]

[0094]

[0095] Where V1 and V2 are the feature vectors obtained after expanding the first-scale training feature map and the second-scale training feature map, respectively, and W1 and W2 are the weight matrices of the classifier for the feature vectors obtained after expanding the first-scale training feature map and the second-scale training feature map, respectively. ReLU(·) represents the ReLU activation function, and Sigmoid(·) represents the Sigmoid activation function. This represents matrix multiplication, and d(·,·) represents the Euclidean distance between the two vectors. This represents the loss function value for the internalization learning of the sequence response rules. That is, by using a squeeze-and-encouragement channel attention mechanism on the weight matrices of different sequences of feature vectors V1 and V2, the classifier obtains enhanced discriminative ability between the sequences of feature vectors. By training the network with this loss function, the causal relationship features with better discriminativeness between responsive sequences can be recovered, enabling the internalization learning of the cause-and-effect response rules between the sequences of feature vectors V1 and V2. This enhances the intrinsic response relationship between feature vectors V1 and V2 as feature sequences, improves the calculation accuracy of the difference feature map between the first-scale feature map and the second-scale feature map, and thus improves the classification accuracy. In this way, the crushing effect of the raw coal can be monitored in real time according to the actual situation, and crushing control can be implemented when the crushing effect does not meet the predetermined standards, thereby improving the efficiency of the raw coal crushing operation.

[0096] In summary, the image processing-based coal crushing operation monitoring method based on the embodiments of this application is explained. It employs machine learning-based artificial intelligence monitoring technology to extract multi-scale implicit feature distribution information in a high-dimensional feature space after dust removal from images of raw coal crushing operations. The crushing effect characteristics of the raw coal after the crushing operation are represented based on the differences between the multi-scale raw coal crushing image features. Furthermore, the presence of large raw coal pieces of different sizes than the small raw coal pieces is detected based on the global difference feature distribution, thereby enabling intelligent monitoring of the raw coal crushing effect. In this way, the crushing effect of the raw coal can be monitored in real time according to the actual situation, allowing for crushing control when the crushing effect does not meet predetermined standards, thus improving the efficiency of the raw coal crushing operation.

[0097] Exemplary System

[0098] Figure 7 This is a block diagram of an image processing-based crushing operation monitoring system according to an embodiment of this application. Figure 7As shown, the image processing-based crushing operation monitoring system 300 according to an embodiment of this application includes: an image acquisition module 310; a dust removal module 320; a multi-scale convolution module 330; a difference module 340; a feature vector generation module 350; a context encoding module 360; and a classification result generation module 370.

[0099] The image acquisition module 310 is used to acquire images of the crushing operation after the operation captured by a camera; the dust removal module 320 is used to pass the images of the crushing operation after the operation through a dust removal generator based on a generative adversarial network to obtain a generated images of the crushing operation after the operation; the multi-scale convolution module 330 is used to pass the generated images of the crushing operation after the operation through a dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain first-scale feature maps and second-scale feature maps, wherein the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate. The kernel; the difference module 340 is used to calculate the difference feature map between the first scale feature map and the second scale feature map; the feature vector generation module 350 is used to expand each feature matrix along the channel dimension of the difference feature map into feature vectors to obtain multiple local difference feature vectors; the context encoding module 360 ​​is used to pass the multiple local difference feature vectors through a converter-based context encoder to obtain a global difference feature vector; and the classification result generation module 370 is used to pass the global difference feature vector through a classifier to obtain a classification result, the classification result being used to indicate whether the raw coal crushing effect meets a predetermined standard.

[0100] In one example, in the above-mentioned image processing-based crushing operation monitoring system 300, the dust removal module 320 includes: a generative adversarial network including a discriminator and a generator, wherein the step of passing the post-crushing operation image through the dust removal generator based on the generative adversarial network to obtain a generated post-crushing operation image includes: inputting the post-crushing operation image into the dust removal generator based on the generative adversarial network so that the generator of the generative adversarial network generates the generated post-crushing operation image by deconvolution encoding.

[0101] In one example, in the aforementioned image processing-based crushing operation monitoring system 300, the multi-scale convolution module 330 includes: performing convolution processing on the input data using the layers of the first convolutional neural network during the forward propagation of the layers; performing convolution processing on the input data to obtain a convolutional feature map; performing pooling processing on the convolutional feature map based on a local feature matrix to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activation feature map; wherein the output of the last layer of the first convolutional neural network is the first-scale feature map, and the first convolutional neural network... The input to the first layer of the network is the image generated after the crushing operation; and, each layer of the second convolutional neural network performs convolution processing on the input data with a second convolutional kernel during the forward propagation of the layer: convolution processing is performed on the input data to obtain a convolutional feature map; pooling processing based on the local feature matrix is ​​performed on the convolutional feature map to obtain a pooled feature map; and nonlinear activation is performed on the pooled feature map to obtain an activation feature map; wherein, the output of the last layer of the second convolutional neural network is the second scale feature map, and the input to the first layer of the second convolutional neural network is the image generated after the crushing operation.

[0102] In one example, in the image processing-based crushing operation monitoring system 300 described above, the differential module 340 includes: calculating a differential feature map between the first-scale feature map and the second-scale feature map using the following formula; wherein, the formula is:

[0103]

[0104] Where F1 represents the first scale feature map, F2 represents the second scale feature map, and F c This represents the difference feature map. This indicates differentiation based on position.

[0105] In one example, in the above-mentioned image processing-based crushing operation monitoring system 300, the feature vector generation module 350 includes: expanding each feature matrix along the channel dimension of the differential feature map into feature vectors along row vectors or column vectors to obtain the plurality of local differential feature vectors.

[0106] In one example, in the image processing-based crushing operation monitoring system 300 described above, the context encoding module 360 ​​includes: arranging the plurality of local difference feature vectors in one dimension to obtain a global difference feature vector; calculating the product between the global difference feature vector and the transpose of each difference feature vector in the plurality of local difference feature vectors to obtain a plurality of self-attention association matrices; standardizing each self-attention association matrix in the plurality of self-attention association matrices to obtain a plurality of standardized self-attention association matrices; applying a Softmax classification function to each standardized self-attention association matrix in the plurality of standardized self-attention association matrices to obtain a plurality of probability values; weighting each difference feature vector in the plurality of local difference feature vectors using each probability value as a weight to obtain a plurality of context semantic difference feature vectors; and concatenating the plurality of context semantic difference feature vectors to obtain the global difference feature vector.

[0107] In one example, in the above-mentioned image processing-based crushing operation monitoring system 300, the classification result generation module 370 includes: processing the global difference feature vector using the classifier with the following formula to obtain the classification result, the formula being:

[0108] softmax{(M c B c )}, softmax{(M c B c )|X}, where M c B is the weight matrix of the fully connected layer. c X represents the bias vector of the fully connected layer, and X is the global difference feature vector.

[0109] In summary, the image processing-based coal crushing operation monitoring system based on the embodiments of this application is explained. It employs machine learning-based artificial intelligence monitoring technology to extract multi-scale implicit feature distribution information in a high-dimensional feature space after dust removal from images of raw coal crushing operations. The system then represents the crushing effect characteristics of the raw coal after the crushing operation based on the differences between the multi-scale raw coal crushing image features. Furthermore, it detects the presence of large raw coal pieces of different sizes from the small raw coal pieces based on the global difference feature distribution, thereby intelligently monitoring the raw coal crushing effect. In this way, the crushing effect of the raw coal can be monitored in real time according to the actual situation, allowing for crushing control when the crushing effect does not meet predetermined standards, thus improving the efficiency of the raw coal crushing operation.

[0110] Exemplary electronic devices

[0111] Below, for reference Figure 8This describes an electronic device according to embodiments of the present application.

[0112] Figure 8 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0113] like Figure 8 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0114] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0115] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the functions of the image processing-based crushing operation monitoring method of the various embodiments of this application described above, and / or other desired functions. Various contents such as local differential feature vectors may also be stored in the computer-readable storage medium.

[0116] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0117] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0118] The output device 14 can output various information to the outside, including classification results. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0119] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0120] Exemplary computer program products and computer-readable storage media

[0121] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the image processing-based crushing operation monitoring method according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0122] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0123] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the image processing-based crushing operation monitoring method according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0124] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0125] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0126] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0127] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0128] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0129] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for monitoring crushing operations based on image processing, characterized in that, include: Acquire images of the crushing operation captured by a camera; The image after the crushing operation is processed by a dust removal generator based on a generative adversarial network to obtain a new image after the crushing operation. The generated fragmentation image is passed through a dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map. The first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate. Calculate the difference feature map between the first scale feature map and the second scale feature map; The feature matrices along the channel dimension of the differential feature map are expanded into feature vectors to obtain multiple local differential feature vectors. The plurality of local difference feature vectors are passed through a converter-based context encoder to obtain a global difference feature vector; and The global difference feature vector is passed through a classifier to obtain a classification result, which is used to indicate whether the raw coal crushing effect meets the predetermined standard. The process of obtaining a global difference feature vector by passing the multiple local difference feature vectors through a transducer-based context encoder includes: arranging the multiple local difference feature vectors in one dimension to obtain a global difference feature vector; calculating the product between the global difference feature vector and the transpose of each difference feature vector in the multiple local difference feature vectors to obtain multiple self-attention association matrices; standardizing each self-attention association matrix in the multiple self-attention association matrices to obtain multiple standardized self-attention association matrices; passing each standardized self-attention association matrix in the multiple standardized self-attention association matrices through a Softmax classification function to obtain multiple probability values; weighting each difference feature vector in the multiple local difference feature vectors using each probability value as a weight to obtain multiple context semantic difference feature vectors; and concatenating the multiple context semantic difference feature vectors to obtain the global difference feature vector.

2. The image processing-based crushing operation monitoring method according to claim 1, characterized in that, The adversarial generative network includes a discriminator and a generator. The step of passing the post-crushing image through a dust removal generator based on the adversarial generative network to obtain a generated post-crushing image includes: inputting the post-crushing image into the dust removal generator based on the adversarial generative network so that the generator of the adversarial generative network generates the generated post-crushing image by deconvolutional encoding.

3. The image processing-based crushing operation monitoring method according to claim 2, characterized in that, The step of passing the generated fragmentation image through a two-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map includes: Each layer of the first convolutional neural network performs convolution processing on the input data using a first convolutional kernel during the forward propagation of the layer: convolution processing is performed on the input data to obtain a convolutional feature map; pooling processing based on the local feature matrix is ​​performed on the convolutional feature map to obtain a pooled feature map; and non-linear activation is performed on the pooled feature map to obtain an activation feature map; wherein, the output of the last layer of the first convolutional neural network is the first scale feature map, and the input of the first layer of the first convolutional neural network is the image generated after the crushing operation; and Each layer of the second convolutional neural network performs convolution processing on the input data using a second convolutional kernel during the forward propagation of the layer: convolution processing is performed on the input data to obtain a convolutional feature map; pooling processing based on the local feature matrix is ​​performed on the convolutional feature map to obtain a pooled feature map; and non-linear activation is performed on the pooled feature map to obtain an activation feature map; wherein, the output of the last layer of the second convolutional neural network is the second scale feature map, and the input of the first layer of the second convolutional neural network is the image generated after the crushing operation.

4. The image processing-based crushing operation monitoring method according to claim 3, characterized in that, The calculation of the difference feature map between the first scale feature map and the second scale feature map includes: calculating the difference feature map between the first scale feature map and the second scale feature map using the following formula; The formula is as follows: Where F1 represents the first scale feature map, F2 represents the second scale feature map, and F c This represents the difference feature map. This indicates differentiation based on position.

5. The image processing-based crushing operation monitoring method according to claim 4, characterized in that, The step of expanding the feature matrices along the channel dimension of the differential feature map into feature vectors to obtain multiple local differential feature vectors includes: The feature matrices along the channel dimension of the differential feature map are expanded into feature vectors along row or column vectors to obtain the multiple local differential feature vectors.

6. The image processing-based crushing operation monitoring method according to claim 5, characterized in that, The step of passing the global difference feature vector through a classifier to obtain a classification result includes: processing the global difference feature vector using the classifier with the following formula to obtain the classification result, wherein the formula is: softmax{(M c B c )}, softmax{(M c B c )|X}, where M c B is the weight matrix of the fully connected layer. c X represents the bias vector of the fully connected layer, and X is the global difference feature vector.

7. The image processing-based crushing operation monitoring method according to claim 6, characterized in that, It also includes training steps: The dual-stream detection network, the converter-based context encoder, and the classifier are trained. The training steps include: Acquire training data, which includes images after the crushing operation and classification results of whether the raw coal crushing effect meets predetermined standards. The training image after the crushing operation is passed through the dust removal generator based on the adversarial generative network to obtain the training generated image after the crushing operation. The image generated after the training is processed is passed through the dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map. The first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate. Calculate the training difference feature map between the training first-scale feature map and the training second-scale feature map; The feature matrices along the channel dimension of the trained differential feature map are expanded into feature vectors to obtain multiple trained local difference feature vectors. The plurality of trained local difference feature vectors are passed through the converter-based context encoder to obtain a trained global difference feature vector; and The trained global difference feature vector is passed through the classifier to obtain the classification loss function value; The loss function value for the sequence pair sequence response rule internalization learning is calculated based on the distance between the training first-scale feature map and the training second-scale feature map; and The weighted sum of the classification loss function value and the sequence-to-sequence response rule internalization learning loss function value is calculated as the loss function value to train the two-stream detection network, the converter-based context encoder, and the classifier.

8. The image processing-based crushing operation monitoring method according to claim 7, characterized in that, The step of calculating the sequence-to-sequence response rule internalization learning loss function value based on the distance between the trained first-scale feature map and the trained second-scale feature map includes: The loss function value for the sequence-to-sequence response rule internalization learning is calculated based on the distance between the first-scale training feature map and the second-scale training feature map using the following formula: The formula is as follows: Where V1 and V2 are the feature vectors obtained after expanding the first-scale training feature map and the second-scale training feature map, respectively, and W1 and W2 are the weight matrices of the classifier for the feature vectors obtained after expanding the first-scale training feature map and the second-scale training feature map, respectively. ReLU(·) represents the ReLU activation function, and Sigmoid(·) represents the Sigmoid activation function. This represents matrix multiplication, and d(·,·) represents the Euclidean distance between the two vectors. This represents the value of the sequence-to-sequence response rule internalization learning loss function.

9. A crushing operation monitoring system based on image processing, characterized in that, include: The image acquisition module is used to acquire images of the crushing operation captured by the camera. The dust removal module is used to generate a post-crushing image by passing the post-crushing image through a dust removal generator based on a generative adversarial network. A multi-scale convolution module is used to pass the image generated after the fragmentation operation through a dual-stream detection network containing a first convolutional neural network model and a second convolutional neural network model to obtain a first-scale feature map and a second-scale feature map, wherein the first convolutional neural network model uses a first dilated convolutional kernel with a first dilation rate, and the second convolutional neural network model uses a second dilated convolutional kernel with a second dilation rate. The difference module is used to calculate the difference feature map between the first scale feature map and the second scale feature map; The feature vector generation module is used to expand each feature matrix along the channel dimension of the difference feature map into feature vectors to obtain multiple local difference feature vectors. A context encoding module is used to pass the plurality of local difference feature vectors through a converter-based context encoder to obtain a global difference feature vector; and The classification result generation module is used to pass the global differential feature vector through a classifier to obtain a classification result, which is used to indicate whether the raw coal crushing effect meets the predetermined standard. The process of obtaining a global difference feature vector by passing the multiple local difference feature vectors through a transducer-based context encoder includes: arranging the multiple local difference feature vectors in one dimension to obtain a global difference feature vector; calculating the product between the global difference feature vector and the transpose of each difference feature vector in the multiple local difference feature vectors to obtain multiple self-attention association matrices; standardizing each self-attention association matrix in the multiple self-attention association matrices to obtain multiple standardized self-attention association matrices; passing each standardized self-attention association matrix in the multiple standardized self-attention association matrices through a Softmax classification function to obtain multiple probability values; weighting each difference feature vector in the multiple local difference feature vectors using each probability value as a weight to obtain multiple context semantic difference feature vectors; and concatenating the multiple context semantic difference feature vectors to obtain the global difference feature vector.

Images