A Method and System for Monitoring Bucket Wheel Performance Based on Convolutional Neural Networks

Abstract

This application relates to the field of performance monitoring technology, specifically disclosing a bucket wheel performance monitoring method and system based on convolutional neural networks. First, a detection image and a reference image of the bucket wheel are acquired using a camera. Then, a convolutional neural network model is used as a feature extractor to map the detection image and the reference image into a high-dimensional feature space. The difference between the feature distributions of the two images in the high-dimensional feature space is calculated to obtain a differential feature map. Next, the differential feature map is passed through a residual dual attention mechanism model to obtain an enhanced differential feature map. Finally, the enhanced differential feature map is used as a classification feature map and passed through a classifier to obtain a classification result indicating whether the performance of the bucket wheel meets predetermined requirements. In this way, the performance of the bucket wheel is accurately monitored to prevent damage and accidents.

Description

Technical Field

[0001] This application relates to the field of performance monitoring technology, and more specifically, to a bucket wheel performance monitoring method and system based on convolutional neural networks. Background Technology

[0002] Bucket wheel excavators are crucial excavating equipment in continuous bucket wheel systems. Because the bucket wheel of a bucket wheel excavator comes into direct contact with ore, sand, and rock during operation, it is a critical and vulnerable component. Monitoring the performance of the bucket wheel is particularly important during the operation of the excavator. This is because if the bucket wheel's performance fails to meet application requirements and continues to be used, it will affect work efficiency. Furthermore, damage to the bucket wheel can lead to accidents and injuries.

[0003] Therefore, a bucket wheel performance monitoring solution is needed. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a method and system for monitoring bucket wheel performance based on a convolutional neural network. First, a detection image and a reference image of the bucket wheel are acquired using a camera. Then, a convolutional neural network model is used as a feature extractor to map the detection image and the reference image into a high-dimensional feature space. The difference between the feature distributions of the two images in the high-dimensional feature space is calculated to obtain a differential feature map. Preferably, considering the limited ability of the convolutional neural network model to selectively use features, the differential feature map is passed through a residual dual attention mechanism model to obtain an enhanced differential feature map. This enhanced differential feature map is then used as a classification feature map by a classifier to obtain a classification result indicating whether the performance of the bucket wheel meets predetermined requirements. In this way, the performance of the bucket wheel is accurately monitored to prevent damage and accidents.

[0005] According to one aspect of this application, a method for monitoring the performance of bucket wheels based on a convolutional neural network is provided, comprising:

[0006] Acquire bucket wheel detection images and bucket wheel reference images captured by the camera;

[0007] The bucket wheel detection image and the bucket wheel reference image are processed through a twin network model containing a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map, wherein the first image encoder and the second image encoder have the same network structure;

[0008] Calculate the difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map;

[0009] The differential feature map is processed through a residual dual attention mechanism model to obtain an enhanced differential feature map;

[0010] The enhanced differential feature map is passed through a classifier to obtain a classification result, which is used to indicate whether the performance of the bucket wheel to be tested meets the predetermined requirements.

[0011] According to another aspect of this application, a bucket wheel performance monitoring system based on a convolutional neural network is provided, comprising:

[0012] The image acquisition module is used to acquire the bucket wheel detection image and the bucket wheel reference image captured by the camera.

[0013] The twin coding module is used to pass the bucket wheel detection image and the bucket wheel reference image through a twin network model containing a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map, wherein the first image encoder and the second image encoder have the same network structure;

[0014] The difference module is used to calculate the difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map;

[0015] The feature enhancement module is used to pass the difference feature map through a residual dual attention mechanism model to obtain an enhanced difference feature map; and

[0016] The classification module is used to pass the enhanced differential feature map through a classifier to obtain a classification result, which is used to indicate whether the performance of the bucket wheel to be tested meets the predetermined requirements.

[0017] According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing computer program instructions, which, when executed by the processor, cause the processor to perform the bucket wheel performance monitoring method based on a convolutional neural network as described above.

[0018] According to another aspect of this application, a computer-readable medium is provided having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the bucket wheel performance monitoring method based on a convolutional neural network as described above.

[0019] Compared with existing technologies, this application provides a bucket wheel performance monitoring method and system based on convolutional neural networks. First, a bucket wheel detection image and a bucket wheel reference image are acquired using a camera. Then, a convolutional neural network model is used as a feature extractor to map the bucket wheel detection image and the bucket wheel reference image into a high-dimensional feature space. The difference between the feature distributions of the two images in the high-dimensional feature space is calculated to obtain a differential feature map. Preferably, considering the limited ability of the convolutional neural network model to selectively use features, the differential feature map is passed through a residual dual attention mechanism model to obtain an enhanced differential feature map. This enhanced differential feature map is then used as a classification feature map by a classifier to obtain a classification result indicating whether the performance of the bucket wheel meets predetermined requirements. In this way, the performance of the bucket wheel is accurately monitored to prevent damage to the bucket wheel, which could lead to accidents and injuries. Attached Figure Description

[0020] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0021] Figure 1 The illustration shows an application scenario of a bucket wheel performance monitoring method and system based on a convolutional neural network according to an embodiment of this application.

[0022] Figure 2 The illustration shows a flowchart of a bucket wheel performance monitoring method based on a convolutional neural network according to an embodiment of this application.

[0023] Figure 3 The figure shows a schematic diagram of the system architecture of the bucket wheel performance monitoring method based on a convolutional neural network according to an embodiment of this application.

[0024] Figure 4 The illustration shows a flowchart of the process of obtaining an enhanced differential feature map by passing the differential feature map through a residual dual attention mechanism model in the bucket wheel performance monitoring method based on a convolutional neural network according to an embodiment of this application.

[0025] Figure 5 The illustration shows a flowchart of a bucket wheel performance monitoring method based on a convolutional neural network according to an embodiment of this application, in which the differential feature map is passed through the spatial attention module of the residual dual attention mechanism model to obtain a spatial attention map.

[0026] Figure 6 The figure shows a block diagram of a bucket wheel performance monitoring system based on a convolutional neural network according to an embodiment of this application.

[0027] Figure 7 A block diagram of an electronic device according to an embodiment of this application is illustrated. Detailed Implementation

[0028] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0029] Scene Overview

[0030] As mentioned above, monitoring the performance of bucket wheels is particularly important during the operation of bucket wheel excavators. This is because if the performance of the bucket wheels fails to meet application requirements and they continue to be used, it will affect work efficiency. Furthermore, damage to the bucket wheels can cause accidents and injuries. Therefore, a bucket wheel performance monitoring solution is needed.

[0031] Traditional bucket wheel performance monitoring methods rely on static or dynamic testing of the bucket wheel's mechanical characteristics. This is time-consuming, labor-intensive, and requires specialized equipment and facilities, failing to meet application needs. In contrast, the present application addresses this issue by recognizing that changes in bucket wheel performance are characterized by their appearance. For example, oxidation alters surface color and texture; structural stress and impacts can cause cracks and scratches. Therefore, based on these observations, the applicant proposes a machine vision-based bucket wheel performance monitoring solution.

[0032] Specifically, the process begins by acquiring a bucket wheel detection image and a bucket wheel reference image captured by a camera. The purpose of introducing the bucket wheel reference image is to compare the differences between the bucket wheel detection image and the reference image, and to use this as a basis to determine whether the performance of the bucket wheel under test meets predetermined requirements. However, during testing, the applicant discovered that the bucket wheel surface contains numerous interfering elements during the acquisition of the bucket wheel detection image. For example, the bucket wheel surface may contain mud. While the introduction of these interfering elements widens the difference between the bucket wheel detection image and the bucket wheel reference image in the image source domain, this amplification of the difference actually leads to a decrease in the accuracy of performance detection. In other words, when comparing the bucket wheel detection image and the bucket wheel reference image, the focus should be on comparing image features that reflect the bucket wheel's performance, rather than directly comparing pixels one by one in the image source domain.

[0033] To address the aforementioned problems, the technical solution of this application uses a twin network model comprising a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map. The first image encoder and the second image encoder have the same network structure. Those skilled in the art should understand that training a deep neural network model in an appropriate manner can give the model specific functions. Accordingly, in the technical solution of this application, the deep neural network models used by the first image encoder and the second image encoder are trained using training data so that the first image encoder and the second image encoder can focus on extracting local image features that reflect the performance of the bucket wheel.

[0034] After obtaining the bucket wheel detection feature map and the bucket wheel reference feature map, a difference feature map is calculated between the bucket wheel detection feature map and the bucket wheel reference feature map to represent the difference in feature distribution between the two in the high-dimensional feature space, that is, the difference in feature distribution between the image feature distributions used to represent the bucket wheel performance. However, after a series of convolutions, the network obtains some feature information, but it does not automatically distinguish the detailed information between high and low frequencies and the differences between features of different categories. The network's ability to selectively use features is limited. Given that the attention mechanism can select the focus position and generate more discriminative feature representations, and that the features after adding the attention module will adaptively change as the network deepens, the technical solution of this application introduces an attention mechanism to enhance the features of the difference feature map.

[0035] Furthermore, based on the introduction of the attention mechanism, a residual structure is introduced and combined with a dual attention network to construct a residual dual attention model. This model combines spatial attention and channel attention in parallel, enabling a large amount of effective information of different types to be captured, which can effectively enhance the feature discrimination learning ability. During network training, the task processing system focuses more on finding significant useful information in the input data that is relevant to the current output, thereby improving the quality of the output. Moreover, the increasing number of attention modules will bring continuous performance improvement.

[0036] In other words, in the technical solution of this application, after obtaining the differential feature map, the differential feature map is further processed through a residual dual attention mechanism model to obtain an enhanced differential feature map. The enhanced differential feature map is then used as a classification feature map by a classifier to obtain a classification result indicating whether the performance of the bucket wheel to be tested meets predetermined requirements.

[0037] In the residual dual-attention mechanism, the difference feature map is first processed by a spatial attention module and a channel attention module to obtain a spatial attention map and a channel attention map, respectively, and then fused to obtain a weighted feature map. Then, using the residual cascading concept, the weighted feature map and the input feature map, i.e., the difference feature map, are fused to obtain the enhanced difference feature map. Here, the weighted feature map F′ and the input feature map, i.e., the difference feature map (denoted as F), are fused by point addition between feature maps. Therefore, it is desired that the feature distributions of the weighted feature map F′ and the difference feature map F have as small a spatial position error as possible within the high-dimensional feature space, thereby improving the fusion effect of the weighted feature map F′ and the difference feature map F.

[0038] In the technical solution of this application, considering that the weighted feature map F′ is obtained from the difference feature map F through a residual dual attention mechanism, the weighted feature map F′ and the difference feature map F are essentially from the same source, thus having a certain correspondence in feature distribution. Therefore, the weighted feature map F′ and the difference feature map F can be corrected by relative class angle probability information representation, as follows:

[0039]

[0040]

[0041]

[0042] in and These are the feature values ​​of the difference feature map F and the weighted feature map F′, respectively. and It is the mean of all feature values ​​of the difference feature map F and the weighted feature map F′.

[0043] Here, the relative class angle probability information representation correction uses the relative class angle probability information representation between the differential feature map F and the weighted feature map F′ to geometrically dilute the spatial position error of the feature distribution in the high-dimensional feature space of the differential feature map F and the weighted feature map F′. Thus, given a certain correspondence between the differential feature map F and the weighted feature map F′, based on the distribution constraints of the feature value distribution at each position of the differential feature map F and the weighted feature map F′ relative to their overall distribution, implicit contextual correspondence correction of features is performed through point-by-point regression at position, thereby improving the fusion effect of the differential feature map F and the weighted feature map F′, and correspondingly improving the accuracy of the classification result of the enhanced differential feature map. In other words, it improves the accuracy of bucket wheel performance monitoring.

[0044] Based on this, this application provides a method for monitoring bucket wheel performance based on a convolutional neural network, comprising: acquiring a bucket wheel detection image and a bucket wheel reference image captured by a camera; passing the bucket wheel detection image and the bucket wheel reference image through a Siamese network model containing a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map, wherein the first image encoder and the second image encoder have the same network structure; calculating a difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map; passing the difference feature map through a residual dual attention mechanism model to obtain an enhanced difference feature map; and passing the enhanced difference feature map through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the performance of the bucket wheel to be detected meets predetermined requirements.

[0045] Figure 1 The illustration shows an application scenario of a bucket wheel performance monitoring method and system based on a convolutional neural network according to an embodiment of this application. For example... Figure 1 As shown, in this application scenario, through a camera (e.g., Figure 1 The C) shown is a bucket wheel excavator (e.g., Figure 1 The bucket wheel (e.g., W) shown in the diagram Figure 1 The B) detection image and the bucket wheel reference image shown are then input into a server deployed with a bucket wheel performance monitoring algorithm based on a convolutional neural network (e.g., Figure 1 As shown in S), the server is able to use the convolutional neural network-based bucket wheel performance monitoring algorithm to process the bucket wheel detection image and the bucket wheel reference image to generate bucket wheel performance monitoring results.

[0046] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0047] Exemplary methods

[0048] Figure 2 The illustration shows a flowchart of a bucket wheel performance monitoring method based on a convolutional neural network according to an embodiment of this application. Figure 2As shown, the bucket wheel performance monitoring method based on convolutional neural networks according to an embodiment of this application includes: S110, acquiring a bucket wheel detection image and a bucket wheel reference image captured by a camera; S120, passing the bucket wheel detection image and the bucket wheel reference image through a Siamese network model including a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map, wherein the first image encoder and the second image encoder have the same network structure; S130, calculating a difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map; S140, passing the difference feature map through a residual dual attention mechanism model to obtain an enhanced difference feature map; S150, passing the enhanced difference feature map through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the performance of the bucket wheel to be detected meets predetermined requirements.

[0049] Figure 3 The illustration shows a schematic diagram of the system architecture of a bucket wheel performance monitoring method based on a convolutional neural network according to an embodiment of this application. Figure 3 As shown in the system architecture of the bucket wheel performance monitoring method based on convolutional neural networks described in this application embodiment, firstly, a bucket wheel detection image captured by a camera is acquired, and the bucket wheel detection image is passed through a first image encoder of a Siamese network model to obtain a bucket wheel detection feature map. Simultaneously, a bucket wheel reference image captured by the camera is acquired, and the bucket wheel reference image is passed through a second image encoder of the Siamese network model to obtain a bucket wheel reference feature map. Then, a difference feature map is calculated between the bucket wheel detection feature map and the bucket wheel reference feature map, and the difference feature map is passed through a residual dual attention mechanism model to obtain an enhanced difference feature map. Finally, the enhanced difference feature map is passed through a classifier to obtain a classification result, which indicates whether the performance of the bucket wheel to be detected meets predetermined requirements.

[0050] In S110 of this embodiment, a bucket wheel detection image and a bucket wheel reference image captured by a camera are acquired. As mentioned above, traditional bucket wheel performance monitoring schemes are implemented by performing static or dynamic tests on the mechanical characteristics of the bucket wheel, which is not only time-consuming and labor-intensive, but also requires specialized equipment and facilities, failing to meet application requirements. Accordingly, in the technical solution of this application, considering that changes in bucket wheel performance can be characterized at the appearance level, for example, its surface color and texture may change due to oxidation; and its surface may develop cracks and scratches due to structural stress and collisions. That is to say, performance can be detected through images. Of course, considering that the human eye cannot accurately observe and judge changes in bucket wheel performance through images, this application attempts to construct a bucket wheel performance monitoring scheme based on machine vision.

[0051] Specifically, the process begins by acquiring a bucket wheel detection image and a bucket wheel reference image captured by a camera. The purpose of introducing the bucket wheel reference image is to compare the differences between the bucket wheel detection image and the reference image, and to use this as a basis to determine whether the performance of the bucket wheel under test meets predetermined requirements. However, during testing, the applicant discovered that the bucket wheel surface contains numerous interfering elements during the acquisition of the bucket wheel detection image. For example, the bucket wheel surface may contain mud. While the introduction of these interfering elements widens the difference between the bucket wheel detection image and the bucket wheel reference image in the image source domain, this amplification of the difference actually leads to a decrease in the accuracy of performance detection. In other words, when comparing the bucket wheel detection image and the bucket wheel reference image, the focus should be on comparing image features that reflect the bucket wheel's performance, rather than directly comparing pixels one by one in the image source domain.

[0052] In S120 of this embodiment, the bucket wheel detection image and the bucket wheel reference image are processed through a Siamese network model including a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map, respectively. The first image encoder and the second image encoder have the same network structure. It should be understood that, considering the possibility of mud stains on the bucket wheel surface affecting the detection results, the technical solution of this application maps the bucket wheel detection image and the bucket wheel reference image to a high-dimensional feature space using an image encoder. Specifically, the bucket wheel detection image and the bucket wheel reference image are processed through a Siamese network model including a first image encoder and a second image encoder to obtain the bucket wheel detection feature map and the bucket wheel reference feature map. Simultaneously, considering that the bucket wheel detection image and the bucket wheel reference image have the same data volume and data distribution at the source domain, the first image encoder and the second image encoder use the same network structure, thereby unifying the dimension and size of the feature maps obtained by the first image encoder and the second image encoder, facilitating subsequent calculation of feature distribution differences in the high-dimensional feature space.

[0053] In a specific embodiment of this application, the first image encoder and the second image encoder are deep convolutional neural network models. It should be understood that, considering the excellent performance of deep convolutional neural network models in image feature extraction, which can extract the essential features of an image, this embodiment further uses a convolutional neural network model as a feature extractor to map the bucket wheel detection image and the bucket wheel reference image to a high-dimensional feature space. Preferably, those skilled in the art should know that the deep neural network model can also be trained in an appropriate way to give it specific functions. Accordingly, in the technical solution of this application, the deep neural network models used by the first image encoder and the second image encoder are trained using training data so that the first image encoder and the second image encoder can focus on extracting local image features that reflect the performance of the bucket wheel.

[0054] In a specific embodiment of this application, the step of obtaining a bucket wheel detection feature map and a bucket wheel reference feature map by passing the bucket wheel detection image and the bucket wheel reference image through a Siamese network model including a first image encoder and a second image encoder includes: performing convolution processing, pooling processing, and nonlinear activation processing on the input data using each layer of the first image encoder during the forward pass of the layer, so that the bucket wheel detection feature map is output by the last layer of the first image encoder; and performing convolution processing, pooling processing, and nonlinear activation processing on the input data using each layer of the second image encoder during the forward pass of the layer, so that the bucket wheel reference feature map is output by the last layer of the second image encoder.

[0055] More specifically, in this embodiment, during the forward propagation of each layer of the first image encoder, the input image is processed as follows: a two-dimensional convolutional kernel slides across the input image, and values ​​are calculated at each position to extract high-dimensional local latent features of the input image to obtain the convolutional feature map; then, the convolutional feature map is processed by mean pooling or max pooling based on the local feature matrix to obtain the pooled feature map. Through global pooling, the main features can be extracted, while reducing the number of parameters and overfitting. Then, an activation function is selected to activate the pooled feature map to obtain the activated feature map, such as the sigmoid activation function. The activation function introduces nonlinear factors to increase the representational power of the entire network.

[0056] In S130 of this embodiment, a difference feature map is calculated between the bucket wheel detection feature map and the bucket wheel reference feature map. It should be understood that, considering that directly inputting the bucket wheel detection feature map into the classifier would lead to a decrease in the accuracy of the detection results due to the lack of comparison with the bucket wheel reference feature map, after obtaining the bucket wheel detection feature map and the bucket wheel reference feature map, a difference feature map is calculated between them to represent the difference in feature distribution between the two in the high-dimensional feature space, that is, the difference in feature distribution between the image feature distributions used to represent the bucket wheel performance.

[0057] In a specific embodiment of this application, calculating the difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map includes: calculating the difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map using the following formula;

[0058] The formula is as follows:

[0059]

[0060] Among them, F a This represents the bucket wheel detection feature map. F represents the difference based on position. b This represents the reference feature diagram of the bucket wheel, and F c This represents the differential feature map.

[0061] In S140 of this embodiment, the differential feature map is processed through a residual dual attention mechanism model to obtain an enhanced differential feature map. It should be understood that, considering that after a series of convolutions, the network obtains some feature information, but does not automatically distinguish detailed information between high and low frequencies or differences between features of different categories, the network's ability to selectively use features is limited. Given that the attention mechanism can select the focus position, generating more discriminative feature representations, and that the features after adding the attention module will adaptively change as the network deepens, the technical solution of this application introduces an attention mechanism to enhance the features of the differential feature map.

[0062] Furthermore, based on the introduction of the attention mechanism, a residual structure is introduced and combined with a dual-attention network to construct a residual dual-attention model. This model combines spatial attention and channel attention in parallel, enabling the capture of a large amount of effective information of different types, which can effectively enhance the feature discrimination learning ability. During network training, the task processing system focuses more on finding significant useful information in the input data that is relevant to the current output, thereby improving the quality of the output, and the increasing number of attention modules will bring continuous performance improvement. That is, in the technical solution of this application, after obtaining the differential feature map, the differential feature map is further processed through the residual dual-attention mechanism model to obtain an enhanced differential feature map.

[0063] Figure 4 The illustration shows a flowchart of the process of obtaining an enhanced differential feature map by passing the differential feature map through a residual dual attention mechanism model in a bucket wheel performance monitoring method based on a convolutional neural network according to an embodiment of this application. Figure 4 As shown, in a specific embodiment of this application, the step of obtaining an enhanced differential feature map by passing the differential feature map through a residual dual attention mechanism model includes: S210, passing the differential feature map through the spatial attention module of the residual dual attention mechanism model to obtain a spatial attention map; S220, passing the differential feature map through the channel attention module of the residual dual attention mechanism model to obtain a channel attention map; S230, fusing the spatial attention map and the channel attention map to obtain a weighted feature map; and S240, fusing the differential feature map and the weighted feature map to obtain the enhanced differential feature map.

[0064] Figure 5 The illustration shows a flowchart of a bucket wheel performance monitoring method based on a convolutional neural network according to an embodiment of this application, in which the differential feature map is passed through the spatial attention module of the residual dual attention mechanism model to obtain a spatial attention map. Figure 5 As shown, in a specific embodiment of this application, the step of obtaining a spatial attention map by passing the differential feature map through the spatial attention module of the residual dual attention mechanism model includes: S310, performing average pooling and max pooling along the channel dimension on the differential feature map to obtain an average feature matrix and a maximum feature matrix; S320, concatenating the average feature matrix and the maximum feature matrix and adjusting the channels to obtain a channel feature matrix; S330, using the convolutional layer of the spatial attention module to perform convolutional encoding on the channel feature matrix to obtain a convolutional feature matrix; S340, passing the convolutional feature matrix through the Softmax function to obtain a spatial attention score matrix; and S350, performing positional dot multiplication of the spatial attention score matrix with each feature matrix along the channel dimension of the differential feature map to obtain the spatial attention map.

[0065] In a specific embodiment of this application, the step of obtaining a channel attention map by passing the differential feature map through the channel attention module of the residual dual attention mechanism model includes: performing global mean pooling on each feature matrix along the channel dimension of the differential feature map to obtain a channel feature vector; passing the channel feature vector through the Softmax function to obtain a normalized channel feature vector; and using the feature values ​​at each position in the normalized channel feature vector as weights to weight the feature matrix along the channel dimension of the differential feature map to obtain the channel attention map.

[0066] Specifically, in the technical solution of this application, in the residual dual attention mechanism, the differential feature map is first processed by a spatial attention module and a channel attention module respectively to obtain a spatial attention map and a channel attention map, and then the two are fused to obtain a weighted feature map. Then, using the residual cascading idea, the weighted feature map and the input feature map, i.e., the differential feature map, are fused to obtain the enhanced differential feature map. Here, the weighted feature map F′ and the input feature map, i.e., the differential feature map (denoted as F), are fused by point addition between feature maps. Therefore, it is desired that the feature distributions of the weighted feature map F′ and the differential feature map F have as small a spatial position error as possible within the high-dimensional feature space, thereby improving the fusion effect of the weighted feature map F′ and the differential feature map F. In the technical solution of this application, considering that the weighted feature map F′ is obtained from the difference feature map F through the residual dual attention mechanism, the weighted feature map F′ and the difference feature map F are essentially from the same source, and thus have a certain correspondence in feature distribution. Therefore, the weighted feature map F′ and the difference feature map F can be corrected by relative class angle probability information representation.

[0067] In one specific embodiment of this application, fusing the spatial attention map and the channel attention map to obtain a weighted feature map includes: calculating the weighted feature map by position points between the spatial attention map and the channel attention map.

[0068] In a specific embodiment of this application, fusing the differential feature map and the weighted feature map to obtain the enhanced differential feature map includes: based on the weighted feature map, performing relative class angle probability information representation correction on the differential feature map using the following formula to obtain the corrected differential feature map;

[0069] The formula is as follows:

[0070]

[0071] in and These are the feature values ​​of the difference feature map F and the weighted feature map F′, respectively. and It is the mean of all feature values ​​of the difference feature map F and the weighted feature map F′, where log represents the logarithmic function value with base 2; based on the difference feature map, the weighted feature map is corrected by relative class angle probability information representation using the following formula to obtain the corrected weighted feature map;

[0072] The formula is as follows:

[0073]

[0074] in and These are the feature values ​​of the difference feature map F and the weighted feature map F′, and and It is the mean of all eigenvalues ​​of the difference feature map F and the weighted feature map F′, where log represents the logarithmic function value to the base 2; and the enhanced difference feature map is obtained by calculating the position points between the corrected difference feature map and the corrected weighted feature map.

[0075] Here, the relative class angle probability information representation correction uses the relative class angle probability information representation between the differential feature map F and the weighted feature map F′ to geometrically dilute the spatial position error of the feature distribution in the high-dimensional feature space of the differential feature map F and the weighted feature map F′. Thus, given a certain correspondence between the differential feature map F and the weighted feature map F′, based on the distribution constraints of the feature value distribution at each position of the differential feature map F and the weighted feature map F′ relative to their overall distribution, implicit contextual correspondence correction of features is performed through point-by-point regression at position, thereby improving the fusion effect of the differential feature map F and the weighted feature map F′, and correspondingly improving the accuracy of the classification result of the enhanced differential feature map. In other words, it improves the accuracy of bucket wheel performance monitoring.

[0076] In S150 of this application embodiment, the enhanced differential feature map is passed through a classifier to obtain a classification result, which is used to indicate whether the performance of the bucket wheel to be detected meets the predetermined requirements.

[0077] In a specific embodiment of this application, the step of passing the enhanced difference feature map through a classifier to obtain a classification result includes: processing the enhanced difference feature map using the classifier with the following formula to obtain the classification result; wherein, the formula is: O = softmax{(W c B c)|Project(F)}, where Project(F) represents projecting the enhanced differential feature map into a vector, W c Let B be the weight matrix. c This represents the bias vector.

[0078] That is, the enhanced difference feature map is projected into a one-dimensional classification feature vector. Then, the fully connected layer of the classifier is used to fully encode the classification feature vector to fully utilize the information at each position in the difference feature map, thereby obtaining the encoded classification feature vector. Next, the Softmax function value of the encoded classification feature vector is calculated, that is, the probability value of the encoded classification feature vector belonging to each classification label. In this embodiment, the classification labels include whether the performance of the bucket wheel to be detected meets predetermined requirements (first label) and whether the performance of the bucket wheel to be detected does not meet predetermined requirements (second label). Finally, the label corresponding to the larger probability value is taken as the classification result.

[0079] In summary, the bucket wheel performance monitoring method based on convolutional neural networks of this application has been clarified. First, a bucket wheel detection image and a bucket wheel reference image are acquired using a camera. Then, a convolutional neural network model is used as a feature extractor to map the bucket wheel detection image and the bucket wheel reference image into a high-dimensional feature space. The difference between the feature distributions of the two images in the high-dimensional feature space is calculated to obtain a differential feature map. Preferably, considering the limited ability of the convolutional neural network model to selectively use features, the differential feature map is passed through a residual dual attention mechanism model to obtain an enhanced differential feature map. This enhanced differential feature map is then used as a classification feature map by a classifier to obtain a classification result indicating whether the performance of the bucket wheel to be detected meets predetermined requirements. In this way, the performance of the bucket wheel is accurately monitored to prevent damage to the bucket wheel, which could lead to accidents and injuries.

[0080] Exemplary System

[0081] Figure 6 The diagram illustrates a block diagram of a bucket wheel performance monitoring system based on a convolutional neural network according to an embodiment of this application. Figure 6As shown, the bucket wheel performance monitoring system 100 based on a convolutional neural network according to an embodiment of this application includes: an image acquisition module 110, used to acquire a bucket wheel detection image and a bucket wheel reference image captured by a camera; a twin coding module 120, used to pass the bucket wheel detection image and the bucket wheel reference image through a twin network model including a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map, wherein the first image encoder and the second image encoder have the same network structure; a difference module 130, used to calculate a difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map; a feature enhancement module 140, used to pass the difference feature map through a residual dual attention mechanism model to obtain an enhanced difference feature map; and a classification module 150, used to pass the enhanced difference feature map through a classifier to obtain a classification result, wherein the classification result is used to indicate whether the performance of the bucket wheel to be detected meets a predetermined requirement.

[0082] In one specific embodiment of this application, the first image encoder and the second image encoder are deep convolutional neural network models.

[0083] In one specific embodiment of this application, the twin coding module includes: a first coding unit, configured to perform convolution processing, pooling processing, and nonlinear activation processing on the input data using each layer of the first image encoder during the forward pass of the layers, so as to output the bucket wheel detection feature map by the last layer of the first image encoder; and a second coding unit, configured to perform convolution processing, pooling processing, and nonlinear activation processing on the input data using each layer of the second image encoder during the forward pass of the layers, so as to output the bucket wheel reference feature map by the last layer of the second image encoder.

[0084] In one specific embodiment of this application, the differential module includes: calculating a differential feature map between the bucket wheel detection feature map and the bucket wheel reference feature map using the following formula;

[0085] The formula is as follows:

[0086]

[0087] Among them, F a This represents the bucket wheel detection feature map. F represents the difference based on position. b This represents the reference feature diagram of the bucket wheel, and F c This represents the differential feature map.

[0088] In a specific embodiment of this application, the feature enhancement module includes: a spatial attention encoding unit, used to pass the differential feature map through the spatial attention module of the residual dual attention mechanism model to obtain a spatial attention map; a channel attention encoding unit, used to pass the differential feature map through the channel attention module of the residual dual attention mechanism model to obtain a channel attention map; a weighting unit, used to fuse the spatial attention map and the channel attention map to obtain a weighted feature map; and a fusion unit, used to fuse the differential feature map and the weighted feature map to obtain the enhanced differential feature map.

[0089] In a specific embodiment of this application, the spatial attention encoding unit includes: a pooling subunit, used to perform average pooling and max pooling along the channel dimension on the difference feature map to obtain an average feature matrix and a maximum feature matrix; an aggregation subunit, used to concatenate the average feature matrix and the maximum feature matrix and adjust the channels to obtain a channel feature matrix; a convolution subunit, used to perform convolutional encoding on the channel feature matrix using the convolutional layer of the spatial attention module to obtain a convolutional feature matrix; an activation subunit, used to pass the convolutional feature matrix through the Softmax function to obtain a spatial attention score matrix; and a spatial attention application subunit, used to perform positional dot multiplication of the spatial attention score matrix with each feature matrix along the channel dimension of the difference feature map to obtain the spatial attention map.

[0090] In a specific embodiment of this application, the channel attention encoding unit includes: a global pooling subunit, used to perform global mean pooling on each feature matrix along the channel dimension of the difference feature map to obtain a channel feature vector; a normalization subunit, used to pass the channel feature vector through the Softmax function to obtain a normalized channel feature vector; and a channel attention application subunit, used to weight the feature matrix along the channel dimension of the difference feature map using the feature values ​​at each position in the normalized channel feature vector as weights to obtain a channel attention map.

[0091] In a specific embodiment of this application, the weighting unit includes: calculating the weighted feature map by position points between the spatial attention map and the channel attention map.

[0092] In a specific embodiment of this application, the fusion unit includes: a first correction subunit, used to perform relative class angle probability information representation correction on the differential feature map based on the weighted feature map using the following formula to obtain a corrected differential feature map;

[0093] The formula is as follows:

[0094]

[0095] in and These are the feature values ​​of the difference feature map F and the weighted feature map F′, respectively. and It is the mean of all feature values ​​of the difference feature map F and the weighted feature map F′, where log represents the logarithmic function value with base 2; the second correction subunit is used to perform relative class angle probability information representation correction on the weighted feature map based on the difference feature map using the following formula to obtain the corrected weighted feature map;

[0096] The formula is as follows:

[0097]

[0098] in and These are the feature values ​​of the difference feature map F and the weighted feature map F′, and and It is the mean of all feature values ​​of the differential feature map F and the weighted feature map F′, where log represents the logarithmic function value to base 2; and a point addition sub-unit is used to calculate the enhanced differential feature map by adding the position points between the corrected differential feature map and the corrected weighted feature map.

[0099] In a specific embodiment of this application, the classification module includes: processing the enhanced differential feature map using the classifier with the following formula to obtain the classification result; wherein, the formula is: O = softmax{(W c B c )|Project(F)}, where Project(F) represents projecting the enhanced differential feature map into a vector, W c Let B be the weight matrix. c This represents the bias vector.

[0100] Here, those skilled in the art will understand that the specific functions and operations of each unit and module in the above-mentioned convolutional neural network-based bucket wheel performance monitoring system have been referenced above. Figures 1 to 5 The description of the bucket wheel performance monitoring method based on convolutional neural networks is detailed here, and therefore, its repeated description will be omitted.

[0101] As described above, the convolutional neural network-based bucket wheel performance monitoring system 100 according to the embodiments of this application can be implemented in various terminal devices, such as servers deploying convolutional neural network-based bucket wheel performance monitoring algorithms. In one example, the convolutional neural network-based bucket wheel performance monitoring system 100 can be integrated into the terminal device as a software module and / or hardware module. For example, the convolutional neural network-based bucket wheel performance monitoring system 100 can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the convolutional neural network-based bucket wheel performance monitoring system 100 can also be one of many hardware modules of the terminal device.

[0102] Alternatively, in another example, the convolutional neural network-based bucket wheel performance monitoring system 100 and the terminal device can also be separate devices, and the convolutional neural network-based bucket wheel performance monitoring system 100 can be connected to the terminal device via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0103] Exemplary electronic devices

[0104] Below, for reference Figure 7 This describes an electronic device according to embodiments of the present application.

[0105] Figure 7 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0106] like Figure 7 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0107] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0108] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the bucket wheel performance monitoring and / or other desired functions of the various embodiments of this application described above. Various contents, such as bucket wheel detection images and bucket wheel reference images captured by a camera, may also be stored in the computer-readable storage medium.

[0109] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0110] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0111] The output device 14 can output various information to the outside, including classification results. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0112] Exemplary computer program products and computer-readable storage media

[0113] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the convolutional neural network-based bucket wheel performance monitoring method according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0114] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0115] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the bucket wheel performance monitoring method based on a convolutional neural network according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0116] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0117] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0118] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0119] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0120] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0121] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for monitoring the performance of bucket wheels based on convolutional neural networks, characterized in that, include: Acquire bucket wheel detection images and bucket wheel reference images captured by the camera; The bucket wheel detection image and the bucket wheel reference image are processed through a twin network model containing a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map, wherein the first image encoder and the second image encoder have the same network structure; Calculate the difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map; The differential feature map is processed through a residual dual attention mechanism model to obtain an enhanced differential feature map, including: The difference feature map is passed through the spatial attention module of the residual dual attention mechanism model to obtain a spatial attention map; The difference feature map is passed through the channel attention module of the residual dual attention mechanism model to obtain the channel attention map; The spatial attention map and the channel attention map are fused to obtain a weighted feature map; The process of fusing the differential feature map and the weighted feature map to obtain the enhanced differential feature map includes: based on the weighted feature map, performing relative class angle probability information representation correction on the differential feature map using the following formula to obtain the corrected differential feature map; The formula is as follows: in and These are the feature values ​​of the differential feature map F and the weighted feature map F', respectively. and It is the mean of all feature values ​​of the difference feature map F and the weighted feature map F', and log represents the logarithmic function value with base 2; Based on the difference feature map, the weighted feature map is corrected by relative class angle probability information representation using the following formula to obtain the corrected weighted feature map; The formula is as follows: in and These are the feature values ​​of the differential feature map F and the weighted feature map F', respectively. and It is the mean of all feature values ​​of the difference feature map F and the weighted feature map F', and log represents the logarithmic function value with base 2; The enhanced differential feature map is obtained by calculating the position points between the corrected differential feature map and the corrected weighted feature map; The enhanced differential feature map is passed through a classifier to obtain a classification result, which is used to indicate whether the performance of the bucket wheel to be tested meets the predetermined requirements.

2. The bucket wheel performance monitoring method based on convolutional neural networks according to claim 1, characterized in that, The first image encoder and the second image encoder are deep convolutional neural network models.

3. The bucket wheel performance monitoring method based on convolutional neural networks according to claim 2, characterized in that, The step of passing the bucket wheel detection image and the bucket wheel reference image through a Siamese network model containing a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map includes: Each layer of the first image encoder performs convolution, pooling, and nonlinear activation processing on the input data during the forward pass of the layer to output the bucket wheel detection feature map from the last layer of the first image encoder; and Each layer of the second image encoder performs convolution, pooling, and nonlinear activation processing on the input data during the forward pass of the layer to output the bucket wheel reference feature map by the last layer of the second image encoder.

4. The bucket wheel performance monitoring method based on convolutional neural networks according to claim 3, characterized in that, The calculation of the difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map includes: calculating the difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map using the following formula; The formula is as follows: Among them, F a This represents the bucket wheel detection feature map. F represents the difference based on position. b This represents the reference feature diagram of the bucket wheel, and F c This represents the differential feature map.

5. The bucket wheel performance monitoring method based on convolutional neural networks according to claim 4, characterized in that, The step of passing the differential feature map through the spatial attention module of the residual dual attention mechanism model to obtain a spatial attention map includes: The difference feature map is subjected to average pooling and max pooling along the channel dimension to obtain the average feature matrix and the max feature matrix, respectively; The average feature matrix and the maximum feature matrix are concatenated and channel-adjusted to obtain the channel feature matrix; The convolutional layer of the spatial attention module is used to perform convolutional encoding on the channel feature matrix to obtain the convolutional feature matrix; The convolutional feature matrix is ​​passed through the Softmax function to obtain the spatial attention score matrix; The spatial attention map is obtained by performing positional multiplication of the spatial attention score matrix with each feature matrix of the differential feature map along the channel dimension.

6. The bucket wheel performance monitoring method based on convolutional neural networks according to claim 5, characterized in that, The step of passing the differential feature map through the channel attention module of the residual dual attention mechanism model to obtain a channel attention map includes: Global mean pooling is performed on each feature matrix along the channel dimension of the difference feature map to obtain the channel feature vector; The channel feature vectors are passed through the Softmax function to obtain normalized channel feature vectors; The feature matrix along the channel dimension of the differential feature map is weighted using the feature values ​​at each position in the normalized channel feature vector as weights to obtain the channel attention map.

7. The bucket wheel performance monitoring method based on convolutional neural networks according to claim 6, characterized in that, Fusing the spatial attention map and the channel attention map to obtain a weighted feature map includes: calculating the weighted feature map by position points between the spatial attention map and the channel attention map.

8. The bucket wheel performance monitoring method based on convolutional neural networks according to claim 7, characterized in that, The step of passing the enhanced differential feature map through a classifier to obtain a classification result includes: processing the enhanced differential feature map using the classifier with the following formula to obtain the classification result; The formula is: O = softmax{(W c B c )|Project(F)}, where Project(F) represents projecting the enhanced differential feature map into a vector, W c Let B be the weight matrix. c This represents the bias vector.

9. A bucket wheel performance monitoring system based on a convolutional neural network, characterized in that, include: The image acquisition module is used to acquire the bucket wheel detection image and bucket wheel reference image captured by the camera; The twin coding module is used to pass the bucket wheel detection image and the bucket wheel reference image through a twin network model containing a first image encoder and a second image encoder to obtain a bucket wheel detection feature map and a bucket wheel reference feature map, wherein the first image encoder and the second image encoder have the same network structure; The difference module is used to calculate the difference feature map between the bucket wheel detection feature map and the bucket wheel reference feature map; The feature enhancement module is used to pass the difference feature map through a residual dual attention mechanism model to obtain an enhanced difference feature map, including: The difference feature map is passed through the spatial attention module of the residual dual attention mechanism model to obtain a spatial attention map; The difference feature map is passed through the channel attention module of the residual dual attention mechanism model to obtain the channel attention map; The spatial attention map and the channel attention map are fused to obtain a weighted feature map; The process of fusing the differential feature map and the weighted feature map to obtain the enhanced differential feature map includes: based on the weighted feature map, performing relative class angle probability information representation correction on the differential feature map using the following formula to obtain the corrected differential feature map; The formula is as follows: in and These are the feature values ​​of the differential feature map F and the weighted feature map F', respectively. and It is the mean of all feature values ​​of the difference feature map F and the weighted feature map F', and log represents the logarithmic function value with base 2; Based on the difference feature map, the weighted feature map is corrected by relative class angle probability information representation using the following formula to obtain the corrected weighted feature map; The formula is as follows: in and These are the feature values ​​of the differential feature map F and the weighted feature map F', respectively. and It is the mean of all feature values ​​of the difference feature map F and the weighted feature map F', and log represents the logarithmic function value with base 2; The enhanced differential feature map is obtained by calculating the position points between the corrected differential feature map and the corrected weighted feature map; The classification module is used to pass the enhanced differential feature map through a classifier to obtain a classification result, which is used to indicate whether the performance of the bucket wheel to be tested meets the predetermined requirements.

Images