Real-time monitoring method and system for the attitude of the bucket crossbeam of a gantry excavator

Abstract

This invention discloses a method and system for real-time monitoring of the attitude of the bucket crossbeam of a gantry bucket excavator. First, it acquires an image of the bucket crossbeam captured by a camera. Next, it acquires the bending deformation of the bucket crossbeam at multiple predetermined time points within a predetermined time period, collected by a displacement sensor. Then, it arranges the bending deformation at these multiple predetermined time points into a bending deformation time-series input vector. Next, it performs image feature analysis on the bucket crossbeam image to obtain a bucket crossbeam feature map. Then, it performs cross-modal fusion feature analysis on the bucket crossbeam feature map and the bending deformation time-series input vector to obtain a bucket crossbeam feature with fused bending deformation time-series features. Finally, based on the bucket crossbeam feature with fused bending deformation time-series features, it determines whether a bucket crossbeam safety warning should be generated. The monitoring method and system of this invention can improve monitoring efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of intelligent monitoring, and more specifically, to a method and system for real-time monitoring of the attitude of the crossbeam of a gantry bucket wheel excavator. Background Technology

[0002] A gantry bucket wheel excavator is a large mechanical device used for loading and unloading bulk materials (such as coal and ore). It consists of a large gantry structure and bucket wheels suspended on the gantry structure, and is widely used in thermal power plants and other fields. The bucket wheel crossbeam of the bucket wheel excavator is a key component that bears the load and bending moment, and its safety is crucial to ensuring the smooth progress of loading and unloading operations.

[0003] However, due to long-term changes in the working environment and load, the bucket wheel crossbeam may suffer damage such as deformation and fatigue cracks, affecting the safety and reliability of the gantry bucket wheel excavator. Therefore, real-time monitoring of the attitude and deformation of the bucket wheel crossbeam, and timely detection and early warning of abnormalities, are important measures to ensure the normal and safe operation of the gantry bucket wheel excavator.

[0004] However, traditional monitoring methods for the bucket wheel crossbeam of gantry bucket excavators mainly rely on regular inspections and monitoring by professional personnel. This approach requires significant manpower and time, and has low monitoring efficiency and accuracy. Furthermore, traditional methods can only obtain the status information of the bucket wheel crossbeam during regular inspections, and cannot monitor the attitude and deformation of the crossbeam in real time. Therefore, during inspections, some potential problems or anomalies may be missed, and if an anomaly occurs, the opportunity to handle it may be delayed, increasing safety risks.

[0005] Therefore, a real-time monitoring scheme for the attitude of the crossbeam of the bucket wheel of a gantry bucket excavator is desired. Summary of the Invention

[0006] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a method and system for real-time monitoring of the attitude of the bucket wheel crossbeam of a gantry bucket wheel excavator. This method enables real-time monitoring of the attitude and deformation of the bucket wheel crossbeam, improving monitoring efficiency and accuracy. This helps to promptly detect abnormalities in the bucket wheel crossbeam and take corresponding measures to ensure the safe operation of the bucket wheel excavator.

[0007] According to one aspect of this application, a method for real-time monitoring of the attitude of the crossbeam of a gantry bucket excavator is provided, comprising:

[0008] Acquire images of the bucket crossbeam captured by a camera;

[0009] The bending deformation of the bucket crossbeam at multiple predetermined time points within a predetermined time period is obtained by a displacement sensor.

[0010] The bending deformation at the multiple predetermined time points is arranged according to the time dimension to form a bending deformation time sequence input vector;

[0011] Image feature analysis is performed on the image of the bucket wheel crossbeam to obtain a feature map of the bucket wheel crossbeam;

[0012] The bucket crossbeam feature map and the bending deformation time-series input vector are subjected to cross-modal element fusion feature analysis to obtain the bucket crossbeam feature with fused bending deformation time-series features; and

[0013] Based on the characteristics of the bucket crossbeam with the fused bending deformation time sequence, it is determined whether a safety warning for the bucket crossbeam should be generated.

[0014] According to another aspect of this application, a real-time monitoring system for the attitude of the bucket crossbeam of a gantry bucket excavator is provided, comprising:

[0015] The image acquisition module is used to acquire images of the bucket beam captured by the camera.

[0016] The bending deformation acquisition module is used to acquire the bending deformation of the bucket beam at multiple predetermined time points within a predetermined time period, which is collected by the displacement sensor.

[0017] The vectorization module is used to arrange the bending deformation at the multiple predetermined time points into a bending deformation time sequence input vector according to the time dimension.

[0018] The image feature analysis module is used to perform image feature analysis on the image of the bucket crossbeam to obtain a feature map of the bucket crossbeam;

[0019] A cross-modal element fusion feature analysis module is used to perform cross-modal element fusion feature analysis on the feature map of the bucket crossbeam and the time-series input vector of the bending deformation to obtain the bucket crossbeam feature with fused bending deformation time-series features; and

[0020] The safety analysis module is used to determine whether a safety warning for the bucket crossbeam should be generated based on the characteristics of the bucket crossbeam with the fused bending deformation time sequence characteristics.

[0021] Compared with existing technologies, the real-time monitoring method and system for the attitude of the bucket wheel crossbeam of the gantry bucket excavator provided in this application first acquires an image of the bucket wheel crossbeam captured by a camera. Next, it acquires the bending deformation of the bucket wheel crossbeam at multiple predetermined time points within a predetermined time period, collected by a displacement sensor. Then, it arranges the bending deformation at these multiple predetermined time points into a bending deformation time-series input vector according to the time dimension. Next, it performs image feature analysis on the bucket wheel crossbeam image to obtain a bucket wheel crossbeam feature map. Then, it performs cross-modal fusion feature analysis on the bucket wheel crossbeam feature map and the bending deformation time-series input vector to obtain a bucket wheel crossbeam feature with fused bending deformation time-series features. Finally, based on the bucket wheel crossbeam feature with fused bending deformation time-series features, it determines whether a bucket wheel crossbeam safety warning should be generated. This improves monitoring efficiency and accuracy. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The following drawings are not deliberately drawn to scale according to the actual size, but are intended to show the main idea of ​​this application.

[0023] Figure 1 This is a flowchart of a method for real-time monitoring of the attitude of the crossbeam of a gantry bucket wheel excavator according to an embodiment of this application.

[0024] Figure 2 This is a schematic diagram of the architecture of the real-time monitoring method for the attitude of the crossbeam of the gantry bucket wheel excavator according to an embodiment of this application.

[0025] Figure 3 This is a flowchart of sub-step S150 of the method for real-time monitoring of the attitude of the bucket crossbeam of a gantry bucket excavator according to an embodiment of this application.

[0026] Figure 4 This is a flowchart of sub-step S160 of the method for real-time monitoring of the attitude of the bucket crossbeam of a gantry bucket excavator according to an embodiment of this application.

[0027] Figure 5 This is a flowchart of sub-step S161 of the method for real-time monitoring of the attitude of the bucket crossbeam of a gantry bucket excavator according to an embodiment of this application.

[0028] Figure 6 This is a flowchart of sub-step S162 of the method for real-time monitoring of the attitude of the bucket crossbeam of a gantry bucket excavator according to an embodiment of this application.

[0029] Figure 7 This is a block diagram of a real-time monitoring system for the attitude of the bucket crossbeam of a portal bucket excavator according to an embodiment of this application.

[0030] Figure 8This is an application scenario diagram of the real-time attitude monitoring method for the crossbeam of the gantry bucket wheel excavator according to an embodiment of this application. Detailed Implementation

[0031] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are also within the scope of protection of this application.

[0032] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0033] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0034] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0035] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0036] To address the aforementioned technical problems, the technical concept of this application is to acquire images of the bucket wheel crossbeam using a camera, and to monitor and acquire the bending deformation of the bucket wheel crossbeam in real time using a displacement sensor. Then, a data processing and analysis algorithm is introduced at the back end to perform a collaborative analysis of the temporal distribution of the bucket wheel crossbeam images and the bending deformation of the bucket wheel crossbeam. This allows for real-time monitoring and early warning of the bucket wheel crossbeam status of the gantry bucket wheel excavator. In this way, real-time monitoring of the attitude and deformation of the bucket wheel crossbeam of the gantry bucket wheel excavator can be achieved, improving monitoring efficiency and accuracy. This helps to promptly detect abnormalities in the bucket wheel crossbeam and take corresponding measures to ensure the safe operation of the bucket wheel excavator.

[0037] Figure 1 This is a flowchart of a method for real-time monitoring of the attitude of the crossbeam of a gantry bucket wheel excavator according to an embodiment of this application. Figure 2 This is a schematic diagram of the architecture of a real-time monitoring method for the attitude of the bucket crossbeam of a gantry bucket excavator according to an embodiment of this application. Figure 1 and Figure 2 As shown, the real-time monitoring method for the attitude of the bucket wheel crossbeam of a gantry bucket excavator according to an embodiment of this application includes the following steps: S110, acquiring an image of the bucket wheel crossbeam captured by a camera; S120, acquiring the bending deformation of the bucket wheel crossbeam at multiple predetermined time points within a predetermined time period, collected by a displacement sensor; S130, arranging the bending deformation at the multiple predetermined time points into a bending deformation time-series input vector according to the time dimension; S140, performing image feature analysis on the bucket wheel crossbeam image to obtain a bucket wheel crossbeam feature map; S150, performing cross-modal element fusion feature analysis on the bucket wheel crossbeam feature map and the bending deformation time-series input vector to obtain a bucket wheel crossbeam feature with fused bending deformation time-series features; and S160, determining whether a bucket wheel crossbeam safety warning should be generated based on the bucket wheel crossbeam feature with fused bending deformation time-series features.

[0038] Specifically, in the technical solution of this application, firstly, an image of the bucket beam is acquired by a camera, and the bending deformation of the bucket beam at multiple predetermined time points within a predetermined time period is acquired by a displacement sensor. Next, considering that the bending deformation of the bucket beam changes continuously over time, meaning that the bending deformation has a dynamic temporal variation pattern, the bending deformation at the multiple predetermined time points needs to be arranged into a bending deformation time-series input vector according to the time dimension. This integrates the temporal distribution information of the bending deformation, facilitating temporal analysis and feature characterization of the bending deformation of the bucket beam.

[0039] Then, a wheel bucket beam posture feature extractor based on a convolutional neural network model, which has excellent performance in extracting latent features of images, is used to perform feature mining on the wheel bucket beam image to extract the posture feature distribution information of the wheel bucket beam in the image, thereby obtaining the wheel bucket beam feature map.

[0040] Accordingly, in step S140, image feature analysis is performed on the bucket crossbeam image to obtain a bucket crossbeam feature map, including: passing the bucket crossbeam image through a bucket crossbeam posture feature extractor based on a convolutional neural network model to obtain the bucket crossbeam feature map.

[0041] It's worth noting that a Convolutional Neural Network (CNN) is a deep learning model specifically designed to process data with a grid-like structure, such as images and videos. CNNs extract image features layer by layer by combining convolution operations on the input data with non-linear activation functions. It has several important components: 1. Convolutional Layers: Convolutional layers perform convolution operations on the input data using a sliding window approach, extracting local features. Each convolutional layer contains multiple convolutional kernels (or filters), each learning different features. 2. Pooling Layers: Pooling layers reduce the spatial size of feature maps, decreasing the number of parameters while preserving important features. Common pooling operations include max pooling and average pooling. 3. Non-linear Activation Functions: Activation functions introduce non-linear transformations, increasing the network's expressive power. Common activation functions include ReLU, Sigmoid, and Tanh. 4. Fully Connected Layer: The fully connected layer flattens the features extracted by the preceding convolutional and pooling layers and performs tasks such as classification or regression through fully connected operations. Convolutional neural network models have wide applications in image processing. For images of bucket wheel crossbeams, feature maps of the bucket wheel crossbeams can be extracted using a bucket wheel crossbeam pose feature extractor based on a convolutional neural network model. These feature maps can contain information such as the shape, texture, and edges of the bucket wheel crossbeam, which helps in subsequent bucket wheel crossbeam recognition, classification, or other related tasks. By training a convolutional neural network model, the feature representation of the bucket wheel crossbeam can be automatically learned, improving the understanding and processing capabilities of bucket wheel crossbeam images.

[0042] It should be understood that the bucket crossbeam feature map is an attitude feature of the bucket crossbeam extracted through a convolutional neural network model. It can provide information about the shape, position, and angle of the bucket crossbeam, which helps to determine whether there are any abnormalities in the bucket crossbeam. The bending deformation time-series input vector is a set of time-series distributions of the bending deformation of the bucket crossbeam at multiple predetermined time points within a predetermined time period, collected by a displacement sensor. It can reflect the deformation of the bucket crossbeam, including the degree of bending and the rate of deformation, which helps to determine whether there are any deformation anomalies in the bucket crossbeam. Therefore, by fusing these two different types of feature information about the bucket crossbeam, the attitude and deformation of the bucket crossbeam can be comprehensively considered, thereby providing a more comprehensive description of the state characteristics of the bucket crossbeam and improving the accuracy and reliability of bucket crossbeam monitoring. Based on this, in the technical solution of this application, the bucket crossbeam feature map and the bending deformation time-series input vector are further processed through a cross-modal fusion module to obtain a bucket crossbeam feature map that fuses the bending deformation time-series features. It should be understood that the cross-modal element fusion module can effectively fuse two different types of feature information about the bucket wheel crossbeam, so that the attitude features and deformation features of the bucket wheel crossbeam can complement each other, thereby obtaining a feature representation with more information and discriminative power, which is conducive to more accurate safety detection and early warning of the bucket wheel crossbeam.

[0043] Accordingly, in step S150, the bucket crossbeam feature map and the bending deformation time-series input vector are subjected to cross-modal fusion feature analysis to obtain bucket crossbeam features with fused bending deformation time-series features. This includes: passing the bucket crossbeam feature map and the bending deformation time-series input vector through the cross-modal fusion module to obtain a bucket crossbeam feature map with fused bending deformation time-series features as the bucket crossbeam features with fused bending deformation time-series features. It should be understood that the cross-modal fusion module is used to fuse features from different modalities (e.g., images and time-series input vectors) to obtain a fused feature representation. In this case, the cross-modal fusion module is used to fuse the bucket crossbeam feature map and the bending deformation time-series input vector to generate bucket crossbeam features with fused bending deformation time-series features. The role of the cross-modal fusion module is to combine information from different modalities to obtain a more comprehensive and richer feature representation. By fusing features from different modalities, their complementarity can be leveraged to provide more information and context, thereby improving the performance of subsequent tasks. In the application of bucket wheel crossbeams, the cross-modal meta-fusion module can fuse the bucket wheel crossbeam feature map and the temporal input vector of bending deformation to generate bucket wheel crossbeam features that incorporate the temporal features of bending deformation. Such fused features can simultaneously include image features such as the shape and texture of the bucket wheel crossbeam, as well as temporal information on bending deformation. By comprehensively considering these features from different modalities, the state and deformation of the bucket wheel crossbeam can be better understood, improving the accuracy and robustness of bucket wheel crossbeam analysis, recognition, or other related tasks.

[0044] More specifically, such as Figure 3 As shown, the wheel bucket crossbeam feature map and the bending deformation time-series input vector are passed through a cross-modal meta-fusion module to obtain a wheel bucket crossbeam feature map with fused bending deformation time-series features, which is used as the wheel bucket crossbeam feature map with fused bending deformation time-series features. This includes: S151, passing the bending deformation time-series input vector through a one-dimensional convolutional layer of the cross-modal meta-fusion module to obtain a bending deformation time-series feature vector; and S152, using the bending deformation time-series feature vector as a channel weighting vector to perform weighting processing along the channel dimension on the wheel bucket crossbeam feature map to obtain the wheel bucket crossbeam feature map with fused bending deformation time-series features.

[0045] It's worth noting that a 1D convolutional layer is a type of layer in convolutional neural networks used to process one-dimensional sequential data, such as time-series data or text data. Similar to 2D convolutional layers (used for image data), 1D convolutional layers perform convolution operations on the input data using a sliding window to extract local features. The sliding window of a 1D convolutional layer moves only in one dimension, typically the time dimension. The input to a 1D convolutional layer is usually a tensor of shape (sequence length, feature dimension). The convolutional layer contains multiple convolutional kernels (or filters), each learning different features. In a 1D convolution operation, the kernels perform convolution operations on the input along the sequence length, introducing a non-linear transformation through a non-linear activation function. The output of a 1D convolutional layer is typically a feature map of shape (output length, feature dimension). The output length depends on the hyperparameter settings of the convolutional layer and can be controlled by adjusting the kernel size, stride, and padding. One-dimensional convolutional layers offer the following advantages when processing sequential data: 1. Local feature extraction: One-dimensional convolutional layers can extract local features of sequential data through a sliding window approach, capturing patterns and structures at different locations. 2. Parameter sharing: The convolutional kernels of one-dimensional convolutional layers share parameters at different locations in the sequence, reducing the number of model parameters and improving model efficiency. 3. Translation invariance: By performing convolution operations on the sequence, one-dimensional convolutional layers can achieve translation invariance to a certain extent, meaning that similar feature representations can be obtained for the same pattern at different locations in the sequence. By using one-dimensional convolutional layers in the cross-modal meta-fusion module, convolution operations can be performed on the temporal input vector of bending deformation to extract the temporal features of the sequence. These features can contain information such as the dynamic changes, trends, and periodicity of the sequence, which is helpful for subsequent analysis and processing tasks.

[0046] Subsequently, the feature map of the bucket wheel crossbeam, which incorporates the temporal features of bending deformation, is processed by a classifier to obtain a classification result. This classification result indicates whether a safety warning for the bucket wheel crossbeam is generated. In other words, the cross-modal fusion correlation feature information between the attitude features and bending deformation features of the bucket wheel crossbeam is used for classification processing, thereby enabling anomaly detection and safety warnings for the bucket wheel crossbeam's state. This method allows for real-time monitoring of the attitude and deformation of the gantry bucket wheel excavator's bucket wheel crossbeam, improving monitoring efficiency and accuracy, and facilitating the timely detection of anomalies in the bucket wheel crossbeam and the implementation of corresponding measures, thus ensuring the safe operation of the bucket wheel excavator.

[0047] Accordingly, such as Figure 4As shown, in step S160, based on the wheel bucket crossbeam features fused with bending deformation time-series features, determining whether a wheel bucket crossbeam safety warning is generated includes: S161, optimizing the feature distribution of the wheel bucket crossbeam feature map fused with bending deformation time-series features to obtain an optimized wheel bucket crossbeam feature map fused with bending deformation time-series features; and S162, passing the optimized wheel bucket crossbeam feature map fused with bending deformation time-series features through a classifier to obtain a classification result, the classification result being used to indicate whether a wheel bucket crossbeam safety warning is generated. It should be understood that in step S161, by optimizing the feature distribution of the wheel bucket crossbeam feature map fused with bending deformation time-series features, the expressive and discriminative power of the features can be further improved. Feature distribution optimization can include various techniques, such as feature normalization, feature selection, and feature dimensionality reduction. By optimizing the feature distribution, the features can be made more discriminative and distinguishable, which is helpful for subsequent classification tasks. In step S162, by inputting the optimized feature map of the bucket crossbeam, which incorporates the time-series features of bending deformation, into a classifier, a safety warning for the bucket crossbeam can be determined. The classifier can be a trained machine learning model, such as a Support Vector Machine (SVM), Random Forest, or a deep learning model (such as a Convolutional Neural Network). The classifier will classify the input feature map and output a classification result indicating whether a safety warning for the bucket crossbeam has been generated. Through the combination of these two steps, feature optimization and classification can be performed based on the bucket crossbeam features incorporating the time-series features of bending deformation to determine whether a safety warning for the bucket crossbeam has been generated. Such a safety warning system can help monitor and detect the state of the bucket crossbeam, promptly identify abnormalities, and take corresponding measures to ensure the safe operation of the bucket crossbeam.

[0048] Among them, such as Figure 5 As shown, in step S161, the feature distribution of the wheel bucket crossbeam feature map with fused bending deformation time-series features is optimized to obtain an optimized wheel bucket crossbeam feature map with fused bending deformation time-series features, including: S1611, calculating the global mean of each feature matrix of the wheel bucket crossbeam feature map along the channel dimension to obtain the wheel bucket crossbeam feature vector; S1612, optimizing the bending deformation time-series feature vector with the wheel bucket crossbeam feature vector to obtain an optimized bending deformation time-series feature vector; and S1613, performing weighted processing along the channel dimension on the wheel bucket crossbeam feature map with the optimized bending deformation time-series feature vector to obtain the optimized wheel bucket crossbeam feature map with fused bending deformation time-series features.

[0049] Specifically, in the technical solution of this application, when obtaining the wheel bucket crossbeam feature map with fused bending deformation time-series features by passing the wheel bucket crossbeam feature map and the bending deformation time-series input vector through the cross-modal element fusion module, the bending deformation time-series feature vector obtained by passing the bending deformation time-series input vector through a one-dimensional convolutional layer is used as the channel weighting vector to weight the wheel bucket crossbeam feature map along the channel, thereby obtaining the wheel bucket crossbeam feature map with fused bending deformation time-series features.

[0050] Based on this, the applicant of this application considers that each feature matrix of the bucket crossbeam feature map expresses the local image neighborhood features of the bucket crossbeam image, and the feature matrices follow the channel distribution of the convolutional neural network model. The bending deformation time-series feature vector expresses the local temporal correlation features of bending deformation between samples. Therefore, the feature distribution intensity of the bucket crossbeam feature map in its channel dimension is stronger than the feature distribution intensity of the bending deformation time-series feature vector. That is, the feature distribution intensity of the bucket crossbeam feature map and the bending deformation time-series feature vector is unbalanced relative to the target distribution of the classification result, thereby affecting the accuracy of the classification result obtained by the classifier from the bucket crossbeam feature map with fused bending deformation time-series features.

[0051] Therefore, preferably, the global mean of each feature matrix of the feature map of the bucket crossbeam is first calculated to obtain the bucket crossbeam feature vector, for example, denoted as V1. Then, the bucket crossbeam feature vector V1 is used to optimize the bending deformation time-series feature vector, for example, denoted as V2, to obtain the optimized bending deformation time-series feature vector, for example, denoted as V2′.

[0052] Accordingly, in one example, optimizing the time-series feature vector of bending deformation using the feature vector of the bucket wheel crossbeam to obtain an optimized time-series feature vector of bending deformation includes: optimizing the time-series feature vector of bending deformation using the feature vector of the bucket wheel crossbeam to obtain the optimized time-series feature vector of bending deformation using the following optimization formula; wherein, the optimization formula is:

[0053]

[0054] Wherein, V1 represents the characteristic vector of the bucket crossbeam, and V2 represents the time-series characteristic vector of the bending deformation. and Let I represent the reciprocals of the global mean of the characteristic vector V1 of the bucket crossbeam and the time-series characteristic vector V2 of the bending deformation, respectively, where I is a unit vector and ⊙ represents the dot product by position. This represents vector addition. V2′ represents vector subtraction, and V2′ represents the time-series feature vector of the optimized bending deformation.

[0055] In other words, considering the optimization based on the difference in feature distribution intensity, if the feature vector V1 of the bucket beam is regarded as the feature distribution enhancement input of the time-series feature vector V2 of bending deformation, then considering the loss of target distribution information of the target feature of the time-series feature vector V2 of bending deformation in the class space, it may lead to the loss of the class regression objective. Therefore, by applying cross-penalty to the feature distribution relative to each other's outlier distribution, a self-supervised balance between feature enhancement and regression robustness can be achieved during feature interpolation fusion. The time-series feature vector V2 of bending deformation is optimized based on the feature vector V1 of the bucket beam. Then, the bucket beam feature map is weighted along the channel with the optimized time-series feature vector V2′ of bending deformation to obtain the bucket beam feature map with fused bending deformation time-series features. This can improve the accuracy of the classification result obtained by the classifier from the bucket beam feature map with fused bending deformation time-series features. In this way, the state of the bucket wheel beam can be monitored and warned in real time based on the attitude and deformation of the bucket wheel beam of the gantry bucket excavator. This improves the efficiency and accuracy of monitoring and helps to detect abnormalities in the bucket wheel beam in a timely manner and take corresponding measures, thereby ensuring the safe operation of the bucket wheel excavator.

[0056] Furthermore, such as Figure 6 As shown, in step S162, the bucket crossbeam feature map with optimized fusion of bending deformation time-series features is processed by a classifier to obtain a classification result. The classification result is used to indicate whether a bucket crossbeam safety warning is generated. This includes: S1621, expanding the bucket crossbeam feature map with optimized fusion of bending deformation time-series features into an optimized classification feature vector according to row vectors or column vectors; S1622, using the fully connected layer of the classifier to perform fully connected encoding on the optimized classification feature vector to obtain an encoded classification feature vector; and S1623, inputting the encoded classification feature vector into the Softmax classification function of the classifier to obtain the classification result.

[0057] In other words, in the technical solution of this application, the classifier's labels include generating a bucket wheel crossbeam safety warning (first label) and not generating a bucket wheel crossbeam safety warning (second label). The classifier uses a soft-maximum function to determine which label the optimized, fused bending deformation time-series feature map of the bucket wheel crossbeam belongs to. It is worth noting that the first label p1 and the second label p2 here do not contain artificially defined concepts. In fact, during the training process, the computer model does not have the concept of "whether a bucket wheel crossbeam safety warning is generated." It simply has two classification labels and outputs the probability of the feature under these two labels, i.e., the sum of p1 and p2 is one. Therefore, the classification result of whether a bucket wheel crossbeam safety warning is generated is actually transformed into a binary probability distribution conforming to natural laws through the classification labels. Essentially, it uses the physical meaning of the natural probability distribution of the labels, rather than the linguistic textual meaning of "whether a bucket wheel crossbeam safety warning is generated."

[0058] As you can understand, the role of a classifier is to learn classification rules and classifiers using given categories and known training data, and then classify (or predict) unknown data. Logistic regression and SVM are commonly used to solve binary classification problems. For multi-class classification problems, logistic regression or SVM can also be used, but multiple binary classifications are needed to form the multi-class classification. However, this is prone to errors and inefficient. A commonly used multi-class classification method is the Softmax classification function.

[0059] In summary, the real-time monitoring method for the attitude of the bucket wheel beam of the gantry bucket wheel excavator based on the embodiments of this application has been clarified. It can realize the real-time monitoring of the attitude and deformation of the bucket wheel beam of the gantry bucket wheel excavator, improve the monitoring efficiency and accuracy, which helps to detect abnormalities of the bucket wheel beam in a timely manner and take corresponding measures to ensure the safe operation of the bucket wheel excavator.

[0060] Figure 7 This is a block diagram of a real-time monitoring system 100 for the attitude of the bucket crossbeam of a gantry bucket wheel excavator according to an embodiment of this application. Figure 7As shown, the real-time monitoring system 100 for the attitude of the bucket wheel beam of a gantry bucket excavator according to an embodiment of this application includes: an image acquisition module 110, used to acquire images of the bucket wheel beam captured by a camera; a bending deformation acquisition module 120, used to acquire bending deformation amounts of the bucket wheel beam at multiple predetermined time points within a predetermined time period, collected by a displacement sensor; a vectorization module 130, used to arrange the bending deformation amounts at the multiple predetermined time points into a bending deformation time-series input vector according to the time dimension; an image feature analysis module 140, used to perform image feature analysis on the bucket wheel beam image to obtain a bucket wheel beam feature map; a cross-modal element fusion feature analysis module 150, used to perform cross-modal element fusion feature analysis on the bucket wheel beam feature map and the bending deformation time-series input vector to obtain a bucket wheel beam feature with fused bending deformation time-series features; and a safety analysis module 160, used to determine whether a bucket wheel beam safety warning is generated based on the bucket wheel beam feature with fused bending deformation time-series features.

[0061] In one example, in the above-mentioned real-time monitoring system 100 for the attitude of the bucket wheel beam of a gantry bucket excavator, the image feature analysis module 140 is used to: pass the image of the bucket wheel beam through a bucket wheel beam attitude feature extractor based on a convolutional neural network model to obtain the feature map of the bucket wheel beam.

[0062] In one example, in the above-mentioned real-time monitoring system 100 for the attitude of the bucket wheel beam of a gantry bucket excavator, the cross-modal element fusion feature analysis module 150 is used to: pass the bucket wheel beam feature map and the bending deformation time sequence input vector through the cross-modal element fusion module to obtain a bucket wheel beam feature map with fused bending deformation time sequence features as the bucket wheel beam feature with fused bending deformation time sequence features.

[0063] Here, those skilled in the art will understand that the specific functions and operations of each module in the above-mentioned real-time monitoring system 100 for the attitude of the bucket crossbeam of the gantry bucket excavator have been referenced above. Figures 1 to 6 The method for real-time monitoring of the attitude of the bucket crossbeam of a gantry bucket wheel excavator has been described in detail, and therefore, its repeated description will be omitted.

[0064] As described above, the real-time monitoring system 100 for the attitude of the gantry bucket wheel beam according to embodiments of this application can be implemented in various wireless terminals, such as servers with real-time monitoring algorithms for the attitude of the gantry bucket wheel beam. In one example, the real-time monitoring system 100 for the attitude of the gantry bucket wheel beam according to embodiments of this application can be integrated into a wireless terminal as a software module and / or a hardware module. For example, the real-time monitoring system 100 for the attitude of the gantry bucket wheel beam can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the real-time monitoring system 100 for the attitude of the gantry bucket wheel beam can also be one of many hardware modules of the wireless terminal.

[0065] Alternatively, in another example, the real-time monitoring system 100 for the attitude of the gantry bucket wheel beam and the wireless terminal can also be separate devices, and the real-time monitoring system 100 for the attitude of the gantry bucket wheel beam can be connected to the wireless terminal via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0066] Figure 8 This is an application scenario diagram of the real-time monitoring method for the attitude of the crossbeam of a gantry bucket wheel excavator according to an embodiment of this application. For example... Figure 8 As shown, in this application scenario, firstly, an image of the bucket beam captured by a camera is obtained (e.g., ...). Figure 8 As shown in the figure, D1), and the bending deformation of the bucket beam at multiple predetermined time points within a predetermined time period, collected by a displacement sensor (e.g., Figure 8 As shown in D2), the image of the bucket wheel beam and the bending deformation at the multiple predetermined time points are then input to a server equipped with a real-time monitoring algorithm for the attitude of the gantry bucket wheel beam (e.g., D2). Figure 8 In the S shown, the server is able to use the real-time monitoring algorithm for the attitude of the gantry bucket wheel beam to process the image of the bucket wheel beam and the bending deformation at the multiple predetermined time points to obtain a classification result indicating whether a safety warning for the bucket wheel beam has been generated.

[0067] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Accordingly, aspects of this application can be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. All of the above hardware or software may be referred to as a “data block,” “module,” “engine,” “unit,” “component,” or “system.” Furthermore, aspects of this application may manifest as a computer product located on one or more computer-readable media, the product including computer-readable program code.

[0068] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in a common dictionary shall be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0069] The foregoing description is a illustrative description of the present application and should not be construed as limiting it. Although several exemplary embodiments of the present application have been described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without departing from the novel teachings and advantages of the present application. Therefore, all such modifications are intended to be included within the scope of the present application as defined by the claims. It should be understood that the foregoing description is a illustrative description of the present application and should not be construed as limiting it to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present application is defined by the claims and their equivalents.

Claims

1. A method for real-time monitoring of the attitude of the crossbeam of a gantry bucket excavator, characterized in that, include: Acquire images of the bucket crossbeam captured by a camera; The bending deformation of the bucket crossbeam at multiple predetermined time points within a predetermined time period is obtained by a displacement sensor. The bending deformation at the multiple predetermined time points is arranged according to the time dimension to form a bending deformation time sequence input vector; Image feature analysis is performed on the image of the bucket wheel crossbeam to obtain a feature map of the bucket wheel crossbeam; The wheel bucket crossbeam feature map and the bending deformation time-series input vector are passed through the cross-modal element fusion module to obtain the wheel bucket crossbeam feature map with fused bending deformation time-series features as the wheel bucket crossbeam feature with fused bending deformation time-series features; Specifically, this includes: passing the bending deformation time-series input vector through the one-dimensional convolutional layer of the cross-modal meta-fusion module to obtain a bending deformation time-series feature vector; and using the bending deformation time-series feature vector as a channel weighting vector to perform weighted processing along the channel dimension on the wheel bucket crossbeam feature map to obtain the wheel bucket crossbeam feature map with fused bending deformation time-series features; And based on the wheel bucket crossbeam features of the fused bending deformation time-series features, determining whether a wheel bucket crossbeam safety warning is generated specifically includes: optimizing the feature distribution of the wheel bucket crossbeam feature map of the fused bending deformation time-series features to obtain an optimized wheel bucket crossbeam feature map of the fused bending deformation time-series features; and passing the optimized wheel bucket crossbeam feature map of the fused bending deformation time-series features through a classifier to obtain a classification result, the classification result being used to indicate whether a wheel bucket crossbeam safety warning is generated; wherein, optimizing the feature distribution of the wheel bucket crossbeam feature map of the fused bending deformation time-series features to obtain an optimized wheel bucket crossbeam feature map of the fused bending deformation time-series features includes: calculating the global mean of each feature matrix of the wheel bucket crossbeam feature map along the channel dimension to obtain a wheel bucket crossbeam feature vector; optimizing the bending deformation time-series feature vector with the wheel bucket crossbeam feature vector to obtain an optimized bending deformation time-series feature vector; and weighting the wheel bucket crossbeam feature map along the channel dimension with the optimized bending deformation time-series feature vector to obtain an optimized wheel bucket crossbeam feature map of the fused bending deformation time-series features.

2. The method for real-time monitoring of the attitude of the bucket crossbeam of a gantry bucket excavator according to claim 1, characterized in that, Image feature analysis is performed on the image of the bucket wheel crossbeam to obtain a feature map of the bucket wheel crossbeam, including: The image of the bucket crossbeam is processed by a bucket crossbeam pose feature extractor based on a convolutional neural network model to obtain the bucket crossbeam feature map.

3. The method for real-time monitoring of the attitude of the crossbeam of a gantry bucket wheel excavator according to claim 2, characterized in that, The optimized and fused bending deformation time-series feature map of the bucket crossbeam is passed through a classifier to obtain a classification result. The classification result is used to indicate whether a safety warning for the bucket crossbeam is generated, including: The optimized and fused bending deformation time-series feature map of the bucket crossbeam is expanded into an optimized classification feature vector according to row vectors or column vectors; The optimized classification feature vector is fully encoded using the fully connected layer of the classifier to obtain the encoded classification feature vector; and The encoded classification feature vector is input into the Softmax classification function of the classifier to obtain the classification result.

4. A real-time monitoring system for the attitude of the bucket beam of a gantry bucket excavator, used for implementing the real-time monitoring method for the attitude of the bucket beam of a gantry bucket excavator according to any one of claims 1-3, characterized in that, include: The image acquisition module is used to acquire images of the bucket beam captured by the camera. The bending deformation acquisition module is used to acquire the bending deformation of the bucket beam at multiple predetermined time points within a predetermined time period, which is collected by the displacement sensor. The vectorization module is used to arrange the bending deformation at the multiple predetermined time points into a bending deformation time sequence input vector according to the time dimension. The image feature analysis module is used to perform image feature analysis on the image of the bucket crossbeam to obtain a feature map of the bucket crossbeam; The cross-modal element fusion feature analysis module is used to perform cross-modal element fusion feature analysis on the feature map of the bucket crossbeam and the time-series input vector of the bending deformation to obtain the bucket crossbeam feature with fused bending deformation time-series features; as well as The safety analysis module is used to determine whether a safety warning for the bucket crossbeam should be generated based on the characteristics of the bucket crossbeam with the fused bending deformation time sequence characteristics.

5. The real-time monitoring system for the attitude of the bucket crossbeam of a gantry bucket excavator according to claim 4, characterized in that, The image feature analysis module is used for: The image of the bucket crossbeam is processed by a bucket crossbeam pose feature extractor based on a convolutional neural network model to obtain the bucket crossbeam feature map.

6. The real-time monitoring system for the attitude of the bucket crossbeam of a gantry bucket excavator according to claim 5, characterized in that, The cross-modal meta-fusion feature analysis module is used for: The bucket crossbeam feature map and the bending deformation time-series input vector are passed through a cross-modal element fusion module to obtain a bucket crossbeam feature map with fused bending deformation time-series features, which is used as the bucket crossbeam feature with fused bending deformation time-series features.

Images