System and method for identifying safety state of pin shaft of tower crane

Abstract

The invention relates to the field of pin shaft safety state recognition, and particularly discloses a tower crane pin shaft safety state recognition system and method. Firstly, a tower crane pin shaft monitoring image collected by a camera, a pin shaft position change moving track collected by a position sensor and a tower crane pin shaft vibration signal collected by a vibration sensor are obtained, then feature extraction and correlation analysis are carried out on the three images by using a deep learning technology, and finally, the tower crane pin shaft vibration signal is obtained through a classifier. Therefore, potential abnormity or fault signs can be detected in time, and the safety and the reliability of equipment can be improved.

Description

Technical Field

[0001] The present application relates to the field of pin shaft safety status identification, and more specifically, to a tower crane pin shaft safety status identification system and method. Background Art

[0002] The pin of a tower crane is a key component that connects the main boom and the counter-arm, carrying important structural loads and power transmission functions. It is usually made of high-strength alloy steel with excellent wear resistance and corrosion resistance to ensure the stable operation of the crane under high-intensity working conditions. The design and manufacture of the pin must strictly comply with mechanical engineering standards to ensure its reliability and safety under long-term, high-load working conditions.

[0003] Nowadays, due to the increase in lifting performance and structural size of tower cranes, the connecting pins between structural parts are also larger. Due to the excessive weight of the pins and the high-altitude operations, the pins are difficult to fix and are prone to slipping, causing safety problems.

[0004] Therefore, a tower crane pin shaft safety status identification system and method are desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a tower crane pin shaft safety status recognition system and method, which first obtains the tower crane pin shaft monitoring image collected by the camera, the pin shaft position change operation trajectory collected by the position sensor, and the tower crane pin shaft vibration signal collected by the vibration sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether to issue a tower crane pin shaft safety warning, so as to detect potential abnormalities or fault signs in time, thereby improving the safety and reliability of the equipment.

[0006] According to one aspect of the present application, a tower crane pin shaft safety state identification system is provided, which includes:

[0007] The tower crane pin shaft data acquisition module is used to acquire the tower crane pin shaft monitoring image collected by the camera, the pin shaft position change operation trajectory collected by the position sensor, and the tower crane pin shaft vibration signal collected by the vibration sensor;

[0008] A tower crane pin data extraction module is used to extract a multi-modal correlation feature vector of a tower crane pin state and a tower crane pin vibration waveform feature vector from the tower crane pin monitoring image collected by the camera, the pin position change running trajectory collected by the position sensor, and the tower crane pin vibration signal collected by the vibration sensor;

[0009] The pin shaft safety warning judgment module is used to judge whether to issue a tower crane pin shaft safety warning based on the multi-modal association feature vector of the tower crane pin shaft state and the tower crane pin shaft vibration waveform feature vector.

[0010] According to another aspect of the present application, a method for identifying a safe state of a tower crane pin is provided, comprising:

[0011] Acquire the monitoring image of the tower crane pin shaft collected by the camera, the position change running track of the pin shaft collected by the position sensor, and the vibration signal of the tower crane pin shaft collected by the vibration sensor;

[0012] Extracting a multimodal correlation feature vector of a tower crane axle pin state and a tower crane axle pin vibration waveform feature vector from the tower crane axle pin monitoring image acquired by the camera, the position change running track of the axle pin acquired by the position sensor, and the tower crane axle pin vibration signal acquired by the vibration sensor;

[0013] Based on the multi-modal association feature vector of the tower crane axle pin state and the vibration waveform feature vector of the tower crane pin shaft, it is determined whether to issue a tower crane pin shaft safety warning.

[0014] Compared with the prior art, the present application provides a tower crane pin shaft safety status identification system and method, which first obtains the tower crane pin shaft monitoring image collected by a camera, the pin shaft position change trajectory collected by a position sensor, and the tower crane pin shaft vibration signal collected by a vibration sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the three, and finally uses a classifier to determine whether to issue a tower crane pin shaft safety warning, thereby timely detecting potential abnormalities or fault signs, thereby improving the safety and reliability of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a block diagram of a tower crane pin shaft safety status identification system according to an embodiment of the present application.

[0017] Figure 2 It is a block diagram of a tower crane pin data extraction module in a tower crane pin safety status identification system according to an embodiment of the present application.

[0018] Figure 3 It is a block diagram of a multimodal feature extraction unit for axle pin state in a tower crane axle pin safety state identification system according to an embodiment of the present application.

[0019] Figure 4 It is a block diagram of a pin shaft safety warning judgment module in a pin shaft safety status identification system for a tower crane according to an embodiment of the present application.

[0020] Figure 5 The present invention is a flowchart of a tower crane pin shaft safety status identification method according to an embodiment of the present application.

[0021] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] Figure 1 : is a block diagram of a tower crane pin shaft safety status identification system according to an embodiment of the present application. Figure 1 As shown, according to the tower crane pin safety state identification system 100 of the embodiment of the present application, it includes: a tower crane pin data acquisition module 110, which is used to obtain the tower crane pin monitoring image collected by the camera, the pin position change running trajectory collected by the position sensor and the tower crane pin vibration signal collected by the vibration sensor; a tower crane pin data extraction module 120, which is used to extract the tower crane axle pin state multimodal correlation feature vector and the tower crane pin vibration waveform feature vector from the tower crane pin monitoring image collected by the camera, the pin position change running trajectory collected by the position sensor and the tower crane pin vibration signal collected by the vibration sensor; a pin safety warning judgment module 130, which is used to judge whether to issue a tower crane pin safety warning based on the tower crane axle pin state multimodal correlation feature vector and the tower crane pin vibration waveform feature vector.

[0024] In the above-mentioned tower crane pin shaft safety state identification system 100, the tower crane pin shaft data acquisition module 110 is used to obtain the tower crane pin shaft monitoring image collected by the camera, the position change operation trajectory of the pin shaft collected by the position sensor, and the tower crane pin shaft vibration signal collected by the vibration sensor. It should be understood that the pin shaft of the tower crane is an important component connecting the main arm and the balance arm, carrying the key structural load and power transmission tasks. It is usually made of high-strength alloy steel with excellent wear resistance and corrosion resistance to ensure stability under high-load conditions. The pin shaft must meet strict mechanical engineering standards to ensure its reliability under long-term and high-load operating conditions. However, with the increase in the lifting capacity of the tower crane and the increase in the structural size, the pin shaft also increases accordingly. This leads to an increase in the weight of the pin shaft, and the problem of difficulty in fixing during high-altitude operations makes the pin shaft easy to slide, thereby causing safety hazards. Therefore, in the technical solution of the present application, by acquiring the monitoring image of the tower crane pin collected by the camera, the position change trajectory of the pin collected by the position sensor, and the vibration signal of the tower crane pin collected by the vibration sensor, and combining with deep learning technology, it is evaluated whether it is necessary to issue a safety warning for the tower crane pin, so as to timely identify potential abnormalities or faults, thereby enhancing the safety and reliability of the equipment.

[0025] Specifically, obtaining data collected by cameras, position sensors, and vibration sensors is crucial for identifying the safe status of tower crane pins. These three data sources each provide unique and complementary information, which helps to comprehensively monitor and evaluate the safety status of pins. Among them, the monitoring images collected by the camera can provide visual information of the pin, which is very important for detecting obvious physical damage, wear or abnormal conditions. The image data is processed by computer vision technology to detect problems such as cracks, corrosion or deformation on the surface of the pin, which may affect the stability and overall safety of the pin. The pin position change operation trajectory collected by the position sensor provides accurate displacement data of the pin during operation. These data are used to monitor the motion trajectory and position changes of the pin to confirm whether its performance in operation meets the design standards or operating requirements. The vibration signal collected by the vibration sensor is used to detect the vibration characteristics of the pin during operation. Vibration signal analysis can reveal the operating status of the pin and potential mechanical problems, such as imbalance, resonance, or loose components. Abnormal vibration usually indicates that there is a fault in the equipment. Vibration analysis can detect these problems by extracting the waveform characteristics of the vibration signal and provide additional safety warnings. By integrating this information, the accurate judgment of the safety status of the pin shaft can be improved, the risk of accidents can be reduced, and the stability and safety of the tower crane can be ensured.

[0026] In the above-mentioned tower crane pin shaft safety state identification system 100, the tower crane pin shaft data extraction module 120 is used to extract the multimodal correlation feature vector of the tower crane shaft pin state and the tower crane pin shaft vibration waveform feature vector from the tower crane pin shaft monitoring image collected by the camera, the position change running trajectory of the pin shaft collected by the position sensor, and the tower crane pin shaft vibration signal collected by the vibration sensor. It should be understood that extracting feature vectors from different sensor data can integrate static visual information, position change information and dynamic vibration information, thereby forming a more comprehensive and accurate pin shaft state evaluation system. This multimodal data fusion method improves the monitoring capability of the pin shaft state, can more effectively prevent failures, and ensure the safe and reliable operation of the equipment.

[0027] Figure 2 FIG. 1 is a block diagram of a tower crane pin data extraction module in a tower crane pin safety state identification system according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the tower crane pin data extraction module 120 includes: a pin state multimodal feature extraction unit 121, which is used to extract features from the tower crane pin monitoring image collected by the camera and the pin position change operation trajectory collected by the position sensor to obtain the multimodal associated feature vector of the tower crane pin state; a pin vibration signal feature extraction unit 122, which is used to extract features from the tower crane pin vibration signal collected by the vibration sensor to obtain the tower crane pin vibration waveform feature vector.

[0028] It should be understood that the monitoring images collected by the camera provide visual information of the pin. Through image analysis, physical defects on the pin surface, such as cracks, wear or corrosion, can be identified. These visual features are an important basis for judging the state of the pin. However, relying solely on visual information may not fully understand the performance of the pin in actual work, because the image can only show the static physical state, but cannot reflect the dynamic behavior of the pin. The pin position change trajectory collected by the position sensor provides dynamic position data of the pin during work. These data can reveal the actual motion path of the pin and any deviation or anomaly in the position. For example, the position sensor can detect whether the pin moves within the predetermined trajectory range, whether there is deviation or abnormal vibration. Position change information can help identify position offsets caused by improper installation, wear or other mechanical failures. Feature extraction of these data and combining them to form a multimodal correlation feature vector is to comprehensively consider visual and dynamic information, thereby providing a more comprehensive pin state assessment. By combining image features (such as surface damage) with position trajectory features (such as position deviation), the health status of the pin can be more accurately judged.

[0029] Furthermore, feature extraction of vibration signals of tower crane pins is performed to deeply analyze the dynamic behavior of the pins and identify potential mechanical problems. Vibration signals provide key information about the operating status of the pins, because vibration features can reflect many potential faults or anomalies. Among them, vibration sensors can monitor the vibration of the pins in real time during operation. These vibration signals contain the dynamic characteristics of the pins, such as vibration frequency, amplitude, and mode. These features are an effective tool for identifying mechanical faults, because different faults usually lead to specific vibration modes. For example, imbalance, wear or looseness of the pins can cause abnormal vibration frequency and amplitude. By extracting features from vibration signals, complex vibration data can be converted into feature vectors that are easy to analyze. These feature vectors can describe the main features of vibration signals, such as spectrum, time domain characteristics, and vibration amplitude. Among them, the feature extraction process includes Fourier transform, wavelet transform and other analyses of the vibration signals to extract frequency domain features and time domain features. These features can reveal key patterns and trends in vibration signals.

[0030] Figure 3 FIG. 1 is a block diagram of a multi-modal feature extraction unit for axle pin state in a tower crane pin safety state identification system according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the axle pin state multimodal feature extraction unit 121 includes: a pin monitoring feature encoding subunit 1211, which is used to feature encode the tower crane pin monitoring image collected by the camera to obtain a tower crane pin monitoring feature vector; a pin position change feature encoding subunit 1212, which is used to feature encode the position change operation trajectory of the pin collected by the position sensor to obtain a position change operation trajectory semantic feature vector; an axle pin state multimodal feature association subunit 1213, which is used to associate the tower crane pin monitoring feature vector with the position change operation trajectory semantic feature vector to obtain the tower crane axle pin state multimodal association feature vector.

[0031] It should be understood that raw image data usually contains a lot of redundant information. Using these images directly for analysis is not only computationally complex but also inefficient. Therefore, it is necessary to feature encode the images for more efficient data analysis and fault detection. The main purpose of feature encoding is to extract the key visual features in the image and convert them into feature vectors. These feature vectors represent the core information of the image, such as the shape, color, texture, and structure of the object. Through feature encoding, noise and irrelevant information in the image can be removed, and key features related to the pin state can be retained. Common feature encoding techniques include convolutional neural networks (CNNs), which can automatically extract and encode high-level features in images, such as edges, corners, and texture patterns.

[0032] Furthermore, feature encoding of the pin position change trajectory collected by the position sensor is to extract meaningful features from the complex trajectory data, so as to facilitate the analysis and understanding of the dynamic behavior of the pin. The data collected by the position sensor provides information on the position change of the pin during operation. This information includes dynamic parameters such as the motion trajectory, velocity, acceleration, and position offset of the pin. Although the raw data can reflect the motion state of the pin, its complexity and data volume are large, and direct processing of this data may lead to excessive consumption of computing resources and low analysis efficiency. The purpose of feature encoding is to convert complex motion trajectory data into concise and easy-to-process feature vectors. This process usually includes data preprocessing, dimensionality reduction, and feature extraction to extract key features that reflect trajectory changes.

[0033] Furthermore, the monitoring feature vector of the tower crane pin is associated with the semantic feature vector of the position change operation trajectory, mainly to achieve a more comprehensive and accurate assessment of the pin state. This process combines the advantages of visual and dynamic data to provide deeper insights into the health of the equipment. Among them, the monitoring feature vector comes from the images collected by the camera, which mainly reflects the static state of the pin, such as surface damage, cracks or wear. This information is crucial for detecting physical defects, but they mainly focus on the surface condition of the pin and cannot fully describe the dynamic performance of the pin in actual work. The position change operation trajectory feature vector is the data collected by the position sensor, which reflects the dynamic behavior of the pin, such as position offset, motion trajectory and speed change. These dynamic features reveal the actual performance of the pin during operation and can detect problems caused by irregular motion or position deviation. Associating these two feature vectors, that is, fusing static monitoring information with dynamic operation data, can comprehensively consider the entire state of the pin. Correlation analysis can reveal the connection between static damage and dynamic anomalies.

[0034] In a specific embodiment of the present application, the pin shaft monitoring feature encoding subunit 1211 includes: passing the tower crane pin shaft monitoring image collected by the camera through a tower crane pin shaft monitoring convolutional neural network as a denoiser to obtain a denoised tower crane pin shaft monitoring image; passing the denoised tower crane pin shaft monitoring image through a tower crane pin shaft monitoring convolutional neural network as a feature encoder to obtain the tower crane pin shaft monitoring feature vector.

[0035] It should be understood that denoising surveillance images is an important step in image processing, especially in industrial surveillance applications. Images captured by cameras are usually affected by various noises, including sensor noise, ambient light interference, image compression distortion, etc. These noises reduce the clarity of the image and make it difficult to identify details in the image, thus affecting subsequent analysis and fault detection. Convolutional neural networks are widely used in image processing tasks because they can effectively extract features from images and perform pattern recognition. Using convolutional neural networks as denoisers can remove noise from images while maintaining important information. This network gradually extracts high-level features of the image through multiple convolutional layers and pooling layers, and restores the clarity of the image through deconvolution layers or upsampling layers. Specifically, the tower crane pin shaft monitoring image captured by the camera is input into the encoder of the tower crane pin shaft monitoring convolutional neural network serving as a denoiser, wherein the encoder uses a convolutional layer to perform explicit spatial encoding on the tower crane pin shaft monitoring image captured by the camera to obtain image features; and the image features are input into the decoder of the tower crane pin shaft monitoring convolutional neural network serving as a denoiser, wherein the decoder uses a deconvolution layer to perform deconvolution processing on the image features to obtain the denoised tower crane pin shaft monitoring image.

[0036] Furthermore, the de-noised tower crane pin monitoring images are used as feature encoders through convolutional neural networks in order to extract meaningful feature information from clear images for more efficient analysis and decision-making. The de-noised images have removed the noise that affects the analysis and present clearer details and structures. These images contain the actual state information of the pins, such as surface defects, wear or cracks. This information is crucial for subsequent feature extraction, because these details can only be accurately identified and analyzed on clear images. The role of convolutional neural networks as feature encoders lies in their powerful feature extraction capabilities. By learning from a large amount of labeled training data, convolutional neural networks can identify important patterns in images and convert them into high-dimensional feature vectors. This feature vector not only retains the core information of the image, but also simplifies the data representation, making subsequent analysis and processing more efficient. Specifically, each layer of the tower crane pin monitoring convolutional neural network used as a feature encoder performs convolution processing, mean pooling processing based on the local feature matrix and nonlinear activation processing on the input data in the forward pass of the layer, so that the last layer of the tower crane pin monitoring convolutional neural network used as a feature encoder outputs the tower crane pin monitoring feature vector, wherein the input of the tower crane pin monitoring convolutional neural network used as a feature encoder is the denoised tower crane pin monitoring image.

[0037] In a specific embodiment of the present application, the pin shaft position change feature encoding subunit 1212 includes: passing the position change trajectory of the pin shaft collected by the position sensor through a position change trajectory word segmentation processor to obtain a position change trajectory word sequence; passing the position change trajectory word sequence through a position change trajectory text convolutional neural network as a semantic encoder to obtain the position change trajectory semantic feature vector.

[0038] It should be understood that the position change running trajectory data is collected by sensors in real time and contains information about the position change of the pin during operation. These data usually exist in the form of time series, including dynamic parameters such as position, speed, acceleration, etc. Although these data can record the running status of the pin in detail, their original form may be complex and difficult to directly analyze effectively. The role of the word segmentation processor is to decompose continuous dynamic data into meaningful fragments or "words". In natural language processing, the purpose of word segmentation is to decompose long text into smaller units for easy analysis. Similarly, the "word segmentation" processing of the position change running trajectory is to divide the continuous dynamic data into fragments with specific meanings. These fragments may represent specific motion patterns, speed change intervals, or position offsets. The position change running trajectory word sequence can convert complex continuous data into a more structured sequence. These word sequences not only simplify the representation of data, but also retain the key dynamic characteristics of the original data.

[0039] Furthermore, the position change running trajectory word sequence is processed as a semantic encoder through the position change running trajectory text convolutional neural network, mainly to extract more representative features from the sequence data and improve the understanding of motion patterns and behaviors. Among them, the position change running trajectory word sequence is obtained by word segmentation processing of sensor data, and these sequences provide a structured expression for dynamic data. These word sequences can effectively represent the different states and behavioral characteristics of the pin during operation. However, the original form of these word sequences still lacks a deep understanding and refinement of the potential semantics in the data. As a semantic encoder, the text convolutional neural network has been proven to be effective in extracting semantic features from text in the field of natural language processing (NLP). For the word sequence of the position change running trajectory, the convolutional neural network can automatically learn and extract important patterns and features in the data through multiple convolutional layers and pooling layers. These convolution operations can capture local dependencies and semantic patterns in the sequence, helping to identify complex dynamic behaviors and motion laws. Specifically, the embedding layer of the position change trajectory text convolutional neural network as a semantic encoder is used to map each trajectory word in the position change trajectory word sequence into a word embedding vector to obtain a sequence of trajectory word embedding vectors; the converter-based Bert model of the position change trajectory text convolutional neural network as a semantic encoder is used to perform global context semantic encoding on the sequence of trajectory word embedding vectors to obtain multiple trajectory feature vectors; and the multiple trajectory feature vectors are cascaded to obtain the position change trajectory semantic feature vector.

[0040] In a specific embodiment of the present application, the pin shaft vibration signal feature extraction unit 122 includes: extracting a waveform graph of the tower crane pin shaft vibration signal from the tower crane pin shaft vibration signal collected by the vibration sensor; obtaining a tower crane pin shaft vibration waveform feature graph by using a tower crane pin shaft vibration waveform feature encoder using a void convolutional neural network to obtain the tower crane pin shaft vibration waveform feature graph; pooling the tower crane pin shaft vibration waveform feature graph to obtain a tower crane pin shaft vibration waveform feature vector.

[0041] It should be understood that vibration signals usually exist in the form of time series, recording the vibration information generated by the tower crane pin during operation. These signals contain various dynamic behaviors of the equipment during operation, including normal operating vibration patterns and potential abnormal signals. However, directly analyzing the raw vibration data can be very complicated because they may contain a lot of noise and interference, and it is not easy to directly identify potential abnormal patterns in the time domain. The waveform diagram shows the change of the vibration signal over time, clearly reflecting the amplitude, frequency, periodicity and other dynamic characteristics of the vibration. This visualization makes it easier to identify abnormal behaviors, such as sudden vibration peaks, periodic vibration patterns or continuous vibration deviations, which may be early warning signs of equipment failure or performance degradation. The main advantage of extracting the waveform diagram is that it can provide detailed time domain features, which facilitates the identification and analysis of characteristic patterns in the vibration signal. Through the waveform diagram, the vibration performance of the equipment under different working conditions can be monitored, and the vibration characteristics under normal and abnormal conditions can be compared to make effective judgments.

[0042] Furthermore, the waveform of the extracted vibration signal of the tower crane pin is processed by a dilated convolutional neural network to obtain a vibration waveform feature map, in order to extract deep and highly representative features from the time series data to enhance the ability of fault detection and equipment monitoring. It should be understood that the vibration waveform map is generated from the vibration sensor data, showing the time domain characteristics of the vibration of the tower crane pin during operation. Although these waveforms can intuitively reflect the amplitude and frequency changes of the vibration, advanced feature extraction methods are still required to extract useful features from them and conduct effective analysis. Among them, the dilated convolutional neural network is a convolutional neural network that uses the dilated convolution technology. Unlike traditional convolution, the dilated convolution introduces holes in the convolution operation, that is, there is a gap between the elements of the convolution kernel, so that the receptive field can be expanded without increasing the amount of calculation. This method can capture a wider range of contextual information and help to better understand the long-range dependencies in complex time series data. Specifically, each layer of the first convolutional neural network model in the tower crane pin shaft vibration waveform feature encoder using the atrous convolutional neural network performs atrous convolution processing based on the first convolution kernel, pooling processing along the channel dimension, and nonlinear activation processing on the input data in the forward transfer of the layer, so as to output a first feature matrix from the last layer of the first convolutional neural network model; each layer of the second convolutional neural network model in the tower crane pin shaft vibration waveform feature encoder using the atrous convolutional neural network performs atrous convolution processing based on the second convolution kernel, pooling processing along the channel dimension, and nonlinear activation processing on the input data in the forward transfer of the layer, so as to output a second feature matrix from the last layer of the second convolutional neural network model; the first feature matrix and the second feature matrix are fused to obtain the tower crane pin shaft vibration waveform feature map.

[0043] Furthermore, the vibration waveform feature map of the tower crane pin is pooled mainly to transform the high-dimensional feature map into a more compact and information-rich feature representation, thereby improving the efficiency and accuracy of data analysis. The feature map is extracted from the vibration waveform map by using a dilated convolutional neural network, showing the deep-level characteristics of the vibration signal. These feature maps contain a lot of local and global information, but their dimensions may be very high. For subsequent analysis and processing, such as fault detection or state prediction, high-dimensional feature maps may lead to an increase in the amount of calculation, a decrease in processing speed, and easy introduction of noise and redundant information. Pooling operation is a common dimensionality reduction technique in convolutional neural networks, which aims to simplify data by reducing the spatial dimension of feature maps while retaining important feature information. Pooling is usually divided into methods such as max pooling and average pooling. Max pooling represents the characteristics of a local area by taking the maximum value in the local area, while average pooling takes the average value in the local area. This operation can effectively reduce the size of the feature map while retaining important feature information.

[0044] In the above-mentioned tower crane pin shaft safety state identification system 100, the pin shaft safety warning judgment module 130 is used to determine whether to issue a tower crane pin shaft safety warning based on the multimodal correlation feature vector of the tower crane pin shaft state and the vibration waveform feature vector of the tower crane pin shaft. It should be understood that the multimodal correlation feature vector of the pin shaft state includes data provided by different sensors or monitoring systems, such as temperature sensors, load sensors, wear sensors, etc., which together describe the overall health status of the pin shaft. These feature vectors can provide global information about the pin shaft state, such as whether there is excessive wear, abnormal temperature or excessive load. The vibration waveform feature vector provides dynamic information in the vibration signal. By analyzing the changes in the vibration mode, possible abnormal conditions such as abnormal vibration peaks, frequency offsets, etc. can be identified. This information can reveal potential mechanical problems of the pin shaft, such as wear, looseness or misalignment. Combining these two feature vectors can obtain a comprehensive assessment of the equipment status. The multimodal feature vector provides a global perspective on the equipment status, while the vibration feature vector focuses on dynamic changes and abnormal modes. By integrating this information, it is possible to more accurately determine whether the equipment has potential safety hazards.

[0045] Figure 4 FIG. 1 is a block diagram of a pin shaft safety warning judgment module in a tower crane pin shaft safety state identification system according to an embodiment of the present application. Figure 4As shown, in a specific embodiment of the present application, the pin shaft safety warning judgment module 130 includes: a tower crane pin shaft safety feature fusion unit 131, which is used to fuse the multi-modal correlation feature vector of the tower crane shaft pin state and the tower crane pin shaft vibration waveform feature vector to obtain a tower crane pin shaft safety warning judgment classification feature vector; a tower crane pin shaft safety feature optimization unit 132, which is used to maximize the entropy degradation of the prediction parameter space of the tower crane pin shaft safety warning judgment classification feature vector based on the classifier to obtain an optimized tower crane pin shaft safety warning judgment classification feature vector; a tower crane pin shaft safety warning classification judgment unit 133, which is used to pass the optimized tower crane pin shaft safety warning judgment classification feature vector through a classifier to obtain a classification result, and the classification result is used to determine whether to issue a tower crane pin shaft safety warning.

[0046] It should be understood that by combining the multimodal feature vector of the shaft pin state with the vibration feature vector, a comprehensive classification feature vector can be obtained, which not only takes into account the static state of the equipment, but also combines the dynamic vibration mode. This comprehensive feature vector can more accurately reflect the overall health of the equipment, thereby improving the accuracy of fault detection and safety warnings. Ultimately, the generation of classification feature vectors enables the early warning system to classify and judge the equipment status more accurately. By comprehensively analyzing these features, potential safety risks can be more effectively identified, and early warnings can be issued in a timely manner to prevent equipment failures and ensure the safe operation of the equipment.

[0047] In particular, in the technical solution of the present application, it is considered that data from different sensors (such as monitoring images, position change trajectories, and vibration signals) may provide similar information in some aspects. In the feature extraction stage, similar features may be extracted from the tower crane pin monitoring feature vector and the vibration waveform feature vector. Redundant features may be introduced when associating the tower crane pin monitoring feature vector and the position change operation trajectory semantic feature vector. If these features overlap in some aspects, it may cause repeated information to be contained in the final multimodal association feature vector. When pooling the vibration waveform feature map, the pooling operation may not be able to completely eliminate redundant information, especially when there are already a large number of redundant features in the original features, the final feature vector may still retain repeated information, and feature redundancy makes the feature distributions of different categories (such as normal, warning, fault, etc.) similar. This similarity will make it difficult for the classifier to clearly distinguish different categories when making classification judgments, thereby causing a run between categories and affecting the classification iteration effect. In order to solve this problem, in the technical solution of the present application, the tower crane pin shaft safety warning judgment classification feature vector is optimized by maximizing the entropy degradation of the prediction parameter space of the classifier based on the tower crane pin shaft safety warning judgment classification feature vector.

[0048] Among them, the entropy degradation of the prediction parameter space of the classifier based on the tower crane pin shaft safety warning judgment classification feature vector is maximized to obtain the optimized tower crane pin shaft safety warning judgment classification feature vector, including: extracting each row vector in the prediction parameter space of the classifier to obtain a set of tower crane pin shaft safety warning prediction parameter row vectors; calculating the class probability entropy degradation simulation value between the tower crane pin shaft safety warning judgment classification feature vector and each tower crane pin shaft safety warning prediction parameter row vector in the set of tower crane pin shaft safety warning prediction parameter row vectors to obtain a set of tower crane pin shaft safety warning class probability entropy degradation simulation values; calculating the square root of the sum of squares of the tower crane pin shaft safety warning class probability entropy degradation simulation values ​​to obtain a tower crane pin shaft safety warning entropy degradation modulation value; based on the tower crane pin shaft safety warning entropy degradation modulation value, the tower crane pin shaft safety warning judgment classification feature vector is subjected to feature distribution modulation to obtain the optimized tower crane pin shaft safety warning judgment classification feature vector.

[0049] The optimization formula is:

[0050] P={p1,p2,...,p j ,...,p n}

[0051] D j =Softmax(M m v c +H m p j +b m )

[0052]

[0053] Among them, P represents the set of row vectors of tower crane pin shaft safety warning prediction parameters, p1, p2, p j 、p n represents the first, second, jth and nth tower crane pin shaft safety warning prediction parameter row vectors in the set of tower crane pin shaft safety warning prediction parameter row vectors, v c represents the tower crane pin shaft safety warning judgment classification feature vector, Softmax represents the normalized exponential function, M m represents the first weight matrix, H m represents the second weight matrix, b m represents the bias vector, D jrepresents the jth tower crane pin safety warning class probability entropy degradation simulation value of the set of tower crane pin safety warning class probability entropy degradation simulation values, t represents the tower crane pin safety warning entropy degradation modulation value, v c ⊙t Represents the calculation of the tower crane pin shaft safety warning judgment classification feature vector v c Each eigenvalue of the t-th power, represents the i-th eigenvalue of the tower crane pin shaft safety early warning judgment classification feature vector, α represents a predetermined hyperparameter, ⊕ represents addition by position, and V′ represents the optimized tower crane pin shaft safety early warning judgment classification feature vector.

[0054] Specifically, in the technical solution of the present application, the entropy degradation of the tower crane pin shaft safety warning judgment classification feature vector based on the prediction parameter space of the classifier is maximized, and each row vector is first extracted from the prediction parameter space of the classifier to form a row vector set. Then, the class probability entropy degradation simulation value between the tower crane pin shaft safety warning judgment classification feature vector and each row vector in the above row vector set is calculated to measure the uncertainty or information entropy change of the tower crane pin shaft safety warning judgment classification feature vector under given parameters. Then, all the tower crane pin shaft safety warning class probability entropy degradation simulation values ​​are squared and summed, and then the square root is taken to obtain a comprehensive tower crane pin shaft safety warning entropy degradation modulation value, which reflects the degree of entropy change of the entire system when processing feature vectors. Finally, the tower crane pin shaft safety warning entropy degradation modulation value obtained above is used to adjust the tower crane pin shaft safety warning judgment classification feature vector. Specifically, the attribute set of the tower crane pin shaft safety warning judgment classification feature vector is represented in the form of a power series based on the tower crane pin shaft safety warning entropy degradation modulation value, and a set aggregated bounded deviation represented by an aggregation radius based on each attribute value of the tower crane pin shaft safety warning judgment classification feature vector is added, so as to perform attribute selection and sparse optimization on the tower crane pin shaft safety warning judgment classification feature vector under a unified tower crane pin shaft safety warning entropy degradation modulation value restriction standard, so as to ensure that the tower crane pin shaft safety warning judgment classification feature vector contains sufficient data but is not too cumbersome so as to introduce unnecessary interference or redundancy.

[0055] Furthermore, the optimized tower crane pin shaft safety warning judgment classification feature vector is formed after comprehensive analysis by integrating the multi-modal correlation characteristics and vibration waveform characteristics of the tower crane pin shaft state. The optimized tower crane pin shaft safety warning judgment classification feature vector contains a comprehensive description of the health status of the equipment, including static state information (such as temperature, load) and dynamic information (such as vibration mode). The significance of the classification result is to transform complex feature information into actual decision-making basis. The classification result can clearly indicate whether the current state of the equipment is normal and whether there are potential safety risks. For example, if the classifier marks the optimized tower crane pin shaft safety warning judgment classification feature vector as "warning", the system will issue a corresponding safety warning to remind the operator to take measures to prevent possible equipment damage or safety accidents.

[0056] In summary, the embodiment of the present application first obtains the tower crane pin monitoring image collected by the camera, the pin position change trajectory collected by the position sensor, and the tower crane pin vibration signal collected by the vibration sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the three. Finally, a classifier is used to determine whether to issue a tower crane pin safety warning, so as to detect potential abnormalities or fault signs in time, thereby improving the safety and reliability of the equipment.

[0057] As described above, the tower crane pin shaft safety state identification system 100 according to the embodiment of the present application can be implemented in various terminal devices. In one example, the tower crane pin shaft safety state identification system 100 can be integrated into the terminal device as a software module and / or a hardware module. For example, the tower crane pin shaft safety state identification system 100 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the tower crane pin shaft safety state identification system 100 can also be one of the many hardware modules of the terminal device.

[0058] Alternatively, in another example, the tower crane pin shaft safety status identification system 100 and the terminal device may also be separate devices, and the tower crane pin shaft safety status identification system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0059] Figure 5 FIG. 1 is a flow chart of a method for identifying the safe state of a tower crane pin shaft according to an embodiment of the present application. Figure 5As shown, according to the method for identifying the safe state of a tower crane pin shaft in an embodiment of the present application, it includes: S110, acquiring a monitoring image of a tower crane pin shaft collected by a camera, a position change running trajectory of the pin shaft collected by a position sensor, and a vibration signal of a tower crane pin shaft collected by a vibration sensor; S120, extracting a multi-modal correlation feature vector of a tower crane pin shaft state and a tower crane pin shaft vibration waveform feature vector from the monitoring image of a tower crane pin shaft collected by the camera, the position change running trajectory of the pin shaft collected by the position sensor, and the vibration signal of a tower crane pin shaft collected by the vibration sensor; S130, judging whether to issue a safety warning for a tower crane pin shaft based on the multi-modal correlation feature vector of the tower crane pin shaft state and the vibration waveform feature vector of the tower crane pin shaft.

[0060] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned tower crane pin shaft safety state identification method have been described in the above reference. Figures 1 to 4 The description of the tower crane pin shaft safety state identification system has been introduced in detail, and therefore, its repeated description will be omitted.

[0061] Below, reference Figure 6 To describe an electronic device according to an embodiment of the present application.

[0062] like Figure 6 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 14 and the output interface 15, respectively, and then connected to other components of the electronic device 10.

[0063] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.

[0064] In one embodiment, Figure 6 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the tower crane pin shaft safety status identification methods described in the above embodiments.

[0065] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0066] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0067] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A tower crane pin shaft safety status identification system, characterized in that: include: The tower crane pin shaft data acquisition module is used to acquire the tower crane pin shaft monitoring image collected by the camera, the pin shaft position change operation trajectory collected by the position sensor, and the tower crane pin shaft vibration signal collected by the vibration sensor; A tower crane pin data extraction module is used to extract a multi-modal correlation feature vector of a tower crane pin state and a tower crane pin vibration waveform feature vector from the tower crane pin monitoring image collected by the camera, the pin position change running trajectory collected by the position sensor, and the tower crane pin vibration signal collected by the vibration sensor; The pin shaft safety warning judgment module is used to judge whether to issue a tower crane pin shaft safety warning based on the multi-modal association feature vector of the tower crane pin shaft state and the tower crane pin shaft vibration waveform feature vector.

2. The tower crane pin safety status identification system according to claim 1 is characterized in that: The tower crane pin shaft data extraction module comprises: A multimodal feature extraction unit for axle pin state, used for extracting features from the monitoring image of the tower crane pin collected by the camera and the position change running track of the pin collected by the position sensor to obtain a multimodal correlation feature vector of the tower crane pin state; The pin shaft vibration signal feature extraction unit is used to extract features from the tower crane pin shaft vibration signal collected by the vibration sensor to obtain the tower crane pin shaft vibration waveform feature vector.

3. The tower crane pin safety status identification system according to claim 2 is characterized in that: The axle pin state multimodal feature extraction unit comprises: A pin shaft monitoring feature encoding subunit is used to perform feature encoding on the tower crane pin shaft monitoring image acquired by the camera to obtain a tower crane pin shaft monitoring feature vector; A pin position change feature encoding subunit, used for performing feature encoding on the position change running track of the pin collected by the position sensor to obtain a semantic feature vector of the position change running track; The axle pin state multimodal feature association subunit is used to associate the tower crane axle pin monitoring feature vector with the position change operation trajectory semantic feature vector to obtain the tower crane axle pin state multimodal association feature vector.

4. The tower crane pin shaft safety status identification system according to claim 3 is characterized in that: The pin shaft monitoring feature encoding subunit comprises: The tower crane pin monitoring image collected by the camera is passed through a tower crane pin monitoring convolutional neural network as a noise reducer to obtain a tower crane pin monitoring image after noise reduction; The denoised tower crane pin monitoring image is passed through a tower crane pin monitoring convolutional neural network as a feature encoder to obtain the tower crane pin monitoring feature vector.

5. The tower crane pin safety status identification system according to claim 4, characterized in that: The pin position change characteristic encoding subunit comprises: The position change running track of the pin shaft collected by the position sensor is passed through a position change running track word segmentation processor to obtain a position change running track word sequence; The position change running trajectory word sequence is passed through a position change running trajectory text convolutional neural network as a semantic encoder to obtain the position change running trajectory semantic feature vector.

6. The tower crane pin safety status identification system according to claim 5, characterized in that: The pin shaft vibration signal feature extraction unit comprises: Extracting a waveform diagram of the tower crane pin vibration signal from the tower crane pin vibration signal collected by the vibration sensor; The waveform diagram of the extracted tower crane pin vibration signal is obtained by using a tower crane pin vibration waveform feature encoder using a hole convolutional neural network to obtain a tower crane pin vibration waveform feature diagram; The tower crane pin shaft vibration waveform feature map is pooled to obtain a tower crane pin shaft vibration waveform feature vector.

7. The tower crane pin safety status identification system according to claim 6, characterized in that: The pin shaft safety early warning judgment module includes: A tower crane pin shaft safety feature fusion unit, used to fuse the multi-modal correlation feature vector of the tower crane pin shaft state and the tower crane pin shaft vibration waveform feature vector to obtain a tower crane pin shaft safety early warning judgment classification feature vector; A tower crane pin shaft safety feature optimization unit, used for maximizing the entropy degradation of the prediction parameter space of the tower crane pin shaft safety warning judgment classification feature vector based on the classifier to obtain an optimized tower crane pin shaft safety warning judgment classification feature vector; The tower crane pin shaft safety warning classification judgment unit is used to pass the optimized tower crane pin shaft safety warning judgment classification feature vector through a classifier to obtain a classification result, and the classification result is used to judge whether to issue a tower crane pin shaft safety warning.

8. The tower crane pin shaft safety status identification system according to claim 7, characterized in that: The tower crane pin shaft safety feature optimization unit comprises: Extracting each row vector in the prediction parameter space of the classifier to obtain a set of row vectors of tower crane pin shaft safety warning prediction parameters; Calculating the class probability entropy degradation simulation value between the tower crane pin shaft safety warning judgment classification feature vector and each tower crane pin shaft safety warning prediction parameter row vector in the set of the tower crane pin shaft safety warning prediction parameter row vector to obtain a set of tower crane pin shaft safety warning class probability entropy degradation simulation values; Calculating the square root of the sum of squares of the tower crane pin shaft safety warning class probability entropy degradation simulation values ​​to obtain the tower crane pin shaft safety warning entropy degradation modulation value; Based on the tower crane pin shaft safety warning entropy degradation modulation value, the tower crane pin shaft safety warning judgment classification feature vector is subjected to feature distribution modulation to obtain the optimized tower crane pin shaft safety warning judgment classification feature vector.

9. A method for identifying the safe state of a tower crane pin shaft, characterized in that: include: Acquire the monitoring image of the tower crane pin shaft collected by the camera, the position change running track of the pin shaft collected by the position sensor, and the vibration signal of the tower crane pin shaft collected by the vibration sensor; Extracting a multimodal correlation feature vector of a tower crane axle pin state and a tower crane axle pin vibration waveform feature vector from the tower crane axle pin monitoring image acquired by the camera, the position change running track of the axle pin acquired by the position sensor, and the tower crane axle pin vibration signal acquired by the vibration sensor; Based on the multi-modal association feature vector of the tower crane axle pin state and the vibration waveform feature vector of the tower crane pin shaft, it is determined whether to issue a tower crane pin shaft safety warning.

10. The tower crane pin safety state identification method according to claim 9, characterized in that: Extracting a multimodal correlation feature vector of a tower crane shaft pin state and a tower crane shaft pin vibration waveform feature vector from the tower crane shaft pin monitoring image acquired by the camera, the shaft pin position change running track acquired by the position sensor, and the tower crane shaft pin vibration signal acquired by the vibration sensor, including: Performing feature extraction on the monitoring image of the tower crane pin shaft collected by the camera and the position change running trajectory of the pin shaft collected by the position sensor to obtain a multi-modal correlation feature vector of the tower crane pin shaft state; The vibration signal of the tower crane pin shaft collected by the vibration sensor is subjected to feature extraction to obtain a vibration waveform feature vector of the tower crane pin shaft.

Images