Crane operation monitoring method based on neural network architecture search
By using neural network architecture search and cross-modal self-supervised learning, multi-source data from lifting machinery is adaptively processed, solving the problems of noise and data misalignment. This achieves high-precision data purification and alignment, improving the accuracy and reliability of the monitoring system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SICHUAN JINGZHUN SPECIAL EQUIP INSPECTION CO LTD
- Filing Date
- 2026-03-10
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies for monitoring lifting machinery suffer from inaccurate data processing due to noise, outliers, and data misalignment. They also ignore the physical correlation between multiple data sources and struggle to adaptively handle noise patterns under complex operating conditions.
We employ neural network architecture search and cross-modal self-supervised learning to construct an adaptive preprocessing network. By automatically optimizing the topology, we can clean and align multi-source data streams and perform adaptive processing by utilizing the physical correlations between data.
It enables high-precision data purification and alignment under complex working conditions without the need for manual annotation, improving the accuracy and reliability of the monitoring system, reducing false alarms and missed alarms, and ensuring the safety of lifting machinery.
Smart Images
Figure CN121834356B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of crane monitoring technology, and specifically relates to a crane operation monitoring method based on neural network architecture search. Background Technology
[0002] In the field of intelligent monitoring of large industrial equipment such as cranes, it is common practice to deploy various types of heterogeneous sensors on the equipment to acquire massive amounts of operational data for health status assessment, fault early warning, and safety risk identification. However, due to electromagnetic interference in the acquisition environment, the physical characteristics of the sensors themselves, and complex operating conditions, these raw working data often contain noise, outliers, and anomalies. Furthermore, there are often problems such as inconsistent sampling frequencies or misaligned sampling timestamps caused by transmission delays between different data streams. Therefore, effective purification and alignment preprocessing of the raw data is a crucial prerequisite for ensuring the accurate operation of subsequent advanced analysis algorithms. In existing technologies, data preprocessing mainly employs signal processing or statistical methods, such as Kalman filters, wavelet transforms, or moving averages. Once the algorithm models and core parameters of these methods are set, they are difficult to change, relying on extensive engineering experience for manual selection and optimization. This approach cannot adaptively address the complex and variable noise patterns exhibited by cranes under different operating conditions, environments, and aging levels. Furthermore, these methods typically process individual data channels independently, ignoring the physical correlation between different data streams. This can easily lead to the misinterpretation of real but dramatic signal steps as noise and the smoothing out of the signal, or to incorrect judgments during time alignment, resulting in the loss of critical information. Summary of the Invention
[0003] To address the shortcomings of existing technologies, this invention provides a method for monitoring the operation of lifting machinery based on neural network architecture search, thereby solving the aforementioned technical problems.
[0004] The method for monitoring the operation of lifting machinery based on neural network architecture search includes the following steps:
[0005] Raw working data is acquired on the lifting machinery using heterogeneous sensors. The raw working data includes noise, outliers, abnormal values, and sampling delays caused by the acquisition environment and equipment characteristics. The raw working data of the lifting machinery includes hoisting data and movement data. The hoisting data includes hoisting motor power, load data, and hook image data. The movement data includes travel motor power, environmental image data, and lifting machinery position data.
[0006] The raw working data is cleaned and aligned by an adaptive preprocessing network equipped with a neural network architecture search algorithm to generate a clean data stream.
[0007] The clean data stream is subjected to threshold detection processing, and the output result is the security monitoring result.
[0008] Preferably, the construction and application of the adaptive preprocessing network specifically includes:
[0009] A neural network architecture search algorithm is applied to automatically optimize within a predefined search space to determine the internal topology of the adaptive preprocessing network.
[0010] The network is trained using the original working data as input by employing cross-modal self-supervised learning. The task of the self-supervised learning is to apply random distortion to the data of one or more input data streams within a set time window during the training process, and drive the adaptive preprocessing network to reconstruct the distorted data based on other unaffected related data streams with the optimization objective of minimizing the reconstruction error.
[0011] Preferably, when applying the neural network architecture search algorithm to automatically optimize within a predefined search space to determine the internal topology of the adaptive preprocessing network, the method further includes the following steps:
[0012] A hypernetwork containing multiple candidate operators and connection paths is constructed as the search space, wherein the multiple candidate operators include at least: a convolution operator for extracting local spatial features, a self-attention operator for capturing long-distance temporal dependencies, and a pooling operator for feature fusion;
[0013] By applying a search strategy based on reinforcement learning or evolutionary algorithms, the selection of candidate operators and connection paths in the supernetwork are optimized, thereby determining the internal topology with the best performance index.
[0014] Preferably, determining the internal topology with optimal performance indicators specifically includes the following steps:
[0015] During training, a candidate subnetwork architecture is periodically sampled from the supernetwork;
[0016] Using a reserved validation dataset, the candidate sub-network architecture is evaluated using a cross-modal self-supervised learning approach. Specifically, the data stream in the validation dataset is distorted, and the data is reconstructed using the candidate sub-network architecture. The loss function value of the candidate sub-network architecture on the data reconstruction task is calculated, and the loss function value is used as the performance index. The lower the loss function value, the better the performance index.
[0017] Preferably, when applying random distortion to one or more input data streams within a set time window during training, the specific steps are as follows:
[0018] Based on the original working data, the real-time operating status of the lifting machinery is divided into at least one core working characteristic, which includes at least: load conversion stage, handling stage and empty hook stage;
[0019] Based on the identified core operational characteristics, select one or more data streams as the data streams to be distorted or masked.
[0020] Preferably, when determining the working characteristics of the lifting machinery based on its original working data, the specific steps include:
[0021] When the hoisting motor power exhibits a significant starting peak or a step change, while the movement data remains constant, the operating characteristic is the load transition phase.
[0022] If the position data of the lifting machinery changes continuously while the load data remains stable at a non-zero value, then the working characteristic is the handling stage.
[0023] When the load data stabilizes at a low level corresponding to the empty hook's own weight, the working characteristic is the empty hook stage.
[0024] Preferably, during the load conversion phase, distortion is preferentially applied to the load data;
[0025] During the transport phase, distortion is preferentially applied to the hook image data or environmental image data;
[0026] During the empty hook stage, it is preferable to apply distortion or masking to the crane position data or travel motor power.
[0027] Preferably, when selecting one or more data streams as the data streams to be distorted or masked, the method further includes the following steps:
[0028] The working intensity of the lifting machinery is determined based on the original working data, and the working intensity includes at least low-intensity working conditions, medium-intensity working conditions, and high-intensity working conditions;
[0029] Based on the workload, adjust the amplitude or duration of the random distortion.
[0030] Preferably, the threshold detection process for the clean data stream includes the following steps:
[0031] The non-image data streams in the clean data stream are compared with their respective preset safety thresholds in real time to identify threshold-type risks, including overload, speeding, or exceeding limits. The non-image data streams include hoisting motor power, load data, travel motor power, and crane position data.
[0032] The image data stream in the clean data stream is analyzed using an artificial intelligence image analysis model to identify visual risks, including swaying of the suspended object or path intrusion. The image data stream includes hook image data and environmental image data.
[0033] Preferably, when the output processing result is a security monitoring result, the process specifically includes the following steps:
[0034] Establish a risk fusion decision-making model;
[0035] The occurrence, severity, and combination relationship of the threshold-type risks and visual risks are taken as input, and the risk fusion decision model is used for querying and reasoning to determine the output result as the comprehensive risk level.
[0036] Based on the comprehensive risk level, differentiated safety monitoring results are output, including at least: issuing early warning prompts to the operating terminal or issuing deceleration or shutdown commands to the control system.
[0037] The beneficial effects of this invention are as follows: By introducing neural network architecture search and cross-modal self-supervised learning, it overcomes the shortcomings of traditional methods, such as reliance on human experience and model rigidity. It adaptively constructs and optimizes a preprocessing model that deeply understands the physical relationships between multi-source data, thereby achieving adaptive, high-precision integrated processing of noise and anomalies in complex working conditions without manual annotation. Furthermore, because this method can continuously output high-quality, clean data streams, it greatly improves the input quality of subsequent risk identification modules such as threshold detection, thus significantly improving the accuracy and reliability of the entire monitoring system. This effectively reduces false alarms and missed alarms caused by data quality issues, ensuring the inherent safety of lifting machinery. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 A flowchart illustrating the crane operation monitoring method based on neural network architecture search provided by this invention;
[0040] Figure 2 This is a schematic diagram of the risk fusion and decision processing of the lifting machinery operation monitoring method based on neural network architecture search provided by the present invention. Detailed Implementation
[0041] The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Of course, these are merely examples and are not intended to limit the present invention.
[0042] The embodiments of the invention will now be described in detail with reference to the accompanying drawings.
[0043] like Figure 1 As shown, the crane operation monitoring method based on neural network architecture search includes the following steps:
[0044] Raw working data is acquired on the lifting machinery using heterogeneous sensors. The raw working data includes noise, outliers, abnormal values, and sampling delays caused by the acquisition environment and equipment characteristics. The raw working data of the lifting machinery includes hoisting data and movement data. The hoisting data includes hoisting motor power, load data, and hook image data. The movement data includes travel motor power, environmental image data, and lifting machinery position data.
[0045] The raw working data is cleaned and aligned by an adaptive preprocessing network equipped with a neural network architecture search algorithm to generate a clean data stream.
[0046] The clean data stream is subjected to threshold detection processing, and the output result is the security monitoring result.
[0047] In its specific implementation, this invention first deploys various heterogeneous sensors, including power, load, position, and image sensors, on the lifting machinery to collect raw operational data in real time, which is mixed with noise, anomalies, and time delays. Next, this method constructs and applies an adaptive preprocessing network equipped with a neural network architecture search algorithm. This network does not employ a fixed, manually designed structure, but rather automatically optimizes within a vast search space containing various basic computational units, based on the characteristics of the actual input data, to determine an optimal end-to-end network topology capable of collaboratively processing all data streams. This network is trained using a cross-modal self-supervised learning paradigm to reconstruct the distorted data by applying random occlusion or distortion to one or more data streams within a specific time window, with the goal of minimizing reconstruction error. The network learns and utilizes the inherent physical correlations of other unaffected related data streams. In this way, the adaptive network can not only adaptively design the purification and alignment model best suited to the current working conditions, but also comprehensively consider global data information during the processing, thereby achieving integrated, high-precision purification and alignment of multi-source heterogeneous data, and finally generating a reliable clean data stream, which is then input into the downstream threshold detection process to output accurate safety monitoring results.
[0048] This invention, by introducing neural network architecture search and cross-modal self-supervised learning, overcomes the reliance on human experience and the rigidity of traditional methods. It adaptively constructs and optimizes a preprocessing model that deeply understands the physical relationships between multi-source data, thereby achieving adaptive, high-precision integrated processing of noise and anomalies in complex operating conditions without manual annotation. Furthermore, because this method continuously outputs high-quality, clean data streams, it significantly improves the input quality of subsequent risk identification modules such as threshold detection, thereby significantly enhancing the accuracy and reliability of the entire monitoring system. This effectively reduces false alarms and missed alarms caused by data quality issues, ensuring the inherent safety of lifting machinery.
[0049] More specifically, the construction and application of the adaptive preprocessing network includes:
[0050] A neural network architecture search algorithm is applied to automatically optimize within a predefined search space to determine the internal topology of the adaptive preprocessing network.
[0051] The network is trained using the original working data as input by employing cross-modal self-supervised learning. The task of the self-supervised learning is to apply random distortion to the data of one or more input data streams within a set time window during the training process, and drive the adaptive preprocessing network to reconstruct the distorted data based on other unaffected related data streams with the optimization objective of minimizing the reconstruction error.
[0052] Because the data from different sensors on lifting machinery are not isolated during operation, they exhibit various correlations based on physical laws: in the mechanical correlation structure, a surge in the power of the hoisting motor necessarily corresponds to an increase in load and a rise in the Z-axis position of the hook (i.e., the hoisting motor power); in the kinematic correlation structure, a change in the current of the travel motor necessarily corresponds to a change in the crane's position coordinates (i.e., a change in the travel motor power); in the dynamic correlation structure, a change in the travel motor necessarily corresponds to a change in the environment surrounding the lifting machinery and a periodic change in structural stress (i.e., the crane's position data). In traditional preprocessing, these correlations are ignored. However, through the reconstruction process, if the neural network wants to minimize the reconstruction error, it can only learn and master these physical laws hidden behind the data. During training, the system randomly selects one or more data streams from the input raw working data (e.g., hoist motor power, load data, hook image data, and crane position data), and applies artificial random distortion or complete masking to the data within a certain time window. The network is then tasked with accurately reconstructing the distorted load data based on other unaffected related data streams, such as hoist motor power and hook images. By minimizing this reconstruction error as the optimization objective, the network is forced to deeply learn and understand the inherent and profound spatiotemporal correlations between different physical quantities during crane operation.
[0053] If the reconstructed data from the network is highly consistent with the original, masked data, it means the network has essentially learned the physical relationships between the data. Its reconstruction capability is strong, and it can accurately infer missing information based on other data. This signifies that the training process was successful and the model has converged.
[0054] If the data reconstructed from the network is inconsistent with the original, masked data, it usually means that:
[0055] The original data is noisy: there are spikes or peaks in the original data that should not exist, while the network reconstructs a smooth, spike-free theoretical value based on other smooth data.
[0056] The original data was an outlier: a sensor momentarily malfunctioned, giving an abnormal reading. The network reconstructed a theoretical value that conformed to common sense physics based on data from other working sensors, thus correcting the anomaly.
[0057] The raw data has a delay: for example, the data from sensor A is half a second slower than that from sensor B. When reconstructing data A, the network automatically references data from sensor B from half a second earlier, thus correcting the timestamp of data A in the reconstruction result.
[0058] Therefore, when the reconstructed data differs from the original data, the data reconstructed using other reference data is purer and closer to the actual physical process, making it more reliable than the original, unprocessed data.
[0059] More specifically, when applying the neural network architecture search algorithm to automatically optimize within a predefined search space to determine the internal topology of the adaptive preprocessing network, the method further includes the following steps:
[0060] A hypernetwork containing multiple candidate operators and connection paths is constructed as the search space, wherein the multiple candidate operators include at least: a convolution operator for extracting local spatial features, a self-attention operator for capturing long-distance temporal dependencies, and a pooling operator for feature fusion;
[0061] By applying a search strategy based on reinforcement learning or evolutionary algorithms, the selection of candidate operators and connection paths in the supernetwork are optimized, thereby determining the internal topology with the best performance index.
[0062] In this scheme, a supernetwork with multiple candidate operators pre-configured specifically for the multi-source heterogeneous data characteristics of cranes is first constructed. These include a convolution operator for efficiently extracting local spatial features from data such as hook images and environmental images; a self-attention operator for deeply understanding long-range temporal dependencies in time-series data such as hoisting motor power, load data, and crane position data; and a pooling operator for efficiently fusing information from different sensors. Next, an intelligent search strategy based on reinforcement learning or evolutionary algorithms is introduced. This strategy aims to continuously select operators within the supernetwork according to preset performance indicators and attempt to combine and connect them. Through thousands of iterations of optimization, a specific internal topology that can complete the data preprocessing task with optimal performance is ultimately determined.
[0063] More specifically, determining the internal topology with optimal performance indicators includes the following steps:
[0064] During training, a candidate subnetwork architecture is periodically sampled from the supernetwork;
[0065] Using a reserved validation dataset, the candidate sub-network architecture is evaluated using a cross-modal self-supervised learning approach. Specifically, the data stream in the validation dataset is distorted, and the data is reconstructed using the candidate sub-network architecture. The loss function value of the candidate sub-network architecture on the data reconstruction task is calculated, and the loss function value is used as the performance index. The lower the loss function value, the better the performance index.
[0066] To determine the internal topology with optimal performance metrics, this scheme periodically samples a candidate subnetwork from the supernetwork during the training iterations of the neural network architecture search. The performance of this network is evaluated on a reserved validation set through a self-supervised learning-based agent task. The evaluation process distorts one or more modalities in the validation data stream and requires the candidate network to reconstruct the distorted data using the remaining undistorted data. Finally, the loss function value between the reconstructed data and the original data is calculated; this loss value is defined as the performance metric of the candidate subnetwork and fed back to the neural network architecture search strategy to guide subsequent sampling. A lower loss value indicates better performance. Compared to existing technologies, this implementation replaces the complete supervised training required to evaluate candidate networks with a computationally inexpensive self-supervised reconstruction task, reducing the evaluation cycle of a single architecture from hours to minutes. Furthermore, it can accurately reconstruct networks with missing physical quantities based on partial data, inherently forming an effective representation of the physical causal relationships between multimodal data, thus ensuring the validity of the search results.
[0067] More specifically, when applying random distortion to one or more input data streams within a set time window during training, the specific steps are as follows:
[0068] Based on the original working data, the real-time operating status of the lifting machinery is divided into at least one core working characteristic, which includes at least: load conversion stage, handling stage and empty hook stage;
[0069] Based on the identified core operational characteristics, select one or more data streams as the data streams to be distorted or masked.
[0070] In this approach, by reconstructing the masked "result" data using "causal" data in each specific physical scenario, a learning task containing physical causal relationships is created, improving training efficiency and model convergence speed. Secondly, this targeted distortion based on operational characteristics guides the model to learn the core physical models of lifting machinery at different stages in a structured manner, rather than forming an undifferentiated and difficult-to-interpret hybrid model. This makes the final representation more robust and interpretable. Furthermore, by constructing a self-supervised training framework strongly coupled with the physical process, this invention ensures that the network learns not only data correlations but also a deep understanding of the equipment's operating mechanism.
[0071] More specifically, determining the working characteristics of a lifting machine based on its original working data includes the following steps:
[0072] When the hoisting motor power exhibits a significant starting peak or a step change, while the movement data remains constant, the operating characteristic is the load transition phase.
[0073] If the position data of the lifting machinery changes continuously while the load data remains stable at a non-zero value, then the working characteristic is the handling stage.
[0074] When the load data stabilizes at a low level corresponding to the empty hook's own weight, the working characteristic is the empty hook stage.
[0075] During the load transition phase, a significant peak or abrupt change in the hoisting motor power is the most direct electrical characteristic signal of a rapid energy conversion in the system, as the hoisting motor is the only actuator directly resisting gravity. However, this condition alone is insufficient to eliminate interference from other operating conditions. Therefore, the constraint of "unchanged movement data" serves to eliminate interference, ensuring that the aforementioned power change is not caused by the horizontal movement of the crane's trolley or crane crane itself, thus eliminating power fluctuation interference during horizontal acceleration and deceleration. When both conditions are met simultaneously, the reasonable physical scenario is that the crane hook is performing a purely vertical lifting or lowering motion, i.e., the load transition phase where the load state undergoes a fundamental change.
[0076] During the handling phase, "continuous changes in the crane's position data" is a direct kinematic indicator of spatial displacement of the system. This condition confirms that the crane is in a dynamic, non-static operation phase, thus distinguishing it from other static operation phases such as static lifting or waiting for instructions. However, displacement alone is insufficient. Therefore, the condition "load data stabilizes at a non-zero value" is added for qualitative analysis. The "non-zero value" confirms that the crane is under load, ruling out empty hook operation; while "stable" confirms that there is no significant vertical movement, i.e., no load lifting or lowering, effectively separating this phase from the step change process of the "load transfer phase." When both conditions are met simultaneously, the physical scenario corresponding to the system is: a load of constant weight that has already been lifted is being moved by the crane from one location to another.
[0077] During the empty hook phase, the single condition that "the load data is stable at a low level corresponding to the empty hook's own weight" provides two clear pieces of physical information: the load value matches the known empty hook's own weight, providing physical evidence that the system is in an unloaded state, thus ruling out all loaded operation phases; the stability of the load value proves that the hook has no acceleration in the vertical direction, thus ruling out dynamic processes of lifting or lowering. Therefore, this single condition is sufficient to limit the empty hook phase to when the lifting machinery is only moving horizontally or stationary.
[0078] More specifically, during the load conversion phase, distortion is preferentially applied to the load data;
[0079] During the transport phase, distortion is preferentially applied to the hook image data or environmental image data;
[0080] During the empty hook stage, it is preferable to apply distortion or masking to the crane position data or travel motor power.
[0081] During the load transition phase, distortion is preferentially applied to the load data: the core physical process at this stage is the conversion between force and work. The step change in the load data is the final effect of this process. By masking this data, the model is forced to perform causal reasoning: the masked load weight is reconstructed based on three factors: the instantaneous peak power of the hoisting motor, the change in the hook's Z-axis coordinate, and the change in the wire rope tension state in the image. This enables the network to learn the dynamic mechanical model of the crane, namely the quantitative relationship between motor power, mechanical motion, and load changes.
[0082] During the handling phase, image data distortion is preferentially applied: the core physical processes at this stage are motion and inertia. The swaying of the suspended load is the most significant visual effect in this process. By masking the image, the model infers the three-dimensional spatial dynamics from one-dimensional time-series data: based on non-visual factors such as the acceleration / deceleration curves of the traveling motor current, the rate of change of position data, and the periodic oscillations of the structural stress sensor, the amplitude and frequency of the suspended load's swaying in the masked image are reconstructed. This enables the network to learn the inertial swaying model of the crane.
[0083] During the empty hook phase, distortion is preferentially applied to the position data or travel motor power: the core physical process at this stage is the mapping between command and execution. Changes in position data are the final result of the motor executing control commands. By masking the position data, the model learns the machine's own kinematic model: based on the control commands from the operating handle and the power / current readings of the travel motor, combined with known empty hook load conditions, the masked position coordinates are reconstructed. This allows the network to accurately characterize the transfer function from the control signal to the actual displacement, i.e., the crane's own motion characteristics.
[0084] The core logic of this scheme is to shield the most critical result data at each specific working stage, thereby enabling the neural network to reverse-engineer and reconstruct the result by analyzing the remaining causal data streams, thus learning the underlying physical model of that stage.
[0085] More specifically, when selecting one or more data streams as the data streams to be distorted or masked, the method further includes the following steps:
[0086] The working intensity of the lifting machinery is determined based on the original working data, and the working intensity includes at least low-intensity working conditions, medium-intensity working conditions, and high-intensity working conditions;
[0087] Based on the workload, adjust the amplitude or duration of the random distortion.
[0088] In this scheme, the scale of data distortion in self-supervised learning is dynamically adjusted according to the actual working intensity of the lifting machinery, so that the distortion task matches the physical event characteristics of interest under that intensity, thereby creating an efficient and relevant learning objective for the model.
[0089] In practice, the determination of workload and the adjustment of distortion strategies are as follows:
[0090] High-intensity working conditions: The load data is detected to instantaneously exceed 80% of the rated load, the position data of the walking mechanism shows an extremely high rate of acceleration or deceleration change, or the motor current shows a peak far exceeding the norm.
[0091] Distortion adjustment strategy: Employ a short-duration, high-sampling-rate micro-window, such as 0.5-5 seconds. Within this window, apply a very short masking time to the data to be distorted, such as 0.1-0.5 seconds. Because the core event under high-intensity conditions is pulse-like, the distortion duration must be short enough to accurately isolate the pulse, forcing the model to reconstruct this critical event from other synchronously occurring high-frequency signals. If the distortion time is too long, the entire event itself will be completely erased, losing its learning significance.
[0092] Medium-intensity working conditions: Load data is stable in the range of 30%-80% of the rated load, and the position and power data show a complete working cycle of continuous acceleration-uniform speed-deceleration.
[0093] Distortion adjustment strategy: Use a medium-length time window, such as 10-60 seconds. This window just covers a complete standard motion. The distortion duration is also adjusted accordingly to a medium length, such as 5-20 seconds. For example, in a 30-second horizontal transport, the image data for the first 15 seconds after braking begins is masked. This duration is sufficient to cover the complete process from the start of the swing to the point of stabilization, allowing the model to learn and reconstruct the entire inertial swing process.
[0094] Low-intensity working conditions: The load data is equal to or close to the empty hook self-weight, or the power of each motor is close to zero for a long time.
[0095] Distortion adjustment strategy: Use a large time window, such as 5-30 minutes. In this scenario, two distortion methods can be used: one is long-duration masking, for example, in a 5-minute slow movement of the empty hook positioning, masking 1 minute of position data, forcing the model to learn the kinematic model based on the long-term stable motor power; the other is high-amplitude distortion, that is, not masking the data, but superimposing a large amplitude of random noise on it for a period of time. The task is to make the model reconstruct a smooth original signal, thereby learning the inherent noise characteristics and basic operating model of the system in steady state.
[0096] like Figure 2As shown, more specifically, when performing threshold detection processing on the clean data stream, the following steps are included:
[0097] The non-image data streams in the clean data stream are compared with their respective preset safety thresholds in real time to identify threshold-type risks, including overload, speeding, or exceeding limits. The non-image data streams include hoisting motor power, load data, travel motor power, and crane position data.
[0098] The image data stream in the clean data stream is analyzed using an artificial intelligence image analysis model to identify visual risks, including swaying of the suspended object or path intrusion. The image data stream includes hook image data and environmental image data.
[0099] For non-image data streams, including hoist motor power, load, travel motor power, and position data, the processing logic is based on the one-dimensional scalar or vector characteristics of these data as direct measures of physical state. The risks associated with this type of data, such as overload, speeding, or exceeding limits, are essentially deterministic events where physical quantities exceed preset safety limits. Therefore, using fixed numerical thresholds for real-time comparison is a detection method with the lowest computational cost, fastest response speed, and highly reliable results. It can directly and efficiently identify this type of threshold-based risk.
[0100] For image data streams, including images of hooks and the environment, the processing logic is based on the characteristics of this type of high-dimensional unstructured data, where pixels themselves do not directly carry physical meaning. The corresponding visual risks, such as swaying of the suspended object or path intrusion, are complex spatiotemporal patterns that cannot be defined by any single numerical threshold. Therefore, it is necessary to leverage artificial intelligence image analysis models, utilizing their feature extraction and pattern recognition capabilities, to decode complex semantic information such as swaying or intrusion from the pixel stream in order to make risk assessments. This is the necessary and only effective technical approach to handling such unstructured visual risks.
[0101] More specifically, when the output processing result is a security monitoring result, it specifically includes the following steps:
[0102] Establish a risk fusion decision-making model;
[0103] The occurrence, severity, and combination relationship of the threshold-type risks and visual risks are taken as input, and the risk fusion decision model is used for querying and reasoning to determine the output result as the comprehensive risk level.
[0104] Based on the comprehensive risk level, differentiated safety monitoring results are output, including at least: issuing early warning prompts to the operating terminal or issuing deceleration or shutdown commands to the control system.
[0105] In one implementation, the risk fusion decision model can be a Bayesian network. Each independent risk event is treated as a node in the network, and a conditional probability table is established between nodes based on historical data and expert experience. This allows for probabilistic inference of the most likely comprehensive risk level based on currently occurring risks. The core of this solution lies in overcoming the limitations of single risks and quantitatively assessing scenarios with multiple concurrent risks. The actual dangerous state of a crane is often not determined by a single overspeed or sway, but by the severity of multiple risk events and their specific combinations, such as heavy load and sudden braking. The model's function is to map multiple heterogeneous risk inputs from the threshold detection module and visual analysis module into a unified, standardized comprehensive risk level scalar through a pre-set rule base or decision logic. Based on this comprehensive risk level, differentiated safety monitoring results are output to achieve safety response and operational efficiency. It directly links the system's intervention behavior to the actual degree of danger. Low overall risk levels trigger only non-intrusive warnings, ensuring operational continuity; medium risk levels may trigger rate-limiting commands, providing proactive but gentle intervention; and only when the overall risk level reaches a high level or a critical threshold will the system execute mandatory safety actions with the highest authority, such as issuing a shutdown command. This approach avoids unnecessary downtime due to isolated, low-severity risk events, while ensuring the system can react quickly and decisively in truly dangerous complex operating conditions.
[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.
Claims
1. A method for monitoring the operation of lifting machinery based on neural network architecture search, characterized in that, Includes the following steps: Raw working data is acquired on the lifting machinery using heterogeneous sensors. The raw working data includes noise, outliers, abnormal values, and sampling delays caused by the acquisition environment and equipment characteristics. The raw working data of the lifting machinery includes hoisting data and movement data. The hoisting data includes hoisting motor power, load data, and hook image data. The movement data includes travel motor power, environmental image data, and lifting machinery position data. The raw working data is cleaned and aligned by an adaptive preprocessing network equipped with a neural network architecture search algorithm to generate a clean data stream. The clean data stream is subjected to threshold detection processing, and the output processing result is the security monitoring result. The construction and application of the adaptive preprocessing network specifically include: A neural network architecture search algorithm is applied to automatically optimize within a predefined search space to determine the internal topology of the adaptive preprocessing network. The network is trained using the original working data as input by applying cross-modal self-supervised learning. The task of the self-supervised learning is to apply random distortion to the data of one or more input data streams within a set time window during the training process, and drive the adaptive preprocessing network to reconstruct the distorted data based on other unaffected related data streams with the optimization objective of minimizing the reconstruction error. When applying random distortion to one or more input data streams within a set time window during training, specifically: Based on the original working data, the real-time operating status of the lifting machinery is divided into at least one core working characteristic, which includes at least: load conversion stage, handling stage and empty hook stage; Based on the identified core operational characteristics, select one or more data streams as the data streams to be distorted or masked; Determining the working characteristics of a lifting machine based on its original working data includes the following steps: When the hoisting motor power exhibits a significant starting peak or a step change, while the movement data remains constant, the operating characteristic is the load transition phase. If the position data of the lifting machinery changes continuously while the load data remains stable at a non-zero value, then the working characteristic is the handling stage. When the load data stabilizes at a low level corresponding to the empty hook's own weight, the working characteristic is the empty hook stage. During the load conversion phase, distortion is preferentially applied to the load data; During the transport phase, distortion is preferentially applied to the hook image data or environmental image data; During the empty hook stage, it is preferable to apply distortion or masking to the crane position data or travel motor power.
2. The method for monitoring the operation of lifting machinery based on neural network architecture search according to claim 1, characterized in that, When applying the neural network architecture search algorithm to automatically optimize within a predefined search space to determine the internal topology of the adaptive preprocessing network, the method further includes the following steps: A hypernetwork containing multiple candidate operators and connection paths is constructed as the search space, wherein the multiple candidate operators include at least: a convolution operator for extracting local spatial features, a self-attention operator for capturing long-distance temporal dependencies, and a pooling operator for feature fusion; By applying a search strategy based on reinforcement learning or evolutionary algorithms, the selection of candidate operators and connection paths in the supernetwork are optimized, thereby determining the internal topology with the best performance index.
3. The method for monitoring the operation of lifting machinery based on neural network architecture search according to claim 2, characterized in that, Determining the internal topology with optimal performance indicators specifically includes the following steps: During training, a candidate subnetwork architecture is periodically sampled from the supernetwork; Using a reserved validation dataset, the candidate sub-network architecture is evaluated using a cross-modal self-supervised learning approach. Specifically, the data stream in the validation dataset is distorted, and the data is reconstructed using the candidate sub-network architecture. The loss function value of the candidate sub-network architecture on the data reconstruction task is calculated, and the loss function value is used as the performance index. The lower the loss function value, the better the performance index.
4. The method for monitoring the operation of lifting machinery based on neural network architecture search according to claim 1, characterized in that, When selecting one or more data streams as the data streams to be distorted or masked, the method further includes the following steps: The working intensity of the lifting machinery is determined based on the original working data, and the working intensity includes at least low-intensity working conditions, medium-intensity working conditions, and high-intensity working conditions; Based on the workload, adjust the amplitude or duration of the random distortion.
5. The method for monitoring the operation of lifting machinery based on neural network architecture search according to claim 1, characterized in that, When performing threshold detection processing on the clean data stream, the specific steps include: The non-image data streams in the clean data stream are compared with their respective preset safety thresholds in real time to identify threshold-type risks, including overload, speeding, or exceeding limits. The non-image data streams include hoisting motor power, load data, travel motor power, and crane position data. The image data stream in the clean data stream is analyzed using an artificial intelligence image analysis model to identify visual risks, including swaying of the suspended object or path intrusion. The image data stream includes hook image data and environmental image data.
6. The method for monitoring the operation of lifting machinery based on neural network architecture search according to claim 5, characterized in that, When the output processing result is a security monitoring result, the specific steps include: Establish a risk fusion decision-making model; The occurrence, severity, and combination relationship of the threshold-type risks and visual risks are taken as input, and the risk fusion decision model is used for querying and reasoning to determine the output result as the comprehensive risk level. Based on the comprehensive risk level, differentiated safety monitoring results are output, including at least: issuing early warning prompts to the operating terminal or issuing deceleration or shutdown commands to the control system.