Crane dynamic instability early warning method based on multi-modal visual wind-induced oscillation decoupling

Abstract

The application discloses a kind of based on multi-modal visual wind-induced swing decoupling crane dynamic instability early warning method, this method includes, the stabilization processing of crane operation video, eliminate the false displacement of picture caused by camera installation platform vibration;In stabilized video, the position of boom end and hoist is tracked, and the relative swing displacement of hoist relative to hanging point is determined according to the coordinate difference of two;According to the operation signal of each action mechanism of crane, the operation silent window is identified, and the swing amplitude is extracted after the waiting duration after skipping the start of window, and the wind-induced swing amplitude sample is obtained;According to time sequence, the amplitude growth rate of adjacent sample is calculated, when the amplitude growth rate of continuous setting number of times is all greater than 1, it is determined that wind-induced swing continues to grow and issues instability early warning.The application can reduce the interference of camera vibration, arm end displacement and operation swing to wind-induced swing identification.

Description

Technical Field

[0001] This disclosure relates to the field of image recognition and processing technology, specifically to a method for early warning of crane dynamic instability based on multimodal visual wind-induced sway decoupling. Background Technology

[0002] Cranes are widely used in power construction, building installation, and equipment hoisting. During hoisting, the load is flexibly suspended by wire ropes, and is prone to lateral or combined swaying under the influence of gusts, continuous winds, and changes in wind direction. When the sway amplitude continues to increase, it may cause collisions of the load, load sway, and changes in the working radius, which can seriously affect the stability of the entire machine. Existing hoisting safety monitoring usually uses anemometers, torque limiters, boom angle sensors, and on-site personnel observation, and stops the machine based on wind speed thresholds, rated loads, or allowable working amplitudes. The above methods can reflect the environmental wind conditions or the overall load status of the machine, but wind speed and the actual sway of the load are also affected by factors such as the windward area, rope length, load shape, and working posture, making it difficult to determine whether the load has entered a continuous instability process based solely on wind speed. Some systems use inertial sensors or machine vision to measure the sway amplitude of the load, but the additional sensors have requirements for installation, power supply, and communication maintenance; visual monitoring is also susceptible to vibration of the camera mounting platform, boom end displacement, target obstruction, and changes in lighting. Meanwhile, crane slewing, hoisting, and luffing operations themselves can cause transient swaying, which existing methods often struggle to effectively distinguish from wind-induced swaying. Relying solely on single-moment sway amplitude or fixed thresholds for early warning can easily lead to false alarms during operation, or alarms may only be triggered when the sway amplitude approaches the safety boundary, making it difficult to promptly identify the dynamic instability risk resulting from the continuous growth of wind-induced swaying. Summary of the Invention

[0003] The purpose of this application is to provide a method for early warning of crane dynamic instability based on multimodal visual wind-induced sway decoupling, so as to solve the problems mentioned in the background art.

[0004] According to the first aspect of this application, a method for early warning of crane dynamic instability based on multimodal visual wind-induced sway decoupling is provided, comprising the following steps: stabilizing the crane operation video captured by the camera to eliminate false displacement of the image caused by vibration of the camera mounting platform; tracking the positions of the boom tip and the suspended object in the stabilized video, and using the coordinate difference between the position of the suspended object and the position of the boom tip as the relative sway displacement of the suspended object relative to the suspension point; identifying a continuous time period in which all motion mechanisms are in a non-working state and the duration is not less than the set minimum window duration, based on the operation signals of each motion mechanism output by the crane controller, and using this continuous time period as an operation silence window; within each operation silence window, skipping the set waiting time after the start of the window, extracting the sway amplitude within the skipped time period from the relative sway displacement as the wind-induced sway amplitude sample of that window; arranging the wind-induced sway amplitude samples corresponding to each operation silence window in chronological order, and calculating the amplitude growth rate between two adjacent wind-induced sway amplitude samples; when the amplitude growth rate is greater than 1 for a set number of consecutive times, determining that the wind-induced sway is in a continuous growth state and issuing an instability warning.

[0005] Preferably, the stabilization process includes: extracting reference feature points on a fixed rigid structural surface within the camera's field of view using a corner detection method; tracking the inter-frame displacement of the reference feature points in each subsequent frame using a sparse optical flow method; estimating the homography matrix from the current frame to the previous frame using a random sampling consistency method, constrained by the corresponding coordinates of the reference feature points in two adjacent frames; and applying an inverse transformation of the homography matrix to the current frame to obtain the corresponding frame in the stabilized video.

[0006] Preferably, the operation signals include the angular velocity of the slewing mechanism, the linear velocity of the hoisting mechanism, and the angular velocity of the luffing mechanism; when the absolute values ​​of the angular velocity of the slewing mechanism, the linear velocity of the hoisting mechanism, and the angular velocity of the luffing mechanism are all lower than their respective stop judgment thresholds, it is determined that all the moving mechanisms are in a non-working state; each of the stop judgment thresholds is determined based on the no-load creep speed of the corresponding moving mechanism, and the no-load creep speed is obtained from the crane technical parameter table or calibrated on-site by running it under no-load at the lowest gear.

[0007] Preferably, both the shortest window duration and the set waiting time are set to twice the natural swing period of the suspended object; the natural swing period is determined according to the effective suspension length of the rope at the hook based on the simple pendulum period relationship, and the effective suspension length is read from the crane torque limiter system or estimated by a visual method based on the pixel distance between the boom end and the hook combined with camera calibration parameters.

[0008] Preferably, the extraction of the swing amplitude within the skipped period includes: taking the modulus of the horizontal and vertical components of the relative swing displacement to obtain a time series of the synthesized swing amplitude; detecting each peak point in the time series of the synthesized swing amplitude; and taking the median of each peak point as the wind-induced swing amplitude sample of the window.

[0009] Preferably, the instability warning includes two levels: a first-level warning and a second-level warning, with the second-level warning having a higher priority than the first-level warning. The first-level warning is issued when the amplitude growth rate is greater than 1 for a set number of consecutive times and the current wind-induced sway amplitude sample is less than the product of the allowable sway amplitude and the first proportional coefficient. The second-level warning is issued when the current wind-induced sway amplitude sample reaches or exceeds the product of the allowable sway amplitude and the first proportional coefficient.

[0010] Preferably, the allowable swing amplitude is the difference between the maximum allowable working amplitude under the current working condition and the current actual working amplitude multiplied by a safety factor. The maximum allowable working amplitude and the current actual working amplitude are read from the crane torque limiter system, and the safety factor ranges from 0.5 to 0.8.

[0011] Preferably, when the amplitude growth rate is less than 1 for a set number of consecutive times, and the current wind-induced sway amplitude sample is lower than the product of the allowable sway amplitude and the second proportional coefficient, the issued first-level warning or second-level warning is released.

[0012] Preferably, before taking the median of each peak point, the method further includes calibrating the credibility of each peak point: calculating the single-cycle attenuation rate between two adjacent peak points; when the single-cycle attenuation rate is less than a set attenuation threshold, marking the wind-induced sway amplitude sample of the window as a low-credibility sample, and reducing the weight of the low-credibility sample when calculating the amplitude growth rate.

[0013] In a second aspect, this application also provides a crane dynamic instability early warning system based on multimodal visual wind-induced sway decoupling, comprising: a video stabilization module for stabilizing crane operation videos captured by a camera to eliminate false displacement of the image caused by vibration of the camera mounting platform; a relative sway displacement determination module for tracking the positions of the boom tip and the suspended load in the stabilized video, using the coordinate difference between the position of the suspended load and the position of the boom tip as the relative sway displacement of the suspended load relative to the suspension point; and an operation silence window recognition module for identifying, based on the operation signals of each action mechanism output by the crane controller, that all action mechanisms are in a non-working state for a considerable period of time. Within a set shortest window duration, a continuous time period is designated as the operation silence window. A wind-induced sway amplitude extraction module is used to, within each operation silence window, skip the set waiting time after the window starts and extract the sway amplitude within the skipped period from the relative sway displacement, as a wind-induced sway amplitude sample for that window. An instability warning module is used to arrange the wind-induced sway amplitude samples corresponding to each operation silence window in chronological order, calculate the amplitude growth rate between two adjacent wind-induced sway amplitude samples, and when the amplitude growth rate is greater than 1 for a set number of consecutive times, it is determined that the wind-induced sway is in a state of continuous growth and an instability warning is issued.

[0014] This application stabilizes crane operation videos and determines the relative swing displacement based on the coordinate difference between the load position and the boom tip position, reducing the impact of camera mounting platform vibration and boom deformation due to wind on swing amplitude measurement results. Based on this, it identifies the operation silence window according to the operation signals of the slewing, hoisting, and luffing mechanisms, and extracts the wind-induced swing amplitude after skipping the residual swing decay period after the operation ends, effectively distinguishing between transient swing caused by crane operation and swing caused by continuous wind action. By comparing the wind-induced swing amplitude and its growth rate corresponding to continuous operation silence windows, a continuous growth trend can be identified before the swing amplitude reaches the danger limit, reducing false alarms and warning lags caused by judging solely based on a single swing amplitude or a fixed wind speed threshold. Simultaneously, it determines the allowable swing amplitude by combining the current actual working amplitude, the maximum allowable working amplitude, and the safety factor, and implements graded warnings, ensuring that the warning results adapt to changes in boom posture and operating conditions, providing operators with sufficient time to take action such as stopping, lowering the hook, or reducing the boom.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0016] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 A schematic diagram of a crane dynamic instability early warning method based on multimodal vision wind-induced sway decoupling according to an exemplary embodiment of the present disclosure is shown. Figure 2 A flowchart illustrating the determination of window wind-induced sway amplitude samples according to an exemplary embodiment of the present disclosure is shown; Figure 3 A schematic diagram of a crane dynamic instability early warning system based on multimodal vision-induced sway decoupling is shown according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0020] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0021] The method provided in this disclosure is applicable to the safety monitoring of crane lifting operations at power construction sites. A truck crane or crawler crane is configured at the construction site for lifting power equipment such as transmission tower components, transformers, or cable reels. A visible light camera is mounted on the crane body or a nearby temporary support, covering the boom tip and the area of ​​the lifted object's movement. Large rigid structures such as transmission towers, concrete poles, or existing foundation piers are present within the camera's field of view. Preferably, it is recommended to mount the camera on the top of the crane body, with the field of view facing forward and downward, covering the boom tip and the lifted object area. It is also recommended to ensure that a portion of the crane body's own rigid structure (such as the top edge of the cab, the corner of the counterweight, etc.) is always visible within the field of view as a fixed reference point to improve the robustness of the stabilization process.

[0022] The crane is equipped with a programmable logic controller (PLC) or a controller area network (CLAN) bus interface, capable of outputting operation signals for each actuator. The video capture frame rate is, for example, no less than 25 frames per second, and the resolution is, for example, no less than 1920×1080 pixels; the operation signal sampling period is, for example, no greater than 100 milliseconds. The video signal and the operation signal are aligned via timestamps, and their time deviation does not exceed one video frame interval.

[0023] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A schematic diagram of a crane dynamic instability early warning method based on multimodal visual wind-induced sway decoupling according to an exemplary embodiment of the present disclosure is shown, including the following method steps: In step S101, the crane operation video captured by the camera is stabilized to eliminate false displacement of the image caused by the vibration of the camera mounting platform.

[0025] According to an embodiment of this disclosure, the camera mounting platform vibrates under wind load, causing all targets in the image to exhibit false displacement. Specifically, in the first frame of the video, a textured area is selected from the surface of a fixed rigid structure within the scene, and reference feature points are extracted using a corner detection method. Optional fixed rigid structures include the leg joints of transmission towers, the ends of tower crossarms, and the corners of cast-in-place foundations. The deformation of these structures under wind load is on the order of millimeters, negligible compared to the centimeter to decimeter displacement of the suspended load.

[0026] One corner detection method employed is the Shi-Tomasi corner detection method. This method calculates the gradient covariance matrix of each pixel's neighborhood in the grayscale image, and takes the smaller of the two eigenvalues ​​of this matrix as the corner response value. Pixels with response values ​​higher than a set threshold are identified as corners. The number of reference feature points is, for example, no less than eight. In each subsequent frame, a sparse optical flow method is used to track the inter-frame displacement of the reference feature points. For example, the Lucas-Kanade optical flow algorithm is used. This algorithm assumes that the motion of each pixel within the neighborhood of a feature point is consistent between adjacent frames, and solves for the inter-frame displacement vector of each feature point by minimizing the sum of squares of the grayscale differences within the neighborhood window.

[0027] For example, the input for corner detection is a single-channel grayscale image of the first frame of the video after grayscale conversion, with grayscale values ​​ranging from 0 to 255. The calculation process for Shi-Tomasi corner detection is as follows: for each pixel... Calculate its Gradient covariance matrix in the neighborhood: in , Each pixel is located at and The gray-level gradient in the direction (calculated using the Sobel operator). Take the matrix. Two eigenvalues , The smaller one is used as the corner response value. ,when Greater than the set threshold Points are identified as corners. The initial number of extracted reference feature points is set to 50, evenly distributed across the surface area of ​​the fixed rigid structure. The position of each feature point is specified in pixel coordinates. express, ,in This represents the number of valid feature points in the current frame. In each frame, if the number of valid feature points... If the number of corner points is less than 30, the corner points will be re-detected in the current frame to increase the number to 50.

[0028] In this embodiment, the homography matrix from the current frame to the previous frame is estimated using the corresponding coordinates of the reference feature points in two adjacent frames as constraints. ; for The homography matrix describes the projective transformation relationship caused by camera motion. The RANSAC method randomly selects a minimum number of matching pairs from feature point pairs to calculate candidate homography matrices, counts the number of interior points satisfying the reprojection error constraint, and after multiple iterations, selects the candidate matrix with the most interior points as the final estimation result. An application is then applied to the current frame. The inverse transformation corrects the image to a coordinate system consistent with the camera posture of the previous frame. The fixed rigid structure remains stationary in the corrected image, and the false displacement of the image caused by the vibration of the camera mounting platform is eliminated, resulting in a stable video.

[0029] For example, when RANSAC estimates the homography matrix, it randomly selects 4 pairs of matching feature points each time to calculate the candidate homography matrix. (4 pairs of points for calculation) The minimum requirement for a homography matrix, because It has 8 degrees of freedom, with 2 constraint equations provided for each pair of points. The reprojection error threshold is set to... Pixel, that is, for a pair of feature points ,like The point pair is then determined to be an interior point. The maximum number of RANSAC iterations is set to 2000. When the number of interior points in a candidate model exceeds 60% of the total number of feature point pairs and the absolute number of interior points is not less than 20, the iteration is terminated early and the model is output. If the required interior point ratio is still not met after all iterations, the candidate matrix with the most interior points is selected as the final estimation result, and a confidence flag indicating the estimation quality is returned to the system. When the interior point ratio is below 30%, the stabilization result of that frame is marked as low confidence, and the data in that frame is discarded or interpolated in subsequent relative sway displacement calculations.

[0030] Optionally, when some reference feature points are lost due to occlusion or motion blur, new corner points are detected in subsequent frames to maintain the number of reference feature points.

[0031] In step S102, the positions of the boom end and the suspended object are tracked in the stabilized video, and the coordinate difference between the position of the suspended object and the position of the boom end is used as the relative swing displacement of the suspended object relative to the suspension point.

[0032] In this embodiment, camera vibration has been eliminated in the stabilized video, and the motion of each target in the image corresponds to its actual motion in physical space. The boom bends and deforms under wind load, causing displacement at the boom tip, i.e., the suspension point of the load. This displacement is a translation of the suspension point as a whole, rather than the swing of the load itself. If this displacement is not separated, it will lead to an excessively large subsequent swing measurement.

[0033] In this process, two targets are tracked in the stabilized video: the boom tip and the load. For the boom tip, the outline of the boom pulley system is used as the tracking feature; for the load, the outline of the hook or balance beam is used as the tracking feature. Specifically, a pre-trained lightweight object detection model can be used to locate the center coordinates of these two targets in each frame, such as the YOLOv5s model. YOLOv5s is a single-stage object detection network whose backbone consists of multiple layers of deeply separable convolutions and cross-stage local connections. Each layer outputs feature maps at different scales, and the detection head predicts the bounding box coordinates and class probabilities of the targets at three scales. The model input is a 3-channel RGB image, normalized to 640×640 pixels; the model output includes the center coordinates, width, height, and class confidence of each detection box. The training data consists of crane operation video frames collected at a power construction site, with bounding boxes labeled for the hook and boom pulley system, categorized as "boom tip" and "load".

[0034] During training, the CIoU loss function is used to measure the cross-union ratio deviation between the predicted bounding box and the labeled bounding box, and a binary cross-entropy loss is combined to calculate the category and target confidence losses. The total loss is a weighted sum of the above terms. The optimizer uses stochastic gradient descent (SGD), with an initial learning rate of, for example, 0.01, gradually reduced to one percent of the initial value using a cosine annealing strategy. The number of training epochs is, for example, 100 to 200, and the batch size is, for example, 16. The center coordinates of the object detection bounding boxes in the model are the pixel positions of the arm tip and the suspended object in the current frame. The above object detection is a common application of general vision tools, used to provide the positional data required for subsequent processing.

[0035] For example, the training dataset collected crane operation videos from 10 different construction sites, with each site containing 2 hours of video footage. Keyframes were extracted at a sampling rate of 1 frame per second, resulting in approximately 72,000 frames. Bounding boxes were manually annotated for the boom and the suspended object in each frame using the Label Img tool, generating annotation files in PASCALVOC format. After annotation, all annotated samples were randomly divided into training, validation, and test sets in a 7:2:1 ratio. The training set contained approximately 50,400 frames, the validation set approximately 14,400 frames, and the test set approximately 7,200 frames, maintaining a relatively balanced distribution of samples across the different sets. An early stopping strategy was employed during training: training was stopped when the validation set loss did not decrease within 10 consecutive training epochs to prevent overfitting. The model weights with the highest average accuracy on the validation set were selected as the final deployment model.

[0036] According to embodiments of this disclosure, the arm end position is obtained in each frame. and the position of the suspended object All coordinates are in pixels. The coordinate difference between the position of the suspended object and the position of the boom tip is the relative swing displacement of the suspended object relative to the suspension point. in for The relative oscillation displacement vector at time t. and These are the horizontal and vertical components of the image, respectively. After this step, the displacement caused by the bending of the boom due to wind load is separated and removed, and the resulting relative sway displacement contains only two components: the transient sway caused by crane operation and the forced sway caused by the wind directly acting on the suspended load.

[0037] In step S103, based on the operation signals of each action mechanism output by the crane controller, a continuous time period in which all action mechanisms are in a non-working state and the duration is not shorter than the set minimum window duration is identified, and this continuous time period is used as the operation silent window.

[0038] Specifically, hoisting operations in power construction are not continuous; there are natural pauses between slewing, lifting, and luffing maneuvers, during which operators observe the load's position, await ground control signals, or adjust the hoisting posture. The operational silence window refers to the continuous period during which all crane mechanisms are inactive. Within this window, the crane outputs no operational stimuli, and all swaying of the load originates solely from the attenuated residual force of the previous operation and the continued effect of wind. Utilizing this characteristic, wind-induced sway components can be distinguished from the total sway without introducing additional wind field measurement equipment.

[0039] The identification of the silent operation window is based on the operation signals of each actuator output from the crane controller. Specifically, the operation signals include the angular velocity of the slewing mechanism. The unit is degrees per second; the linear velocity of the hoisting mechanism. The unit is meters per second; the angular velocity of the luffing mechanism. The unit is degrees per second. When the absolute values ​​of the above three operation signals are simultaneously lower than their respective stop judgment thresholds, all moving mechanisms are determined to be in a non-working state. The judgment condition is: in , , The shutdown thresholds are for the three mechanisms: slewing, hoisting, and luffing.

[0040] In this embodiment, each shutdown threshold is set to one-tenth of the no-load creep speed of the corresponding actuator. The no-load creep speed refers to the speed value of each actuator when running at its lowest stable speed under no-load conditions. This speed can be obtained from the technical parameter table provided by the crane manufacturer or calibrated on-site: Place each mechanism of the crane in its lowest gear and run it under no-load conditions. After the speed stabilizes, record the speed reading; this is the no-load creep speed. The shutdown threshold is calculated as follows: in The no-load creep angular velocity of the rotary mechanism is expressed in degrees per second. The unloaded creep linear velocity of the hoisting mechanism is expressed in meters per second. This represents the no-load creep angular velocity of the luffing mechanism, measured in degrees per second. The value of one-tenth is used to set the threshold well below any intentional operating speed, while remaining above the sensor's zero-point noise amplitude. For example, the no-load creep angular velocity of a 50-ton truck crane is 0.3 degrees per second, corresponding to a stop threshold of 0.03 degrees per second.

[0041] According to the embodiments of this disclosure, after determining that all actuators are in a non-working state, the duration of this state must not be shorter than the set minimum window duration. The minimum window duration is set to twice the natural swing period of the suspended object. The suspended object is suspended at the lower end of the rope, and its small-amplitude swing approximates a simple pendulum motion. The natural swing period is determined as follows: in The effective suspension length of the rope at the hook, in meters; Let be the acceleration due to gravity, and take . Effective overhang length The height can be obtained from the difference between the current lifting height and the boom tip height, read from the crane torque limiter system; alternatively, it can be calculated using a visual method based on the pixel distance between the boom tip and the hook in the stabilized video, combined with camera calibration parameters. As an example, when... hour, seconds, the shortest window duration is Seconds. Set to The consideration is that after the operation, the suspended object has an initial swing amplitude generated by the operation excitation. This swing amplitude decreases cycle by cycle during the free decay process. After two swing cycles, the residual swing amplitude of the operation has been greatly reduced. The swing in the latter part of the window is mainly wind-induced.

[0042] In this embodiment, the start and end times of the silent window are determined according to the following rules: window start time The moment when the absolute values ​​of the three operation signals simultaneously drop below their respective shutdown thresholds; the window ends. The moment when the absolute value of any one of the three operation signals exceeds the corresponding shutdown threshold again. Only when If the continuous time period is deemed a valid silent operation window, the process proceeds to the next step; otherwise, it is considered a short pause between operations and is not used.

[0043] In step S104, within each of the operation silent windows, the set waiting time after the start of the skip window is skipped, and the swing amplitude during the skipped period is extracted from the relative swing displacement as a wind-induced swing amplitude sample for that window.

[0044] Figure 2 A flowchart illustrating the window wind-induced sway amplitude sample determination according to an exemplary embodiment of the present disclosure is shown. In step S201, within each of the operation silent windows, the set waiting time after the window starts is skipped.

[0045] According to embodiments of this disclosure, within each operational silence window, the relative oscillation displacement comprises two components: operational residual decay oscillation and wind-induced forced oscillation. The operational residual component decays periodically after the start of the window, while the wind-induced component remains constant or even increases under continuous wind load. To extract the wind-induced oscillation component, a set waiting period after the start of each operational silence window is skipped. This set waiting period is also set to twice the natural oscillation period of the suspended load. .jump over Afterward, the residual swing amplitude has been significantly reduced; for a typical wire rope suspended load system, the damping ratio is usually between 0.02 and 0.08. After two complete cycles, the residual swing amplitude decays to about 60% to 85% of the initial value. At this point, the proportion of wind-induced components in the total swing has increased significantly.

[0046] In step S202, during the time period after skipping the set waiting time and before the window ends, the swing amplitude is extracted from the relative swing displacement. First, the time series of the synthesized swing amplitude is calculated, where the synthesized swing amplitude at each moment is the magnitude of the relative swing displacement vector: in and These are the horizontal and vertical components of the relative oscillating displacement.

[0047] Optionally, before detecting the peak, a moving average filter is applied to the time series of the synthesized swing amplitude, with a filter window width smaller than one natural swing period of the suspended object. The moving average filter replaces the value at each moment with the arithmetic mean of all sampled values ​​within a window centered at that moment. For example, a window width of 5 frames corresponds to approximately 0.2 seconds at 25 frames per second, which is much smaller than one swing period. This will not significantly change the amplitude and phase of the swing waveform, but it can smooth high-frequency fluctuations caused by image noise or tracking jitter.

[0048] In this embodiment, peak points are detected in the filtered synthetic swing amplitude sequence. The criterion for determining peak points is: at a certain moment... The combined oscillation amplitude at the location Also greater than the previous frame and the next frame The combined oscillation amplitude at the point, where This represents the video frame interval.

[0049] Optionally, before taking the median of each peak point, a calibration confidence assessment is performed on each peak point. Specifically, the single-cycle decay rate between two adjacent peak points is calculated, which is the composite oscillation amplitude of the later peak point divided by the composite oscillation amplitude of the earlier peak point. The single-cycle decay rate reflects the degree of change in the oscillation amplitude between adjacent cycles within the window: if the value is close to 1, it indicates that the amplitude tends to stabilize and the operational residue has been basically decayed; if the value is significantly lower than 1, it indicates that there is still a significant decay component remaining. When the single-cycle decay rate is less than a set decay threshold, the wind-induced oscillation amplitude sample of that window is marked as a low-confidence sample. The set decay threshold is, for example, 0.90, meaning that when the ratio between two adjacent peaks is less than 0.90, the operational residue component is considered not to have decayed sufficiently, and the calibration result has low confidence. In the subsequent calculation of the amplitude growth rate, the weight of the low-confidence sample is reduced. Specifically, the amplitude growth rate corresponding to the low-confidence sample is not included in the valid number of consecutive growth counts, or the amplitude growth rate corresponding to the low-confidence sample is multiplied by a reduction factor less than 1 before participating in the assessment.

[0050] In step S203, the median of each peak point is taken as the wind-induced sway amplitude sample for the silent operation window. The peaks at the beginning of the window may still contain residual operational components and thus appear higher; the median is not sensitive to such deviations. For example, if five peaks are detected within a silent window after skipping a set waiting time, with composite swing amplitudes of 38, 35, 33, 34, and 33 pixels respectively, then... Take the median of 34 pixels.

[0051] As construction progresses, the system sequentially collects wind-induced sway amplitude samples within each effective operating silent window, forming an amplitude sequence arranged in chronological order. ,in The calibration time for each window is the midpoint of the calibration time period within that window. The interval between two adjacent calibration times depends on the frequency of operational breaks during construction, typically ranging from 2 to 8 minutes in typical power hoisting operations.

[0052] It is understandable that the above calibration process utilizes the inherent intermittency of crane operation itself. It requires no artificially created downtime or specific operational sequences from operators. As long as a natural operational pause occurs during construction that meets the minimum window duration requirement, the system automatically collects samples. Specifically, the calibration time is defined as the time taken for each valid operational silent window. Skip window start Set the waiting time, remaining time period This is called the calibration time period, which is the actual time interval used to extract wind-induced sway amplitude samples. The calibration time of this window... Defined as the midpoint of the calibrated time period.

[0053] In step S105, the wind-induced sway amplitude samples corresponding to each of the operation silent windows are arranged in chronological order, and the amplitude growth rate between two adjacent wind-induced sway amplitude samples is calculated; when the amplitude growth rate is greater than 1 for a set number of consecutive times, it is determined that the wind-induced sway is in a state of continuous growth and an instability warning is issued.

[0054] According to embodiments of this disclosure, dynamic instability physically manifests as follows: the swaying energy continuously input by the wind to the suspended object exceeds the energy consumed by the system damping, causing the wind-induced sway amplitude to gradually increase. The changing trend of the wind-induced sway amplitude samples across multiple operational silent windows directly reflects the aforementioned energy balance. Based on the time-ordered sequence of wind-induced sway amplitude samples obtained in the preceding steps, the amplitude growth rate between two adjacent wind-induced sway amplitude samples is calculated: in and Samples of wind-induced sway amplitude calibrated for two adjacent silent operation windows; This indicates that the wind-induced sway is increasing, and the wind energy input exceeds the damping consumption; This indicates that the wind-induced sway amplitude is decreasing. As an example, if... Pixels pixels, then .

[0055] single Wind-induced oscillations may be caused by measurement fluctuations or short-term gusts, which are insufficient to determine a trend of increase. When the amplitude growth rate of a consecutive set number of observations is greater than 1, it is determined that the wind-induced oscillation is in a state of continuous increase and an instability warning is issued. For example, a set number of observations is 3. In power construction scenarios where the operation interval is typically 2 to 8 minutes, 3 observations cover a time span of approximately 6 to 24 minutes. If the value is too small, a single fluctuation may trigger the judgment; if the value is too large, the judgment period is too long and the warning is too late. In some embodiments, the set number of observations can be configured as an integer between 2 and 5.

[0056] According to some implementations, instability warnings are divided into two levels: Level 1 and Level 2, with Level 2 warnings having higher priority than Level 1 warnings. The determination logic involves the allowable swing amplitude. The two parameters are the first proportionality coefficient.

[0057] Among them, the allowable swing amplitude This value is obtained by multiplying the difference between the maximum permissible working radius and the current actual working radius under the current operating conditions by a safety factor. The maximum permissible working radius and the current actual working radius are read from the crane torque limiter system, and the safety factor... The value ranges from 0.5 to 0.8; for example, 0.7 reflects the margin maintained between the rated limit and the warning trigger. The calculation method is as follows: in For the maximum permissible working range, The current actual working range is measured in meters. The safety factor is dimensionless. The dimension of is meters. As an example, when , , hour, If the wind-induced sway amplitude sample In pixels, then The values ​​are then converted to pixel values ​​using the same camera calibration parameters before comparison. (Safety factor) The specific value can be preset according to the risk level of the construction site and the type of crane; the higher the risk level, the smaller the value. When the boom's working radius or load changes, the maximum allowable working radius and the current actual working radius output by the torque limiter system are updated accordingly, allowing for a certain swing amplitude. The corresponding calculations should be recalculated.

[0058] The first proportionality coefficient is, for example, 0.8, multiplied by the allowable swing amplitude. These constitute the criteria for determining the graded early warning system.

[0059] The criteria for determining a Level II warning are: the current wind-induced sway amplitude sample. Reaching or exceeding the allowable swing amplitude The product of the first proportionality coefficient, i.e. When this condition is met, a Level II warning will be issued directly, regardless of whether the amplitude growth rate sequence meets the continuous growth condition. The Level II warning prompts an immediate halt to hoisting operations and the implementation of measures such as lowering the hook or boom.

[0060] The criteria for a Level 1 warning are: the amplitude growth rate for the most recent consecutive set number of times is greater than 1, and the current wind-induced sway amplitude sample... Less than the allowable swing amplitude The product of the first proportionality coefficient, i.e. A Level 1 warning will be issued when both conditions are met simultaneously. A Level 1 warning will alert operators to changes in wind conditions, reduce operating speed, and shorten boom length.

[0061] In the above-mentioned judgment logic, the system first determines whether the conditions for a Level 2 warning are met: if they are met, a Level 2 warning is output directly, and the conditions for a Level 1 warning are no longer judged; if they are not met, the Level 1 warning conditions are judged. The Level 2 warning is based on the absolute level of the amplitude, while the Level 1 warning is based on the growth trend, with the additional constraint that the amplitude has not reached Level 2.

[0062] Optionally, when the amplitude growth rate for a consecutive set number of times is less than 1, and the current wind-induced oscillation amplitude sample Below the allowable swing amplitude When multiplied by the second proportional coefficient, the issued Level 1 or Level 2 warning is lifted, and normal monitoring is restored. For example, the second proportional coefficient is 0.3. The warning can only be lifted when the value of the warning decreases continuously. The second proportional coefficient is lower than the first proportional coefficient, so that the amplitude threshold for lifting the warning is significantly lower than the trigger threshold, avoiding repeated triggering and lifting near the amplitude boundary.

[0063] The warning information is displayed on the control room terminal to show the current warning level, and is simultaneously pushed to the mobile devices of ground command personnel via wireless communication. The current swing direction and amplitude of the suspended object are superimposed on the video screen, and the changing trend of several recent wind-induced swing amplitude samples is displayed in the form of a line graph.

[0064] The overall computational resources required by this method include projective transformation of image frames, frame-by-frame extraction of target center coordinates, peak statistics of swing amplitude, and division comparison operations. The computational load is concentrated in the video stabilization and target tracking stages. In some embodiments, the entire processing flow is deployed using an embedded edge computing device with a computing power of no less than 1 TOPS fixed-point operation per second, which can complete real-time processing at a video input rate of 25 frames per second without relying on a cloud server.

[0065] By applying the above method, the non-wind-induced components of camera mounting platform vibration and boom displacement are removed layer by layer from the composite motion of the suspended object observed by the camera. The intermittent pauses naturally present in crane operation are used as calibration windows for wind-induced sway. After skipping the residual attenuation segment of operation within the silent operation window, wind-induced sway amplitude samples are extracted. Then, the sway is determined to be in a state of continuous growth based on the growth rate sequence of wind-induced sway amplitude across multiple windows. Compared with the method of directly setting a fixed threshold for the total sway amplitude, this method changes the warning criterion from the absolute amplitude at the current moment to the trend growth across windows. It can identify the trend of energy accumulation when the wind-induced sway amplitude is still within the allowable range, giving operators time to respond. At the same time, transient sway caused by operation is no longer included in the wind-induced sway amplitude, reducing false alarms triggered by operation sway in power construction scenarios where cranes are frequently operated.

[0066] According to one aspect of this disclosure, a crane dynamic instability early warning system based on multimodal visual wind-induced sway decoupling is also provided. Figure 3 A block diagram of a crane dynamic instability early warning system based on multimodal vision-induced wind-induced sway decoupling, according to an exemplary embodiment of this disclosure, is shown. Figure 3 As shown, the system specifically includes: a video stabilization module 301, used to stabilize the crane operation video captured by the camera, eliminating false displacement of the image caused by vibration of the camera mounting platform; a relative swing displacement determination module 302, used to track the positions of the boom tip and the suspended object in the stabilized video, using the coordinate difference between the position of the suspended object and the position of the boom tip as the relative swing displacement of the suspended object relative to the suspension point; and an operation silence window recognition module 303, used to identify continuous periods when all motion mechanisms are in a non-working state and the duration is not less than the set minimum window duration, based on the operation signals of each motion mechanism output by the crane controller. The interval is defined as an operation silent window; the wind-induced sway amplitude extraction module 304 is used to extract the sway amplitude within the skipped period from the relative sway displacement within each operation silent window after skipping the set waiting time after the window starts, as the wind-induced sway amplitude sample of that window; the instability warning module 305 is used to arrange the wind-induced sway amplitude samples corresponding to each operation silent window in chronological order, calculate the amplitude growth rate between two adjacent wind-induced sway amplitude samples; when the amplitude growth rate is greater than 1 for a set number of consecutive times, it is determined that the wind-induced sway is in a continuous growth state and an instability warning is issued.

[0067] It should be noted that the working process of each module in the crane dynamic instability early warning system based on multimodal vision wind-induced sway decoupling described in this embodiment can refer to the working process of the crane dynamic instability early warning method based on multimodal vision wind-induced sway decoupling described in the above embodiment. The technical effect achieved is also the same as that of the crane dynamic instability early warning method based on multimodal vision wind-induced sway decoupling described in the above embodiment, and will not be repeated here.

[0068] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0069] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for early warning of crane dynamic instability based on multimodal vision-induced sway decoupling, characterized in that, The process includes the following steps: stabilizing the crane operation video captured by the camera to eliminate false displacement caused by vibration of the camera mounting platform; tracking the positions of the boom tip and the suspended object in the stabilized video, using the coordinate difference between the object's position and the boom tip position as the relative swing displacement of the object relative to the suspension point; identifying a continuous time period where all mechanisms are in a non-working state and the duration is not shorter than the set minimum window duration, based on the operation signals output by the crane controller; within each operation silence window, skipping the set waiting time after the window starts, extracting the swing amplitude within the skipped period from the relative swing displacement as the wind-induced swing amplitude sample for that window; arranging the wind-induced swing amplitude samples corresponding to each operation silence window in chronological order, calculating the amplitude growth rate between two adjacent wind-induced swing amplitude samples; when the amplitude growth rate is greater than 1 for a set number of consecutive times, determining that the wind-induced swing is in a continuously increasing state and issuing an instability warning.

2. The method according to claim 1, characterized in that, The stabilization process includes: extracting reference feature points on a fixed rigid structural surface within the camera's field of view using a corner detection method; tracking the inter-frame displacement of the reference feature points in each subsequent frame using a sparse optical flow method; estimating the homography matrix from the current frame to the previous frame using a random sampling consistency method, constrained by the corresponding coordinates of the reference feature points in two adjacent frames; and applying an inverse transformation of the homography matrix to the current frame to obtain the corresponding frame in the stabilized video.

3. The method according to claim 1, characterized in that, The operation signals include the angular velocity of the slewing mechanism, the linear velocity of the hoisting mechanism, and the angular velocity of the luffing mechanism. When the absolute values ​​of the angular velocities of the slewing mechanism, the linear velocity of the hoisting mechanism, and the angular velocity of the luffing mechanism are all lower than their respective stop judgment thresholds, it is determined that all the moving mechanisms are in a non-working state. Each stop judgment threshold is determined based on the no-load creep speed of the corresponding moving mechanism. The no-load creep speed is obtained from the crane's technical parameter table or calibrated on-site by running it under no-load conditions at the lowest gear.

4. The method according to claim 1, characterized in that, Both the shortest window duration and the set waiting time are set to twice the natural swing period of the suspended object; the natural swing period is determined according to the effective suspension length of the rope at the hook based on the relationship of a single pendulum period, and the effective suspension length is read from the crane torque limiter system or estimated by a visual method based on the pixel distance between the boom end and the hook combined with camera calibration parameters.

5. The method according to claim 1, characterized in that, The extraction of the swing amplitude within the skipped period includes: taking the modulus of the horizontal and vertical components of the relative swing displacement to obtain a time series of the synthesized swing amplitude; detecting each peak point in the time series of the synthesized swing amplitude; and taking the median of each peak point as the wind-induced swing amplitude sample of the window.

6. The method according to claim 1, characterized in that, The instability warning includes two levels: Level 1 and Level 2, with Level 2 having a higher priority than Level 1. When the amplitude growth rate is greater than 1 for a set number of consecutive times, and the current wind-induced sway amplitude sample is less than the product of the allowable sway amplitude and the first proportional coefficient, a first-level warning is issued; when the current wind-induced sway amplitude sample reaches or exceeds the product of the allowable sway amplitude and the first proportional coefficient, a second-level warning is issued.

7. The method according to claim 6, characterized in that, The allowable swing amplitude is the difference between the maximum allowable working amplitude under the current working condition and the current actual working amplitude, multiplied by a safety factor. The maximum allowable working amplitude and the current actual working amplitude are read from the crane torque limiter system, and the safety factor ranges from 0.5 to 0.

8.

8. The method according to claim 6, characterized in that, When the amplitude growth rate is less than 1 for a set number of consecutive times, and the current wind-induced sway amplitude sample is lower than the product of the allowable sway amplitude and the second proportional coefficient, the issued first-level warning or second-level warning is lifted.

9. The method according to claim 5, characterized in that, Before taking the median of each peak point, the method further includes calibrating the credibility of each peak point: calculating the single-cycle attenuation rate between two adjacent peak points; when the single-cycle attenuation rate is less than a set attenuation threshold, the wind-induced sway amplitude sample of the window is marked as a low credibility sample, and the weight of the low credibility sample is reduced when calculating the amplitude growth rate.

10. A crane dynamic instability early warning system based on multimodal vision-induced sway decoupling, characterized in that, include: The video stabilization module is used to stabilize the crane operation video captured by the camera and eliminate false displacement of the image caused by the vibration of the camera mounting platform; The relative swing displacement determination module is used to track the positions of the boom tip and the suspended object in the stabilized video, and use the coordinate difference between the position of the suspended object and the position of the boom tip as the relative swing displacement of the suspended object relative to the suspension point; the operation silence window recognition module is used to identify, based on the operation signals of each action mechanism output by the crane controller, a continuous time period in which all action mechanisms are in a non-working state and the duration is not less than the set minimum window duration, and use this continuous time period as the operation silence window; the wind-induced swing amplitude extraction module is used to, within each operation silence window, skip the set waiting time after the start of the window, and extract the swing amplitude within the skipped time period from the relative swing displacement as the wind-induced swing amplitude sample of that window; The instability warning module is used to arrange the wind-induced sway amplitude samples corresponding to each of the operation silent windows in chronological order, calculate the amplitude growth rate between two adjacent wind-induced sway amplitude samples, and determine that the wind-induced sway is in a state of continuous growth and issue an instability warning when the amplitude growth rate is greater than 1 for a set number of consecutive times.

Images