A double-channel vision-based load swing detection method for bridge cranes

Abstract

This invention discloses a method for detecting load sway in bridge cranes based on dual-channel vision. The invention employs an active optical filtering strategy, utilizing short exposure combined with infrared illumination and reflective markers to suppress ambient light interference. At the algorithm level, a dual-channel detection architecture is constructed: the high-speed channel uses a classic vision algorithm based on region of interest (ROI) prediction and sub-pixel localization to ensure real-time performance; the redundant channel uses a quantized lightweight deep learning network to ensure robustness under complex operating conditions. An adaptive fusion strategy based on confidence evaluation dynamically coordinates the outputs of the two channels, and the redundant channel is used to update the prior information of the high-speed channel. This invention achieves high-precision, high-real-time, and high-reliability load sway detection on an embedded edge computing platform, effectively solving the detection challenges caused by changes in lighting, occlusion, and strong interference sources in industrial environments.

Description

Technical Field

[0001] This invention relates to the field of crane automation control technology, specifically to a method for detecting load sway of a bridge crane based on dual-channel vision. Background Technology

[0002] Bridge cranes, as large material handling equipment, are widely used in industries such as metallurgy, machinery manufacturing, and port logistics. To achieve automated operation and safe operation of the cranes, it is essential to accurately detect the load's sway angle to implement closed-loop anti-sway control and precise positioning.

[0003] Traditional contact sensors (such as potentiometers and encoders) theoretically possess a certain level of accuracy, but in practical applications, they suffer from problems such as wear, signal jitter, complex installation, and susceptibility to vibration interference. While non-contact laser scanners offer higher accuracy, their high cost and sensitivity to ambient light and reflective surfaces hinder their widespread adoption.

[0004] In recent years, non-contact detection schemes based on monocular vision have attracted attention due to their low cost and ease of installation. However, in complex industrial environments, traditional image processing algorithms are prone to failure, leading to detection loss or large errors and insufficient robustness. While pure deep learning methods exhibit better robustness in target recognition, they often struggle to meet the real-time requirements of high frame rate and low latency for closed-loop anti-sway control of cranes on resource-constrained embedded edge computing devices. Therefore, there is an urgent need for a detection method that can guarantee both high real-time performance and strong robustness. Summary of the Invention

[0005] The purpose of this invention is to solve the problem that existing crane load sway detection methods struggle to achieve high precision, high real-time performance, and high robustness in complex industrial environments, and to provide a crane load sway detection method based on dual-channel vision.

[0006] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0007] A crane load sway detection method based on dual-channel vision includes the following steps:

[0008] Step S1, Image Acquisition: Using short exposure combined with an infrared light source, photograph the reflective markers installed on the crane lifting device to acquire the original image;

[0009] Step S2, High-speed detection channel primary detection: Construct a high-speed detection channel based on classic vision algorithms, predict the region of interest (ROI) of the current frame based on the motion information of the previous frame, perform image segmentation and feature extraction within the ROI, and calculate the preliminary sub-pixel centroid coordinates of the backlight reflection markers;

[0010] Step S3, Confidence Assessment and Branch Decision: Calculate the confidence score of the primary detection output of the high-speed detection channel and compare it with a preset threshold: when the confidence score is high, the primary detection result of the high-speed detection channel is determined to be reliable, and the preliminary sub-pixel centroid coordinates obtained in step S2 are directly output as the detection result of the current frame; otherwise, the primary detection result of the high-speed detection channel is determined to be unreliable, and proceed to step S4.

[0011] Step S4, Redundant Detection Channel Correction and Update: Construct a redundant detection channel based on a lightweight deep learning model to perform large-scale or global target detection on the image, obtain the corrected coordinates of the backlight reflection markers as the detection result of the current frame, and use the corrected coordinates to reset the ROI prior information of the high-speed detection channel.

[0012] Step S5, Coordinate Mapping and Output: Establish the mapping relationship between the zero point coordinates of the image and the rope length, convert the image pixel coordinates into physical space coordinates, and output the load swing angle or displacement information to the anti-sway controller.

[0013] This invention employs a master-slave dual-channel collaborative detection method. The master channel, based on a classic visual algorithm, is a high-speed detection channel that operates in most situations, ensuring the system's ultra-high real-time performance and low latency. The slave channel utilizes a highly robust deep learning redundant detection channel to correct biases and restore the state when the confidence level of the high-speed detection channel is insufficient, thus improving detection accuracy and reliability. Simultaneously, an active optical filtering method using short exposure combined with an infrared light source effectively suppresses ambient light interference.

[0014] Furthermore, the specific method for image acquisition in step S1 is as follows: A narrow-band high-power infrared lamp is used as the light source, facing the reflective marker (such as a marker covered with a reflective film to enhance the contrast of the target in the image). A monochrome industrial camera with the infrared cutoff filter removed is used, and the camera exposure time is set to the microsecond level for shooting. A short exposure strategy is adopted, and by controlling the camera exposure time, ambient background light and high-brightness interference sources are suppressed, retaining only the high-brightness features of the reflective marker.

[0015] The prediction method for the ROI in step S2 is as follows: The position of the reflective marker in the next frame is predicted using a uniform velocity model to generate the ROI. Simultaneously, the image segmentation and feature extraction process within the ROI is as follows: Adaptive binarization is performed using the Otsu method; isolated noise points are removed using erosion and dilation operations; valid contours are selected using geometric constraints (area, aspect ratio); and preliminary sub-pixel centroid coordinates of the reflective marker are calculated using image moments.

[0016] Among them, the confidence score in step S3 The calculation includes outline sharpness, shape integrity, and / or temporal consistency, and the specific formula is as follows:

[0017] ,

[0018] in, Represents the quality of the outline. Represents shape integrity. Represents the stability of key points. Represents consistency over time; These are weighting coefficients, and , , , The sum is 1 ( + + + =1).

[0019] Furthermore, the specific calculation methods for each sub-item of the confidence score are as follows:

[0020] Contour quality This is used to assess the level of interference from potential targets in an image: based on the number of valid contours detected in the current frame. With total outline quantity The ratio is determined by the following formula: ;

[0021] Shape integrity This is used to assess whether markers are occluded or missing: The calculation formula is as follows: ,in The area of ​​the region where the retroreflective marker is missing. The expected complete area of ​​the retroreflective marker;

[0022] Key point stability Used to evaluate the spatial positional jitter of the detection results: The calculation formula is as follows ,in For the number of key points, and The first The key points are located at their coordinates in the current frame and the previous frame. Normalization factor;

[0023] Time Consistency This is used to evaluate the smoothness of feature changes of the detected target over time: The calculation formula is as follows. ,in and These represent the image feature vectors of the current frame and the previous frame, respectively. This refers to the sensitivity parameter.

[0024] In step S4, the deep learning model uses an INT8 quantized YOLOv11n network model embedded in an edge computing device to reduce memory usage and accelerate inference. The specific method is as follows: A linear asymmetric quantization method is used to convert the model weights and activation values ​​from 32-bit floating-point numbers to 8-bit integers (INT8). The quantization formula is:

[0025] ,

[0026] in, It is a floating-point value. To quantize the step size, Zero point and For quantification range.

[0027] Furthermore, the method for establishing the mapping relationship between the image zero-point coordinates and the rope length in step S5 is as follows: Static zero-point samples are collected under different rope length conditions to establish the zero point in the image coordinate system. With rope length A nonlinear empirical mapping relationship is used to compensate for imaging distortion of industrial cameras at different depths. The expression is as follows:

[0028] ,

[0029] Where A, B, C, and D are the fitting parameters.

[0030] The present invention has the following advantages over the prior art:

[0031] This invention employs an architecture of "active optical filtering + master-slave dual-channel collaboration." At the hardware level, ambient light interference is effectively suppressed through short exposure and backlight reflection characteristics. At the algorithm level, an innovative hierarchical processing mechanism is designed: in most cases, only the high-speed classical algorithm channel with extremely low computational load is run, ensuring the system's ultra-high real-time performance and low latency; only when sudden changes in illumination or occlusion cause insufficient confidence in the high-speed channel is the highly robust deep learning redundant channel activated as needed for correction and state recovery. This mechanism perfectly balances speed and stability on resource-constrained embedded platforms, achieving both the high-frequency response required for closed-loop control and ensuring reliable all-weather operation under complex conditions. Attached Figure Description

[0032] Figure 1 This is a flowchart of the overall dual-channel detection algorithm used in the crane load sway detection method of the present invention;

[0033] Figure 2 shows the actual effect of the high-speed detection channel in anti-interference positioning and detection of reflective markers under complex industrial conditions with strong light interference (such as welding spatter and arc light) in Embodiment 2 of the present invention.

[0034] Figure 3 shows the movement trajectory of the detection coordinates within 5 hours under static accuracy testing in Embodiment 2 of the present invention;

[0035] Figure 4 is a graph showing the absolute error curves of the true and measured values ​​of each offset under the dynamic error test in Embodiment 2 of the present invention. Detailed Implementation

[0036] The technical solution of this patent will be further described in detail below with reference to specific embodiments.

[0037] The embodiments of this patent are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described in the drawings are exemplary and are only used to explain this patent, and should not be construed as limiting this patent.

[0038] See Figure 1 This embodiment provides a crane load sway detection method based on dual-channel vision. The hardware platform of this method includes an industrial camera mounted on the bottom of the crane trolley, an infrared light source used in conjunction, a reflective marker plate (i.e., a reflective marker) mounted on the lifting device, and an edge computing device. The specific implementation steps are as follows:

[0039] Step 1: Active optical filtering imaging.

[0040] The industrial camera removes the infrared cutoff filter to enhance its sensitivity to infrared light. A narrow-band, high-power infrared lamp is used as the light source, positioned directly towards the reflective marker on the hanger. The camera exposure time is set to the microsecond level. Due to the strong directional reflective properties of the reflective film, and the weak energy of ambient background light during short exposures, the marker appears bright white in the image, while the background is almost completely black. This hardware preprocessing significantly improves the signal-to-noise ratio.

[0041] Step 2: High-speed channel detection and confidence assessment.

[0042] The system first runs a high-speed detection channel. This channel, based on classic computer vision algorithms, uses the target position and velocity detected in the previous frame to predict the region of interest (ROI) where the target may appear in the current frame. Within the ROI, adaptive binarization, morphological denoising, and contour extraction are performed, and then sub-pixel-level centroid coordinates are calculated.

[0043] Subsequently, the system calculates the confidence level of the detection result in real time. The confidence function takes into account the clarity of the contour (such as edge gradient), the integrity of the shape (whether it is close to the standard geometry), and the temporal consistency of motion.

[0044] The specific formula is as follows:

[0045] ,

[0046] in, Represents the quality of the outline. Represents shape integrity. Represents the stability of key points. Represents consistency over time; These are weighting coefficients, and , , , The sum is 1 ( + + + =1).

[0047] The specific calculation methods for each of the above sub-indicators are as follows:

[0048] Contour quality This is used to assess the level of interference from potential targets in an image: based on the number of valid contours detected in the current frame. With total outline quantity The ratio is determined by the following formula: ;

[0049] Shape integrity This is used to assess whether markers are occluded or missing: The calculation formula is as follows: ,in The area of ​​the region where the retroreflective marker is missing. The expected complete area of ​​the retroreflective marker;

[0050] Key point stability Used to evaluate the spatial positional jitter of the detection results: The calculation formula is as follows ,in For the number of key points, and The first The key points are located at their coordinates in the current frame and the previous frame. Normalization factor;

[0051] Time Consistency This is used to evaluate the smoothness of feature changes of the detected target over time: The calculation formula is as follows. ,in and These represent the image feature vectors of the current frame and the previous frame, respectively. This refers to the sensitivity parameter.

[0052] Step 3: Adaptive confidence fusion and decision making.

[0053] The system calculates the confidence level. With preset threshold Make branch decisions:

[0054] 1. High confidence (tracking mode): If This indicates that the current detection results of the high-speed channel are accurate and reliable (e.g., stable lighting and no occlusion). In this mode, the system does not activate the deep learning model and directly uses the sub-pixel coordinates calculated by the high-speed channel as the final output. This mode results in minimal system computation and lowest latency, meeting real-time control requirements.

[0055] 2. Low confidence (recovery mode): If This indicates that the high-speed channel may have failed (e.g., due to a sudden change in illumination, occlusion of markers, or misidentification of background noise). In this case, the system triggers the redundant detection channel. This channel loads a quantized lightweight deep learning object detection model (such as INT8-quantized YOLOv11n) to perform a full-image or wide-range search of the current frame image to relocate the markers.

[0056] If the redundant channel detection is successful, its output coordinates are used as the final result for the current frame. At the same time, the ROI predictor of the high-speed channel is reinitialized using the detection results of the redundant channel, "pulling" the high-speed channel back to the correct position so that it can resume normal tracking in the next frame.

[0057] Step 4: Coordinate mapping and output.

[0058] Because variations in crane rope length cause changes in camera magnification, and industrial cameras suffer from distortion, this embodiment pre-calculates images with different rope lengths. Collect static zero-point samples to establish the zero point in the image coordinate system. With rope length The nonlinear empirical mapping relationship is used. During system operation, the pixel coordinates output in step three are converted into physical deviation values ​​relative to zero by combining the real-time rope length information. This physical deviation value is sent as a feedback signal to the anti-sway controller of the crane PLC to control the acceleration and deceleration of the trolley and the crane itself, thereby suppressing load swaying.

[0059] In summary, this invention employs a dual-channel master-slave collaborative mechanism, primarily using high-speed classical algorithms and supplementing with deep learning algorithms only when necessary for correction. This strategy avoids the computational overhead and latency accumulation associated with running a deep learning model for every frame, while leveraging the robustness of deep learning to address the issue of classical algorithms easily losing target information. This results in high-precision, low-latency, and highly reliable crane load sway detection on embedded devices.

[0060] Example 2

[0061] This embodiment provides a specific application test process and results for load sway detection of an industrial bridge crane. The test was conducted on a 20-ton industrial bridge crane. The edge computing device adopted an RK3576 platform with a quad-core Cortex-A72 and quad-core Cortex-A53 architecture. The industrial camera had a resolution of 1440×1080 and supported a maximum frame rate of 75.7 FPS.

[0062] Based on the steps in Example 1, the specific experimental process and test results are as follows:

[0063] 1. Image acquisition and processing under complex working conditions: In actual industrial sites, there is often interference from strong background light spots or high-intensity point light sources such as welding arc light. For example... Figure 2 The image shown is an image acquired and processed by the system under typical strong interference conditions (electric welding operation environment).

[0064] Figure 2 The high robustness of the present invention is fully demonstrated: a large area of ​​bright spatter and arc light (background noise) caused by welding exists in the lower right corner of the image. Following steps S1 and S2 of Example 1, the system first suppresses ambient diffuse light through short exposure and active infrared filtering; then, in the high-speed detection channel, fine hazy noise is removed using Otsu adaptive binarization and morphological (erosion and dilation) operations; finally, using geometric constraints (such as the area and aspect ratio of the target rectangle), the algorithm successfully removes the extremely bright non-target artifact in the lower right corner and accurately locates and extracts the reflective marker composed of two rhombuses (such as...) in the upper left corner. Figure 2 (As shown in the green ROI box and red outline), and accurately calculated and output the sub-pixel level centroid coordinates (the line connecting the centers of the markers and the center point): (425.26, 246.46). Simultaneously, the confidence level of the detection results was calculated. .

[0065] 2. Redundancy Collaboration Test of Deep Learning Channels: In continuous operation experiments, the confidence score of high-speed channel output is tested when short-term strong occlusion is artificially introduced or extreme changes in lighting occur. Below the preset threshold Time (preset threshold in this embodiment) (With a confidence level of 0.3), the system enters low-confidence recovery mode. Following steps S3 and S4, the system triggers the YOLOv11n redundant detection channel based on INT8 quantization within milliseconds. Experiments show that this lightweight network can not only quickly re-acquire targets over a wide range on edge platforms, but also successfully reset the ROI prior information of the high-speed channel using its corrected coordinates, preventing the propagation of errors to the anti-shake controller.

[0066] 3. Coordinate Mapping and Anti-sway Control: Following step S5, the system uses a pre-calibrated empirical mapping relationship (for different rope lengths) to perform coordinate mapping and anti-sway control. The zero-point data collected is fitted with equations. In this embodiment, the fitted equations are: x = -157188 ​​ / H + 624.76, y = 562074 / H + 305.35. The calculated pixel coordinates are converted into physical offsets (in millimeters) (at this time, the physical offsets are (160.203, 199.408)). This offset is used as feedback input in the crane's "open-loop input shaping + closed-loop PID" anti-sway control system.

[0067] 4. Test Data and Results To verify the comprehensive performance of the present invention, the following quantitative tests were conducted, all of which yielded excellent results: (1) Static accuracy test: such as Figure 3 As shown, during a continuous 5-hour static high-frequency sampling test, an aluminum plate was artificially placed to simulate strong reflection. The test results showed that the maximum fluctuation of the target centroid coordinates was only ±0.12 pixels, and the standard deviation was less than 0.04 pixels, proving that this method achieves extremely high sub-pixel level positioning accuracy.

[0068] (2) Dynamic Error Test: Keeping the suspension length of the hanger constant, move the load in a plane perpendicular to the optical axis. The measured physical offset is compared to the actual offset; the maximum error is controlled within 0.86mm, and the average absolute error is only 0.53mm. Figure 4 As shown.

[0069] (3) Real-time performance and robustness comparison test: The system was continuously operated for 12 hours in an industrial environment with drastic changes in lighting, strong reflections, and welding interference. The target effective detection rate (Availability Rate) of the system reached 100% throughout the entire test period. At the same time, since the system relies on the high-speed classical algorithm channel for most of the time, the average processing time per frame is only 0.008 seconds (8ms). The processing speed and environmental adaptability are significantly better than the traditional HSV color space method (27ms processing time, 65.4% efficiency) and the pure YOLOv5s deep learning method (45ms processing time).

Claims

1. A method for detecting load sway of a bridge crane based on dual-channel vision, characterized in that, Includes the following steps: Step S1, Image Acquisition: Using short exposure combined with an infrared light source, photograph the reflective markers installed on the crane lifting device to acquire the original image; Step S2, High-speed detection channel primary detection: Construct a high-speed detection channel based on classic vision algorithms, predict the ROI of the current frame based on the motion information of the previous frame, perform image segmentation and feature extraction within the ROI, and calculate the preliminary sub-pixel centroid coordinates of the backlight reflection markers; Step S3, Confidence Assessment and Branch Decision: Calculate the confidence score of the primary detection output of the high-speed detection channel and compare it with a preset threshold: when the confidence score is high, the primary detection result of the high-speed detection channel is determined to be reliable, and the preliminary sub-pixel centroid coordinates obtained in step S2 are directly output as the detection result of the current frame; otherwise, the primary detection result of the high-speed detection channel is determined to be unreliable, and proceed to step S4. Step S4, Redundant Detection Channel Correction and Update: Construct a redundant detection channel based on a lightweight deep learning model to perform large-scale or global target detection on the image, obtain the corrected coordinates of the backlight reflection markers as the detection result of the current frame, and use the corrected coordinates to reset the ROI prior information of the high-speed detection channel. Step S5, Coordinate Mapping and Output: Establish the mapping relationship between the zero point coordinates of the image and the rope length, convert the image pixel coordinates into physical space coordinates, and output the load swing angle or displacement information to the anti-sway controller.

2. The method for detecting load sway of a bridge crane according to claim 1, characterized in that, The specific method for image acquisition in step S1 is as follows: a narrow-band high-power infrared lamp is used as the light source, facing the backlight reflection marker. A monochrome industrial camera with the infrared cutoff filter removed is used, and the camera exposure time is set to the microsecond level for shooting.

3. The method for detecting load sway of a bridge crane according to claim 1, characterized in that, The method for predicting the ROI in step S2 is as follows: the position of the backlight reflection marker in the next frame is predicted using a uniform velocity model to generate the ROI.

4. The method for detecting load sway of a bridge crane according to claim 3, characterized in that, The high-speed detection channel in step S2 performs image segmentation and feature extraction as follows: adaptive binarization is performed using the Otsu method; isolated noise points are removed using erosion and dilation operations; and valid contours are selected using geometric constraints, including area and aspect ratio. The preliminary subpixel centroid coordinates of the retroreflection markers are calculated using image moments.

5. The method for detecting load sway of a bridge crane according to claim 1, characterized in that, Confidence score in step S3 The calculation formula is: ， in, Represents the quality of the outline. Represents shape integrity. Represents the stability of key points. Represents consistency over time; These are weighting coefficients, and , , , The sum is 1.

6. The method for detecting load sway of a bridge crane according to claim 5, characterized in that, The specific calculation methods for each sub-item of the confidence score are as follows: Contour quality Based on the number of valid contours detected in the current frame. With total outline quantity The ratio is determined by the following formula: ; Shape integrity The calculation formula is: ,in The area of ​​the region where the retroreflective marker is missing. The expected complete area of ​​the retroreflective marker; Key point stability The calculation formula is: ,in For the number of key points, and The first The key points are located at their coordinates in the current frame and the previous frame. Normalization factor; Time Consistency The calculation formula is: ,in and These represent the image feature vectors of the current frame and the previous frame, respectively. This refers to the sensitivity parameter.

7. The method for detecting load sway of a bridge crane according to claim 1, characterized in that, In step S4, the deep learning model uses an INT8 quantized YOLOv11n network model. The specific method is as follows: A linear asymmetric quantization method is used to convert the model weights and activation values ​​from 32-bit floating-point numbers to 8-bit integers. The quantization formula is: ， in, It is a floating-point value. To quantize the step size, Zero point and For quantification range.

8. The method for detecting load sway of a bridge crane according to claim 1, characterized in that, The method for establishing the mapping relationship between the image zero-point coordinates and the rope length in step S5 is as follows: Static zero-point samples are collected under different rope length conditions to establish the zero point in the image coordinate system. With rope length Nonlinear empirical mapping relationship: ， Where A, B, C, and D are the fitting parameters.

Images