Vision-assisted method and device for aligning and correcting the load of a double-girder tire gantry crane

By separating the visual detection space and constructing an anchor point constraint network, the problems of poor stability and low accuracy caused by swaying and background interference in visual-assisted load alignment are solved, and efficient and accurate load alignment correction is achieved.

CN123105461APending Publication Date: 2026-11-17SHANDONG MINGDAO HEAVY IND MACHINERY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611005475.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-07
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing visual-assisted lifting and alignment technologies are affected by swaying, occlusion, changes in lighting, and background interference during lifting, moving, and alignment, resulting in poor stability of visual detection anchor points, low accuracy of deviation recognition, and lag in alignment correction response.

Method used

The visual detection space of the lifting operation area is obtained by a visual sensor and separated into foreground and background spaces. Spatial anchor points with high stability are extracted, an anchor point constraint network is constructed, and image deviation is analyzed in real time to generate correction control commands.

Benefits of technology

It improves the accuracy and efficiency of load alignment, reduces the impact of swaying and background interference on visual inspection, and achieves high-precision automatic deviation control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123105461A_ABST
    Figure CN123105461A_ABST
Patent Text Reader

Abstract

This invention discloses a vision-assisted method and device for aligning and correcting the load of a double-girder tire-mounted gantry crane, relating to the field of crane control technology. The method includes: acquiring the visual detection space of the load, separating it into foreground and background visual detection spaces; acquiring stability anchor points in the foreground and background spaces respectively; constructing a constraint network for the foreground and background anchor points; acquiring real-time visual detection images; performing deviation-assisted analysis based on the anchor point constraint network to obtain a multi-dimensional deviation vector and output correction control commands. This invention solves the technical problems of poor stability of visual detection anchor points, low deviation recognition accuracy, and delayed alignment correction response caused by swinging, occlusion, changes in lighting, and background interference during lifting, movement, and alignment of the load. It achieves the technical effect of enhancing the spatiotemporal constraint capability of visual features under dynamic working conditions by constructing a dual-level stability anchor point and constraint network for the foreground and background, thereby improving the anti-interference capability, control accuracy, and operational safety of the load alignment correction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of crane control technology, specifically to a vision-assisted method and device for aligning and correcting the load of a double-girder tire gantry crane. Background Technology

[0002] Double-girder tire-mounted gantry cranes are indispensable heavy-duty loading and unloading equipment in ports, railway freight yards, and large logistics hubs. Their lifting and positioning accuracy directly determines operational efficiency and safety. In actual operation, the crane needs to precisely move the hoisted containers or bulk cargo to the designated position. However, due to the combined movements of the trolley travel, luffing, and hoisting mechanisms, coupled with wind loads, inertial forces, and the flexible coupling effect of the wire ropes, the hoisted load often experiences complex spatial swaying and deflection, leading to positioning difficulties. Traditional manual visual-assisted positioning methods heavily rely on operator experience, resulting in high labor intensity, low efficiency, and difficulty in meeting the demands of high-frequency, high-precision automated operations. In recent years, automatic positioning technology based on vision sensors has been initially applied. By collecting image information of the hoisted load and its surrounding environment, feature matching and position calculation are performed to provide a reference for correction control. However, existing vision solutions generally have significant drawbacks: on the one hand, the unstable swing of the suspended load itself causes the foreground feature points to change constantly, which can easily lead to feature loss or mismatch; on the other hand, the complex and variable background of the work site, the interference factors such as lighting conditions, ground texture and surrounding equipment further reduce the robustness of visual positioning, seriously affecting the safe positioning of the suspended load and the efficiency of the operation. Summary of the Invention

[0003] This application provides a vision-assisted method and device for aligning and correcting the lifting load of a double-girder tire gantry crane, which solves the technical problems of poor stability of visual detection anchor points, low deviation recognition accuracy, and delayed alignment and correction response caused by the influence of swing, obstruction, changes in lighting and background interference on the lifting, moving and alignment of the load.

[0004] The first aspect of this application provides a vision-assisted method for aligning and correcting the load of a double-girder tire gantry crane, the method comprising: A visual inspection space for the load lifted by a double-girder tire-mounted gantry crane is obtained, and the visual inspection space is separated into a foreground visual inspection space and a background visual inspection space. Multiple foreground space stability anchor points and multiple background space stability anchor points based on the foreground visual inspection space are obtained. An anchor point constraint network is constructed for the multiple foreground space stability anchor points and the multiple background space stability anchor points. A real-time visual inspection image of the load lifted by the double-girder tire-mounted gantry crane is obtained, and image deviation auxiliary analysis is performed on the real-time visual inspection image according to the anchor point constraint network to obtain a multi-dimensional deviation vector. Based on the multi-dimensional deviation vector, a correction control command for the load control terminal is output.

[0005] A second aspect of this application provides a vision-assisted double-girder tire gantry crane load alignment and correction device, the device comprising: Visual space separation component: acquires the visual detection space of the load being lifted by the double-girder tire-mounted gantry crane, and separates the visual detection space into a foreground visual detection space and a background visual detection space; Spatial anchor point acquisition component: acquires multiple foreground spatial stability anchor points based on the foreground visual detection space and multiple background spatial stability anchor points based on the background visual detection space; Anchor point constraint construction component: constructs an anchor point constraint network for the multiple foreground spatial stability anchor points and the multiple background spatial stability anchor points; Deviation correction control component: acquires real-time visual detection images of the load being lifted by the double-girder tire-mounted gantry crane, performs image deviation auxiliary analysis on the real-time visual detection images according to the anchor point constraint network, obtains a multi-dimensional deviation vector, and outputs deviation correction control commands for the load control terminal based on the multi-dimensional deviation vector.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages: First, visual information of the lifting operation area of ​​the double-girder tire-mounted gantry crane is acquired, and the foreground region related to the lifting motion and the relatively stable background region are distinguished. Then, highly stable spatial anchor points are extracted from both the foreground and background regions, and these anchor points are used to establish constraint relationships between the foreground and background. Next, during actual crane operation, the real-time acquired images of the lifted load are matched and analyzed with the anchor point constraint network to determine the deviation of the lifted load in the swing, rotation, horizontal position, and height directions, forming a multi-dimensional deviation vector. Based on this, a correction control command is generated and sent to the lifting load control terminal to achieve assisted alignment and automatic correction of the lifted load. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 A schematic diagram of the visual-assisted double-beam tire gantry crane lifting alignment and correction method provided in the embodiments of this application.

[0009] Figure 2 A schematic diagram of the structure of a vision-assisted double-beam tire gantry crane lifting alignment and correction device provided in the embodiments of this application.

[0010] Figure 3This is a schematic diagram illustrating the training convergence verification of the visual separation network provided in an embodiment of this application.

[0011] Figure 4 This is a schematic diagram comparing the stability of anchor points under different working conditions, provided for embodiments of this application.

[0012] Explanation of reference numerals in the attached figures: Visual space separation component 11, spatial anchor point acquisition component 12, anchor point constraint construction component 13, and deviation control component 14. Detailed Implementation

[0013] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0014] Example 1, as Figure 1 As shown, this application provides a vision-assisted method for aligning and correcting the load of a double-girder tire gantry crane. The method includes: The visual inspection space of the load lifted by the double-girder tire gantry crane is obtained, and the visual inspection space is separated to obtain the foreground visual inspection space and the background visual inspection space.

[0015] In this embodiment, a vision sensor installed on a double-girder tire gantry crane is used to acquire images of the lifting operation area. The target area in the image related to the alignment and correction of the lifting load is used as the visual detection space. This visual detection space includes at least the lifting device, the four wire ropes connecting the lifting device and the lifting load, the four corners of the lifting load, the Twist Lock head, the container corner fittings, the outer contour of the lifting load, the edge of the double-girder structure, and ground reference marks. The lifting device, the four wire ropes, the four corners of the lifting load, the Twist Lock head, the container corner fittings, and the lifting load contour are used to reflect the real-time position, attitude, swing state, and rotation state of the lifting load body and its connecting structure in the image. The edge of the double-girder structure and the ground reference marks are used to provide a relatively fixed spatial reference. After acquiring the visual inspection space, foreground and background separation processing is performed. This separation can be based on a pre-trained visual separation module, thus defining the foreground visual inspection space as the position of the lifting equipment, wire rope, four corners of the load, Twist Lock, container corner fittings, and the outline of the load changes with the movement, swinging, or rotation of the load. The background visual inspection space is defined as the edges of the double-beam structure, ground reference marks, and other fixed environmental reference areas that remain stable or have minimal changes relative to the lifting operation. Through this processing, the foreground visual inspection space can be used to characterize the real-time motion state of the load, and the background visual inspection space can be used to provide a stable spatial reference, thereby providing basic data for the subsequent extraction of foreground and background space stability anchor points and load deviation analysis.

[0016] Furthermore, the visual detection space is separated into a foreground visual detection space and a background visual detection space, the method including: The original visual detection sample space data of the crane's lifting area is acquired using a vision sensor deployed on a double-girder tire gantry crane. A temporal rolling window is used to obtain the stability index of each visual object in the original visual detection sample space data based on consecutive frames. The stability index is normalized to obtain a stability training label. The original visual detection sample space data is input into a visual separation network. The stability training label is used as a supervisory constraint to construct a supervised loss function. The supervised loss function is used to perform gradient backpropagation on the visual separation network, iteratively updating the weight parameters of the visual separation network to obtain a trained visual separation module. The visual detection space is separated according to the visual separation module to obtain a foreground visual detection space and a background visual detection space.

[0017] Preferably, when acquiring and separating the visual inspection space, a set of binocular vision sensors is fixedly installed under the trolley frame of the double-girder tire gantry crane. The field of view of the binocular vision sensors covers the lifting device, the four wire ropes, the four corners of the load, the Twist Lock head, the container corner fittings, the outline of the load, the edge of the double-girder structure, and the ground reference markers. RGB images and depth images of the load area are continuously acquired at a preset acquisition frequency. The RGB image, depth image, and image acquisition timestamp at the same moment are combined to form a frame of original visual inspection sample. Multiple frames of original visual inspection samples acquired continuously are arranged in chronological order to form original visual inspection sample spatial data. Subsequently, using a preset number of frames as a time-series rolling window, such as 16 frames, multiple consecutive frames of images are sequentially extracted from the original visual inspection sample spatial data. Within each time-series rolling window, object numbers are established for the lifting device, the four wire ropes, the four corners of the load, the Twist Lock head, the container corner fittings, the outline of the load, the edge of the double-girder structure, and the ground reference markers. The position, scale, pose, occlusion, and features of the same object number across multiple consecutive frames are statistically analyzed, and a stability index for each visual object within the temporal scrolling window is calculated through weighted fusion. The obtained stability index is then normalized using a minimum-maximum value method, mapping it uniformly to a value range of 0 to 1. This normalized stability index is used as a stability training label, where a stability training label close to 0 indicates that the visual object belongs to a foreground region that changes with the movement of the suspended weight, while a stability training label close to 1 indicates that the visual object belongs to a relatively fixed background region. The original visual detection sample spatial data is then input into a visual separation network consisting of an image encoding layer, a temporal feature fusion layer, and a pixel classification decoding layer. The image encoding layer extracts edge, texture, contour, and depth features from each frame; the temporal feature fusion layer extracts motion change features of the same visual object across 16 consecutive frames; and the pixel classification decoding layer outputs the temporal stability probability for each pixel unit. When training the visual separation network, stability training labels are assigned to the pixel regions covered by the corresponding visual objects. The temporal stability probability output by the pixel classification decoding layer is calculated with the stability training labels to construct a supervised loss function. Then, gradient backpropagation is performed on the visual separation network according to the supervised loss function, and the weight parameters of the image encoding layer, temporal feature fusion layer and pixel classification decoding layer are updated layer by layer until the decrease of the supervised loss function in multiple rounds of training is less than the preset convergence threshold, and the trained visual separation module is obtained.In practical use, the real-time visual detection space of the suspended load, acquired by the binocular vision sensor, is input into the visual separation module. The visual separation module outputs the temporal stability probability of each pixel unit. Pixel units with a temporal stability probability less than or equal to a preset temporal stability threshold are classified as the foreground visual detection space, while those with a temporal stability probability greater than the preset threshold are classified as the background visual detection space. This results in a foreground visual detection space characterizing the motion state of the suspended load and a background visual detection space providing a fixed reference benchmark. Through this processing, the motion region related to the suspended load is stably distinguished from the fixed environmental reference region at the pixel level, reducing the impact of suspended load swaying, partial occlusion, and background texture interference on the visual detection results, and improving the accuracy of subsequent anchor point extraction and suspended load deviation analysis.

[0018] Furthermore, the method for obtaining the stability index of each visual object in the original visual detection sample spatial data based on consecutive frames using a temporal rolling window includes: A temporal rolling window is used to acquire N consecutive frames of visual images of the original visual detection sample spatial data; inter-frame correlation analysis is performed on the N consecutive frames of visual images to obtain the continuous frame tracking trajectory corresponding to each visual object; the position change rate, scale change rate, pose change rate, occlusion persistence rate, and feature preservation rate of each visual object are calculated based on the continuous frame tracking trajectory; the position change rate, scale change rate, pose change rate, occlusion persistence rate, and feature preservation rate are weighted and averaged to obtain the stability index.

[0019] Optionally, when using a temporal rolling window to obtain the stability index, the original visual detection sample spatial data is first arranged into a continuous frame sequence according to the acquisition time order, and the length and sliding step of the temporal rolling window are set, such as a length of 16 frames and a sliding step of 1 frame. For any temporal rolling window, the continuous frame visual images within the window are read sequentially, and target detection is performed on the lifting device, four wire ropes, four corners of the load, Twist Lock head, container corner fittings, load outline, double beam structure edge, and ground reference mark in each frame visual image, to obtain the target box, outline region, center point coordinates, outline area, principal direction angle, visible area, and number of corner features for each visual object in each frame. Subsequently, using the visual object detected in the first frame as the initial object, the detection results of the same visual object in subsequent frames are correlated according to the matching rules of consistent object category, minimum center point distance and maximum contour overlap area, forming a continuous frame tracking trajectory corresponding to each visual object. The tracking state of the i-th visual object in the t-th frame is denoted as Pi,t=(xi,t, yi,t, Ai,t, θi,t, Vi,t, Fi,t), where xi,t and yi,t represent the center point coordinates, Ai,t represents the contour area, θi,t represents the principal direction angle, Vi,t represents the visible area, and Fi,t represents the number of corner features. Subsequently, based on the continuous frame tracking trajectory, the position change rate, scale change rate, pose change rate, occlusion persistence rate, and feature preservation rate of the visual object are calculated. Among them, the position change rate is the ratio of the average displacement of the center point of consecutive adjacent frames to the length of the image diagonal; the scale change rate is the ratio of the average absolute value of the difference in contour area of ​​consecutive adjacent frames to the average contour area; the pose change rate is the ratio of the average absolute value of the difference in principal direction angle of consecutive adjacent frames to 180°; the occlusion persistence rate is the ratio of the number of frames in which the visible area is less than 60% of the initial visible area of ​​the visual object to the total number of frames in the window; and the feature preservation rate is the ratio of the number of corner features preserved in consecutive adjacent frames to the initial number of corner features. After completing the above calculations, the position change rate, scale change rate, pose change rate, and occlusion persistence rate are truncated and normalized to limit their values ​​to between 0 and 1, and the feature preservation rate is also limited to between 0 and 1. Then, a weighted average is performed according to preset weights to obtain the stability index of the visual object within the current temporal scrolling window. This stability index satisfies: S = wp × (1-Rp) + ws × (1-Rs) + wθ × (1-Rθ) + wo × (1-Ro) + wf × Rf, where S represents the stability index; wp, ws, wθ, wo, and wf are preset weights for the position change rate, scale change rate, pose change rate, and occlusion persistence rate, and the sum of the weights is 1; Rp represents the position change rate; Rs represents the scale change rate; Rθ represents the pose change rate; Ro represents the occlusion persistence rate; and Rf represents the feature preservation rate.Therefore, when the position, scale, pose, and occlusion of a visual object change little in consecutive frames and its corner features remain high, its stability index is close to 1; when the visual object undergoes significant movement, swinging, rotation, occlusion, or feature loss due to the load, its stability index is close to 0. Through this process, the moving objects related to the load and the fixed reference objects can be quantitatively distinguished at the temporal stability level, providing an accurate basis for subsequent generation of stability training labels and training of the visual separation network.

[0020] like Figure 4 As shown in the figure, the average stability index of different types of anchor points is displayed under various working conditions such as static alignment, small swing, strong light interference, partial occlusion, and complex background. As can be seen from the figure, the foreground stable anchor points and the background stable anchor points maintain high stability under various working conditions, which is significantly better than ordinary corner anchor points. Among them, the background stable anchor points have the highest overall stability and can provide a reliable fixed spatial reference. The foreground stable anchor points still have good continuity under swing and occlusion conditions, indicating that the extracted anchor points can effectively reduce the impact of load movement and environmental interference on deviation analysis.

[0021] Furthermore, the method for iteratively training the visual separation network also includes: The stability change sequence is calculated based on the changing trend of the stability index corresponding to each visual object in consecutive frames; a temporal consistency constraint is constructed according to the stability change sequence, which is used to constrain the stability index change of the same visual object to remain continuous and consistent in consecutive frames; a temporal consistency constraint loss function is constructed based on the temporal consistency constraint, and gradient backpropagation is performed on the visual separation network using the supervised loss function and the temporal consistency constraint loss function.

[0022] Optionally, the visual separation network adopts a network structure in which an image coding layer, a temporal feature fusion layer, and a pixel classification decoding layer are connected sequentially. The image coding layer is used to extract spatial features from each frame of the visual image, including a first convolutional unit, a second convolutional unit, and a third convolutional unit. In one feasible approach, the kernel size of the first convolutional unit can be set to 3×3, the number of output channels can be set to 32, and the stride can be set to 1; the kernel size of the second convolutional unit can be set to 3×3, the number of output channels can be set to 64, and the stride can be set to 2; the kernel size of the third convolutional unit can be set to 3×3, the number of output channels can be set to 128, and the stride can be set to 2. Each convolutional unit is followed by a batch normalization layer and a ReLU activation function to extract the edge, texture, contour, and depth variation features of the lifting device, wire rope, lifting contour, double beam structure edge, and ground reference marker. The temporal feature fusion layer is used for… To fuse motion change features between consecutive frames, a single-layer ConvLSTM structure can be used. The number of hidden state channels can be set to 128, the convolutional kernel size can be set to 3×3, and the input sequence length can be set to 16 frames. This is used to learn the position changes, scale changes, pose changes, occlusion changes, and feature preservation changes of the same visual object in consecutive frames. The pixel classification decoding layer is used to restore the fused temporal features to the spatial resolution corresponding to the input image. It includes a first upsampling unit, a second upsampling unit, and a classification output unit. The first upsampling unit enlarges the feature map by 2 times and adjusts the number of output channels to 64. The second upsampling unit further enlarges the feature map by 2 times and adjusts the number of output channels to 32. The classification output unit uses a 1×1 convolution to adjust the number of channels to 1 and outputs the predicted temporal stability probability of each pixel unit through the Sigmoid function. The value of this predicted temporal stability probability ranges from 0 to 1.

[0023] Within each time-series rolling window, the stability index corresponding to each visual object is read sequentially according to its visual object number. The stability indices of the same visual object in adjacent time-series rolling windows are arranged in chronological order to form a stability change sequence for that visual object. For the i-th visual object, its stability index in the k-th time-series rolling window is denoted as Si,k. The stability change sequence of this visual object is then represented as Qi={Si,1,Si,2,……,Si,K}, where K represents the number of time-series rolling windows involved in the training. Subsequently, the stability index difference ΔSi,k=Si,k-Si,k-1 between two adjacent time-series rolling windows is calculated, and multiple consecutive stability index differences are arranged in chronological order to obtain a stability change trend sequence. If the same visual object does not experience significant occlusion, decoupling, target loss, or sudden large swings in consecutive frames, its stability index should change continuously over time, rather than abruptly between adjacent windows. Therefore, the absolute value of the difference in stability index between adjacent windows is less than or equal to the preset stability change threshold Ts as a temporal consistency constraint. This stability change threshold can be set to 0.12, that is, when |Si,k-Si,k-1|≤Ts, the visual object is determined to have stable temporal continuity within adjacent temporal windows.

[0024] Next, the predicted temporal stability probability of the i-th visual object output by the visual separation network in the k-th temporal rolling window is denoted as Pi,k, and the difference in predicted temporal stability probabilities between adjacent windows is denoted as Pi,k-Pi,k-1. A temporal consistency constraint loss function is constructed based on the temporal consistency constraint condition. This function penalizes abrupt changes in the predicted temporal stability probability of the same visual object in adjacent temporal rolling windows. Its calculation method is as follows: when |Si,k-Si,k-1|≤Ts, calculate (Pi,k-Pi,k-1). 2The difference between the stability training label and the predicted temporal stability probability output by the visual separation network is used as the supervision loss function. The supervision loss function and the temporal consistency constraint loss function are fused according to a preset ratio to obtain the total training loss function L = Lsup + 0.35Ltime, where Lsup represents the supervision loss function and Ltime represents the temporal consistency constraint loss function. During training, the input image size is uniformly adjusted to 640×480, the batch size is set to 4, the initial learning rate is set to 0.001, the Adam optimizer is used, and the training epochs are set to 80. When the total training loss decreases by less than 0.001 for 5 consecutive epochs, the visual separation network is considered to have reached convergence, and the converged network parameters are solidified into the visual separation module. Finally, using the total training loss function as the network optimization objective, gradient backpropagation is performed on the visual separation network to sequentially update the weight parameters of the image encoding layer, temporal feature fusion layer, and pixel classification decoding layer. This ensures that the visual separation network learns the differences between foreground and background regions while maintaining the temporal continuity of the separation results for the same visual object in consecutive frames. This process reduces foreground / background misseparation caused by instantaneous illumination changes, local occlusion, and single-frame detection errors, thereby improving the stability and continuity of spatial separation results in visual detection.

[0025] like Figure 3 As shown in the figure, the loss of the visual separation network changes during 80 training rounds. As can be seen from the figure, the supervised training loss, validation loss, and temporal consistency loss all gradually decrease with the increase of training rounds and tend to stabilize in the later stage of training. This indicates that the visual separation network can gradually learn the difference features between the foreground visual detection space and the background visual detection space. At the same time, the validation loss and the training loss maintain a similar decreasing trend, indicating that the model does not have obvious overfitting and has good convergence and generalization ability.

[0026] Furthermore, the method for separating the visual detection space into a foreground visual detection space and a background visual detection space based on the visual separation module includes: The visual separation module calculates the temporal distribution weights of each pixel unit in the visual detection space and outputs a temporal stability probability distribution. Based on the temporal stability probability distribution, pixel units less than or equal to a preset temporal stability threshold are identified and output as the foreground visual detection space. Based on the visual detection space and the foreground visual detection space, a background visual detection space is constructed.

[0027] Optionally, when separating the visual detection space according to the visual separation module, the visual detection images of the suspended load acquired in real time by the binocular vision sensor are first uniformly adjusted to an input size of 640×480, and the RGB image and depth image corresponding to the same acquisition time are channel-stitched to form the visual detection space input data. After the visual detection space is input into the trained visual separation module, the image coding layer extracts the edge features, texture features, contour features and depth features corresponding to each pixel unit. The temporal feature fusion layer combines the changes of the neighborhood of the same pixel in the current frame and the previous 15 frames to calculate the temporal distribution weight of each pixel unit in the continuous frames. This temporal distribution weight is used to represent the stability of the pixel unit in the continuous image frames. Subsequently, the pixel classification decoding layer of the visual separation module outputs the temporal stability probability corresponding to each pixel unit according to the temporal distribution weight, and the temporal stability probability distribution map is composed of the temporal stability probabilities of all pixel units. The smaller the temporal stability probability, the more likely the pixel unit belongs to the region that changes with the movement, swinging or rotation of the load. The larger the temporal stability probability, the more likely the pixel unit belongs to the relatively fixed environmental reference region.

[0028] After obtaining the temporal stability probability distribution map, a preset temporal stability threshold is acquired, and the temporal stability probability is compared pixel by pixel with the preset threshold. When the temporal stability probability of a pixel is less than or equal to the preset threshold, the pixel is marked as a foreground pixel; when the temporal stability probability of a pixel is greater than the preset threshold, it is not marked as a foreground pixel. After thresholding all pixel units, connected component merging is performed on adjacent foreground pixel units, and isolated noise regions with an area smaller than the preset pixel are removed to obtain the foreground visual detection space. This foreground visual detection space corresponds to the image regions in the lifting device, four wire ropes, four corners of the load, Twist Lock head, container corner fittings, and load contour that change position with the lifting operation status. Then, pixel units in the visual detection space that are not included in the foreground visual detection space are used as background candidate pixel units, and connected component simplification and boundary smoothing are performed on the background candidate pixel units to obtain the background visual detection space. This background visual detection space corresponds to the edge of the double beam structure, ground reference markers, and fixed reference regions with small position and texture changes in consecutive frames. Therefore, the visual detection space is divided into a foreground visual detection space for characterizing the motion state of the suspended load, and a background visual detection space for providing a stable spatial reference. Through the above processing, the interference of suspended load swaying, single-frame noise, and local occlusion on the separation results can be eliminated at the pixel level, making the boundary between the foreground and background regions more continuous and stable, and providing a reliable foundation for the subsequent extraction of foreground and background spatial stability anchor points.

[0029] Obtain multiple foreground spatial stability anchor points based on the foreground visual detection space and multiple background spatial stability anchor points based on the background visual detection space.

[0030] In one embodiment, after obtaining the foreground visual detection space and the background visual detection space, the pixel units in the two spaces are screened for stability. For the foreground visual detection space, pixel units that change with the movement of the suspended load, such as the outline of the load, the four corners of the load, and the area near the lock head, are extracted, and the changes in the curvature of their outlines in consecutive frames are analyzed to obtain the corresponding curvature change rate. If the curvature change rate of a certain pixel unit or local pixel region is small, it indicates that the region can maintain relatively stable morphological features even when the load swings or its posture changes slightly, and it is then identified as a foreground space stability anchor point. For the background visual detection space, pixel units in relatively fixed areas such as the edge of the double beam structure and ground reference marks are extracted, and the changes in their texture gradients in consecutive frames are analyzed to obtain the corresponding texture gradient change rate. If the texture gradient change rate of a certain pixel unit or local pixel region is small, it indicates that the region is less affected by the movement of the load, lighting fluctuations, or image noise, and has good background reference stability, and it is then identified as a background space stability anchor point. Thus, auxiliary anchor points with high stability can be screened from the foreground motion region and the background reference region respectively, providing a reliable spatial benchmark for subsequent anchor point constraint network construction and load deviation analysis.

[0031] Furthermore, the curvature change rate of pixel units in the foreground visual detection space is obtained, and pixel units with curvature change rates less than a preset curvature threshold are marked as foreground space stability anchor points; the texture gradient change rate of pixel units in the background visual detection space is obtained, and pixel units with texture gradient change rates less than a preset gradient change rate threshold are marked as background space stability anchor points.

[0032] Optionally, when extracting foreground and background spatial stability anchor points, the foreground visual detection space output by the visual separation module is first read, and contour extraction is performed on the foreground visual detection space to obtain the foreground contour point set corresponding to the spreader, Twist Lock head, container corner pieces, and the lifting weight contour. Since the four upper corner pieces of the container, the edge of the spreader's lock head, and the regular corner areas in the lifting weight contour are rigid structures that move with the overall lifting weight, although these structures will translate, swing, or rotate with the lifting weight, their local geometry remains basically unchanged in consecutive frames, so their local curvature change rate is low. Based on this feature, a pixel neighborhood is established with each pixel unit in the foreground visual detection space as the center, such as a 5×5 neighborhood. The curvature value of the local contour where the pixel unit is located in the current frame is calculated, and the corresponding contour position of the pixel unit in the previous 15 frames is read to obtain the curvature sequence within 16 consecutive frames. Subsequently, the absolute average of the differences in curvature values ​​between adjacent frames is calculated, and this average is used as the curvature change rate of the pixel unit. When the curvature change rate of a pixel unit is less than a preset curvature threshold, and the pixel unit is visible for no less than 12 frames in 16 consecutive frames, the pixel unit is determined to be a geometrically stable foreground candidate anchor point. Then, connected component clustering is performed on the foreground candidate anchor points, merging those with a distance less than a preset pixel into one anchor point region, and using the geometric center of this region as the foreground spatial stability anchor point. Following this method, multiple foreground spatial stability anchor points are extracted preferentially at the four upper corners of the container, the edge of the Twist Lock head of the spreader, the lower edge of the spreader, and the corners of the lifting contour. The pixel coordinates, depth value, foreground object number, curvature change rate, and number of consecutive frames visible for each foreground spatial stability anchor point are recorded.

[0033] Next, the background visual detection space output by the visual separation module is read, and texture gradient calculations are performed on the double-beam structure edges, ground reference markers, and fixed reference line regions within the background visual detection space. Specifically, a pixel neighborhood is established centered on each pixel unit in the background visual detection space, and the gradient intensity in the horizontal and vertical directions of this neighborhood is calculated to obtain the texture gradient value of that pixel unit in the current frame. The texture gradient values ​​of the corresponding positions of this pixel unit in the previous 15 frames are then read, forming a texture gradient sequence of 16 consecutive frames. Then, the absolute average of the differences in texture gradient values ​​between adjacent frames is calculated, and this average is used as the texture gradient change rate of that pixel unit. Since the double-beam structure edges and ground reference markers are fixed reference objects, their texture direction and edge intensity change relatively little in consecutive frames. Therefore, when the texture gradient change rate of a pixel unit is less than a preset gradient change rate threshold, and the temporal stability probability of that pixel unit is greater than a preset temporal stability threshold, that pixel unit is marked as a background candidate anchor point. Finally, connected component merging is performed on the candidate background anchor points, and isolated regions with an area smaller than a preset pixel are removed. The geometric center of the retained connected components is used as the background spatial stability anchor point. Following this method, multiple background spatial stability anchor points are extracted at the intersection of the double-beam structure edge, the ground reference mark edge, and the fixed reference line. The pixel coordinates, depth value, background region number, texture gradient change rate, and number of consecutive frames visible for each anchor point are recorded. Through this processing, the foreground rigid structure's local curvature stability can be utilized to select foreground anchor points that move with the load but maintain stable shape. Simultaneously, the stable texture gradient of the background reference region can be used to select fixed background anchor points, thus providing a reliable spatial reference basis for subsequent anchor point constraint network construction and load deviation analysis.

[0034] Construct an anchor point constraint network for the plurality of foreground space stability anchor points and the plurality of background space stability anchor points.

[0035] In one embodiment, when constructing the anchor point constraint network, multiple foreground spatial stability anchor points and multiple background spatial stability anchor points are first read, and the pixel coordinates, depth values, and stability evaluation values ​​of each anchor point are recorded. Then, the foreground spatial stability anchor points located on the same rigid structure of the suspended load are connected, and the initial distance, included angle, and depth difference between adjacent foreground anchor points are calculated as internal topological geometric constraints, generating a foreground rigid deformation constraint subnet. Next, with each foreground spatial stability anchor point as the center, the closest background spatial stability anchor point with a satisfactory stability evaluation value is selected as a reference anchor point. The horizontal offset, vertical offset, and depth difference of the foreground anchor point relative to the background anchor point are calculated to form a relative spatial mapping relationship, generating a relative displacement reference subnet. Finally, the foreground rigid deformation constraint subnet and the relative displacement reference subnet are fused together according to the same foreground anchor points to obtain the anchor point constraint network. This network simultaneously possesses the rigid structural constraints of the suspended load itself and the displacement reference capability relative to a fixed background, providing a benchmark for subsequent image deviation auxiliary analysis.

[0036] Furthermore, the method for constructing the anchor point constraint network of the plurality of foreground space stability anchor points and the plurality of background space stability anchor points includes: Establish internal topological geometric constraints among the multiple foreground spatial stability anchor points to generate a foreground rigid deformation constraint subnet; establish relative spatial mapping relationships between the multiple foreground spatial stability anchor points and their neighboring background spatial stability anchor points to form a relative displacement reference subnet; merge the foreground rigid deformation constraint subnet and the relative displacement reference subnet to construct the anchor point constraint network.

[0037] Preferably, multiple extracted foreground spatial stability anchor points and multiple background spatial stability anchor points are first read, and foreground anchor point sets and background anchor point sets are established respectively. Then, using the foreground spatial stability anchor points on the same rigid structure of the lifting load as nodes, internal topological geometric constraints are established according to the geometric connection relationship of the lifting load itself. Specifically, the four upper corner anchor points of the container are connected in the order of upper left front corner, upper right front corner, upper right rear corner, and upper left rear corner, and two diagonals are connected to form a rectangular topology on the upper surface of the container; the four Twist Lock edge anchor points are connected to the corresponding upper corner anchor points of the container to form lock-corner anchor point correspondence constraints; the lower edge anchor points of the spreader are connected to the adjacent Twist Lock edge anchor points to form spreader-lock structure constraints. For each foreground anchor point connection edge, the pixel distance, three-dimensional spatial distance, depth difference, and connection direction angle between the two foreground anchor points in the initial frame are recorded, and the above parameters are used as the initial topological geometric parameters of the connection edge. In subsequent real-time monitoring, if the changes in distance, depth difference, and orientation angle between two corresponding foreground anchor points are all within the preset allowable range, then the connecting edge is determined to maintain consistent rigidity. All foreground anchor points and the connecting edges between them are combined into a foreground rigid deformation constraint subnet, which is used to describe the geometric relationships that should be maintained between the spreader, lock head, container corner fittings, and lifting contour under rigid body motion.

[0038] Subsequently, taking each foreground spatial stability anchor point as the center, the three background spatial stability anchor points with the closest spatial distance to it are found in the set of background anchor points, and a relative spatial mapping relationship is established between the foreground spatial stability anchor point and the three neighboring background spatial stability anchor points. Specifically, based on the depth value output by the binocular vision sensor, the pixel coordinates of the foreground anchor point and the background anchor point are converted into three-dimensional coordinates, and the horizontal relative displacement, vertical relative displacement, depth relative displacement, and relative orientation angle of the foreground anchor point relative to each neighboring background anchor point are calculated. The above relative displacement parameters obtained in the initial alignment state are used as the relative spatial mapping reference. For the i-th foreground anchor point and the j-th background anchor point, their relative spatial mapping relationship is recorded as Mij={ΔXij, ΔYij, ΔZij, αij, wij}, where ΔXij represents the relative displacement in the horizontal direction, ΔYij represents the relative displacement in the vertical direction, ΔZij represents the relative displacement in the depth direction, αij represents the relative direction angle from the foreground anchor point to the background anchor point, and wij represents the weight of this mapping relationship. This weight wij is determined based on the stability evaluation value of the background anchor point and the distance between the foreground anchor point and the background anchor point. The higher the stability evaluation value of the background anchor point and the closer the distance, the greater the weight. In the above manner, a relative spatial mapping relationship is established between each foreground spatial stability anchor point and its neighboring background spatial stability anchor points. All foreground anchor points, background anchor points, and the relative mapping edges from the foreground anchor point to the background anchor point are combined to form a relative displacement reference subnet, which is used to describe the spatial offset relationship of the moving part of the suspended load relative to the fixed environmental reference part.

[0039] Next, the foreground rigid deformation constraint subnet and the relative displacement reference subnet are fused. During fusion, the foreground anchor nodes and internal topological connection edges in the foreground rigid deformation constraint subnet are retained, while the background anchor nodes and relative mapping connection edges in the relative displacement reference subnet are superimposed on the same foreground anchor nodes. This ensures that each foreground anchor is subject to both the geometric constraints of other foreground anchors within the same rigid structure of the suspended load and the fixed reference constraints of neighboring background anchors. The fused anchor constraint network includes a node layer, a foreground rigid constraint edge layer, and a background reference constraint edge layer. The node layer stores the coordinates, depth values, and stability evaluation values ​​of the foreground and background anchors; the foreground rigid constraint edge layer stores the initial distance, initial angle, and initial depth difference between foreground anchors; and the background reference constraint edge layer stores the initial relative displacement and mapping weight of the foreground anchors relative to the background anchors. After fusion, the anchor constraint network is saved as a reference constraint model. During real-time suspended load alignment and correction, the anchor state of the current frame is matched and compared with the reference constraint model to determine whether the suspended load has swaying, rotation, horizontal offset, or height offset. By employing the above processing, the rigid geometric stability between foreground anchor points and the fixed spatial reference provided by background anchor points can be utilized simultaneously to improve the stability of the load deviation analysis under occlusion, swaying, and image jitter conditions.

[0040] A real-time visual inspection image of the load being lifted by a double-girder tire gantry crane is acquired. Image deviation auxiliary analysis is performed on the real-time visual inspection image according to the anchor point constraint network to obtain a multi-dimensional deviation vector. Based on the multi-dimensional deviation vector, a correction control command for the load control terminal is output.

[0041] In one embodiment, when a double-girder tire-mounted gantry crane performs a lifting and positioning operation, a binocular vision sensor acquires real-time visual inspection images of the lifting area. The foreground and background spatial stability anchor points, corresponding to the anchor point constraint network, are identified from these real-time visual inspection images. The anchor point positions in the real-time images are matched with the pre-constructed anchor point constraint network to determine whether the geometric relationship between the foreground anchor points has changed, and whether the spatial mapping relationship between the foreground anchor points and the background anchor points has shifted. During the matching analysis, if the distance, angle, or direction relationship between the foreground spatial stability anchor points deviates from the reference relationship in the constraint network, it can be determined that the lifting weight is swinging, rotating, or experiencing an attitude shift. If the positional relationship between the foreground spatial stability anchor points and the background anchor points changes, it can be determined that the lifting weight has a horizontal or height deviation relative to the target working area. Thus, deviation results of different directions and types are fused to form a multi-dimensional deviation vector that characterizes the current spatial deviation state of the lifting weight. Subsequently, the multidimensional deviation vector is input into the crane dynamics compensation model to calculate the motion compensation amounts for the trolley traveling mechanism, gantry traveling mechanism, and hoisting mechanism of the double-girder tire-mounted gantry crane. Corresponding correction control commands are then generated to control the trolley traveling mechanism, gantry traveling mechanism, and hoisting mechanism to perform corresponding motion compensations, such as correcting horizontal position deviations, reducing sway amplitude, adjusting hoisting height, or limiting operating speed, thereby assisting the suspended load to gradually approach the target alignment position. Through the above process, the deviation state of the suspended load in real-time visual images can be stably identified using an anchor point constraint network, and the visual analysis results can be converted into executable control commands, improving the correction accuracy, response speed, and operational safety during the suspended load alignment process.

[0042] Furthermore, the method for performing image deviation auxiliary analysis on the real-time visual detection image according to the anchor point constraint network to obtain a multi-dimensional deviation vector includes: Anchor point topology matching is performed on the real-time visual detection image based on the anchor point constraint network, including calculating the nonlinear swing amplitude deviation and rotation angle deviation of the suspended weight based on the multiple foreground spatial stability anchor points according to the foreground rigid deformation constraint subnet, and calculating the horizontal alignment deviation and height deviation of the suspended weight relative to the multiple foreground spatial stability anchor points according to the relative displacement reference subnet; the nonlinear swing amplitude deviation, rotation angle deviation, horizontal alignment deviation and height deviation are combined to construct the multidimensional deviation vector.

[0043] Optionally, when performing image deviation-assisted analysis on real-time visual inspection images according to the anchor point constraint network, the real-time visual inspection images are first input into the visual separation module to obtain the foreground visual inspection space and background visual inspection space of the current frame, and the foreground space stability anchor points and background space stability anchor points in the current frame are identified according to the aforementioned anchor point extraction method. Subsequently, each foreground space stability anchor point identified in the current frame is topologically matched with the foreground anchor point nodes in the anchor point constraint network. The matching is based on the object number to which the anchor point belongs, the relative order of the anchor points, the adjacency relationship of the anchor points, and the depth value of the anchor points, so that the four upper corner anchor points of the container, the edge anchor point of the spreader Twist Lock, the lower edge anchor point of the spreader, and the corner anchor point of the lifting contour in the current frame are respectively mapped to the reference foreground anchor point nodes in the foreground rigid deformation constraint subnet. After matching is completed, the initial 3D distance, initial angle and initial depth difference between the foreground anchor points stored in the foreground rigid deformation constraint subnet are read, and the real-time 3D distance, real-time angle and real-time depth difference between the corresponding foreground anchor points in the current frame are calculated. The difference between the real-time 3D distance and the initial 3D distance is used as the distance deformation deviation, the difference between the real-time angle and the initial angle is used as the angle deformation deviation, and the difference between the real-time depth difference and the initial depth difference is used as the depth deformation deviation. Since the four upper corner pieces of the container, the edge of the spreader lock head, and the outline of the load are rigid structures that move with the load as a whole, their internal topological geometry should remain relatively stable when there is no obvious swaying or rotation. Therefore, when the distance deformation deviation and depth deformation deviation between multiple foreground anchor points increase periodically, it is determined that the load has nonlinear swaying. The distance deformation deviation and depth deformation deviation of the connecting edges of each foreground anchor point are weighted and averaged according to the anchor point stability evaluation value to obtain the nonlinear swaying amplitude deviation of the load. When the rectangular topology formed by the four upper corner pieces of the container undergoes an overall directional deflection relative to the reference rectangular topology, the difference between the main direction angle of the current frame rectangular topology and the main direction angle of the reference rectangular topology is calculated, and this difference is used as the rotation angle deviation of the load.

[0044] Next, the foreground and background spatial stability anchor points in the current frame are matched with the relative displacement reference subnet. For each foreground spatial stability anchor point, its three corresponding neighboring background spatial stability anchor points in the anchor point constraint network are read. Based on the current frame depth image, the pixel coordinates of the foreground and background anchor points are converted into three-dimensional coordinates. Then, the real-time horizontal relative displacement, real-time vertical relative displacement, and real-time depth relative displacement of the current frame's foreground anchor point relative to its neighboring background anchor points are calculated, and the differences are calculated with the initial horizontal relative displacement, initial vertical relative displacement, and initial depth relative displacement saved in the reference alignment state. Then, the horizontal relative displacement differences corresponding to multiple foreground anchor points are weighted and fused according to the mapping weight to obtain the horizontal alignment deviation of the suspended weight; the depth relative displacement differences and vertical relative displacement differences corresponding to multiple foreground anchor points are weighted and fused to obtain the height deviation of the suspended weight. After completing the above calculations, the nonlinear swing amplitude deviation, rotation angle deviation, horizontal alignment deviation, and height deviation are combined in a preset order to construct a multidimensional deviation vector D={dSwing, dAngle, dX, dH}, where dSwing represents the nonlinear swing amplitude deviation of the suspended weight in the current frame; dAngle represents the rotation angle deviation of the suspended weight relative to the reference alignment state; dX represents the horizontal alignment deviation of the suspended weight relative to the target alignment position; and dH represents the height deviation of the suspended weight relative to the target landing point or target height. Through the above process, the changes in the anchor point topology in the real-time visual detection image can be converted into quantified results of the suspended weight's swing, rotation, horizontal offset, and height offset, enabling subsequent correction control commands to be generated based on explicit multidimensional deviation parameters.

[0045] Furthermore, the method for outputting correction control commands for the load control terminal based on the multidimensional deviation vector includes: The multidimensional deviation vector is input into a preset crane dynamics compensation model to calculate the motion compensation amount for the trolley traveling mechanism, gantry traveling mechanism, and hoisting mechanism of a double-girder tire gantry crane. Based on the motion compensation amount, a pulse smoothing control signal and a path correction signal are generated and sent to the load control terminal as the correction control command. The correction control command includes a speed limit safety control command that restricts the operating speed of the trolley traveling mechanism, gantry traveling mechanism, and hoisting mechanism.

[0046] Optionally, after obtaining the multidimensional deviation vector, the multidimensional deviation vector is input into a preset crane dynamics compensation model. This crane dynamics compensation model pre-stores the operating parameters of the trolley traveling mechanism, the hoisting mechanism, and the crane's gantry crane, including the current trolley traveling speed, the current hoisting speed, the maximum allowable acceleration of the trolley, the maximum allowable acceleration of the hoisting mechanism, the braking response time, and the lifted weight. After inputting the multidimensional deviation vector, the crane dynamics compensation model first determines whether each deviation component exceeds a preset correction threshold. The horizontal alignment deviation threshold can be set to 30mm; the height deviation threshold can be set to 20mm; the nonlinear sway amplitude deviation threshold can be set to 50mm; and the rotation angle deviation threshold can be set to 2°.

[0047] When dX exceeds the horizontal alignment deviation threshold, the horizontal motion compensation of the trolley traveling mechanism and the gantry traveling mechanism is calculated based on the direction and magnitude of dX. Specifically, according to the coordinate system of the double-girder tire gantry crane, the horizontal alignment deviation dX is decomposed into a deviation component dG along the trolley traveling direction and a deviation component dT along the gantry traveling direction, where dG=dX×cosβ, dT=dX×sinβ, and β represents the angle between the horizontal deviation direction of the load and the trolley traveling direction. Then, dG is converted into the trolley compensation displacement ΔSG, and dT is converted into the gantry compensation displacement ΔST, i.e., ΔSG=-kG×dG, ΔST=-kT×dT, where kG and kT are the trolley compensation coefficient and the gantry compensation coefficient, respectively, and their values ​​can be set to 0.8 to 1.0. The negative sign indicates that the compensation direction is opposite to the deviation direction. The speed correction is calculated by combining the current speed of the main trolley (VG), the current speed of the trolley (VT), and the braking response time (tb). The speed correction for the main trolley is ΔVG = ΔSG / (tc+tb) - VG, and the speed correction for the trolley is ΔVT = ΔST / (tc+tb) - VT, where tc represents the preset correction execution time, which can be set to 1s to 3s. This yields the horizontal motion compensation for the main trolley and trolley running mechanisms.

[0048] When dH exceeds the height deviation threshold, the height compensation amount of the hoisting mechanism is calculated based on the direction and magnitude of dH, and this height compensation amount is converted into the hoisting mechanism's lifting speed correction amount and stop position correction amount. Specifically, the height deviation dH is used as the basic height compensation amount of the hoisting mechanism, and the lifting height compensation amount is calculated according to ΔSH=-kH×dH, where kH is the height compensation coefficient, which can be set to 0.7 to 1.0. The negative sign indicates that the compensation direction of the hoisting mechanism is opposite to the height deviation direction. Combined with the current lifting speed VH of the hoisting mechanism and the preset height correction execution time tH, the lifting speed correction amount ΔVH=ΔSH / tH-VH is calculated. At the same time, the current target stop position H0 is corrected to H1=H0+ΔSH, obtaining the stop position correction amount. Thus, the hoisting mechanism can adjust the descent speed and stop height according to the height deviation of the suspended load relative to the target landing point.

[0049] When dSwing exceeds the nonlinear swing amplitude deviation threshold, the crane dynamics compensation model generates a short-term reverse compensation amount opposite to the swing direction based on the swing direction, and distributes this short-term reverse compensation amount to the trolley traveling mechanism and the hoist traveling mechanism to counteract the swing inertia of the load. Specifically, the nonlinear swing amplitude deviation dSwing is decomposed into a trolley-direction swing component dSwing-G and a hoisting direction swing component dSwing-T according to the swing direction. Then, the short-term reverse compensation amount ΔPG=-ks×dSwing-G for the trolley and ΔPT=-ks×dSwing-T for the hoisting are calculated respectively, where ks is the swing smoothing coefficient, which can be set to 0.3 to 0.6. Next, the pulse duration tp=t0+ks1×dSwing is determined according to the swing amplitude, where t0 is the base pulse time, which can be set to 0.3s, ks1 is the time scaling factor, and tp is limited to between 0.3s and 0.8s. This results in short-term reverse micro-motion compensation, which causes the trolley traveling mechanism and the gantry traveling mechanism to output a small displacement or speed change in the opposite direction of the swing in a short period of time, so as to reduce the swing amplitude of the load.

[0050] When dAngle exceeds the rotation angle deviation threshold, the differential micro-motion compensation amount of the trolley traveling mechanism and the trolley traveling mechanism is calculated according to the positive or negative direction of the rotation angle deviation, so that the load generates a correction trend opposite to the rotation deviation during horizontal movement. Specifically, taking the center point of the load as the rotation reference point, the angle compensation amount Δθ = -kθ × dAngle is calculated according to the rotation angle deviation dAngle, where kθ is the angle compensation coefficient, which can be set to 0.5 to 0.9. Then, the corresponding horizontal micro-motion compensation amount is calculated according to the load length L and width W. The differential micro-motion compensation amount along the trolley direction is ΔRG = (L / 2) × sinΔθ, and the differential micro-motion compensation amount along the trolley direction is ΔRT = (W / 2) × sinΔθ. If dAngle is positive, the trolley traveling mechanism and the trolley traveling mechanism are controlled to output micro-motion compensation in the direction opposite to the positive rotation; if dAngle is negative, the trolley traveling mechanism and the trolley traveling mechanism are controlled to output micro-motion compensation in the direction opposite to the negative rotation. This allows the rotation angle deviation to be gradually offset during the horizontal alignment of the suspended weight.

[0051] After completing the above calculations, the motion compensation amounts ΔCg, ΔCt, and ΔCh of the trolley traveling mechanism, respectively, are obtained. ΔCg includes the trolley compensation displacement and trolley speed correction; ΔCt includes the trolley compensation displacement and trolley speed correction; and ΔCh includes the lifting height compensation and lifting speed correction. Subsequently, when the nonlinear swing amplitude deviation or rotation angle deviation exceeds the corresponding threshold, a pulse smoothing control signal is generated based on the trolley traveling mechanism motion compensation amounts ΔCg and ΔCt. This pulse smoothing control signal includes the pulse direction, pulse duration, and pulse amplitude. The pulse direction is opposite to the direction of the load swing or rotational deviation; the pulse duration is set to 0.3s to 0.8s; and the pulse amplitude is determined proportionally to the magnitudes of dSwing and dAngle, used to reduce the load swing amplitude and rotational deviation through short-time micro-motion control. When the horizontal alignment deviation exceeds the horizontal alignment deviation threshold, a path correction signal is generated based on the trolley compensation displacement and the hoisting mechanism compensation displacement. This path correction signal includes the target correction distance for the trolley, the target correction distance for the hoisting mechanism, the correction speed for the trolley, and the correction speed for the hoisting mechanism, and is used to drive the suspended load closer to the target alignment position. When the height deviation exceeds the height deviation threshold, a height correction signal is generated based on the hoisting mechanism motion compensation amount ΔCh. This height correction signal includes the hoisting mechanism descent correction distance, descent speed correction amount, and stop position correction amount, and is used to control the suspended load to descend smoothly to the target height or stop. Meanwhile, to prevent excessive swaying of the suspended load due to excessive speed during the alignment process, the alignment control command also includes a speed limit safety control command. Specifically, when dSwing is greater than the first sway speed limit threshold and less than or equal to the second sway speed limit threshold, the operating speeds of the trolley and crane traveling mechanisms are limited to the first horizontal speed limit ratio, such as 60%. When dSwing is greater than the second sway speed limit threshold or dAngle is greater than the rotation speed limit threshold, the operating speeds of the trolley and crane traveling mechanisms are limited to the second horizontal speed limit ratio, such as 30%, and the descent speed of the hoisting mechanism is limited to the hoisting speed limit ratio, such as 40%. When dX, dH, dSwing, and dAngle all fall back to within their corresponding thresholds, the speed limit safety control command is released, and the system returns to normal alignment control. Finally, the pulse smoothing control signal, path correction signal, height correction signal, and speed limit safety control command are encapsulated according to the command format recognizable by the control terminal to form a correction control command for the crane control terminal, and sent to the crane control terminal. The crane control terminal then controls the trolley traveling mechanism, the hoisting mechanism, and the hoisting mechanism to perform corresponding speed adjustment, displacement correction, lifting correction, and safety speed limit actions.Through the above process, the multidimensional deviation results obtained from visual detection can be converted into motion compensation control quantities that can be executed by each actuator of the crane, thereby realizing load sway suppression, path correction, height correction and safe speed limiting, and improving the real-time performance, stability and safety of load alignment correction.

[0052] In summary, the embodiments of this application have at least the following technical effects: First, the visual inspection space of the load lifted by the double-girder tire-mounted gantry crane is obtained, and this visual inspection space is separated into a foreground visual inspection space and a background visual inspection space. Then, multiple foreground space stability anchor points based on the foreground visual inspection space and multiple background space stability anchor points based on the background visual inspection space are obtained. Next, an anchor point constraint network is constructed for the multiple foreground space stability anchor points and the multiple background space stability anchor points. Finally, a real-time visual inspection image of the load lifted by the double-girder tire-mounted gantry crane is obtained, and image deviation-assisted analysis is performed on the real-time visual inspection image according to the anchor point constraint network to obtain a multi-dimensional deviation vector. Based on the multi-dimensional deviation vector, a correction control command is output for the load control terminal. This invention addresses the technical challenges of poor stability of visual inspection anchor points, low accuracy of deviation recognition, and delayed response to alignment correction caused by swinging, obstruction, changes in lighting, and background interference during the lifting, moving, and alignment of loads. It achieves the technical effect of enhancing the spatiotemporal constraint capability of visual features under dynamic working conditions by constructing a dual-level stability anchor point and constraint network of foreground and background, thereby improving the anti-interference capability, control accuracy, and operational safety of load alignment correction.

[0053] Example 2, based on the same inventive concept as the vision-assisted double-beam tire gantry crane lifting alignment and correction method in the aforementioned examples, such as... Figure 2 As shown, this application provides a vision-assisted double-girder tire gantry crane load alignment and correction device, wherein the device includes: Visual space separation component 11: acquires the visual detection space of the load being lifted by the double-girder tire gantry crane, and separates the visual detection space to obtain a foreground visual detection space and a background visual detection space; Spatial anchor point acquisition component 12: acquires multiple foreground spatial stability anchor points based on the foreground visual detection space and multiple background spatial stability anchor points based on the background visual detection space; Anchor point constraint construction component 13: constructs an anchor point constraint network for the multiple foreground spatial stability anchor points and the multiple background spatial stability anchor points; Deviation correction control component 14: acquires a real-time visual detection image of the load being lifted by the double-girder tire gantry crane, performs image deviation auxiliary analysis on the real-time visual detection image according to the anchor point constraint network, obtains a multi-dimensional deviation vector, and outputs a deviation correction control command for the load control terminal based on the multi-dimensional deviation vector.

[0054] Furthermore, the visual spatial separation component 11 is used to perform the following method: The original visual detection sample space data of the crane's lifting area is acquired using a vision sensor deployed on a double-girder tire gantry crane. A temporal rolling window is used to obtain the stability index of each visual object in the original visual detection sample space data based on consecutive frames. The stability index is normalized to obtain a stability training label. The original visual detection sample space data is input into a visual separation network. The stability training label is used as a supervisory constraint to construct a supervised loss function. The supervised loss function is used to perform gradient backpropagation on the visual separation network, iteratively updating the weight parameters of the visual separation network to obtain a trained visual separation module. The visual detection space is separated according to the visual separation module to obtain a foreground visual detection space and a background visual detection space.

[0055] Furthermore, the visual spatial separation component 11 is used to perform the following method: A temporal rolling window is used to acquire N consecutive frames of visual images of the original visual detection sample spatial data; inter-frame correlation analysis is performed on the N consecutive frames of visual images to obtain the continuous frame tracking trajectory corresponding to each visual object; the position change rate, scale change rate, pose change rate, occlusion persistence rate, and feature preservation rate of each visual object are calculated based on the continuous frame tracking trajectory; the position change rate, scale change rate, pose change rate, occlusion persistence rate, and feature preservation rate are weighted and averaged to obtain the stability index.

[0056] Furthermore, the visual spatial separation component 11 is used to perform the following method: The stability change sequence is calculated based on the changing trend of the stability index corresponding to each visual object in consecutive frames; a temporal consistency constraint is constructed according to the stability change sequence, which is used to constrain the stability index change of the same visual object to remain continuous and consistent in consecutive frames; a temporal consistency constraint loss function is constructed based on the temporal consistency constraint, and gradient backpropagation is performed on the visual separation network using the supervised loss function and the temporal consistency constraint loss function.

[0057] Furthermore, the visual spatial separation component 11 is used to perform the following method: The visual separation module calculates the temporal distribution weights of each pixel unit in the visual detection space and outputs a temporal stability probability distribution. Based on the temporal stability probability distribution, pixel units less than or equal to a preset temporal stability threshold are identified and output as the foreground visual detection space. Based on the visual detection space and the foreground visual detection space, a background visual detection space is constructed.

[0058] Furthermore, the spatial anchor point acquisition component 12 is used to perform the following method: Obtain the curvature change rate of pixel units in the foreground visual detection space, and mark pixel units with curvature change rates less than a preset curvature threshold as foreground space stability anchor points; obtain the texture gradient change rate of pixel units in the background visual detection space, and mark pixel units with texture gradient change rates less than a preset gradient change rate threshold as background space stability anchor points.

[0059] Furthermore, the anchor point constraint construction component 13 is used to perform the following method: Establish internal topological geometric constraints among the multiple foreground spatial stability anchor points to generate a foreground rigid deformation constraint subnet; establish relative spatial mapping relationships between the multiple foreground spatial stability anchor points and their neighboring background spatial stability anchor points to form a relative displacement reference subnet; merge the foreground rigid deformation constraint subnet and the relative displacement reference subnet to construct the anchor point constraint network.

[0060] Furthermore, the correction control component 14 is used to perform the following method: Anchor point topology matching is performed on the real-time visual detection image based on the anchor point constraint network, including calculating the nonlinear swing amplitude deviation and rotation angle deviation of the suspended weight based on the multiple foreground spatial stability anchor points according to the foreground rigid deformation constraint subnet, and calculating the horizontal alignment deviation and height deviation of the suspended weight relative to the multiple foreground spatial stability anchor points according to the relative displacement reference subnet; the nonlinear swing amplitude deviation, rotation angle deviation, horizontal alignment deviation and height deviation are combined to construct the multidimensional deviation vector.

[0061] Furthermore, the correction control component 14 is used to perform the following method: The multidimensional deviation vector is input into a preset crane dynamics compensation model to calculate the motion compensation amount for the trolley traveling mechanism, gantry traveling mechanism, and hoisting mechanism of a double-girder tire gantry crane. Based on the motion compensation amount, a pulse smoothing control signal and a path correction signal are generated and sent to the load control terminal as the correction control command. The correction control command includes a speed limit safety control command that restricts the operating speed of the trolley traveling mechanism, gantry traveling mechanism, and hoisting mechanism.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A vision-assisted method for aligning and correcting the load on a double-girder tire gantry crane, characterized in that, The method includes: The visual inspection space of the load lifted by the double-girder tire gantry crane is obtained, and the visual inspection space is separated to obtain the foreground visual inspection space and the background visual inspection space. Obtain multiple foreground spatial stability anchor points based on the foreground visual detection space and multiple background spatial stability anchor points based on the background visual detection space; Construct an anchor point constraint network for the plurality of foreground space stability anchor points and the plurality of background space stability anchor points; A real-time visual inspection image of the load being lifted by a double-girder tire gantry crane is acquired. Image deviation auxiliary analysis is performed on the real-time visual inspection image according to the anchor point constraint network to obtain a multi-dimensional deviation vector. Based on the multi-dimensional deviation vector, a correction control command for the load control terminal is output.

2. The method as described in claim 1, characterized in that, The method for separating the visual detection space into a foreground visual detection space and a background visual detection space includes: The original visual inspection sample spatial data of the crane's lifting area is obtained by using vision sensors deployed on a double-girder tire gantry crane; A time-series rolling window is used to obtain the stability index of each visual object in the original visual detection sample spatial data based on consecutive frames, and the stability index is normalized to obtain the stability training label. The original visual detection sample spatial data is input into the visual separation network. The stability training labels are used as supervision constraints to construct a supervision loss function. The supervision loss function is used to perform gradient backpropagation on the visual separation network. The weight parameters of the visual separation network are iteratively updated to obtain the trained visual separation module. The visual detection space is separated into a foreground visual detection space and a background visual detection space by the visual separation module.

3. The method as described in claim 2, characterized in that, The method for obtaining the stability index of each visual object in the original visual detection sample spatial data based on consecutive frames using a temporal rolling window includes: A time-series rolling window is used to acquire N consecutive frames of visual images of the original visual detection sample spatial data; Perform inter-frame correlation analysis on the N consecutive visual images to obtain the continuous frame tracking trajectory corresponding to each visual object; The position change rate, scale change rate, pose change rate, occlusion persistence rate, and feature preservation rate of each visual object are calculated based on the continuous frame tracking trajectory. The stability index is obtained by weighted averaging and fusing the position change rate, scale change rate, pose change rate, occlusion persistence rate, and feature preservation rate.

4. The method as described in claim 2, characterized in that, The method for iteratively training the visual separation network further includes: Calculate the stability change sequence based on the changing trend of the stability index corresponding to each visual object in consecutive frames; A temporal consistency constraint is constructed based on the stability change sequence, and the temporal consistency constraint is used to constrain the stability index change of the same visual object to remain continuous and consistent in consecutive frames. A temporal consistency constraint loss function is constructed based on the temporal consistency constraint conditions, and gradient backpropagation is performed on the visual separation network using the supervised loss function and the temporal consistency constraint loss function.

5. The method as described in claim 2, characterized in that, The visual detection space is separated into a foreground visual detection space and a background visual detection space according to the visual separation module, and the method includes: The temporal distribution weights of each pixel unit in the visual detection space are calculated based on the visual separation module, and the temporal stability probability distribution is output. Pixel units less than or equal to a preset temporal stability threshold are identified based on the temporal stability probability distribution, and the output is the foreground visual detection space. Based on the visual detection space and the foreground visual detection space, a background visual detection space is constructed.

6. The method as described in claim 5, characterized in that, Obtain the curvature change rate of pixel units in the foreground visual detection space, and mark pixel units with curvature change rates less than a preset curvature threshold as foreground space stability anchor points; The texture gradient change rate of pixel units in the background visual detection space is obtained, and pixel units with texture gradient change rates less than a preset gradient change rate threshold are marked as background space stability anchor points.

7. The method as described in claim 1, characterized in that, The method for constructing the anchor point constraint network of the plurality of foreground space stability anchor points and the plurality of background space stability anchor points includes: Establish internal topological geometric constraints among the multiple foreground space stability anchor points to generate a foreground rigid deformation constraint subnet. Establish relative spatial mapping relationships between the multiple foreground spatial stability anchor points and their neighboring background spatial stability anchor points to form a relative displacement reference subnet; The anchor point constraint network is constructed by integrating the foreground rigid deformation constraint subnet and the relative displacement reference subnet.

8. The method as described in claim 7, characterized in that, The method involves performing image deviation auxiliary analysis on the real-time visual detection image according to the anchor point constraint network to obtain a multi-dimensional deviation vector, including: The anchor point constraint network is used to perform anchor point topology matching on the real-time visual detection image, including calculating the nonlinear swing amplitude deviation and rotation angle deviation of the suspended weight based on the multiple foreground spatial stability anchor points according to the foreground rigid deformation constraint subnet, and calculating the horizontal alignment deviation and height deviation of the suspended weight relative to the multiple foreground spatial stability anchor points according to the relative displacement reference subnet. The nonlinear swing amplitude deviation, rotation angle deviation, horizontal alignment deviation, and height deviation are combined to construct the multidimensional deviation vector.

9. The method according to claim 8, characterized in that, The method includes outputting correction control commands for the load control terminal based on the multidimensional deviation vector, and the method further includes: The multidimensional deviation vector is input into the preset crane dynamics compensation model to calculate the motion compensation amount for the trolley traveling mechanism, the gantry traveling mechanism and the hoisting mechanism of the double-girder tire gantry crane. Based on the motion compensation amount, a pulse smoothing control signal and a path correction signal are generated and sent to the load control terminal as the correction control command. The correction control command includes a speed limit safety control command that restricts the operating speed of the trolley traveling mechanism, the hoisting mechanism, and the crane mechanism.

10. A vision-assisted double-girder tire gantry crane load alignment and correction device, characterized in that, The apparatus for implementing the vision-assisted double-girder tire gantry crane lifting alignment and correction method according to any one of claims 1-9, the apparatus comprising: Visual space separation component: acquire the visual detection space of the load being lifted by the double-beam tire gantry crane, and separate the visual detection space to obtain the foreground visual detection space and the background visual detection space; Spatial anchor point acquisition component: acquires multiple foreground spatial stability anchor points based on the foreground visual detection space and multiple background spatial stability anchor points based on the background visual detection space; Anchor point constraint construction component: Constructs an anchor point constraint network for the plurality of foreground space stability anchor points and the plurality of background space stability anchor points; Correction control component: acquires real-time visual inspection images of the load lifted by the double-girder tire gantry crane, performs image deviation auxiliary analysis on the real-time visual inspection images according to the anchor point constraint network, obtains multi-dimensional deviation vectors, and outputs correction control commands for the load control terminal based on the multi-dimensional deviation vectors.