A visual automatic alignment system for tire container cranes

Abstract

The present invention discloses a visual automatic alignment system for a tire-type container crane, comprising: a target detection and segmentation unit for obtaining the position of an operating target in an image; a motion decision unit for receiving the output result of the target detection and segmentation unit and outputting the control instructions required by the tire crane control system; and a human-computer interaction interface for displaying key parameters, so that the debugging personnel can adjust the parameters for different tire cranes. The present invention solves the problem of automatic alignment control of a tire-type container crane and realizes the integration of perception and motion decision.

Description

Technical Field

[0001] The invention relates to port automation technology, and more specifically to a visual automatic alignment system for a tire-type container crane. Background Art

[0002] The tire crane is a common port machinery and equipment. It is widely used in ports around the world because of its low requirements for the work site. Perception positioning is one of the keys to the automation of the tire crane. For a long time, the tire crane has mainly used traditional laser, magnetic nail and other technologies to achieve positioning. The advantages of this technology are simplicity and reliability, and the disadvantages are single usage scenarios, limited information provided and high construction costs.

[0003] In recent years, the transformation of ports to intelligence has become an inevitable trend in the industry. Traditional technologies alone are no longer able to meet current development needs. With the research of scholars around the world, deep learning has become the mainstream perception technology that uses convolutional neural networks to extract image features. Image processing technology based on deep learning has the advantages of low sensor requirements, rich perception information, and effective reduction of production costs, making this technology gradually become a new solution.

[0004] Another key to achieving port machinery automation is motion decision-making. At present, position-based visual control is widely used at home and abroad. In the 3D Cartesian coordinate system, camera calibration and coordinate transformation are used to obtain the position of the object being measured in the Cartesian space, and form a deviation with the expected position. Then, the control algorithm is designed according to the deviation to control the movement of the mechanism. This method has the following disadvantages:

[0005] 1) No matter how you calibrate the camera, coordinate transformation will bring certain errors;

[0006] 2) Port machinery is heavy machinery. When lifting dozens of tons of cargo, the machine will be deformed, which will also cause errors in coordinate conversion.

[0007] Due to the existence of errors, port machinery parameters need to be adjusted frequently. Therefore, a large number of debugging personnel have to be deployed at the operation site to adjust the parameters, which increases the operation and maintenance costs. Summary of the invention

[0008] In view of the above-mentioned defects existing in the prior art, the purpose of the present invention is to provide a visual automatic alignment system for a tire-type container crane, which solves the problem of automatic alignment control of the tire-type container crane and realizes the integration of perception and motion decision-making.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A visual automatic alignment system for a tire container crane, comprising:

[0011] The target detection and segmentation unit is used to obtain the position of the target in the image;

[0012] A motion decision unit, used to receive the output result of the target detection and segmentation unit, and output the control instructions required by the tire crane control system;

[0013] The human-computer interaction interface is used to display key parameters, making it easy for debuggers to adjust parameters for different tire cranes.

[0014] Preferably, the target detection and segmentation unit comprises:

[0015] An image acquisition module, used to acquire a target image;

[0016] A work target foreground segmentation module based on a Mask-RCNN network, used to receive the image data of the image acquisition module and segment the foreground image of the work target;

[0017] The operation target positioning module is used to post-process the foreground image and output the position of the operation target in the image.

[0018] Preferably, the image acquisition module is four cameras, which are respectively arranged at four corners of the sling.

[0019] Preferably, the operating target of the camera is the diagonal line of the container, and the images are collected synchronously.

[0020] Preferably, the Mask-RCNN network includes:

[0021] The backbone network extracts features of the target image through the residual network, and then performs multi-scale feature fusion through the feature pyramid network;

[0022] Sensitive region proposal network, used to judge positive and negative samples of each feature and roughly locate them;

[0023] The sensitive regions are aligned and the value of each feature is calculated by applying bilinear interpolation.

[0024] Preferably, the motion decision unit includes:

[0025] The motion modeling module is used to perform motion modeling on the positioning operation of the tire crane and establish a motion model;

[0026] The logic control module calculates the deviation and the bottom opening operation process according to the motion model and the position of the operation target in the image, and outputs the logic control instructions.

[0027] Preferably, the human-computer interaction interface is developed using HTML web pages.

[0028] The present invention provides a visual automatic alignment system for a tire-type container crane. First, a target detection and segmentation unit is designed to sense and locate the operating target, which fully combines the powerful perception ability of the neural network with the interpretability of the traditional image algorithm, greatly improving the accuracy and reliability of the model. Then, a motion decision unit is designed to convert the perception and positioning results into control signals, which greatly simplifies the camera calibration process and realizes the integration of visual positioning and alignment control. The decision-making accuracy of the present invention is as high as 99.9%. Practical application results show that the operating efficiency has been greatly improved and the failure rate is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of a rubber tire container crane;

[0030] Figure 2 It is a schematic diagram of the framework structure of the visual automatic alignment system of the tire-type container crane of the present invention;

[0031] Figure 3 Schematic diagram of the image acquisition module in the visual automatic alignment system of the rubber-tyred container crane of the present invention, wherein (a) is a schematic diagram of the arrangement, and (b) is a schematic diagram of the shooting angle of view;

[0032] Figure 4 It is a box corner line foreground binary image of the operating target positioning module in the visual automatic alignment system of the tire-type container crane of the present invention, wherein (a) is a box corner line mask image, and (b) is a box corner line foreground binary image;

[0033] Figure 5 It is a canny edge detection result diagram of the operating target positioning module in the visual automatic alignment system of the tire-type container crane of the present invention;

[0034] Figure 6 It is a schematic diagram of the positioning points of the operating target positioning module in the visual automatic positioning system of the tire-type container crane of the present invention during the container positioning operation;

[0035] Figure 7 It is a schematic diagram of the box corner line being blocked;

[0036] Figure 8 It is a schematic diagram of the misdetection of the box corner line;

[0037] Fig. 9 It is a schematic diagram of standard and non-standard box corner lines;

[0038] Fig.10 Schematic diagram of standard box corner line positioning point extraction, where (a) is the original perception result image, and (b) is the positioning point extraction image;

[0039] Fig.11 It is a schematic diagram of the container position calibration points;

[0040] Fig.12 This is a schematic diagram of the ideal camera installation position;

[0041] Fig.13 It is a composite diagram of four cameras in non-ideal camera installation positions;

[0042] Fig.14 This is a schematic diagram of the upper right box corner being blocked;

[0043] Fig.15 It is a schematic diagram of blocking two box corner lines at the same time;

[0044] Fig.16 It is a schematic diagram of the open-bottom motion control curve, where (a) is a schematic diagram of the trolley direction motion control curve, (b) is a schematic diagram of the trolley direction motion control curve, and (c) is a schematic diagram of the rotation direction motion control curve. DETAILED DESCRIPTION

[0045] In order to better understand the above technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0046] Combination Figure 1 As shown in the figure, the tire crane is a crane used for lifting containers. Its main structure consists of three parts: a large car 1, a small car frame 2 and a spreader 3. The large car 1 is equipped with tires and can move in the direction of each row of containers; the small car frame 2 moves in the direction of each column of containers; and the spreader 3 is connected to the small car frame 2 with a wire rope. Through the movement of the large car 1 and the small car frame 2, the spreader 3 can lift containers at any position in the yard. At the same time, the spreader 3 has the function of micro-movement along the direction of the large car, the small car and rotation, which is convenient for the spreader to perform micro-adjustment during the alignment operation.

[0047] Combination Figure 2 As shown, the present invention provides a visual automatic alignment system for a rubber-tyred container crane, comprising:

[0048] The target detection and segmentation unit 4 is used to obtain the position of the working target in the image;

[0049] The motion decision unit 5 is used to receive the output result of the target detection and segmentation unit 4 and output the control instructions required by the tire crane control system;

[0050] The human-machine interaction interface 6 is used to display key parameters, so that the debugging personnel can adjust the parameters for different tire cranes.

[0051] The target detection and segmentation unit 4 first obtains the target image through the image acquisition module 41, and then the target foreground segmentation module 42 based on the Mask-RCNN network segments the target foreground image. Finally, the target positioning module 43 post-processes the foreground image and outputs the position of the target in the image.

[0052] The image acquisition module 41 is composed of four cameras, which are respectively arranged at the four corners of the spreader 3 to collect the image of the working target (for the bottom opening operation, the working target is the box corner line). Figure 3 As shown in (a).

[0053] Camera shooting angle Figure 3 As shown in (b), since the four cameras work independently, in order to ensure that the four cameras collect images synchronously, the present invention adopts a soft synchronization method, and the specific method is as follows:

[0054] 1) The four cameras capture images at a fixed frame rate with timestamps, and the captured images are stored in their respective buffers;

[0055] 2) Taking one of the cameras as the reference, obtain the latest image in its buffer;

[0056] 3) In the buffers of the other three cameras, select the image that is closest in time to the image;

[0057] 4) The four images selected are grouped as a batch and used as the input of the neural network.

[0058] The modified model of the operating target foreground segmentation module 42 based on the Mask-RCNN network is established: In the selection of the perception algorithm research baseline, a more accurate two-stage target detection algorithm is selected, and the detection and segmentation algorithm based on Mask-RCNN is selected to realize the positioning and classification of the operating target.

[0059] The Mask-RCNN detection and segmentation algorithm is used to achieve the foreground segmentation of the box corner line. Target detection uses a rectangular box to locate the target, and its position in the image is determined by only two points (the upper left point and the lower right point), while target segmentation is to separate different targets by region from the perspective of pixels, and use a mask to identify the target, and its position in the image is determined by a set of points (i.e., the coordinates of each pixel). There is a disadvantage of using target detection to locate targets in a continuous image sequence: the detection frame will shake. The jitter of the detection signal is not conducive to the alignment control of the tire crane. Compared with target detection located by two points, target segmentation located by a group of points is more stable.

[0060] The Mask-RCNN network is a two-stage object detection and segmentation network. The first stage extracts image features and generates proposals, and the second stage classifies the proposals and generates bounding boxes and masks. It mainly includes:

[0061] The backbone network extracts features of the target image through the residual network (Resnet50), and then performs multi-scale feature fusion through the Feature Pyramid Network (FPN);

[0062] Sensitive Region Proposal Network (RPN), the network structure is similar to the RPN in Faster-RCNN, the only difference is that the input is the output of FPN, the network is used to judge the positive and negative samples of each anchor feature and roughly locate;

[0063] Compared with RoI Pooling, RoI Align uses bilinear interpolation to calculate the value of each feature point, making the feature aggregation process continuous. When performing Mask regression, pixel position offset is avoided.

[0064] Dataset collection: The training dataset uses the MS-COCO dataset and the box-angle line dataset collected for bottom opening operations. The perception model is pre-trained with the MS-COCO dataset to enable the convolutional neural network to have a certain understanding of the target, and then the box-angle line dataset is used to fine-tune the perception model to improve the recognition and positioning accuracy.

[0065] The box corner line dataset is mainly composed of two parts: 1) The box corner line image data collected from multiple ports. This dataset is used as a general detection and segmentation dataset for box corner line targets, with a total of 10,838 images, including 9,758 training sets and 1,080 validation sets; 2) In the actual port scene of bottom opening operation, the box corner line image data during the bottom opening operation is collected specifically. This dataset is used as a fine-tuning dataset to improve the recognition and positioning accuracy in actual operation scenarios. It has a total of 6,028 images, including 5,004 training sets and 1,024 validation sets.

[0066] Training: During training, the hyperparameters in Mask-RCNN are kept unchanged. The difference is that the resolution of the input image is scaled to 640×640 to improve the real-time performance of the model.

[0067] Initialization: For the Mask-RCNN backbone network, ImageNet1k pre-trained weights were used for initialization. For the one-stage and two-stage classification and regression networks, weights with bias b = 0 and Gaussian variance σ = 0.01 were used for initialization.

[0068] Optimizer: The entire network is trained using the stochastic gradient descent (SGD) optimizer, with momentum set to 0.9 and weight decay set to 0.0001. During training, the batch size is set to 4 and evenly distributed across two 2080Ti graphics cards. The initial learning rate is set to 0.001. As training progresses, the learning rate is divided by 10 at the 4K, 6K, and 8K iterations to achieve a dynamic decrease in the learning rate. The training process is iterated 10K times in total. During training, only horizontal flipping of images is used to augment the dataset. (Note: The above training process uses MS-COCO for pre-training. During the fine-tuning stage, the learning rate is set to 0.0001 throughout the process, and the rest remains unchanged).

[0069] Experimental results: On the general detection and segmentation dataset of box corner targets, the AP accuracy reaches 96.04%, and on the fine-tuning dataset of actual open-bottom conditions, the AP accuracy reaches 99.49%.

[0070] The target foreground segmentation module 42 based on the Mask-RCNN network only outputs the segmentation result, but does not output the position of the desired target in the image. The perception result is analyzed using traditional image processing methods, and the precise position of the target in the image is output.

[0071] Obtaining the box corner line foreground image: The box corner line mask obtained by the operation target foreground segmentation module 42 based on the Mask-RCNN network is saved on the same image and binarized to obtain the box corner line foreground image, such as Figure 4 As shown in (a) and (b).

[0072] Canny edge detection: First, the obtained box corner line foreground binary image is morphologically processed, specifically: 1) corrosion operation is performed to remove some small noise spots; 2) dilation operation is performed to connect the adjacent areas to enhance the integrity of the box corner line. Finally, the pre-processed foreground image is subjected to canny edge detection, and the minimum bounding rectangle of the contour is obtained, such as Figure 5 shown.

[0073] Positioning point selection: The bottom opening operation requires the container to be placed in the area marked by the corner line of the container. Through observation, it is found that when the bottom opening meets the requirements, the corner points of the corner line of the container will not be blocked. Therefore, the present invention uses the corner points of the corner line of the container as the positioning points during the container alignment operation, such as Figure 6 As shown in the points.

[0074] Extraction of standard box corner positioning points: During the bottom opening and alignment process, the extraction of positioning points faces the following problems:

[0075] 1) The container will inevitably block the corner line of the box, causing the corner point of the box corner line to disappear. Figure 7 As shown;

[0076] 2) Although the performance of the neural network is excellent, due to the existence of interference factors (light intensity, electromagnetic interference, etc.) and the "black box" nature of the neural network, the perception results are still uncertain and false detections may occur (such as Figure 8 Even though the probability of false detection is very low, it cannot be ignored in order to improve system reliability;

[0077] 3) There will be multiple box corner lines appearing in the camera's field of view, and the canny edge detection algorithm will detect the outline of each box corner line (such as Figure 5 As shown in FIG. 1 , the desired box corner line needs to be screened out. In view of these problems, the present invention proposes a standard box corner line positioning point extraction algorithm.

[0078] For problems 1) and 2): Analyze each contour detected by canny and only filter out the standard box corners, such as Fig. 9 For the standard box corner line, the present invention provides three criteria: the aspect ratio of the box corner line (i.e. the aspect ratio of the minimum bounding rectangular frame of the box corner line), the opening direction of the box corner line and the shape of the box corner line (L-type or F-type). Only when these three criteria are met at the same time can it be determined as a standard box corner line.

[0079] For the upper left, upper right, lower left, and lower right cameras, the expected opening directions of the box corner line are lower right, lower left, upper right, and upper left, respectively. According to the standard box corner line judgment conditions, the box corner line is judged and the positioning points are extracted (Note: only the standard box corner line is used for positioning point extraction), such as Fig.10 shown.

[0080] Since only the standard box corner line is used for positioning point extraction, the occlusion and false detection problems can be effectively solved. The more stringent the standard box corner line judgment standard is, the lower the false detection rate will be, but the missed detection rate will increase. By adjusting the threshold of the judgment standard, a positioning point extraction algorithm with balanced performance can be obtained.

[0081] If the cameras of one or several viewing angles fail to identify the positioning point, the posture of the container relative to the corner line can be inferred based on the perception results of other cameras. For specific methods, see the establishment of the logic control module.

[0082] Regarding question 3): When the sling lifts the container, the sling and the container, as well as the sling and the camera are rigidly connected, so the relative position of the camera and the container does not change. In other words, the position of the container on the camera screen is fixed. When the container and the corner line of the box are aligned, it is necessary to obtain the position of the corner line of the box and the container. The position of the corner line of the box is the positioning point, and as for the position of the container, since the position of the container on the camera screen is fixed, a point can be directly calibrated on the image as the position of the container. After analysis, the present invention uses the four corners of the container as the calibration points of the container position, such as Fig.11 As shown in the points.

[0083] The operation target positioning module 43 adopts a method that combines deep learning with traditional image processing, which greatly improves the accuracy and reliability of the model and successfully applies vision to the open production environment of the port.

[0084] Refer to Figure 2 As shown, the motion decision unit 5 first performs motion modeling on the tire crane alignment operation through the motion modeling module 51, and then calculates the deviation amount according to the motion model and the position of the operation target in the image. Finally, the logic control module 52 outputs the logic control instruction according to the bottom opening operation process. The target detection and segmentation module 4 and the motion decision module 5 are both adjusted through the human-computer interaction interface 6.

[0085] Motion modeling module 51 Establishing a motion modeling module

[0086] Open-bottom motion analysis: If the cameras are installed at the same height and have the same lens focal length, and the center points of the CMOS target surfaces of the cameras are aligned horizontally and vertically, and the sides of the CMOS target surfaces are parallel, then the image stitched together by the four cameras can be regarded as an image taken by one camera. In this ideal case, the pixel plane coordinate system can be used directly, such as Fig.12 As shown in the figure, the origin of the coordinate is the upper left point of the image, the x direction is the moving direction of the large vehicle, and the y direction is the moving direction of the small vehicle.

[0087] Fig.12 In the figure, Pctl, Pctr, Pcbl, Pcbr are the corner line positioning points, Pbtl, Pbtr, Pbbl, Pbbr are the container position calibration points. Pbc is the center point of the container, and Pcc is the center point of the corner line positioning area.

[0088] During alignment operation, the calculation formula for the trolley direction deviation err_x and the trolley direction deviation err_y is as follows:

[0089]

[0090] The coordinate calculation formulas of point Pbc and point Pcc in the x and y directions are as follows:

[0091]

[0092] Substituting formula (2) into formula (1), we get:

[0093]

[0094] In the four cameras, the calibration points and positioning points are displaced in the x and y directions (see Fig.12 ) is calculated as follows:

[0095]

[0096] Substituting formula (4) into formula (3), we get:

[0097]

[0098] From formula (5), we can see that err_x and err_y are the average values ​​of the displacements of the calibration points and the positioning points in the four cameras in the x and y directions.

[0099] During alignment, the rotation deviation can be calculated by calculating the angle between the line segment (Pbbl, Pbtr) and the line segment (Pcbl, Pctr). However, in actual applications, it is found that the difference in displacement between the calibration point and the positioning point on the same side in the x or y direction can be used to measure it. For example, the difference in displacement between the upper left and upper right cameras in the y direction is used as the rotation deviation, and the formula is as follows:

[0100] err_θ=Petl_y-Pebr_y (6)

[0101] Establishing a multi-camera fusion positioning coordinate system: The above is the calculation of the deviation of the camera under the ideal installation state. However, in actual applications, it is difficult to install the camera in place. Even if it is installed in place, the installation position of the camera will be offset due to vibration as the tire crane cycles. Fig.13 In this case, the pixel plane coordinate system is no longer applicable.

[0102] To solve this problem, a multi-camera fusion positioning coordinate system is proposed, with the vertex of the box corner line (the red point in the figure) as the origin and the two sides of the box corner line as the x-axis and y-axis, as shown in Fig.13 shown.

[0103] The reasons why the coordinate system can be constructed in this way are: 1) No matter how the camera is installed, the relative position relationship between the container and the box corner line in each camera image will not change with the camera installation angle; 2) Although the 4-camera stitching image looks like the container and the box corner line are deformed, they are not actually deformed. Therefore, the calculation formulas for the large vehicle direction deviation err_x, the small vehicle direction deviation err_y, and the rotation deviation _ can still use formulas (5) and (6), but they should be calculated in the multi-camera fusion positioning coordinate system.

[0104] Establishing the logic control module 52

[0105] When the tire crane is aligned, the logical control strategy for the micro-motion of the hoist is: 1) first perform rotational movement until it reaches the desired position; 2) then perform movement in the direction of the trolley until it reaches the desired position; 3) finally perform movement in the direction of the large vehicle until it reaches the desired position.

[0106] The bottom opening operation can be performed according to the logic control strategy and the calculated deviation. However, during the bottom opening operation, the container may block the corner line of the box, resulting in the inability to detect the positioning point. In this case, the position relationship between the container and the corner line of the box can be inferred based on the recognition of the other cameras. The specific strategy is as follows:

[0107] (1) No obstruction

[0108] In this state, the calculation formulas for err_x, err_y, and err_θ are the same as formulas (5) and (6).

[0109] (2) Only one box corner is blocked

[0110] When only one box corner line is blocked and the positioning point cannot be identified, it can be calculated based on the positioning points in the other three cameras. Fig.14 shown.

[0111] The calculation formulas for err_x and err_y are as follows:

[0112]

[0113] The calculation formula of err_θ is as follows:

[0114] err_θ=Pebl_y-Pebr_y (8)

[0115] The calculation methods for the other three cases are similar.

[0116] (3) Only block two box corners

[0117] Due to the geometric relationship between the container and the box corner line, if two box corner lines are blocked at the same time, the blocked box corner lines must be on the same side, such as Fig.15 shown.

[0118] Take the example of blocking the two upper box corners at the same time (i.e. Fig.15 The calculation formula of err_θ is the same as formula (8), and the calculation formula of err_x is as follows:

[0119]

[0120] Since the two positioning points on the same side cannot be identified, the deviation err_y of the car direction cannot be accurately calculated. However, the direction of movement of the car can be inferred by combining the recognition of the four cameras. In this case, setting a step can control the car to move in the correct direction. As the movement progresses, the occlusion will disappear, and then the accurate calculation can be performed according to formula (5) or (7).

[0121] (4) Block three or four box corners

[0122] Due to the geometric relationship between the container and the corner line, this situation will not occur. If this situation occurs, it can be considered that the foreground segmentation module has made a recognition error, and the control will jump from automatic control to remote manual operation.

[0123] Design control rate

[0124] The dimensions of err_x, err_y and err_θ calculated above are all pixels, not the position, speed, etc. required for actual control. How to apply these deviations to the tire crane control system requires control rate design.

[0125] When the tire crane is performing alignment operations, the cargo it lifts weighs dozens of tons, and it is a heavy-duty, low-speed system. The existing manual control logic is: 1) The driver observes the relative position relationship between the current container and the corner line to determine the movement direction of the spreader at the next moment; 2) Operate the corresponding handle to give the switch value to control the tire crane to perform alignment operations. Repeat steps 1) and 2) until the accuracy requirements are met.

[0126] The design of the control rate of the present invention simulates manual work. For the three control quantities err_x, err_y, and err_θ, fixed step sizes are set: step_x, step_y, step_θ. The control rate formula is as follows:

[0127]

[0128]

[0129]

[0130] Experimental verification

[0131] Experiment of logic control module 52: The logic control module 52 is used to make motion decisions for each frame of the image during the bottom opening operation. For the decision results, the present invention adopts a manual verification method. Finally, in 4000 frames of images, the decision accuracy rate reaches 99.9%.

[0132] Motion control experiment: According to the designed control rate, the motion control curve of the tire crane along the trolley, trolley and rotation direction is as follows Fig.16As shown in (a), (b) and (c).

[0133] It can be seen from the figure that the deviation curves in the three directions are all approaching 0. Experiments have proved that this control rate can effectively control the bottom opening operation.

[0134] The human-machine interaction interface 6 is developed using an HTML web page. The debugging personnel only need to open a browser and enter the corresponding URL to log in to the human-machine interaction interface 6. The human-machine interaction interface 6 provides key parameters, which facilitates the debugging personnel to adjust parameters for different tire cranes to ensure that the crane's visual automatic alignment system can work properly.

[0135] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. As long as they are within the spirit of the present invention, any changes or modifications to the above embodiments will fall within the scope of the claims of the present invention.

Claims

1. A visual automatic alignment system for a rubber-tyred container crane, characterized in that: include: The target detection and segmentation unit is used to obtain the position of the target in the image; A motion decision unit, used to receive the output result of the target detection and segmentation unit, and output the control instructions required by the tire crane control system; The human-machine interactive interface is used to display key parameters, making it easier for commissioning personnel to adjust parameters for different tire cranes. The target detection segmentation unit comprises: An image acquisition module, used to acquire a target image; A work target foreground segmentation module based on a Mask-RCNN network, used to receive the image data of the image acquisition module and segment the foreground image of the work target; The operation target positioning module is used to post-process the foreground image and output the position of the operation target in the image. The Mask-RCNN network includes: The backbone network extracts features of the target image through the residual network, and then performs multi-scale feature fusion through the feature pyramid network; Sensitive region proposal network, used to judge positive and negative samples of each feature and roughly locate them; Sensitive areas are aligned and the value of each feature is calculated by using bilinear interpolation. The motion decision unit comprises: The motion modeling module is used to perform motion modeling on the positioning operation of the tire crane and establish a motion model; The logic control module calculates the deviation and the bottom opening process according to the motion model and the position of the operation target in the image, and outputs the logic control instructions. The Mask-RCNN network-based target foreground segmentation module outputs the precise position of the target in the image: Obtaining the box-corner line foreground image: The box-corner line mask obtained by the target foreground segmentation module based on the Mask-RCNN network is saved on the same image and binarized to obtain the box-corner line foreground image. Canny edge detection: First, the obtained box corner line foreground binary image is morphologically processed, specifically: 1) corrosion operation is performed to remove some small noise spots; 2) dilation operation is performed to connect the adjacent areas and enhance the integrity of the box corner line; finally, canny edge detection is performed on the preprocessed foreground image, and the minimum bounding rectangle of the contour is obtained. Positioning point selection: The bottom opening operation requires the container to be placed in the area marked by the corner line of the container. Through observation, it is found that when the bottom opening meets the requirements, the corner point of the corner line of the container will not be blocked. The corner point of the corner line of the container is set as the positioning point for the container alignment operation. Open-bottom motion analysis: If the cameras are installed at the same height and have the same lens focal length, and the center points of the CMOS target surfaces of the cameras are aligned horizontally and vertically, and the sides of the CMOS target surfaces are parallel, then the image stitched together by the four cameras is considered to be an image taken by one camera.

2. The visual automatic alignment system for a rubber-tyred container crane according to claim 1 is characterized in that: The image acquisition module is four cameras, which are respectively arranged at the four corners of the sling.

3. The visual automatic alignment system for a rubber-tyred container crane according to claim 2 is characterized in that: The operating target of the camera is the diagonal line of the container, and the image is collected synchronously.

4. The visual automatic alignment system for a rubber-tyred container crane according to claim 1 is characterized in that: The human-computer interaction interface is developed using HTML web pages.

Images