Ship Scanning and Spread Collision Avoidance System and Method Based on Laser and Vision Fusion

Abstract

The present invention discloses a ship scanning and spreader anti-collision system and method based on the fusion of laser and vision. The system includes a laser scanner, a vision acquisition device, a ship scanning vision controller, a ship scanning laser controller, and a quay crane PLC controller. The laser scanner and the vision acquisition device continuously scan and take images of the ship contour and the container stacking contour on the ship according to the scanning instruction and the shooting instruction. The ship scanning vision controller processes the taken images to obtain a three-dimensional space image of the ship contour. The ship scanning laser controller obtains the ship contour point cloud data according to the received scanning data, and corrects the ship contour point cloud data according to the received three-dimensional space image of the ship contour, so as to realize the accurate scanning and recognition of the ship contour. And through the quay crane PLC controller, it controls the movement of the quay crane trolley and the lifting of the spreader to realize the anti-collision of the quay crane spreader and achieve the soft landing of the container on the sea and land sides.

Description

Technical Field

[0001] The present invention relates to the technical field of quay cranes in container terminals, and particularly to a ship scanning and spreader anti-collision system and method based on the fusion of laser and vision. Background Art

[0002] The quay bridge crane (referred to as quay crane for short) is an important machine for container terminal operations. It is one of the equipment for lifting containers above the ship. Its efficiency, safety, and operation correctness have an important impact on terminal operations. The quay bridge crane has the characteristics of long spreader steel cables, fast hoisting and trolley speeds. When operating above the ship, since the driver cannot clearly see the situation below, it may cause container damage due to too fast container landing, or even collide with the ship's structure, resulting in losses. At the same time, due to the influence of tidal fluctuations, cable slack, weight changes, etc. during ship operations, phenomena such as shaking and drifting away from the shore (offshore) will occur, which have a safety impact on operations.

[0003] At present, for the ship contour scanning system of the ship under the quay crane in container terminals at home and abroad, the ship contour is scanned by using a laser scanner to achieve the anti-collision function, but the function is relatively single; and due to the limitation of the accuracy of the laser scanner, the existing system can only detect obstacles large enough and cannot meet the precise positioning requirements of the ship's tank and other parts, nor can it accurately position the lifting lock holes of the target containers on the ship; in addition, due to the influence of the environment on the laser scanner and the fault factors of the scanning system itself, the credibility of the laser equipment is poor, and it cannot guarantee the realization of soft landing on the sea and land sides. Summary of the Invention

[0004] To solve the above deficiencies of the prior art, the present invention provides a ship scanning and spreader anti-collision system and method based on the fusion of laser and vision. Through laser scanning and visual acquisition, the contour of the ship under the quay bridge is scanned and recognized completely, meeting the precise positioning of the longitudinal displacement of the ship, the ship's upper tank, and the container lock holes, accurately measuring the height value of the quay crane spreader and the position information of the quay crane trolley, realizing the obstacle avoidance of the quay crane spreader, and realizing the soft landing of the container on the sea and land sides.

[0005] In the first aspect, the present disclosure provides a ship scanning and spreader anti-collision system based on the fusion of laser and vision.

[0006] A ship scanning and spreader anti-collision system based on the fusion of laser and vision includes a laser scanner, a visual acquisition device, a ship scanning vision controller, and a ship scanning laser controller;

[0007] The laser scanner is used to receive the scanning instruction issued by the ship scanning laser controller, continuously scan the ship contour and the container stacking contour on the ship according to the scanning instruction, and upload the scanning data to the ship scanning laser controller;

[0008] The visual acquisition device is used to receive the image shooting instruction issued by the ship scanning vision controller, continuously shoot the ship contour and the container stacking contour on the ship according to the shooting instruction, and upload the image data to the ship scanning vision controller;

[0009] The ship scanning vision controller is used to obtain the three-dimensional space image of the ship contour through image processing based on the received image data, and send the three-dimensional space image of the ship contour to the ship scanning laser controller;

[0010] The ship scanning laser controller is used to obtain the ship contour point cloud data according to the received scanning data, and at the same time correct the ship contour point cloud data according to the received three-dimensional space image of the ship contour to obtain the final ship contour.

[0011] In a further technical solution, the system further includes a quay crane PLC controller, a hoisting encoder and a quay crane trolley encoder;

[0012] The quay crane PLC controller is electrically connected to the hoisting encoder and the quay crane trolley encoder, and is used to obtain the encoder information data and the spreader opening / closing state data, and send the obtained data to the ship scanning laser controller;

[0013] The quay crane PLC controller is further used to receive the control instruction issued by the ship scanning laser controller, and control the quay crane trolley encoder and the hoisting encoder according to the control instruction, so as to control the movement of the quay crane trolley and the lifting of the spreader.

[0014] In a further technical solution, the system further includes a DP module. The DP module is a conversion module, which is connected to the quay crane PLC controller and the ship scanning laser controller, and is used for communication protocol conversion and communication data transmission between the quay crane PLC controller and the ship scanning laser controller.

[0015] In a further technical solution, the system further includes an optoelectronic module and a display module. The ship scanning laser controller converts the obtained ship contour into an image through the optoelectronic module and performs visual display through the display module.

[0016] In a further technical solution, the ship scanning vision controller locates the ship guide groove and the target container lock hole according to the three-dimensional space image of the ship contour, and transmits the positioning information of the ship guide groove and the target container lock hole detected visually to the ship scanning laser controller;

[0017] The ship scanning laser controller issues a control instruction to the quay crane PLC controller according to the received positioning information of the ship guide groove and the target container lock hole, combined with the detected quay crane trolley position, spreader height and quay crane status data;

[0018] The quay crane PLC controller controls the quay crane trolley encoder and the hoisting encoder according to the received control instructions, and further controls the movement of the quay crane trolley, the opening and closing, and the lifting and lowering of the spreader.

[0019] In a further technical solution, the ship scanning vision controller detects the height of the highest obstacle at the current working bay according to the three-dimensional spatial image of the ship's contour, and transmits the height of the highest obstacle detected visually to the ship scanning laser controller;

[0020] The ship scanning laser controller detects the position of the quay crane trolley and the spreader height, and detects the height of the highest obstacle at the current working bay according to the ship's contour point cloud data;

[0021] The ship scanning laser controller performs data correction according to the received visually detected highest obstacle height, performs collision judgment according to the corrected highest obstacle height and the spreader height, and issues control instructions and warnings according to the collision judgment result, controlling the movement of the quay crane trolley and the lifting and lowering of the spreader.

[0022] In a further technical solution, the system further includes an alarm module for performing collision warnings according to the warning information issued by the ship scanning laser controller.

[0023] In a second aspect, the present disclosure provides a ship scanning and spreader anti-collision method based on the fusion of laser and vision.

[0024] A ship scanning and spreader anti-collision method based on the fusion of laser and vision is implemented based on the ship scanning and spreader anti-collision system based on the fusion of laser and vision proposed in the first aspect, and includes:

[0025] Initialize the installation parameters of the laser scanner and the quay crane structure parameters;

[0026] Based on the installation parameters of the laser scanner, with the installation position of the laser scanner as the origin, convert the laser scan polar coordinate data to the laser rectangular coordinate system;

[0027] Calibrate the vision acquisition device;

[0028] Based on visual acquisition, obtain continuous two-dimensional images of the ship's contour, and based on the obtained two-dimensional image coordinates and the internal and external parameters of the binocular camera calibration, obtain the three-dimensional spatial image of the ship's contour;

[0029] Based on laser scanning, obtain laser scan data to form ship contour point cloud data;

[0030] Fuse the three-dimensional spatial image of the ship's contour based on visual acquisition and the ship's contour point cloud data based on laser scanning to identify the ship's contour.

[0031] In a further technical solution, it further includes a target container keyhole positioning method, including:

[0032] Based on visual acquisition, continuous two-dimensional images of the container stacking contour on the ship are obtained. Based on the obtained two-dimensional image coordinates and the internal and external parameters calibrated by the binocular camera, a spatial position image of the container stacking on the ship is obtained;

[0033] Preprocess and perform edge detection on the obtained images to identify the target container and determine the three-dimensional coordinates of the container lock hole.

[0034] A further technical solution further includes:

[0035] Based on laser scanning and visual acquisition, detect the position, lifting height, and spreader height of the quay crane trolley during the lifting process, and simultaneously detect the height of the highest obstacle at the current working bay;

[0036] Based on the detection data acquired visually, correct the data of the laser scanning. According to the corrected height of the highest obstacle and the spreader height, perform collision judgment, and issue control instructions and warnings based on the collision judgment results to control the movement of the quay crane trolley and the lifting of the spreader.

[0037] The above one or more technical solutions have the following beneficial effects:

[0038] 1. The present invention provides a ship scanning and spreader anti-collision system and method based on the fusion of laser and vision. Three-dimensional point cloud data is obtained through a planar array laser, and the position, height of the containers at the current bay on the ship, as well as the position and height information of the containers at the adjacent bays are also obtained; RGB images of the ship are obtained through a binocular camera, and based on the vision model, the bounding boxes of the containers at the current bay on the ship and the bounding boxes of the containers at the adjacent bays are determined; the laser point cloud data is corrected according to the data acquired visually, and the various parameter features of the container ship are detected and identified to achieve accurate recognition of the ship contour.

[0039] 2. The present invention obtains the characteristic parameters of the ship through two methods of laser scanning and visual acquisition, realizes accurate determination of the obstacles around the containers on the ship, realizes anti-collision of the quay crane spreader, and realizes soft landing on the sea and land sides of the container, meeting the requirements of safe production.

[0040] 3. The present invention obtains the RGB image of the ship through a binocular camera, and based on the vision model, accurately captures the lock hole positioning coordinates of the target container to assist in the positioning of the quay crane target container. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings forming a part of this invention are used to provide a further understanding of the invention. The schematic embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation to the invention.

[0042] Figure 1Schematic diagram of the ship scanning and spreader anti-collision system based on the fusion of laser and vision according to the embodiments of the present invention;

[0043] Figure 2 Schematic diagram of adjacent bays of containers on a ship in the embodiments of the present invention;

[0044] Figure 3 Right side view of the hardware installation of the system in the embodiments of the present invention;

[0045] Figure 4 Top view of the hardware installation of the system in the embodiments of the present invention;

[0046] Figure 5 Schematic diagram of the installation of the laser reflector on the spreader in the embodiments of the present invention;

[0047] Figure 6 Wiring diagram of the system described in the embodiments of the present invention.

[0048] Among them, 11 is the first 2D laser scanner, 12 is the second 2D laser scanner, 13 is the third 2D laser scanner, 14 is the laser scanner reflector, 21 is the first camera, 22 is the second camera, 3 is the center of the quay crane trolley, 4 is the center of the quay crane gantry, 5 is the driver's cab, and 6 is the spreader upper frame. Detailed implementation manners

[0049] It should be noted that the following detailed descriptions are all exemplary and are intended to provide further descriptions of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0051] Embodiment 1

[0052] This embodiment provides a ship scanning and spreader anti-collision system based on the fusion of laser and vision. As Figure 1 shown, it includes a laser scanner, a vision acquisition device, a ship scanning vision controller, a ship scanning laser controller, an optoelectronic module, a DP module, a quay crane PLC controller, a hoist encoder, a trolley encoder, and a touch screen terminal.

[0053] In this embodiment, a 2D laser scanner is selected as the laser scanner. Three laser scanners are used to receive the scanning instructions sent by the ship scanning laser controller, and continuously scan the ship's outline and the stacking outline of the containers on the ship according to the scanning instructions, and upload the scanning data to the ship scanning laser controller. Among them, as Figure 3 and Figure 4 shown, two 2D laser scanners (the first 2D laser scanner 11 and the second 2D laser scanner 12) are fixedly installed on the quay crane trolley, on the left and right sides of the center 3 of the quay crane trolley, and can move along the sea-land side with the quay crane trolley, so as to synchronously scan the contour data of the container area on the ship downward along the row direction (i.e., the left-right direction of the hull); another 2D laser scanner (the third 2D laser scanner 13) is also fixedly installed on the quay crane trolley, at the relative position of the center 4 of the quay crane gantry, and can move along the sea-land side with the quay crane trolley, so as to scan the contour data of the container area of the container ship downward along the column direction (i.e., the front-back direction of the container hull).

[0054] Furthermore, the 2D laser scanner detects the fixed reference target installed in the direction of the quay crane trolley to obtain the position of the quay crane trolley; the 2D laser scanner detects the fixed reference target installed in the lifting direction to obtain the spreader height. The spreader height and the position information of the quay crane trolley are obtained through the above laser scanning measurement. In this embodiment, the 2D laser scanner scans the fixed reference target installed in the direction of the quay crane trolley, and transmits the scanning data to the ship scanning laser controller. The ship scanning laser controller determines the position of the quay crane trolley according to the detection data; the 2D laser scanner scans the fixed reference target installed in the lifting direction, that is, detects the laser scanner reflector 14 installed on the spreader upper frame, and transmits the scanning data to the ship scanning laser controller. The ship scanning laser controller determines the spreader height according to the detection data.

[0055] It should be noted that the laser scanner used in this embodiment is different from the lidar. Although they use the same principle, lidar is usually used for measuring long distances with poor accuracy, while laser scanners are usually used for three-dimensional modeling with high accuracy. In this embodiment, the laser scanner is used for scanning to obtain the three-dimensional point cloud data of the ship's outline.

[0056] In this embodiment, the installation angle and installation position of the above 2D laser scanner are input into the ship scanning laser controller in the form of a configuration file, so as to detect the above-mentioned highest obstacle and the heights of adjacent containers and container guides.

[0057] The operation processes of the above quay crane trolley and quay crane gantry are as follows: In the cab 5, the quay crane gantry is controlled to move left and right facing the sea. After moving to the working bay row, the quay crane trolley moves on the sea and land sides. When the quay crane trolley moves forward and backward, that is, when the quay crane trolley moves on the sea and land sides, the scanning data of the first 2D laser scanner 11 and the second 2D laser scanner 12 respectively form a line, and the scanning data of the third 2D laser scanner 13 forms a plane; after completing one bay row, the quay crane gantry moves left and right facing the sea. At this time, the quay crane trolley does not move, the scanning data of the third 2D laser scanner 13 forms a line, and the scanning data of the first 2D laser scanner 11 and the second 2D laser scanner 12 forms a plane.

[0058] In this embodiment, the visual acquisition device 13 is selected as a camera. Two cameras are used to receive the image shooting instructions sent by the ship scanning vision controller, and continuously shoot the ship contour and the container stacking contour on the ship according to the shooting instructions, and upload the image data to the ship scanning vision controller. Among them, the two cameras (i.e., the first camera 21 and the second camera 22) are both fixedly installed on the quay crane trolley, and are respectively installed on the left and right sides of the center 3 of the quay crane trolley. The two cameras form a binocular vision and can move on the sea and land sides with the quay crane trolley, so as to shoot the contour data of the container ship box area downward along the bay row direction (i.e., the front and back direction of the container hull).

[0059] The two cameras of the quay crane trolley form a binocular vision. Camera calibration is performed according to the binocular vision stereo imaging principle. The internal and external parameters of the camera are substituted into the reconstruction model and input into the ship scanning vision controller in the form of a configuration file.

[0060] The above quay crane PLC controller is electrically connected to the hoisting encoder and the quay crane trolley encoder, and is used to control the hoisting encoder and the quay crane trolley encoder. By this quay crane PLC controller, the movement of the quay crane trolley, the lifting of the spreader, etc. are controlled to achieve the soft landing of the containers on the ship on the sea and land sides. In this embodiment, this quay crane PLC controller is used to obtain encoder information data and spreader opening and closing state data, and send the obtained data to the ship scanning laser controller. At the same time, it receives the control instructions sent by the ship scanning laser controller and controls the movement of the quay crane trolley, the lifting of the spreader, etc.

[0061] Furthermore, a DP module is also set in this embodiment. As a conversion module, it realizes the communication connection and data transmission between the quay crane PLC controller and the ship scanning laser controller through communication protocol conversion.

[0062] The ship scanning vision controller receives image data, that is, it receives two images of the same cabin or chute obtained by two industrial cameras with different perspectives (i.e., the first camera 21 and the second camera 22). After step-by-step image processing, it calculates the conversion parameters between the corresponding pixel point pairs in the two images in three-dimensional space, and obtains the accurate spatial position of the target object (i.e., the ship) based on the vision between the two cameras. Then it sends the operation result (i.e., the three-dimensional image of the ship's contour) to the ship scanning laser controller.

[0063] The ship scanning laser controller receives the scanning data to obtain the ship contour point cloud data. On this basis, the ship scanning laser controller combines the shore crane trolley position, spreader height, and shore crane status data for subsequent operation processing. The above shore crane status data includes the encoder information data and spreader open / lock status data obtained through the shore crane PLC controller. By summarizing and processing the laser power data corresponding to the trolley position, a point cloud data map of the entire ship contour is obtained. In this embodiment, the ship scanning laser controller forms the contour data of the working container area on the three-dimensional container according to the scanning data of the 2D laser scanner. To reduce the error of a single scanning system, the ship scanning laser controller corrects the scanning data of the first 2D laser scanner 11, the second 2D laser scanner 12, and the third 2D laser scanner 13 with each other, and the three 2D laser scanners are backup to each other. When any one of the scanners fails, the other scanners can continue to meet the operation tasks.

[0064] Considering that there may be a problem of poor accuracy in identifying the ship contour only by using pure laser scanning, the ship scanning laser controller also receives the accurate spatial position information of the target object sent by the ship scanning vision controller, and performs correction and comprehensive processing according to the ship contour information and the accurate spatial position of the target object to obtain the final ship scanning operation result. The operation result is sent to the touch screen terminal, which includes a display and an alarm device, and the operation result is displayed through the display.

[0065] In this embodiment, the ship scanning laser controller detects the heights of the adjacent containers and container guides in the front and back of the current working bay through the contour data of bay A from the 2D laser scanner. The adjacent bay containers on the ship are as Figure 2 shown, and are mutually corrected with the ship data obtained through the ship vision controller to avoid collisions between the spreader and the container lifted by the spreader and obstacles, and ensure the safety of the operation process.

[0066] Furthermore, in this embodiment, the ship contour is recognized by laser and vision, and the trolley position, lifting height, and stacking position and height of the container on the ship are detected in real time.

[0067] The above-mentioned ship scanning vision controller detects the highest obstacle in the current working bay and the positioning of the ship chute through the container ship compartment area contour data (i.e., the three-dimensional spatial image of the ship contour) from the vision acquisition device, and transmits the highest obstacle information and ship chute positioning information detected by vision to the ship scanning laser controller.

[0068] In this embodiment, an RGB image is obtained by a camera and input into a vision model based on an instance segmentation algorithm (the instance segmentation algorithm can be used for "object detection", "object instance segmentation", "object key point detection", etc.) to determine the boundary boxes of the containers in the current bay and the boundary boxes of the containers in the adjacent bay, and at the same time, the positioning coordinates of the keyholes of the target container to be grabbed can also be determined.

[0069] The ship scanning laser controller detects the trolley position and spreader height information through the scanning data from the 2D laser scanner, and detects the height information of the highest obstacle in the current working bay according to the ship contour point cloud data. In this embodiment, as Figure 5 shown, the accurate acquisition of the spreader height information is achieved through the laser scanner and the laser scanner reflector 14 installed on the spreader upper frame 6. This embodiment uses the laser scanner reflector marking to realize the continuous tracking of the spreader. In this embodiment, a three-dimensional point cloud map is obtained through a planar array laser and input into the PointNet++ network for detection to determine the position, height, deflection angle of the containers on the upper bay of the current ship and the position and height information of the containers in the adjacent bay.

[0070] Furthermore, the ship scanning laser controller performs data correction through the received highest obstacle information detected by vision, makes a collision judgment according to the corrected highest obstacle height and spreader height, performs anti-collision protection according to the set anti-collision protection distance, and issues control instructions and warnings according to the collision judgment result to control the movement of the quay crane trolley, the lifting of the spreader, etc., to realize the functions of anti-collision protection of the spreader in the trolley running direction, soft landing in the lifting direction, and anti-collision protection of the spreader in the adjacent bay in the trolley running direction.

[0071] Specifically, it detects the positions and heights of the containers in each row under the quay container crane, and detects the trolley position and lifting height; during the lifting process, when the bottom of the lifted container is 0.5 meters higher than the container on the ship, it controls the trolley to move at full speed; when there is a collision tendency, it controls the trolley to stop moving; the system regards the movement of the quay container crane exceeding 30 cm as changing the current column.

[0072] In this embodiment, the ship scanning laser controller is connected to the quay crane remote management system and the quay crane PLC controller. The detection data of the ship scanning laser controller is sent to the quay crane remote management system and the quay crane PLC controller in real time, serving as a reference for the operation path of the spreader, so as to control the movement of the spreader along the optimal path and assist in achieving intelligent deceleration when the spreader is close to the target container.

[0073] This embodiment combines visual means and laser means, and respectively uses visual images and laser data to train effective models to detect and identify the parameter characteristics of the container ship. Then, based on the parameter characteristics obtained by the two recognition methods, it jointly judges the ship contour scanning and anti-collision, and realizes the accurate determination of the obstacles around the containers on the ship, meeting the requirements of safe production.

[0074] The system functional safety design of this embodiment meets the SIL2 requirements, realizes the collision protection between adjacent brackets. The container hold adopts a laser point cloud mode recognition algorithm, through intelligent visual detection, uses a ship detection algorithm based on edge features and straight line models, and realizes the horizontal and vertical division, clustering refinement, and straight line filtering based on gray value variance of the hold edge line, etc., to achieve accurate detection and protection of deep pit operations.

[0075] As another implementation method, this embodiment can also achieve the precise positioning of the lock hole of the target container through the system, including: two cameras on the quay crane trolley form binocular vision. Through the binocular vision stereo imaging principle, camera calibration and correction are carried out, and the internal and external parameters of the camera are substituted into the reconstruction model. Through image preprocessing and edge detection, the target container is recognized, the lock hole circle of the container is extracted, the lock hole is positioned, and the three-dimensional coordinates of the lock hole are obtained to meet the accurate positioning and container grabbing of the ship-side container.

[0076] The specific connection method of the system in this embodiment is as Figure 6 shown. The laser scanner and the visual acquisition device are both connected to the switch through Ethernet cables. The switch is connected to the laser controller and the visual industrial computer through Ethernet cables. The power supply is connected to the laser controller and the visual industrial computer through power cables to supply power to them. The laser controller is connected to the quay crane PLC controller through a data cable for data transmission.

[0077] In summary, the system in this embodiment uses a long-distance laser scanner and a visual acquisition device. By scanning the reference board on the quay crane trolley frame, a fixed coordinate system is established. Through the laser and visual fusion technology, combined with the encoder information provided by the quay crane PLC control system and the laser and visual scanning contours, the ship contour is scanned and recognized and the historical information is complemented. At the same time, using the information such as the symmetry of the ship berthing, the ship width, and the hold depth, a ship container position map is constructed, which can interact with the container number recognition system and the terminal operation system to realize the automatic issuance and verification of operation instructions.

[0078] Embodiment 2

[0079] This embodiment provides a method for ship scanning and spreader anti-collision based on the fusion of laser and vision, which is implemented based on the ship scanning and spreader anti-collision system based on the fusion of laser and vision proposed in the above-mentioned Embodiment 1, and includes the following steps:

[0080] Step S1: Initialize the installation parameters of the laser scanner and the shore crane structure parameters. The installation parameters of the laser scanner include the installation height, installation angle, distance from the laser to the spreader, etc.; the shore crane structure parameters include the saddle beam height, saddle beam width, front end height of the main girder, etc. Among them, the installation parameters of the laser scanner and the shore crane structure parameters are input into the ship scanning laser controller in the form of a configuration file for subsequent data processing by the ship scanning laser controller.

[0081] Step S2: Based on the installation parameters of the laser scanner, taking the installation position of the laser scanner as the origin, convert the laser scan polar coordinate data to the laser rectangular coordinate system. Based on the right-hand coordinate system, with the shore crane trolley running direction as the X-axis, the shore crane crab running direction as the Y-axis, and the shore crane hoisting direction as the Z-axis, the rotation angle (such as trolley tilt) clockwise (viewed from the origin) = positive angle, counterclockwise = negative angle.

[0082] Step S3: Calibrate the vision acquisition device, that is, calibrate the internal and external parameters of the binocular camera.

[0083] Based on the parallax principle, binocular stereo vision is proposed. A similar triangle model is formed between the image planes of the two cameras and the object to be measured. As Figure 3 shown, given the parameters and image coordinate relationship between the two cameras, the three-dimensional coordinate positions of the spatial feature points of the object in the images taken by the two cameras can be calculated. The baseline distance refers to the distance between the projection centers of the two cameras, denoted as B. The two cameras view the same feature point P of the spatial object at the same time, and obtain the image coordinates of point P on the left camera (left eye) and the right camera (right eye) as P left =(X left , Y left ), P right =(X righ , Y righ ). According to the principle of setting two parallel cameras, the two cameras are on the same plane, so the Y coordinates of the image coordinates of the feature point P are the same, that is, Y left =Y righ =Y. Then, according to the principle of triangle similarity, we get:

[0084]

[0085]

[0086]

[0087] The calibration of the vision acquisition device is completed in the following order, including: for camera calibration, based on the theory of the camera imaging model, the Zhang's planar calibration method is adopted, and a planar checkerboard template is used to perform stereo calibration on the binocular camera; the stereo calibration is corrected, and the internal and external parameters and distortion parameters of the camera are obtained through experiments. By comparing the camera parameters before and after correction, the parameter values in the horizontal viewing state are obtained. In this embodiment, the internal and external calibration parameters of the camera are input into the ship scanning vision controller in the form of a configuration file to prepare for subsequent calculation of three-dimensional space coordinates.

[0088] Step S4: Based on vision acquisition, obtain the ship's spatial position data. That is, obtain continuous two-dimensional images of the ship's contour, and based on the obtained two-dimensional image coordinates and the internal and external parameters of the binocular camera calibration, obtain the three-dimensional image of the ship's contour in space.

[0089] The two-dimensional image coordinates are the coordinates of the photo pixel points. For the two-dimensional images obtained based on vision acquisition, the target container is identified from different colored containers and its image center coordinates are obtained. The calculation method of the image coordinates includes: intercepting the target image, target template matching, target image preprocessing, target image binarization, target recognition matching, parameter optimization, target coordinate calculation, and feedback of image coordinates. Among them, the preprocessing of the obtained two-dimensional image includes defogging processing, image segmentation, etc. Among them, for two-dimensional images under foggy weather conditions, defogging processing is performed. The defogging processing process is completed by using dark channel extraction. Through the objective evaluation standard of the defogging effect, the image feature information is extracted from the foggy two-dimensional image; considering the different colors of the containers in the two-dimensional image, image segmentation is realized based on the color model.

[0090] Step S5: Based on laser scanning, obtain the ship's contour point cloud data. That is, obtain laser scanning data to form the ship's contour point cloud data, that is, obtain the contour data of the working box area on the three-dimensional container.

[0091] When the trolley of the quay container crane moves, the ship scanning laser controller receives the signal of "trolley movement signal". At this time, the ship scanning laser controller clears the previous box area configuration file data according to the received signal.

[0092] When the quay container crane reaches a new area, the "trolley movement signal" becomes 0. At this time, the ship scanning laser controller rotates the 2D laser scanner forward by a certain angle and scans the contours of adjacent brackets and the working bracket, and the two 2D laser scanners also scan and update the configuration file in real time.

[0093] When the quay crane trolley moves forward, the 2D laser scanner rotates forward by a certain angle to detect obstacles and update the contours of the current and adjacent compartments; when the quay crane trolley moves backward, the 2D laser scanner rotates vertically towards the ground to update the obstacle contours of the current and adjacent compartments.

[0094] Step S6: Integrate the stereo space image of the ship's contour acquired based on vision and the point cloud data of the ship's contour acquired based on laser scanning.

[0095] Vision acquisition obtains stereo images, preprocesses the images, identifies the targets in the images, and extracts feature points to obtain the information in the images; laser scanning performs coordinate transformation, plane segmentation, and plane fitting on the acquired target point cloud data to obtain the plane where the containers on the ship are placed; data integration is performed through feature point depth information restoration, etc.

[0096] Step S7: Analyze and process the integrated data based on the encoder information data and in combination with the quay crane contour data acquired by scanning to determine the final ship's contour.

[0097] In this embodiment, the integrated data obtained in step S6 is divided by using the encoder information in combination with the quay crane contour information acquired by laser scanning, the points on the ship are determined, all the points on the ship are counted, and their corresponding positions in the grid map are determined, thereby constituting the ship's contour and completing the accurate recognition of the ship's contour.

[0098] As another implementation manner, on the basis of the above ship's contour recognition, this embodiment also provides a method for positioning the locking holes of the target container, including:

[0099] Based on vision acquisition, continuous two-dimensional images of the container stacking contour on the ship are acquired, and based on the coordinates of the acquired two-dimensional images and the internal and external parameters of the binocular camera calibration, the spatial position image of the container stacking on the ship is obtained;

[0100] The acquired images are preprocessed and edge detected to identify the target container and determine the three-dimensional coordinates of the container locking hole, so as to meet the accurate positioning and container grabbing of the target container on the ship.

[0101] As another implementation manner, on the basis of the above ship's contour recognition, this embodiment also provides a method for preventing the spreader from colliding, including:

[0102] Based on laser scanning and vision acquisition, the position, lifting height, and spreader height of the quay crane trolley during the lifting process are detected, and at the same time, the height of the highest obstacle in the current working bay is detected respectively;

[0103] Based on the detection data collected visually, the data of the laser scan is corrected. According to the highest obstacle height and the spreader height after correction, collision judgment is performed, and control instructions and alarms are issued according to the collision judgment results to control the movement of the quay crane trolley and the lifting of the spreader, so as to realize the anti-collision protection of the spreader in the trolley running direction, the soft landing in the lifting direction, and the anti-collision protection of the spreader in the adjacent bays in the trolley running direction.

[0104] In this embodiment, during the lifting process, when the bottom of the lifted container is 0.5 meters higher than the container on the ship, the trolley can move at full speed; when there is a tendency of collision, the trolley is controlled to stop moving; the system regards the movement of the quay crane on the shore exceeding 30 cm as changing the current row.

[0105] In addition, based on laser scanning and visual acquisition, the position of the quay crane trolley and the stacking position and height of the containers on the ship are detected in real time. At the same time, the positions and heights of the containers in each row under the quay crane on the shore are detected in real time respectively to assist in realizing the soft landing of the containers on the sea and land sides.

[0106] Each step involved in the above Embodiment 2 corresponds to that in Embodiment 1. For the specific implementation manner, reference may be made to the relevant description part of Embodiment 1.

[0107] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0108] Although the specific implementation manner of the present invention is described above in conjunction with the drawings, it is not a limitation on the protection scope of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or deformations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.

Claims

1. A ship scanning and spreader anti-collision system based on the fusion of laser and vision, characterized in that It includes a laser scanner, a vision acquisition device, a ship scanning vision controller, and a ship scanning laser controller; The laser scanner is used to receive the scanning instruction issued by the ship scanning laser controller, continuously scan the ship contour and the container stacking contour on the ship according to the scanning instruction, and upload the scanning data to the ship scanning laser controller; The laser scanner selects a 2D laser scanner, and 3 laser scanners are adopted; The first 2D laser scanner and the second 2D laser scanner are fixedly installed on the shore bridge trolley, on the left and right sides of the center of the shore bridge trolley, and move along the sea-land side with the shore bridge trolley, so as to synchronously scan the contour data of the upper box area on the ship downward along the bay direction; The third 2D laser scanner is also fixedly installed on the shore bridge trolley, at the relative position of the center of the shore bridge gantry, and moves along the sea-land side with the shore bridge trolley, so as to scan the contour data of the container ship box area downward along the bay row direction; The vision acquisition device is used to receive the image shooting instruction issued by the ship scanning vision controller, continuously take images of the ship contour and the container stacking contour on the ship according to the shooting instruction, and upload the image data to the ship scanning vision controller; The vision acquisition device selects a camera, and 2 cameras are adopted; The first camera and the second camera are both fixedly installed on the shore bridge trolley, respectively installed on the left and right sides of the center of the shore bridge trolley, and the 2 cameras form binocular vision, and move along the sea-land side with the shore bridge trolley, so as to take the contour data of the container ship box area downward along the bay row direction; The ship scanning vision controller is used to calculate the conversion parameters between the pixel point pairs corresponding to each other in the two images in the three-dimensional space through image processing according to the received image data, that is, receive two images of the same cabin or ship tank obtained by the first camera and the second camera with different perspectives, obtain the three-dimensional space image of the ship contour, and send the three-dimensional space image of the ship contour to the ship scanning laser controller; The ship scanning laser controller is used to obtain the ship contour point cloud data according to the received scanning data, and at the same time correct the ship contour point cloud data according to the received three-dimensional space image of the ship contour to obtain the final ship contour; The ship scanning vision controller detects the height of the highest obstacle at the current working bay according to the three-dimensional space image of the ship contour, and transmits the visually detected height of the highest obstacle to the ship scanning laser controller; The ship scanning laser controller detects the position of the shore bridge trolley and the height of the spreader, and detects the height of the highest obstacle at the current working bay according to the ship contour point cloud data; The ship scanning laser controller performs data correction according to the received visually detected height of the highest obstacle, makes a collision judgment according to the corrected height of the highest obstacle and the height of the spreader, and issues a control instruction and an alarm according to the collision judgment result to control the movement of the shore bridge trolley and the lifting of the spreader.

2. The ship scanning and spreader anti-collision system based on laser and vision fusion according to claim 1, characterized in that, It also includes a shore bridge PLC controller, a hoist encoder, and a shore bridge trolley encoder; The shore bridge PLC controller is electrically connected to the hoist encoder and the shore bridge trolley encoder, and is used to obtain encoder information data and spreader open / close lock state data, and send the obtained data to the ship scanning laser controller; The quay crane PLC controller is also used to receive the control instructions issued by the ship scanning laser controller, and according to the control instructions, control the quay crane trolley encoder and the hoisting encoder, and then control the movement of the quay crane trolley and the lifting and lowering of the spreader.

3. The ship scanning and spreader anti-collision system based on laser and vision fusion according to claim 2, characterized in that, It also includes a DP module. The DP module is a conversion module, which is connected to the quay crane PLC controller and the ship scanning laser controller, and is used for communication protocol conversion and communication data transmission between the quay crane PLC controller and the ship scanning laser controller.

4. The ship scanning and spreader anti-collision system based on laser and vision fusion according to claim 1, characterized in that It also includes an optoelectronic module and a display module. The ship scanning laser controller converts the obtained ship profile into an image through the optoelectronic module and performs visual display through the display module.

5. The ship scanning and spreader anti-collision system based on laser and vision fusion according to claim 2, characterized in that, The ship scanning vision controller locates the ship guide groove and the target container lock hole according to the ship profile three-dimensional space image, and transmits the positioning information of the ship guide groove and the target container lock hole detected by vision to the ship scanning laser controller; The ship scanning laser controller issues control instructions to the quay crane PLC controller according to the received positioning information of the ship guide groove and the target container lock hole, combined with the detected position of the quay crane trolley, the spreader height and the quay crane status data; The quay crane PLC controller controls the quay crane trolley encoder and the hoisting encoder according to the received control instructions, and then controls the movement of the quay crane trolley and the opening / closing and lifting of the spreader.

6. The ship scanning and spreader anti-collision system based on laser and vision fusion according to claim 1, characterized in that, The system also includes an alarm module, which is used to perform collision alarm according to the warning information issued by the ship scanning laser controller.

7. A ship scanning and spreader anti-collision method based on the fusion of laser and vision, which is implemented based on the ship scanning and spreader anti-collision system based on the fusion of laser and vision according to any one of claims 1-6, and is characterized by including: Initialize the installation parameters of the laser scanner and the quay crane structure parameters; Based on the installation parameters of the laser scanner, with the installation position of the laser scanner as the origin, convert the laser scan polar coordinate data to the laser rectangular coordinate system; Calibrate the vision acquisition device; Based on vision acquisition, obtain continuous two-dimensional images of the ship profile, and based on the obtained two-dimensional image coordinates and the internal and external parameters calibrated by the binocular camera, obtain the ship profile three-dimensional space image; Based on laser scanning, obtain laser scan data to form ship profile point cloud data; Fuse the ship profile three-dimensional space image based on vision acquisition and the ship profile point cloud data based on laser scanning to identify the ship profile.

8. The method for ship scanning and spreader anti-collision based on laser and vision fusion according to claim 7, characterized in that It also includes a method for positioning the target container lock hole, including: Based on vision acquisition, obtain continuous two-dimensional images of the container stacking profile on the ship, and based on the obtained two-dimensional image coordinates and the internal and external parameters calibrated by the binocular camera, obtain the spatial position image of the container stacking on the ship; Preprocess and perform edge detection on the obtained image to identify the target container and determine the three-dimensional coordinates of the container lock hole.

9. The method for ship scanning and spreader anti-collision based on laser and vision fusion according to claim 7, characterized in that it further Including: Based on laser scanning and vision acquisition, detect the position, hoisting height and spreader height of the quay crane trolley during the hoisting process, and at the same time detect the height of the highest obstacle at the current working bay respectively; Based on the detection data collected visually, the data of the laser scan is corrected. According to the highest obstacle height and the spreader height after correction, collision judgment is performed, and control instructions and alarms are issued according to the collision judgment results to control the movement of the quay crane trolley and the lifting of the spreader.

Images