A high-precision real-time detection system for ship hold position for automated bridge cranes

Abstract

The present invention discloses a high-precision real-time detection system for ship hold spaces for automated bridge cranes. The system uses a data acquisition unit to send the acquired synchronous data to a detection and positioning unit. The synchronous data includes PLC data, images, and point cloud data. The detection and positioning unit calculates the models and positions of all containers or empty spaces on the ship hold space, and then sends the model and position information to a bay layer calculation unit to obtain its bay layer information. The bay map automatic reasoning unit obtains a complete bay map, and finally sends the seaside target container space to a control system through a target container space planning and decision unit. When the dynamic correction unit is actually performing loading and unloading operations, the loading and unloading signals acquired by the data acquisition unit automatically trigger the dynamic correction unit to dynamically correct the bay map. The present invention solves the problems of low detection accuracy and poor adaptability to complex and changing port environments in existing visual or non-visual ship hold space detection systems.

Description

Technical Field

[0001] The present invention relates to the technical field of container freight transport at docks, and in particular to a high-precision real-time detection system for ship cabin space facing automated bridge cranes. Background Art

[0002] In recent years, driven by big data and artificial intelligence technologies, traditional terminals have gradually been automated. Ship hold sensing is a key component of this automation transformation. It not only assists intelligent tallying systems in counting the container cargo currently on board, but also provides the automated control system with the target location for seaside container placement, thereby improving automated operation efficiency. Ship hold positions are typically represented by a bay map, which includes at least the bay number, row number, and layer number of the container. For automated bridge cranes, the bay map should also include information such as the container type and physical position of the container, thereby assisting the spreader in automatically adjusting its size and completing the container placement trajectory.

[0003] The two metrics used to evaluate bay maps are accuracy and positioning accuracy. Accuracy reflects the degree to which the bay layer information in the bay map matches the actual ship map during each operation, while positioning accuracy refers to the position accuracy of each container or empty space in the bay map. These two metrics are of great significance to automated bridge cranes, mainly reflected in the following aspects:

[0004] First, when performing a task, the automated bridge crane automatically determines the target container position for the next operation based on the current bay position map. This target position information is then transmitted to the automatic control system, enabling automated container pickup and release operations on the seaside. Deviations from the bay position map will directly result in discrepancies between the planned target container position and the intended one, making it impossible to accurately assess the vessel's loading situation. In severe cases, this can lead to dangerous accidents such as overloading or capsizing.

[0005] Second, when the ship's hold detection system provides the seaward positioning information for the target container, the control system directs the spreader to the seaward side to complete the pick-up and release operation. To improve operational efficiency, the control system must plan a safe and shortest trajectory based on the position information in the bay map. If the bay map's positioning accuracy is low, friction between the spreader and the hold can occur during the planning process, potentially damaging the container or causing a collision or other safety incident.

[0006] There are two main methods for ship hold detection: non-visual inspection and visual inspection. Non-visual inspection relies solely on PLC data to determine the actual container placement, and is updated after each operation based on existing ship chart information. Visual inspection, on the other hand, utilizes visual sensors (such as cameras or lidar) to perceive the container type in the hold, typically providing richer information.

[0007] Patent CN111170158A discloses a non-visual ship hold position detection method, which uses the actual operation box position of the last operation as a benchmark and uses PLC data (such as trolley position, sling height, etc.) to calculate the box layer information for each operation. Patent CN112462373A discloses a multi-sensor fusion visual detection method, which obtains PLC, camera and lidar data at certain moments, uses point clouds to locate the center of mass of the container, and at the same time gives the container more semantic information based on the image. Patent CN111170158A discloses a container loading bay automatic recognition system and method, which calculates the bay position of the container based on the relative displacement between the current container and the previous container's traveling trolley, using the previous container's bay position as a relative benchmark; at the same time, the system adds automatic and manual bay position verification functions, and manual intervention and correction through software is required for bay position verification failures. Patent CN109795892A also discloses a multi-sensor ship scanning algorithm. This method selects a fixed reference, uses the initial container position as the origin, calculates the new container position information for subsequent loading, and uses a simple height correction formula to correct the floating position of the fixed reference. Although the above methods have certain theoretical significance, they have certain limitations in the harsh port environment with frequent winds, rain, and ebbs and flows. The specific analysis is as follows:

[0008] First, the Bayeux chart has low positioning accuracy, making it prone to safety accidents and unsuitable for automated bridge crane scenarios. On the one hand, rope-driven spreaders are subject to wind and inertia during actual container placement, causing the trolley or gantry to swing. PLC information cannot directly reflect the precise operating position of the container, reducing the accuracy and positioning precision of the Bayeux chart. On the other hand, due to weather conditions such as tides or strong winds, moored ships will experience dynamic fluctuations with the sea surface. When the fluctuation value exceeds the size of a container, the Bayeux chart will become abnormal, causing the automated system to operate inefficiently or even stagnate, and is highly susceptible to safety accidents.

[0009] Secondly, visual perception methods are highly sensitive to adverse weather conditions such as rain, fog, and haze, and it is difficult to achieve a high detection rate for non-standard berthed ships or containers. Camera lenses are easily fogged and blurred, while lidars have the anomaly of missing point clouds when there is water accumulation in the container space or reflective surface materials. At the same time, due to the non-uniform manufacturing standards of berthed ships at ports and the wide variety of ship spaces or containers, even the more advanced AI detection technologies find it difficult to achieve a high detection rate when training data is limited. In addition, the method of patent CN112462373A is highly sensitive to noise. When the sensor vibrates, a single target object may be located at a different container space, which greatly reduces the accuracy of the bay map.

[0010] Third, there is a lack of robust bay map self-calibration function. When the bay map is abnormal, it will interfere with the automatic system's planning and decision-making on the target container position, thus seriously affecting the safety and efficiency of the operation. When the sensing unit has missed detection or missing data, the detection system cannot make a good balance of the loading situation on the ship, thereby sending unexpected seaside target container position information to the control system. In serious cases, dangerous accidents such as overloading or rollover may occur. In addition, when the height information is missing, the path planned by the automatic system may collide with the missed container. Although patent CN111170158A includes automatic and manual verification modules for the bay map, if the verification fails, manual intervention is still required to repair the bay map through software. This strategy requires additional human resources and will seriously affect the efficiency of automatic operations.

[0011] To this end, we propose a high-precision real-time detection system for ship cabin position for automated bridge cranes. Summary of the Invention

[0012] Based on the technical problems existing in the background technology, the present invention proposes a high-precision real-time ship cabin position detection system for automated bridge cranes to solve the problems of low detection accuracy and poor adaptability to complex and changeable port environments of existing visual or non-visual ship cabin position detection systems.

[0013] The present invention provides the following technical solutions: a high-precision real-time detection system for ship cabin positions for automated bridge cranes, comprising a data acquisition unit, a detection and positioning unit, a bay layer calculation unit, a dynamic correction unit, a bay position map automatic reasoning unit, and a target container position planning and decision unit;

[0014] The data acquisition unit is used to send the acquired synchronization data to the detection and positioning unit, and the synchronization data includes PLC data, image and point cloud data;

[0015] The detection and positioning unit is used to calculate the model and position of all containers or empty spaces on the ship's hold, and then send the model and position information to the bay layer calculation unit to obtain its bay layer information, and then obtain a complete bay map through the bay map automatic reasoning unit. Finally, the target container space on the sea side is sent to the control system through the target container space planning and decision unit;

[0016] The dynamic correction unit is used to automatically trigger the dynamic correction unit to dynamically correct the bay position map when the loading and unloading operations are actually carried out.

[0017] Preferably, the data acquisition unit includes a PLC controller, a camera, a laser radar, and a data synchronizer; the PLC controller is used to obtain the operating status of the bridge crane in real time, and the camera and the laser radar can move synchronously with the position of the trolley, and are used to obtain the image I and laser point cloud P of the cabin respectively; the data synchronizer is installed on the control server, and is used to package and obtain the PLC data, image data and point cloud data at the same time.

[0018] Preferably, the detection and positioning unit is composed of a target detection and positioning algorithm, a point cloud segmentation algorithm, and a multi-source multi-frame data fusion algorithm;

[0019] Its input is a continuous multi-frame data packet, each of which contains synchronized PLC data, image I and laser point cloud P;

[0020] It outputs the model and weighted position of the container or empty space in the ship's hold.

[0021] Preferably, the input of the shell layer calculation unit is the model and posture information of the ship hold container or empty position and PLC data. During the initial scan, the loading and unloading signal is not triggered. The shell layer calculation unit selects the reference box and calculates the shell layer number of all box positions to obtain the shell position map for the first time.

[0022] Preferably, the input of the dynamic correction unit is the actual container landing signal of the PLC sea side. When the sea side loading operation is carried out, the loading signal is triggered, the dynamic reference is selected as the actual operating container position, and the floating deviation of the cabin position is calculated in combination with the hoist status, and the bay position map is updated; when the sea side unloading operation is carried out, the sea side unloading signal is triggered, the dynamic reference is selected as the actual operating adjacent bottom container, and the floating deviation of the cabin position is calculated in combination with the hoist status, and the bay position map is updated.

[0023] Preferably, the input of the bay map automatic reasoning unit is the original bay map obtained by perception, and the missed row information is inferred based on the container position information of adjacent rows of the cabin space. At the same time, the inaccurate container height and layer number caused by the missing point cloud are inferred, and finally the optimized bay map is obtained.

[0024] Preferably, the target container position planning and decision-making unit is triggered to run by the PLC land measurement container signal, and the input is the optimized bay position map. Combined with the bay layer and height information in the bay position map, the loading situation of the ship cabin space is weighed, and a safe seaside operation target container position instruction is sent to the control system. The instruction content specifically includes the target container position information, safety height and cabin inclination angle.

[0025] The present invention provides a high-precision real-time detection system for ship cabin position for automated bridge cranes. Compared with the existing technology, the beneficial effects are as follows:

[0026] First, the present invention proposes a robust and high-precision Bayeux mapping strategy. This strategy selects different dynamic benchmarks for the loading and unloading processes, and trains an AI model to detect the swing state of the spreader at the moment of loading and unloading. The Bayeux map is then corrected in real time based on the PLC and the swing amplitude of the spreader in the direction of the trolley and the truck, which greatly improves the correction accuracy and effectively reduces the risk of safety accidents caused by inaccurate Bayeux maps in automated bridge cranes.

[0027] Secondly, a multi-source and multi-frame two-dimensional data fusion method is proposed. Each bin is clustered as a single cluster, and the multi-frame data in each cluster are voted and weighted averaged. This can not only effectively make up for the limitations of single-frame detection, but also eliminate the impact of unknown factors such as sensor vibration on positioning accuracy and reduce sensitivity to noise.

[0028] Third, the present invention proposes a missing bay reasoning method based on neighborhood information, which reduces manual intervention and improves the reliability and security of the automatic system.

[0029] Fourth, the present invention proposes for the first time a target container space planning and decision-making unit, which combines the bay layer and height information in the bay position diagram to weigh the loading situation of the ship's hold space, thereby sending safe seaside operation target container space instructions to the control system to avoid dangerous accidents such as overloading or rollover. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the dynamic ship space detection system combining perception and reasoning according to the present invention;

[0031] Figure 2 Schematic diagram of the data acquisition unit of the present invention;

[0032] Figure 3 is a schematic diagram of a detection and positioning unit of the present invention;

[0033] Figure 4 Schematic diagram of the Bayesian layer calculation unit (right) and the dynamic correction unit (left) of the present invention;

[0034] Figure 5 Schematic diagram of the Bayesian graph automatic reasoning unit of the present invention;

[0035] Figure 6 Schematic diagram of the target slot planning and decision-making unit of the present invention;

[0036] Figure 7 The diagram is a schematic diagram of the hull floating changes before and after the box is placed;

[0037] Figure 8 Schematic diagram of the hull floating changes before and after the grab box;

[0038] Figure 9 Schematic diagram of the planned seaside container storage target location;

[0039] Figure 10 This is a schematic diagram of the planned target container location for seaside container grabbing. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] The present invention provides a technical solution: a high-precision real-time detection system for ship cabin positions for automated bridge cranes, which mainly consists of six parts: a data acquisition unit, a detection and positioning unit, a bay layer calculation unit, a dynamic correction unit, a bay position map automatic reasoning unit, and a target container position planning and decision unit. The system structure diagram is shown in the figure below. Figure 1 shown.

[0042] Overall, the data acquisition unit sends the acquired synchronous data (including PLC data, images and point clouds) to the box detection and positioning unit to calculate the model and posture of all containers or empty spaces on the ship's hold, and then sends the model and posture information to the Bayesian layer calculation unit to further obtain its Bayesian layer information, and finally obtains the complete Bayesian map information through the Bayesian map post-processing unit.

[0043] Data acquisition unit, such as Figure 2 As shown, it includes PLC controller, camera, laser radar and other hardware equipment, as well as data synchronizer. The PLC controller is installed in the bridge crane control room to obtain the bridge crane operation status in real time, including but not limited to the trolley value x car , the car value is y car , spreader height z hanger , spreader extension status size and loading and unloading signal sig l with sig u The camera and lidar, both with their lenses facing vertically downward, are mounted adjacent to each other at the bottom of the crane trolley frame. They are used to acquire the cabin image I and laser point cloud P, respectively. The trolley frame moves synchronously with the trolley's position. A data synchronizer, installed on the control server, packages and captures PLC data, image data, and point cloud data at the same time.

[0044] Box type detection and positioning unit, such as Figure 3As shown in the figure, the input is a continuous multi-frame data packet, each of which contains synchronized PLC data, image I and laser point cloud P. The unit mainly consists of a box detection and positioning module, a point cloud positioning module, and a multi-frame data fusion algorithm, and finally outputs the model and weighted pose of the container or empty space in the ship's hold {(class i ,x i ,y i ,z i Models include but are not limited to 20-foot containers, 40-foot containers, 45-foot containers, double 20-foot containers, and empty cargo spaces.

[0045] Bayesian layer computing unit, such as Figure 4 As shown, the input is the model and position information of the container or empty space in the ship cabin {(class i ,x i ,y i ,z i )} and PLC data, including but not limited to PLC loading and unloading signals sig l with sig u , the car position x car , the cart position y car and spreader height z hanger In the first scan, the loading and unloading signal is not triggered, the reference box is selected and the layer number of all box positions is calculated, and the position map {(class i ,x i ,y i ,z i ,bay i ,row,tier i )}; During the loading and unloading operation, the loading and unloading signal is triggered, the dynamic reference is selected as the actual operating container position, and the floating deviation of the ship's hold position is calculated in combination with the spreader status, and the bay position map is updated.

[0046] Baymap post-processing unit, such as Figure 5 As shown in the figure, the input is the original bay map obtained by perception. The missed row information is inferred based on the container position information of adjacent rows of the cabin. At the same time, the inaccurate information of the container height value z and the floor height tier caused by the missing point cloud is inferred, and finally the optimized bay map is obtained.

[0047] Target slot planning and decision-making unit, such as Figure 6 As shown in the figure, this unit is triggered by a PLC landing signal. Its input is an optimized bay map. Combining the bay layers and height information in the bay map, it weighs the loading status of the hold and sends a safe seaside target position instruction to the control system. The instruction specifically includes the target container position information, the safe height (height), and the hold inclination angle (theta).

[0048] The specific operation of this system includes the following steps:

[0049] Step 1: When the ship docks for the first time and before loading and unloading operations begin, a bay scanning action is required, that is, the automatic bridge crane controls the trolley to move from the land side to the sea side until the trolley's movement track covers the entire ship's hold and then stops moving. During the movement, the data acquisition unit collects data streams in real time and finally obtains a set of data packets {(plc i , I i , P i )}. The PLC data stored in each data packet includes the car value x car , the car value is y car , spreader height z hanger And loading and unloading signal sig l with sig u wait.

[0050] Step 2: Parse the data packets obtained in step 1 frame by frame and pass them into the box detection and positioning unit to detect the box type and position information, and perform weighted fusion on the calculation results of multiple consecutive frames.

[0051] (1) First, the data packet is parsed frame by frame to obtain the single-frame synchronized PLC, image and laser point cloud data, denoted as plc i , I i , P i .

[0052] (2) Using AI box detection algorithm, image I i Input into the trained target detection model and obtain the target box {ROI} and box type information {class′} of all containers or empty spaces in the field of view.

[0053] (3) According to the container or empty space box information, the actual size of the container or empty space surface can be queried. The target positioning algorithm is used to calculate the 6-DOF position of the target relative to the camera {(x c ,y c , z c , rx c ,ry c , rz c )}, where x c ,y c , z c is the translation posture, rx c ,ry c , rz c The pose is transformed into the car coordinate system through the pose transformation matrix of the camera and car coordinate system, which is recorded as {(x v ,y v , z v , rx v ,ry v, rz v )}.

[0054] (4) Using the position transformation relationship between the laser radar and the car coordinate system, the point cloud data P i Convert to the car coordinate system and calculate the point cloud data P i Then, based on the target pose (x v ,y v , z v , rx v ,ry v , rz v ), extract the corresponding point cloud of each target object, and calculate the average height zp of the internal points of the target object. If the number of internal point clouds of the target object is less than the point count threshold δ, it is considered that there is a point cloud missing anomaly, and the average height zp is set to the maximum value of 99m. At the same time, the average height zp of the point cloud is replaced by z in (3). v , get the precise positioning value of all targets, record {(x v ,y v , z p )}, the positioning value only considers the translational degree of freedom.

[0055] (5) Based on the parking space value {x car} and large parking space value {y car}, calculate the positioning value of all targets in the single frame data in the bridge crane coordinate system {(x t ,y t , z t )}, abbreviated as {pose t}, the formula is as follows:

[0056] x t =x v +x car ,

[0057] y t =y v +y car ,

[0058] z t =z p ,

[0059] (6) Repeat (1) to (5) until all frames in the data packet are parsed and processed. Integrate the target object models and poses of all frames together to form a new perception instance set Φ, denoted as {(class′, pose t )}, using the Euclidean clustering algorithm, the instances with pose spacing less than the threshold ε are grouped into a series of subsets {Φ′}. Using the voting strategy, the model with the largest number of appearances in each subset is used as the box type of the subset:

[0060] class=MaxCount(class′),

[0061] (7) And perform weighted fusion on the target pose of each subset, the formula is as follows:

[0062]

[0063]

[0064]

[0065] Where N is the subset size.

[0066] (8) Finally, the real container or empty storage location perception set {(class,x,y,z)} is obtained.

[0067] Step 3: The container or empty position perception set obtained in step 2 is passed into the bay layer calculation unit to calculate the original perception bay map.

[0068] (1) According to the target car position x, the perception set is rearranged in ascending order.

[0069] (2) From the rearranged perception set, select the information of the i-th target object one by one (class i ,x i ,y i ,z i ), when first selected, it needs to be selected as the reference box, and its shell number bay0, row number row0 and tier number tier0 are preliminarily calculated according to the following formula:

[0070]

[0071] row0=0,

[0072]

[0073] (3) where c x , c y with c z Here, [ ] represents the width, length, and height of a standard 20-foot high-cube container, respectively. The formula [·] is rounded to the nearest integer. Therefore, the base container is chosen as (class0, x0, y0, z0, bay0, row0, tier0). It's worth noting that we use a standard 20-foot high-cube container as the baseline for calculations. Although container height standards may vary, the difference between high-cube and low-cube containers is generally less than 15%.

[0074] (4) If the reference box already exists, the Bayesian layer estimation formula for the subsequent calculation of the reference box is as follows:

[0075]

[0076]

[0077]

[0078] (5) Therefore, the cabin position information obtained during the initial bay scan is {(class,x,y,z,bay,row,tier)}.

[0079] Step 4: The ship cabin position information obtained during the initial bay position scan in step 3 is transferred to the bay position map post-processing unit to calculate the inferred bay position map.

[0080] (1) The cabin information in the original bay map is rearranged according to the row number from small to large, where the smallest row number is the base row number 0 and the largest row number is row max .

[0081] (2) Determine whether there is a valid container space in row 1. If not, there is a missed inspection in the current row. Query the valid container space information of the nearest row in the bitmap (class v ,x v ,y v ,z v ,bay v ,row v ,tier v ) and fill the valid information into the current invalid ranking.

[0082] (3) If there is a valid bin in the first row, it is necessary to further determine whether there is a point cloud missing in the current bin. There are the following situations:

[0083] a. If there is a missing point cloud and the box type is a container:

[0084] The first step is to query the nearest container of the same model in the bay map, and replace the z value and tier value of the original container with its height and layer number; if there is no other container of the same model, replace it with the height and layer number of the nearest neighboring container; if there is no other container, replace it with the height and layer number of the nearest neighboring empty container plus a standard container height c. z When replacing the current container height, the empty storage space height plus 1 is used as the layer number of the current container.

[0085] b. If there is a missing point cloud and the box type is an empty position:

[0086] Then first query the other empty locations closest to the container in the bay map, and replace the z value and tier value of the original container with its height and layer number; if there is no other valid empty location, but there is a container, then subtract the height of a standard container c from the height of the nearest neighbor container. z to replace the current height, and use its layer number minus 1 as the layer number of the current empty storage location; if there is no other empty storage location or container, the default layer number is -1.

[0087] c. If there is no missing point cloud, the current box position does not need to be corrected.

[0088] (4) Add 1 to the row number and repeat the process (3) until the maximum row number is found. max Finally, the corrected Bayer map will be obtained.

[0089] Step 5: After the loading and unloading operation begins, monitor the PLC's loading signal sig in real time l and unloading signal sig u , and is divided into the following two cases:

[0090] Case 1. PLC loading signal sig l If it suddenly changes from false to true, it means that the packing operation has just been completed. Get the synchronized PLC and image data at that moment.

[0091] (1) Input the image into the spreader AI detection service to obtain the target frame of the spreader, and use the target positioning algorithm to obtain the swing posture of the spreader relative to the camera. Convert the spreader swing posture to the trolley coordinate system and obtain the swing value of the spreader in the trolley direction as offset x , the trolley direction swing value is offset y At the same time, the state of the spreader is read from the PLC to obtain the actual position of the current operation box (x r ,y r , z r ), calculated as follows:

[0092] x r =x car +offset x ,

[0093] y r =ycar+offset y ,

[0094] z r =z hanger ,

[0095] (2) According to the actual position of the operating box, search the box position information with the closest horizontal distance from the current position map. The search formula is:

[0096] min(x r -x j ) 2 +(y r -y j ) 2 ,

[0097] Among them, (class j , x j ,y j , z j ,bay j , row j , tier j ) is any box location information in the current bay map. After searching and matching, the box location information with the closest horizontal distance can be obtained as (class n , x n ,y n , z n ,bay n , row n , tier n ).

[0098] (3) Since this is a loading operation, this operation box is superimposed on the original container position, so the calculation formula for the container position information of the current operation box is:

[0099] bay r =bay n ,

[0100] row r =row n ,

[0101] tier r =tier n +1,

[0102] The box type information can be directly obtained from the spreader extension state size in the PLC, that is,

[0103] class r =size,

[0104] In the bitmap, the box position information to be covered is deleted and the box position information of the current operation box is added (class r , x r ,y r , z r ,bay r , row r , tier r ).

[0105] (4) Since the position of the actual operation box is updated in real time and is not affected by special conditions such as overload, environment or tide, the reference box is dynamically updated to the current operation box, and the positions of other boxes in the bay map are corrected. Figure 7 Taking the height floating error as an example, the floating change of the hull before and after the box is placed is shown. The error calculation formula corresponding to the three degrees of freedom is as follows:

[0106] Δz=(z r -c Z )-z n ,

[0107] Δx=x r -x n ,

[0108] Δy=y r -y n ,

[0109] Therefore, the posture correction formula for other boxes in the Bay Area Map is:

[0110] x′ i =x i +Δx,

[0111] y′ i =y i +Δy,

[0112] z′ i =z i +Δz,

[0113] Case 2. PLC unloading signal sig u If the value suddenly changes from false to true, it means that the unloading operation has just been completed. Get the synchronized PLC and image data at that moment.

[0114] (1) Input the image into the spreader AI detection service to obtain the target frame of the spreader, and use the target positioning algorithm to obtain the swing posture of the spreader relative to the camera. Convert the spreader swing posture to the trolley coordinate system and obtain the swing value of the spreader in the trolley direction as offsetx and the swing value in the truck direction as offset y At the same time, the state of the spreader is read from the PLC to obtain the actual position of the current operation box (x r ,y r , z r ), calculated as follows:

[0115] x r =x car +offset x ,

[0116] y r =ycar +offset y ,

[0117] z r =z hanger ,

[0118] (2) According to the actual position of the operating box, search the box position information with the closest horizontal distance from the current position map. The search formula is:

[0119] min(x r -x j ) 2 +(y r -y j ) 2 ,

[0120] Among them, (class j , x j ,y j , z j ,bay j , row j , tier j ) is any box location information in the current bay map. After searching and matching, the box location information with the closest horizontal distance can be obtained as (class n , x n ,y n , z n ,bay n , row n , tier n ).

[0121] (3) Since it is an unloading operation, after the current container is taken away, the bottom container position is exposed. The calculation formula for the newly exposed bottom container position information is:

[0122] bay k =bay n ,

[0123] row k =row n ,

[0124] tier k =tier n -1,

[0125] x k =x r ,

[0126] y k =y r ,

[0127] z k =z r-c z ,

[0128] Its box type information class n Given once by the AI ​​box detection algorithm.

[0129] In the bay map, the captured location information is deleted and the newly exposed location information is added (class k , x k ,y k , z k ,bay k , row k , tier k ).

[0130] (4) Since the cabin space is floating and updated in real time, the reference box is dynamically updated to the current newly exposed container space, and the positions of other container spaces in the bay map are corrected. Figure 8 Taking the height floating error as an example, the floating change of the hull before and after the grab box is shown. The error calculation formula corresponding to the three degrees of freedom is as follows:

[0131] Δz=z k -(z n -c z ),

[0132] Δx=x k -x n ,

[0133] Δy=y k -y n ,

[0134] Therefore, the posture correction formula for other boxes in the Bay Area Map is:

[0135] x′ i =x i +Δx,

[0136] y′ i =y i +Δy,

[0137] z′ i =z i +Δz,

[0138] Step 6: After the loading and unloading operation begins, monitor the landside container grabbing signal of the PLC in real time and release signal The signal is transmitted to the target container space planning and decision-making unit, which reads the current cabin position map information and calculates the current cabin inclination angle θ. The formula is as follows:

[0139]

[0140] Where Q is the top container in the first row, T' is the container in the last row with the same floor height as the top container in the first row, which is determined by the top container in the last row, T, and the floor height difference between Q and T. The formula is:

[0141] T′ x =T x ,

[0142] T′ z =T Z +C Z ×(Q row -T row ),

[0143] At this point, there are two situations to consider:

[0144] Scenario 1. If Figure 9 As shown, PLC land side box grab signal When it suddenly changes from false to true, it means that the container grabbing action on the land side has just been completed. The control system will control the spreader to move to the sea side to complete the container placement according to the target container position instruction.

[0145] (1) If the cabin inclination angle θ satisfies

[0146] -θ0≤θ≤θ0,

[0147] In the formula, θ0 is the tilt angle threshold, which means that the cabin tilt angle is small and the containers can be placed layer by layer and row by row. drop The box position of the trajectory is the best target box position, and the trajectory safety height is row drop The maximum height of all slots within a row.

[0148] (2) If the cabin inclination angle θ satisfies

[0149] θ≤-θ0,

[0150] This means that the cabin has a large inclination angle and the last row of cabins sinks, so the first row of containers is placed first. min The box is the best target box position of the trajectory, and the trajectory safety height is row min The maximum height of all slots within a row.

[0151] (3) If the cabin inclination angle θ satisfies

[0152] θ≥θ0,

[0153] This means that the cabin tilt is large and the first row of cabins is sinking, so the last row of cabins will be placed first. max The box is the best target box position of the trajectory, and the trajectory safety height is row maxThe maximum height of all boxes in a row.

[0154] Case 2. If Figure 10 As shown, PLC land side box release signal When it suddenly changes from false to true, it means that the container release action on the land side has just been completed. The control system will control the spreader to go to the sea side to complete the container grabbing according to the target container position instruction.

[0155] (1) If the cabin inclination angle θ satisfies

[0156] -θ0≤θ≤θ0, which means the cabin tilt angle is small, and the cabin can be captured layer by layer and row by row. grab The box is the best target box position of the trajectory, and the trajectory safety height is row grab The maximum height of all slots within a row.

[0157] (2) If the cabin inclination angle θ satisfies

[0158] θ≤-θ0,

[0159] This means that the cabin tilt is large and the last row of cabins is sinking, so the last row of boxes will be grabbed first. max The box is the best target box position of the trajectory, and the trajectory safety height is row max The maximum height of all slots within a row.

[0160] (3) If the cabin inclination angle θ satisfies

[0161] θ≥θ0,

[0162] This means that the cabin has a large inclination angle and the first row of cabins is sinking, so the boxes in the first row are grabbed first. min The box is the best target box position of the trajectory, and the trajectory safety height is row min The maximum height of all slots within a row.

[0163] In order to verify the performance of the present invention, this embodiment conducted relevant experiments at the Shanghai Container Terminal.

[0164] This embodiment compares the purely visual method of ship hold position perception with the method of combining perception and reasoning of the present invention. In the experiment, the experimental scheme is completely consistent with the evaluation criteria. For the evaluation method, we have developed two evaluation indicators, which are used to evaluate the accuracy of the bay map (including box type and bay layer information) and the positioning accuracy of the box posture (that is, the position of the box in the direction of the trolley, truck and height). We make the following provisions: when the bay map information output by the system is completely consistent with the actual ship map, the current bay map is considered accurate, and the accuracy of the system is calculated through a large number of bay scans or loading and unloading processes; and the positioning accuracy of the bay map can be indirectly represented by the actual operating efficiency on the sea side.

[0165] Table 1 lists the detailed experimental results of this example.

[0166] Table 1: Comparative experiment between bay position map and operation efficiency

[0167] Experimental methods Accuracy (%) Average operating efficiency (box / h) Pure Vision 75.2 13.2 The perception and reasoning combination method of the present invention 94.6 17.6

[0168] (1) In order to solve the problem of low accuracy of the bay position map in severe weather such as tides or strong winds, the present invention proposes a robust high-precision bay position mapping strategy. This strategy selects different dynamic benchmarks for the loading and unloading processes, and trains an AI model to detect the swing state of the spreader at the time of loading and unloading. The bay position map is then corrected in real time by integrating the PLC and the swing amplitude of the spreader in the direction of the trolley and the truck, which greatly improves the correction accuracy and effectively reduces the risk of safety accidents caused by inaccurate bay position maps of automated bridge cranes.

[0169] (2) In view of the sensitivity and limitations of single-frame perception results to noise under special working conditions, a multi-source and multi-frame two-dimensional data fusion method is proposed. Each bin is clustered as a single cluster, and the multi-frame data in each cluster are voted and weighted averaged, thereby effectively compensating for the limitations of single-frame detection and eliminating noise interference such as sensor vibration.

[0170] (3) In response to abnormal situations such as missed bins or missing point cloud heights, the present invention proposes an automatic reasoning method for missing bins based on neighborhood information, which reduces manual intervention and improves the reliability and safety of the automatic system.

[0171] (4) The present invention proposes a target container space planning and decision-making unit for the first time. It combines the bay layer and height information in the bay map, weighs the loading situation of the ship's hold, and sends safe seaside operation target container space instructions to the control system to avoid dangerous accidents such as overloading or rollover.

[0172] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A high-precision real-time detection system for ship cabin position for automated bridge cranes, characterized by: It includes data acquisition unit, detection and positioning unit, bay layer calculation unit, dynamic correction unit, bay map automatic reasoning unit, and target box location planning and decision unit; The data acquisition unit is used to send the acquired synchronization data to the detection and positioning unit, and the synchronization data includes PLC data, image and point cloud data; The detection and positioning unit is used to calculate the model and position of all containers or empty spaces on the ship's hold, and then send the model and position information to the bay layer calculation unit to obtain its bay layer information, and then obtain a complete bay map through the bay map automatic reasoning unit. Finally, the target container space on the sea side is sent to the control system through the target container space planning and decision unit; The dynamic correction unit is used to automatically trigger the dynamic correction unit to dynamically correct the bay position map when the loading and unloading operation is actually carried out; The data acquisition unit includes a PLC controller, a camera, a laser radar, and a data synchronizer. The PLC controller is used to obtain the operating status of the bridge crane in real time. The camera and laser radar can move synchronously with the position of the trolley to respectively obtain the image I and laser point cloud P of the cabin space. The data synchronizer is installed on the control server and is used to package and obtain the PLC data, image data, and point cloud data at the same time. The detection and positioning unit is composed of a target detection and positioning algorithm, a point cloud segmentation algorithm, and a multi-source multi-frame data fusion algorithm; The input of the detection and positioning unit is a continuous multi-frame data packet, each of which contains synchronized PLC data, image I and laser point cloud P; It outputs the model and weighted position of the container or empty space in the ship's hold; The dynamic correction unit takes as input the actual container landing signal from the PLC sea side. When the sea side loading operation is carried out, the loading signal is triggered, the dynamic reference is selected as the actual operating container position, and the floating deviation of the ship hold position is calculated in combination with the spreader status to update the bay position map. When the seaside unloading operation is carried out, the seaside unloading signal is triggered, the dynamic benchmark is selected as the adjacent bottom box in actual operation, and the floating deviation of the ship's cabin position is calculated in combination with the spreader status to update the bay position map.

2. The high-precision real-time detection system for ship hold positions for automated bridge cranes according to claim 1 is characterized by: The input of the bay layer calculation unit is the model and position information of the ship hold container or empty warehouse space and PLC data. During the initial scan, the loading and unloading signal is not triggered. The bay layer calculation unit selects the reference box and calculates the bay layer number of all container positions to obtain the bay position map for the first time.

3. The high-precision real-time detection system for ship hold positions for automated bridge cranes according to claim 1 is characterized by: The input of the bay map automatic reasoning unit is the original bay map obtained by perception. The missed row information is inferred based on the container position information of adjacent rows in the cabin. At the same time, the inaccurate container height and layer number caused by the missing point cloud are inferred, and finally the optimized bay map is obtained.

4. The high-precision real-time detection system for ship hold positions for automated bridge cranes according to claim 1 is characterized by: The target container position planning and decision-making unit is triggered to run by the PLC landside container signal. The input of the target container position planning and decision-making unit is the optimized bay position map. Combined with the bay layer and height information in the bay position map, the loading situation of the ship cabin space is weighed and a safe seaside operation target container position instruction is sent to the control system. The instruction content specifically includes the target container position information, safety height and cabin inclination angle.

Images