An automated container wharf truck operation safety management and control method and system

Abstract

The application discloses a kind of automated container terminal truck operation safety control method and system, the video image data and three-dimensional space point cloud data of operation area are synchronously collected;The collected data is spatio-temporal synchronization and fusion, and fusion perception data is obtained;The fusion perception data is input into collision risk identification model, and the collision detection result of big belly cabinet, high-low cabinet, double box and equipment is output;The driver gets off and personnel intrusion detection result is output through personnel illegal detection model;According to the risk level of collision risk and personnel illegal detection result evaluation, and through the communication interface of equipment host, the linkage control instruction is issued, and the corresponding safety response measure is executed.The application realizes the all-round, intelligent safety monitoring of truck operation area, supports the accurate detection of multiple collision risks and the real-time identification of personnel illegal behavior, has the risk grading response and equipment linkage control ability, significantly improves the operation safety and management efficiency of automated container terminal.

Description

Technical Field

[0001] This invention belongs to the field of intelligent safety management and control technology for port terminals. Specifically, it relates to an automated container terminal truck operation safety management and control method and system. Background Technology

[0002] With the rapid development of global trade, the throughput of container terminals continues to grow, placing higher demands on terminal operation efficiency and safety. Automated container terminals employ unmanned operation methods, using automated equipment to realize container loading, unloading, and transportation, significantly improving operational efficiency. However, during truck operations at automated terminals, the lack of on-site supervision presents several safety risks, including: oversized containers in wide-body container configurations are prone to collisions with surrounding equipment or other containers during loading and unloading; containers of different heights in low-profile container configurations pose a collision risk during stacking or transportation; when two containers are lifted simultaneously in double-container operations, collisions may occur between the containers; there is a risk of equipment collisions between containers and spreaders, trucks, etc.; and personnel safety hazards arise from drivers leaving the vehicle without authorization or unauthorized personnel entering the work area.

[0003] Currently, the existing dock safety monitoring system has the following problems: (1) It mainly relies on fixed-location monitoring cameras for safety monitoring. This method has problems such as many blind spots, poor real-time performance, and lack of intelligent analysis capabilities due to the fixed location of the monitoring equipment; video monitoring also has difficulty in accurately judging the distance between objects and the risk of collision. (2) The existing system mostly uses simple image processing algorithms and lacks special recognition algorithms for special working conditions, resulting in insufficient intelligent analysis capabilities; there is a lack of real-time linkage mechanism between risk warning and equipment control, and the linkage control is lagging. (3) There is a lack of effective monitoring and early warning capabilities for personnel behavior in the work area. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for safety management and control of automated terminal truck operations that can provide comprehensive coverage, specific intelligent analysis capabilities, and support real-time linkage control, thereby improving the safety management level and operational efficiency of automated terminals.

[0005] The present invention is implemented using the following technical solutions: A method for safety management and control of automated container terminal truck operations is proposed, including: S1, synchronously collect video image data and three-dimensional spatial point cloud data of the work area; wherein, the video image data is collected by multiple high-definition cameras, and the three-dimensional spatial point cloud data is collected by a laser sensor; the multiple high-definition cameras and the laser sensor are installed on the cantilever side of the equipment; S2 performs spatiotemporal synchronization and data fusion of video image data and 3D spatial point cloud data to obtain fused perception data; S3 integrates the perception data into the pre-trained collision risk identification model, performs multi-target detection and tracking of containers, loading and unloading equipment and trucks in the truck operation area, and outputs collision risk detection results; the collision risk identification model includes a large container collision detection sub-model, a high and low container collision detection sub-model, a double container collision detection sub-model and an equipment collision detection sub-model. S4, which integrates sensory data input into a pre-trained personnel violation detection model, detects personnel behavior within the work area and outputs personnel violation detection results; S5. Based on the collision risk detection results and personnel violation detection results, conduct a risk level assessment to determine the current risk level; S6, based on the current risk level, issues linkage control commands through the communication interface with the device host to execute corresponding safety response measures.

[0006] In some embodiments of the present invention, S2 specifically includes: The video image data is subjected to denoising and enhancement processing; The three-dimensional spatial point cloud data is filtered and downsampled. The video image data and the three-dimensional spatial point cloud data are spatiotemporally aligned based on timestamps; The subsequent video image data and 3D spatial point cloud data will be fused at the feature level to obtain the fused perception data.

[0007] In some embodiments of the present invention, the collision detection sub-model for large-bellied cabinets detects the collision risk of large-bellied cabinets, specifically including: A deep learning-based target detection algorithm is used to identify the large-bellied cabinet target from the fused perception data; The actual width of the large cabinet target was calculated by combining three-dimensional spatial point cloud data; Obtain the real-time distance between the large cabinet target and surrounding obstacles; The real-time distance is compared with a preset safe distance threshold. When the real-time distance is less than the safe distance threshold, a collision warning signal for the large cabinet is output.

[0008] In some embodiments of the present invention, the point cloud data of the large-bellied cabinet is further completed by using a large-bellied cabinet outline reconstruction sub-model, specifically including: Based on sparse point cloud data from the side of a container, and by integrating depth estimation results from a high-definition camera, the two-dimensional depth map is back-projected onto a three-dimensional world coordinate system through a sensor extrinsic calibration matrix. This data is then registered and fused with spatial point cloud data scanned by a laser sensor to generate an initial multimodal point cloud. Based on the PointNet++ encoder-decoder architecture, local geometric features are extracted and the global shape of the sparse point cloud is predicted. A differentiable standard box-shaped size constraint module is added to the decoder output layer to ensure that the completion result meets the standard length / height size constraints of the container. An ultra-wide regression branch is added at the end of the network to directly output the absolute deviation of the container width from the standard value, which serves as the basis for collision safety judgment. A simulation data augmentation strategy is adopted to apply random occlusion, noise, and ultra-wide deformation to the standard point cloud data to generate training samples covering different occlusion rates to train the network. The initial multimodal point cloud network is completed using a pre-trained network based on the standard box-shaped prior constraints.

[0009] In some embodiments of the present invention, when implementing collision detection for large containers, the method further includes calculating the current occlusion rate of the container point cloud and determining the point cloud data to be input into the large container collision detection sub-model based on the occlusion rate, including: The occlusion rate of the current container point cloud is calculated in real time. When the occlusion rate is less than a first threshold, the current container point cloud data is used. When the occlusion rate is between the first and second thresholds, the current container point cloud data is weighted and fused with the point cloud data completed by the large container outline reconstruction sub-model. When the occlusion rate exceeds the second threshold, the point cloud data completed by the large container outline reconstruction sub-model is used, and a low confidence marker is triggered.

[0010] In some embodiments of the present invention, the high-low cabinet collision detection sub-model detects the risk of high-low cabinet collisions, specifically including: The height information of different containers within the work area is obtained through laser ranging. Combine visual recognition algorithms to determine the type of each container; Based on the height information and the container type, a three-dimensional spatial model of the high-low container operation scenario is established; Based on the three-dimensional spatial model, the motion trajectory of each container is predicted, the collision risk is assessed, and the collision risk detection results are output.

[0011] In some embodiments of the present invention, the dual-box collision detection sub-model predicts the risk of dual-box collisions, specifically including: The position of each container is determined based on 3D spatial point cloud data, and the axial dimensions of each container are extracted by L-shaped corner fitting algorithm to construct the minimum outer rectangle of each container. Extended Kalman filtering is used to estimate the motion state of the spreader and the unloaded container; A wire rope swing inertia compensation model is adopted to correct the trajectory prediction deviation of the un-dropped container caused by the rope swing; the wire rope swing inertia compensation model is based on the simple pendulum dynamics equation. Based on the state mean and covariance estimated by Kalman filter, the three-dimensional motion trajectory of the undropped container within the next T seconds is predicted, and an ellipsoidal trajectory confidence envelope is generated. The predicted envelope is intersected with the smallest outer rectangle of the already placed container in three-dimensional space. If there is an overlap, it is determined that there is a collision risk, and the collision time is predicted and graded warnings are issued.

[0012] In some embodiments of the present invention, the equipment collision detection sub-model is used to detect the collision risk between containers and loading / unloading equipment and container trucks, specifically including: Define all monitored devices in the scene as nodes. When the shortest distance between the 3D bounding boxes of two targets is less than the security perception threshold, establish a bidirectional directed edge between the two nodes. A temporal graph-based attention network is used to predict the future trajectory distribution of each node. Based on the predicted trajectory distribution of each node, the collision probability between all targets is calculated, and a joint collision probability matrix is ​​output. A dock physical constraint layer is added after the network prediction layer to constrain the prediction results using prior knowledge. The first-order collision and second-order chain collision probabilities are calculated based on the joint collision probability matrix, and then differentiated management of multi-target collision risks is implemented after sorting.

[0013] In some embodiments of the present invention, the personnel violation detection model includes a driver exit detection sub-model and a personnel intrusion detection sub-model; The driver exit detection sub-model monitors the truck cab area based on visual target detection and behavior recognition algorithms. When it detects that the driver has left the cab during operation, it outputs a driver violation exit alarm signal. The personnel intrusion detection sub-model continuously monitors the work area, uses a human detection algorithm to identify personnel entering the dangerous area, combines a personnel tracking algorithm for continuous monitoring, and outputs a personnel intrusion alarm signal.

[0014] An automated container terminal truck safety management system is proposed, comprising: The perception layer includes multiple high-definition cameras and laser sensors installed on the cantilever side of the equipment, used to synchronously collect video image data and three-dimensional spatial point cloud data of the work area; The analysis layer includes an edge computing unit, a collision risk identification unit, and a personnel violation detection unit. The edge computing unit performs spatiotemporal synchronization and data fusion on video image data and 3D spatial point cloud data to obtain fused perception data. The collision risk identification unit inputs the fused perception data into a pre-trained collision risk identification model to perform multi-target detection and tracking of containers, loading and unloading equipment, and container trucks within the container truck operating area, and outputs collision risk detection results. The collision risk identification model includes a large container collision detection sub-model, a high-low container collision detection sub-model, a double container collision detection sub-model, and an equipment collision detection sub-model. The personnel violation detection unit inputs the fused perception data into a pre-trained personnel violation detection model to detect personnel behavior within the operating area and outputs personnel violation detection results. The control layer is used to assess the risk level based on the collision risk detection results and personnel violation detection results, determine the current risk level, and issue linkage control commands through the communication interface with the equipment host. The execution layer is used to execute the instructions issued by the control layer and to carry out the corresponding security response measures.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are: (1) By deploying a combination of multi-source heterogeneous sensors, including a high-definition camera and a laser sensor, on the cantilever side of the equipment, the sensors move with the working equipment and always maintain the best monitoring angle. Compared with the traditional fixed monitoring method, this effectively eliminates the monitoring blind spots of the traditional fixed installation scheme, achieves 360-degree all-round coverage of the working area, and improves the sensing range and data quality.

[0016] (2) Simultaneous acquisition of video images and 3D point cloud data, through spatiotemporal alignment and feature-level fusion, overcomes the limitations of a single sensor, enhances the perception of complex working environments, and provides a high-quality data foundation for subsequent risk identification.

[0017] (3) For typical high-risk working conditions at the terminal, a collision detection sub-model for large container, a collision detection sub-model for high and low container, a collision detection sub-model for double container and a collision detection sub-model for equipment were designed respectively, which realized accurate identification and early warning of collisions between ultra-wide containers, high and low containers, double container operations and equipment.

[0018] (4) Combining multimodal point cloud fusion with point cloud completion network based on standard box type prior constraints, and introducing an adaptive fusion strategy for occlusion rate, it effectively solves the problem of sparse point cloud and large measurement error caused by occlusion of large cabinets. It can still maintain high measurement accuracy when the occlusion rate exceeds 20%, and significantly reduces the risk of false alarms and missed alarms.

[0019] (5) An extended Kalman filter and a wire rope swing inertia compensation model are used to accurately predict the motion trajectory of the unlanded container. The ellipsoidal trajectory confidence envelope and the minimum outer rectangle of the landed container are combined to perform three-dimensional intersection, realizing a graded early warning based on the predicted collision time, and improving the anti-collision capability from passive perception to active prediction.

[0020] (6) The operation scenario is modeled as a dynamic directed graph. The temporal graph attention network is used to predict the future trajectory distribution of each target and output the joint collision probability matrix. Combined with the physical hard constraint layer of the dock and the chain collision risk reasoning, the joint prediction of direct collision and indirect secondary collision between multiple targets is realized.

[0021] (7) By using the driver exit detection sub-model and the personnel intrusion detection sub-model, the behavior of the driver leaving the cab in violation of regulations and the entry of unrelated personnel into dangerous areas can be identified in real time, thereby reducing the safety hazards in the human-computer interaction process.

[0022] (8) The risk level is comprehensively assessed based on the collision risk detection results and the personnel violation detection results, and the linkage control command is issued through the communication interface with the equipment host to realize differentiated safety responses such as graded early warning, automatic deceleration, and emergency stop, which shortens the response time from risk identification to control execution and improves the real-time performance and effectiveness of overall safety management.

[0023] (9) An integrated closed-loop management and control architecture of "perception-analysis-linkage" was constructed, forming a complete closed-loop system from front-end data collection, middle-platform intelligent analysis to back-end equipment linkage control, realizing real-time linkage between risk detection and equipment control, and the response time from risk detection to linkage control can be controlled within 500 milliseconds.

[0024] Other features and advantages of the present invention will become clearer after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. Attached Figure Description

[0025] The accompanying drawings, as part of this invention, are provided to further illustrate the invention. The illustrative embodiments and descriptions are used to explain the invention but do not constitute an undue limitation thereof. Clearly, the drawings described below are merely some embodiments; those skilled in the art can obtain other drawings based on these drawings without creative effort. Figure 1 Figure 1 This is a flowchart illustrating the automated container terminal truck operation safety management method proposed in this invention. Figure 2 This is an architecture diagram of the automated container terminal truck operation safety management system proposed in this invention; Figure 3This is a schematic diagram showing the sensor hardware installation location in the system of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0027] Automated container terminal operations involve multiple stages, including container loading, unloading, and transportation. The work site involves frequent interactions between various objects such as large machinery, containers, and trucks, resulting in complex and variable safety risks. Collision risks are significantly increased, especially under special operating conditions such as large containers, high-low containers, and double containers. Traditional terminal safety monitoring systems primarily rely on fixed-position surveillance cameras. Due to their fixed monitoring angles and lack of three-dimensional spatial perception capabilities, they struggle to effectively cover all critical locations within the work area and cannot accurately identify and provide real-time warnings of collision risks under special operating conditions.

[0028] Based on this, such as Figure 2 As shown, this application provides an automated container terminal truck operation safety management system with a four-layer architecture of "perception layer - analysis layer - control layer - execution layer". This system utilizes multi-source heterogeneous sensors deployed on the cantilever side of the equipment, combined with... Figure 1 The intelligent analysis and linkage control shown enable real-time detection and intelligent early warning of various collision risks and personnel violations during truck operations.

[0029] The perception layer is responsible for synchronously and in real-time acquiring multi-source data from the work area. This multi-source data includes at least video image data and 3D spatial point cloud data. The hardware system of the perception layer is mainly installed on the cantilever side of the equipment, such as... Figure 3 As shown, it specifically includes a high-definition camera and a laser sensor.

[0030] Four high-definition cameras are installed on the equipment cantilever, positioned at the front left, front right, rear left, and rear right, providing 360-degree all-around visual monitoring coverage. The cameras are industrial-grade high-definition network cameras with a resolution of at least 1920×1080 and a frame rate of at least 25fps. They are waterproof and dustproof with a rating of at least IP67, enabling stable operation in the harsh environment of the dock. The installation positions of the four cameras are carefully designed to ensure complete coverage of the entire working area below and around the equipment cantilever, eliminating blind spots common in traditional fixed camera solutions.

[0031] Two laser sensors are installed on the left and right sides of the equipment cantilever, respectively, to collect 3D point cloud data of the work area. The laser sensors employ industrial-grade lidar or laser rangefinders, with a detection range of no less than 50 meters, an angular resolution of no more than 0.5 degrees, and a scanning frequency of no less than 10Hz. The laser sensors can accurately acquire the 3D spatial position information of each target object within the work area, compensating for the shortcomings of simple visual monitoring in distance measurement and providing a reliable data foundation for accurate collision risk calculation.

[0032] By integrating the sensors of the safety detection system onto the cantilever side of the equipment rather than the fixed facility side, this mobile sensor deployment solution allows the sensors to move with the operating equipment, always maintaining the best monitoring perspective. This avoids the blind spot problem caused by fixed installation methods and greatly improves the comprehensiveness and effectiveness of monitoring.

[0033] The analysis layer includes edge computing units, collision risk identification units, and personnel violation detection units.

[0034] The edge computing unit, installed on the main body of the equipment, is responsible for the acquisition, preprocessing, and fusion of sensor data. It connects to each sensor via high-speed industrial Ethernet, enabling high-speed, low-latency data transmission. Simultaneously, the edge computing unit communicates in real-time with the operation control system via industrial Ethernet, uploading processed data and analysis results to the control system and receiving control commands from it.

[0035] The collision risk identification unit integrates perception data into a pre-trained collision risk identification model to perform multi-target detection and tracking of containers, loading and unloading equipment, and container trucks within the truck operation area, and outputs collision risk detection results. The collision risk identification model includes a large container collision detection sub-model, a high and low container collision detection sub-model, a double container collision detection sub-model, and an equipment collision detection sub-model.

[0036] The personnel violation detection unit integrates the sensor data and inputs it into a pre-trained personnel violation detection model to detect personnel behavior within the work area and output the personnel violation detection results.

[0037] The control layer is used to assess the risk level based on the collision risk detection results and personnel violation detection results, determine the current risk level, and issue linkage control commands through the communication interface with the equipment host. The execution layer executes the commands issued by the control layer and implements corresponding safety response measures. The execution layer's response measures include four types: audible and visual alarms, equipment deceleration, emergency stop, and remote monitoring alarms. Audible and visual alarms are triggered by sound and flashing signals emitted from alarms installed on the equipment; equipment deceleration is achieved by controlling the operating parameters of the equipment's host computer; emergency stop is achieved by sending an emergency stop signal to the equipment host computer; and remote monitoring alarms push alarm information to a remote monitoring center via the network, allowing management personnel to promptly understand the on-site situation.

[0038] Based on the above-described safety management system architecture for container truck operations, the safety management method for container truck operation areas provided in this application embodiment is as follows: Figure 1 As shown, it includes the following steps: S1: Synchronously acquire video image data and 3D spatial point cloud data of the work area.

[0039] After the system starts up, multiple sensors simultaneously acquire video image data and laser point cloud data. Four high-definition cameras acquire video image data synchronously at a preset frame rate, while two laser sensors acquire 3D spatial point cloud data synchronously at a preset scanning frequency. The edge computing unit timestamps the data from each sensor to ensure precise time synchronization of data acquired by different sensors.

[0040] S2: Perform spatiotemporal synchronization and data fusion on video image data and 3D spatial point cloud data to obtain fused perception data.

[0041] The edge computing unit preprocesses and fuses the acquired data. The preprocessing includes: denoising the video image data to eliminate image noise caused by factors such as dust and water mist in the dock environment, and performing image enhancement to improve image quality; filtering the 3D spatial point cloud data to remove invalid points caused by ground reflection and multipath effects, and performing downsampling to reduce data volume and improve subsequent processing speed. The data fusion process includes: spatiotemporally aligning the video image data and 3D spatial point cloud data based on sensor calibration parameters and timestamps to ensure precise matching of the two types of data in the spatial coordinate system; then, performing feature-level fusion of the aligned 2D image features and 3D spatial features to obtain fused perception data containing rich visual information and precise spatial information.

[0042] S3: Input the fused perception data into the pre-trained collision risk identification model to perform multi-target detection and tracking of containers, loading and unloading equipment and trucks in the truck operation area, and output the collision risk detection results.

[0043] The collision risk identification unit and personnel violation detection unit in the analysis layer perform collision risk detection and personnel violation detection in parallel. The collision risk identification unit includes four dedicated collision detection sub-models, each targeting one of the four most common collision risk scenarios in automated terminal truck operations.

[0044] 1. Oversized Container Collision Detection Sub-model: Oversized containers refer to extra-wide containers whose width exceeds the standard container size. During loading and unloading, due to their excessive width, they are prone to collisions with surrounding equipment or other containers. The oversized container collision detection sub-model first identifies the oversized container target from fused perception data based on deep learning target detection algorithms (such as YOLO series algorithms); then, combined with laser point cloud data, it accurately calculates the actual width of the oversized container target through point cloud segmentation and size measurement algorithms; simultaneously, it obtains the real-time distance between the oversized container target and surrounding obstacles (including other containers, equipment support columns, trucks, etc.); finally, it compares the real-time distance with a preset safe distance threshold. When the real-time distance is less than the safe distance threshold, it outputs an oversized container collision warning signal and transmits the warning signal to the risk assessment module.

[0045] In actual operation, the oversized portion of a large container is easily obstructed by crane structural components, spreader frames, or adjacent containers, resulting in large areas of voids in the laser point cloud (no effective echo in the obstructed area). Existing solutions directly measure the width of the sparse and incomplete point cloud, with errors as high as ±15cm when the obstruction rate exceeds 20%, making it impossible to accurately determine the safe clearance between the oversized container and adjacent equipment. This implementation provides a sub-model for reconstructing the outline of a large container to complete the point cloud data of the large container, specifically including: (1) Based on the sparse point cloud data of the side of the container, the depth estimation results of the high-definition camera are fused, and the two-dimensional depth map is back-projected to the three-dimensional world coordinate system through the sensor extrinsic calibration matrix. The two-dimensional depth map is then registered and fused with the spatial point cloud data scanned by the laser sensor to generate the initial multimodal point cloud. Based on the sparse point cloud data collected by laser scanners #1 and #2 on the side of the container, the depth estimation results of cameras #1 to #4 are simultaneously fused (using a monocular depth estimation network or binocular parallax estimation). The 2D depth map is back-projected to the 3D world coordinate system through the sensor extrinsic calibration matrix and registered and fused with the laser scanner point cloud to generate an initial multimodal point cloud (with a higher density than the pure laser point cloud, but still containing occlusion holes).

[0046] (2) Container point cloud compensation is performed using standard container type prior constraints.

[0047] This embodiment proposes a point cloud completion network (PC-Net, Prior-Constrained Completion Network) based on the prior constraints of standard ISO container dimensions. The physical dimension parameters of the container are incorporated as hard constraints into the network architecture. The main technical elements are as follows: Network backbone: Based on PointNet++ encoder-decoder architecture, it extracts local geometric features and predicts the global shape of sparse point cloud.

[0048] Prior constraints: Add a differentiable standard container size constraint module to the decoder output layer to ensure that the completion result meets the standard container length / height size constraints (only the width dimension is not constrained and is freely predicted by the network), thus eliminating completion errors in the length and height directions.

[0049] Large container inspection head: An ultra-wide regression branch is added at the end of the network to directly output the absolute deviation (in mm) of the container width exceeding the standard value, which serves as the basis for collision safety judgment.

[0050] Training data: Using a simulation data augmentation strategy, random occlusion, noise, and ultra-wide deformation are applied to standard point cloud data to generate training samples covering different occlusion rates (0-60%).

[0051] (3) Calculate the current container point cloud occlusion rate and determine the point cloud data to be input into the large container collision detection sub-model based on the occlusion rate.

[0052] Calculate the occlusion rate (effective points / theoretical full points) of the current container point cloud in real time, and dynamically adjust the completion strategy based on the occlusion rate: When the occlusion rate is less than the first threshold (20%), the current container point cloud data is used to complete the network as an auxiliary verification; when the occlusion rate is between the first threshold (20%) and the second threshold (40%), the current container point cloud data is weighted and fused with the point cloud data completed by the large container outline reconstruction sub-model; when the occlusion rate exceeds the second threshold (40%), the point cloud data completed by the large container outline reconstruction sub-model is used, and a low confidence marker is triggered.

[0053] Finally, based on the completed and accurate 3D contour, the clearance between the dangerous side (overwidth direction) of the large container and adjacent containers and crane structural components is calculated. When the clearance is less than the safety threshold (default 150mm, configurable), a collision warning is triggered.

[0054] 2. High-low container collision detection sub-model: High-low containers refer to the situation where containers of different heights exist in the same working environment, such as the simultaneous presence of 20-foot standard containers and 40-foot high cube containers. The high-low container collision detection sub-model first obtains the precise height information of different containers in the working area through laser ranging; then, it combines visual recognition algorithms to determine the type of each container (such as standard container, high cube container, open top container, etc.); based on the height information and container type, a three-dimensional spatial model of the high-low container working environment is established. This model fully describes the position, size, and interrelationship of each container in the working area; finally, based on the three-dimensional spatial model and the motion state of the containers and equipment, the model predicts the motion trajectory of each target, assesses the risk level of possible collisions, and outputs the collision risk detection results.

[0055] 3. Dual-Container Collision Detection Sub-model: Dual-container operations refer to scenarios where a truck needs to carry two 20-foot containers. When the spreader lowers the second container into position, existing systems only trigger an alarm when the real-time distance is below a threshold, resulting in an extremely short response time window and a lack of proactive prediction capabilities. This embodiment uses a dual-container collision detection sub-model to proactively predict dual-container collisions and trigger a three-level early warning system, thus improving the system's collision avoidance capability from passive perception to proactive prediction.

[0056] Specifically, including: (1) The position of each container is determined based on the three-dimensional spatial point cloud data, and the axial dimensions of each container are extracted by the L-shaped corner fitting algorithm to construct the minimum outer rectangle of each container. The spreader lidar is used to scan in real time to obtain the three-dimensional pose (position coordinates + deflection angle) of the first container that has been placed (the container that has been placed), as well as the real-time spatial position of the second container currently suspended by the spreader (the container that has not been placed). The outer contour bounding boxes of the two containers are obtained with the assistance of laser scanners #1 and #2. The axial dimensions are accurately extracted by L-shaped corner fitting algorithm to construct the three-dimensional minimum outer bounding rectangle (3D OBB, Oriented Bounding Box) of each container.

[0057] (2) Extended Kalman filter is used to estimate the motion state of the spreader and the unloaded container. The motion states (position, velocity, acceleration) of the spreader and the unloaded container are optimally estimated using an extended Kalman filter (EKF), and the model is as follows: State vector: X=[x,y,z,vx,vy,vz,ax,ay,az] (3D position, velocity, acceleration).

[0058] Process noise: Calibrated based on the acceleration control characteristics of the crane control system.

[0059] Observation model: Multi-sensor state fusion is performed by fusing the centroid coordinates of the lidar point cloud with the depth back projection results of the camera 2D detection frame.

[0060] (3) The wire rope swing inertia compensation model is used to correct the trajectory prediction deviation of the un-dropped container caused by the rope swing.

[0061] A swing inertia compensation model for the spreader wire rope is introduced to correct the trajectory prediction deviation of the unreleased container caused by the rope swing effect. The swing model is based on the dynamic equation of a simple pendulum, where: Swing length parameter: The distance from the spreader to the top of the container is measured in real time by lidar, and the swing length L is dynamically updated.

[0062] Swing angle: Calculated in real time by the horizontal deviation of the center of mass of the unloaded container relative to the center of the spreader.

[0063] Oscillation compensation: An oscillation inertial force term is superimposed in the Kalman filter prediction step to correct the prediction trajectory deviation.

[0064] (4) Based on the state mean and covariance estimated by Kalman filter, predict the three-dimensional motion trajectory of the undropped container within the next T seconds and generate an ellipsoidal trajectory confidence envelope.

[0065] Based on the state mean and covariance of the Kalman filter output, the three-dimensional motion trajectory of the undropped container is predicted in the next T seconds (T=3~5 seconds, configurable) by rolling forward, and an ellipsoidal trajectory confidence envelope considering the prediction uncertainty is generated.

[0066] (5) Perform a three-dimensional spatial intersection operation between the envelope and the smallest outer rectangle of the already placed container. If there is an overlap, it is determined that there is a collision risk, and the collision time is predicted and graded for early warning.

[0067] In this embodiment, a three-level early warning system is implemented based on the predicted collision time: Yellow alert (3s-5s): Interface alarm prompts, reducing the descent speed of the lifting device by 50%.

[0068] Orange alert (2s-3s): Audible and visual alarm, sling descent speed reduced to 20%, operator notification.

[0069] Red alert (less than 2 seconds): Emergency stop, immediately halt all movement of the lifting equipment, and await manual confirmation.

[0070] 4. Equipment Collision Detection Sub-model: The equipment collision detection sub-model is responsible for detecting the collision risk between containers and loading / unloading equipment and trucks.

[0071] In complex container terminal operations, multiple moving targets, including spreader equipment, multiple containers (including the first and second containers in a dual-container configuration), and truck chassis, coexist in a limited three-dimensional space. Existing systems perform collision detection on each pair of targets independently, completely ignoring the physical interactions and relationships between them. Chain collision effect: After target A collides with target B, the direction of movement of target B changes immediately, which may lead to a secondary collision with target C. Existing systems cannot predict this kind of indirect collision.

[0072] Blind spots in cooperative motion: Indirect collision risks caused by multi-target cooperative motion (such as changes in the relative position of containers caused by the displacement of trucks) are not effectively detected by existing systems.

[0073] Lack of risk prioritization: Isolated pairwise collision probabilities cannot support the system in ranking and differentiating responses to multiple simultaneous collision risks.

[0074] This embodiment addresses the problems in existing collision detection systems by configuring the following method in the device collision detection sub-model to achieve joint prediction and risk priority ranking of direct collisions and indirect secondary collisions between multiple targets: (1) Define all monitored devices in the scene as nodes. When the shortest distance between the three-dimensional bounding boxes of two targets is less than the security perception threshold, establish a bidirectional directed edge between the two nodes.

[0075] Node set V: contains all monitored targets in the scene, including crane spreaders (node ​​type: spreader), suspended containers (node ​​type: air container), ground containers (node ​​type: landing container), and container truck chassis (node ​​type: carrier); the feature vector of each node includes target type embedding, 3D position, velocity vector, and 3D bounding box size.

[0076] Edge set E: When the shortest distance between the three-dimensional bounding boxes of two targets is less than the safety perception threshold (default 5m), a bidirectional directed edge is established between the two nodes; the feature vector of the edge includes the distance between the two targets, the relative velocity vector, the historical distance change rate, and the physical connection relationship (such as the connection between the lifting device and the aerial box through a steel wire rope, which is marked as a strongly constrained edge).

[0077] Graph update frequency: Based on the laser sensor frame rate (10Hz), the graph structure (node ​​features and edge set) is updated once per frame.

[0078] (2) A time-series graph-based attention network is used to predict the future trajectory distribution of each node. Based on the predicted trajectory distribution of each node, the collision probability between all targets is calculated, and the joint collision probability matrix is ​​output. Among them, a dock physical constraint layer is added after the network prediction layer to constrain the prediction results using prior knowledge.

[0079] In this embodiment, the main technical elements of the Temporal Graph Attention Network (TGAT) are as follows: Temporal modeling: A graph convolutional network (GCN) is used to aggregate the interaction information of neighboring nodes in the spatial dimension, and a Transformer temporal encoder is used to capture the motion pattern history of each node in the temporal dimension (time window T=2 seconds).

[0080] Graph attention mechanism: The edge attention weights are jointly determined by the feature similarity and physical proximity of the two endpoints, enabling the model to automatically give higher attention to target pairs with stronger physical connections (such as a lifting device and a container connected by a wire rope).

[0081] Node state prediction: The model predicts the motion trajectory distribution (mean + covariance) of each node within the next T seconds.

[0082] Joint collision probability matrix output: Based on the predicted trajectory distribution of each node, calculate the collision probability between all target pairs and output an N×N dimensional joint collision probability matrix P (N is the number of targets in the current scene).

[0083] This embodiment adds a wharf physical hard constraint layer after the TGAT prediction layer, which significantly reduces the prediction search space by utilizing prior knowledge. The main constraints include: Movement direction constraint: The predicted trajectory of the truck node is constrained in the direction of the lane centerline (allowing ±0.5m lateral deviation).

[0084] Crane motion constraints: The horizontal displacement of the spreader nodes is constrained in the direction of the crane track, and the vertical displacement is limited by the speed range controlled by the winch.

[0085] Container rigid body constraint: Ground container (positioning container) nodes are assumed to be stationary targets (velocity is forced to zero) unless the lidar detects that it has actually moved.

[0086] The above constraints are implemented using a differentiable projection layer, without compromising the model's end-to-end training capability.

[0087] (3) Calculate the first-order collision and second-order chain collision probabilities based on the joint collision probability matrix, and implement differentiated management of multi-target collision risks after sorting.

[0088] Based on the joint collision probability matrix P, the system performs chain-like collision risk reasoning: First-order collision: The off-diagonal element P(i,j) of matrix P directly gives the direct collision probability between target i and target j.

[0089] Second-order chain collision: If both P(i,j) and P(j,k) exceed the probability threshold θ1, then calculate the chain collision probability P_chain(i→j→k) of target i being transmitted to target k through target j.

[0090] Risk Priority Ranking: All detected collision risks (including direct collisions and chain collisions) are ranked according to their comprehensive risk scores, and the highest risk target pairs are responded to first, so as to achieve differentiated management of multi-target collision risks.

[0091] S4: The fused sensing data is input into the pre-trained personnel violation detection model to detect personnel behavior within the work area and output the personnel violation detection results. In the personnel violation detection unit of the analysis layer, the personnel violation detection model includes a driver getting out of the vehicle detection sub-model and a personnel intrusion detection sub-model.

[0092] Driver Exit Detection Sub-model: In some operational scenarios at automated terminals, truck drivers are required to remain in the cab to assist with operations. Unauthorized departure from the cab during operations constitutes a serious violation. This sub-model, based on visual object detection and behavior recognition algorithms, continuously monitors the truck cab area. First, it uses object detection algorithms to locate the cab area and driver's position in video images. Then, it analyzes the driver's actions and postures using behavior recognition algorithms. When it detects actions such as opening a door or exiting the truck, it determines this as unauthorized exit and immediately outputs a driver violation warning signal.

[0093] Personnel Intrusion Detection Sub-model: The automated terminal's operating area is a hazardous area, and unauthorized personnel are strictly prohibited from entering. The personnel intrusion detection sub-model continuously monitors the operating area, employing a deep learning-based human detection algorithm (YOLOv5) to identify personnel entering the hazardous area, and combining this with a personnel tracking algorithm (a Kalman filter-based tracking algorithm) to continuously track detected personnel, recording their movement trajectory and position changes in real time. When the human detection algorithm first detects personnel entering the operating area, it immediately outputs a personnel intrusion alarm signal and initiates continuous tracking mode until the personnel leave the hazardous area.

[0094] S5: Conduct a risk level assessment based on the collision risk detection results and personnel violation detection results to determine the current risk level.

[0095] If the collision risk identification model or the personnel violation detection model detects a risk, the detected risk is then identified by type. Risk types include collision risk of large cabinets, collision risk of high and low cabinets, collision risk of two cabinets, equipment collision risk, risk of driver unauthorized exit from the vehicle, and risk of personnel intrusion.

[0096] S6: Based on the current risk level, issue a linkage control instruction through the communication interface with the device host to execute corresponding safety response measures. The risk assessment module evaluates the risk level based on the detection results. The risk levels are divided into three levels: high risk, medium risk, and low risk. High risk corresponds to the situation where a collision is about to occur or it has been detected that a person has entered a high-risk area; medium risk corresponds to the situation where the collision probability is relatively high but there is still a buffer distance or a person is in a warning area; low risk corresponds to the situation where there is a potential collision possibility but the distance is far or a person is in a general monitoring area. The evaluation of the risk level comprehensively considers multiple factors such as collision probability, distance between targets, movement speed, and acceleration.

[0097] The control layer executes corresponding linkage control according to the risk level. The linkage control is achieved through the communication interface with the device host. The specific control strategies are as follows: When the risk level is high risk, the control layer issues an emergency stop instruction to the device host through the communication interface, causing the loading and unloading equipment to stop running immediately to prevent the occurrence of a collision accident; at the same time, trigger an audible and visual alarm to remind the on-site personnel to pay attention to safety; the system automatically records detailed event logs, including information such as risk type, risk level, detection time, and sensor data snapshot, for post-event analysis and traceability.

[0098] When the risk level is medium risk, the control layer issues an equipment deceleration instruction to the device host to reduce the running speed of the loading and unloading equipment, reserving more time and space for risk handling; at the same time, issue a prompt warning signal to remind the operator to pay attention to the current risk status; the system also records event logs.

[0099] When the risk level is low risk, the system only records event logs, does not execute equipment control actions, and continues to maintain a cyclic monitoring state.

[0100] If neither the collision risk identification model nor the personnel violation detection model detects a risk, the system continues cyclic monitoring and returns to S1 to re-perform data collection and analysis.

[0101] The safety control system provided by the embodiments of this application has the following technical advantages: 360-degree full coverage is achieved through 4 cameras installed on the cantilever side of the equipment, and with the three-dimensional space perception of the laser sensor, effectively eliminating the monitoring blind spots of traditional solutions; multi-source data fusion and dedicated AI algorithms make the collision risk detection accuracy rate reach more than 95%, and the personnel violation detection accuracy rate reach more than 98%; the response time from risk detection to linkage control is less than 500 milliseconds; it is expected to reduce the collision accident incidence rate in the container yard operation area by more than 80%; the intelligent safety control system can reduce the need for manual patrols and improve operation efficiency.

[0102] It should be noted that, in the specific implementation process, the above-mentioned control part can be implemented by a hardware processor executing computer-executable instructions in software form stored in memory, which will not be elaborated here. The programs corresponding to the actions performed by the above control circuit can all be stored in the computer-readable storage medium of the system in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0103] The computer-readable storage media mentioned above may include volatile memory, such as random access memory; may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; and may also include combinations of the above types of memory.

[0104] The term "processor" as mentioned above can also refer to a collective of multiple processing elements. For example, a processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor, and it can also be a special-purpose processor.

[0105] It should be noted that the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for safety management and control of automated container terminal truck operations, characterized in that, include: S1, synchronously collect video image data and three-dimensional spatial point cloud data of the work area; wherein, the video image data is collected by multiple high-definition cameras, and the three-dimensional spatial point cloud data is collected by a laser sensor; the multiple high-definition cameras and the laser sensor are installed on the cantilever side of the equipment; S2 performs spatiotemporal synchronization and data fusion of video image data and 3D spatial point cloud data to obtain fused perception data; S3 integrates the perception data into the pre-trained collision risk identification model, performs multi-target detection and tracking of containers, loading and unloading equipment and trucks in the truck operation area, and outputs collision risk detection results; the collision risk identification model includes a large container collision detection sub-model, a high and low container collision detection sub-model, a double container collision detection sub-model and an equipment collision detection sub-model. S4, which integrates sensory data input into a pre-trained personnel violation detection model, detects personnel behavior within the work area and outputs personnel violation detection results; S5. Based on the collision risk detection results and personnel violation detection results, conduct a risk level assessment to determine the current risk level; S6, based on the current risk level, issues linkage control commands through the communication interface with the device host to execute corresponding safety response measures.

2. The method for safety management and control of automated container terminal truck operations according to claim 1, characterized in that, S2 specifically includes: The video image data is subjected to denoising and enhancement processing; The three-dimensional spatial point cloud data is filtered and downsampled. The video image data and the three-dimensional spatial point cloud data are spatiotemporally aligned based on timestamps; The subsequent video image data and 3D spatial point cloud data will be fused at the feature level to obtain the fused perception data.

3. The method for safety management and control of automated container terminal truck operations according to claim 1, characterized in that, The collision detection sub-model for large-bellied cabinets detects collision risks, specifically including: A deep learning-based target detection algorithm is used to identify the large-bellied cabinet target from the fused perception data; The actual width of the large cabinet target was calculated by combining three-dimensional spatial point cloud data; Obtain the real-time distance between the large cabinet target and surrounding obstacles; The real-time distance is compared with a preset safe distance threshold. When the real-time distance is less than the safe distance threshold, a collision warning signal for the large cabinet is output.

4. The method for safety management and control of automated container terminal truck operations according to claim 3, characterized in that, The method also completes the point cloud data of the large-bellied cabinet by reconstructing the cabinet's outline using a sub-model, specifically including: Based on sparse point cloud data from the side of a container, and by integrating depth estimation results from a high-definition camera, the two-dimensional depth map is back-projected onto a three-dimensional world coordinate system through a sensor extrinsic calibration matrix. This data is then registered and fused with spatial point cloud data scanned by a laser sensor to generate an initial multimodal point cloud. Based on the PointNet++ encoder-decoder architecture, local geometric features are extracted and the global shape of the sparse point cloud is predicted. A differentiable standard box-shaped size constraint module is added to the decoder output layer to ensure that the completion result meets the standard length / height size constraints of the container. An ultra-wide regression branch is added at the end of the network to directly output the absolute deviation of the container width from the standard value, which serves as the basis for collision safety judgment. A simulation data augmentation strategy is adopted to apply random occlusion, noise, and ultra-wide deformation to the standard point cloud data to generate training samples covering different occlusion rates to train the network. The initial multimodal point cloud network is completed using a pre-trained network based on the standard box-shaped prior constraints.

5. The method for safety management and control of automated container terminal truck operations according to claim 4, characterized in that, When implementing collision detection for large containers, the process also includes calculating the current occlusion rate of the container point cloud and determining the point cloud data to be input into the large container collision detection sub-model based on the occlusion rate. This includes: The occlusion rate of the current container point cloud is calculated in real time. When the occlusion rate is less than a first threshold, the current container point cloud data is used. When the occlusion rate is between the first and second thresholds, the current container point cloud data is weighted and fused with the point cloud data completed by the large container outline reconstruction sub-model. When the occlusion rate exceeds the second threshold, the point cloud data completed by the large container outline reconstruction sub-model is used, and a low confidence marker is triggered.

6. The method for safety management and control of automated container terminal truck operations according to claim 1, characterized in that, The collision detection sub-model for high and low cabinets detects the risk of collisions between high and low cabinets, specifically including: The height information of different containers within the work area is obtained through laser ranging. Combine visual recognition algorithms to determine the type of each container; Based on the height information and the container type, a three-dimensional spatial model of the high-low container operation scenario is established; Based on the three-dimensional spatial model, the motion trajectory of each container is predicted, the collision risk is assessed, and the collision risk detection results are output.

7. The method for safety management and control of automated container terminal truck operations according to claim 1, characterized in that, The dual-box collision detection sub-model predicts the risk of dual-box collisions, specifically including: The position of each container is determined based on 3D spatial point cloud data, and the axial dimensions of each container are extracted by L-shaped corner fitting algorithm to construct the minimum outer rectangle of each container. Extended Kalman filtering is used to estimate the motion state of the spreader and the unloaded container; A wire rope swing inertia compensation model is adopted to correct the trajectory prediction deviation of the un-dropped container caused by the rope swing; the wire rope swing inertia compensation model is based on the simple pendulum dynamics equation. Based on the state mean and covariance estimated by Kalman filter, the three-dimensional motion trajectory of the undropped container within the next T seconds is predicted, and an ellipsoidal trajectory confidence envelope is generated. The three-dimensional space intersection calculation is performed between the envelope and the smallest outer rectangle of the already placed container. If there is an overlap, it is determined that there is a collision risk, and the collision time is predicted and graded warnings are issued.

8. The method for safety management and control of automated container terminal truck operations according to claim 1, characterized in that, The equipment collision detection sub-model is used to detect the collision risk between containers and loading / unloading equipment and trucks, specifically including: Define all monitored devices in the scene as nodes. When the shortest distance between the 3D bounding boxes of two targets is less than the security perception threshold, establish a bidirectional directed edge between the two nodes. A temporal graph-based attention network is used to predict the future trajectory distribution of each node. Based on the predicted trajectory distribution of each node, the collision probability between all targets is calculated, and a joint collision probability matrix is ​​output. A dock physical constraint layer is added after the network prediction layer to constrain the prediction results using prior knowledge. The first-order collision and second-order chain collision probabilities are calculated based on the joint collision probability matrix, and then differentiated management of multi-target collision risks is implemented after sorting.

9. The method for safety management and control of automated container terminal truck operations according to claim 1, characterized in that, The personnel violation detection model includes a driver exit detection sub-model and a personnel intrusion detection sub-model. The driver exit detection sub-model monitors the truck cab area based on visual target detection and behavior recognition algorithms. When it detects that the driver has left the cab during operation, it outputs a driver violation exit alarm signal. The personnel intrusion detection sub-model continuously monitors the work area, uses a human detection algorithm to identify personnel entering the dangerous area, combines a personnel tracking algorithm for continuous monitoring, and outputs a personnel intrusion alarm signal.

10. An automated container terminal truck safety management and control system, characterized in that, include: The perception layer includes multiple high-definition cameras and laser sensors installed on the cantilever side of the equipment, used to synchronously collect video image data and three-dimensional spatial point cloud data of the work area; The analysis layer includes an edge computing unit, a collision risk identification unit, and a personnel violation detection unit; the edge computing unit performs spatiotemporal synchronization and data fusion of video image data and 3D spatial point cloud data to obtain fused perception data; The collision risk identification unit integrates perception data into a pre-trained collision risk identification model to perform multi-target detection and tracking of containers, loading and unloading equipment, and container trucks within the truck operation area, and outputs collision risk detection results. The collision risk identification model includes a large container collision detection sub-model, a high-low container collision detection sub-model, a double container collision detection sub-model, and an equipment collision detection sub-model. The personnel violation detection unit integrates perception data into a pre-trained personnel violation detection model to detect personnel behavior within the operation area and outputs personnel violation detection results. The control layer is used to assess the risk level based on the collision risk detection results and personnel violation detection results, determine the current risk level, and issue linkage control commands through the communication interface with the equipment host. The execution layer is used to execute the instructions issued by the control layer and to carry out the corresponding security response measures.

Images