A shore crane anti-ramming tracking method and device based on machine vision and PLC linkage, and a medium

Abstract

The present application belongs to the technical field of port machinery safety monitoring, and particularly relates to a shore-to-ship crane anti-smashing tracking method, device and medium based on machine vision and PLC linkage, which deploys a first fixed camera, a second fixed camera and a pan-tilt camera. Video stream and spreader state data are collected in real time; a target detection and tracking model is run to generate target identification and trajectory, and a spreader projection danger area is calculated. The second camera target is mapped to the first camera screen to generate a virtual warning box. A pan-tilt smooth tracking instruction is generated in combination with the target trajectory; the pan-tilt is driven to rotate and zoom, so that the target is centered and a preset screen ratio is maintained. When the target enters the danger area and the spreader is below the safety threshold, a linkage control signal is sent to the PLC and an audible and visual alarm is started. The present application realizes the calculation of the spreader projection danger area, and reduces the anti-smashing false alarm rate. The pan-tilt tracking based on PLC signal and visual trajectory fusion realizes automatic safety protection and improves the safety of shore-to-ship crane operation.

Description

Technical Field

[0001] This invention belongs to the field of port machinery safety monitoring technology, and specifically relates to a method, equipment and medium for tracking and preventing collisions with quay cranes based on machine vision and PLC linkage. Background Technology

[0002] Quay cranes (shore container cranes) are equipment used for loading and unloading operations at the forefront of container terminals. During quay crane operations, the operator is located in the cab tens of meters above the ground, and controls the movement of the trolley, the operation of the spreader, and the lifting and lowering of the spreader through the operating handles to complete the grabbing and stacking of containers.

[0003] Due to the complex operating environment, involving various dynamic factors such as the movement of container trucks and the obstruction caused by stacked containers, the driver's field of vision is severely limited. Especially in quay crane operations, the structure of the spreader itself inevitably obstructs the area directly below it, creating blind spots for fixed cameras. Although additional cameras can be added to cover this area, the driver needs to simultaneously observe multiple independent video feeds, potentially overlooking hazardous targets in a particular feed, thus increasing the risk of accidents during the lifting operation.

[0004] Furthermore, in port quay crane operations, distant personnel targets occupy a small pixel area in images, resulting in a limited sample size and indistinct features. Conversely, close-range targets have a larger sample size. Directly using these samples for model training can lead to biased detection results towards large-scale targets, while small-scale personnel are prone to being missed. The model's reliability in complex operating environments cannot meet on-site safety requirements. The model's detection network lacks sufficient perception of the target's operating environment and relative position. When a target is partially obscured by lifting equipment or cargo, it cannot rely on surrounding environmental information to assist in determining the target's category and location, resulting in insufficient continuous target recognition capabilities. Summary of the Invention

[0005] This invention provides a method for tracking and preventing collisions on quay cranes based on machine vision and PLC linkage. The method enables collaborative perception of the movement status of the spreader and the surrounding environment, upgrading safety protection from passive response to active prediction and intervention, improving the accuracy and timeliness of early warning, and enhancing human-machine interaction efficiency and operational continuity.

[0006] The methods include: S1. Install a first fixed camera for anti-collision monitoring, a second fixed camera for covering the blind spot under the spreader, and a PTZ camera for automatic tracking at the designated positions of the sea-side portal frame, land-side portal frame, and boom of the quay crane, respectively. Load the pre-trained target detection and tracking model into the data processing center. S2. Real-time video streams of the quay crane operation area are collected using the first and second fixed cameras, and the status data of the spreader is obtained in real time through the quay crane PLC control system. S3. The data processing center runs a target detection and tracking model to detect and track targets in the two video streams in real time. It assigns a unique identifier to each target and generates the target's actual motion trajectory. Based on the spreader status data, it maps the spatial position of the spreader to the ground and analyzes and updates the real-time projected danger zone of the spreader. S4. The image coordinates of the blind zone target identified by the second fixed camera are mapped to the image coordinate system of the first fixed camera through the pre-calibrated spatial transformation relationship between cameras. A virtual warning box is generated at the corresponding coordinate position, and the virtual warning box is superimposed on the monitoring screen of the first fixed camera to achieve unified visualization of the blind zone target on the main monitoring screen. S5. The data processing center receives the spreader motion control signals from the quay crane PLC control system in real time, analyzes them to obtain the spreader's motion direction and speed, and combines them with the actual target motion trajectory obtained in S3 to select high-risk targets related to the spreader operation as tracking objects. Based on this, it configures the gimbal motion path and generates a control command sequence. S6. The data processing center sends the generated control command sequence to the PTZ camera to drive the PTZ camera to run and track the target. S7. The data processing center will merge and display the images from the first fixed camera with the superimposed virtual warning frame, the tracking images from the PTZ camera, and the target risk information on the driver's cab monitor. Based on the spatial overlap between the actual detection position of the target and the real-time projected danger zone, the center will make a judgment. When the target is determined to have entered the real-time projected danger zone and the actual height of the spreader is lower than the preset safety threshold, the center will send a linkage control signal to the quay crane PLC control system to trigger the spreader to decelerate or stop operation.

[0007] According to another embodiment of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the quay crane anti-collision tracking method based on machine vision and PLC linkage.

[0008] According to another embodiment of this application, a storage medium is also provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the quay crane anti-collision tracking method based on machine vision and PLC linkage are implemented.

[0009] As can be seen from the above technical solutions, the present invention has the following advantages: The present invention provides a quay crane anti-collision tracking method based on machine vision and PLC linkage. A first fixed camera, a second fixed camera, and a pan-tilt camera are installed at designated locations on the sea-side gate frame, land-side gate frame, and boom of the quay crane. The first fixed camera is responsible for anti-collision monitoring, the second fixed camera covers the blind spot below the spreader, and the pan-tilt camera automatically tracks operation-related targets, achieving multi-device collaborative monitoring. A pre-trained target detection and tracking model is loaded into the data processing center. The model training data uses video frames collected from actual port operation scenarios, adapting to target recognition requirements under different lighting and working conditions. During model operation, feature extraction, multi-scale target detection, pose recognition, and multi-target tracking are performed frame-by-frame on the video stream. A unique identifier is assigned to each target, and its motion trajectory is recorded.

[0010] This invention pre-calibrates the fixed cameras to establish a spatial transformation relationship between their image coordinate systems. The target image coordinates identified by the second fixed camera are mapped to the first fixed camera's image coordinate system through this spatial transformation, generating a virtual warning box associated with the target's state. This warning box is then overlaid on the first fixed camera's monitoring screen, enabling visualization of targets in blind spots. The data processing center determines the real-time projected danger zone of the lifting equipment based on its status data and dimensions. Combining the positional relationship between the target and the danger zone with the lifting equipment height, a risk classification is determined. When the target enters the danger zone and the lifting equipment height reaches a preset safety threshold, a linkage control signal is sent to the PLC control system, triggering a deceleration or stop command for the lifting equipment and driving an audible and visual alarm to issue a graded alarm. Attached Figure Description

[0011] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 Flowchart of a quay crane anti-collision tracking method based on machine vision and PLC linkage; Figure 2 A diagram showing the camera setup; Figure 3 A schematic diagram illustrating the framework for implementing a tracking method to prevent collisions with quay cranes; Figure 4 This is a schematic diagram of an electronic device. Detailed Implementation

[0013] This invention provides a quay crane anti-collision tracking method based on machine vision and PLC linkage. It constructs a perception network by installing multi-view cameras and communicates in real-time with the quay crane's PLC system. The data processing center integrates video streams and PLC data, runs a target detection model to identify and track personnel and vehicles, and calculates the dangerous area projected onto the spreader. The target coordinates identified by the blind-spot cameras are mapped to the main monitoring screen for enhanced display. Based on PLC action signals and target trajectories, instructions are generated to drive the PTZ camera to perform adaptive smooth tracking and intelligent mapping of high-risk targets. Finally, the data is displayed, and when a target is determined to have entered a dangerous area, a linkage control signal is sent to the PLC to trigger deceleration or stoppage and activate an audible and visual alarm. This invention achieves collaborative perception of the spreader's movement status and the surrounding environment, upgrading safety protection from passive response to proactive prediction and intervention, improving the accuracy and timeliness of early warnings, and enhancing human-machine interaction efficiency and operational continuity.

[0014] It should be understood that, when used in this specification, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0015] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figures 1 to 3 The diagram shows a flowchart of a quay crane anti-collision tracking method based on machine vision and PLC linkage in a specific embodiment. The method includes: S1. Install a first fixed camera 1 for anti-collision monitoring, a second fixed camera 2 for covering the blind spot under the spreader, and a PTZ camera 3 for automatic tracking at the designated positions of the quay crane's sea-side portal frame, land-side portal frame, and boom. Establish a communication connection between the first fixed camera 1, the second fixed camera 2, the PTZ camera 3, the quay crane's PLC control system 4, and the data processing center 5, and load a pre-trained target detection and tracking model into the data processing center.

[0018] In some embodiments, the first fixed camera 1 is a high-definition network camera equipped with a wide-angle lens, installed in the middle of the crossbeams of the seaside and landside portal frames of the quay crane, with the lens facing the operating area of ​​the spreader to ensure coverage of the side and surrounding 20 meters of the spreader.

[0019] Furthermore, the second fixed camera 2 is a high-definition network camera, installed on the bracket below the rear of the driver's cab, with the lens pointing vertically downwards, covering a blind spot of 2 meters × 2 meters directly below the lifting device.

[0020] Furthermore, the PTZ camera 3 supports 360° continuous rotation and is installed on the gantry crane boom near the trolley, with the lens facing the working area of ​​the spreader. Communication is via industrial Ethernet, establishing communication connections between the PTZ camera, the first fixed camera, the second fixed camera, and the data processing center.

[0021] The PTZ camera is installed at a designated position on the boom of the quay crane. The position is selected in an area that can completely cover the working range of the spreader and can follow the movement of the spreader to achieve full-area tracking, taking into account both the field of view of the work-related targets and the area where the spreader is operating.

[0022] Optionally, a linkage communication is established based on the PLC control system 4 of the quay crane based on the PLC signal and data processing center 5.

[0023] Furthermore, the target detection and tracking model is pre-trained on an offline server. The training data consists of video frames collected from actual port quay crane operation scenarios, including targets such as humans, container trucks, and forklifts under different lighting conditions and based on different working conditions such as lifting, unloading, and moving. During the training process, the gradient descent algorithm is used to optimize the model parameters, and the training is completed after multiple iterations. The trained model is then configured in the data processing center.

[0024] S2. Video streams of the quay crane operation area are collected in real time by the first fixed camera and the second fixed camera, and the spreader status data is obtained in real time by the quay crane PLC control system. The spreader status data includes at least the spreader height, trolley position and crane position.

[0025] In some embodiments, the video streams captured by both the first fixed camera and the second fixed camera are encoded in H.264 format and transmitted in real time to the data processing center via industrial Ethernet.

[0026] The quay crane PLC control system collects various status data of the spreader in real time through position sensors and height sensors. The spreader height is collected by a pull-wire displacement sensor, and the trolley position and the crane position are collected by encoders, which record the distance the trolley moves along the boom and the distance the crane moves along the track, respectively.

[0027] After receiving the video stream and the status data of the lifting device, the data processing center stores the data in association according to the timestamp.

[0028] S3. The data processing center runs a target detection and tracking model to detect and track targets in the two video streams in real time. It assigns a unique identifier to each target and generates the target's actual movement trajectory. Based on the spreader status data, it maps the spatial position of the spreader to the ground and analyzes and updates the real-time projected danger zone of the spreader.

[0029] The following is a specific embodiment of step S3 in the present invention, which specifically includes the following steps: S31. Input the video streams collected by the first fixed camera and the second fixed camera into the target detection and tracking model respectively, extract the target position coordinates, target category and current frame timestamp of each frame image, assign a unique tracking identifier to each target, and receive the real-time speed data of the spreader trolley and the trolley from the PLC control system of the quay crane. Use the speed of the spreader trolley and the speed of the trolley as external motion field constraints to predict the direction of the target motion state, so that the predicted offset of the target in the direction of spreader movement is positively correlated with the actual speed of spreader movement.

[0030] In some embodiments, the video streams from the first fixed camera and the second fixed camera are independently input into the target detection and tracking model. The processing output of each frame image is a set of detection boxes. Each detection box contains six fields: the x-coordinate of the top left pixel, the y-coordinate of the top left pixel, the x-coordinate of the bottom right pixel, the y-coordinate of the bottom right pixel, the target category confidence score, and the target category number.

[0031] Furthermore, the target detection and tracking model assigns candidate tracking labels to each detection box in the output detection box. The assignment rule for candidate tracking labels is as follows: the detection box of the current frame is matched with the detection boxes of the previous frame that have been assigned tracking labels using the intersection-union ratio (IU). If the IU is greater than a preset threshold of 0.45, they are considered to be the same target and the tracking label of the previous frame is retained. If the IU is less than or equal to the preset threshold of 0.45, they are considered to be newly appearing targets and new tracking labels are assigned.

[0032] Optionally, the tracking identifier uses a 64-bit unsigned integer encoding, with the high 16 bits recording the camera number, the middle 16 bits recording the date sequence number of the first appearance, and the low 32 bits recording the cumulative appearance sequence number for that day.

[0033] The quay crane PLC control system pushes real-time speed data of the spreader trolley and the main trolley to the data processing center. The speed data is in millimeters per second and has a positive or negative sign. A positive value indicates that the trolley is moving towards the sea or the main trolley is moving towards the land, and a negative value indicates that it is moving in the opposite direction.

[0034] Optionally, the state vector of the Kalman filter is designed as an eight-dimensional vector x=[u,v, , ,w,h, , ] Where u and v are the pixel coordinates of the center point of the detection box, and Let w be the pixel velocity component, and h be the width and height of the detection box. and The width and height change rate is given. The state transition matrix F of the Kalman filter is set to a constant velocity model, assuming that the target maintains a constant velocity between adjacent frames, i.e., u = uprev + prev*Δt.

[0035] Furthermore, the state transition matrix F is modified, and the speed V of the spreader trolley is adjusted. trolley The speed of the trolley and the speed of the spreader trolley, Vgantry, are superimposed after normalization transformation. and In terms of components, the position prediction term in the corrected state transition matrix F' is u=u prev +( prev +α*V trolley )*Δt.

[0036] Where α is a preset scaling factor, ranging from 0.03 to 0.08, determined by the camera's installation height and viewing angle parameters, and Δt is the time interval between two adjacent frames. This correction ensures that the predicted motion direction of the target in the image plane aligns with the physical motion direction of the rigging. When the rigging moves the camera's field of view as a whole, the predicted position of the target shifts accordingly, reducing tracking loss caused by the camera's movement with the rigging.

[0037] Furthermore, the calibration of the proportionality coefficient α was completed through on-site testing. The specific method was as follows: when the trolley moved at different constant speeds, the pixel displacement velocity of the stationary target in the image was collected, the ratio of the pixel displacement velocity to the trolley's moving speed was calculated, and the average value of multiple measurements was taken as the α value.

[0038] S32. Align the target trajectories obtained by the first fixed camera and the second fixed camera according to the timestamp, and calculate the position of the target trajectory with the same tracking mark in the respective image coordinate system of the two cameras.

[0039] When a target with a tracking identifier is continuously lost in the field of view of the first fixed camera for more than a preset frame number threshold, the trajectory data of the target in the last five frames before it is lost is stored in the occlusion temporary storage area, including the position coordinates, width and height dimensions and velocity vector of each frame. The target in the occlusion temporary storage area will no longer participate in the matching calculation of the current frame in subsequent frames. When the target reappears in the field of view of the second fixed camera, the last stored trajectory data in the occlusion temporary storage area is extracted and matched with the reappearing detection box for feature vector similarity. If the similarity exceeds the preset threshold, the original tracking identifier is reassigned to the target, and the virtual warning box display of the target in the first fixed camera's view is restored.

[0040] In some embodiments, the first fixed camera and the second fixed camera each independently run the dual-modal target trajectory prediction and ID allocation mechanism described in S31 to generate two independent trajectory sets, each trajectory set containing multiple tracking identifiers and their corresponding historical location sequences.

[0041] For a target whose tracking marker appears in the field of view of two cameras simultaneously, its pixel coordinates P1 in the first fixed camera frame and P2 in the second fixed camera frame are recorded respectively, and both coordinates are retained in the target trajectory history record.

[0042] Furthermore, when the number of consecutive frames lost by a target with a certain tracking identifier in the field of view of the first fixed camera exceeds a preset frame threshold, the target is determined to be in an occlusion state. The counting rule for consecutive lost frames is as follows: if there is no detection box matching the tracking identifier in the current frame detection result, the count is incremented by 1. If there is a matching detection box, the count is reset to zero. After determining that the target is in an occlusion state, the trajectory data of the last five frames before the loss is extracted and stored in the occlusion temporary storage area. The tracking identifier is removed from the active trajectory set, and the prediction and matching of the target are stopped. Each target in the occlusion temporary storage area records its last position before loss, target category, feature vector, and loss timestamp.

[0043] The original tracking identifier and the last five frames of trajectory data of the target are retrieved from the occlusion buffer. The original tracking identifier is assigned to the new target, and the velocity vector in the last five frames of trajectory data is used as the initial velocity value of the Kalman filter to achieve a smooth recovery of the tracking state. The virtual warning box is displayed as follows: when the target in the occlusion buffer is re-matched, the current position of the target in the second fixed camera frame is mapped to the first fixed camera frame through the spatial transformation relationship described in S4, generating a virtual warning box and displaying it overlaid until the target leaves the field of view of the second fixed camera.

[0044] S33. The system acquires real-time data on the spreader's height, trolley position, and gantry position, as well as its roll, pitch, and yaw angles, from the quay crane's PLC control system. Based on the spreader's geometric parameters, it establishes the three-dimensional coordinates of the four corner points of the spreader's bottom surface in the world coordinate system. The four corner points are then projected onto the ground plane to obtain four ground projection points. Connecting these four ground projection points forms a real-time projection danger zone polygon for the spreader. During the projection process, the offset of the projection points is corrected based on the spreader's roll and pitch angles. When the spreader has a non-zero roll or pitch angle, a horizontal offset occurs between the vertical projection points of the four corner points of the spreader's bottom surface and the actual projection points. The horizontal offset is calculated by multiplying the sine values ​​of the roll and pitch angles by the spreader's height.

[0045] In some embodiments, the quay crane PLC control system outputs spreader status data. This data includes spreader height H, trolley position, and crane position, as well as roll angle φ, pitch angle θ, and yaw angle ψ output by a six-axis inertial measurement unit mounted above the spreader. The system reads the corresponding dimensional parameters based on the specifications of the container being handled. The coordinates of the four corner points of the spreader's bottom surface in the world coordinate system are calculated using the spreader's center point as a reference. The world coordinates of the spreader's center point are determined by the trolley and crane positions, and the height coordinates of the spreader's center point are the spreader height H.

[0046] Furthermore, let the length of the spreader's base be L and the width be W. The offsets of the four corner points relative to the spreader's center point in the spreader's own coordinate system are respectively. These four offsets are then transformed by roll angle φ and pitch angle θ to obtain the offsets in the world coordinate system.

[0047] The formula for rotation transformation is: the component of offset in the X direction of the world coordinate system = the component of offset in the X direction of the spreader coordinate system × cosθ + the component of offset in the Z direction of the spreader coordinate system × sinθ.

[0048] The component of the offset in the Y direction of the world coordinate system = the component of the offset in the Y direction of the spreader coordinate system × cosφ + the component of the offset in the Z direction of the spreader coordinate system × sinφ × (the coupling term between the X and Y directions of the spreader coordinate system).

[0049] The world coordinates of the four corner points of the spreader's bottom surface are the sum of the world coordinates of its center point and the offset after rotation. The ground plane is set as the Z=0 plane, and the projection of the four corner points of the spreader's bottom surface onto the ground plane is divided into two steps: vertical projection and actual projection.

[0050] The vertical projection point is projected vertically downwards along the Z-axis, meaning the X and Y coordinates of the projection point are the same as those of the corner point. The actual projection point takes into account the light obstruction caused by the lifting device's tilt. When the lifting device has a roll angle φ, the shadow area of ​​the lifting device's bottom edge on the ground shifts relative to the vertical projection area in the direction of the lifting device's tilt, with a shift distance droll = H × tanφ.

[0051] When the spreader has a pitch angle θ, the offset distance dpitch = H × tanθ. The actual offset of the projection point in the X direction is dpitch × sinψcomp, and the offset in the Y direction is droll × cosψcomp, where ψcomp is the combined angle of the yaw angle and the spreader direction.

[0052] The four actual projection points, when connected, form a quadrilateral, which represents the real-time projected danger zone polygon of the lifting device. When both the roll and pitch angles are zero, the actual projection area coincides with the vertical projection area. When either the roll or pitch angle is not zero, the actual projection area shifts relative to the vertical projection area, and its shape changes from rectangular to trapezoidal. Each time the system receives new lifting device status data, it recalculates the projected danger zone polygon and outputs the vertex coordinate sequence of this polygon to the anti-collision judgment criteria.

[0053] S34. Determine the spatial inclusion relationship between the real-time projected danger zone polygon of the lifting device calculated in S33 and the positions of each target after the tracking marker is restored in S32. Obtain the current descent speed data of the lifting device and calculate the time required for the lifting device to descend from the current height to the ground at the current descent speed as the warning time margin. When the target enters the projected danger zone polygon and the warning time margin is less than the preset emergency braking time threshold, trigger the emergency braking warning signal. When the target enters the projected danger zone polygon but the warning time margin is greater than or equal to the preset emergency braking time threshold and less than the preset deceleration warning time threshold, trigger the deceleration warning signal. When the target does not enter the projected danger zone polygon but the closest distance between the predicted extension direction of its trajectory and the boundary of the projected danger zone polygon is less than the preset approach distance threshold, trigger the approach warning signal.

[0054] In some embodiments, the target location point is taken from the target trajectory data after the tracking marker is recovered in S32, and the bottom center point of the target detection box in the current frame is taken as the target location point, which corresponds to the position where the target contacts the ground. The projected danger zone polygon is formed by connecting the four vertices output in S33 in a clockwise order.

[0055] Furthermore, a horizontal ray is emitted to the right from the target point, and the number of intersections between the ray and each side of the polygon is calculated. An odd number of intersections indicates the point is inside the polygon, and an even number indicates the point is outside the polygon. The current descent speed Vdown of the spreader is obtained in real time from the quay crane PLC control system; a positive value indicates descent, and a negative value indicates ascent.

[0056] Furthermore, the formula for calculating the warning time margin Twarning is Twarning=Hcurrent / Vdown, where Hcurrent is the current height of the spreader and Vdown is the current descent speed of the spreader.

[0057] Furthermore, when the spreader rises (Vdown) and becomes negative, no warning time margin is calculated, and no warning is triggered. When the emergency braking warning signal is triggered, the data processing center sends a stop operation command to the quay crane PLC control system, the spreader stops descending and is prohibited from further descent, and the audible and visual alarm emits a continuous buzzing sound.

[0058] Optionally, the proximity warning signal is triggered when: the target has not entered the projected danger zone polygon, but the shortest distance between the target's current direction of movement and the boundary of the projected danger zone polygon is less than a proximity distance threshold. The proximity warning signal allows operators to be aware of potential risks in advance, avoiding stressful operations caused by sudden alarms.

[0059] S4. The image coordinates of the blind zone target identified by the second fixed camera are mapped to the image coordinate system of the first fixed camera through the pre-calibrated spatial transformation relationship between cameras. A virtual warning box is generated at the corresponding coordinate position, and the virtual warning box is superimposed on the monitoring screen of the first fixed camera to achieve unified visualization of the blind zone target on the main monitoring screen.

[0060] In some embodiments, a spatial transformation relationship is established through joint camera calibration to convert target coordinates in different camera coordinate systems, mapping blind spot targets onto the main monitoring screen. Target status and risk level are converted into visual signals, enabling the overlay of warning boxes onto the main screen, allowing the driver to observe blind spot targets directly on the monitoring screen without switching between multiple screens.

[0061] Furthermore, the second fixed camera covers the blind spot beneath the spreader, which can be areas that cannot be directly observed by the first fixed camera. By mapping the coordinates of targets in the blind spot to the image coordinate system of the first fixed camera and generating a virtual warning box, the invisible targets in the blind spot can be displayed on the main monitoring screen viewed by the driver, eliminating the inherent blind spots in quay crane operations. The driver can intuitively grasp the location and status of targets in the blind spot without switching between multiple monitoring screens.

[0062] S5. The data processing center receives the spreader motion control signals from the quay crane PLC control system in real time, analyzes them to obtain the spreader's motion direction and speed, and combines them with the actual target motion trajectory obtained in S3 to select high-risk targets related to the spreader operation as tracking objects. Based on this, it configures the gimbal motion path and generates a control command sequence.

[0063] In some embodiments, the spreader motion control signals issued by the quay crane PLC control system include four motion types: spreader lifting, lowering, trolley movement, and trolley movement. The data processing center parses the movement direction and speed of the spreader and extracts the timestamp from the signal to ensure time synchronization with the video stream and spreader status data.

[0064] Furthermore, the data processing center extracts all target motion trajectories, risk levels, and attitude information generated by S3, and filters out targets related to the current lifting operation.

[0065] Optionally, during a container quay crane operation, the data processing center detects three human targets simultaneously within the operation area in real time. The first target is located inside the real-time projection danger zone of the spreader; the system determines its risk level to be dangerous, and the target is set as the highest priority tracking object. The PTZ camera immediately turns and locks onto the target, maintaining continuous tracking. The second target is located at the edge of the danger zone, with a risk level of warning; it is set as medium priority and only included in the PTZ tracking range after the highest priority target disappears. The third target is far from the danger zone, with a risk level of safe; it is set as low priority, and the system does not allocate PTZ tracking resources to it.

[0066] Optionally, based on the motion trajectory of the tracked target, the predicted position of the target for the next 5 frames is fitted, and the speed and direction of the spreader are combined with the rotational angular velocity of the gimbal, and arranged in chronological order to form a sequence of control commands.

[0067] Furthermore, in quay crane operation scenarios, the degree of danger of a target is tied to the movement state of the spreader. The direction and speed of the spreader's movement determine the changing trend of the real-time projected danger zone, and also determine which targets are operation-related targets requiring focused attention. After analyzing the spreader's motion parameters, high-risk targets near the moving spreader can be identified, while distant targets unrelated to the operation can be excluded. The spreader's motion parameters allow the gimbal tracking action to remain synchronized with the rhythm of the on-site operation, improving tracking stability.

[0068] S6. The data processing center sends the control command sequence generated by S5 to the PTZ camera, driving the PTZ camera to rotate continuously and automatically adjust the focal length so that the tracked target is always kept in the center of the screen and occupies the preset screen ratio range. The tracked target is the current operation associated target determined by S5 based on the target's motion trajectory and the lifting device action control signal.

[0069] In some embodiments, the data processing center sends the control command sequence generated by S5 to the PTZ camera's driver module one item at a time. After receiving the commands, the PTZ camera driver module parses parameters such as rotation angle, angular velocity, and focal length adjustment, and controls the PTZ to rotate, adjusting the horizontal and vertical rotation angles.

[0070] Furthermore, real-time images captured by the PTZ camera are collected, the outlines of the currently associated targets in the images are extracted, the proportion of the target's current area in the total image area is calculated, and compared with a preset proportion range. Optionally, the proportions are 2 / 3 to 3 / 4 for the lifting equipment, 1 / 2 to 2 / 3 for the human body, and 3 / 5 to 4 / 5 for the container truck head.

[0071] If the target proportion is less than the preset lower limit, gradually increase the focal length. If the target proportion is greater than the preset upper limit, gradually decrease the focal length while keeping the gimbal rotation speed constant.

[0072] Each time the focus is adjusted, a gimbal rotation feedback signal is collected, and the actual rotation parameters are compared with the command parameters. If there is a deviation, the rotation angular velocity is fine-tuned to ensure that the target moves smoothly towards the center of the image. Each frame is captured and the target position is identified. The offset between the target center and the image center is calculated. If the offset exceeds a preset threshold, the gimbal rotation angle and focus are fine-tuned, selectively for loaders ≤5 pixels, human bodies ≤8 pixels, and truck heads ≤6 pixels, until the target returns to the preset range.

[0073] If the target is obstructed, maintain the current gimbal attitude, start the area scanning mode, control the gimbal to scan the area within the working radius of the lifting device, combine the target motion trajectory prediction in S3, focus on scanning the area where the target may appear, and immediately resume normal tracking after re-capturing the target.

[0074] As can be seen, the gimbal drive module parses commands and controls the gimbal rotation. Focus adjustment works in tandem with gimbal rotation, dynamically adjusting the focus based on the target's proportion in the image to ensure target clarity. Anomalies such as occlusion are handled by maintaining the target's orientation and scanning key areas to ensure tracking continuity.

[0075] S7. The data processing center will merge and display the images from the first fixed camera with the superimposed virtual warning frame, the tracking images from the PTZ camera, and the target risk information on the driver's cab monitor. Based on the spatial overlap between the actual detection position of the target and the real-time projected danger zone, the center will make a judgment. When the target is determined to have entered the real-time projected danger zone and the actual height of the spreader is lower than the preset safety threshold, the center will send a linkage control signal to the quay crane PLC control system to trigger the spreader to decelerate or stop operation and drive the audible and visual alarm to sound an alarm.

[0076] In some embodiments, the data processing center can display the monitoring screen of the first fixed camera with the virtual warning box superimposed on it and the real-time tracking screen of the PTZ camera in a split screen, which can display the screen of the first fixed camera and the PTZ tracking screen at the same time.

[0077] The information bar on the display screen shows the positional relationship between each target determined in S3 and the real-time projected danger zone, the target risk level, and the real-time status of the lifting equipment. Colors can be matched with target risk levels to ensure that the driver can see them clearly.

[0078] Furthermore, the preset safety threshold can be set according to the actual needs of port operations. The data processing center compares the positional relationship between the target and the real-time projected danger zone, as well as the spreader height with the preset safety threshold in real time. If the target is detected to have entered the real-time projected danger zone and the spreader height is less than or equal to 5-10 meters, a linkage control signal is sent to the quay crane PLC control system, including control for deceleration / stop, target entering the danger zone, and spreader height being too low. After receiving the signal, the quay crane PLC control system triggers a deceleration or stop operation command for the spreader. If the target partially overlaps with the danger zone, a deceleration command is triggered; if the target completely overlaps with the danger zone, a stop command is triggered, locking the spreader movement until the operator confirms safety and unlocks it.

[0079] Furthermore, the judgment process is based on actual detection and data collection. The target detection and tracking model identifies the target's actual position in real time. Combined with measured data such as the spreader height, trolley position, and crane position collected by the control system, the true real-time projected danger zone of the spreader is obtained. The spatial overlap between the target's actual position and this danger zone is then compared. The actual height of the spreader is then read to determine if it is below a preset safety threshold.

[0080] The data processing center activates the audible and visual alarm. When a hazard level is triggered, the alarm emits a flashing red light and a continuous buzzing sound. When a warning level is triggered, it emits a flashing orange light and an intermittent buzzing sound to alert the driver. The buzzing sound can be manually turned off via the control panel in the driver's cab.

[0081] In one embodiment of the present invention, the method of training the target detection and tracking model in the data processing center in step S3 includes the following steps: S11. Construct a multi-scenario sample database. Collect operation videos under different lighting conditions, weather conditions, different spreader heights, and different degrees of obstruction from fixed cameras deployed at different locations on the quay crane. Extract image frames containing personnel, truck cabs, spreaders, and other operation vehicles. Label the category labels and location coordinate boxes of various targets appearing in each image frame. Divide the labeled image frames into training set, validation set, and test set according to a preset ratio.

[0082] Among them, a scale-layered labeling strategy is established for the scale range of human targets in the image. The human targets are divided into three levels: small scale, medium scale and large scale according to the pixel area, and the sample distribution is statistically analyzed in each level to keep the sample number of each level balanced.

[0083] In some embodiments, the bounding rectangle of each target is labeled during annotation, and the coordinates of the four vertices of the rectangle are accurate to the pixel level.

[0084] Furthermore, the specific implementation of the scale-layered annotation strategy is as follows: sort the pixel areas of the people targets in the image from smallest to largest, with 30 to 80 pixels as the small-scale interval, 81 to 200 pixels as the medium-scale interval, and above 201 pixels as the large-scale interval. The number of samples in each interval is counted independently. Data augmentation techniques are used to expand the samples in the small-scale interval, including random rotation, brightness adjustment, and contrast stretching of the image region where the small-scale target is located, to generate new training samples, so that the ratio of the number of samples in the three intervals is controlled within the range of 1:1:1 to 1:2:1.

[0085] Furthermore, after annotation, the samples are allocated to the training set, validation set, and test set in a ratio of 7:2:1. The training set is used for model parameter learning, the validation set is used for hyperparameter tuning and model selection, and the test set is used for final performance evaluation.

[0086] S12. Construct a dual-branch parallel feature extraction network. The network consists of a backbone feature extraction module, a target enhancement branch, and a context-aware branch connected in parallel. The backbone feature extraction module extracts high-level semantic features of the image step by step. The target enhancement branch connects to a spatial attention unit at the output of the first two layers of the feature map of the backbone module. The spatial attention unit performs global average pooling and global max pooling on the input feature map in the channel dimension to generate a spatial attention weight map. The spatial attention weight map is multiplied element-wise with the original feature map to output an enhanced small-scale feature map. The context-aware branch connects to a dilated convolutional pyramid unit at the output of the middle layer of the feature map of the backbone module. The dilated convolutional pyramid unit contains three parallel dilated convolutional layers with different dilation rates. The output feature maps of the three dilated convolutional layers are concatenated in the channel dimension and then dimensionality-reduced by 1×1 convolution to output a context feature map. The enhanced small-scale feature map output by the target enhancement branch is concatenated with the context feature map output by the context-aware branch in the channel dimension, and then a fused feature map is output through a feature fusion layer.

[0087] In some embodiments, the backbone feature extraction module adopts a cross-stage local network structure, and the input image size is set to 1280×732 pixels. The size selection is based on the constraint that the minimum pixel size of personnel targets in the port scene is 30×80 pixels. After 32 times downsampling, small-scale targets can still retain at least 1×2 pixel response areas on the feature map.

[0088] Furthermore, the backbone module comprises four downsampling stages, with each stage outputting feature maps of sizes 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input size, respectively. The target enhancement branch connects the 1 / 4-size feature map output from the first downsampling stage and the 1 / 8-size feature map output from the second downsampling stage of the backbone module. These two feature maps preserve the spatial detail information of small-scale targets.

[0089] The specific structure of the spatial attention unit is as follows: For the input feature map F with size H×W×C, global average pooling and global max pooling are first performed in the channel dimension to obtain two pooling results of H×W×1. These two results are concatenated in the channel dimension and then passed through a 7×7 convolutional layer and a sigmoid activation function to generate a spatial attention weight map M with size H×W×1. M is then multiplied element-wise with F to obtain the enhanced feature map.

[0090] Furthermore, the context-aware branch connects to the 1 / 16-size feature map output from the third downsampling stage of the main module. This feature map has medium resolution, preserving both spatial information and high-level semantic expression. The dilation rates of the three dilated convolutional layers in the dilated convolutional pyramid unit are set to 2, 4, and 6, respectively, with a kernel size of 3×3. The number of output channels for the three parallel convolutions is one-third of the number of channels in the original feature map. After concatenating the three output feature maps along the channel dimension, the number of channels is restored to the original number. Then, a 1×1 convolution is performed for cross-channel information fusion to output the context feature map.

[0091] The feature fusion layer upsamples the enhanced small-scale feature map to the same size as the context feature map through bilinear interpolation, then concatenates it with the context feature map in the channel dimension, and then outputs a fused feature map through a 3×3 convolutional layer and a batch normalization layer. The fused feature map contains both the fine spatial information of the small-scale target and the contextual environment information of the medium-scale target.

[0092] S13. Input the fused feature map output from S12 into the detection head network. The detection head network outputs the target classification confidence, target position regression offset, and target scale level attribution probability. Train the three outputs using a loss function, which is composed of a weighted sum of classification loss, regression loss, and scale level perception loss.

[0093] The scale-level perception loss term uses the scale-level labels marked in S11 as supervision signals and the cross-entropy between the true scale level of the target and the predicted scale level probability as the loss value. During backpropagation, the gradient simultaneously updates the detection head network and the dual-branch parallel feature extraction network in S12. The training is iterated until the joint loss function converges, resulting in the trained target detection and tracking model.

[0094] In some embodiments, the detection head network treats each feature point on the fused feature map as a prediction unit, and each feature point directly predicts the target center point location, target width and height, and target category.

[0095] Furthermore, the target classification confidence uses a binary cross-entropy loss function, where positive samples are defined as points where the feature point falls within the target's central region, and negative samples are defined as points where the feature point falls outside the target's central region. The target location regression offset uses a smoothed L1 loss function, with the regression target being the distance from the feature point coordinates to the four sides of the target's bounding box.

[0096] Furthermore, the scale-level perceptual loss term is constructed as follows: During the training sample labeling stage, each target is labeled with its corresponding scale level label according to the scale-level labeling strategy in S11, with small-scale levels labeled as 0, medium-scale levels as 1, and large-scale levels as 2. A three-branch fully connected subnetwork is added to the detection head network, with the input being the feature vector corresponding to the target center point position in the fused feature map, and the output being the probability distribution of the three scale levels.

[0097] Furthermore, during the training process, 16 images were input in each batch, the initial learning rate was set to 0.001, and the learning rate was dynamically adjusted using a cosine annealing strategy. A total of 200 training rounds were conducted, and the model was evaluated on the validation set every 10 rounds. The model with the smallest loss value on the validation set was selected as the object detection and tracking model.

[0098] Furthermore, step S3 also includes: predefining the target detection and tracking model in the data processing center, and performing inference and recognition based on real-time lifting device status data, specifically including the following steps: S14. Perform intrinsic and extrinsic parameter calibration on each fixed camera deployed in S11, obtain the intrinsic parameter matrix, distortion parameters, rotation matrix and translation vector of each camera relative to the world coordinate system of the quay bridge, establish the perspective transformation relationship between the image coordinate systems of different cameras based on the calibration results, and encapsulate the perspective transformation relationship into a coordinate mapping lookup table.

[0099] In some embodiments, intrinsic parameter calibration involves acquiring 20 sets of images at different poses within the camera's field of view using a calibration board. The intrinsic parameter matrix K and distortion parameters for each camera are calculated using Zhang's calibration method. The intrinsic parameter matrix includes focal length parameters and principal point coordinates, while the distortion parameters include radial distortion coefficients and tangential distortion coefficients. Extrinsic parameter calibration involves selecting at least six non-collinear control points on the quay crane's working surface. A high-precision total station is used to measure the three-dimensional coordinates of these control points in the quay crane's world coordinate system. The pixel coordinates of these control points are then marked in the images acquired by each camera. The rotation matrix and translation vector of each camera relative to the quay crane's world coordinate system are analyzed.

[0100] The perspective transformation relationship is established based on the assumption of ground coplanarity, meaning that all targets within the work area lie on the ground plane or approximately on the ground plane. For any two cameras A and B, given a point pA in the image of camera A, the three-dimensional coordinates of its corresponding ground plane point P are obtained by backprojecting the extrinsic and intrinsic parameter matrices of camera A. Then, P is projected onto the image plane of camera B to obtain the mapped point pB. This transformation process can be represented by the homography matrix HAB, satisfying pB = HAB * pA.

[0101] Furthermore, for the n fixed cameras in the quay crane operation scenario, the homography matrices between all pairs of cameras are pre-calculated, resulting in a total of n×(n-1) homography matrices. The parameters in these homography matrices are stored as coordinate mapping lookup tables in floating-point form, with the source camera number and target camera number as indexes.

[0102] S15. Deploy the trained target detection and tracking model and coordinate mapping lookup table to the data processing center. The data processing center receives video streams from each fixed camera in real time, performs target detection and tracking on each frame of image, and outputs the category, position coordinates, unique identifier and motion trajectory of each target.

[0103] It can be seen that by performing intrinsic and extrinsic parameter calibration on the first and second fixed cameras and establishing a perspective transformation relationship based on ground plane constraints, a coordinate mapping lookup table that can be directly called can be formed.

[0104] Furthermore, acquiring camera intrinsic parameters can eliminate positional shifts caused by lens distortion, making the pixel coordinates of the target in the image closer to its real spatial location. Extrinsic parameter calibration is achieved by relying on ground control points collected by a total station, which can uniformly anchor the image coordinates of each camera to the quay crane's world coordinate system, giving the target position a unified spatial reference benchmark from different perspectives.

[0105] Based on the homography matrix transformation relationship constructed on the ground plane, the coordinate mapping calculation between multiple cameras is simplified, ensuring sufficient accuracy when mapping targets in blind spots from the second fixed camera to the first fixed camera. After organizing all homography matrices into a coordinate mapping lookup table and deploying it along with the target detection and tracking model to the data processing center, the system can directly look up the table to complete the coordinate transformation during real-time inference, eliminating the need to repeatedly perform complex matrix operations and improving the processing speed of target recognition and coordinate fusion.

[0106] In one embodiment of the present invention, based on step S4, the image coordinates of the target identified by the second fixed camera are mapped to the image coordinate system of the first fixed camera through a pre-calibrated spatial transformation relationship to generate a virtual warning box. The following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0107] S41. Extract the single-frame target detection result output by the second fixed camera, and obtain the target's unique ID, the image coordinates of the four corners of the detection box, the acquisition timestamp, the target category, and the pose label; traverse the timestamp sequence of the video frames of the first fixed camera, and select video frames whose absolute difference with the target timestamp does not exceed the threshold as alignment frames to complete the time synchronization of the two video streams; call the spatial transformation relationship obtained in advance through joint calibration, and transform the coordinates of the four corners of the target detection box in the image coordinate system of the second fixed camera point by point to obtain the preliminary mapped coordinates in the image coordinate system of the first fixed camera.

[0108] In some embodiments, the second fixed camera outputs target detection results in real time, with each frame containing one or more target information entries. The target image coordinates use a pixel coordinate system, recording the coordinate values ​​of the four vertices of the detection box (top left, top right, bottom left, and bottom right) to locate the target's position within the second fixed camera's view. Target categories include human bodies, container trucks, forklifts, etc.

[0109] The posture labels target human targets and include four types: standing, bending, squatting, and partially occluded. Spatial transformation relationships are pre-observed through joint camera calibration. During calibration, a standard calibration board is placed in the hoisting operation area, and images of the calibration board are captured from the first and second fixed cameras at different angles and distances. The focal length, principal point coordinates, rotation matrix, and translation matrix of the two cameras are analyzed and integrated to form the spatial transformation relationship. This transformation relationship is then used to substitute the coordinates of the four corners of the target detection box under the second fixed camera into the transformation formula, completing the coordinate transformation and obtaining the preliminary mapped coordinates in the image coordinate system of the first fixed camera, ensuring the basic accuracy of the coordinate transformation.

[0110] S42. Extract all target coordinates and ID information identified in the current alignment frame of the first fixed camera, and calculate the overlap between the preliminary mapped coordinates obtained in S41 and the coordinates of each identified target. If the overlap exceeds a preset threshold and the target categories are consistent, associate the ID and attitude label of the blind spot target with the corresponding identified target and use the relevant parameters of the identified target. If the overlap does not exceed the threshold or the categories are inconsistent, confirm the preliminary mapped coordinates as the coordinates of the newly added blind spot target, assign a temporary ID to it and associate it with the corresponding attitude and category label.

[0111] In some embodiments, the data processing center extracts all target information identified in the current aligned frame of the first fixed camera, including the detection box coordinates, unique ID, and category label of each target.

[0112] Furthermore, the overlap between the initial mapped coordinates obtained from S41 and the detection boxes of each identified target from the first fixed camera is analyzed. The preset overlap threshold is 0.8, which was determined through multiple port operation scenario tests. If the overlap between an identified target and the initial mapped coordinates exceeds 0.8, and the target categories of the two are the same, then they are determined to be the same target. The ID and attitude label of the blind spot target are associated with the identified target, and the risk level, occlusion status, and other parameters of the identified target are used. No new virtual warning boxes are added. If the overlap between all identified targets and the initial mapped coordinates does not exceed 0.8, or the overlap exceeds 0.8 but the categories are different, then the initial mapped coordinates are determined to correspond to a new blind spot target. A temporary ID is assigned to it, and its attitude and category labels extracted from S41 are associated with it, marking it as a new blind spot target. A virtual warning box is then configured for it separately. The entire determination process is automatically executed by the target association module of the data processing center, and is completed synchronously with each frame to ensure the real-time nature of the determination results.

[0113] S43. Extract the risk level, posture, and occlusion status parameters of the corresponding target; configure the visual attributes of the virtual warning box according to the target status parameters, and specify the specific parameters of color, line width, flashing frequency, and transparency; superimpose the configured virtual warning box onto the initial mapping coordinate position of the first fixed camera alignment frame to complete the visualization of the blind spot target.

[0114] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0115] like Figure 4 As shown, this application also provides an electronic device, including a display module 103, a memory 102, a processor 101, a communication module 104, and a computer program stored in the memory and executable on the processor 101. When the processor 101 executes the program, it implements the steps of a quay crane anti-collision tracking method based on machine vision and PLC linkage.

[0116] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the embodiments described and / or claimed herein.

[0117] In this embodiment, processor 101 may be implemented using at least one of an application-specific integrated circuit, a programmable logic device, a field-programmable gate array, a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such an implementation may be implemented within a controller. For software implementation, implementations such as processes or functions may be implemented with separate software modules that allow the performance of at least one function or operation. Software code may be implemented by a software application (or program) written in any suitable programming language, and the software code may be stored in memory and executed by the controller.

[0118] The display module 103 is used to display information input by the user or information provided to the user. The display module 103 may include a display panel, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like.

[0119] The memory 102 can be used to store software programs and various data. The memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0120] The communication module 104 transmits radio signals to and / or receives radio signals from at least one of a base station, an external terminal, and a server. Such radio signals may include voice call signals, video call signals, or various types of data sent and / or received according to text and / or multimedia messages.

[0121] The present invention also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the quay crane anti-collision tracking method based on machine vision and PLC linkage.

[0122] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0123] The storage medium stores a program product capable of implementing the methods described above in this specification. In some possible implementations, various aspects of this disclosure may also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0124] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for tracking and preventing collisions on quay cranes based on machine vision and PLC linkage, characterized in that the method... include: S1. Install a first fixed camera for anti-collision monitoring, a second fixed camera for covering the blind spot under the spreader, and a pan-tilt camera at the location of the observation operation path of the quay crane's boom at the designated positions of the sea-side portal frame, land-side portal frame, and boom of the quay crane, and load a pre-trained target detection and tracking model at the data processing center. S2. Real-time video streams of the quay crane operation area are collected using the first and second fixed cameras, and the status data of the spreader is obtained in real time through the quay crane PLC control system. S3. The data processing center runs a target detection and tracking model to detect and track targets in the two video streams in real time. It assigns a unique identifier to each target and generates the target's actual motion trajectory. Based on the spreader status data, it maps the spatial position of the spreader to the ground and analyzes and updates the real-time projected danger zone of the spreader. S4. The image coordinates of the blind zone target identified by the second fixed camera are mapped to the image coordinate system of the first fixed camera through the pre-calibrated spatial transformation relationship between cameras. A virtual warning box is generated at the corresponding coordinate position and the virtual warning box is superimposed on the monitoring screen of the first fixed camera. S5. The data processing center receives the spreader motion control signals from the quay crane PLC control system in real time, analyzes them to obtain the spreader's motion direction and speed, and combines them with the actual target motion trajectory obtained in S3 to select high-risk targets related to the spreader operation as tracking objects. Based on this, it configures the gimbal motion path and generates a control command sequence. S6. The data processing center sends the generated control command sequence to the PTZ camera to drive the PTZ camera to run and track the target. S7. The data processing center will merge and display the images from the first fixed camera with the superimposed virtual warning frame, the tracking images from the PTZ camera, and the target risk information on the driver's cab monitor. The system makes a judgment based on the spatial overlap between the actual detection location of the target and the real-time projected danger zone. When the target is determined to have entered the real-time projected danger zone and the actual height of the lifting device collected is lower than the preset safety threshold, a linkage control signal is sent to the quay crane PLC control system to trigger the lifting device to decelerate or stop operation.

2. The anti-collision tracking method for quay cranes based on machine vision and PLC linkage according to claim 1, characterized in that, Step S3, which involves the data processing center training the target detection and tracking model, includes the following steps: S11. Construct a multi-scene sample database, label the various targets appearing in the acquired image frames with category labels and location coordinate boxes, and divide them into training set, validation set and test set according to a preset ratio; S12. Construct a dual-branch parallel feature extraction network, which consists of a main feature extraction module, a target enhancement branch, and a context-aware branch connected in parallel. The target enhancement branch connects a spatial attention unit at the output of the first two layers of the feature map of the main module to output an enhanced small-scale feature map. The context-aware branch connects a dilated convolutional pyramid unit at the output of the middle layer of the feature map of the main module to output a context feature map. The feature maps output by the two branches are concatenated in the channel dimension and then output as a fused feature map through a feature fusion layer. S13. Input the fused feature map into the detection head network. The detection head network outputs the target classification confidence, target position regression offset, and target scale level attribution probability. Train the model using a loss function consisting of a weighted sum of classification loss, regression loss, and scale level perception loss. The scale level perception loss is calculated using the scale level label marked in S11 as the supervision signal to calculate the cross-entropy loss. During backpropagation, update both the detection head network and the dual-branch parallel feature extraction network. Iterate until the loss function converges to obtain the target detection and tracking model.

3. The anti-collision tracking method for quay cranes based on machine vision and PLC linkage according to claim 2, characterized in that, Step S3 further includes: predefining the target detection and tracking model in the data processing center, and performing inference and recognition based on real-time lifting device status data, specifically including the following steps: The intrinsic and extrinsic parameters of each fixed camera are calibrated separately to obtain the intrinsic parameter matrix, distortion parameters, rotation matrix and translation vector of each camera relative to the world coordinate system of the quay crane. Based on the calibration results, the perspective transformation relationship between the image coordinate systems of different cameras is established and the perspective transformation relationship is encapsulated into a coordinate mapping lookup table. The trained target detection and tracking model and coordinate mapping lookup table are deployed to the data processing center. The data processing center receives video streams from each fixed camera in real time, performs target detection and tracking on each frame of the image, and outputs the category, position coordinates, unique identifier and motion trajectory of each target.

4. The anti-collision tracking method for quay cranes based on machine vision and PLC linkage according to claim 1, characterized in that, In S4, the image coordinates of the target identified by the second fixed camera are mapped to the image coordinate system of the first fixed camera through a pre-defined spatial transformation relationship to generate a virtual warning box. This process specifically includes the following steps: S41. Synchronize the target detection results output by the second fixed camera with the video frames acquired by the first fixed camera in time, and map the target detection box coordinates in the image coordinate system of the second fixed camera to the image coordinate system of the first fixed camera to obtain the preliminary mapped coordinates; S42. Perform overlap matching between the preliminary mapped coordinates and the coordinates of the identified targets in the first fixed camera. Based on the matching results, establish a connection between the blind zone targets and the identified targets or assign temporary identifiers to the blind zone targets. S43. Configure the visual attributes of the virtual warning box according to the risk level, posture and occlusion status of the associated target, and overlay the configured virtual warning box on the mapped coordinate position in the first fixed camera screen.

5. The anti-collision tracking method for quay cranes based on machine vision and PLC linkage according to claim 1, characterized in that, In S3, the data processing center runs a target detection and tracking model to identify targets in the video stream in real time and generate unique identifiers and motion trajectories for each target. Based on the spreader status data, it dynamically calculates the real-time projected danger zone of the spreader, specifically including the following steps: S31. Input the video streams collected by the first fixed camera and the second fixed camera into the target detection and tracking model respectively, extract the target position coordinates, target category and current frame timestamp of each frame image, assign a unique tracking identifier to each target, and receive the real-time speed data of the spreader trolley and the trolley from the PLC control system of the quay crane, use the speed of the spreader trolley and the speed of the trolley as external motion field constraints to predict the direction of the target motion state, so that the predicted offset of the target in the direction of spreader movement is positively correlated with the actual speed of spreader movement; S32. Align the target trajectories obtained by the first fixed camera and the second fixed camera according to the timestamp, and calculate the position of the target trajectory with the same tracking identifier in the respective image coordinate system of the two cameras' fields of view; S33. Obtain spreader data in real time from the quay crane PLC control system, establish the three-dimensional coordinates of the four corner points of the spreader bottom surface in the world coordinate system according to the geometric dimension parameters of the spreader, project the four corner points onto the ground plane to obtain four ground projection points, and connect the four ground projection points to form the real-time projection danger zone polygon of the spreader. S34. Determine the spatial inclusion relationship between the real-time projected dangerous area polygon and each target location, and based on the current descent speed data of the lifting equipment, analyze the time required for the lifting equipment to descend from the current height to the projection contact ground at the current descent speed as the warning time margin, and trigger the corresponding process based on the warning time margin status.

6. The anti-collision tracking method for quay cranes based on machine vision and PLC linkage according to claim 1, characterized in that, S32 also includes the following steps: When a target of a certain tracking identifier is continuously lost in the field of view of the first fixed camera for more than a preset frame number threshold, the trajectory data of the target in the last five frames before the loss is stored in the occlusion temporary storage area. The target in the occlusion temporary storage area will no longer participate in the matching calculation of the current frame in subsequent frames. When the target reappears in the field of view of the second fixed camera, the last stored trajectory data in the occlusion temporary storage area is extracted and matched with the reappearing detection box for feature vector similarity. If the similarity exceeds a preset threshold, the original tracking label is reassigned to the target, and the virtual warning box display of the target in the first fixed camera's view is restored.

7. The anti-collision tracking method for quay cranes based on machine vision and PLC linkage according to claim 1, characterized in that, Step S6 specifically includes the data processing center sending the control command sequence generated in S5 to the PTZ camera, driving the PTZ camera to rotate continuously and automatically adjust the focal length so that the tracked target is always kept in the center of the screen and occupies the preset screen ratio range. The tracked target is the current operation associated target determined in S5 based on the target's motion trajectory and the lifting device action control signal.

8. The anti-collision tracking method for quay cranes based on machine vision and PLC linkage according to claim 1, characterized in that, Step S7 further integrates the monitoring images of the first fixed camera with the virtual warning frame superimposed, the real-time tracking images of the PTZ camera, and the positional relationship information of the target and the real-time projected danger zone determined in S3, based on the data processing center, and displays it on the driver's cab monitor.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the anti-collision tracking method for quay cranes based on machine vision and PLC linkage as described in any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the anti-collision tracking method for quay cranes based on machine vision and PLC linkage as described in any one of claims 1 to 8.

Images