A method and system for preventing ramming of a quay crane shore side operation spreader

Abstract

This invention provides a method and system for preventing falling objects from under quay crane lifting equipment, relating to the fields of vision and deep learning technologies. It aims to accurately identify and locate the position of an obstructed object before it falls or is obstructed. The method includes: detecting video frames using a YOLOv5x visual detection network to obtain detection results; performing tracking prediction based on the detection results using the DeepSort algorithm and a re-identification algorithm to obtain tracking prediction results; labeling the tracking prediction results based on the similarity between the detection results and the tracking prediction results; projecting the tracking prediction results onto the plane of the image captured by a trolley-mounted camera to obtain the projected tracking prediction results from the trolley-mounted camera's perspective; calculating the ground projection area of ​​the lifting equipment from the trolley-mounted camera's perspective based on the lifting equipment size, determining whether there is a risk of falling objects, and sending an alarm message to the quay crane. This system is applied to the method of preventing falling objects from under quay crane lifting equipment.

Description

Technical Field

[0001] This invention is applicable to the port industry and relates to the fields of machine vision and deep learning technology, and in particular to a method and system for preventing falling objects from under the quay crane's side-operation spreader. Background Technology

[0002] A quay crane, or quayside container crane, is a specialized piece of equipment at a container terminal used for loading and unloading container ships. When a quay crane is lowering a container onto the inner truck, the operator controls the spreader from the overhead cab. During this operation, the following situations can cause objects to appear near the lowering container: ① Port workers are standing next to the inner truck, preparing to remove latches from the container; ② The inner truck cab is in front of the inner truck cab, significantly higher than the cab, and the two are very close. However, because the quay crane operator is positioned high and the spreader directly below obstructs their view, the operator may have difficulty seeing these personnel and the truck cab, potentially leading to a collision between the container and these objects, resulting in a serious safety accident.

[0003] In recent years, with the rapid development of various sensor technologies and machine vision analysis technologies, these technologies have been widely applied in the field of intelligent control. In the port industry, relevant technical departments have already applied these technologies to try to solve the problem of quay cranes hitting trucks during container unloading operations on the quay side, and have designed related internal truck guidance systems. Existing internal truck guidance systems mainly fall into the following categories: ① Laser-based internal truck alignment, which uses a laser mounted on the spreader to illuminate the quay side, thereby obtaining the precise positioning coordinates and attitude of the container. Then, alignment is performed based on the relative position of the spreader and the container to avoid collisions between the truck and the container; ② Machine vision-based truck alignment, which uses a foreground / background detection method to first detect and locate the truck's position, and then accurately guides the truck to the designated position based on the relative position of the spreader and the truck, ensuring that the container does not collide with the truck's cab during unloading. However, the existing methods have the following problems: (1) Laser-based truck alignment technology is difficult to accurately identify port workers, other types of vehicles and other objects working under the quay crane spreader; (2) Machine vision detection algorithm-based truck alignment technology is difficult to identify objects under the spreader when it is blocked; (3) Foreground and background detection algorithms rely on background modeling and are greatly affected by the weather. In rainy or snowy weather, it is difficult to accurately distinguish between the foreground and background. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention provides a method and system for preventing falling objects from under the lifting equipment used in shore-side operations of quay cranes. When the lifting equipment falls or even blocks the area below, it can accurately identify and locate the position of the object being blocked before it was blocked.

[0005] This invention provides a method for preventing falling objects from under lifting equipment used in shore-side operations of quay cranes. The method includes:

[0006] Step 1: Install a ground-facing camera at the bottom of the driver's cab and the bottom of the trolley frame on both sides of the quay crane spreader to obtain video data of the area below the quay crane spreader;

[0007] Step 2: Decode the video data of the area below the quay crane spreader into video frames, and perform detection on the video frames based on the Yolov5x visual inspection network to obtain the detection results;

[0008] Step 3: Based on the DeepSort algorithm and the re-identification algorithm, perform tracking prediction on the detection results to obtain tracking prediction results. Based on the similarity between the detection results and the tracking prediction results, label the tracking prediction results.

[0009] Step 4: Project the tracking prediction result onto the plane where the small vehicle frame camera is shooting, and obtain the tracking prediction result projected from the perspective of the small vehicle frame camera;

[0010] Step 5: Calculate the ground projection area of ​​the spreader as seen from the camera on the trolley frame based on the spreader size. If the center point of the tracking prediction result of the projection falls on the ground projection area of ​​the spreader, it is determined that there is a risk of falling and an alarm message is sent to the quay crane.

[0011] Preferably, step 1 includes: installing a ground-illuminating camera at the bottom rear of the driver's cab and the bottom front of the trolley frame on both sides of the quay crane spreader to acquire video data of the area below the quay crane spreader. The objects in the video data of the area below the quay crane spreader include: pedestrians, truck cabs, quay crane spreaders, and other vehicles.

[0012] Preferably, step 3 includes:

[0013] Step 3.1: Based on Kalman filtering, track and predict the motion trajectory of the detected object in the detection result to obtain the tracking prediction result. Calculate the first similarity between the detection result and the tracking prediction result based on Mahalanobis distance and cosine distance, and judge the first similarity.

[0014] Step 3.2: If the first similarity is greater than the first similarity threshold, then the detected object in the detection result and the tracked object in the tracking prediction result are matched as the same object, and the tracking prediction result is marked as matched and tracked.

[0015] Step 3.2: If the first similarity is less than or equal to the first similarity threshold, then obtain the unmatched detected objects in the detection results, obtain the unmatched tracked objects in the tracking prediction results, calculate the second similarity between the feature vectors of the unmatched detected objects and the feature vectors of the unmatched tracked objects based on cosine distance, and judge the second similarity.

[0016] If the second similarity is greater than the second similarity threshold, the unmatched detected object and the unmatched tracked object are matched as the same object, and the tracking prediction result is marked as matched tracking; if the second similarity is less than or equal to the second similarity threshold, the unmatched detected object is used as a candidate new tracked object, and the tracking prediction result is marked as new tracking.

[0017] Step 3.3: If the first similarity is less than or equal to the first similarity threshold, then obtain the unmatched tracking objects in the tracking prediction results, and determine whether the unmatched tracking objects are near the quay crane spreader. If so, the unmatched tracking objects are occluded tracking objects, and the tracking prediction results are marked as occluded tracking. Otherwise, the unmatched tracking objects are candidate lost objects, and the tracking prediction results are marked as lost tracking.

[0018] Preferably, the Kalman filter formula is:

[0019] x′=Fx(1)

[0020] P′=FPF T +Q(2)

[0021] Where x represents the target information in the previous frame, x = [u, v, gama, h, x1, y1, gama1, h1], (u, v) are the center coordinates of the target anchor box, gama is the aspect ratio of the target anchor box, h is the height of the target anchor box, x1, y1, gama1, h1 are the motion information of the target in the image coordinates, P is the target estimation error, F is the state transition matrix, Q is the system error, x′ is the object state information of the next frame predicted using the Kalman model, and P′ is the estimation error predicted using the Kalman model.

[0022] y=z-Hx′(3)

[0023] S=HP′H T +R(4)

[0024] K=P′H T S -1 (5)

[0025] x=x′+Ky(6)

[0026] P=(1-KH)P′ (7)

[0027] Where z is the detected mean vector, z = [u, v, gama, h], H is called the measurement matrix, R is the detector noise matrix, K is the Kalman gain, and x and P are the updated mean vector and covariance matrix, respectively.

[0028] Preferably, the Mahalanobis distance formula is:

[0029]

[0030] Where, d j Indicates the position of the j-th detection box; y i S represents the predicted position of the target by the i-th tracker; i This represents the covariance matrix between the detection box and the tracking box;

[0031] The formula for the cosine distance is:

[0032] Where, d j Let y represent the feature vector of the j-th detected target (obtained by the preceding CNN network). i This represents the target feature vector stored in the i-th tracker.

[0033] Preferably, step 4 includes: projecting the detection result and the tracking prediction result onto the plane of the image captured by the trolley camera using an affine transformation, to obtain the tracking prediction result projected from the trolley camera's viewpoint, wherein the affine transformation matrix is:

[0034]

[0035] Where x and y represent the original coordinates to be transformed, and x′ and y′ represent the transformed results. Let be the affine transformation matrix.

[0036] Preferably, step 4 includes: determining whether the projection of the past tracking prediction result coincides with the vehicle frame detection result based on the IoU algorithm.

[0037] If the IoU result exceeds the preset threshold, the projected past tracking prediction result overlaps with the small vehicle frame detection result, and the small vehicle frame detection result is retained;

[0038] If the object projected from the rear of the driver's cab onto the trolley frame does not coincide with any object in the trolley frame detection results, it is sent back as a new object.

[0039] If the object in the trolley frame detection result does not coincide with any object projected onto the trolley frame from behind the driver's cab, it is sent backward as a new object.

[0040] Compared with existing technologies, the present invention provides a method for preventing falling objects from under quay crane spreaders, which has the following advantages: A ground-illuminating camera is installed at the bottom of the driver's cab and the bottom of the trolley frame on both sides of the quay crane spreader to acquire video data of the area below the spreader; the video data of the area below the spreader is decoded into video frames, and the video frames are detected using a Yolov5x visual detection network to obtain detection results; based on the DeepSort algorithm and re-identification algorithm, the detection results are tracked and predicted to obtain tracking prediction results; the tracking prediction results are labeled based on the similarity between the detection results and the tracking prediction results; the tracking prediction results are projected onto the plane where the trolley frame camera captures the image, obtaining the projected tracking prediction results from the trolley frame camera's perspective; the ground projection area of ​​the spreader appearing in the trolley frame camera's perspective is calculated based on the spreader size; if the center point of the projected tracking prediction results falls within the ground projection area of ​​the spreader, it is determined that there is a risk of falling objects, and an alarm message is sent to the quay crane. By using a re-identification algorithm to track and predict the detection results, it can accurately determine whether a newly appearing person was previously obscured by the hoist or is a newly appearing person when the hoist falls or even when it is blocked from view.

[0041] The present invention also provides a system for preventing falling objects from under lifting equipment used in shore-side operations of quay cranes, the system comprising:

[0042] The video data acquisition module is used to install a ground-facing camera at the bottom of the driver's cab and the bottom of the trolley frame on both sides of the quay crane spreader to acquire video data of the area below the quay crane spreader.

[0043] The detection result acquisition module is used to decode the video data of the area below the quay crane spreader into video frames, detect the video frames based on the Yolov5x visual detection network, and obtain the detection results.

[0044] The tracking prediction and labeling module is used to perform tracking prediction on the detection results based on the DeepSort algorithm and the re-identification algorithm, obtain tracking prediction results, and label the tracking prediction results based on the similarity between the detection results and the tracking prediction results;

[0045] The projection module is used to project the detection results and the tracking prediction results onto the plane where the small vehicle frame camera is shooting, so as to obtain the tracking prediction results projected from the perspective of the small vehicle frame camera.

[0046] The anti-collision judgment module is used to calculate the ground projection area of ​​the spreader in the view of the trolley frame camera based on the size of the spreader. If the center point of the tracking prediction result of the projection falls on the ground projection area of ​​the spreader, it is judged that there is a risk of collision and an alarm message is sent to the quay crane.

[0047] Compared with the prior art, the beneficial effects of the anti-collision system for quay crane side-operation lifting equipment provided by the present invention are the same as the beneficial effects of the anti-collision method for quay crane side-operation lifting equipment described in the above technical solution, and will not be repeated here.

[0048] The present invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are connected via the bus. When the computer program is executed by the processor, it implements the steps of the above-described method for preventing falling objects from a quay crane side-operation lifting device.

[0049] Compared with the prior art, the beneficial effects of the electronic device provided by the present invention are the same as those of the method for preventing falling objects from a quay crane side-operation lifting device described in the above technical solution, and will not be repeated here.

[0050] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method for preventing falling objects from a quay crane side-operation lifting device.

[0051] Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as those of the anti-collision method for the lifting equipment under the shore-side operation of the quay bridge described in the above technical solution, and will not be repeated here.

[0052] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart of a method for preventing falling debris from lifting equipment during shore-side operations on a quay bridge, provided by an embodiment of the present invention, is shown.

[0055] Figure 2 This diagram illustrates the installation of cameras behind the driver's cab and in front of the trolley frame according to an embodiment of the present invention.

[0056] Figure 3 The diagram illustrates the principle of a method for preventing falling objects from a lifting device used in shore-side operations of a quay bridge, as provided in an embodiment of the present invention.

[0057] Figure 4 A flowchart of the tracking and prediction process based on the re-identification algorithm provided in an embodiment of the present invention is shown;

[0058] Figures 5-11 This diagram illustrates the tracking and prediction based on the re-identification algorithm provided in an embodiment of the present invention. Detailed Implementation

[0059] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0060] In this embodiment, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations, intended to present related concepts in a specific manner, and should not be construed as superior or more advantageous than other embodiments or designs.

[0061] Figure 1 The flowchart illustrates a method for preventing falling debris from lifting equipment during shore-side operations on a quay bridge, as provided in an embodiment of the present invention. Figure 3 This diagram illustrates the principle of a method for preventing falling debris from lifting equipment used in shore-side operations of a quay crane, as provided in an embodiment of the present invention. Figure 1 and Figure 3 The present invention provides a method for preventing falling objects from under lifting equipment used in shore-side operations of quay cranes, the method comprising:

[0062] Step S1: Install a ground-facing camera at the bottom of the driver's cab and the bottom of the trolley frame on both sides of the quay crane spreader to obtain video data of the area below the quay crane spreader.

[0063] It should be noted that the anti-collision method for quay crane side-operation spreaders provided in this embodiment of the invention is mainly applied to port quay crane operations. A ground-illuminating camera is installed at the rear bottom of the driver's cab and the front bottom of the trolley frame on both sides of the quay crane spreader to acquire video data of the area below the quay crane spreader. Objects in the video data of the area below the quay crane spreader include: pedestrians, truck cabs, the quay crane spreader, other vehicles, etc.

[0064] Specifically, such as Figure 2As shown, two 2-megapixel network cameras can be used to illuminate the ground from the quay crane operator's cab and the trolley frame. Mounting the two cameras at the rear bottom of the operator's cab and the front bottom of the trolley frame respectively offers two advantages. First, it allows the cameras to move with the trolley, and since the spreader is also mounted on the trolley frame, the two cameras are positioned horizontally relative to the spreader, facilitating the determination of the relative relationship between objects and the spreader. Second, with one camera in front of the spreader and the other behind, and the optical axes of the two cameras ultimately intersecting at the center of the spreader's projection onto the ground, it ensures that the area below the spreader is detected with as few blind spots as possible during its descent.

[0065] Step S2: Decode the video data of the area below the quay crane spreader into video frames, and perform detection on the video frames based on the Yolov5x visual inspection network to obtain the detection results.

[0066] It should be noted that after decoding the video data of the area under the quay crane captured by the two network cameras into video frames, the video frames are fed into a computer vision detection network to detect the position and type of objects in the video frames. Given the presence of multiple objects in the scene and the significant background variations, a Yolov5x network was chosen.

[0067] The reasons for using Yolov5x as the detection network are as follows:

[0068] (1) As shown in Table 1, compared with the traditional Darknet-53 base network of Yolov3, the base network of Yolov5x uses the CSP architecture, which can reduce the amount of computation while slightly improving the accuracy.

[0069] Table 1: Performance comparison between existing methods and Yolov5x

[0070] Method Backbone Size FPS #Parameter AP50 AP75 APS YoloV3 Darknet53 608 30 62.3M 57.9 34.4 18.3 YoloV3(SPP) Darknet53 608 30 62.9M 60.6 38.2 20.6 PANet(SSP) CSPResNeXt50 608 35 56.9M 60.6 41.6 22.1

[0071] (2) Compared to the 512 channels of YOLOv5s, YOLOv5 has 1280 channels. More channels mean more complex semantic expression capabilities, that is, more accurate detection capabilities can be obtained in shallow feature maps of complex environments, thus improving the detection accuracy.

[0072] Meanwhile, considering that the scale of port staff on the shore is 50×50 (unit pixels) in the camera and the viewing angle is from top to bottom, making it difficult to judge the shape of a normal human body, the input size was determined to be 1280×732 pixels.

[0073] Specifically, the selection of the input size is based on:

[0074] a. Determine the video resolution.origin ×w origin If the resolution is rectangular, then w origin Given the longer side of the video's width and height, the minimum size w of the object to be detected. min ×w min Pixel.

[0075] b. Since the detection coordinates of Yolov5 are determined by a pyramid-shaped feature map, the principle of which will be explained in detail in step three. According to scale invariance, if an object can be detected, it needs to have significant features at the corresponding position in the feature map at the corresponding scale. Therefore, the size of the object on the corresponding feature map needs to be at least 1×1 pixels.

[0076] c. To reliably detect tiny objects lacking human features, it can be stipulated that the object must be at least 1×1 pixels on the feature map with a stride of 32. If the input size of the model is w... input ×w input The conditions that the input dimensions must meet are shown in the following figure:

[0077]

[0078] After derivation, we can obtain

[0079]

[0080] d. Since typical video frame images are not square, if the detection model is set to a square input size w... input There will be areas at the top and bottom that cannot be effectively utilized, which leads to wasted computation. Therefore, in this embodiment of the invention, the input size ratio is made as close as possible to the ratio of the video frame image.

[0081] Here, since the input raw video frame size w has already been determined... origin The image size is 1920×1080 pixels, and the dimensions of the shore-side workers in the image are w. min Given a resolution of 50×50 pixels and the requirement that the input size be a multiple of 32, the final model's input size w can be determined. input It is 1280×732 pixels.

[0082] This detection network can identify objects such as truck cabs, people, crane spreaders, and other vehicles. Its average detection accuracy can reach 95%.

[0083] Step S3: Based on the DeepSort algorithm and the re-identification algorithm, perform tracking prediction on the detection results to obtain tracking prediction results, and label the tracking prediction results based on the similarity between the detection results and the tracking prediction results.

[0084] It should be noted that step S3 includes:

[0085] Step 3.1: Based on Kalman filtering, track and predict the motion trajectory of the detected object in the detection result to obtain the tracking prediction result. Calculate the first similarity between the detection result and the tracking prediction result according to Mahalanobis distance and cosine distance, and judge the first similarity.

[0086] Step 3.2: If the first similarity is less than or equal to the first similarity threshold, then obtain the unmatched detected objects in the detection results, obtain the unmatched tracked objects in the tracking prediction results, calculate the second similarity between the feature vectors of the unmatched detected objects and the feature vectors of the unmatched tracked objects based on cosine distance, and judge the second similarity.

[0087] If the second similarity is greater than the second similarity threshold, the unmatched detected object and the unmatched tracked object are matched as the same object, and the tracking prediction result is marked as matched tracking; if the second similarity is less than or equal to the second similarity threshold, the unmatched detected object is used as a candidate new tracked object, and the tracking prediction result is marked as new tracking.

[0088] Step 3.3: If the first similarity is less than or equal to the first similarity threshold, then obtain the unmatched tracking objects in the tracking prediction results, and determine whether the unmatched tracking objects are near the quay crane spreader. If so, the unmatched tracking objects are occluded tracking objects, and the tracking prediction results are marked as occluded tracking. Otherwise, the unmatched tracking objects are candidate lost objects, and the tracking prediction results are marked as lost tracking.

[0089] First, let's explain the DeepSORT algorithm. DeepSORT is an improved version of the SORT multi-object tracking algorithm. It designs a new association method to improve the accuracy of tracking objects that are occluded for a long time. Compared to the SORT algorithm, DeepSORT achieves association measurement by combining more accurate metrics of action and appearance information, uses a CNN network to extract features, and increases robustness to tracking missing and occluded objects.

[0090] Specifically, the DeepSORT algorithm flow is as follows:

[0091] (1) For the input detection results, the image region corresponding to each detection coordinate is first extracted, and the CNN network is used to extract the image features of the extracted image as the feature parameters of the detected object in the detection results.

[0092] (2) For the original tracking trajectory, Kalman filtering is used to predict the original tracking trajectory. Kalman filtering is an algorithm that uses the state equation of a linear system to make the optimal estimate of the system state through the system input and output observation data. DeepSORT uses eight parameters u, v, gama, h, x1, y1, gama1, h1 to describe the motion state of the target, where (u, v) are the center coordinates of the target anchor box, gama is the aspect ratio of the target anchor box, h represents the height of the target anchor box, and the other four variables are the motion information of the target in the image coordinates.

[0093] The Kalman filter operation steps are as follows:

[0094] x′=Fx(1)

[0095] P′=FPF T +Q(2)

[0096] Where x is the target information in the previous frame, x = [u,v,gama,h,x1,y1,gama1,h1], P is the target estimation error, F is the state transition matrix, Q is the system error, x′ is the object state information predicted using the Kalman model in the next frame, and P′ is the estimation error predicted using the Kalman model.

[0097] y=z-Hx′(3)

[0098] A=HP′H T +R(4)

[0099] K=P′H T S -1 (5)

[0100] x=x′+Ky(6)

[0101] P=(1-KH)P′(7)

[0102] In Formula 3, z is the detection mean vector, which does not include velocity change values, i.e., z = [u, v, gama, h]. H is called the measurement matrix, which maps the tracking mean vector x′ to the detection space. Formula 3 is used to calculate the tracking mean error.

[0103] In Formula 4, R is the noise matrix of the detector. It is a 4×4 diagonal matrix. The values ​​on the diagonal are the two coordinates of the center point and the noise of the width and height, respectively. It is initialized with arbitrary values. Generally, the noise of the width and height is set to be greater than the noise of the center point. This formula first maps the covariance matrix P′ to the detection space, and then adds the noise matrix R.

[0104] Formula 5 calculates the Kalman gain K, which is used to estimate the significance of the error.

[0105] Formulas 6 and 7 yield the updated mean vector x and covariance matrix P. The first four terms of x, u, v, gama, and h, represent the object tracking results predicted by the Kalman filter.

[0106] Based on Kalman filtering, the motion trajectory of the detected object in the detection result is tracked and predicted to obtain the tracking prediction result. After obtaining the detection result and the tracking prediction result, the first similarity between the detection result and the tracking prediction result is calculated according to Mahalanobis distance and cosine distance, that is, the first similarity between each detected object and each tracked object, and the first similarity is judged.

[0107] Specifically, Mahalanobis distance is the Euclidean distance in a normalized principal component space. A normalized principal component space is created by performing principal component analysis on some data, decomposing it into principal components, and then normalizing all the principal component decomposition axes to form new coordinate axes. The space spanned by these coordinate axes is the normalized principal component space.

[0108] In DeepSORT, the Mahalanobis distance is expressed as:

[0109]

[0110] Where, d j Indicates the position of the j-th detection box; y i S represents the predicted position of the target by the i-th tracker; i This represents the covariance matrix between the detection box and the tracking box. Two vectors d j y i To find the difference, due to the correlation and order of magnitude, we need to use the covariance matrix S. i The data is normalized to obtain the Mahalanobis distance between the i-th track and the j-th detection. If the Mahalanobis distance of a given association is less than a specified threshold, the two objects are associated as the same object.

[0111] Cosine distance: This is a distance used to represent the directional relationship between two vectors. The larger the cosine distance, the more inconsistent the directions of the two vectors are.

[0112] In DeepSORT, the expression for cosine distance is:

[0113]

[0114] Where, d j The feature vector of the j-th detected target is obtained by the preceding CNN network, y i This represents the target feature vector stored in the i-th tracker. Calculate the cosine similarity between two vectors, then subtract that value from 1 to obtain the distance. If the distance is less than a specified threshold, the two objects are considered to be the same object.

[0115] (4) If the similarity between the detected object and the tracked object is higher than a certain threshold, i.e. the distance is lower than a certain threshold, the two objects are associated as the same object, and the coordinates of the tracker and the object feature information are updated.

[0116] (5) After the matching tracking results are associated, the objects that were not matched in the detection are taken out as new tracking objects. If the new tracking object can be matched for N consecutive frames, it is considered a stable new tracking object.

[0117] (6) While performing step (5), remove objects that do not match the detection during tracking and record them as candidate lost objects. If the object is still marked as lost for N consecutive frames, it is considered a lost object and its trajectory is eliminated.

[0118] The DeepSORT algorithm can stably track multiple targets, but when an object is occluded by a large object for a long time and then reappears in a different location, the algorithm may lose track of that object, resulting in a different ID before and after occlusion. In practical applications, for example, a port worker might be obscured when entering from the left side of a quay crane and then reappearing from the right side after a period of time. In this case, the worker would be identified as two different people.

[0119] To address the aforementioned problems, this invention introduces a re-identification technique. Re-identification is a sub-problem of image retrieval, requiring the retrieval of images of the same object from other cameras, given an image of an object. Here, the previously occluded object serves as the given input; its identity needs to be identified and retrieved when it reappears in other locations.

[0120] Therefore, such as Figure 4 As shown, the following modifications need to be made to DeepSORT:

[0121] (1) During the initial DeepSORT network step, for objects that do not match the detection during tracking, it is determined whether the previous matching tracking result of the object is located near the lifting device. The lifting device position information is obtained through the detection network in the previous step. If the previous matching tracking result of the object is located near the lifting device, it is marked as an occluded object, distinguishing it from other lost objects.

[0122] (2) When tracking is performed in the next frame, Kalman filtering is not performed on objects marked as occluded.

[0123] (3) After successful tracking and matching in the next frame, objects that were not matched in the detection are identified. These unmatched detected targets are then matched with the objects marked as occluded in the previous frame using cosine distance. If the cosine distance of the feature matching is below a certain threshold, the pair of objects is considered to be the same object, and the rematch is successful. Mahalanobis distance is not used for occluded object matching because its calculation is related to the position between objects; the greater the distance, the larger the Mahalanobis distance. However, when an occluded object appears, it is usually far from its disappearance position, resulting in a large Mahalanobis distance, which is difficult to use as a discrimination criterion. Therefore, re-identification matching can only use cosine distance as the discrimination basis. Therefore, further improvements to the DeepSORT algorithm are needed.

[0124] like Figure 4 As shown, the improved DeepSORT algorithm flow is as follows:

[0125] (1) Similar to the original DeepSORT, extract image features from the input detection results and use them as feature parameters of the detected object.

[0126] (2) For unobstructed tracked objects, Kalman filtering is used to predict their motion trajectory.

[0127] (3) Calculate the cosine distance and Mahalanobis distance between the unoccluded tracked object and the detected object.

[0128] (4) If the similarity between the detected object and the unoccluded tracked object is higher than a certain threshold, i.e. the distance is lower than a certain threshold, the two objects are associated as the same object, and the coordinates of the tracker and the object feature information are updated.

[0129] (5) After the matching of non-occluded tracking results is completed, the objects that were not matched in the detection are first extracted, and the cosine distance between their feature vectors and the feature vectors of the occluded tracked objects is calculated. If the cosine distance is lower than a certain threshold, the two objects are associated as the same object, and the coordinates and object feature information of the tracker are updated at the same time.

[0130] (6) After all the matching in (4) and (5) is completed, the detected objects that have no matching are taken as the candidate new tracking objects. If the candidate new tracking object can be matched for N consecutive frames, it is considered to be a stable new tracking object.

[0131] (7) Extract objects that do not match the detection during tracking and determine their positional relationship with the quay crane spreader. Specifically, determine whether the previous tracking match of the object was located near the spreader. The spreader's position information has been obtained through the detection network in the previous step. If the previous tracking match of the object was located near the spreader, mark it as an occluded tracking object, distinguishing it from other lost objects. Otherwise, record it as a candidate lost object. If a candidate lost object is still marked as lost for N consecutive frames, it is considered a lost object, and its trajectory is eliminated.

[0132] like Figure 5 As shown, the two workers marked with green letters 90 and 79 in the middle of the white detection box appear in the lower left of the field of view. Figure 6 After the device moves below the spreader, it cannot be tracked. Since its disappearance point is near the spreader, it is judged as disappearing due to obstruction, and the coordinates of its disappearance are recorded. Figure 7 As shown, after disappearing for a period of time, the two people reappeared under the hoisting equipment and were then re-identified as the previously hidden and disappeared individuals.

[0133] Step 4: Project the detection results and the tracking prediction results onto the plane where the small vehicle frame camera is located, and obtain the tracking prediction results projected from the perspective of the small vehicle frame camera.

[0134] It should be noted that the detection and tracking results of the two cameras are projected onto a single plane. In practice, the detection and tracking results from the rear camera in the driver's cab and the front camera on the trolley frame are all projected onto the plane where the image captured by the trolley frame camera is located.

[0135] The algorithm that projects the detection results from the rear camera in the driver's cab onto the vehicle frame is an affine transformation. An affine transformation, in geometry, refers to a linear transformation of a vector space followed by a translation, transforming it into another vector space. In two-dimensional images, an affine transformation is a linear transformation that translates two-dimensional coordinates (x, y) in one plane to two-dimensional coordinates (u, v) in another plane; it is also called image warping.

[0136] It should be understood that the reason for using affine transformation is that during quay crane operations, if the object on the quay side is stationary, its three-dimensional positional relationship relative to the trolley frame and the camera behind the driver's cab remains constant; when the object moves, only the horizontal positional relationship changes, but its height relationship with the camera remains unchanged. Therefore, this embodiment of the invention simplifies the projection between cameras into a linear transformation problem in the horizontal direction, i.e., affine transformation.

[0137] The matrix representation of an affine transformation is as follows:

[0138]

[0139] In this diagram, x and y on the left correspond to the original coordinates to be transformed, while x′ and y′ correspond to the transformed results. The middle 3×3 matrix... This is the affine transformation matrix, which is calculated using the OpenCV function `getAffineTransform`. The expression for `getAffineTransform` in OpenCV is:

[0140] Mat cv::getAffineTransform(const Point2f src[],const Point2fdst[])

[0141] Where src[] represents a series of points selected from the image to be transformed; dst[] represents the position of the point corresponding to src[] on the new plane. In practical applications of quay cranes, the method for obtaining these two sets of points is as follows:

[0142] (1) Capture video frames from the camera behind the driver's cab. Select M points (M>3) from the lane markings in the video frames that are not on a straight line, and use them as the src[] of the getAffineTransform function.

[0143] (2) Capture video frames with the same timestamp from the trolley camera. Find the coordinate point of src[] in (1) corresponding to the trolley camera from the lane marking line in the video frame, and mark it as dst[].

[0144] (3) Use the above function getAffineTransform to perform the calculation. The return value of this function is the affine transformation matrix. Then use the matrix to calculate the position of the object detected behind the driver's cab projected onto the small frame camera.

[0145] After projection, the IoU algorithm is used to determine whether the projected tracking prediction result matches the detection result of the vehicle frame, and the following three cases are considered:

[0146] (1) If the IoU result exceeds a certain threshold, the two objects are considered to overlap, that is, they are judged to be the same object, and the detection result of the small vehicle frame is retained.

[0147] (2) If the object projected from the driver’s cab onto the trolley frame does not coincide with any object inside the trolley frame, it is sent backward as a new object.

[0148] (3) If the object on the trolley frame does not coincide with any object projected behind the driver’s cab, it is sent backward as a new object.

[0149] Once integrated, the results are sent as the final output, which is the detection result from the perspective of the small vehicle frame. Figure 8The final fusion result, showing the combination of the trolley frame and driver's cab, is presented in the view of the trolley frame. Figure 9 The simultaneous view behind the driver's cab is shown. Understandably, since the two figures in the image were added artificially for tracking performance testing, their effects from the two perspectives are difficult to accurately superimpose. Therefore, the two figures behind the driver's cab are directly mapped to the green markings "25" and "4" in the center of the white detection frame on the trolley frame. The mapping result shows that the projected coordinates are accurate. Simultaneously, the truck cab, which originally appeared in the image, is accurately merged into a single cab from both perspectives.

[0150] Step 5: Calculate the ground projection area of ​​the spreader as seen from the camera on the trolley frame based on the spreader size. If the center point of the tracking prediction result of the projection falls on the ground projection area of ​​the spreader, it is determined that there is a risk of falling and an alarm message is sent to the quay crane.

[0151] It should be noted that, based on the PLC information, the size of the spreader used for the container unloading operation is determined: 20 feet, 40 feet, or 45 feet. The ground projection area of ​​the spreader as seen from the trolley frame camera is calculated based on the spreader size. If a tracking result shows its center point falling within the spreader's ground projection area, it is considered that the container is at risk of colliding with an object, and the algorithm will send an alarm to the quay crane PLC.

[0152] Another possible implementation:

[0153] Step 1: Collect video data from two network cameras at the port quay crane operation site, including pedestrians in different clothing, container trucks in different positions, and quay crane spreaders in different positions and states. Convert the video into images and divide them into test and training datasets. Train a Yolov5 detection network using the trained classifier and apply it to the detection process.

[0154] Step 2: As Figures 10-11 As shown, two network HD cameras are installed at two locations: one behind the quay crane trolley and the other behind the driver's cab. An industrial control computer is installed on the quay crane, and a pre-compiled target detection program is installed on the industrial control computer. After the equipment is installed, the cameras on the trolley and behind the driver's cab are adjusted to illuminate the container from the sea side to the shore side, with the center point of the image positioned in the center of the spreader that is lifting the container and placing it on the ground.

[0155] Step 3: Based on the position of the corresponding road markings in the image, calculate the radial transformation matrix from the driver's cab to the trolley frame.

[0156] Step 4: After completing the monitoring model training and equipment calibration, the program can be run to perform target detection and tracking, and finally all detection and tracking results will be mapped onto the screen of the trolley frame.

[0157] Compared with the prior art, the anti-collision method for the lifting equipment used in shore-side operations of quay cranes provided in this embodiment of the invention has the following beneficial effects:

[0158] 1. When the lifting equipment is falling or even obstructing the view, the system can accurately identify and locate the position of the obstructed object before it was obstructed. This improves the tracking algorithm's ability to catch up with personnel again after they are obstructed by a specific object, which is of great significance for tracking and identifying personnel after they are obstructed by the lifting equipment.

[0159] 2. A coordinate transformation method was designed after the trolley was mounted on the driver's cab. This method can integrate the results from multiple cameras into a single reference frame. If a certain viewpoint cannot be detected, the results from another viewpoint can be used to complement the tracking results, thereby improving the robustness of the tracking results.

[0160] This invention provides an anti-collision system for lifting equipment used on the shore side of a quay crane, the system comprising:

[0161] The video data acquisition module is used to install a ground-facing camera at the bottom of the driver's cab and the bottom of the trolley frame on both sides of the quay crane spreader to acquire video data of the area below the quay crane spreader.

[0162] The detection result acquisition module is used to decode the video data of the area below the quay crane spreader into video frames, detect the video frames based on the Yolov5x visual detection network, and obtain the detection results.

[0163] The tracking prediction and labeling module is used to perform tracking prediction on the detection results based on the DeepSort algorithm and the re-identification algorithm, obtain tracking prediction results, and label the tracking prediction results based on the similarity between the detection results and the tracking prediction results;

[0164] The projection module is used to project the detection results and the tracking prediction results onto the plane where the small vehicle frame camera is shooting, so as to obtain the tracking prediction results projected from the perspective of the small vehicle frame camera.

[0165] The anti-collision judgment module is used to calculate the ground projection area of ​​the spreader in the view of the trolley frame camera based on the size of the spreader. If the center point of the tracking prediction result of the projection falls on the ground projection area of ​​the spreader, it is judged that there is a risk of collision and an alarm message is sent to the quay crane.

[0166] Compared with the prior art, the beneficial effects of the anti-collision system for quay crane side-operation lifting equipment provided in this embodiment of the invention are the same as the beneficial effects of the anti-collision method for quay crane side-operation lifting equipment described in the above technical solution, and will not be repeated here.

[0167] In addition, this invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are connected via the bus. When the computer program is executed by the processor, it implements the various processes of the above-described method for preventing falling objects from a quay crane side-operation lifting device and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0168] Furthermore, this embodiment of the invention also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of the method for preventing falling objects from a quay crane side-operation lifting device, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0169] Computer-readable storage media include: permanent and non-permanent, removable and non-removable media, which are tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media include: electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, and any suitable combination thereof. Computer-readable storage media include: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape storage, magnetic disk storage or other magnetic storage devices, memory sticks, mechanical encoding devices (e.g., punched cards or raised structures in grooves on which instructions are recorded), or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in the embodiments of the present invention, computer-readable storage media do not include temporary signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, electronic devices, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to solve the problems addressed by the embodiments of the present invention, depending on actual needs.

[0172] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (including: a personal computer, a server, a data center, or other network device) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media listed above that can store program code.

[0174] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of preventing a shore-based container crane from being struck by a load, characterized in that, The method comprises the following steps: Step 1: A camera for ground irradiation is arranged at the bottom of the driver's cabin and the bottom of the trolley frame on both sides of the quay crane spreader to obtain quay crane spreader area video data; Step 2: The quay crane spreader area video data is decoded into video frames, the video frames are detected based on a Yolov5x visual detection network, and a detection result is obtained; Step 3: The detection result is tracked and predicted based on a DeepSort algorithm and a re-identification algorithm, a tracking prediction result is obtained, and the tracking prediction result is labeled based on the similarity between the detection result and the tracking prediction result; Step 4: The tracking prediction result is projected to the plane where the trolley frame camera shot is located, and the tracking prediction result projected to the trolley frame camera visual angle is obtained; Step 5: The size of the spreader is calculated to obtain the ground projection area of the spreader in the trolley frame camera visual angle, if the center point of the projected tracking prediction result falls within the ground projection area of the spreader, it is judged that there is a risk of anti-smashing, and an alarm information is sent to the quay crane. The step 3 comprises: Step 3.1: The motion trajectory of the detected object in the detection result is tracked and predicted based on Kalman filtering to obtain a tracking prediction result, the first similarity between the detection result and the tracking prediction result is calculated according to the Mahalanobis distance and the cosine distance, and the first similarity is judged; Step 3.2: If the first similarity is less than or equal to the first similarity threshold, the unmatched detected object in the detection result is obtained, the unmatched tracked object in the tracking prediction result is obtained, the second similarity between the feature vector of the unmatched detected object and the feature vector of the unmatched tracked object is calculated according to the cosine distance, and the second similarity is judged, If the second similarity is greater than the second similarity threshold, the unmatched detected object and the unmatched tracked object are matched as the same object, and the tracking prediction result is labeled as a matched tracking; if the second similarity is less than or equal to the second similarity threshold, the unmatched detected object is taken as a new tracking object, and the tracking prediction result is labeled as a new tracking; Step 3.3: If the first similarity is less than or equal to the first similarity threshold, the unmatched tracked object in the tracking prediction result is obtained, it is judged whether the unmatched tracked object is near the quay crane spreader, if yes, the unmatched tracked object is a blocked tracked object, and the tracking prediction result is labeled as a blocked tracking, otherwise, the unmatched tracked object is a candidate lost object, and the tracking prediction result is labeled as a lost tracking.

2. The quay crane shore side operation spreader anti-smashing method according to claim 1, wherein the step 1 comprises: arranging a camera for ground irradiation at the rear bottom of the driver's cabin and the front bottom of the trolley frame on both sides of the quay crane spreader to obtain quay crane spreader area video data, and the objects in the quay crane spreader area video data comprise pedestrians, truck heads, quay crane spreaders and other vehicles. ​ 3. The method according to claim 2, characterized in that, The Kalman filtering formula is: x' = Fx (1) Wherein, x is the information of the target in the last frame, x = [u, v, gama, h, x1, y1, gama1, h1], (u, v) is the center coordinate of the target anchor box, gama is the aspect ratio of the target anchor box, h is the height of the target anchor box, x1, y1, gama1, h1 are the motion information of the target in the image coordinates, P is the estimation error of the target, F is the state transition matrix, Q is the system error, x' is the next frame object state information predicted using the Kalman model, and P' is the estimation error predicted using the Kalman model. P' = FPF T + Q (2) y = z - Hx' (3) x = x' + Ky (6) S = HP' H T + R (4) K = P'H T S -1 (5) P = (1 - KH)P' (7) Wherein, z is the detected mean vector, z = [u, v, gama, h], H is called the measurement matrix, R is the noise matrix of the detector, K is the Kalman gain, x and P are the updated mean vector and covariance matrix respectively.

4. The method according to claim 3, characterized in that, The Mahalanobis distance formula is:

5. The method according to claim 1, characterized in that, Step 4 includes projecting the tracking prediction result to the plane where the trolley frame camera captures the picture using affine transformation to obtain the tracking prediction result projected in the trolley frame camera view, and the affine transformation matrix is:

6. The method according to claim 5, characterized in that, wherein d j represents the position of the jth detection box; y i represents the predicted position of the target by the ith tracker; S i represents the covariance matrix between the detection box and the tracking box; The cosine distance formula is: where d j represents the feature vector of the jth detection target, y i represents the target feature vector stored by the ith tracker. Step 4 includes judging whether the tracking prediction result projected in the trolley frame overlaps with the trolley frame detection result based on the IoU algorithm, If the IoU result exceeds the preset threshold, the tracking prediction result projected in the trolley frame overlaps with the trolley frame detection result, and the trolley frame detection result is retained; Wherein, x, y represent the original coordinates that need to be transformed, x', y' represent the result after transformation, is an affine transformation matrix. If the object projected from the cab to the trolley frame does not overlap with any object in the trolley frame detection result, it is sent back as a new object; If the object in the trolley frame detection result does not overlap with any object projected from the cab to the trolley frame, it is sent back as a new object. It includes: A video data acquisition module is arranged at the bottom of the cab and the trolley frame on both sides of the quay crane to obtain video data of the area below the quay crane. A detection result acquisition module is arranged to decode the video data of the area below the quay crane into video frames, detect the video frames based on the Yolov5x visual detection network, and obtain the detection result. A tracking prediction marking module is arranged to track and predict the detection result based on the DeepSort algorithm and the re-identification algorithm, obtain the tracking prediction result, and mark the tracking prediction result based on the similarity between the detection result and the tracking prediction result.

7. A shore side landing gear anti-punching system for a shore-based container crane, characterized in that ​ ​ ​ ​ The method comprises: tracking and predicting a motion trajectory of a detected object in the detection result based on Kalman filtering to obtain a tracking prediction result, calculating a first similarity between the detection result and the tracking prediction result according to a Mahalanobis distance and a cosine distance, and judging the first similarity; If the first similarity is less than or equal to the first similarity threshold, an unmatched detected object in the detection result is obtained, an unmatched tracked object in the tracking prediction result is obtained, a second similarity between a feature vector of the unmatched detected object and a feature vector of the unmatched tracked object is calculated according to a cosine distance, and the second similarity is judged, If the second similarity is greater than a second similarity threshold, the unmatched detected object and the unmatched tracked object are matched as the same object, and the tracking prediction result is marked as a matched tracking; if the second similarity is less than or equal to the second similarity threshold, the unmatched detected object is taken as an alternative new tracked object, and the tracking prediction result is marked as a new tracking. If the first similarity is less than or equal to the first similarity threshold, an unmatched tracked object in the tracking prediction result is obtained, it is judged whether the unmatched tracked object is near a spreader of a shore-based container crane, if yes, the unmatched tracked object is a blocked tracked object, and the tracking prediction result is marked as a blocked tracking, otherwise, the unmatched tracked object is an alternative lost object, and the tracking prediction result is marked as a lost tracking. A projection module is configured to project the detection result and the tracking prediction result to a plane where a picture taken by a trolley frame camera is located, and obtain a tracking prediction result projected to the plane under a view of the trolley frame camera. A smashing prevention judging module is configured to calculate a ground projection area of a spreader under the view of the trolley frame camera according to a size of the spreader, and if a center point of the projected tracking prediction result falls in the ground projection area of the spreader, it is judged that there is a smashing risk, and an alarm information is sent to the shore-based container crane.

8. An electronic device comprising a bus, a transceiver, a memory, a processor, and a computer program stored on the memory and executable on the processor, the transceiver, the memory and the processor being connected by the bus, characterized in that, The computer program is executed by the processor to implement the steps in the method for preventing smashing under a spreader of a shore-based container crane on a shore side according to any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps in the method for preventing smashing under a spreader of a shore-based container crane on a shore side according to any one of claims 1-6.

Images