Intelligent control method for gate seat type ship loading and unloading machine for intelligent port

Abstract

The invention relates to the technical field of scheduling of ship loading and unloading machines of intelligent ports, in particular to an intelligent control method of a gate seat type ship loading and unloading machine for the intelligent ports, which comprises the following steps: S1, acquiring a working video of the gate seat type ship loading and unloading machine and a preset working track of a preset grab bucket; s2, decomposing the working video of the gate seat type ship loading and unloading machine into a plurality of ship loading and unloading machine image frames; s3, grab bucket target detection is carried out through the target detection model, and a grab bucket target detection result is output; s4, grab bucket target tracking and trajectory generation are carried out in the target detection model, and a grab bucket target prediction trajectory is output; s5, calculating a corresponding track matching degree based on the grab bucket target prediction track and the grab bucket preset working track; and S6, the grab bucket control strategy of the gate seat type ship loading and unloading machine is adjusted based on the track matching degree. Through the synergistic effect of target detection, trajectory tracking and matching degree calculation, high-efficiency intelligent control is realized while the control accuracy is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent port ship unloader scheduling, and particularly relates to an intelligent control method for a portal ship unloader of an intelligent port. BACKGROUND

[0002] With the continuous growth of global trade and the rapid development of the shipping industry, the construction of intelligent ports has become the core direction of the transformation and upgrading of modern ports. As the key equipment for port loading and unloading operations, portal ship unloaders play an important role in bulk and miscellaneous cargo loading and unloading scenarios due to their large lifting capacity, high operating efficiency, and ability to adapt to complex working conditions.

[0003] The working logic of a portal ship unloader is usually as follows: through the coordinated movement of the lifting, luffing, and rotating mechanisms, the grab bucket is driven to complete the material grabbing, transporting, and unloading operations in three-dimensional space. The entire operation process must strictly follow the preset operation path and safety specifications to ensure operation efficiency and equipment safety.

[0004] In the field of intelligent control of portal ship unloaders, pre-generating a grab bucket preset working trajectory and adjusting the grab bucket control strategy based on the trajectory are the core means to achieve precise operation. The control system plans the complete movement path of the grab bucket from the starting position to the target position in advance according to factors such as cargo characteristics, ship structure, and environmental conditions, forming a preset working trajectory containing parameters such as spatial coordinates, speed, and acceleration. During the operation process, the control system needs to monitor the actual running state of the grab bucket in real time and adjust the driving parameters (such as motor speed and braking timing) of the lifting, luffing, and rotating mechanisms to ensure that the actual trajectory of the grab bucket is highly consistent with the preset trajectory, thereby avoiding safety risks such as collision and overload and improving operation accuracy.

[0005] However, the existing technology faces key challenges in implementing the above control logic: due to the complex operating environment of portal ship unloaders (such as strong winds, dust, and changes in lighting) and the multi-degree-of-freedom coordination involved in grab bucket movement, traditional trajectory monitoring methods based on manual observation or fixed sensor arrays cannot accurately determine whether the actual running trajectory of the grab bucket matches the preset trajectory. Manual observation has visual blind spots and subjective judgment errors, making it difficult to capture small deviations in the grab bucket. Although fixed sensors (such as laser range finders and encoders) can obtain some position data, they are limited by their installation location and cannot completely cover the three-dimensional movement trajectory of the grab bucket, and are easily affected by environmental interference, leading to measurement errors. The above problems directly lead to the blindness of the adjustment of the grab bucket control strategy or the system oscillation caused by excessive correction, or the trajectory deviation caused by insufficient correction, ultimately affecting operation efficiency and safety.

[0006] Therefore, how to accurately match the actual running trajectory of the grab bucket with the preset trajectory in a complex operating environment has become a technical problem that needs to be solved in the field of intelligent control of portal ship unloaders. SUMMARY

[0007] In view of the above problems of the prior art, the technical problem to be solved by the present application is to provide a gantry ship loader intelligent control method for a smart port, which realizes efficient intelligent control while ensuring control accuracy through the synergistic effect of target detection, trajectory tracking and matching degree calculation, thereby providing strong technical support and a new technical direction for the automation and intelligent operation of the gantry ship loader.

[0008] To solve the above technical problem, the present application adopts the following technical solution:

[0009] A gantry ship loader intelligent control method for a smart port, comprising:

[0010] S1: obtaining a working video of the gantry ship loader and a preset work trajectory of a grab bucket;

[0011] S2: decomposing the working video of the gantry ship loader into a plurality of ship loader image frames;

[0012] S3: inputting the ship loader image frames into a trained target detection model for grab bucket target detection, and outputting the grab bucket target detection results in the corresponding ship loader image frames;

[0013] S4: inputting the grab bucket target detection results in the ship loader image frames into the trained target detection model for grab bucket target tracking and trajectory generation, and outputting a grab bucket target prediction trajectory;

[0014] S5: calculating the corresponding trajectory matching degree based on the grab bucket target prediction trajectory and the grab bucket preset work trajectory;

[0015] S6: adjusting the grab bucket control strategy of the gantry ship loader based on the trajectory matching degree.

[0016] Preferably, in step S2, the working video is decoded to obtain original ship loader image data in the working video; the original ship loader image data in the working video is extracted frame by frame in time sequence, at fixed intervals or in key frame mode, to obtain a plurality of ship loader image frames.

[0017] Preferably, in step S3, the processing steps of training the target detection model include:

[0018] S301: obtaining a ship loader image frame as a training sample, and its corresponding real position information and class label;

[0019] S302: performing image enhancement processing on the grab bucket part of the ship loader image frame to obtain an enhanced image frame;

[0020] S303: input the enhanced image frame into the target detection model for grab target detection, and output the grab target detection result in the ship unloader image frame;

[0021] S304: calculate the loss function based on the grab target detection result, the real position information and the category label, and optimize the parameters of the target detection model;

[0022] S305: repeat steps S301 to S304 to iteratively train the target detection model until the model converges.

[0023] Preferably, in step S302, the processing steps for performing the image enhancement processing on the grab part of the ship unloader image frame to obtain the enhanced image frame include:

[0024] S3021: performing contour extraction on the grab part of the ship unloader image frame to obtain a grab target region;

[0025] S3022: generating a corresponding grab mask image based on the grab target region;

[0026] The formula is:

[0027]

[0028] In the formula, I m represents the grab mask image; i represents the pixel value of the corresponding position of the original ship unloader image frame; A represents the grab target region obtained by contour extraction;

[0029] S3023: superimposing and smoothing the ship unloader image frame and the grab mask image to obtain the enhanced image frame;

[0030] The formula is:

[0031] I n = (I mean · I m + I s ) · a;

[0032] In the formula, I n represents the enhanced image frame; I s represents the original ship unloader image frame; I mean represents the image obtained by smoothing and averaging the ship unloader image frame I s ; and a represents the set coefficient.

[0033] Preferably, in step S303, the processing steps of the target detection model include:

[0034] S3031: inputting the ship unloader image frame into the target detection model;

[0035] S3032: Multi-scale feature extraction is performed on the ship loader image frame through the backbone network to obtain a multi-scale feature map;

[0036] The formula of the backbone network is as follows:

[0037] F l = σ(conv(F l-1 , K l )+b l );

[0038] In the formula, F l represents the feature map output by the lth layer; conv represents a convolution operation; K l represents a convolution kernel weight; b l represents a bias term, and σ represents a SiLU activation function.

[0039] S3033: Top-down and bottom-up fusion is performed on the multi-scale feature map to obtain a multi-scale fused feature map.

[0040] The formula of the top-down fusion is as follows:

[0041]

[0042] The formula of the bottom-up fusion is as follows:

[0043]

[0044] In the formula, P l represents the feature map of the lth scale; and P represents the fused feature map of the lth and (l-1)th scales; Upsample and Downsample represent up-sampling and down-sampling.

[0045] S3034: The multi-scale fused feature map is input into a corresponding prediction head to obtain a multi-scale grab bucket target detection result.

[0046] S3035: The multi-scale grab bucket target detection results are integrated through global non-maximum suppression to output a merged grab bucket target detection result.

[0047] Preferably, in step S3, the grab bucket target detection result output by the target detection model includes a detection box position coordinate and a confidence score of the grab bucket target.

[0048] Preferably, in step S4, the processing steps of the target detection model include:

[0049] S401: A grab bucket target in a current ship loader image frame is taken as an input of the target detection model.

[0050] S402: tracking the grab bucket target in the current ship loader image frame by Kalman filtering, predicting the state of each grab bucket target in the next ship loader image frame, and predicting the detection box of the grab bucket target;

[0051] S403: matching the detection box of the grab bucket target in the current ship loader image frame with the detection box of the known grab bucket target in the previous ship loader image frame by the Hungarian algorithm, if the matching is successful, obtaining the unique annotation ID and motion trajectory of the grab bucket target in the previous ship loader image frame;

[0052] Wherein, the unique data ID includes three states of missing state, matching success state and creating detection state;

[0053] The missing state means that the unique data ID in the database continuously loses the matching for a number of times reaching the set mismatch threshold;

[0054] The matching success state means that the unique annotation ID of the grab bucket target matches the corresponding unique data ID in the database;

[0055] The creating detection state means that the unique annotation ID of the grab bucket target does not match the corresponding unique data ID in the database;

[0056] S404: matching the motion trajectory that is not matched successfully in the previous ship loader image frame with the detection box that is not matched successfully in the current ship loader image frame by cascading matching, if the matching is successful, associating the corresponding detection box with the motion trajectory matched successfully;

[0057] S405: outputting the unique annotation ID and the updated motion trajectory of the grab bucket target in the current ship loader image frame.

[0058] Preferably, in step S5, before calculating the trajectory matching degree, the grab bucket target predicted trajectory and the grab preset working trajectory are subjected to trajectory alignment, trajectory smoothing and trajectory format unification;

[0059] 1) Trajectory alignment:

[0060] Synchronize the time axis of the grab bucket target predicted trajectory and the grab preset working trajectory by timestamp or key event;

[0061] Convert the coordinates of the grab bucket target predicted trajectory and the grab preset working trajectory to the same spatial reference system by coordinate conversion;

[0062] 2) Trajectory smoothing

[0063] Smooth the grab bucket target predicted trajectory by Kalman filtering, moving average or Gaussian filtering;

[0064] Identify and eliminate outliers in the grab target prediction trajectory and the grab preset working trajectory;

[0065] 3) Uniform trajectory format

[0066] Uniform sampling rate of the grab target prediction trajectory and the grab preset working trajectory to ensure consistent number of trajectory points.

[0067] Preferably, in step S5, the processing step of calculating the corresponding trajectory matching degree based on the grab target prediction trajectory and the grab preset working trajectory comprises:

[0068] S501: Extracting the speed feature vector, the acceleration feature vector and the curvature feature vector from the grab target prediction trajectory and the grab preset working trajectory respectively;

[0069] S502: Calculating the cosine similarity of the speed feature vectors of the grab target prediction trajectory and the grab preset working trajectory;

[0070] S503: Calculating the cosine similarity of the acceleration feature vectors of the grab target prediction trajectory and the grab preset working trajectory;

[0071] S504: Calculating the cosine similarity of the curvature feature vectors of the grab target prediction trajectory and the grab preset working trajectory;

[0072] S505: Weighted calculation of the cosine similarity of the speed feature vector, the acceleration feature vector and the curvature feature vector to obtain the trajectory matching degree of the grab target prediction trajectory and the grab preset working trajectory.

[0073] Preferably, in step S6, the processing step of adjusting the control strategy of the gantry ship unloader based on the trajectory matching degree comprises:

[0074] 1) If the trajectory matching degree is greater than the first threshold value, it is judged that the grab target prediction trajectory and the grab preset working trajectory are highly consistent, and the gantry ship unloader is in a stable control state, and the grab control strategy does not need to be adjusted;

[0075] 2) If the trajectory matching degree is less than the first threshold value but greater than the second threshold value, it is judged that there is deviation between the grab target prediction trajectory and the grab preset working trajectory, and the grab control strategy of the gantry ship unloader is fine-tuned to suppress error accumulation;

[0076] 3) If the trajectory matching degree is less than the second threshold value, it is judged that there is a significant error between the grab target prediction trajectory and the grab preset working trajectory, and the grab control strategy of the gantry ship unloader is adjusted or the grab preset working trajectory is re-planned.

[0077] Compared with the prior art, the intelligent control method of the gantry ship unloader for the smart port in the application has the following beneficial effects:

[0078] This invention utilizes a target detection model to detect grab targets in image frames of a gantry crane's operational video. This model accurately identifies key features such as the grab's position and shape in each frame. Based on the detection results, a target tracking model further tracks and generates the grab's trajectory, ensuring that the predicted grab target trajectory matches the actual grab's running trajectory. This avoids trajectory deviation problems caused by human visual judgment errors or insufficient sensor accuracy in traditional control methods. Furthermore, by training the target detection and tracking models, the invention can effectively adapt to the complex operating environment of ports. Simultaneously, by calculating the matching degree between the predicted grab target trajectory and the preset working trajectory, the degree of deviation between the grab's operating state and the preset path can be quantitatively assessed, providing precise data for adjusting control strategies and thus improving the scientific rigor and accuracy of gantry crane control.

[0079] This invention employs real-time video processing technology, which can quickly decompose working videos into image frames and input them into a target detection model for batch processing. This significantly shortens the data acquisition and processing time cycle. Furthermore, the automated implementation of target detection and trajectory generation processes eliminates the need for manual intervention, significantly improving operational efficiency. In addition, the mechanism for adjusting the grab control strategy based on trajectory matching can dynamically optimize control parameters according to the actual operating status of the grab, enabling the control strategy to quickly adapt to different operating scenarios and conditions. This achieves efficient operation throughout the entire process from data acquisition to control strategy adjustment, thereby improving the overall efficiency and response speed of the gantry-type ship loader control. Attached Figure Description

[0080] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0081] Figure 1 This is a logic block diagram of an intelligent control method for gantry-type ship loaders used in smart ports. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0083] It should be noted that similar reference numerals and letters refer to like items in the accompanying drawings, and once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings. In the description of the application, it needs to be explained that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the application is used, which is only for the convenience of describing the application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application. In addition, the terms "first", "second", "third", and the like are only used for differentiation and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", and the like do not mean that the components must be absolutely horizontal or vertical, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined. In the description of the application, it also needs to be explained that unless otherwise explicitly specified and limited, the terms "arrangement", "installation", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be connected inside two elements. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0084] The specific embodiments are further described in detail below:

[0085] Embodiment:

[0086] A gantry ship loader intelligent control method for a smart port is disclosed in this embodiment.

[0087] As shown in the drawings, Figure 1 A gantry ship loader intelligent control method for a smart port, comprising:

[0088] S1: obtaining a working video of the gantry ship loader (when loading or unloading materials), and a preset working trajectory of the grab bucket;

[0089] In this embodiment, the preset working trajectory of the grab bucket is an optimal path generated by combining a path planning algorithm based on the position coordinates of the gantry ship loader and its grab bucket, the position coordinates of the material loading point (such as a ship cabin or a material storage yard), and the position coordinates of the material unloading point (such as a vehicle compartment or a material storage area).

[0090] S2: decompose the working video of the portal ship loader into a plurality of ship loader image frames;

[0091] S3: input the ship loader image frame into the trained target detection model for grab target detection, and output the grab target detection result in the corresponding ship loader image frame;

[0092] S4: input the grab target detection result in the ship loader image frame into the trained target detection model for grab target tracking and trajectory generation, and output the grab target prediction trajectory;

[0093] S5: calculate the corresponding trajectory matching degree based on the grab target prediction trajectory and the preset grab trajectory.

[0094] S6: adjust the grab control strategy of the portal ship loader based on the trajectory matching degree.

[0095] The application can accurately identify the position, shape and other key features of the grab in each image frame through the target detection model for grab target detection of the image frames of the ship loader working video, and further realize the tracking and trajectory generation of the grab target based on the detection result through the target tracking model, so that the output grab target prediction trajectory is consistent with the actual running trajectory of the grab, avoiding the trajectory deviation problem caused by manual visual judgment error or insufficient sensor accuracy in the traditional control method, and through the training of the target detection model and the target tracking model, the complex working environment of the port can be effectively adapted. At the same time, by calculating the matching degree of the grab target prediction trajectory and the preset working trajectory, the deviation degree of the grab running state and the preset path can be quantitatively evaluated, and accurate data basis is provided for control strategy adjustment, so as to improve the scientificity and accuracy of the portal ship loader control.

[0096] The application adopts real-time video processing technology, can quickly decompose the working video into image frames and input the target detection model for batch processing, greatly shortens the time period of data acquisition and processing, and realizes the automation of target detection and trajectory generation process without manual intervention, significantly improves the working efficiency, and the mechanism of adjusting the grab control strategy based on the trajectory matching degree can dynamically optimize the control parameters according to the actual running state of the grab, so that the control strategy quickly adapts to different working scenes and working condition changes, realizes the efficient operation of the whole process from data acquisition to control strategy adjustment, and improves the overall efficiency and response speed of the portal ship loader control.

[0097] In the specific implementation process, the working video is decoded by FFmpeg or OpenCV tools to obtain the original image data of the ship loader in the working video; the original image data of the ship loader in the working video is extracted frame by frame in time sequence, at fixed intervals or in key frame mode to obtain a plurality of ship loader image frames.

[0098] In this embodiment, metadata such as timestamps and frame numbers can be added to each frame of the ship loading and unloading machine image to facilitate alignment with the time axis during subsequent trajectory analysis.

[0099] In the specific implementation process, the processing steps for training the target detection model include:

[0100] S301: Obtain the image frames of the ship loading and unloading machine used as training samples, along with their corresponding real location information and category labels;

[0101] S302: Perform image enhancement processing on the grab bucket portion of the ship loader / unloader image frame to obtain an enhanced image frame;

[0102] S303: Input the enhanced image frame into the target detection model to perform grab target detection, and output the grab target detection result in the corresponding ship loader / unloader image frame;

[0103] S304: Calculate the loss function based on the target detection results of the grab bucket, combined with the real location information and category labels, and optimize the parameters of the target detection model;

[0104] S305: Repeat steps S301 to S304 to iteratively train the object detection model until the model converges.

[0105] In specific implementation, the processing steps for enhancing the grab portion of the ship loading / unloading machine image frame to obtain the enhanced image frame include:

[0106] S3021: Extract the contour of the grab portion of the ship loader / unloader image frame to obtain the grab (area of ​​interest) target region;

[0107] In this embodiment, the contour extraction process includes: preprocessing the loading and unloading machine image frame by denoising and contrast enhancement; converting the preprocessed loading and unloading machine image frame into a binary image; using a contour detection algorithm on the binary image to identify the contour of the grab bucket target area, and performing contour filtering to obtain the grab bucket (area of ​​interest) target area.

[0108] S3022: Generate the corresponding grab mask image based on the grab (area of ​​interest) target region;

[0109] The formula is expressed as:

[0110]

[0111] In the formula: I m represents the grab bucket mask image; i represents the pixel value at the corresponding position in the original ship loader image frame; A represents the grab bucket target area obtained by contour extraction;

[0112] S3023: superimpose and smooth the ship unloader image frame with the grab mask image to obtain an enhanced image frame;

[0113] The formula is:

[0114] I n = (I mean ·I m +I s )·a;

[0115] In the formula, I n represents the enhanced image frame; I s represents the original ship unloader image frame; I mean represents the image obtained by smoothing and averaging the ship unloader image frame I s ; and a represents a set coefficient, such as 0.5.

[0116] The present application realizes the contour enhancement of the grab by superimposing and smoothing the ship unloader image frame with the grab mask image, which is conducive to better highlighting the grab features in the key area of the ship unloader image frame in the training process, reducing the interference of dust, rain, fog, strong light and other interference of the port scene on the model training, thereby improving the training effect of the target detection model, and further assisting in improving the accuracy of the intelligent control of the portal ship unloader.

[0117] In the specific implementation process, the Yolov5 network model is used as the target detection model for grab target detection, and the grab target detection result in the corresponding ship unloader image frame is output.

[0118] The processing steps of the target detection model include:

[0119] S3031: input the ship unloader image frame into the target detection model;

[0120] S3032: perform multi-scale feature extraction on the ship unloader image frame through the backbone network to obtain a multi-scale feature map;

[0121] The formula of the backbone network is:

[0122] F l = σ(conv(F l-1 ,K l )+b l );

[0123] In the formula, F l represents the feature map output by the lth layer; conv represents convolution operation; K l represents the convolution kernel weight; b l represents the bias term, and sigma represents the SiLU activation function (Sigmoid-Weighted Linear Unit).

[0124] In this embodiment, the backbone network enhances the feature extraction capability through the residual structure and the CSP (Cross Stage Partial) connection, and finally outputs three scale feature maps P3, P4 and P5, which correspond to small, medium and large target detection respectively. P3, P4 and P5 are cross-scale fusion feature maps processed by a feature pyramid (such as PANet), which are generated after up-sampling, down-sampling and channel adjustment of multiple Fl (such as F3, F4 and F5), and have the characteristics of multi-scale semantic enhancement.

[0125] S3033: top-down and bottom-up fusion of the multi-scale feature maps to obtain a multi-scale fusion feature map;

[0126] The formula of top-down fusion is:

[0127]

[0128] The formula of bottom-up fusion is:

[0129]

[0130] In the formula, P l represents the feature map of the lth scale; represents the fusion feature map of the lth and (l-1)th scales; Upsample and Downsample represent up-sampling and down-sampling;

[0131] Specifically, top-down fusion means starting from the high semantic feature map (P5), adding the middle layer feature map (P4) through up-sampling (2 times), and then generating the fusion feature map P4_fused through convolution adjustment of the channel number; then the fusion feature map P4_fused is up-sampled and fused with the low layer feature map (P3) to generate the fusion feature map P3_fused. Bottom-up fusion means starting from the low layer feature map (P3), adding the middle layer fusion feature map (P4_fused) through down-sampling (2 times) to generate the final fusion feature map P4_fused_final; then the final fusion feature map P4_fused_final is down-sampled and fused with the high semantic feature map P5 to generate the fusion feature map P5_fused.

[0132] In this embodiment, after the processing of top-down fusion and bottom-up fusion, each fusion feature map retains the original scale spatial resolution, but fuses the semantic information of other scales (such as P3_fused fuses the context information of P4 and P5 to enhance the small target detection capability; P5_fused fuses the detail information of P3 and P4 to improve the large target positioning accuracy).

[0133] S3034: input the multi-scale fused feature maps into the corresponding prediction heads respectively to obtain multi-scale grab bucket target detection results;

[0134] In this embodiment, the output structure of each prediction head is the same, and each prediction head includes a detection box parameter, a target confidence, and a category probability. The grab bucket target detection results of the three scales cover the detection requirements of small, medium, and large targets respectively. For example, the high resolution (80x80) of P3_fused is suitable for detecting small size grab buckets (such as a long distance scene), and the low resolution (20x20) of P5_fused is suitable for detecting large size grab buckets (such as a close distance scene). The grab bucket target detection results of each scale will first undergo non-maximum suppression within the scale to filter redundant detection boxes (such as boxes with low confidence or high IoU) under the same scale.

[0135] S3035: integrate the multi-scale grab bucket target detection results through global non-maximum suppression to obtain merged grab bucket target detection results.

[0136] In this embodiment, all detection boxes of the three scales are summarized and sorted in descending order of confidence. Then, global non-maximum suppression is performed to delete cross-scale redundant boxes: if the IoU of two boxes (from different scales) is greater than a threshold (such as 0.6), the box with higher confidence is retained and the other box is suppressed.

[0137] In the specific implementation process, the grab bucket target detection results output by the target detection model include the detection box and the confidence score of the grab bucket target.

[0138] In the specific implementation process, the Deep Sort algorithm model is used as the target detection model to track the grab bucket target and generate a trajectory, and the grab bucket target prediction trajectory is output.

[0139] The processing steps of the target detection model include:

[0140] S401: input the grab bucket target in the current ship loader image frame into the target detection model;

[0141] S402: track the grab bucket target in the current ship loader image frame through Kalman filtering, predict the state of each grab bucket target in the next ship loader image frame, and predict and generate a detection box of the grab bucket target;

[0142] S403: match the detection box of the grab bucket target in the current ship loader image frame with the detection box of the known grab bucket target in the previous ship loader image frame through the Hungarian algorithm, and if the matching is successful, obtain the unique annotation ID and motion trajectory of the grab bucket target in the previous ship loader image frame for the grab bucket target;

[0143] The unique data ID includes a loss state, a matching success state, and a creation detection state.

[0144] The lost state refers to that the number of continuous loss of matching of the unique data ID in the database reaches a set loss threshold;

[0145] The matching success state refers to that the unique marking ID of the grab bucket target is matched to the corresponding unique data ID in the database;

[0146] The created detection state refers to that the unique marking ID of the grab bucket target is not matched to the corresponding unique data ID in the database;

[0147] S404: Match the motion trajectory that is not matched successfully in the previous ship loader image frame with the detection box that is not matched successfully in the current ship loader image frame through cascading matching, and if the matching is successful, associate the corresponding detection box with the motion trajectory matched successfully;

[0148] S405: Output the unique marking ID of the grab bucket target in the current ship loader image frame and the updated motion trajectory.

[0149] In the specific implementation process, before calculating the trajectory matching degree, the grab bucket target predicted trajectory and the grab preset working trajectory are subjected to trajectory alignment, trajectory smoothing and trajectory format unification;

[0150] 1) Trajectory alignment:

[0151] The time axes of the grab bucket target predicted trajectory and the grab preset working trajectory are aligned through timestamp synchronization or key events (such as the starting point of the grab bucket lifting / descending);

[0152] The coordinates of the grab bucket target predicted trajectory and the grab preset working trajectory are converted to the same spatial reference system through coordinate conversion;

[0153] 2) Trajectory smoothing

[0154] The grab bucket target predicted trajectory is subjected to smoothing processing through Kalman filtering, moving average or Gaussian filtering;

[0155] Based on statistical methods (such as the 3σ principle) or clustering algorithms (such as DBSCAN), outliers (such as false detection points caused by dust shielding) in the grab bucket target predicted trajectory and the grab preset working trajectory are identified and removed;

[0156] 3) Trajectory format unification

[0157] The sampling rates of the grab bucket target predicted trajectory and the grab preset working trajectory are unified through linear interpolation, spline interpolation or Fourier transform reconstruction to ensure that the number of points of the two trajectories is consistent.

[0158] Before calculating the trajectory matching degree, the trajectory alignment, trajectory smoothing and trajectory format unification are performed on the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket, so that the accuracy of the trajectory matching degree calculation can be improved, the reliability of subsequent control strategy adjustment (such as PID parameter optimization and fuzzy rule updating) can be ensured, and finally the precise intelligent control of the grab bucket movement of the gantry ship loader can be realized.

[0159] In the implementation process, the processing steps for calculating the corresponding trajectory matching degree based on the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket include:

[0160] S501: Extracting the speed feature vector, the acceleration feature vector and the curvature feature vector from the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket respectively.

[0161] In this embodiment, the speed feature vector is a vector composed of the speed values of each trajectory point on the trajectory, wherein the speed value of each trajectory point is calculated by dividing the coordinate difference between adjacent trajectory points by the sampling time interval. The acceleration feature vector is a vector composed of the acceleration values of each trajectory point on the trajectory, wherein the acceleration value of each trajectory point is calculated by dividing the speed difference between adjacent trajectory points by the sampling time interval. The curvature feature vector is a vector composed of the curvature values of each trajectory point on the trajectory, wherein the first derivative (speed) and the second derivative (acceleration) of the trajectory point are first calculated, then the curvature values of each trajectory point are calculated through the curvature formula, and finally the curvature values of each trajectory point are arranged in order to form the curvature feature vector reflecting the geometric shape of the trajectory.

[0162] S502: Calculating the cosine similarity of the speed feature vectors of the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket.

[0163] In this embodiment, the cosine similarity of the speed feature vectors is calculated by dividing the vector dot product result of the speed feature vectors of the two trajectories by the vector dot product result of the vector lengths of the speed feature vectors of the two trajectories. The cosine similarity of the acceleration feature vectors and the curvature feature vectors is calculated in the same way.

[0164] S503: Calculating the cosine similarity of the acceleration feature vectors of the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket.

[0165] S504: Calculating the cosine similarity of the curvature feature vectors of the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket.

[0166] S505: Weighted calculation of the cosine similarities of the speed feature vector, the acceleration feature vector and the curvature feature vector to obtain the trajectory matching degree of the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket.

[0167] The present application realizes accurate quantification of trajectory matching through multi-dimensional dynamic feature extraction and weighted similarity calculation. The speed, acceleration and curvature features respectively reflect the movement speed, change rate and path curvature, and after comprehensive consideration, the dynamic characteristics of the trajectory can be fully captured. By calculating the cosine similarity and weighting fusion, the key feature influence can be highlighted, the robustness of trajectory matching can be improved, and the accuracy and stability of subsequent gantry crane control can be improved.

[0168] In the specific implementation process, the processing steps of adjusting the control strategy of the gantry crane based on the trajectory matching degree include:

[0169] 1) If the trajectory matching degree is greater than the first threshold value, it is determined that the target predicted trajectory of the grab bucket and the preset working trajectory of the grab bucket are consistent in height, and the gantry crane is in a stable control state, and the control strategy of the grab bucket does not need to be adjusted;

[0170] 2) If the trajectory matching degree is less than the first threshold value but greater than the second threshold value, it is determined that there is a deviation between the target predicted trajectory of the grab bucket and the preset working trajectory of the grab bucket, and the control strategy of the grab bucket of the gantry crane is fine-tuned to suppress error accumulation;

[0171] 3) If the trajectory matching degree is less than the second threshold value, it is determined that there is a significant error between the target predicted trajectory of the grab bucket and the preset working trajectory of the grab bucket, and the control strategy of the grab bucket of the gantry crane is adjusted or the preset working trajectory of the grab bucket is re-planned.

[0172] Among them, the first threshold value and the second threshold value are adjusted in real time according to environmental parameters (such as wind speed, load weight), for example, the matching degree threshold value is increased when the load is heavy to enhance the fault tolerance of the system.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit the technical solutions. Those of ordinary skill in the art should understand that those who modify or equivalently replace the technical solutions of the present application without departing from the purpose and scope of the technical solutions should be covered in the scope of the claims of the present application.

Claims

1. A gantry crane intelligent control method for a smart port, characterized in that, The method comprises the following steps: S1: obtaining a working video of a portal ship loader and a preset working track of a grab bucket; S2: decomposing the working video of the portal ship loader into a plurality of ship loader image frames; S3: inputting the ship loader image frames into a trained target detection model for grab bucket target detection, and outputting a grab bucket target detection result in the corresponding ship loader image frame; S4: inputting the grab bucket target detection result in the ship loader image frame into the trained target detection model for grab bucket target tracking and track generation, and outputting a grab bucket target prediction track; S5: calculating a corresponding track matching degree based on the grab bucket target prediction track and the grab bucket preset working track; S6: adjusting a grab bucket control strategy of the portal ship loader based on the track matching degree.

2. The intelligent control method for the portal crane in the smart port according to claim 1, characterized in that: In step S2, the working video is decoded to obtain original image data of the ship loader in the working video; the original image data of the ship loader in the working video is extracted frame by frame in a time sequence, at a fixed interval or in a key frame mode, to obtain a plurality of ship loader image frames. 3.The intelligent control method for the portal-type ship unloader of the smart port according to claim 1, characterized in that: In step S3, the processing steps of training the target detection model comprise: S301: obtaining a ship loader image frame as a training sample, and corresponding real position information and a category label thereof; S302: performing image enhancement processing on the grab bucket part of the ship loader image frame to obtain an enhanced image frame; S303: inputting the enhanced image frame into the target detection model for grab bucket target detection, and outputting a grab bucket target detection result in the corresponding ship loader image frame; S304: calculating a loss function based on the grab bucket target detection result, the real position information and the category label, and optimizing parameters of the target detection model; S305: iteratively training the target detection model by repeating steps S301 to S304 until the model converges.

4. The intelligent control method for the portal crane in the smart port according to claim 3, characterized in that: In step S302, the processing steps of obtaining the enhanced image frame by performing image enhancement processing on the grab bucket part of the ship loader image frame comprise: S3021: performing contour extraction on the grab bucket part of the ship loader image frame to obtain a grab bucket target region; S3022: generating a corresponding grab bucket mask image based on the grab bucket target region; S3023: superimposing and smoothing the ship loader image frame and the grab bucket mask image to obtain the enhanced image frame; wherein: I m represents the image of the grab mask; i represents the pixel value of the corresponding position of the original ship unloader image frame; A represents the target region of the grab obtained by contour extraction; In step S303, the processing steps of the target detection model comprise: S3031: inputting the ship loader image frame into the target detection model; I n = (I mean · I m + I s ) · a; wherein: I n represents an enhanced image frame; I s represents an original ship unloader image frame; I mean represents an image after smoothing average of the ship unloader image frame I s ; a represents a set coefficient.

5. The intelligent control method for the portal crane in the smart port according to claim 3, characterized in that: S3032: performing multi-scale feature extraction on the ship loader image frame by a backbone network to obtain a multi-scale feature map; S3033: performing top-down and bottom-up fusion on the multi-scale feature map to obtain a multi-scale fusion feature map; S3034: inputting the multi-scale fusion feature map into corresponding prediction heads to obtain multi-scale grab bucket target detection results; S3035: integrating the multi-scale grab bucket target detection results by global non-maximum suppression to output merged grab bucket target detection results. F l = σ(conv(F l-1 , K l ) + b l ); In the formula, F l represents the feature map output by the lth layer; conv represents a convolution operation; K l represents the convolution kernel weight; b l represents a bias term, and σ represents a SiLU activation function. ​ ​ ​ wherein: P l represents the feature map of the l-th scale; represents the fused feature map of the l-th and l-1-th scale; Upsample, Downsample represent up-sampling and down-sampling; ​ ​ 6.The intelligent control method for the portal-type ship unloader of the smart port of claim 1, wherein: In step S3, the grab target detection result output by the target detection model includes the position coordinates of the detection box of the grab target and the confidence score thereof.

7. The intelligent control method for the portal crane in the smart port according to claim 6, characterized in that: In step S4, the processing steps of the target detection model include: S401: taking the grab target in the current ship loader image frame as the input of the target detection model; S402: performing target tracking on the grab target in the current ship loader image frame by Kalman filtering, predicting the state of each grab target in the next ship loader image frame, and predicting the detection box of the grab target; S403: matching the detection box of the grab target in the current ship loader image frame with the detection box of the known grab target in the previous ship loader image frame by the Hungarian algorithm, and if the matching is successful, obtaining the unique annotation ID and motion trajectory of the grab target in the previous ship loader image frame; wherein the unique data ID includes a lost state, a matched successful state and a created detection state; the lost state refers to that the unique data ID in the database continuously loses the matching for a number of times reaching a set mismatch threshold; the matched successful state refers to that the unique annotation ID of the grab target is matched to the corresponding unique data ID in the database; the created detection state refers to that the unique annotation ID of the grab target is not matched to the corresponding unique data ID in the database; S404: matching the motion trajectory that is not matched successfully in the previous ship loader image frame with the detection box that is not matched successfully in the current ship loader image frame by cascading matching, and if the matching is successful, associating the corresponding detection box with the motion trajectory matched successfully; S405: outputting the unique annotation ID and the updated motion trajectory of the grab target in the current ship loader image frame. 8.The intelligent control method for the portal-type ship unloader of the smart port of claim 1, wherein: Before calculating the trajectory matching degree in step S5, the grab target predicted trajectory and the preset working trajectory of the grab are subjected to trajectory alignment, trajectory smoothing and trajectory format unification. 1) trajectory alignment: synchronizing the time axes of the grab target predicted trajectory and the preset working trajectory of the grab by time stamp or key event alignment; converting the coordinates of the grab target predicted trajectory and the preset working trajectory of the grab to the same spatial reference system by coordinate conversion; 2) trajectory smoothing smoothing the grab target predicted trajectory by Kalman filtering, moving average or Gaussian filtering; identifying and removing outliers in the grab target predicted trajectory and the preset working trajectory of the grab; 3) trajectory format unification unifying the sampling rates of the grab target predicted trajectory and the preset working trajectory of the grab to ensure that the number of points of the two trajectories is consistent. 9.The intelligent control method for the portal-type ship unloader of the smart port of claim 1, wherein: In step S5, the processing steps for calculating the corresponding trajectory matching degree based on the grab target predicted trajectory and the preset working trajectory of the grab include: S501: extracting the velocity feature vector, the acceleration feature vector and the curvature feature vector from the grab target predicted trajectory and the preset working trajectory of the grab, respectively; S502: calculating the cosine similarity of the velocity feature vectors of the grab target predicted trajectory and the preset working trajectory of the grab; S503: calculating the cosine similarity of the acceleration feature vectors of the grab target predicted trajectory and the preset working trajectory of the grab; S504: calculating the cosine similarity of the curvature feature vectors of the grab target predicted trajectory and the preset working trajectory of the grab; S505: The cosine similarity of the speed feature vector, the acceleration feature vector and the curvature feature vector is weighted to obtain the trajectory matching degree of the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket. 10.The intelligent control method for the portal ship-to-shore crane of the smart port of claim 1, wherein: In step S6, the processing steps of adjusting the gantry crane grab bucket control strategy based on the trajectory matching degree include: 1) If the trajectory matching degree is greater than the first threshold value, it is judged that the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket are highly consistent, and the gantry crane is in a stable control state, so there is no need to adjust the grab bucket control strategy; 2) If the trajectory matching degree is less than the first threshold value but greater than the second threshold value, it is judged that there is a deviation between the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket, and the grab bucket control strategy of the gantry crane is fine-tuned to suppress error accumulation; 3) If the trajectory matching degree is less than the second threshold value, it is judged that there is a significant error between the target prediction trajectory of the grab bucket and the preset working trajectory of the grab bucket, and the grab bucket control strategy of the gantry crane is adjusted or the preset working trajectory of the grab bucket is re-planned.

Images