Automatic ship berthing and leaving method based on video monitoring and image recognition

Abstract

The invention discloses an automatic ship berthing and leaving method based on video monitoring and image recognition, which comprises the following steps: S1, acquiring real-time video data, and carrying out image preprocessing; s2, extracting target feature information by using a target detection model based on an OfficientDet algorithm; s3, generating environment sensing information by using a multi-modal sensor fusion technology; s4, based on the environment perception information, constructing and updating a three-dimensional dynamic map of the surrounding environment of the ship in real time; s5, performing global path planning by adopting a fast extension random tree-star algorithm; s6, local dynamic obstacle avoidance is carried out in combination with a two-stage neural network algorithm; s7, based on the global path planning result and the local dynamic obstacle avoidance strategy, the adaptive control system accurately adjusts the speed, course and attitude of the ship according to the real-time environment change; and S8, performing big data analysis and historical data mining by using the cloud computing platform. According to the invention, image recognition and deep learning technologies are utilized to realize automatic berthing and unberthing of the ship, and the method has the advantages of high efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image recognition and deep learning technology, and in particular to an automatic berthing and unberthing method for ships based on video monitoring and image recognition. Background Art

[0002] With the rapid development of the global shipping industry, the demand for automation and intelligence in ports and terminals is growing. Traditional ship berthing and unberthing operations mainly rely on manual control, and operators complete complex berthing and unberthing operations through visual observation and experience judgment. This method is not only labor-intensive and difficult to operate, but also easily affected by environmental factors such as weather, light and water flow, resulting in a high risk of ship operation. At the same time, the efficiency and precision of manual operation are difficult to meet the needs of efficient operation of modern ports, and an automated solution is urgently needed to improve the safety and efficiency of ship berthing and unberthing.

[0003] At present, the technology of automated ship berthing and unberthing has made some progress, among which sensor-based environmental perception and path planning technology are the key. However, there are some obvious defects and shortcomings in the existing technology. First, the environmental perception capability of a single sensor is limited, and it is difficult to provide comprehensive and accurate perception information in a complex and dynamic environment. For example, cameras perform poorly in low light and bad weather conditions. Although radar and lidar can make up for the shortcomings of visual perception to a certain extent, the data of a single sensor is difficult to fully reflect the complexity of the actual environment.

[0004] Secondly, existing image recognition and target detection algorithms still face challenges in terms of real-time performance and accuracy in complex scenarios. Although deep learning algorithms, such as YOLO and SSD, perform well in the field of target detection, they still need to be further optimized and improved in application scenarios with high real-time requirements such as automatic berthing and unberthing of ships. In addition, traditional path planning algorithms, such as A* and Dijkstra algorithms, although they can find better paths, perform poorly in dynamic obstacle avoidance and real-time adjustment, and are difficult to adapt to complex and changing environments.

[0005] Therefore, how to provide a method for automatic berthing and unberthing of ships based on video monitoring and image recognition is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0006] One object of the present invention is to propose a method for automatic berthing and unberthing of ships based on video surveillance and image recognition. The present invention utilizes multimodal sensor fusion, stereo vision technology and deep learning algorithms to achieve automatic berthing and unberthing of ships. Through real-time video surveillance and target detection, combined with Kalman filtering and rapidly expanding random tree-star (RRT*) algorithm, path planning is performed to ensure safe, efficient and accurate navigation of ships in complex environments. This method improves the comprehensiveness and accuracy of environmental perception, has the ability of adaptive learning and continuous optimization, significantly improves the safety and reliability of automated operations, and is suitable for the intelligent development of modern ports and docks.

[0007] According to an embodiment of the present invention, a method for automatic berthing and unberthing of a ship based on video monitoring and image recognition comprises the following steps:

[0008] S1. Deploy cameras around ships and docks to obtain real-time video data and perform image preprocessing, including denoising, contrast enhancement, and edge detection;

[0009] S2. Use the target detection model based on the EfficientDet algorithm to perform real-time target detection on the preprocessed images, identify and locate ships, docks, other ships and dynamic obstacles, and extract target feature information;

[0010] S3, using multimodal sensor fusion technology to fuse the real-time data of the camera, radar and inertial navigation system with the extracted target feature information, and integrate multi-sensor data through Kalman filtering to generate environmental perception information;

[0011] S4. Based on the fused environmental perception information, the depth information of obstacles is calculated using stereo vision technology to construct and update a three-dimensional dynamic map of the ship's surroundings in real time;

[0012] S5. Use the fast-expanding random tree-star algorithm for global path planning, comprehensively consider the ship's dynamic characteristics, environmental constraints and safety distance, and calculate the optimal berthing or unberthing path;

[0013] S6, combined with a two-stage neural network algorithm, uses environmental perception information and target feature information to perform local dynamic obstacle avoidance and adjust the speed and heading of the ship in real time;

[0014] S7, based on the global path planning results and the local dynamic obstacle avoidance strategy, the adaptive control system accurately adjusts the speed, heading and attitude of the ship according to the real-time environmental changes;

[0015] S8. Use cloud computing platforms to conduct big data analysis and historical data mining to continuously optimize global path planning and local dynamic obstacle avoidance strategies.

[0016] Furthermore, the S2 specifically includes:

[0017] S21, input the preprocessed image data into the target detection model based on the EfficientDet algorithm for processing, and extract feature information in the image through a convolutional neural network;

[0018] S22. In the EfficientDet model, a double-layer pyramid structure is used for multi-scale feature fusion, and feature maps of different scales are weighted fused:

[0019]

[0020] Among them, F out is the fused feature map, N1 represents the total number of scales of the feature map, and F i represents the feature map of the i-th scale, w i Represents the weight coefficient of the feature map of the i-th scale, satisfying

[0021] S23. In the detection head of the EfficientDet model, the anchor box mechanism is used to perform step-by-step regression and classification on the feature map to obtain the three-dimensional position, scale and confidence score of the detection box;

[0022] S24. The loss function of the target detection model consists of classification loss and bounding box regression loss. The total loss function is expressed as:

[0023]

[0024] Where N2 represents the total number of detected targets, p i represents the model's predicted classification probability for the i-th target, represents the true classification label of the i-th target, t i represents the model's predicted bounding box parameters for the i-th target, represents the true bounding box parameters of the i-th target, L represents the total loss function value, Represents the classification loss of the i-th target:

[0025]

[0026] represents the bounding box regression loss of the i-th object:

[0027]

[0028] Among them, t ij It indicates the value of the i-th target predicted by the model on the j-th parameter. represents the true value of the i-th target on the j-th parameter, smooth L1(x) is used to smooth the difference between the predicted value and the true value;

[0029] S25, further extracting feature information of the target according to the classification result and the detection frame information, including three-dimensional position (x, y, z), scale (w, h), confidence score a, speed v and direction θ, where x, y represent the center coordinates of the target in the image, z represents the height of the target, w represents the width of the target image, and h represents the length of the target image;

[0030] S26, performing non-maximum suppression on the output of the target detection model, selecting the optimal detection frame according to the confidence score a, and eliminating overlapping detection frames to obtain the precise location, size, category and motion state information of the ship, dock, other ships and dynamic obstacles;

[0031] S27. Obtain target feature information, and store the extracted target feature information in a feature vector f = [x, y, z, w, h, v, θ, a].

[0032] Furthermore, the S3 specifically includes:

[0033] S31, using the Kalman filter to take the real-time data of the camera, radar and inertial navigation system and the target feature information extracted by S27 as input, and perform multimodal sensor fusion, and the initial state vector is x0;

[0034] S32, in the prediction stage of the Kalman filter, the state x at the next moment is predicted based on the dynamic model of the system k|k-1 and the covariance matrix P k|k-1 :

[0035] x k|k-1 =Fx k-1 +Bu k-1 ;

[0036] P k|k-1 =FP k-1 F T +Q;

[0037] Where F represents the state transfer matrix, x k-1 represents the prior state vector, P k-1 represents the prior covariance matrix, B represents the control input matrix, u k-1 represents the control vector, Q represents the process noise covariance matrix;

[0038] S33, in the update phase, using the sensor observation vector z k Update the state estimate and covariance matrix:

[0039] y k =z k -Hxk|k-1 ;

[0040] S k =HP k|k-1 H T +R;

[0041]

[0042] x k =x k|k-1 +K k y k ;

[0043] P k =(IK k H)P k|k-1 ;

[0044] Among them, y k represents the measurement residual vector, H represents the observation matrix, S k represents the measurement residual covariance matrix, R represents the measurement noise covariance matrix, H T represents the transpose of the observation matrix, K k represents the Kalman gain matrix, represents the inverse of the measurement residual covariance matrix, x k represents the updated state vector, which is the optimal estimate after combining the predicted state and the observed information, P k represents the updated covariance matrix, and I represents the identity matrix, which is used to maintain the consistency of matrix dimensions;

[0045] S34, finally generating environmental perception information, including three-dimensional position, scale, confidence score, speed and direction.

[0046] Furthermore, the S4 specifically includes:

[0047] S41. Using stereo vision technology, the left and right images are obtained by using a binocular camera installed on the ship, and the feature points in the image pair are extracted and matched. The coordinates of the left image point are (u L ,v L ), the coordinates of the right image point are (u R ,v R );

[0048] S42, calculate the parallax δ of the feature point between the left and right images:

[0049]

[0050] S43, according to the parallax δ and the baseline distance B of the binocular camera, combined with the radial and tangential distortion parameters of the lens, calculate the depth Z of the feature point:

[0051]

[0052] Where f represents the focal length of the camera, k1 and k2 represent the radial distortion coefficients, and p1 and p2 represent the tangential distortion coefficients;

[0053] S44. According to the depth Z of the feature point and the image point coordinates (u, v), the camera intrinsic parameter matrix is ​​used to convert the image coordinates into three-dimensional space coordinates (X, Y, Z):

[0054]

[0055] Among them, c x and c y represents the camera principal point coordinates, f x and f y Represents the scaling factor of the camera focal length in the x and y directions, r 2 =u 2 +v 2 ;

[0056] S45, matching and fusing the calculated three-dimensional spatial coordinates with the environmental perception information fused by the multimodal sensor, and updating the three-dimensional position, scale, confidence score, speed and direction of the target;

[0057] S46, constructing a three-dimensional dynamic map of the vessel's surroundings, including all detected targets and their precise locations, sizes, motion states, and category information;

[0058] S47. Update the three-dimensional dynamic map in real time, and continuously adjust and optimize the three-dimensional environmental model based on the new data obtained by the sensor and the fused environmental perception information to ensure the accuracy and real-time performance of the environmental map.

[0059] Furthermore, the S5 specifically includes:

[0060] S51, set the starting point of the global path planning to the current position of the ship (x0, y0, z0), and the end point to the target berthing position (x t ,y t ,z t );

[0061] S52, using the fast expanding random tree-star algorithm for global path planning, initializing a tree containing the starting point (x0, y0, z0), and gradually expanding the nodes to cover the entire search space until a feasible path from the starting point to the end point is found;

[0062] S53. During the path extension process, the cost function is used to evaluate the path cost of each node:

[0063] J(s i ,s j)=α·dist(s i ,s j )+β·risk(s j )+γ·dyn(s i ,s j );

[0064] Among them, J(s i ,s j ) represents node s i To node s j The path cost, α represents the weight coefficient of the distance cost, dist(s i ,s j ) represents node s i To node s j The distance between them, β represents the weight coefficient of risk cost, risk(s j ) represents node s j The risk value of the location, γ represents the weight coefficient of the dynamic cost, dyn(s i ,s j ) represents node s i To node s j The cost of ship dynamic characteristics;

[0065]

[0066] Among them, m represents the mass of the ship, a ij Represents node s i To node s j acceleration, J1 represents the ship's moment of inertia, ω ij Represents node s i To node s j angular velocity, η represents the hydrodynamic drag coefficient, F ij Represents node s i To node s j The thrust between ij Indicates the angle between the thrust direction and the heading;

[0067] S54, expand the nodes of the tree, and after the path search is completed, the optimal path P from the starting point to the end point is obtained. t}, each node contains the status information of the ship;

[0068] S55. The position and status information of each node in the optimal path is transmitted to the control system to guide the ship to navigate and adjust according to the path planning results.

[0069] Furthermore, the S54 specifically includes:

[0070] S541, randomly sample a state srand ;

[0071] S542, find the same as s in the tree rand The nearest nodes nearest ;

[0072] S543, generate new node s new , so that new Under the control input, from s nearest To s rand Extensions;

[0073] S544、Inspection new Is it within the obstacle range? If not, s new Add to the tree and update the path cost;

[0074] S545. Repeat steps S541-S544.

[0075] Furthermore, the S6 specifically includes:

[0076] S61. Define the reward function R(s t ,a t ) is used to evaluate the score of each state-action pair:

[0077] R(s t ,a t )=r goal -(α·dist(s t ,s goal )+β·risk(s t )+γ·dyn(s t ,a t ));

[0078] Among them, R(s t ,a t ) means in state s t Next, perform action a t The reward obtained, r goal Represents the reward value for reaching the target state, dist(s t ,s goal ) indicates the current state s t With the target state s goal The distance between risk(s t ) indicates the current state s t The risk value, dyn(s t ,a t ) indicates state s t Next, perform action a t The cost of ship dynamics characteristics;

[0079] S62, randomly initialize two neural networks Q and The weight parameter, is the target network, which has the same structure as Q, but the parameters are updated independently;

[0080] S63. Select action a based on ∈-greedy strategy t :

[0081]

[0082] in, represents the optimal action estimated by the current Q network;

[0083] S64. Perform action a in the environment t , observe the new state s t+1 and reward R(s t ,a t );

[0084] S65、(s t ,a t ,R(s t ,a t ),s t+1 ) is stored in the experience replay buffer;

[0085] S66, randomly sampling small batches of data from the experience replay buffer, and updating the Q network parameters using the mean square error loss function;

[0086] S67, during the ship's navigation, use the trained two-level neural network model to select the optimal action in real time t , dynamically adjust the speed and course of the ship to ensure that the ship can berth and leave safely and accurately in complex environments.

[0087] The beneficial effects of the present invention are:

[0088] First, the present invention deploys high-definition cameras to acquire real-time video data and performs image preprocessing, including denoising, contrast enhancement, and edge detection, which significantly improves the image quality and the accuracy of feature extraction. Using a target detection model based on the EfficientDet algorithm, real-time target detection is performed on the preprocessed images, which can accurately identify and locate ships, docks, other vessels, and dynamic obstacles, and extract detailed target feature information. The innovative algorithm selection and multi-scale feature fusion technology in this process ensure efficient and accurate target detection and recognition, even in complex and dynamic environments.

[0089] Secondly, the real-time data of the camera, radar and inertial navigation system are fused with the extracted target feature information by using multimodal sensor fusion technology, and the multi-sensor data is integrated through the Kalman filter to generate accurate environmental perception information. This fusion process improves the comprehensiveness and accuracy of environmental perception and can maintain robustness under different lighting, weather and environmental conditions. At the same time, based on the fused environmental perception information, the depth information of obstacles is calculated using stereo vision technology, and a three-dimensional dynamic map of the ship's surrounding environment is constructed and updated in real time, providing reliable basic data for subsequent path planning and dynamic obstacle avoidance.

[0090] Finally, in terms of path planning, the present invention adopts the Rapidly Expanding Random Tree-Star (RRT*) algorithm, which comprehensively considers the dynamic characteristics of the ship, environmental constraints and safety distance, and calculates the optimal berthing or unberthing path. This method can not only find the optimal path in a complex environment, but also dynamically adjust to adapt to environmental changes, ensuring the safety and efficiency of ship navigation. Combined with the double-level neural network (Double DQN) algorithm, the system uses environmental perception information and target feature information for local dynamic obstacle avoidance. The system can adjust the speed and heading of the ship in real time, effectively avoid potential collision risks, and improve the efficiency and reliability of dynamic obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0092] Figure 1 A flow chart of a method for automatic berthing and unberthing of ships based on video monitoring and image recognition proposed by the present invention;

[0093] Figure 2 A global path planning flow chart of a method for automatic berthing and unberthing of ships based on video monitoring and image recognition proposed by the present invention;

[0094] Figure 3 The present invention provides a local dynamic obstacle avoidance flow chart of a method for automatic berthing and unberthing of ships based on video monitoring and image recognition. DETAILED DESCRIPTION

[0095] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0096] refer to Figure 1-3 , a method for automatic berthing and unberthing of a ship based on video monitoring and image recognition, comprising the following steps:

[0097] S1. Deploy cameras around ships and docks to obtain real-time video data and perform image preprocessing, including denoising, contrast enhancement, and edge detection;

[0098] S2. Use the target detection model based on the EfficientDet algorithm to perform real-time target detection on the preprocessed images, identify and locate ships, docks, other ships and dynamic obstacles, and extract target feature information;

[0099] S3, using multimodal sensor fusion technology to fuse the real-time data of the camera, radar and inertial navigation system with the extracted target feature information, and integrate multi-sensor data through Kalman filtering to generate environmental perception information;

[0100] S4. Based on the fused environmental perception information, the depth information of obstacles is calculated using stereo vision technology to construct and update a three-dimensional dynamic map of the ship's surroundings in real time;

[0101] S5. Use the fast-expanding random tree-star algorithm for global path planning, comprehensively consider the ship's dynamic characteristics, environmental constraints and safety distance, and calculate the optimal berthing or unberthing path;

[0102] S6, combined with a two-stage neural network algorithm, uses environmental perception information and target feature information to perform local dynamic obstacle avoidance and adjust the speed and heading of the ship in real time;

[0103] S7, based on the global path planning results and the local dynamic obstacle avoidance strategy, the adaptive control system accurately adjusts the speed, heading and attitude of the ship according to the real-time environmental changes;

[0104] S8. Use cloud computing platforms to conduct big data analysis and historical data mining to continuously optimize global path planning and local dynamic obstacle avoidance strategies.

[0105] In this implementation, S2 specifically includes:

[0106] S21, input the preprocessed image data into the target detection model based on the EfficientDet algorithm for processing, and extract feature information in the image through a convolutional neural network;

[0107] S22. In the EfficientDet model, a double-layer pyramid structure is used for multi-scale feature fusion, and feature maps of different scales are weighted fused:

[0108]

[0109] Among them, F out is the fused feature map, N1 represents the total number of scales of the feature map, and F i represents the feature map of the i-th scale, wi Represents the weight coefficient of the feature map of the i-th scale, satisfying

[0110] S23. In the detection head of the EfficientDet model, the anchor box mechanism is used to perform step-by-step regression and classification on the feature map to obtain the three-dimensional position, scale and confidence score of the detection box;

[0111] S24. The loss function of the target detection model consists of classification loss and bounding box regression loss. The total loss function is expressed as:

[0112]

[0113] Where N2 represents the total number of detected targets, p i represents the model's predicted classification probability for the i-th target, represents the true classification label of the i-th target, t i represents the model's predicted bounding box parameters for the i-th target, represents the true bounding box parameters of the i-th target, L represents the total loss function value, Represents the classification loss of the i-th target:

[0114]

[0115] represents the bounding box regression loss of the i-th object:

[0116]

[0117] Among them, t ij It indicates the value of the i-th target predicted by the model on the j-th parameter. represents the true value of the i-th target on the j-th parameter, smooth L1 (x) is used to smooth the difference between the predicted value and the true value;

[0118] S25, further extracting feature information of the target according to the classification result and the detection frame information, including three-dimensional position (x, y, z), scale (w, h), confidence score a, speed v and direction θ, where x, y represent the center coordinates of the target in the image, z represents the height of the target, w represents the width of the target image, and h represents the length of the target image;

[0119] S26, performing non-maximum suppression on the output of the target detection model, selecting the optimal detection frame according to the confidence score a, and eliminating overlapping detection frames to obtain the precise location, size, category and motion state information of the ship, dock, other ships and dynamic obstacles;

[0120] S27. Obtain target feature information, and store the extracted target feature information in a feature vector f = [x, y, z, w, h, v, θ, a].

[0121] In this implementation, S3 specifically includes:

[0122] S31, using the Kalman filter to take the real-time data of the camera, radar and inertial navigation system and the target feature information extracted by S27 as input, and perform multimodal sensor fusion, and the initial state vector is x0;

[0123] S32, in the prediction stage of the Kalman filter, the state x at the next moment is predicted based on the dynamic model of the system k|k-1 and the covariance matrix P k|k-1 :

[0124] x k|k-1 =Fx k-1 +Bu k-1 ;

[0125] P k|k-1 =FP k-1 F T +Q;

[0126] Where F represents the state transfer matrix, x k-1 represents the prior state vector, P k-1 represents the prior covariance matrix, B represents the control input matrix, u k-1 represents the control vector, Q represents the process noise covariance matrix;

[0127] S33, in the update phase, using the sensor observation vector z k Update the state estimate and covariance matrix:

[0128] y k =z k -Hx k|k-1 ;

[0129] S k =HP k|k-1 H T +R;

[0130]

[0131] x k =x k|k-1 +K k y k ;

[0132] P k =(IK k H)P k|k-1 ;

[0133] Among them, y k represents the measurement residual vector, H represents the observation matrix, S k represents the measurement residual covariance matrix, R represents the measurement noise covariance matrix, H T represents the transpose of the observation matrix, K k represents the Kalman gain matrix, represents the inverse of the measurement residual covariance matrix, x k represents the updated state vector, which is the optimal estimate after combining the predicted state and the observed information, P k represents the updated covariance matrix, and I represents the identity matrix, which is used to maintain the consistency of matrix dimensions;

[0134] S34, finally generating environmental perception information, including three-dimensional position, scale, confidence score, speed and direction.

[0135] In this implementation, S4 specifically includes:

[0136] S41. Using stereo vision technology, the left and right images are obtained by using a binocular camera installed on the ship, and the feature points in the image pair are extracted and matched. The coordinates of the left image point are (u L ,v L ), the coordinates of the right image point are (u R ,v R );

[0137] S42, calculate the parallax δ of the feature point between the left and right images:

[0138]

[0139] S43, according to the parallax δ and the baseline distance B of the binocular camera, combined with the radial and tangential distortion parameters of the lens, calculate the depth Z of the feature point:

[0140]

[0141] Where f represents the focal length of the camera, k1 and k2 represent the radial distortion coefficients, and p1 and p2 represent the tangential distortion coefficients;

[0142] S44. According to the depth Z of the feature point and the image point coordinates (u, v), the camera intrinsic parameter matrix is ​​used to convert the image coordinates into three-dimensional space coordinates (X, Y, Z):

[0143]

[0144] Among them, c x and c y represents the coordinates of the camera principal point, f x and f yRepresents the scaling factor of the camera focal length in the x and y directions, r 2 =u 2 +v 2 ;

[0145] S45, matching and fusing the calculated three-dimensional spatial coordinates with the environmental perception information fused by the multimodal sensor, and updating the three-dimensional position, scale, confidence score, speed and direction of the target;

[0146] S46, constructing a three-dimensional dynamic map of the vessel's surroundings, including all detected targets and their precise locations, sizes, motion states, and category information;

[0147] S47. Update the three-dimensional dynamic map in real time, and continuously adjust and optimize the three-dimensional environmental model based on the new data obtained by the sensor and the fused environmental perception information to ensure the accuracy and real-time performance of the environmental map.

[0148] In this implementation manner, S5 specifically includes:

[0149] S51, set the starting point of the global path planning to the current position of the ship (x0, y0, z0), and the end point to the target berthing position (x t ,y t ,z t );

[0150] S52, using the fast expanding random tree-star algorithm for global path planning, initializing a tree containing the starting point (x0, y0, z0), and gradually expanding the nodes to cover the entire search space until a feasible path from the starting point to the end point is found;

[0151] S53. During the path extension process, the cost function is used to evaluate the path cost of each node:

[0152] J(s i ,s j )=α·dist(s i ,s j )+β·risk(s j )+γ·dyn(s i ,s j );

[0153] Among them, J(s i ,s j ) represents node s i To node s j The path cost, α represents the weight coefficient of the distance cost, dist(s i ,s j ) represents node s i To node s jThe distance between them, β represents the weight coefficient of risk cost, risk(s j ) represents node s j The risk value of the location, γ represents the weight coefficient of the dynamic cost, dyn(s i ,s j ) represents node s i To node s j The cost of ship dynamic characteristics;

[0154]

[0155] Among them, m represents the mass of the ship, a ij Represents node s i To node s j acceleration, J1 represents the ship's moment of inertia, ω ij Represents node s i To node s j angular velocity, η represents the hydrodynamic drag coefficient, F ij Represents node s i To node s j The thrust between ij Indicates the angle between the thrust direction and the heading;

[0156] S54, expand the nodes of the tree, and after the path search is completed, the optimal path P from the starting point to the end point is obtained. t}, each node contains the status information of the ship;

[0157] S55. The position and status information of each node in the optimal path is transmitted to the control system to guide the ship to navigate and adjust according to the path planning results.

[0158] In this implementation manner, the S54 specifically includes:

[0159] S541, randomly sample a state s rand ;

[0160] S542, find the same as s in the tree rand The nearest nodes nearest ;

[0161] S543, generate new node s new , so that new Under the control input, from s nearest To s rand Extensions;

[0162] S544、Inspection new Is it within the obstacle range? If not, s new Add to the tree and update the path cost;

[0163] S545. Repeat steps S541-S544.

[0164] In this implementation manner, S6 specifically includes:

[0165] S61. Define the reward function R(s t ,a t ) is used to evaluate the score of each state-action pair:

[0166] R(s t ,a t )=r goal -(α·dist(s t ,s goal )+β·risk(s t )+γ·dyn(s t ,a t ));

[0167] Among them, R(s t ,a t ) means in state s t Next, perform action a t The reward obtained, r goal Represents the reward value for reaching the target state, dist(s t ,s goal ) indicates the current state s t With the target state s goal The distance between risk(s t ) indicates the current state s t The risk value, dyn(s t ,a t ) indicates state s t Next, perform action a t The cost of ship dynamics characteristics;

[0168] S62, randomly initialize two neural networks Q and The weight parameter, is the target network, which has the same structure as Q, but the parameters are updated independently;

[0169] S63. Select action a based on ∈-greedy strategy t :

[0170]

[0171] in, represents the optimal action estimated by the current Q network;

[0172] S64. Perform action a in the environment t , observe the new state st+1 and reward R(s t ,a t );

[0173] S65、(s t ,a t ,R(s t ,a t ),s t+1 ) is stored in the experience replay buffer;

[0174] S66, randomly sampling small batches of data from the experience replay buffer, and updating the Q network parameters using the mean square error loss function;

[0175] S67, during the navigation process of the ship, the trained two-level neural network model is used to select the optimal action in real time. t , dynamically adjust the speed and course of the ship to ensure that the ship can berth and leave safely and accurately in complex environments.

[0176] Embodiment 1:

[0177] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a large container port, which receives a large number of international cargo ships berthing and unberthing every day. The traditional manual operation mode is not only inefficient, but also has a high safety risk, especially in bad weather and complex environment, the berthing and unberthing operation of ships becomes more difficult and dangerous. In order to improve the safety and efficiency of berthing and unberthing, the port decided to introduce an automatic berthing and unberthing system for ships based on video monitoring and image recognition.

[0178] In the main dock area of ​​the port, multiple high-definition cameras are deployed to cover the entire berthing and unberthing area. These cameras capture video data of ships and the surrounding environment in real time and transmit this data to the port's control center. The video data is first pre-processed, including denoising, contrast enhancement, and edge detection, to improve image quality. Then, the object detection model based on the EfficientDet algorithm analyzes the pre-processed images in real time to identify and locate ships, docks, other vessels and dynamic obstacles, and extract detailed target feature information.

[0179] These characteristic information are fused with the data from radar and inertial navigation systems, and the multi-sensor data is integrated through the Kalman filter to generate accurate environmental perception information. Based on these fused data, the depth information of obstacles is calculated using stereo vision technology to build and update a three-dimensional dynamic map of the ship's surroundings in real time.

[0180] In terms of path planning, the system uses the Rapidly Expanding Random Tree-Star (RRT*) algorithm. This algorithm comprehensively considers the ship's dynamic characteristics, environmental constraints and safety distance to calculate the optimal berthing or unberthing path. In actual operation, the system gradually covers the entire search space by continuously expanding nodes until a feasible path from the starting point to the end point is found. At the same time, combined with the double-level neural network (Double DQN) algorithm, the system uses real-time environmental perception information and target feature information to perform local dynamic obstacle avoidance, and adjusts the ship's speed and heading in real time to avoid potential collision risks.

[0181] Table 1 Comparison of experimental data between the present invention and the traditional method

[0182]

[0183] In this test, we conducted a detailed comparison between the traditional method and the automatic berthing and unberthing system of the present invention under three different environmental conditions, including sunny days and daytime, night and foggy weather, and bad weather (strong winds and heavy rain). Table 1 shows the detailed experimental data. Under the test conditions of sunny days and daytime, a total of 60 cargo ships performed berthing operations. Under the traditional method, the average time for each berthing operation is 45 minutes, and 1 accident occurred in these 60 operations. After using the automatic berthing and unberthing system of the present invention, the average berthing time was significantly reduced to 25 minutes, the efficiency was improved by 44%, and no accidents occurred. This shows that under standard operating conditions, the present invention not only greatly improves the operating efficiency, but also significantly improves safety.

[0184] At night and in foggy weather conditions, the average berthing time of the traditional method was 60 minutes, and 2 accidents occurred in 60 operations. After adopting the system of the present invention, the average berthing time was reduced to 35 minutes, and the efficiency was improved by 42%. During this period, the system of the present invention was still able to accurately identify the environment and avoid obstacles under low visibility conditions, ensuring the safety of each operation without any accidents. This result fully demonstrates the reliability and effectiveness of the system of the present invention under complex environmental conditions.

[0185] Under the most challenging severe weather conditions (such as strong winds and heavy rain), the average berthing time of the traditional method was 75 minutes, and 3 accidents occurred in 60 operations. In contrast, using the system of the present invention, the average berthing time was reduced to 40 minutes, the efficiency was improved by 47%, and no accidents occurred in all operations. This further proves the robustness and efficiency of the system of the present invention in extreme environments, and can ensure that ships can complete berthing operations safely and quickly in severe weather.

[0186] In general, the automatic berthing and unberthing system of the present invention performed well under various test conditions, significantly improving the efficiency and safety of berthing and unberthing. Compared with traditional methods, the present invention not only shortens the operation time and improves work efficiency, but also greatly reduces the accident rate during operation. These results fully verify the technical advantages and practical value of the present invention, and provide strong technical support for the intelligent and automated development of modern ports and docks.

[0187] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for automatic berthing and unberthing of ships based on video monitoring and image recognition, characterized in that: The steps include: S1. Deploy cameras around ships and docks to obtain real-time video data and perform image preprocessing, including denoising, contrast enhancement, and edge detection; S2. Use the target detection model based on the EfficientDet algorithm to perform real-time target detection on the preprocessed images, identify and locate ships, docks, other ships and dynamic obstacles, and extract target feature information; S3, using multimodal sensor fusion technology to fuse the real-time data of the camera, radar and inertial navigation system with the extracted target feature information, and integrate multi-sensor data through Kalman filtering to generate environmental perception information; S4. Based on the fused environmental perception information, the depth information of obstacles is calculated using stereo vision technology to construct and update a three-dimensional dynamic map of the ship's surroundings in real time; S5. Use the fast-expanding random tree-star algorithm for global path planning, comprehensively consider the ship's dynamic characteristics, environmental constraints and safety distance, and calculate the optimal berthing or unberthing path; S6, combined with a two-stage neural network algorithm, uses environmental perception information and target feature information to perform local dynamic obstacle avoidance and adjust the speed and heading of the ship in real time; S7, based on the global path planning results and the local dynamic obstacle avoidance strategy, the adaptive control system accurately adjusts the speed, heading and attitude of the ship according to the real-time environmental changes; S8. Use cloud computing platforms to conduct big data analysis and historical data mining to continuously optimize global path planning and local dynamic obstacle avoidance strategies.

2. The method for automatic berthing and unberthing of a ship based on video surveillance and image recognition according to claim 1, characterized in that: The S2 specifically includes: S21, input the preprocessed image data into the target detection model based on the EfficientDet algorithm for processing, and extract feature information in the image through a convolutional neural network; S22. In the EfficientDet model, a double-layer pyramid structure is used for multi-scale feature fusion, and feature maps of different scales are weighted fused: Among them, F out is the fused feature map, N1 represents the total number of scales of the feature map, and F i represents the feature map of the i-th scale, w i Represents the weight coefficient of the feature map of the i-th scale, satisfying S23. In the detection head of the EfficientDet model, the anchor box mechanism is used to perform step-by-step regression and classification on the feature map to obtain the three-dimensional position, scale and confidence score of the detection box; S24. The loss function of the target detection model consists of classification loss and bounding box regression loss. The total loss function is expressed as: Where N2 represents the total number of detected targets, p i represents the model's predicted classification probability for the i-th target, represents the true classification label of the i-th target, t i represents the model's predicted bounding box parameters for the i-th target, represents the true bounding box parameters of the i-th target, L represents the total loss function value, Represents the classification loss of the i-th target: represents the bounding box regression loss of the i-th object: Among them, t ij It indicates the value of the i-th target predicted by the model on the j-th parameter. Represents the true value of the i-th target on the j-th parameter; S25, further extracting feature information of the target according to the classification result and the detection frame information, including three-dimensional position (x, y, z), scale (w, h), confidence score a, speed v and direction θ, where x, y represent the center coordinates of the target in the image, z represents the height of the target, w represents the width of the target image, and h represents the length of the target image; S26, performing non-maximum suppression on the output of the target detection model, selecting the optimal detection frame according to the confidence score a, and eliminating overlapping detection frames to obtain the precise location, size, category and motion state information of the ship, dock, other ships and dynamic obstacles; S27. Obtain target feature information, and store the extracted target feature information in a feature vector f = [x, y, z, w, h, v, θ, a].

3. The method for automatic berthing and unberthing of a ship based on video surveillance and image recognition according to claim 1, characterized in that: The S3 specifically includes: S31, using the Kalman filter to take the real-time data of the camera, radar and inertial navigation system and the target feature information extracted by S27 as input, and perform multimodal sensor fusion, and the initial state vector is x0; S32, in the prediction stage of the Kalman filter, the state x at the next moment is predicted based on the dynamic model of the system k|k-1 and the covariance matrix P k|k-1 : x k|k-1 =Fx k-1 +Bu k-1 4 P k|k-1 =FP k-1 F T +Q; Where F represents the state transfer matrix, x k-1 represents the prior state vector, P k-1 represents the prior covariance matrix, B represents the control input matrix, u k-1 represents the control vector, Q represents the process noise covariance matrix; S33, in the update phase, using the sensor observation vector z k Update the state estimate and covariance matrix: y k =z k -Hx k|k-1 ; S k =HP k|k-1 H T +R; x k =x k|k-1 +K k y k ; P k =(I-K k H)P k|k-1 ; Among them, y k represents the measurement residual vector, H represents the observation matrix, S k represents the measurement residual covariance matrix, R represents the measurement noise covariance matrix, H T represents the transpose of the observation matrix, K k represents the Kalman gain matrix, represents the inverse of the measurement residual covariance matrix, x k represents the updated state vector, P k represents the updated covariance matrix, and I represents the identity matrix; S34, finally generating environmental perception information, including three-dimensional position, scale, confidence score, speed and direction.

4. The method for automatic berthing and unberthing of a ship based on video surveillance and image recognition according to claim 1, characterized in that: The S4 specifically includes: S41. Using stereo vision technology, the left and right images are obtained by using a binocular camera installed on the ship, and the feature points in the image pair are extracted and matched. The coordinates of the left image point are (u L ,v L ), the coordinates of the right image point are (u R ,v R ); S42, calculate the parallax δ of the feature point between the left and right images: S43, according to the parallax δ and the baseline distance B of the binocular camera, combined with the radial and tangential distortion parameters of the lens, calculate the depth Z of the feature point: Where f represents the focal length of the camera, k1 and k2 represent the radial distortion coefficients, and p1 and p2 represent the tangential distortion coefficients; S44. According to the depth Z of the feature point and the image point coordinates (u, v), the camera intrinsic parameter matrix is ​​used to convert the image coordinates into three-dimensional space coordinates (X, Y, Z): Among them, c x and c y represents the coordinates of the camera principal point, f x and f y Represents the scaling factor of the camera focal length in the x and y directions, r 2 =u 2 +v 2 ; S45, matching and fusing the calculated three-dimensional spatial coordinates with the environmental perception information fused by the multimodal sensor, and updating the three-dimensional position, scale, confidence score, speed and direction of the target; S46, constructing a three-dimensional dynamic map of the vessel's surroundings, including all detected targets and their precise locations, sizes, motion states, and category information; S47. Update the three-dimensional dynamic map in real time, and continuously adjust and optimize the three-dimensional environmental model based on the new data obtained by the sensor and the fused environmental perception information.

5. The method for automatic berthing and unberthing of a ship based on video surveillance and image recognition according to claim 1, characterized in that: The S5 specifically includes: S51, set the starting point of the global path planning to the current position of the ship (x0, y0, z0), and the end point to the target berthing position (x t ,y t ,z t ); S52, using the fast expanding random tree-star algorithm for global path planning, initializing a tree containing the starting point (x0, y0, z0), and gradually expanding the nodes to cover the entire search space until a feasible path from the starting point to the end point is found; S53. During the path extension process, the cost function is used to evaluate the path cost of each node: J(s i ,s j )=α·dist(s i ,s j )+β·risk(s j )+γ·dyn(s i ,s j ); Among them, J(s i ,s j ) represents node s i To node s j The path cost, α represents the weight coefficient of the distance cost, dist(s i ,s j ) represents node s i To node s j The distance between them, β represents the weight coefficient of risk cost, risk(s j ) represents node s j The risk value of the location, γ represents the weight coefficient of the dynamic cost, dyn(s i ,s j ) represents node s i To node s j The cost of ship dynamic characteristics; Among them, m represents the mass of the ship, a ij Represents node s i To node s j acceleration, J1 represents the ship's moment of inertia, ω ij Represents node s i To node s j angular velocity, η represents the hydrodynamic drag coefficient, F ij Represents node s i To node s j The thrust between ij Indicates the angle between the thrust direction and the heading; S54, expand the nodes of the tree, and after the path search is completed, the optimal path P from the starting point to the end point is obtained. t }, each node contains the status information of the ship; S55. The position and status information of each node in the optimal path is transmitted to the control system to guide the ship to navigate and adjust according to the path planning results.

6. The method for automatic berthing and unberthing of a ship based on video monitoring and image recognition according to claim 5, characterized in that: The S54 specifically includes: S541, randomly sample a state s rand ; S542, find the same as s in the tree rand The nearest nodes nearest ; S543, generate new node s new , so that new Under the control input, from s nearest To s rand Extensions; S544、Inspection new Is it within the obstacle range? If not, s new Add to the tree and update the path cost; S545. Repeat steps S541-S544.

7. The method for automatic berthing and unberthing of a ship based on video surveillance and image recognition according to claim 5, characterized in that: The S6 specifically includes: S61. Define the reward function R(s t ,a t ) is used to evaluate the score of each state-action pair: R(s t ,a t )=r goal -(α·dist(s t ,s goal )+β·risk(s t )+γ·dyn(s t ,a t )); Among them, R(s t ,a t ) means in state s t Next, perform action a t The reward obtained, r goal Represents the reward value for reaching the target state, dist(s t ,s goal ) indicates the current state s t With the target state s goal The distance between risk(s t ) indicates the current state s t The risk value, dyn(s t ,a t ) indicates state s t Next, perform action a t The cost of ship dynamics characteristics; S62, randomly initialize two neural networks Q and The weight parameter, is the target network, which has the same structure as Q, but the parameters are updated independently; S63. Select action a based on ∈-greedy strategy t : in, represents the optimal action estimated by the current Q network; S64. Perform action a in the environment t , observe the new state s t+1 and reward R(s t ,a t ); S65、(s t ,a t ,R(s t ,a t ),s t+1 ) is stored in the experience replay buffer; S66, randomly sampling small batches of data from the experience replay buffer, and updating the Q network parameters using the mean square error loss function; S67, during the navigation process of the ship, the trained two-level neural network model is used to select the optimal action in real time. t , dynamically adjust the ship's speed and heading.

Images