Unmanned ship autonomous following method based on combination of machine vision and reinforcement learning

Abstract

The invention relates to the technical field of underwater autonomous navigation, in particular to an unmanned ship autonomous following method based on combination of machine vision and reinforcement learning, and the method comprises the following steps: 1, carrying out the target recognition through employing YOLOv5; step 2, carrying out target trajectory tracking by using deepsort; 3, stabilizing the target position; and step 4, deep reinforcement learning (DRL) is used as an algorithm of the control module, and the algorithm of the control module is used as an algorithm of the control module. The method has the remarkable advantages of small volume and low cost, can be easily carried in a light-weight environment, and can also realize sufficient production by using a small amount of resources and put into use.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater autonomous navigation, and in particular to an autonomous following method for an unmanned ship based on the combination of machine vision and reinforcement learning. Background Art

[0002] At present, a Chinese patent discloses a method and device for detecting surface targets based on improved Yolov5, application number: 202310231374.3, which includes the following steps: 1) collect images, and construct a surface target set after data enhancement; 2) use Gaussian filter to smooth shallow feature maps to achieve image preprocessing; 3) embed a multi-angle attention mechanism in the Yolov5 network to make the extracted features richer and improve the resolution reduction and local information loss problems caused by ordinary methods; 4) feature scale conversion migration fusion Yolov5 network for feature processing; 5) use Distance-IoU loss method to calculate the loss, and finally calculate the weighted loss of the three losses to optimize the network and obtain more accurate results. This method combines computer vision technology and detection technology to improve the efficiency and accuracy of surface target detection, and avoids the problems of low detection accuracy, slow recognition speed, and unsatisfactory multi-target detection caused by existing methods. This technology can better realize the detection of surface targets, but it cannot cooperate with hardware to realize the tracking and control of unmanned ships. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides an autonomous following method for an unmanned ship based on the combination of machine vision and reinforcement learning.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions: an unmanned ship autonomous following method based on machine vision and reinforcement learning, comprising the following steps: Step 1: using YOLOv5 for target recognition;

[0005] Step 2: Use deepsort to track the target trajectory;

[0006] Step 3: The target position is stable;

[0007] Step 4: Use Deep Reinforcement Learning (DRL) as the control module algorithm.

[0008] Preferably, the specific steps of step 2 are: continuous tracking and target detection through deepsort, using a target detection algorithm to identify target objects in video frames, and providing bounding boxes and category information for each target object, and then extracting features for each detected target object through feature extraction, using a deep convolutional neural network to extract its appearance features, these features are used to calculate the similarity between target objects for target association and re-identification.

[0009] Preferably, in step 3, Kalman filtering is used to stabilize the target position.

[0010] Preferably, the state prediction equation of the Kalman filter is:

[0011]

[0012]

[0013] Among them, F is the system state transfer matrix and B is the control matrix.

[0014] The state update equation is:

[0015]

[0016]

[0017]

[0018] Among them, H is the measurement system parameter, Q is the process noise covariance matrix, and R is the measurement noise covariance matrix;

[0019] In the marine environment, the rolling and pitching of ships are mainly composed of simple harmonic motions; and the target ship's trajectory is a chaotic system and cannot be predicted; in order to filter the simple harmonic oscillation of the target position caused by the rolling and pitching of the follower ship, this technical solution predicts that the position of the target ship at the next moment will remain unchanged, that is, the state transfer matrix F is taken as:

[0020]

[0021] The influence of the following ship control signal on the roll and pitch cannot be predicted, so the control matrix B is taken as a second-order zero matrix:

[0022]

[0023] This technical solution directly filters the target ship position (x, y) obtained by YOLOv5, so the measurement system parameter matrix H takes the second-order unit matrix:

[0024]

[0025] Finally, according to the average sea conditions of the unmanned ship to which the technical solution is applicable, the values ​​of the process noise covariance matrix Q and the measurement noise covariance matrix R are adjusted to achieve the best filtering effect.

[0026] Preferably, in step 4, the neural network is trained in a simulation environment so that it can automatically follow the identified target ship, and after confirming that the algorithm is mature and stable, it is transplanted into the physical unmanned ship.

[0027] Preferably, in step 4, DDPG uses two target networks: one for the Actor's target strategy and one for the Critic's target value function. The parameters of these target networks are updated from the main network by soft update, that is, the target network parameters are slightly updated at each step;

[0028] The Critic network uses the Q-learning method to update the Q-value function to estimate the cumulative reward after performing a specific action;

[0029] The Actor network updates the strategy based on the gradient of the Critic network's output about the action, with the goal of maximizing the Critic network's estimate of the Actor's output action value;

[0030] DDPG adopts the Actor-Critic framework, with a total of four networks: the Actor network μ_θ(s) with parameter θ and input as state s; and the Critic network Q_ω(s,a) with parameter ω and input as state-action pair (s,a), as well as their respective target networks, namely the target Actor network μ_(θ^-)(s,a) with parameter θ^-, and the target Critic network Q_(ω^-)(s,a) with parameter ω^-.

[0031] The beneficial effects of the present invention are as follows: the target detection and tracking method using YOLOv5+DeepSort is used to obtain the information of the tracked ship by means of machine vision, which can avoid being detected by active / passive detection means of the target and can be applied in the military field; the position and trajectory of the tracked ship and the tracking ship posture are obtained from the image, providing a basis for the optimization and adjustment of the subsequent tracking path; compared with models such as Faster R-CNN, this method has excellent computing speed and can make a rapid response while ensuring a certain degree of accuracy.

[0032] Using the Kalman filter, the speed and direction of the tracking ship are adjusted in real time according to the target position and trajectory as well as the tracking ship's posture, so that the target is placed in the center of the field of view as much as possible and maintains a suitable distance to obtain better tracking capabilities.

[0033] This method has the significant advantages of small size and low cost. It can be easily carried in a lightweight environment, and can also be produced and put into use in sufficient quantities using a small amount of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a system block diagram of the present invention;

[0035] Figure 2 This is the annotation diagram of the YOLOv5 data set in the present invention;

[0036] Figure 3a and Figure 3b A target position stabilization diagram is achieved by Kalman filtering in the present invention;

[0037] Figure 4 It is the control module algorithm structure diagram of the present invention;

[0038] Figure 5 A diagram defining the reward mechanism in the present invention;

[0039] Figure 6 It is the generated image of the detection frame of the YOLO algorithm in the present invention;

[0040] Figure 7 This is the structure diagram of the YOLOv5 model in the present invention. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0042] Combine the following Figure 1 Describe the algorithm process and principles.

[0043] An autonomous following method for an unmanned ship based on machine vision and reinforcement learning comprises the following steps:

[0044] 1. Unmanned Ship Machine Vision

[0045] (1) Using YOLOv5 for target recognition

[0046] like Figure 6 and Figure 7 As shown in the figure, the YOLOv5 algorithm is a target detection algorithm based on deep learning, which is continuously updated from the YOLO (You Only Look Once) series. Its core idea is to regard target detection as a regression problem and directly predict the location and category of the target through a convolutional neural network. Compared with traditional target detection algorithms, YOLOv5 has higher detection speed and accuracy, and is particularly suitable for real-time target detection tasks.

[0047] The basic principle of YOLO is to extract and use the features of the entire image to simultaneously predict the bounding boxes of all categories of objects in the image. First, the input image is scaled to a fixed size; then it is input into a convolutional neural network for processing; and non-maximum suppression is performed to obtain the prediction result.

[0048] For an input picture, the neural network divides it into S × S grids. If the center point of an object falls within a grid, then the grid is responsible for detecting the object.

[0049] Each grid predicts B bounding boxes and their corresponding confidences. The confidence consists of two parts: the first part is the confidence that the bounding box predicted by the model contains the object; the second part is the prediction accuracy of the bounding box. The accuracy of the bounding box prediction is expressed by the intersection over union (IOU) between the predicted bounding box and the actual bounding box of the object. Therefore, the confidence of the bounding box is expressed as In this way, each bounding box actually contains 5 prediction values, x, y, w, h and confidence. x, y, w, h represent the horizontal and vertical coordinates of the center point of the bounding box and the width and height of the bounding box relative to the entire image.

[0050] Each grid also predicts the probability of C object categories, which is based on the conditional probability that the bounding box already contains the object. It is denoted as Pr(Class i |Object). The original YOLO algorithm only predicts the conditional probability of a set of object categories corresponding to one of the bounding boxes, even if it predicts B bounding boxes in a grid. After subsequent optimization, the conditional probability of each set of object categories corresponding to the B bounding boxes can be predicted separately.

[0051] Combining the category conditional probability and the confidence of the bounding box, we get the category probability confidence, expressed as It represents the probability of a certain type of object being contained in the bounding box and the prediction accuracy of the bounding box.

[0052] The prediction result will be expressed as a tensor with a size of (S×S×(5×B+C)). For the following picture, to evaluate the performance of YOLO on the PASCAL VOC dataset, assume that it is divided into 7×7 grids, each grid predicts 2 bounding boxes, and the PASCAL VOC dataset has 20 categories, so the tensor size of the prediction result is 7×7×30.

[0053] The problem of a target being detected multiple times is solved using the non-maximum suppression algorithm (NMS). First, select the detection box with the highest confidence from all the detection boxes of an object. Then calculate the IOU between it and a remaining detection box. If the IOU is greater than a certain threshold, it means that the overlap between the two detection boxes is too high, and the remaining detection box is removed. The above process is performed on each remaining detection box, and finally all the remaining detection boxes are the final result.

[0054] Compared with the previous version, YOLOv5 further improves the model performance and adds new features such as hyperparameter optimization, integrated experiment tracking, and automatic export to common export formats. The YOLOv5 model structure is shown in the figure below:

[0055] On the input side, YOLOv5 proposes some improvement ideas, enriches the data set, and improves the training and reasoning speed of the entire algorithm.

[0056] (1) Mosaic+RandomAffine+Mixup data enhancement: Mosaic randomly splices 4 images each time it runs. First, the coordinates of the splicing points of the 4 images are randomly generated within the range of 1280×1280 pixels; 4 images are randomly selected and embedded in the upper left, lower left, upper right, and lower right positions respectively; the parts that exceed the range are cropped; the remaining blank parts are filled with 114 pixel values. RandomAffine random affine transformation randomly translates, scales, cuts, and rotates the images spliced ​​by Mosaic, and restores them to the 640×640 size set by YOLOv5. Randomly select two images processed by Mosaic and RandomAffine for random mixing by Mixup. After the above processing, the capacity of the dataset can be greatly enriched.

[0057] (2) Adaptive anchor box calculation: When YOLOv5 is trained using a public dataset, it can calculate the best initial anchor based on the dataset used, without having to recalculate the initial anchor each time the dataset is changed.

[0058] In the Backtone module, some new ideas from other detection algorithms are integrated.

[0059] (1) Focus structure: The three-layer convolution downsampling layer is improved to the Focus structure, which uses alternate pixel sampling to slice the original 640×640, 3-channel image into a 320×320, 12-channel feature map, and then performs a convolution operation. Compared with the traditional downsampling method, the overall number of parameters and calculations is slightly reduced, the running speed is improved, and a feature map with no information loss is obtained.

[0060] (2) CSP structure: Use the CSP structure to divide the original input into two branches, so that the number of channels of the two branches is halved. After performing additional operations on one of the branches, it is input into the convolution, and the other branch is directly input into the convolution. Then the channels of the two branches are fused so that the number of output channels is restored to the same as the input. The purpose of using the CSP module is to enable the model to learn more feature values. There are two designs in YOLOv5, CSP1_X and CSP2_X. One of the branches of CSP1_X passes through the CBS module and X residual structures in succession before entering the convolution; one of the branches of CSP2_X passes through 1 CBS module and 2×X CBS modules in succession.

[0061] In the Neck module, the FPN+PAN structure is improved.

[0062] (1) The FPN structure transfers the semantic features of the high-level layer to the low-level layer through upsampling; the PAN structure transfers the positioning features from the low-level layer to the high-level layer while upsampling from the high-level layer to the low-level layer, and cascades the results of the fusion of the upper and lower adjacent layers after sampling. Combining the FPN and PAN structures can enhance both semantic features and positioning features.

[0063] In the Head output layer, the CIOU loss function is used, and the IOU threshold of NMS during detection box screening is improved to further improve the detection accuracy of the algorithm.

[0064] 1.DIOU: The calculation formula of CIOU loss function is as follows:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072] Among them, ρ represents the distance between the center point of the detection box and the center point of the object's true bounding box; c represents the diagonal length of the smallest rectangular box that can enclose the detection box and the true box. DIOU can reflect the loss when the true box completely contains the detection box.

[0073] 2.CIOU: CIOU is an optimization of DIOU, taking into account the aspect ratio of the detection box and the object's true bounding box. Its calculation formula is as follows:

[0074]

[0075]

[0076]

[0077] Where v represents the normalized difference between the length and width ratio of the detection box and the object's true bounding box; α represents the balance factor that weighs the length and width ratio loss and the IOU loss.

[0078] (2) Using deepsort to track target trajectories

[0079] Through deepsort, continuous tracking and target detection are performed. The target detection algorithm is used to identify the target objects in the video frame and provide bounding box and category information for each target object. Then, through feature extraction, a deep convolutional neural network is used to extract the appearance features of each detected target object. These features are used to calculate the similarity between target objects for target association and re-identification.

[0080] 2. Target position is stable

[0081] Due to interference factors such as wind and waves in the ocean, the hull of the follower ship will inevitably shake. The monocular camera is rigidly connected to the hull, and the shaking of the hull will cause noise in the position of the target ship in the field of view, which cannot accurately reflect the relative position relationship between the follower ship and the target ship. If the image is not processed, it will affect the decision of the control module. In addition, the recognition of the target detection module will also have errors, and the position of the target ship provided by it may jitter. Therefore, it is very important to stabilize the position of the target ship in the field of view.

[0082] This technical solution uses Kalman filtering to stabilize the target position. Kalman filter is a state estimator that uses sensor fusion and information fusion to improve the accuracy of the system. Kalman filtering can optimally combine the state transfer equation and the sensor measurement results, making the uncertainty of the estimated state less than either of them.

[0083] The state prediction equation of Kalman filter is:

[0084]

[0085]

[0086] Among them, F is the system state transfer matrix and B is the control matrix.

[0087] The state update equation is:

[0088]

[0089]

[0090]

[0091] Among them, H is the measurement system parameter, Q is the process noise covariance matrix, and R is the measurement noise covariance matrix.

[0092] In the marine environment, the rolling and pitching of ships are mainly composed of simple harmonic motion; and the trajectory of the target ship is a chaotic system and cannot be predicted. In order to filter the simple harmonic oscillation of the target position caused by the rolling and pitching of the follower ship, this technical solution predicts that the position of the target ship will remain unchanged at the next moment, that is, the state transfer matrix F is taken as:

[0093]

[0094] The influence of the following ship control signal on the roll and pitch cannot be predicted, so the control matrix B is taken as a second-order zero matrix:

[0095]

[0096] This technical solution directly filters the target ship position (x, y) obtained by YOLOv5, so the measurement system parameter matrix H takes the second-order unit matrix:

[0097]

[0098] Finally, according to the average sea conditions of the unmanned ship to which the technical solution is applicable, the values ​​of the process noise covariance matrix Q and the measurement noise covariance matrix R are adjusted to achieve the best filtering effect.

[0099] 3. Unmanned ship control module based on DDPG

[0100] This technical solution uses deep reinforcement learning (DRL) as the control module algorithm. By training the neural network in a simulation environment, it can automatically follow the identified target ship. After confirming that the algorithm is mature and stable, it is transplanted to the actual unmanned ship, reducing the cost of experiments and trial and error.

[0101] (1) DDPG algorithm control

[0102] Autonomous following of unmanned ships is a decision-making process based on continuous state space and continuous action space. For example, the speed and heading angle of the ship involve continuous action space. The DDPG algorithm introduces deterministic policy gradient based on the DQN algorithm to solve the problem of continuous action. Through deep neural networks, continuous actions can be directly output without discretizing the action space. The DDPG algorithm improves the sample utilization efficiency through technologies such as experience replay and target network, and performs efficient learning with a relatively small number of samples. The use of target network and delayed update strategy improves the stability of the algorithm, which can meet the requirements of system stability and reliability in practical applications.

[0103] (2) DDPG algorithm principle

[0104] DDPG is based on the Actor-Critic architecture, which includes an Actor network and a Critic network. The Actor network learns an approximate optimal policy, while the Critic network evaluates the value function of this policy. Unlike traditional reinforcement learning algorithms, DDPG learns deterministic policies, that is, it directly outputs a certain action in a given state, rather than an action distribution. DDPG uses Experience Replay to solve data correlation and stability problems. It stores the experience tuples (state, action, reward, next_state) of the agent's interaction with the environment in the replay buffer and randomly samples them for training.

[0105] To improve the stability of training, DDPG uses two target networks: one for the Actor's target policy and one for the Critic's target value function. The parameters of these target networks are updated from the main network through soft updates, that is, the target network parameters are slightly updated at each step.

[0106] The Critic network uses Q-learning to update the Q-value function to estimate the cumulative reward after performing a specific action.

[0107] The Actor network updates the strategy based on the gradient of the Critic network's output with respect to the action, with the goal of maximizing the Critic network's estimate of the Actor's output action value.

[0108] DDPG uses the Actor-Critic framework, with four networks: the Actor network μ_θ(s) with parameter θ and input state s; the Critic network Q_ω(s,a) with parameter ω and input state-action pair (s,a), and their respective target networks, namely the target Actor network μ_(θ^-)(s,a) with parameter θ^-, and the target Critic network Q_(ω^-)(s,a) with parameter ω^-.

[0109] (3) DDPG algorithm pseudo code:

[0110] Randomly initialize the critic network model Q(s,a|θ_Q) and the actor network model π(s|θ_π)

[0111] Construct the shadow critic network model Q'(s,a|θ_Q') and the actor network model π'(s|θ_π')

[0112] Initialize the size of the experience replay pool D to N

[0113]

[0114]

[0115] (4) Markov decision model

[0116] The Markov Decision Process (MDP) is a mathematical framework used to describe the decision-making process in reinforcement learning problems and how to trade off between different states and actions to achieve the maximum reward in the long run. For an agent, if its state at time t+1 is only related to the state at time t, and has nothing to do with the state and action at other times, then the state transition has Markov properties.

[0117] Described by a five-element array: D = [S, A, P, R, γ]

[0118] Among them, S is the state space; A is the action space; P is the state transition probability; R is the reward function; γ is the discount factor.

[0119] (5) State space design

[0120] The state space is carefully designed to improve the generalization ability of the model and reduce the interference of redundant information on the learning process. The state space consists of the motion information of the ship and the information of the target point. The motion information of the ship includes the information of the current position, while the information of the target point includes the distance between the current position and the target point. These two kinds of information are combined to form the state S_t at a moment.

[0121] (6) Action Space

[0122] During the navigation process of a ship, frequent changes in speed will cause damage to the engine, so tracking control is mainly performed by changing the heading angle. The heading change Δφ∈[-π / 2,π / 2] is used as the action space.

[0123] Embodiment 1:

[0124] like Figure 1 As shown in the figure, the unmanned boat and its auxiliary hardware equipment mainly consist of the hull, propulsion motor, steering gear, battery, camera, flight control, analog image transmission, Raspberry Pi and other auxiliary equipment. The camera is mainly used to record and reflect the motion state of the boat in real time, providing motion images for the decision-making of the control module. Raspberry Pi is used to run the control module program, receive the images taken by the camera, and perform real-time processing and image recognition. After the automatic control model based on reinforcement learning is deployed, it can control the rudder and motor to change the direction of the boat according to the input image, thereby realizing visual following.

[0125] like Figure 2 As shown in the figure, the data set comes from the motion pictures of the ship in the self-built simulation environment and the motion pictures of four physical unmanned ships. By extracting the video sequence frame by frame, a large amount of data can be quickly obtained. After the pictures are pre-processed by cropping and labeling, they are randomly allocated to the training set, verification set, and test set in a ratio of 6:2:2, and the production of the data set is completed. The work of making large-scale data sets is repetitive and time-consuming. In order to solve this problem and obtain better learning effects, a sufficient amount of data is first selected for manual labeling and directly put into training. The predicted results after training are regarded as the results of automatic labeling, and manual verification and adjustment are performed, and then put into training, forming a cycle of labeling-training-prediction-labeling adjustment; at the same time, try to use the existing data set for automatic labeling. The production of large-scale data sets is completed by combining manual and automatic methods.

[0126] like Figure 3a and Figure 3b As shown in Figure 3, the Kalman filter can stabilize the trajectory of an object by setting simple parameters. As shown in Figure 3, the blue line is the straight line and sine wave trajectory with two-dimensional Gaussian noise added, and the orange line is the trajectory after filtering. It can be seen that the Kalman filter has a good effect on filtering random disturbances in the target motion process.

[0127] like Figure 4As shown, the control of an unmanned ship usually involves a continuous action space, such as the speed and heading angle of the ship, etc. DDPG is suitable for dealing with such a continuous action space. Through a deep neural network, it can directly output continuous actions without discretizing the action space. The DDPG algorithm improves the sample utilization efficiency through techniques such as experience replay and target network, and conducts efficient learning with a relatively small number of samples. The use of the target network and the delayed update strategy improves the stability of the algorithm and can meet the requirements for system stability and reliability in practical applications.

[0128] As Figure 5 shown, the reward function determines the quality of the DRL effect. In this technical solution, a coordinate system is established with the horizon in the monocular camera's field of view as the X-axis and the line of sight as the Y-axis. Let the distance from the Y-axis to the right edge of the screen be 1, and the distance from the X-axis to the lower edge of the screen be 1. In the coordinate system, the area where -0.4 < x < 0.4 and -0.5 < y < -0.1 is set as Area 1, and the rest of the area is Area -1. Take the midpoint of the lower edge of the recognition box of the target ship as the judgment point, and make a judgment every 0.1 seconds: If the judgment point is in Area 1, then R(x, y) returns 1; if it is in Area -1, it returns -1; if the target ship's field of view is lost during the judgment, this judgment returns -10. This is the reward function R(x, y).

[0129] As described above, only the preferred specific implementation manners of the present invention are provided, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and all should be covered by the protection scope of the present invention.

Claims

1. An autonomous following method for an unmanned ship based on machine vision and reinforcement learning, characterized in that: The following steps are included: 1: Use YOLOv5 for target recognition; Step 2: Use deepsort to track the target trajectory; Step 3: The target position is stable; Step 4: Use Deep Reinforcement Learning (DRL) as the control module algorithm.

2. The unmanned ship autonomous following method based on machine vision and reinforcement learning according to claim 1 is characterized by: The specific steps of step 2 are: continuous tracking and target detection through deepsort, using the target detection algorithm to identify the target object in the video frame, and provide bounding box and category information for each target object, and then extract the appearance features of each detected target object through feature extraction using a deep convolutional neural network. These features are used to calculate the similarity between target objects for object association and re-identification.

3. The unmanned ship autonomous following method based on machine vision and reinforcement learning according to claim 1 is characterized by: In step 3, Kalman filtering is used to stabilize the target position.

4. The unmanned ship autonomous following method based on machine vision and reinforcement learning according to claim 3 is characterized by: The state prediction equation of Kalman filter is: Among them, F is the system state transfer matrix and B is the control matrix. The state update equation is: Among them, H is the measurement system parameter, Q is the process noise covariance matrix, and R is the measurement noise covariance matrix; In the marine environment, the rolling and pitching of ships are mainly composed of simple harmonic motions; and the target ship's trajectory is a chaotic system and cannot be predicted; in order to filter the simple harmonic oscillation of the target position caused by the rolling and pitching of the follower ship, this technical solution predicts that the position of the target ship at the next moment will remain unchanged, that is, the state transfer matrix F is taken as: The influence of the following ship control signal on the roll and pitch cannot be predicted, so the control matrix B is taken as a second-order zero matrix: This technical solution directly filters the target ship position (x, y) obtained by YOLOv5, so the measurement system parameter matrix H takes the second-order unit matrix: Finally, according to the average working sea conditions of the unmanned ship to which the technical solution is applicable, the values ​​of the process noise covariance matrix Q and the measurement noise covariance matrix R are adjusted to achieve the best filtering effect.

5. The unmanned ship autonomous following method based on machine vision and reinforcement learning according to claim 1 is characterized in that: In step 4, the neural network is trained in a simulation environment so that it can automatically follow the identified target ship. After confirming that the algorithm is mature and stable, it is transplanted into the actual unmanned ship.

6. The unmanned ship autonomous following method based on machine vision and reinforcement learning according to claim 1 is characterized by: In step 4, DDPG uses two target networks: one for the Actor’s target policy and one for the Critic’s target value function. The parameters of these target networks are updated from the main network through soft updates, that is, the target network parameters are slightly updated at each step; The Critic network uses the Q-learning method to update the Q-value function to estimate the cumulative reward after performing a specific action; The Actor network updates the strategy based on the gradient of the Critic network's output about the action, with the goal of maximizing the Critic network's estimate of the Actor's output action value; DDPG adopts the Actor-Critic framework, with a total of four networks: the Actor network μ_θ(s) with parameter θ and input as state s; and the Critic network Q_ω(s,a) with parameter ω and input as state-action pair (s,a), as well as their respective target networks, namely the target Actor network μ_(θ^-)(s,a) with parameter θ^-, and the target Critic network Q_(ω^-)(s,a) with parameter ω^-.

Images