A method and control system for autonomous cruise path planning of a surface-underwater dual-mode unmanned vehicle in the sea

Abstract

The application belongs to the technical field of marine cruise, and discloses a marine autonomous cruise path planning method and control system for a water surface and underwater dual-mode unmanned vehicle, which comprises the following steps: S101, training an Actor-Critc-based reinforcement learning network model; S102, planning a Dubins path based on the reinforcement learning network model according to situation information and target coordinates, and then controlling the vehicle to sail according to the Dubins path; S103, the Actor network outputs the optimal action of the vehicle according to the incomplete observation features obtained from the current environment, and then controls the vehicle to sail to a new state; S104, after the vehicle reaches the new state, the current incomplete observation features are completed to obtain complete features; meanwhile, a reward return is obtained; S105, the parameters of the Critic network are updated using the time difference error according to the complete features and the reward return; S106, the parameters of the Critic network are updated after the parameters are updated, and the parameters of the Actor network are updated, so that the application provides a marine autonomous cruise control method and system for a vehicle which can make fast and accurate decisions.

Description

Technical Field

[0001] The present invention belongs to the field of ocean cruising technology, and in particular relates to an ocean autonomous cruising path planning method and control system for a surface and underwater dual-mode unmanned vehicle. Background Art

[0002] With the increasing strategic importance of the ocean, large-scale ocean monitoring has become a top priority for every country with a coastline. Currently, the main methods for ocean reconnaissance and surveying available on the market include: regular manual patrols, helicopter aerial reconnaissance, and deployed underwater vehicle monitoring. These methods often suffer from high labor intensity, high risks, low operational efficiency, and small search ranges. To reduce the risks of maritime patrols and the intensity of manual work, a series of unmanned marine equipment, such as unmanned boats and ships, has begun to be applied in the marine military field. Many countries have begun to use unmanned boats and ships as the primary vehicles for maritime patrol and reconnaissance. The high degree of autonomy and flexibility of unmanned equipment can significantly reduce the labor intensity and risks of human intervention, and its implementation cost is also low, making it more economical and practical.

[0003] However, existing unmanned ocean reconnaissance and monitoring still have many defects in intelligence and cruising performance. For example, the cruise detection path still uses traditional preset routes; due to the limitation of battery capacity, the cruise time is short, the reconnaissance radius is small, and the exploration efficiency is low. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for controlling ocean autonomous cruising of a surface and underwater dual-mode unmanned vehicle that can make rapid and accurate decisions.

[0005] Based on the above purpose, the present invention adopts the following technical solutions:

[0006] A method for planning a path for autonomous ocean cruising of a surface and underwater dual-mode unmanned vehicle comprises the following steps:

[0007] S101, training an Actor-Critic-based reinforcement learning network model; the reinforcement learning network model includes an Actor network and a Critic network; the training process uses a random distillation network mechanism to reduce the sparsity of reward returns; in order to prevent strong correlation in time series data during training, an experience replay pool mechanism is introduced. During each training, a fixed amount of historical data is extracted from the experience replay pool and then added to the training process together with the existing data, disrupting the correlation between the previous and subsequent data, thereby improving the stability and generalization of the reinforcement learning network model. In order to ensure that the features input to the AC network are sequences of the same length, the variable homogeneous features obtained from the environment are encoded into fixed-size features during training, and a two-dimensional observation feature is compressed into a one-dimensional fixed-size feature using the maximum pooling principle; a feature embedding module based on the pooling layer is implemented to complete the incomplete information observed by the state observer.

[0008] S102, planning a Dobbins path based on a reinforcement learning network model based on the situation information and target coordinates, and then controlling the surface and underwater dual-mode unmanned vehicle to navigate along the Dobbins path; the situation information includes sea conditions, reefs, ocean depth, and satellite positioning information;

[0009] S103, while the surface and underwater dual-modal unmanned vehicle navigates along the Dobbins path, it obtains incomplete observation features and inputs them into the reinforcement learning network model. The Actor network outputs the optimal action of the surface and underwater dual-modal unmanned vehicle based on the incomplete observation features currently obtained from the environment, and then controls the surface and underwater dual-modal unmanned vehicle to navigate to a new state;

[0010] S104, after the surface and underwater dual-mode unmanned aerial vehicle reaches a new state, it obtains the current incomplete observation features again, and then completes the current incomplete observation features to obtain completed features; and at the same time obtains a reward return;

[0011] S105, based on the completed features and reward returns, the parameters of the critic network are updated using the temporal difference error;

[0012] S106, after the parameters are updated, the Critic network updates the parameters of the Actor network and repeats S101-S104.

[0013] Furthermore, in S104, the method for obtaining the completed feature is: obtaining the incomplete observation feature, then matching each information of the historical incomplete observation feature with the current incomplete observation feature, and then performing average pooling processing on the corresponding information of the historical and current incomplete observation features to obtain the completed feature.

[0014] Furthermore, in S104, the method for obtaining the reward return is: the random distillation network reward function reduces the sparsity of the basic reward return and increases the reward density to obtain the reward return; the paradigm of the random distillation network reward function is:

[0015] R(θ,s)=ψ(n)[R i (θ, s) + R e (θ, s)]

[0016] Where ψ(n) is the chain distillation reward gain; R i (θ, s) is the base reward; R e (θ, s) is the distillation network reward;

[0017] The Distillation Network reward paradigm is as follows:

[0018] R e (θ, s) = -βarctan(Δπ k (s, θ)

[0019] Where β>0 is the gain coefficient; k is the task type; Δπ k (s, θ) is the difference;

[0020] The paradigm of difference is:

[0021]

[0022] Among them, π k (s, θ) is the target strategy applied by the surface and underwater dual-mode unmanned vehicle when it is in situation information s while performing k tasks, A prediction strategy for a dual-mode surface and underwater unmanned vehicle in s situational information while performing k tasks;

[0023] The paradigm of chain distillation reward gain is:

[0024]

[0025] Where n is the number of consecutive decision nodes, N is a fixed consecutive node window, and L is a fixed gain.

[0026] Furthermore, the paradigm of basic rewards is:

[0027] R i (θ, s) = α(Δπ k (s, θ) 2

[0028] Among them, α>0 is the gain factor.

[0029] A processor capable of implementing the above-mentioned path planning method includes a path planning module, the path planning module including a reinforcement learning network model, a Dobbins path module, a perception module, a feature embedding block based on a pooling layer, and a reward return function module; the reinforcement network learning model includes an actor network module, a critic network module, and a critic update module; the perception module is used to obtain situation information and incomplete observation features; the Dobbins path module is used to plan a Dobbins path based on the reinforcement learning network model according to the situation information and target coordinates; the actor network module is used to output the optimal action of a surface and underwater dual-modal unmanned vehicle based on the incomplete observation features currently obtained from the environment; the feature embedding block based on the pooling layer is used to correspond each information of the historical incomplete observation features and the current incomplete observation features, and then average pooling the corresponding information of the historical and current incomplete observation features to obtain a complete feature; the reward return function module is used to reduce the sparsity of the basic reward return and increase the reward density to obtain a reward return; the critic update module is used to update the parameters of the critic network using a temporal difference error based on the complete feature and the reward return; and the critic network is used to update the parameters of the actor network.

[0030] A marine autonomous cruise control system for a surface and underwater dual-mode unmanned vehicle includes a processor of the above-mentioned path planning method and a perception module. The perception module is connected to an onboard computer, and the perception module is used to obtain situation information and transmit the situation information to the onboard computer. The onboard computer is used to determine whether the surface and underwater dual-mode unmanned vehicle is in a safe position based on the situation information, and to issue a surface-up command when the surface and underwater dual-mode unmanned vehicle is in a safe position, and to issue a dive command when the surface and underwater dual-mode unmanned vehicle is in a dangerous position; a buoyancy adjustment system is used to control the surface and underwater dual-mode unmanned vehicle to dive according to the dive command, and also to control the surface and underwater dual-mode unmanned vehicle to float according to the float command; and a mainsail system is used to provide power after the surface and underwater dual-mode unmanned vehicle floats.

[0031] Furthermore, the ocean autonomous cruise control system of the surface and underwater dual-mode unmanned vehicle also includes a buoyancy adjustment system connected to the onboard computer; the buoyancy adjustment system is used to control the surface and underwater dual-mode unmanned vehicle to dive according to the diving instruction, and is also used to control the surface and underwater dual-mode unmanned vehicle to float according to the floating instruction; the mainsail system is used to provide power after the surface and underwater dual-mode unmanned vehicle floats; the buoyancy adjustment system includes a gas ballast device arranged near the bow of the surface and underwater dual-mode unmanned vehicle and two liquid ballast devices arranged near the stern of the surface and underwater dual-mode unmanned vehicle. The ballast device comprises two sets of liquid ballast devices symmetrically arranged on both sides of the center line of the length direction of the surface and underwater dual-mode unmanned aerial vehicle; the gas ballast device comprises an air tank and an air bag with an air pump, the air tank and the air bag are connected by a hose, the air pump is used to fill the buoyancy gas in the air tank into the air bag, and can also compress the buoyancy gas in the air bag into the air tank; the liquid ballast device comprises a water tank and a ballast pump, the water tank and the ballast pump are connected by a hose, the inlet and outlet pipes of the ballast pump pass through the cabin of the surface and underwater dual-mode unmanned aerial vehicle and are connected to the outside, and are used to fill or drain water into the water tank through the ballast pump.

[0032] Furthermore, the mainsail system includes a cover plate, which is provided with a mast that can rotate around its central axis; a foldable sail is rotatably connected to the cover plate; the sail includes a mainsail board fixed to the mast, and a resistance board is rotatably connected to one side of the mainsail board, and the mainsail board is in the same direction as the mast; the mainsail board and the mast are both rotatably connected to the cover plate, and the mainsail board and the mast are rotatably arranged around the cover plate in a vertical plane; the mainsail board is rotatably connected to the cover plate in a vertical plane through a linkage device; the mainsail board and the resistance board are connected to the mainsail board through a belt transmission device, and the resistance board is rotated and folded along the belt transmission device toward the mainsail board, and the folded resistance board fits the board surface of the mainsail board; the belt transmission device includes a connecting plate, and both ends of the connecting plate are rotatably connected to brackets through transmission shafts, and the two brackets are respectively fixed to the mainsail board and the resistance board, and the two transmission shafts are respectively fixed to the brackets, and the two transmission shafts are connected through belt transmission; linkage The device includes a casing, which is a hollow cubic box structure. The casing is rotatably connected to the cover plate in a vertical plane through a flip mechanism, and a rotating mechanism for controlling the mast to rotate around its axis is also provided in the casing; the rotating mechanism is a rotating shaft coaxially arranged with the mast, the rotating shaft extends out of the casing and is fixedly connected to the mast, and the rotating shaft is connected to a rotating motor, which is arranged in the casing; a transition pulley is provided on the mast, and the transition pulley includes a cylindrical sleeve, and pulleys are provided at both ends of the sleeve, one of the pulleys is connected to an adjusting motor, and the adjusting motor is arranged in the casing of the linkage device, and the other pulley is connected to a belt transmission device; a groove is provided on the cover plate along the length direction of the surface and underwater dual-mode unmanned vehicle, the groove is a blind end at the bow end of the surface and underwater dual-mode unmanned vehicle, and an open end at the stern end of the surface and underwater dual-mode unmanned vehicle, and the sail is arranged in the groove.

[0033] Furthermore, the belt transmission device is used to rotate the resistance plate according to the diving command so that it can be folded with the main sail board; the air pump is used to compress the buoyancy gas in the air bag into the air tank according to the diving command, and the ballast pump is used to inject water into the water tank according to the diving command; the air pump is also used to fill the air bag with buoyancy gas according to the floating command, and the ballast pump is also used to drain the water in the water tank according to the floating command; the linkage device is used to drive the main sail board to erect after the dual-mode unmanned vehicle on the surface and underwater floats up; the belt transmission device is used to drive the resistance plate to unfold after the dual-mode unmanned vehicle on the surface and underwater floats up.

[0034] Furthermore, the ocean autonomous cruise control system for the surface and underwater dual-mode unmanned vehicle includes an energy module, which is connected to an onboard computer, and a path planning module is arranged in the onboard computer; the energy module and the onboard computer are connected to a motion controller, a communication module and a perception module; the motion controller and the energy module are connected to the mainsail system, the buoyancy adjustment system and the power propulsion device; the communication module is connected to the near-ground remote operation terminal and the satellite remote operation terminal; the energy module is used to provide power to the control system and convert solar energy into electrical energy after the surface and underwater dual-mode unmanned vehicle surfaces; the onboard computer is used to output navigation instructions according to the Dobins path and optimal action, and transmit the navigation instructions to the motion controller; the motion controller is used to control the mainsail system, the buoyancy adjustment system and the power propulsion device to operate according to the navigation instructions; the mainsail system is used to unfold or fold according to the navigation instructions; the buoyancy adjustment system is used to control the surface and underwater dual-mode unmanned vehicle to surface or dive according to the navigation instructions; the power propulsion device is used to provide power according to the navigation instructions and control the steering of the surface and underwater dual-mode unmanned vehicle; and the communication module is used to transmit data and instructions between the onboard computer and the near-ground remote operation terminal and the satellite remote operation terminal.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The autonomous cruise control system of the present invention efficiently integrates the internal functional modules of a surface and underwater dual-mode unmanned vehicle, innovatively applying an intelligent system to surface and underwater dual-mode unmanned vehicles. The method of the present invention is innovative in that it designs an improved reinforcement learning method, which enhances the intelligence and autonomy of the unmanned system. This method utilizes reinforcement learning networks and multi-sensor fusion technology. Compared with traditional intelligent autonomous navigation and path planning methods for surface and underwater dual-mode unmanned vehicles, this method can assess and make decisions about the surrounding environment. The autonomous decision-making is faster, more reasonable, and has a wider range of applicability. It is also suitable for autonomous path planning and cruising of surface and underwater dual-mode unmanned vehicles on the sea.

[0037] The autonomous cruise reconnaissance method of the present invention innovatively integrates an independently designed control system, which can flexibly switch the surface and underwater dual-mode unmanned aerial vehicle's surfacing patrol and reconnaissance function and the concealment and escape function, greatly improving the survivability and reconnaissance capability of the surface and underwater dual-mode unmanned aerial vehicle.

[0038] Observed features are often incomplete, lacking certain information due to sensor failures, environmental influences, and other factors. Directly inputting these incomplete features into the network can lead to unstable learning and poor learning results. Therefore, to ensure the effectiveness and accuracy of network learning, incomplete features need to be completed.

[0039] Because dual-modal surface and underwater unmanned vehicles often navigate complex, time-varying ocean environments, generating positive and effective reward signals is difficult. A random distillation network reward function mechanism, based on the resulting reward information, can enhance positive reward signals and filter out other ineffective reward signals. This method is used to "distill" the sparse rewards received by dual-modal surface and underwater unmanned vehicles, reducing their sparsity and increasing their density, significantly improving the guidance of reward signals for network training. This reward-reward method, based on a distillation network mechanism, makes it easier for reinforcement learning networks to learn the correct strategy during the training phase or usage phase, accelerating the network training process, reducing training time, and achieving better results.

[0040] The base reward in the reward return defines the fundamental goal of mission completion and is the ultimate learning outcome for the surface and underwater dual-modal unmanned vehicle. However, in complex environments, the base reward only provides sparse rewards in the early stages of training, making it difficult to generate effective and timely feedback. Therefore, a distilled network reward is used to guide the reinforcement learning network of the dual-modal unmanned surface and underwater vehicle. This increases the density of reward returns during training or use, and can guide the network to learn the correct strategy. The chained distillation reward gain is used to reward the network when it makes correct decisions at several consecutive decision points, ensuring the continuity of generating correct strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of a reinforcement learning network model according to Example 1 of the present invention;

[0042] Figure 2 This is a schematic diagram of the Dobbins path of Example 1 of the present invention;

[0043] Figure 3 This is a schematic diagram of the principle of obtaining the completion feature in Example 1 of the present invention.

[0044] Figure 4 This is a schematic diagram of a control system according to embodiment 3 of the present invention;

[0045] Figure 5 This is a working diagram of the surface and underwater dual-mode unmanned vehicle according to embodiment 5 of the present invention;

[0046] Figure 6 This is a schematic diagram of a surface and underwater dual-mode unmanned aerial vehicle patrolling and charging at sea according to embodiment 5 of the present invention;

[0047] Figure 7 This is a schematic diagram of a surface and underwater dual-mode unmanned aerial vehicle diving according to Example 5 of the present invention;

[0048] Figure 8 This is a schematic diagram of the surface and underwater dual-mode unmanned aerial vehicle surfacing according to Example 5 of the present invention. DETAILED DESCRIPTION

[0049] Example 1

[0050] A method for planning a path for autonomous ocean cruising of a surface and underwater dual-mode unmanned vehicle comprises the following steps:

[0051] S101, the parameters of the reinforcement learning network model are trained through course learning to train the reinforcement learning network model based on Actor-Critc; Figure 1 As shown, the reinforcement learning network model includes Actor network and Critic network;

[0052] The training process consists of two phases: 1. Pre-training using the experience pool. The experience replay pool does not store historical training results. Therefore, in the early stages of training, the AC network updates do not utilize the experience pool to break the correlation between previous and subsequent training. In the initial stages, reinforcement learning methods require exploration of the environment to search for high-value rewards, which is then used to train and update network parameters. However, in the early stages of task execution, due to a lack of environmental exploration, it is difficult for an agent to obtain sparse, high-value reward signals. This results in poor training performance for a long period of time in the early stages of training using traditional reinforcement learning methods, with rewards received from the environment remaining at a decreasing level. If the environment is complex and time-varying, the agent may become trapped in a local optimum during learning, or even experience algorithm non-convergence. To increase the density of effective reward signals in the early stages of training, accelerate the convergence of the reinforcement learning algorithm, and avoid being trapped in local optima, a reward function based on a random distillation network is used to provide the agent with positive and effective rewards at each training step during the early stages of training. A random distillation network is a network structure that, based on the reward reports it receives, enhances positive reward signals and filters out other ineffective reward signals. It is used to perform a "distillation" operation on the sparse rewards obtained by the dual-modal unmanned sailboat, reducing the sparsity of the rewards and increasing the density of the rewards, which can greatly improve the guidance of the reward return signal on network training.

[0053] 2. When the experience pool reaches its maximum capacity, each subsequent training step will randomly extract a fixed amount of historical experience from the experience pool and apply it to the current training. This breaks the correlation between previous and subsequent training sequences, improves the stability and generalization of the reinforcement learning network model, and prevents strong correlation between time series data. The historical experience in the experience pool primarily includes the agent's state, position, and velocity information at that moment. The experience replay pool is updated every 10,000 training steps to prevent overfitting to a uniform experience pool and improve training effectiveness. This phase of training continues until the reward value reaches its theoretical constant maximum, at which point the training phase ends.

[0054] The general training steps in the two training phases are the same: Initially, the reinforcement learning network parameters are given initial values. The reinforcement learning network is then applied to a dual-mode surface and underwater unmanned vehicle. The dual-mode surface and underwater unmanned vehicle performs tasks in a certain sea area. During training, certain hazards and obstacles are added to simulate normal cruising. The dual-mode surface and underwater unmanned vehicle then determines a reward function based on the task type. It first trains on similar tasks, then uses different tasks in the training process. Finally, the tasks are randomly combined, making the reinforcement learning network have strong generalization properties. The feature embedding block based on the pooling layer is used to complete and unify the observed feature sequence. Because poor signal or sensor equipment damage often occur during actual task execution, some observations cannot be received. For the reinforcement learning network, inputting observation sequences of inconsistent lengths into the learning network will reduce learning efficiency and speed. Therefore, the feature embedding block based on the pooling layer is used to process the feature sequence to improve the stability and efficiency of the reinforcement learning network training.

[0055] S102, such as Figure 2 As shown in the figure, a Dobbins path based on the reinforcement learning network model is planned according to the situation information and target coordinates, and then the surface and underwater dual-mode unmanned vehicle is controlled to navigate along the Dobbins path; the situation information includes sea conditions, reefs, ocean depth, satellite positioning information, etc.

[0056] S103, while the surface and underwater dual-mode unmanned vehicle navigates along the Dobbins path, it uses a state observer (the perception module of this embodiment uses a state observer) to obtain incomplete observation features and inputs them into the reinforcement learning network model. The pre-trained Actor network independently decides and outputs the optimal action of the surface and underwater dual-mode unmanned vehicle to the external environment based on the incomplete observation features currently obtained from the environment, so that the surface and underwater dual-mode unmanned vehicle navigates to a new state (the new state refers to the surface and underwater dual-mode unmanned vehicle obtaining a new position, speed, heading, new relative position, etc.). The observation features include the current position, speed, heading, information on the surface / underwater and surrounding obstacles of the surface and underwater dual-mode unmanned vehicle, the relative position from the target area, the battery power of the surface and underwater dual-mode unmanned vehicle, and other information. Incomplete observation features are observation features obtained due to incomplete observation of the vehicle's own position information due to poor signal, incomplete observation of dangerous obstacle information, or incomplete sensor information due to a sudden failure of a sensor. The optimal action includes the navigation acceleration generated by the propeller of the surface and underwater dual-mode unmanned vehicle and the angular velocity of the steering rudder.

[0057] S104, after the surface and underwater dual-mode unmanned vehicle reaches a new state, the state observer is used to obtain the current incomplete observation features again, such as Figure 3 As shown in the figure, to ensure consistent feature lengths in the AC network, the feature embedding block based on the pooling layer maps each piece of information from the historical incomplete observation features to the current incomplete observation features. It then performs average pooling on the information at each corresponding position of the historical and current incomplete observation features. This compresses multiple correlated feature sequences in a unified state into a complete feature sequence with complete data dimensions and a fixed size, thus obtaining the completed features. Compared to directly inputting sequences of inconsistent lengths into the reinforcement learning network, this achieves better learning results, greater robustness of the mature network, and faster convergence. A reward function is also used to obtain reward returns.

[0058] The reward return function includes a random distillation network reward function. In S104, the method for obtaining the reward return is: the random distillation network reward function reduces the sparsity of the basic reward return and increases the reward density to obtain the reward return.

[0059] The distillation network is a progressive guided reward method. Based on two identical network structures (including an input layer, 3 fully connected layers, and an output layer), one network is called the target policy network, which is used to output the policy π of the dual-modal unmanned surface and underwater dual-modal unmanned vehicle in each state. θ (s, a), whose network parameter is θ, which remains unchanged during the distillation process. The other network is called the prediction network π θ′(s, a), whose network parameter is θ′.

[0060] The paradigm for defining the reward function of the random distillation network is:

[0061] R(θ,s)=ψ(n)[R i (θ, s) + R e (θ, s)]

[0062] Among them, ψ(n) is the chain distillation reward gain. When the surface and underwater dual-modal unmanned aerial vehicle makes decisions in a series of n states, if the strategy it generates is the same as the target strategy, a reward gain is given, which improves the continuity of the reinforcement learning network in generating the correct strategy and greatly reduces the training time. i (θ, s) is the basic reward, which refers to the reward obtained by the surface and underwater dual-mode unmanned vehicle according to its state in the process of achieving the specified task; R e (θ, s) is the distillation network reward.

[0063] The paradigm for designing distillation network rewards is as follows:

[0064] R e (θ, s) = -βarctan(Δπ k (s, θ)

[0065] Where β>0 is the gain coefficient. When the difference is large, the reward tends to a very small constant value. The smaller the difference, the larger the reward. k is the type of task to be completed (reconnaissance, diving, etc.; k is used to distinguish task types, and different values ​​are given when performing different tasks; the value of k remains unchanged during the execution of each task); Δπ k (s, θ) is the difference; the paradigm of the difference is:

[0066]

[0067] Among them, π k (s, θ) is the target strategy applied by the surface and underwater dual-mode unmanned vehicle when it is in situation information s while performing k tasks, This is the prediction strategy applied by a surface and underwater dual-modal unmanned vehicle when it is in s situation information while performing k tasks. Its essence is probability density. When the difference between the two is large, it is considered that the strategy to be adopted is not consistent with the target strategy. Therefore, we need to guide the reinforcement learning network to learn towards the target network to reduce the difference.

[0068] The paradigm of chain distillation reward gain is:

[0069]

[0070] Where n is the number of continuous decision nodes, indicating that n continuous decision nodes are the same as the corresponding target decision node (a continuous decision point means: at time t-1, the agent makes a decision, generates an optimal action, inputs it into the environment, and enables the agent to reach the next new state; at time t, the agent makes another decision, generates an action, inputs it into the environment, and the agent reaches the next new state again.); N is a fixed continuous node window. During training, the window of length N moves forward one node every time a decision is made; L is a fixed gain.

[0071] Furthermore, the paradigm of basic rewards is:

[0072] R i (θ, s) = α(Δπ k (s, θ) 2

[0073] Among them, α>0 is the gain factor.

[0074] We introduced a distillation network reward in R(θ, s) to guide the reinforcement learning network to learn the correct strategy by reducing the difference between the predicted strategy and the target strategy. We then used the gradient descent principle to update the AC network parameters, making the network converge faster, significantly reducing training time, and improving training accuracy.

[0075] S105, based on the completed features and the reward generated by the last optimal action, the parameters of the critic network are updated using the temporal difference error;

[0076] S106, after the parameters are updated, the Critic network updates the parameters of the Actor network and repeats S101-S104.

[0077] Example 2

[0078] A microprocessor capable of implementing the path planning method of Example 1 includes a path planning module, the path planning module including a reinforcement learning network model, a Dobbins path module, a feature embedding block based on a pooling layer, a reward return function module and a perception module; the reinforcement network learning model includes an Actor network module, a Critic network module and a Critic update module; the Dobbins path module is used to plan a Dobbins path based on the reinforcement learning network model according to situation information and target coordinates; the Actor network module is used to output the optimal action of a surface and underwater dual-modal unmanned vehicle based on incomplete observation features currently obtained from the environment; the feature embedding block based on the pooling layer is used to correspond each information of historical incomplete observation features and current incomplete observation features, and then average pooling the corresponding information of historical and current incomplete observation features to obtain completed features; the reward return function module is used to reduce the sparsity of basic reward returns and increase reward density to obtain reward returns; the Critic update module is used to update the parameters of the Critic network using temporal difference error based on the completed features and reward returns; and the Critic network is used to update the parameters of the Actor network.

[0079] Example 3

[0080] A surface and underwater dual-mode unmanned vehicle ocean autonomous cruise control system, such as Figure 4 As shown, it mainly includes an energy module, the energy module is connected to an onboard computer, and the energy module and the onboard computer are connected to a motion controller, a communication module and a perception module; the motion controller and the energy module are connected to the mainsail system, the buoyancy adjustment system and the power propulsion device; the communication module is connected to the near-ground remote operation terminal and the satellite remote operation terminal; the energy module is used to provide power to the control system and convert solar energy into electrical energy after the surface and underwater dual-mode unmanned vehicle surfaces; the onboard computer is used to output navigation instructions according to the Dobins path and the optimal action, and transmit the navigation instructions to the motion controller; the motion controller is used to control the mainsail system, the buoyancy adjustment system and the power propulsion device to operate according to the navigation instructions; the mainsail system is used to unfold or fold according to the navigation instructions; the buoyancy adjustment system is used to control the surface and underwater dual-mode unmanned vehicle to surface or dive according to the navigation instructions; the power propulsion device is used to provide power according to the navigation instructions and control the steering of the surface and underwater dual-mode unmanned vehicle; the communication module is used to transmit data and instructions between the onboard computer and the near-ground remote operation terminal and the satellite remote operation terminal.

[0081] The energy module includes a solar energy collection panel, an energy conversion device, a battery pack and a voltage stabilization module. It is used for the surface and underwater dual-mode unmanned vehicle to collect solar energy and store it in the battery pack after it safely floats to the surface, and to power other modules to achieve long-term ocean cruising.

[0082] The communication module includes a networking module, a satellite communication module, and antenna equipment. For near-ground communication, wireless networking is used to realize data or command transmission, while for offshore data transmission, satellite communication is used. In addition, through the near-ground / satellite remote operation terminal, end-to-end control command transmission can be realized for surface and underwater dual-mode unmanned vehicles.

[0083] The onboard computer includes the microprocessor, remote operation interface, and bottom-level interaction module of Example 2. The path planning module is stored in the microprocessor. The perception module is used to obtain situation information and incomplete observation features and transmit them to the microprocessor; the situation information includes the surrounding sea conditions (sea conditions include wind, waves, currents, etc.), reefs, the ocean depth of the surface and underwater dual-mode unmanned aerial vehicle, satellite positioning information, etc. The microprocessor is used to transmit the situation information and incomplete observation features transmitted by the perception module to the path planning module. The path planning module is used to plan the navigation path based on the situation information and incomplete observation features and output the optimal action. The remote operation interface directly provides the user with an operational interface for remote control. The bottom-level interaction module is used to transmit data between top-level decision-making and bottom-level control.

[0084] The motion control module includes a central controller and a control drive module. The bottom-level interaction module directly decodes the upper-level decision instructions (optimal actions) to the lower-level central controller, and directly controls the mainsail system, buoyancy adjustment system and power propulsion device based on the driver through the bottom-level central controller.

[0085] The perception module includes a data acquisition module, a data processing module, and a data storage module. The data acquisition module contains an inertial navigation system, a GNSS positioning system, a lidar system, a high-definition camera system, and an altitude and horizontal sensor system. The data acquisition modules obtain the status information of the current dual-modal unmanned surface and underwater dual-modal unmanned vehicle's environment based on sensor information, and use multi-sensor fusion technology and SLAM mapping technology to directly provide real-time and accurate environmental mapping information to the upper-level airborne computer. The data processing module is used to perform data fusion, data mining, and data cognition on the data obtained by the acquisition module, and process the external environment status information into usable feature information based on the AC reinforcement learning network, so as to facilitate the airborne computer to realize decision-making functions. The data storage module is used to store the collected environmental information, communication data, and control instructions. The propulsion device includes a propeller and a steering rudder.

[0086] Example 4

[0087] The other parts of this embodiment are the same as Example 3, with the following differences: the perception module is used to obtain situation information and transmit the situation information to the onboard computer, the onboard computer is used to determine whether the surface and underwater dual-mode unmanned vehicle is in a safe position based on the situation information, and send a floating instruction to the buoyancy adjustment system when the surface and underwater dual-mode unmanned vehicle is in a safe position, and send a diving instruction to the buoyancy adjustment system when the surface and underwater dual-mode unmanned vehicle is in a dangerous position; the buoyancy adjustment system is used to control the surface and underwater dual-mode unmanned vehicle to dive according to the diving instruction, and is also used to control the surface and underwater dual-mode unmanned vehicle to float according to the floating instruction; the mainsail system is used to provide power after the surface and underwater dual-mode unmanned vehicle floats.

[0088] The buoyancy adjustment system includes a gas ballast device arranged near the bow of the surface and underwater dual-mode unmanned vehicle and two liquid ballast devices arranged near the stern of the surface and underwater dual-mode unmanned vehicle. The two liquid ballast devices are symmetrically arranged on both sides of the center line of the surface and underwater dual-mode unmanned vehicle in the length direction; the gas ballast device includes an air tank and an air bag with an air pump. The air tank and the air bag are connected by a hose. The air pump can fill the buoyancy gas in the air tank into the air bag, and can also compress the buoyancy gas in the air bag into the air tank; the liquid ballast device includes a water tank and a ballast pump. The water tank and the ballast pump are connected by a hose. The inlet and outlet pipes of the ballast pump pass through the cabin of the surface and underwater dual-mode unmanned vehicle and are connected to the outside. Water can be added to or discharged from the water tank through the ballast pump.

[0089] The mainsail system includes a cover plate, which is provided with a mast that can rotate around its central axis; the cover plate is rotatably connected to a foldable sail; the sail includes a mainsail board fixed to the mast, one side of the mainsail board is rotatably connected to a resistance board, and the mainsail board is in the same direction as the mast; the mainsail board and the mast are both rotatably connected to the cover plate, and the mainsail board and the mast are rotatably arranged around the cover plate in a vertical plane; the mainsail board is rotatably connected to the cover plate in a vertical plane through a linkage device; the mainsail board and the resistance board are connected by a belt transmission device, and the resistance board is rotated and folded along the belt transmission device toward the mainsail board, and the folded resistance board is fitted with the board surface of the mainsail board; the belt transmission device includes a connecting plate, both ends of the connecting plate are rotatably connected to a bracket through a transmission shaft, the two brackets are respectively fixed to the mainsail board and the resistance board, the two transmission shafts are respectively fixed to the brackets, and the two transmission shafts are connected by a belt transmission; the linkage device includes a connecting plate, The casing is a hollow cubic box structure, and the casing is connected to the cover plate for rotation in a vertical plane through a flip mechanism. A rotating mechanism for controlling the mast to rotate around its axis is also provided in the casing; the rotating mechanism is a rotating shaft arranged coaxially with the mast, and the rotating shaft extends out of the casing and is fixedly connected to the mast, and the rotating shaft is connected to a rotating motor, and the rotating motor is arranged in the casing; a transition pulley is provided on the mast, and the transition pulley includes a cylindrical sleeve, and pulleys are provided at both ends of the sleeve, one of the pulleys is connected to an adjusting motor, and the adjusting motor is arranged in the casing of the linkage device, and the other pulley is connected to the belt transmission device; a groove is provided on the cover plate along the length direction of the surface and underwater dual-mode unmanned vehicle, and the groove is a blind end at the bow end of the surface and underwater dual-mode unmanned vehicle and an open end at the stern end of the surface and underwater dual-mode unmanned vehicle, and the sail is arranged in the groove.

[0090] The belt drive device is used to rotate the resistance plate according to the diving command so that it can be folded with the main sail board; the air pump is used to compress the buoyancy gas in the air bag into the air tank according to the diving command, and the ballast pump is used to inject water into the water tank according to the diving command; the air pump is also used to fill the air bag with buoyancy gas according to the floating command, and the ballast pump is also used to drain the water in the water tank according to the floating command; the linkage device is used to drive the main sail board to erect after the dual-mode unmanned vehicle on the surface and underwater floats up; the belt drive device is used to drive the resistance plate to unfold after the dual-mode unmanned vehicle on the surface and underwater floats up.

[0091] Example 5

[0092] The control method of the surface and underwater dual-mode unmanned vehicle ocean autonomous cruise control system of embodiment 4 is as follows: Figure 5-8 As shown, the following steps are included:

[0093] S100, the onboard computer plans a navigation path (the path planning method of Example 1 may be used for path planning), and controls the surface and underwater dual-mode unmanned aerial vehicle to navigate according to the navigation path;

[0094] S200, evaluate whether the current surface and underwater dual-mode unmanned vehicle has reached the target area. If not, execute S100 to plan the target arrival path. The planned path takes into account the under-actuation of the unmanned surface and underwater dual-mode unmanned vehicle. The planned path is the Dobbins path, and then the dual-mode unmanned surface and underwater dual-mode unmanned vehicle is controlled to navigate. After the surface and underwater dual-mode unmanned vehicle reaches the target area, the onboard computer makes a threat judgment based on the obtained environmental situation information, and determines whether the surface and underwater dual-mode unmanned vehicle is If it is in a safe position, the mainsail system will be opened, surface patrols will be carried out, and energy will be replenished. If it is not in a safe area, the onboard computer will control the surface and underwater dual-mode unmanned vehicle to retract the mainsail system and control the surface and underwater dual-mode unmanned vehicle to dive. If it has dived to a safe position, the onboard computer will be used to plan the underwater gliding path and control the propeller movement to escape the dangerous area. After escaping the dangerous area, the surface and underwater dual-mode unmanned vehicle will be controlled to surface and carry out surface patrols and energy replenishment. The surface and underwater dual-mode unmanned vehicle can use the communication module to send the target area location to the ground receiving terminal, or temporarily store the collected data in the internal memory of the surface and underwater dual-mode unmanned vehicle until the patrol mission and energy replenishment mission are completed.

[0095] The diving process is:

[0096] When the S201 unmanned surface and underwater dual-mode unmanned vehicle is sailing on the surface, after receiving the diving command through the navigation and communication module, it rotates the resistance plate through the belt transmission device to fold it with the main sail plate, and the two plates fit together; it rotates the sail and stores it into the groove on the top surface of the cover plate.

[0097] S202 The bow and stern simultaneously compress the buoyancy gas in the airbag into the air tank through the air pump in the air tank, and fill water into the water tank through the ballast pump until the water surface submerges the entire cabin, at which time the ballast pump is turned off.

[0098] In S203, by continuing to compress the gas in the airbag in the bow, the center of gravity of the surface and underwater dual-mode unmanned vehicle will shift toward the bow, so that the entire surface and underwater dual-mode unmanned vehicle has a certain downward pitch angle, and the air pump is turned off at this time.

[0099] S204 starts the thrusters, and the surface and underwater dual-mode unmanned vehicle will dive quickly; at this time, the surface and underwater dual-mode unmanned vehicle enters the underwater mode.

[0100] The ascent steps include:

[0101] When the S205 unmanned surface and underwater dual-mode unmanned vehicle receives the ascent command through the navigation and communication module, the air pump in the air tank in the bow starts to work and fills the airbag with buoyancy gas. At this time, the center of gravity of the unmanned surface and underwater dual-mode unmanned vehicle will shift toward the stern of the surface and underwater dual-mode unmanned vehicle, so that the entire surface and underwater dual-mode unmanned vehicle has a certain upward pitch angle. At this time, the thruster is started and the unmanned surface and underwater dual-mode unmanned vehicle will quickly ascent.

[0102] S206 When the unmanned surface and underwater dual-mode unmanned vehicle approaches the water surface, the water in the water tank in the stern is emptied through the ballast pump. When the unmanned surface and underwater dual-mode unmanned vehicle reaches the water surface, the pitch angle of the unmanned surface and underwater dual-mode unmanned vehicle is adjusted to be horizontal, and all air pumps and ballast pumps stop working.

[0103] S207 After the surface and underwater dual-mode unmanned vehicle is stably moored on the water surface, the linkage device is started, the sail is erected, the transmission device is activated to deploy the resistance plate, communication is restored, and the solar panel resumes work to charge the lithium battery pack; at this time, the surface and underwater dual-mode unmanned vehicle switches to the surface mode again.

[0104] S300, after completing the surface patrol, the onboard computer begins to plan the return route and control the surface and underwater dual-mode unmanned vehicle to return.

Claims

1. A method for planning a path for autonomous ocean cruising of a dual-mode unmanned vehicle on the surface and underwater, characterized in that: The following steps are involved: S101, training based on Actor-Crit c reinforcement learning network model; S102, planning a Dobbins path based on a reinforcement learning network model according to the situation information and the target coordinates, and then controlling the surface and underwater dual-mode unmanned vehicle to navigate along the Dobbins path; S103, while the surface and underwater dual-modal unmanned vehicle navigates along the Dobbins path, it obtains incomplete observation features and inputs them into the reinforcement learning network model. The Actor network outputs the optimal action of the surface and underwater dual-modal unmanned vehicle based on the incomplete observation features currently obtained from the environment, and then controls the surface and underwater dual-modal unmanned vehicle to navigate to a new state; S104, after the surface and underwater dual-mode unmanned aerial vehicle reaches a new state, it obtains the current incomplete observation features again, and then completes the current incomplete observation features to obtain completed features; and at the same time obtains a reward return; S105, update Criti according to the completed features and reward returns c The parameters of the network; S106, the Critic network after updating the parameters updates the parameters of the Actor network, and repeats S101-S104; In S104, the method for obtaining the completed feature is: obtaining the incomplete observation feature, then matching each information of the historical incomplete observation feature with each information of the current incomplete observation feature, and then performing average pooling processing on the corresponding information of the historical and current incomplete observation features to obtain the completed feature; In S104, the method for obtaining the reward return is: the random distillation network reward function reduces the sparsity of the basic reward return and increases the reward density to obtain the reward return; the paradigm of the random distillation network reward function is: R(θ,s)=ψ(n)[R i (θ,s)+R e (θ,s)] Where ψ(n) is the chain distillation reward gain; R i (θ, s) is the base reward; R e (θ, s) is the distillation network reward; The Distillation Network reward paradigm is as follows: R e (θ,s)=-βarctan(Δπ k (s,θ)) Where β>0 is the gain coefficient; k is the task type; Δπ k (s, θ) is the difference; The paradigm of difference is: Among them, π k (s, θ) is the target strategy applied by the surface and underwater dual-mode unmanned vehicle when it is in situation information s while performing k tasks, A prediction strategy for a dual-mode surface and underwater unmanned vehicle in s situational information while performing k tasks; The paradigm of chain distillation reward gain is: Where n is the number of consecutive decision nodes, N is a fixed consecutive node window, and L is a fixed gain.

2. The method according to claim 1, wherein The basic reward format is: R i (θ,s)=α(Δπ k (s,θ)) 2 Among them, α>0 is the gain factor.

3. A processor capable of implementing the path planning method according to claim 1 or 2, characterized in that: The system comprises a path planning module, which includes a reinforcement learning network model, a Dobbins path module, a feature embedding block based on a pooling layer, and a reward return function module; the reinforcement learning network model includes an Actor network module, a Critic network module, and a Critic update module; the perception module is used to obtain situation information and incomplete observation features; the Dobbins path module is used to plan a Dobbins path based on the reinforcement learning network model according to the situation information and target coordinates; the Actor network module is used to output the optimal action of the surface and underwater dual-modal unmanned vehicle according to the incomplete observation features currently obtained from the environment; The feature embedding block based on the pooling layer is used to correspond each information of the historical incomplete observation features and the current incomplete observation features, and then average pool the corresponding information of the historical and current incomplete observation features to obtain the completed features; the reward return function module is used to reduce the sparsity of the basic reward return and increase the reward density to obtain the reward return; the critic update module is used to update the parameters of the critic network using the temporal difference error according to the completed features and reward returns; the critic network is used to update the parameters of the actor network.

4. A surface and underwater dual-mode unmanned vehicle ocean autonomous cruise control system, comprising a processor of the path planning method according to claim 3, characterized in that: It also includes a perception module, which is connected to an onboard computer, and the onboard computer is connected to a buoyancy adjustment system and a mainsail system; the perception module is used to obtain situation information and transmit the situation information to the onboard computer; the onboard computer is used to determine whether the surface and underwater dual-mode unmanned aircraft is in a safe position based on the situation information, and to issue a surface command when the surface and underwater dual-mode unmanned aircraft is in a safe position, and to issue a dive command when the surface and underwater dual-mode unmanned aircraft is in a dangerous position.

5. The surface and underwater dual-mode unmanned vehicle ocean autonomous cruise control system according to claim 4, characterized in that: It also includes a buoyancy adjustment system connected to the onboard computer; the buoyancy adjustment system is used to control the surface and underwater dual-mode unmanned vehicle to dive according to the diving instruction, and is also used to control the surface and underwater dual-mode unmanned vehicle to float according to the floating instruction; the mainsail system is used to provide power after the surface and underwater dual-mode unmanned vehicle floats; the buoyancy adjustment system includes a set of gas ballast devices arranged near the bow of the surface and underwater dual-mode unmanned vehicle and two sets of liquid ballast devices arranged near the stern of the surface and underwater dual-mode unmanned vehicle, and the two sets of liquid ballast devices are symmetrical It is arranged on both sides of the center line of the surface and underwater dual-mode unmanned aerial vehicle in the length direction; the gas ballast device includes an air tank with an air pump and an air bag, and the air tank and the air bag are connected by a hose. The air pump is used to fill the buoyancy gas in the air tank into the air bag, and is also used to compress the buoyancy gas in the air bag into the air tank; the liquid ballast device includes a water tank and a ballast pump, and the water tank and the ballast pump are connected by a hose. The inlet and outlet pipes of the ballast pump pass through the cabin of the surface and underwater dual-mode unmanned aerial vehicle and are connected to the outside, and are used to fill water or drain water into the water tank through the ballast pump.

6. The surface and underwater dual-mode unmanned vehicle ocean autonomous cruise control system according to claim 5, characterized in that: The mainsail system includes a cover plate, which is provided with a mast that can rotate around its central axis; a foldable sail is rotatably connected to the cover plate; the sail includes a mainsail board fixed to the mast, and a resistance board is rotatably connected to one side of the mainsail board, and the mainsail board is in the same direction as the mast; the mainsail board and the mast are both rotatably connected to the cover plate, and the mainsail board and the mast are rotatably arranged around the cover plate in a vertical plane; the mainsail board is rotatably connected to the cover plate in a vertical plane through a linkage device; the mainsail board is connected to the resistance board through a belt transmission device, and the resistance board is rotated and folded along the belt transmission device onto the mainsail board, and the folded resistance board is in contact with the board surface of the mainsail board; the belt transmission device includes a connecting plate, and both ends of the connecting plate are rotatably connected to brackets through a transmission shaft, and the two brackets are respectively fixed on the mainsail board. On the sailboard and the resistance board, the two transmission shafts are respectively fixed on the brackets, and the two transmission shafts are connected by belt transmission; the linkage device includes a casing, which is a hollow cubic box structure, and the casing is rotatably connected to the cover plate in a vertical plane through a flip mechanism, and a rotating mechanism for controlling the rotation of the mast around its axis is also provided in the casing; the rotating mechanism is a rotating shaft coaxially arranged with the mast, the rotating shaft extends out of the casing and is fixedly connected to the mast, the rotating shaft is connected to a rotating motor, and the rotating motor is arranged in the casing; a transition pulley is provided on the mast, and the transition pulley includes a cylindrical sleeve, and pulleys are provided at both ends of the sleeve, one of the pulleys is connected to an adjusting motor, the adjusting motor is arranged in the casing of the linkage device, and the other pulley is connected to the belt transmission device; A groove is provided on the cover plate along the length direction of the surface and underwater dual-mode unmanned vehicle. The groove is a blind end at the bow end of the surface and underwater dual-mode unmanned vehicle and an open end at the stern end of the surface and underwater dual-mode unmanned vehicle. The sail is arranged in the groove.

7. The surface and underwater dual-mode unmanned vehicle ocean autonomous cruise control system according to claim 6, characterized in that: The belt transmission device is used to rotate the resistance plate according to the diving command so that it can be folded with the main sail board; the air pump is used to compress the buoyancy gas in the air bag into the air tank according to the diving command, and the ballast pump is used to inject water into the water tank according to the diving command; the air pump is also used to fill the air bag with buoyancy gas according to the floating command, and the ballast pump is also used to drain the water in the water tank according to the floating command; the linkage device is used to drive the main sail board to stand up after the dual-mode unmanned vehicle on the surface and underwater floats up; the belt transmission device is used to drive the resistance plate to unfold after the dual-mode unmanned vehicle on the surface and underwater floats up.

8. The surface and underwater dual-mode unmanned vehicle ocean autonomous cruise control system according to claim 7, characterized in that: The vehicle also includes an energy module, which is connected to an onboard computer, and the path planning module is arranged in the onboard computer; the energy module and the onboard computer are connected to a motion controller and a communication module; the motion controller and the energy module are connected to the mainsail system, the buoyancy adjustment system and the power propulsion device; the communication module is connected to the near-ground remote operation terminal and the satellite remote operation terminal; the energy module is used to provide power to the control system and convert solar energy into electrical energy after the surface and underwater dual-mode unmanned vehicle surfaces; the onboard computer is used to output navigation instructions based on the Dobins path and the optimal action, and transmit the navigation instructions to the motion controller; the motion controller is used to control the mainsail system, the buoyancy adjustment system and the power propulsion device to operate in accordance with the navigation instructions; the mainsail system is used to unfold or fold according to the navigation instructions; the buoyancy adjustment system is used to control the surface and underwater dual-mode unmanned vehicle to surface or dive according to the navigation instructions; the power propulsion device is used to provide power according to the navigation instructions and control the steering of the surface and underwater dual-mode unmanned vehicle; and the communication module is used to transmit data and instructions between the onboard computer and the near-ground remote operation terminal and the satellite remote operation terminal.

Images