Underwater glider dynamic path planning training method based on value distribution reinforcement learning

Abstract

The application provides a kind of underwater glider dynamic path planning training method based on value distribution reinforcement learning, which can be applied to underwater glider technical field.The method comprises: inputting training data;Using at least one policy network, obtain the output action of each policy network at time t, the output action at time t+1;Using target evaluation network, obtain the target distribution at time t, the target distribution at time t represents the cumulative return prediction distribution based on quantile of underwater glider in sample state at time t based on policy network after executing action;Using evaluation network, obtain the sample evaluation distribution at time t;According to the sample evaluation distribution at time t and the target distribution at time t, obtain the updated evaluation network;Using the updated evaluation network, obtain the evaluation distribution at time t;According to the evaluation distribution at time t, obtain the target policy network, to control underwater glider to carry out path planning based on target policy network.

Description

Technical Field

[0001] This application relates to the field of underwater glider technology, and more specifically, to a dynamic path planning training method for underwater gliders based on value distribution reinforcement learning. Background Technology

[0002] Underwater gliders can glide in a sawtooth pattern across the ocean by adjusting their net buoyancy and attitude angles, in conjunction with the action of their wings. Each sawtooth motion (a dive and a surfacing motion) constitutes a cross-section. Through these sawtooth movements, underwater gliders can observe and probe the marine environment. For example, they can perform full-cycle tracking and observation of ocean temperature, salinity, and depth. However, due to the unpredictable ocean currents, the erratic movement of marine life, and the complex seabed topography, efficient path planning for underwater gliders is crucial. However, current path planning algorithms for underwater gliders are inefficient, requiring numerous iterations and resulting in slow real-time response. Summary of the Invention

[0003] In view of this, embodiments of this application provide a dynamic path planning training method for underwater gliders based on value distribution reinforcement learning.

[0004] One aspect of this application provides a training method for dynamic path planning of an underwater glider based on value distribution reinforcement learning, comprising: inputting training data, the training data including the underwater glider's sample state at time t, sample action at time t, and sample state at time t+1 obtained after executing the sample action at time t, where t is a positive integer; using at least one policy network to obtain the output action at time t of each policy network in the sample state at time t and the output action at time t+1 of the sample state at time t+1; using a target evaluation network to obtain the target distribution at time t based on the sample state at time t+1 and the output actions at time t+1 of each policy network in the sample state at time t+1, wherein the target distribution at time t represents the cumulative reward of the underwater glider after executing actions based on the policy network in the sample state at time t. The predicted distribution of quantiles is used, where quantiles represent the cumulative reward corresponding to the probability represented by the quantiles. Using the evaluation network, the evaluation distribution of samples at time t is obtained based on the sample state and sample action at time t. This evaluation distribution represents the actual distribution of the cumulative reward of the underwater glider after executing the sample action at time t based on the policy network in the sample state at time t, based on the quantiles. The evaluation network is updated based on the sample evaluation distribution and the target distribution at time t, resulting in an updated evaluation network. Using the updated evaluation network, the evaluation distribution at time t is obtained based on the sample state at time t and the output actions of each policy network in the sample state at time t. The parameters of each policy network are updated based on the evaluation distribution at time t to obtain the target policy network, which is used to control the underwater glider for path planning.

[0005] According to an embodiment of this application, the target evaluation network obtains the target distribution at time t based on the sample state at time t+1 and the output action at time t+1. The target distribution at time t represents the quantile-based predicted distribution of the cumulative reward of the underwater glider after performing the action based on the policy network in the sample state at time t. The evaluation network can obtain the sample evaluation distribution at time t, which represents the quantile-based distribution of the cumulative reward of the underwater glider after performing the sample action based on the policy network in the sample state at time t. Therefore, the output of the evaluation network is no longer the expectation of the cumulative reward. Thus, the target distribution at time t obtained by the target evaluation network and the sample evaluation distribution at time t obtained by the evaluation network can more intuitively reflect the evaluation of the sample action at time t. Updating the evaluation network accordingly can also enable the evaluation network to converge faster. Furthermore, the updated evaluation network yields the evaluation distribution at time t, and updating the parameters of each policy network based on the evaluation distribution at time t can also improve the convergence speed of the policy network and reduce the iteration of the algorithm. Attached Figure Description

[0006] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0007] Figure 1 An exemplary system architecture is shown that can be applied to the value distribution-based reinforcement learning-based underwater glider dynamic path planning training method of this application, according to an embodiment of this application.

[0008] Figure 2 A flowchart is shown of a dynamic path planning training method for underwater gliders based on value distribution reinforcement learning according to an embodiment of this application.

[0009] Figure 3 A schematic diagram showing the current heading angle and desired heading angle of an underwater glider according to an embodiment of this application is provided.

[0010] Figure 4 A schematic diagram of the motion space of an underwater glider according to an embodiment of this application is shown.

[0011] Figure 5 A schematic diagram of the network structure of a policy network according to an embodiment of this application is shown.

[0012] Figure 6 A schematic diagram of the network structure of the evaluation network according to an embodiment of this application is shown.

[0013] Figure 7 A schematic diagram illustrating the updating of the evaluation network according to an embodiment of this application is shown.

[0014] Figure 8 A schematic diagram illustrating the updating of a policy network according to an embodiment of this application is shown.

[0015] Figure 9 A schematic diagram of a target policy network performing a path planning task according to an embodiment of this application is shown.

[0016] Figure 10 The experimental results of the underwater glider dynamic path planning method according to an embodiment of this application in Experiment 1 are shown.

[0017] Figure 11 The experimental results of the underwater glider dynamic path planning method according to an embodiment of this application in Experiment 2 are shown.

[0018] Figure 12 A block diagram of a dynamic path planning training apparatus for an underwater glider based on value distribution reinforcement learning, according to an embodiment of this application, is shown.

[0019] Figure 13A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of this application, is shown. Detailed Implementation

[0020] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0021] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0022] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0023] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0024] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0025] In the embodiments of this application, the user's authorization or consent was obtained before obtaining or collecting the user's personal information.

[0026] Reasonable path planning can enable underwater gliders to reach the target location efficiently and accurately to begin observation tasks. Furthermore, reasonable path planning can help underwater gliders avoid dangerous shallow waters or areas with excessive ocean current interference, thereby reducing drift during navigation.

[0027] In related technologies, underwater glider path planning algorithms suffer from the following problems: The algorithms involve numerous iterations, resulting in slow computation speed and slow real-time response. The communication phase of the underwater glider is also time-consuming, increasing the risk of connection interruption between the glider and the control terminal. Reinforcement learning-based path planning algorithms require a large number of interaction samples, and they suffer from Q-value overestimation, leading to performance degradation and making them unsuitable for long-term observations. The optimal paths calculated by underwater glider path planning algorithms lack consideration for timeliness under long-range observation conditions; as the estimated time of the planned path increases, its timeliness significantly decreases. Furthermore, underwater gliders have low utilization rates of existing ocean information (topographic data, ocean current prediction information) during observation missions, and they face model uncertainties and unknown ocean current interference, preventing them from following predetermined paths.

[0028] This application provides a dynamic path planning training method for underwater gliders based on value distribution reinforcement learning. The method utilizes a target evaluation network to obtain the target distribution at time t based on the sample state and output action at time t+1. The target distribution at time t represents the quantile-based predicted distribution of the cumulative reward after the underwater glider performs an action based on the policy network in the sample state at time t. The evaluation network can also obtain the sample evaluation distribution at time t, which represents the quantile-based distribution of the cumulative reward after the underwater glider performs an action based on the policy network in the sample state at time t. Therefore, the output of the evaluation network is no longer the expected cumulative reward. Thus, the target distribution at time t obtained using the target evaluation network and the sample evaluation distribution at time t obtained using the evaluation network can more intuitively reflect the evaluation of the sample action at time t. Updating the evaluation network based on this distribution allows for faster convergence. Furthermore, the updated evaluation network yields the evaluation distribution at time t, and updating the parameters of each policy network based on this distribution also improves the convergence speed of the policy network and reduces algorithm iterations.

[0029] Figure 1 An exemplary system architecture for training a dynamic path planning method for underwater gliders based on value distribution reinforcement learning, according to embodiments of this application, is shown. It should be noted that... Figure 1The examples shown are merely examples of system architectures that can be applied to the embodiments of this application, in order to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include an underwater glider 101, a network 102, and a terminal device 103. The network 102 serves as a medium for providing a communication link between the underwater glider 101 and the terminal device 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] Terminal device 103 can acquire the state and actions of the underwater glider at various times through network 102, and use this as training data to execute the underwater glider dynamic path planning training method based on value distribution reinforcement learning according to the embodiments of this application. Terminal device 103 can also control the underwater glider to perform path planning based on the target policy network obtained by the underwater glider dynamic path planning training method based on value distribution reinforcement learning according to the embodiments of this application.

[0032] It should be understood that Figure 1 The number of underwater gliders, networks, and terminal devices shown is merely illustrative. Any number of underwater gliders, networks, and terminal devices can be included depending on the implementation requirements.

[0033] Figure 2 A flowchart is shown of a dynamic path planning training method for underwater gliders based on value distribution reinforcement learning according to an embodiment of this application.

[0034] like Figure 2 As shown, the underwater glider dynamic path planning training method based on value distribution reinforcement learning can include operations S210 to S270.

[0035] In operation S210, input training data.

[0036] The training data includes the underwater glider's sample state at time t, sample action at time t, and sample state at time t+1 obtained after performing the sample action at time t, where t is a positive integer.

[0037] In operation S220, using at least one policy network, the output action of each policy network at time t in the sample state at time t and the output action at time t+1 in the sample state at time t+1 are obtained.

[0038] In operation S230, the target evaluation network is used to obtain the target distribution at time t based on the sample state at time t+1 and the output actions of each policy network at time t+1 under the sample state at time t+1.

[0039] The target distribution at time t represents the cumulative reward of the underwater glider after performing actions based on the policy network in the sample state at time t. The predicted distribution is based on quantiles, where quantiles represent the cumulative reward corresponding to the probability represented by the quantiles.

[0040] In operation S240, using the evaluation network, the evaluation distribution of the sample at time t is obtained based on the sample state and sample action at time t.

[0041] The evaluation distribution of the sample at time t represents the cumulative return of the underwater glider after performing the sample action at time t based on the policy network under the sample state at time t, based on the actual distribution of quantiles.

[0042] In operation S250, the evaluation network is updated based on the sample evaluation distribution and the target distribution at time t, resulting in the updated evaluation network.

[0043] In operation S260, using the updated evaluation network, the evaluation distribution at time t is obtained based on the sample state at time t and the output actions of each policy network at time t under the sample state at time t.

[0044] In operation S270, the parameters of each policy network are updated according to the evaluation distribution at time t to obtain the target policy network, so as to control the underwater glider to perform path planning based on the target policy network.

[0045] Training data can be historical data of the underwater glider during its navigation in the marine environment. The sample state of the underwater glider at time t can be the state of its entry point at time t, which is also the state of its exit point at the previous time. The sample action at time t can be the action performed by the underwater glider after entering the water at time t. Each time point can correspond to a profile of the underwater glider. The state and action corresponding to each time point can be viewed as a state-action pair.

[0046] A policy network can be a network that determines the underwater glider's underwater actions based on its current state. There can be at least one policy network, and the parameters of each policy network can be different. Based on the parameters of each policy network, and according to the sample states at time t and time t+1, the corresponding output actions at time t and time t+1 can be obtained for each policy network, respectively.

[0047] The target evaluation network can be used to generate a target distribution, which is a quantile-based predicted distribution of the cumulative reward of an underwater glider after taking actions based on the policy network in the current sample state (e.g., time t). This distribution can be used to characterize the predicted evaluation of the actions taken in the current sample state. The target distribution can include multiple quantiles. For example, if the quantile is 20 (the 0.9 quantile), it means that 90% of the data is less than or equal to 20. The target evaluation network can obtain the target distribution for the current time step based on the sample state and output actions in the next time step (e.g., time t+1). Because the target distribution is based on quantiles, the output of the target evaluation network can be understood more intuitively and robustly.

[0048] Evaluation networks can be used to evaluate the actions of underwater gliders and obtain evaluation-based distributions. When the underwater glider's actions are sample actions, the evaluation network obtains a sample evaluation distribution based on the sample actions. When the underwater glider's actions are output actions obtained based on the policy network, the evaluation network obtains an evaluation distribution based on the output actions.

[0049] The target distribution at time t obtained from the target evaluation network can be used as the label, and the sample evaluation distribution at time t obtained from the evaluation network can be used as the result. The evaluation network is then updated based on the sample evaluation distribution at time t and the target distribution at time t, resulting in the updated evaluation network. The parameters of the target evaluation network and the evaluation network can be the same; therefore, the parameters of the target evaluation network can be softly updated based on the parameters of the evaluation network.

[0050] The evaluation distribution at time t can be obtained from the updated evaluation network. The evaluation distribution at time t is based on the output actions of each policy network at time t. Therefore, the evaluation distribution at time t reflects the quality of the output results of each policy network. Thus, the parameters of the policy network can be updated based on the evaluation distribution at time t to obtain the target policy network.

[0051] According to an embodiment of this application, the target evaluation network obtains the target distribution at time t based on the sample state at time t+1 and the output action at time t+1. The target distribution at time t represents the quantile-based predicted distribution of the cumulative reward of the underwater glider after performing the action based on the policy network in the sample state at time t. The evaluation network can obtain the sample evaluation distribution at time t, which represents the quantile-based distribution of the cumulative reward of the underwater glider after performing the sample action based on the policy network in the sample state at time t. Therefore, the output of the evaluation network is no longer the expectation of the cumulative reward. Thus, the target distribution at time t obtained by the target evaluation network and the sample evaluation distribution at time t obtained by the evaluation network can more intuitively reflect the evaluation of the sample action at time t. Updating the evaluation network accordingly can also enable the evaluation network to converge faster. Furthermore, the updated evaluation network yields the evaluation distribution at time t, and updating the parameters of each policy network based on the evaluation distribution at time t can also improve the convergence speed of the policy network and reduce the iteration of the algorithm.

[0052] In this embodiment of the application, dynamic path planning for the underwater glider can be achieved by constructing a Markov decision process. The construction of the Markov decision process can include the construction of the state space, action space, and reward function.

[0053] According to an embodiment of this application, the sample state of the underwater glider at each moment is determined in a pre-constructed sample state space, which may include terrain data, ocean current data, heading angle data, and the position data of the underwater glider after it emerges from the water.

[0054] The sample state space can be represented by the following formula (1).

[0055] (1);

[0056] in, Represents the sample state space. Represents terrain data, Represents ocean current data, This represents the ocean current velocity component data of the underwater glider at the detection point along the eastward direction, in an inertial coordinate system. This represents the ocean current velocity component data of the underwater glider at the detection point along the northward direction, in an inertial coordinate system. This represents the heading angle data. Position data of the underwater glider after it emerges from the water.

[0057] The terrain data can characterize the terrain data within the first rectangular sea area centered on the underwater glider. The terrain data is represented by a terrain height matrix composed of a first sub-element and a second sub-element. The first sub-element indicates that the absolute value of the terrain true value in the first rectangular sea area is less than or equal to the maximum diving depth of the underwater glider, and the second sub-element indicates that the terrain true value in the first rectangular sea area is less than the maximum diving depth of the underwater glider.

[0058] For example, each element in the terrain height matrix can represent the true value of the terrain corresponding to the coordinate point in the first rectangular sea area, dividing the first rectangular sea area into a grid. This represents the i-th row within the first rectangular sea area. This indicates that the j-th column in the first rectangular sea area can be accessed via... Represents the coordinates of the first rectangular sea area. The true value of the corresponding terrain.

[0059] The boundaries of the first rectangular sea area can be parallel to both longitude and latitude. Assuming the first rectangular sea area is 40km in both length and width, the coordinates within it can be represented by a point with an interval of approximately 0.417km along the longitude and latitude directions. , .

[0060] The first child element can be 1, and the second child element can be 0. The first child element and the second child element can be represented by the following formula (2).

[0061] (2);

[0062] in, This indicates the maximum diving depth of the underwater glider.

[0063] Ocean current data characterizes the ocean current data within the second rectangular sea area centered on the underwater glider. The ocean current data includes the ocean current velocity component data of the underwater glider at the detection point along the eastward direction in the inertial coordinate system and the ocean current velocity component data of the underwater glider at the detection point along the northward direction in the inertial coordinate system. The boundary of the second rectangular sea area is parallel to the longitude and latitude directions, respectively.

[0064] Ocean current data can be obtained through prediction. In an inertial coordinate system, the second rectangular sea area can be divided into grids. This represents the p-th row in the second rectangular sea area. This indicates the q-th column in the second rectangular sea area, where the underwater glider is at the detection point. Normalized ocean current velocity component data along the eastward direction can be obtained through It is stated that the underwater glider is at the detection point Normalized ocean current velocity component data along the northward direction can be obtained through Specifically, it can be represented by the following formula (3).

[0065] (3);

[0066] in, This indicates that the underwater glider is at the detection point in the inertial coordinate system. Eastward-directing ocean current velocity component data, This represents the minimum value of the eastward-directing ocean current velocity component within the second rectangular sea area. This represents the maximum value of the eastward-directing ocean current velocity component within the second rectangular sea area. This indicates that the underwater glider is at the detection point in the inertial coordinate system. Northward ocean current velocity component data, This represents the minimum value of the northward-directed ocean current velocity component within the second rectangular sea area. This represents the maximum value of the northward ocean current velocity component within the second rectangular sea area.

[0067] In ocean current data, the coordinates of the eastward-directing ocean current velocity component data of the underwater glider at the detection point in the inertial coordinate system can be the same as the corresponding coordinates of the northward-directing ocean current velocity component data of the underwater glider at the detection point in the inertial coordinate system. Both can represent the coordinates within the second rectangular sea area centered on the underwater glider. Taking a square sea area with a side length of 20km as an example, the coordinates can be the intersection of meridians and parallels spaced 5km apart eastward and northward. With both the length and width of the second rectangular sea area being 20km, the coordinates within the second rectangular sea area can be represented by coordinates spaced approximately 5km apart in the longitude and latitude directions. , .

[0068] Heading angle data can characterize the deviation between the current heading angle and the desired heading angle of an underwater glider. Both the current heading angle and the desired heading angle are the angles between the fuselage direction and the longitude direction of the underwater glider. The desired heading angle characterizes the target heading angle when the underwater glider is performing path planning.

[0069] The heading angle data can be expressed as the following formula (4).

[0070] (4);

[0071] in, This represents the heading angle data. Indicates the desired heading angle. Let t represent the heading angle at time t. In this embodiment, time t can represent the current time, that is, the heading angle at time t is the current heading angle of the underwater glider.

[0072] Figure 3 A schematic diagram showing the current heading angle and desired heading angle of an underwater glider according to an embodiment of this application is provided.

[0073] like Figure 3 As shown, the desired heading angle between the underwater glider's current position and the mission endpoint, and the desired heading angle based on the underwater glider's current position, is... The current heading angle corresponding to the current heading of the underwater glider based on its current position is: The deviation between the current heading angle and the desired heading angle is The heading angle is the angle between the underwater glider's heading (i.e., the direction of its fuselage) and the X-axis of the inertial coordinate system, which can be parallel to longitude. The mission endpoint is the position the underwater glider is expected to be at at the end of its gliding path planning mission.

[0074] Location data can characterize the distance between the current position of the underwater glider and the destination of the path planning task, as well as the relationship between the starting point and the destination of the path planning task.

[0075] Location data can be expressed as the following formula (5).

[0076] (5);

[0077] in, Represents location data, This indicates the distance between the current position of the underwater glider and the destination of the path planning task. This indicates the starting and ending points of the path planning task.

[0078] The sample motions and output motions of the underwater glider at each moment are determined in a pre-constructed sample motion space, which represents the heading angle adjustment of the underwater glider.

[0079] The sample action space can be expressed as the following formula (6).

[0080] (6);

[0081] in, Represents the sample action space. This represents the action of the underwater glider at time t. , Let represent the heading angle of the underwater glider at time t. This represents the heading angle of the underwater glider at time t+1.

[0082] Figure 4A schematic diagram of the motion space of an underwater glider according to an embodiment of this application is shown.

[0083] like Figure 4 As shown, at the current position of the underwater glider, the heading angle of the underwater glider at time t is... The expected heading can be the desired heading at time t+1, and the desired heading at time t+1 is... From the heading angle is Switch to heading angle The action is the action of the underwater glider at time t. The expected heading at time t+1 can be the predicted heading of the next profile of the underwater glider.

[0084] Furthermore, a reward function for the underwater glider can be constructed. The reward function can be defined based on the heading angle data and position data in the sample state space, and can be expressed as the following formula (7).

[0085] (7);

[0086] Where R represents the reward obtained based on the reward function.

[0087] Figure 5 A schematic diagram of the network structure of a policy network according to an embodiment of this application is shown.

[0088] like Figure 5 As shown, a policy network can be constructed, which includes a policy ocean current feature convolutional layer, a policy terrain feature convolutional layer, a policy feature splicing layer, and a policy fully connected layer.

[0089] The policy network can obtain the output action under the sample state at each time step through the following operations: process ocean current data using a policy ocean current feature convolutional layer to obtain policy ocean current features; process terrain data using a policy terrain feature convolutional layer to obtain policy terrain features; concatenate policy ocean current features, policy terrain features, heading angle data, and position data using a policy feature splicing layer to obtain policy mixed features; and process the policy mixed features using a policy fully connected layer to obtain the output action under the sample state at each time step.

[0090] The input to the policy network is the sample state s of the underwater glider at time t. t The output of the policy network is the output action corresponding to the sample state.

[0091] The strategy current feature convolutional layer is used to extract features from ocean current data. The input is ocean current data, and the output is strategy current features. Taking a second rectangular sea area with a length and width of 20km and a coordinate point in the second rectangular sea area with an interval of approximately 5km in the longitude and latitude directions as an example, the ocean current data can be input into the strategy current feature convolutional layer as two 5×5 input matrices. One input matrix represents the matrix corresponding to the ocean current velocity component data of the underwater glider at the detection point along the eastward direction in the inertial coordinate system, and the other input matrix represents the matrix corresponding to the ocean current velocity component data of the underwater glider at the detection point along the northward direction in the inertial coordinate system.

[0092] The strategy ocean current feature convolutional layer can include two convolutional layers connected in sequence. The input of the first convolutional layer is ocean current data, and the output of the second convolutional layer is ocean current features. The first convolutional layer can include four 3×3 convolutional kernels, and the second convolutional layer can also include four 3×3 convolutional kernels.

[0093] The strategy-based terrain feature convolutional layer is used to extract features from terrain data. The input is terrain data, and the output is strategy-based terrain features. Taking a first rectangular sea area with a length and width of 40km and coordinates within that area spaced approximately 0.417km apart in both longitude and latitude directions, the terrain data can be input as a 97×97 matrix into the strategy-based terrain feature convolutional layer. This layer can consist of two convolutional layers connected in sequence. The first convolutional layer takes terrain data as input, and the second layer outputs terrain features. The first layer may contain eight 10×10 convolutional kernels, and the second layer may contain sixteen 10×10 convolutional kernels.

[0094] The strategy feature stitching layer is used to stitch together strategy ocean current features, strategy topographic features, heading angle data and position data. The output of the strategy feature stitching layer is a strategy hybrid feature, and the dimension of the strategy hybrid feature can be 1×70.

[0095] A policy-fully connected layer can be a fully connected neural network. The input to the policy-fully connected layer is the policy mixture features, and the output is the action 'a' given the sample state. The policy-fully connected layer can consist of an input layer, a hidden layer, and an output layer connected sequentially. The input layer can have 70 neurons, the hidden layer can have 256 neurons, and the output layer can have 1 neuron.

[0096] Figure 6 A schematic diagram of the network structure of the evaluation network according to an embodiment of this application is shown.

[0097] like Figure 6As shown, an evaluation network can be constructed, which may include a convolutional layer for evaluating ocean current features, a convolutional layer for evaluating topographic features, a feature splicing layer for evaluating features, and a fully connected layer for evaluating features.

[0098] The evaluation network obtains the sample distribution under the sample state at each time step through the following operations: processing ocean current data using an evaluation ocean current feature convolutional layer to obtain evaluation ocean current features; processing terrain data using an evaluation terrain feature convolutional layer to obtain evaluation terrain features; splicing evaluation ocean current features, evaluation terrain features, heading angle data, position data, and corresponding sample actions under the sample state at each time step using an evaluation feature splicing layer to obtain evaluation mixed features; and processing the evaluation mixed features using an evaluation fully connected layer to obtain the sample distribution under the sample state at each time step.

[0099] The evaluation network can include convolutional layers for evaluating ocean current features, convolutional layers for evaluating terrain features, feature splicing layers, and fully connected layers. The input to the evaluation network can be the current sample state of the underwater glider and its corresponding sample action or output action. The output of the evaluation network can be the sample distribution corresponding to the sample action or the output distribution corresponding to the output action.

[0100] The structures of the ocean current feature evaluation convolutional layer and the topography feature evaluation convolutional layer can be the same as those of the policy ocean current feature convolutional layer and the policy topography feature convolutional layer in the policy network, and will not be elaborated here.

[0101] The evaluation feature stitching layer is used to stitch together evaluation ocean current features, evaluation topographic features, heading angle data, position data, and actions corresponding to the sample state. The output of the evaluation feature stitching layer is the evaluation mixed feature, and the dimension of the evaluation mixed feature can be 1×71.

[0102] The evaluation of a fully connected layer can be a fully connected neural network. The input to the evaluation of the fully connected layer is the evaluation mixture features, and the output of the evaluation of the fully connected layer is the distribution of actions corresponding to the sample states. The evaluation of the fully connected layer can include an input layer, a hidden layer, and an output layer connected in sequence. The number of neurons in the input layer can be 71, the number of neurons in the hidden layer can be 256, and the number of neurons in the output layer can be 25.

[0103] According to an embodiment of this application, training data can be obtained through the following operations: randomly selecting a region of a preset area from the marine environment to determine the task area; acquiring ocean current task data and terrain task data in the task area; determining the task start point and task end point based on the longitude and latitude of the task area; acquiring the sample state of the underwater glider at time t based on the task start point, ocean current task data, and terrain task data; obtaining the sample action at time t using a policy network based on the sample state at time t; and obtaining the sample state of the underwater glider at time t+1 after the underwater glider executes the sample action at time t.

[0104] A pre-defined ocean area of ​​100km × 100km can be randomly selected from the marine environment as the mission area.

[0105] Acquire ocean current and terrain data corresponding to latitude and longitude coordinates within the task area. Ocean current data may include, for example, the eastward and northward velocity components of the ocean currents. Terrain data may include the seabed height values ​​(i.e., the height from sea level to seabed, which is a negative value) corresponding to latitude and longitude coordinates within the task area.

[0106] Two coordinate points can be randomly selected within the task area based on its longitude and latitude, serving as the task start and end points respectively. The coordinates of the task end point can be... The coordinates of the mission endpoint can be .

[0107] Taking the mission starting point as the current position of the underwater glider, the sample state at the first moment is obtained. , ,in, This represents ocean current data at the first moment. This represents the terrain data at the first moment. This represents the heading angle data at the first moment. This represents the location data at the first moment. It can be obtained by inputting the sample state at the first moment into an untrained policy network. Output the sample action at the first moment. Because the state space and action space are limited, even though the policy network has not been trained, the output sample actions are still within the action space.

[0108] After the underwater glider performs the first-time sample action, the second-time sample state can be obtained. And calculate the first-time sample reward obtained by performing the first-time sample action under the first-time sample. The first sample's state, action, state, and reward are taken as the first sample. .

[0109] If the current task ends, a new start and end point can be selected to obtain new samples. If the task has not ended, samples can be obtained from the next time step. The training data consists of a set of samples C, where each sample is a sample set C. P can be equal to 2,000,000.

[0110] According to embodiments of this application, the evaluation network may include N, where N is an integer greater than or equal to 1. The 1st, ..., nth, ..., Nth evaluation networks can be represented as follows: The parameters corresponding to the 1st, ..., nth, ..., Nth evaluation networks can be expressed as follows: , 1 ≤ n ≤ N, and n is an integer. The output of each evaluation network can be a distribution based on K quantiles, where K is a positive integer. There can be N target evaluation networks. The 1st, ..., nth, ..., Nth target evaluation networks can be represented as follows: The architecture of the target evaluation network and the evaluation network can be the same. The parameters of the 1st, ..., nth, ..., Nth evaluation networks can be used. The network parameters are evaluated for the 1st, ..., nth, ..., Nth targets respectively. During initialization, the output of each target evaluation network can be a target distribution represented by K quantiles.

[0111] The policy network can include M, the 1st, ..., the mth, ..., the Mth policy network can be represented as follows: The parameters of the 1st, ..., mth, ..., Mth policy networks can be expressed as follows: 1 ≤ m ≤ M and m is an integer. In the embodiments of this application, N and M can be 5, and K can be 25.

[0112] A certain number of samples can be randomly selected from the sample set as training data, for example, 256 samples can be selected as training data.

[0113] According to an embodiment of this application, the loss function of the evaluation network can be defined as minimizing the 1-Wasserstein distance between the sample distribution of each evaluation network and the target distribution of the target evaluation network, which can be expressed as the following formula (8).

[0114] (8);

[0115] in, Let n be the loss function of the nth evaluation network. This represents the 1-Wasserstein distance between the distribution of the nth evaluation network output and the target distribution. A state-action pair can be a state-action pair input to the nth evaluation network. The 1-Wasserstein distance can be represented by the following formula (9).

[0116] (9);

[0117] in, Let be the k-th quantile in the distribution of the nth evaluation network output, where 1 ≤ k ≤ K and k is an integer. The distribution of the nth evaluation network output can include K quantiles, which can be expressed as: , Let be the i-th quantile in the target distribution, which includes NK quantiles. The target distribution can be represented as: . This indicates Huber's quantile loss. ,in, Let k represent the probability represented by the k-th quantile. , For Hubel loss with parameter 1, .

[0118] According to an embodiment of this application, using a target evaluation network, the target distribution at time t is obtained based on the sample state at time t+1 and the output actions of each policy network at time t+1 based on the sample state at time t+1. This includes: using at least one target evaluation network, based on the sample state at time t+1 and the output actions of each policy network at time t+1 based on the sample state at time t+1, obtaining the target output distributions at time t+1 corresponding to each policy network and at least one target evaluation network respectively; sorting the target output distributions at time t+1 corresponding to each policy network and at least one target evaluation network respectively to obtain a mixed distribution of target outputs at time t+1 corresponding to each policy network; and averaging the mixed distributions of target outputs at time t+1 corresponding to each policy network to obtain the target distribution at time t.

[0119] Figure 7 A schematic diagram illustrating the updating of the evaluation network according to an embodiment of this application is shown.

[0120] like Figure 7 As shown, for any sample in the sample data, the sample state at time t+1 is... As the m-th policy network The input is based on the parameters of the m-th policy network. Get the output action at time t+1. The first policy network and the Mth policy network The processing procedures for each can be referenced from the m-th policy network. The process will not be elaborated here.

[0121] Based on the parameters of the m-th policy network, the sample state at time t+1 and the output action at time t+1 are... As input to each of the N target evaluation networks, the target output distribution at time t+1 corresponding to each of the N target evaluation networks is obtained. Among them, the target output distribution at time t+1 corresponding to the nth target evaluation network is... The corresponding target output distribution at time t+1 is represented as follows: The target output distribution at time t+1 corresponding to each target evaluation network includes K quantiles. The target output distributions at time t+1 corresponding to each of the N target evaluation networks are then compared. The} are concatenated to obtain the concatenated output distribution at time t+1 based on the NK quantile representations of the m-th policy network. It can be expressed as the following formula (10).

[0122] (10);

[0123] in, It represents the kth quantile in the spliced ​​distribution of the output of the nth objective evaluation network at time t+1, under the parameters of the mth policy network.

[0124] Then, the quantiles in the spliced ​​distribution of the output at time t+1 are sorted from smallest to largest to obtain the mixed distribution of the target output at time t+1 based on the m-th policy network, which is expressed as the following formula (11).

[0125] (11);

[0126] in, This represents the mixed distribution of the target output at time t+1 based on the m-th policy network. Describes the m-th policy network at time t+1 in the mixed distribution of the target output. quantiles.

[0127] The mixed distribution of target outputs at time t+1 corresponding to each of the M policy networks can be averaged to make the output of the target evaluation network averaged. The averaged result is then processed to obtain the target distribution at time t. By averaging the mixed distribution of target outputs at time t+1 corresponding to each of the multiple policy networks, the target distribution at time t can be made more balanced.

[0128] According to an embodiment of this application, averaging the mixed distribution of target outputs at time t+1 corresponding to each policy network to obtain the target distribution at time t may include: averaging the mixed distribution of target outputs at time t+1 corresponding to each policy network to obtain the average distribution of target outputs at time t+1; and shifting and scaling the average distribution of target outputs at time t+1 to obtain the target distribution at time t.

[0129] Continue to refer to Figure 7The average distribution of the target output at time t+1 can be obtained by averaging the mixed distributions of the target outputs corresponding to the M policy networks, which can be expressed as the following formula (12).

[0130] (12);

[0131] in, This represents the average distribution of the target output at time t+1. This represents the m-th policy network at time t+1 in the mixed distribution of the target output. quantiles.

[0132] The average distribution of the target output at time t+1 can be shifted and scaled to obtain the target distribution at time t, which can be expressed as the following formula (13).

[0133] (13);

[0134] in, This represents the target distribution at time t. Describes the first element in the target distribution at time t. quantiles, , Indicates the attenuation factor. Represents the entropy temperature coefficient. This represents the reward at time t.

[0135] According to an embodiment of this application, updating the evaluation network based on the sample evaluation distribution and the target distribution at time t to obtain an updated evaluation network includes: determining the evaluation network loss value of the evaluation network based on the distance between the sample evaluation distribution and the target distribution at time t; and updating the evaluation network based on the evaluation network loss value to obtain the updated evaluation network.

[0136] Continue to refer to Figure 7 The sample state at time t can be... and the sample action at time t The input is fed into each of the N evaluation networks to obtain N evaluation distributions of samples at time t. Since the evaluation network and the target evaluation network have the same structure, the process of obtaining the sample evaluation distribution can refer to the target distribution. The difference is that no translation or scaling is required before the final sample distribution is output in the evaluation network.

[0137] Sample evaluation distribution at time t and the target distribution at time t The distance between them can be the 1-Wasserstein distance. Substituting this into formula (9), the distance between the sample evaluation distribution at time t and the target distribution at time t of the nth evaluation network can be expressed as: It can The evaluation network loss value of the nth evaluation network is used to update the parameters of the nth evaluation network through backpropagation. , which is represented by the following formula (14).

[0138] (14);

[0139] in, This indicates the evaluation of the network learning rate. This represents the gradient of the distance between the evaluation distribution of samples at time t and the target distribution at time t for the nth evaluation network. As an example, .

[0140] After the evaluation network parameters are updated, based on the parameters of each evaluation network... Update the parameters of the target evaluation network corresponding to the evaluation network, and update the parameters of the nth target evaluation network. The update is expressed as follows (15).

[0141] (15);

[0142] In the formula, This represents the update coefficients of the target evaluation network. As an example, .

[0143] According to the embodiments of this application, the parameter updates of each of the N evaluation networks can refer to the parameter update process of the nth evaluation network, and the parameter updates of each of the N target evaluation networks can refer to the parameter update process of the nth target evaluation network.

[0144] According to an embodiment of this application, using an updated evaluation network, an evaluation distribution at time t is obtained based on the sample state at time t and the output actions of each policy network at time t under the sample state at time t. This includes: using at least one updated evaluation network, obtaining an evaluation output distribution at time t corresponding to each policy network based on the sample state at time t and the output actions of each policy network at time t under the sample state at time t; sorting the evaluation output distributions at time t corresponding to each policy network to obtain a mixed distribution of evaluation outputs at time t corresponding to each policy network; and averaging the mixed distributions of evaluation outputs at time t corresponding to each policy network to obtain the evaluation distribution at time t.

[0145] Since the evaluation network and the target evaluation network have the same structure, the output of the evaluation distribution can be referenced from the output of the target distribution. The difference is that the evaluation network does not need to translate or scale the evaluation distribution.

[0146] According to an embodiment of this application, updating the parameters of each policy network based on the evaluation distribution at time t to obtain a target policy network includes: determining the policy network loss value of each policy network based on the evaluation distribution at time t; determining the average policy network loss based on the policy network loss value of each policy network; updating the parameters of each policy network based on the average policy network loss to obtain intermediate policy networks corresponding to each policy network; executing test tasks using the intermediate policy networks corresponding to each policy network to obtain rewards corresponding to each intermediate policy network; and determining the target policy network based on the rewards corresponding to each intermediate policy network.

[0147] Figure 8 A schematic diagram illustrating the updating of the policy network according to an embodiment of this application is shown.

[0148] like Figure 8 As shown, the sample state at time t... The inputs are fed into M policy networks, and the corresponding output action at time t is calculated for each network. The output action at time t for the m-th policy network is represented as follows: N evaluation networks calculate the evaluation output distribution at time t for each policy network based on the sample state and output action at time t. }, sort the evaluation output distributions at time t corresponding to each policy network and at least one updated evaluation network to obtain the mixed distribution of the evaluation outputs at time t for each policy network, where the mixed distribution of the evaluation outputs at time t corresponding to the m-th policy network is expressed as: The average of the mixed distributions of the evaluation outputs of each policy network at time t is obtained by summing the sums. Based on the evaluation distribution at time t, the loss function of each policy network can be established, and the loss value of the policy network can be obtained based on the loss function of the policy network, as shown in the following formula (16).

[0149] (16);

[0150] in, Let m be the loss function of the policy network. This represents the state of the m-th policy network at time t. Output action at time t below The kth quantile in the evaluation distribution at time t is obtained from the output of the nth evaluation network.

[0151] The parameters of each policy network are updated by backpropagation of the average loss of the policy network, as shown in the following formula (17).

[0152] (17);

[0153] in, It is the learning rate of the policy network. This represents the gradient of the average loss of the policy network. As an example, .

[0154] After updating the policy network a certain number of times, M intermediate policy networks can be obtained, for example, 2,000,000 updates. The parameters of the M intermediate policy networks can be expressed as follows: .

[0155] Ten ocean areas, each 100km x 100km in size, can be randomly selected as the mission area. The process involves acquiring ocean current and terrain data corresponding to latitude and longitude coordinates within each task area. Two latitude and longitude coordinates are randomly selected within each task area, serving as the start and end points of the test task, respectively. M intermediate policy networks are then interacted with the test tasks in each of the 10 task areas. The intermediate policy network with the highest average reward among the M networks is selected as the target policy network.

[0156] Figure 9 A schematic diagram of a target policy network performing a path planning task according to an embodiment of this application is shown.

[0157] like Figure 9 As shown, a target policy network is used for path planning, and the action output by the target policy network is taken as the optimal path. The goal of path planning is to guide the underwater glider to the mission endpoint. In this embodiment, the maximum time length of ocean current data is 5 days, meaning that this embodiment can generate the optimal path for the next 5 days. This means the planning interval can be 5 days, the navigation interval can be the distance traveled within 1 day, and the obstacle area can be the area in the ocean current containing objects that obstruct the underwater glider's navigation. On day d, the underwater glider actually navigates for one day according to the action output by the target policy network as the optimal path. After one day of actual navigation, the underwater glider will experience navigation deviation. Therefore, on day d+1, the ocean current data is updated, and the target policy network parameters are used again for path planning. The action output by the target policy network is taken as the optimal path to continue navigation for one day. This process is repeated until the distance from the underwater glider's current position to the mission endpoint is reached. Stop execution.

[0158] The target policy network obtained by the underwater glider dynamic path planning training method based on value distribution reinforcement learning in the embodiments of this application is used for underwater glider dynamic path planning. By regularly updating ocean current data, the dynamic path planning of the underwater glider can be realized, which can ensure the optimality of the output action. At the same time, based on the continuously updated ocean current data, the timeliness of the underwater glider path can be improved, and the navigation error of the underwater glider during the movement along the predetermined path can be reduced.

[0159] As a comparison, 100 path planning tasks can be set up, each covering an ocean area of ​​100 km. Within a 100km radius, two latitude and longitude coordinates are randomly selected within the ocean area of ​​each path planning task, serving as the task start and end points, respectively. The comparison method employs the Soft Actor-Critic (SAC) reinforcement learning algorithm and the Proximal Policy Optimization (PPO) reinforcement learning algorithm. The path planning tasks are performed by the target policy network obtained in this application embodiment and the algorithm. The number of training samples required for each algorithm to converge to the optimal policy, the average reward of the planned navigation path in 100 path planning tasks, the average number of time steps of the planned navigation path in 100 path planning tasks, and the average distance of the planned navigation path in 100 path planning tasks are compared.

[0160] As shown in Table 1, the algorithm in this embodiment requires approximately half the number of samples compared to the SAC reinforcement learning algorithm and the PPO reinforcement learning algorithm, and consumes less time step. Benefiting from the optimization objective of the meta-policy, the underwater glider dynamic path planning method in this embodiment also minimizes the change in yaw angle at each profile, resulting in a less energy-consuming navigation path.

[0161] Table 1

[0162] Figure 10 The experimental results of the underwater glider dynamic path planning method according to an embodiment of this application in Experiment 1 are shown.

[0163] like Figure 10 As shown, G1 represents the underwater glider in Experiment 1. The dynamic path planning method for the underwater glider is compared with the expected trajectory in the actual ocean experiment. Experiment 1 included four days of planning for the underwater glider, with 30 profile movements. The average maximum diving depth of the underwater glider was approximately 700 meters, the average diving time was approximately 80 minutes, the average surfacing time was approximately 80 minutes, and the navigation error of the underwater glider was 1.11 km.

[0164] Figure 11 The experimental results of the underwater glider dynamic path planning method according to an embodiment of this application in Experiment 2 are shown.

[0165] like Figure 11 As shown, G2 represents the underwater glider in Experiment 2. Figure 11 The irregular shapes in the diagram represent obstacles. In Experiment 2, the underwater glider performed 30 profile movements, successfully avoiding the obstacles. The average maximum diving depth of the underwater glider was approximately 700 meters, the average diving time was approximately 80 minutes, the average surfacing time was approximately 80 minutes, and the navigation error of the underwater glider was 1.11 km.

[0166] Figure 12 A block diagram of a dynamic path planning training apparatus for an underwater glider based on value distribution reinforcement learning, according to an embodiment of this application, is shown.

[0167] like Figure 12 As shown, the underwater glider dynamic path planning training device 1200 based on value distribution reinforcement learning includes an input module 1210, a first obtaining module 1220, a second obtaining module 1230, a third obtaining module 1240, a fourth obtaining module 1250, a fifth obtaining module 1260, and a sixth obtaining module 1270.

[0168] The input module 1210 is used to input training data, which includes the underwater glider's sample state at time t, sample action at time t, and sample state at time t+1 obtained after performing the sample action at time t, where t is a positive integer.

[0169] The first obtaining module 1220 is used to obtain the output action of each policy network at time t in the sample state at time t and the output action at time t+1 in the sample state at time t+1 using at least one policy network.

[0170] The second module 1230 is used to obtain the target distribution at time t by using the target evaluation network, based on the sample state at time t+1 and the output actions of each policy network at time t+1 under the sample state at time t+1. The target distribution at time t represents the predicted distribution of the cumulative reward of the underwater glider after performing actions based on the policy network under the sample state at time t, based on the quantiles. The quantiles represent the numerical value of the cumulative reward corresponding to the probability represented by the quantiles.

[0171] The third module 1240 is used to obtain the evaluation distribution of the sample at time t based on the sample state and sample action at time t using the evaluation network. The evaluation distribution of the sample at time t represents the actual distribution of the cumulative reward based on the quantile after the underwater glider performs the sample action at time t based on the policy network in the sample state at time t.

[0172] The fourth module 1250 is used to update the evaluation network based on the sample evaluation distribution and the target distribution at time t, so as to obtain the updated evaluation network.

[0173] The fifth module 1260 is used to obtain the evaluation distribution at time t by utilizing the updated evaluation network and based on the sample state at time t and the output actions of each policy network at time t under the sample state at time t.

[0174] The sixth module 1270 is used to update the parameters of each policy network according to the evaluation distribution at time t, and obtain the target policy network, so as to control the underwater glider to perform path planning based on the target policy network.

[0175] Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented by hardware or firmware in any other reasonable manner by integrating or packaging circuits, or implemented in any one of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0176] For example, any and more of the input module 1210, the first obtaining module 1220, the second obtaining module 1230, the third obtaining module 1240, the fourth obtaining module 1250, the fifth obtaining module 1260, and the sixth obtaining module 1270 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this application, at least one of the input module 1210, the first obtaining module 1220, the second obtaining module 1230, the third obtaining module 1240, the fourth obtaining module 1250, the fifth obtaining module 1260, and the sixth obtaining module 1270 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the input module 1210, the first obtaining module 1220, the second obtaining module 1230, the third obtaining module 1240, the fourth obtaining module 1250, the fifth obtaining module 1260, and the sixth obtaining module 1270 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0177] Figure 13 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of this application, is shown. Figure 13 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0178] like Figure 13 As shown, an electronic device 1300 according to an embodiment of this application includes a processor 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage portion 1308 into a random access memory (RAM) 1303. The processor 1301 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1301 may also include onboard memory for caching purposes. The processor 1301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0179] RAM 1303 stores various programs and data required for the operation of electronic device 1300. Processor 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Processor 1301 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 1302 and / or RAM 1303. It should be noted that the programs may also be stored in one or more memories other than ROM 1302 and RAM 1303. Processor 1301 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0180] According to embodiments of this application, the electronic device 1300 may further include an input / output (I / O) interface 1305, which is also connected to a bus 1304. The electronic device 1300 may also include one or more of the following components connected to the input / output (I / O) interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the input / output (I / O) interface 1305 as needed. A removable medium 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1310 as needed so that computer programs read from it can be installed into the storage section 1308 as needed.

[0181] According to embodiments of this application, the method flow according to embodiments of this application can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1309, and / or installed from removable medium 1311. When the computer program is executed by processor 1301, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0182] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0183] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0184] For example, according to embodiments of this application, a computer-readable storage medium may include the ROM 1302 and / or RAM 1303 described above and / or one or more memories other than ROM 1302 and RAM 1303.

[0185] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this application.

[0186] When the computer program is executed by the processor 1301, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0187] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1309, and / or installed from the removable medium 1311. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0188] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0189] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0190] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A dynamic path planning training method for underwater gliders based on value distribution reinforcement learning, characterized in that, include: Input training data, which includes the underwater glider's sample state at time t, sample action at time t, and sample state at time t+1 obtained after performing the sample action at time t, where t is a positive integer. The sample state of the underwater glider at each time is determined in a pre-constructed sample state space, which includes terrain data, ocean current data, heading angle data, and the position data of the underwater glider after it leaves the water. The sample action of the underwater glider at each time is determined in a pre-constructed sample action space, which represents the heading angle adjustment amount of the underwater glider. Using at least one policy network, the output action of each policy network at time t in the sample state at time t and the output action at time t+1 in the sample state at time t+1 are obtained. Using a target evaluation network, based on the sample state at time t+1 and the output actions of each policy network at time t+1 in the sample state at time t+1, the target distribution at time t is obtained. The target distribution at time t represents the quantile-based predicted distribution of the cumulative reward of the underwater glider after performing actions based on the policy network in the sample state at time t. The quantile represents the cumulative reward corresponding to the probability represented by the quantile. Using an evaluation network, the evaluation distribution of the sample at time t is obtained based on the sample state and the sample action at time t. The evaluation distribution of the sample at time t represents the actual distribution of the cumulative reward of the underwater glider after executing the sample action at time t based on the policy network in the sample state at time t, based on the quantile. The evaluation network is updated based on the sample evaluation distribution at time t and the target distribution at time t to obtain the updated evaluation network. Using the updated evaluation network, the evaluation distribution at time t is obtained based on the sample state at time t and the output actions of each policy network at time t under the sample state at time t. Based on the evaluation distribution at time t, the parameters of each policy network are updated to obtain the target policy network, so as to control the underwater glider to perform path planning based on the target policy network.

2. The method according to claim 1, characterized in that, The terrain data represents the terrain data within a first rectangular sea area centered on the underwater glider. The terrain data is represented by a terrain height matrix composed of a first sub-element and a second sub-element. The first sub-element indicates that the absolute value of the terrain true value in the first rectangular sea area is less than or equal to the maximum diving depth of the underwater glider. The second sub-element indicates that the terrain true value in the first rectangular sea area is less than the maximum diving depth of the underwater glider. The ocean current data represents the ocean current data within a second rectangular sea area centered on the underwater glider. The ocean current data includes the ocean current velocity component data of the underwater glider at the detection point along the eastward direction in the inertial coordinate system and the ocean current velocity component data of the underwater glider at the detection point along the northward direction in the inertial coordinate system. The boundary of the second rectangular sea area is parallel to the longitude and latitude directions, respectively. The heading angle data represents the deviation between the current heading angle and the desired heading angle of the underwater glider. Both the current heading angle and the desired heading angle are the angles between the fuselage direction and the longitude direction of the underwater glider. The desired heading angle represents the target heading angle of the underwater glider when performing path planning. The location data represents the distance between the current position of the underwater glider and the destination of the path planning task, as well as the relationship between the starting point and the destination of the path planning task.

3. The method according to claim 1, characterized in that, The step of using a target evaluation network to obtain the target distribution at time t based on the sample state at time t+1 and the output actions of each policy network at time t+1 in the sample state at time t+1 includes: Using at least one of the target evaluation networks, based on the sample state at time t+1 and the output action of each policy network at time t+1 in the sample state at time t+1, the target output distribution at time t+1 corresponding to each policy network and at least one of the target evaluation networks is obtained respectively. The target output distributions at time t+1 corresponding to each of the policy networks and at least one of the target evaluation networks are sorted to obtain a mixed distribution of the target outputs at time t+1 corresponding to each of the policy networks. The target output mixture distribution at time t+1 corresponding to each of the policy networks is averaged to obtain the target distribution at time t.

4. The method according to claim 3, characterized in that, The step of averaging the mixed distribution of the target output at time t+1 corresponding to each of the policy networks to obtain the target distribution at time t includes: The average distribution of the target output at time t+1 corresponding to each of the policy networks is averaged to obtain the average distribution of the target output at time t+1. The average distribution of the target output at time t+1 is shifted and scaled to obtain the target distribution at time t.

5. The method according to any one of claims 1 to 3, characterized in that, The step of using the updated evaluation network to obtain the evaluation distribution at time t based on the sample state at time t and the output actions of each policy network at time t under the sample state at time t includes: Using at least one updated evaluation network, based on the sample state at time t and the output action of each policy network at time t under the sample state at time t, the evaluation output distribution at time t corresponding to each policy network is obtained; The evaluation output distributions at time t corresponding to each policy network are sorted to obtain the mixed distribution of the evaluation outputs at time t corresponding to each policy network. The evaluation distribution at time t is obtained by averaging the mixed distribution of the evaluation outputs corresponding to each of the policy networks.

6. The method according to any one of claims 1 to 3, characterized in that, The training data was obtained through the following operations: The task area is determined by randomly selecting a pre-defined area from the marine environment. Acquire ocean current and terrain data within the task area; The starting and ending points of the mission are determined based on the longitude and latitude of the mission area. Based on the mission starting point, the ocean current mission data, and the terrain mission data, obtain the sample state of the underwater glider at time t. Based on the sample state at time t, the action of the sample at time t is obtained using the policy network; After the underwater glider performs the sample action at time t, the sample state of the underwater glider at time t+1 is obtained.

7. The method according to any one of claims 1 to 3, characterized in that, Based on the evaluation distribution at time t, the parameters of each policy network are updated to obtain the target policy network, including: Based on the evaluation distribution at time t, determine the policy network loss value for each policy network; Based on the policy network loss values ​​of each policy network, determine the average policy network loss. Based on the average loss of the policy network, the parameters of each policy network are updated to obtain an intermediate policy network corresponding to each policy network. The test task is executed using the intermediate policy network corresponding to each of the policy networks, and the reward corresponding to each of the intermediate policy networks is obtained. The target policy network is determined based on the rewards corresponding to each of the intermediate policy networks.

8. The method according to any one of claims 1 to 3, characterized in that, The step of updating the evaluation network based on the sample evaluation distribution at time t and the target distribution at time t to obtain the updated evaluation network includes: The evaluation network loss value of the evaluation network is determined based on the distance between the sample evaluation distribution at time t and the target distribution at time t. The evaluation network is updated based on the evaluation network loss value to obtain the updated evaluation network.

9. The method according to claim 2, characterized in that, The policy network includes a policy ocean current feature convolutional layer, a policy terrain feature convolutional layer, a policy feature splicing layer, and a policy fully connected layer; The policy network obtains the output action of the sample state at each time step through the following operations: The ocean current data is processed using the strategy current feature convolutional layer to obtain strategy current features; The terrain data is processed using the convolutional layer that describes the terrain features of the strategy to obtain the terrain features of the strategy. The strategy feature stitching layer is used to stitch together the strategy ocean current features, the strategy terrain features, the heading angle data, and the position data to obtain the strategy hybrid features; The policy mixture features are processed using the fully connected layer to obtain the output action under the sample state at each time step.

10. The method according to claim 9, characterized in that, The evaluation network includes a convolutional layer for evaluating ocean current features, a convolutional layer for evaluating topographic features, a feature splicing layer, and a fully connected layer for evaluating features; The evaluation network obtains the sample distribution of the sample states at each time step through the following operations: The ocean current data is processed using the convolutional layer for evaluating ocean current features to obtain the evaluation ocean current features; The terrain data is processed using the convolutional layer for evaluating terrain features to obtain the evaluated terrain features; The evaluation feature layer is used to stitch together the evaluation ocean current features, the evaluation topographic features, the heading angle data, the position data, and the sample actions corresponding to the sample states at each time point to obtain the evaluation hybrid features; The evaluation hybrid features are processed using the fully connected evaluation layer to obtain the sample distribution under the sample state at each time step.

Images