Marine observation station intelligent design method based on deep reinforcement learning

Abstract

The invention discloses an intelligent design method for an ocean observation station based on deep reinforcement learning, which belongs to the technical field of intelligent design of observation stations, is used for intelligent design of observation stations, and comprises the following steps: acquiring multi-source ocean data, making a deep learning sample set, and constructing and training a deep neural network model in combination with deep reinforcement learning. And performing ocean observation station reasoning by using the trained deep neural network. According to the method, rapid, batch and optimal design of regional ocean observation stations is realized, and the accuracy of an ocean observation reconstruction field is improved, so that guarantee and service are provided for related application requirements depending on an ocean environment field.

Description

Technical Field

[0001] The present invention discloses an intelligent design method for ocean observation stations based on deep reinforcement learning, belonging to the technical field of intelligent design of observation stations. Background Art

[0002] Focusing on the demand for obtaining marine information under sudden marine situations, the rapid, batch, and optimized design of marine observation stations is the prerequisite for implementing regional marine rapid mobile network observations and the basis for achieving real-time reconstruction of regional marine three-dimensional fine environmental information. It is of great significance for ensuring fine marine environmental information under sudden marine situations. Marine observation station design is an important factor affecting the observation and reconstruction effects. Reasonable observation station design is a breakthrough for optimizing observation effects and improving reconstruction accuracy. However, existing station design methods have problems such as unstable artificial station design effects, difficult station design in complex sea areas, and traditional optimization algorithms that easily fall into local optimality. Summary of the invention

[0003] The purpose of the present invention is to provide a method for intelligent design of ocean observation stations based on deep reinforcement learning, so as to solve the problem that the intelligent design results of ocean observation stations in the prior art are not ideal.

[0004] An intelligent design method for ocean observation stations based on deep reinforcement learning, including obtaining multi-source ocean data and making a deep learning sample set, combining deep reinforcement learning to build a training deep neural network model, and using the trained deep neural network to perform ocean observation station inference.

[0005] Deep reinforcement learning includes constructing an intelligent agent as the subject of deep reinforcement learning, constructing an environment as the interactive object of the intelligent agent, the intelligent agent takes actions based on the current state of the environment, constructing a value to describe the current state and the quality of the action, and constructing a reward as the value returned to the intelligent agent by the environment after the intelligent agent performs an action, so as to maximize the reward and guide the intelligent agent to take actions according to the strategy;

[0006] Value and strategy are functions that depend on current action, future action, current state, and future state, and are obtained through training and fitting of deep neural networks.

[0007] The deep neural network model includes a policy network and a value network built based on the actor-critic algorithm, which are used to fit the policy function and the value function respectively;

[0008] The policy network uses multi-source ocean data as environmental input, extracts spatial feature information of the ocean environment field through the convolution layer, and outputs action strategies through the fully connected layer and tanh layer. The value network uses multi-source ocean data as environmental input, extracts spatial feature information of the ocean environment field through the convolution layer, and outputs state-action values ​​through the fully connected layer.

[0009] Training the deep neural network model includes using the gradient ascent method to train the policy network and the gradient descent method to train the value network. Multi-source ocean data is used as the state in the training, and the movement of the observation station on the two-dimensional plane is used as the action. The current reward and the closeness of the current state to the true value are combined to obtain the reward. The deep neural network is trained so that the learning of the deep neural network will maximize the reward and obtain the strategy to move the observation station to the optimal position.

[0010] The ocean observation station reasoning is a process of using a trained deep neural network model, inputting multi-source ocean data, and outputting an intelligent design process for ocean observation station locations.

[0011] The actor-critic algorithm includes action, policy network, value network, environment and state. The state and action are input into the policy network and value network. The environment transfers rewards to the value network, and the value network transfers values ​​to the policy network.

[0012] The value network consists of a convolutional network and a fully connected network in turn.

[0013] The policy network includes a convolutional network, a fully connected network, and a tanh activation function in sequence.

[0014] The elements of multi-source ocean data include temperature data, salinity data and density data, and the types of multi-source ocean data include observational data, reanalysis data and numerical simulation data.

[0015] The deep learning sample set includes training set, validation set and test set.

[0016] Compared with the existing technology, the present invention has the following beneficial effects: it realizes the rapid, batch and optimized design of regional ocean observation stations, helps to improve the accuracy of ocean observation reconstruction fields, and thus provides protection and services for related application needs that rely on ocean environmental fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a technical flow chart of the present invention;

[0018] Figure 2 This is a schematic diagram of the actor-critic algorithm principle;

[0019] Figure 3 It is a diagram of the value network;

[0020] Figure 4 is a schematic diagram of the strategy network;

[0021] Figure 5 It is a schematic diagram of a convolutional neural network;

[0022] Figure 6It is the convolution module 1 structure;

[0023] Figure 7 It is the structure of convolution module 2, convolution module 3, convolution module 4 and convolution module 5;

[0024] Figure 8 It is a residual connection structure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] An intelligent design method for ocean observation stations based on deep reinforcement learning, including obtaining multi-source ocean data and making a deep learning sample set, combining deep reinforcement learning to build a training deep neural network model, and using the trained deep neural network to perform ocean observation station inference.

[0027] Deep reinforcement learning includes constructing an intelligent agent as the subject of deep reinforcement learning, constructing an environment as the interactive object of the intelligent agent, the intelligent agent takes actions based on the current state of the environment, constructing a value to describe the current state and the quality of the action, and constructing a reward as the value returned to the intelligent agent by the environment after the intelligent agent performs an action, so as to maximize the reward and guide the intelligent agent to take actions according to the strategy;

[0028] Value and strategy are functions that depend on current action, future action, current state, and future state, and are obtained through training and fitting of deep neural networks.

[0029] The deep neural network model includes a policy network and a value network built based on the actor-critic algorithm, which are used to fit the policy function and the value function respectively;

[0030] The policy network uses multi-source ocean data as environmental input, extracts spatial feature information of the ocean environment field through the convolution layer, and outputs action strategies through the fully connected layer and tanh layer. The value network uses multi-source ocean data as environmental input, extracts spatial feature information of the ocean environment field through the convolution layer, and outputs state-action values ​​through the fully connected layer.

[0031] Training the deep neural network model includes using the gradient ascent method to train the policy network and the gradient descent method to train the value network. Multi-source ocean data is used as the state in the training, and the movement of the observation station on the two-dimensional plane is used as the action. The current reward and the closeness of the current state to the true value are combined to obtain the reward. The deep neural network is trained so that the learning of the deep neural network will maximize the reward and obtain the strategy to move the observation station to the optimal position.

[0032] The ocean observation station reasoning is a process of using a trained deep neural network model, inputting multi-source ocean data, and outputting an intelligent design process for ocean observation station locations.

[0033] The actor-critic algorithm includes action, policy network, value network, environment and state. The state and action are input into the policy network and value network. The environment transfers rewards to the value network, and the value network transfers values ​​to the policy network.

[0034] The value network consists of a convolutional network and a fully connected network in turn.

[0035] The policy network includes a convolutional network, a fully connected network, and a tanh activation function in sequence.

[0036] The elements of multi-source ocean data include temperature data, salinity data and density data, and the types of multi-source ocean data include observational data, reanalysis data and numerical simulation data.

[0037] The deep learning sample set includes training set, validation set and test set.

[0038] The technical process of the present invention is as follows Figure 1 As shown, it includes making a deep learning sample set, building a deep neural network, training the deep neural network, and performing observatory inference.

[0039] Actor-critic algorithms such as Figure 2 As shown, it includes action, policy network, value network, environment and state. The state and action are input into the policy network and value network. The environment transfers rewards to the value network, and the value network transfers values ​​to the policy network.

[0040] Value network Figure 3 As shown, it includes convolutional network and fully connected network in sequence.

[0041] Strategy network Figure 4 As shown, it includes convolutional network, fully connected network and tanh activation function in sequence.

[0042] The convolutional neural network model of the present invention is as follows Figure 5As shown in FIG. 1 , it includes five serial convolutional layers in sequence. The first convolutional layer (convolutional module 1) has a size of 7×7, a channel of 64, and a step size of 2. The second convolutional layer (convolutional module 2) includes a maximum pooling layer and two blocks. The maximum pooling layer has a size of 3×3 and a step size of 2. Each block includes two convolutional layers with a size of 3×3, a channel of 64, and a step size of 1. Residual connections are established between blocks. The third convolutional layer (convolutional module 3) includes two blocks. Each block includes two convolutional layers with a size of 3×3 and a channel of 128. The first convolutional layer in each block The stride is 2, the convolution layer stride is 1, and residual connections are established between blocks; the fourth convolution layer (convolution module 4) includes two blocks, each block includes two convolution layers of size 3×3 and channels 256, the first convolution layer stride in each block is 2, the convolution layer stride is 1, and residual connections are established between blocks; the fifth convolution layer (convolution module 5) includes two blocks, each block includes two convolution layers of size 3×3 and channels 512, the first convolution layer stride in each block is 2, the convolution layer stride is 1, and residual connections are established between blocks. After the fifth convolution layer, the output layer is composed of an average pooling layer and a fully connected layer with an output size of 1×160, which outputs the coordinates of the observation station.

[0043] The structure of convolution module 1 is as follows Figure 6 As shown, it is composed of a convolution layer and a pooling layer. In the embodiment, the convolution layer has a convolution kernel size of 7×7, 64 channels, and a stride of 2; the pooling layer is a maximum pooling layer, and its kernel size is 3×3 and the stride is 2.

[0044] The input of convolution module 1 is X0, and the output is X1, as shown in formula (1):

[0045] X1=Conv1(X0) (1);

[0046] Wherein, Conv1 represents the operation function of convolution module 1, the shape of X0 is h=224, w=224, c=1; the shape of X1 is h=56, w=56, c=64.

[0047] The structures of convolution module 2, convolution module 3, convolution module 4 and convolution module 5 are as follows: Figure 7 As shown, the module consists of four convolutional layers.

[0048] In the convolutional layer module 2, the convolution kernel size of convolutional layer A, convolutional layer B, convolutional layer C, and convolutional layer D is 3×3, the channel is 64, and the stride is 1. The input of convolutional module 2 is X1, and the output is X2, as shown in formula (2):

[0049] X2=Conv2(X1) (2);

[0050] Wherein, Conv2 represents the operation function of convolution module 2, and the shape of X2 is h=56, w=56, c=64.

[0051] In the convolutional layer module 3, the convolution kernel size of convolutional layer A is 3×3, the channel is 128, and the stride is 2. The convolution kernel size of convolutional layer B, convolutional layer C, and convolutional layer D is 3×3, the channel is 128, and the stride is 1. The input of convolutional module 3 is X2, and the output is X3, as shown in formula (3):

[0052] X3=Conv3(X2) (3);

[0053] Wherein, Conv3 represents the operation function of convolution module 3, and the shape of X3 is h=28, w=28, c=128.

[0054] In convolutional layer module 4, the convolution kernel size of convolutional layer A is 3×3, the channel is 256, and the stride is 2. The convolution kernel size of convolutional layer B, convolutional layer C, and convolutional layer D is 3×3, the channel is 256, and the stride is 1. The input of convolutional module 4 is X3, and the output is X4, as shown in formula (4):

[0055] X4=Conv4(X3) (4);

[0056] Wherein, Conv4 represents the operation function of convolution module 4, and the shape of X4 is h=14, w=14, c=256.

[0057] In the convolutional layer module 5, the convolution kernel size of convolutional layer A is 3×3, the channel is 512, and the stride is 2. The convolution kernel size of convolutional layer B, convolutional layer C, and convolutional layer D is 3×3, the channel is 512, and the stride is 1. The input of convolutional module 5 is X4, and the output is X5, as shown in formula (5):

[0058] X5=Conv5(X4) (5);

[0059] Wherein, Conv5 represents the operation function of convolution module 5, and the shape of X5 is h=7, w=7, c=512.

[0060] In this embodiment, the pooling layer uses an average pooling layer and a fully connected layer with an output dimension of 160 to output the observation station coordinates, as shown in formula (6):

[0061] Y = FC(POOL(X5)) (6);

[0062] Where POOL represents the pooling operation function, FC represents the fully connected operation function, and Y is a vector of length 160.

[0063] The residual connection is used as a pluggable module in convolution module 2, convolution module 3, convolution module 4 and convolution module 5. Figure 8 , the residual connection expresses the output as a linear superposition of the input and a nonlinear transformation of the input. The residual connection in the embodiment is shown in formula (7):

[0064] y=Conv(ReLU(Conv(x)))+x (7);

[0065] In the formula, Conv represents the convolution operation function, ReLU is the nonlinear activation function, x represents the input of the residual connection, and y represents the output of the residual connection.

[0066] The present invention uses HYCOM sea surface temperature field data from 2005 to 2015 to produce a sample set with a total number of 100,000; samples are uniformly selected in 10 years with a step length of 1 month, thereby achieving sample time diversity; samples are randomly selected in the global mid- and low-latitude waters and the South China Sea at a ratio of 4:1, thereby achieving the regional diversity of samples and strengthening the characteristic proportion of key areas; three different sizes of sea areas, 100km×100km, 200km×200km, and 300km×300km, are selected at a ratio of 1:3:1, and then scaled to the same size, thereby achieving diversity in sea area scales; all sample sets are stratified and sampled at a ratio of 7:2:1 as training sets, validation sets, and test sets.

[0067] Building a deep neural network model involves the following steps:

[0068] S2.1 inputs the ocean environment field in the sample set into the neural network as the true value field, and sets 80 observation stations on the environment field in a random initialization manner.

[0069] S2.2 uses the observation station information to perform virtual sampling on the environmental field, and the sampling result is used as the observation value. i is the horizontal coordinate of the observation station, y i is the vertical coordinate of the observation station, and (x i ,y i ) corresponds to the attribute value o i , will o i As the observations of virtual sampling.

[0070] S2.3 Use the data assimilation algorithm to calculate the inverted environmental field through the observation value and the station information, wherein the data assimilation algorithm refers to an algorithm for inferring the real environmental field through the sampling of the observation data, including the stepwise correction method, the Kriging interpolation method, the optimal interpolation method, and the variational assimilation method. Specifically, in the embodiment, according to the station coordinates and the observation value (x i ,y i ,oi ), and use ordinary Kriging to invert and reconstruct the environmental field F r , as shown in formula (8):

[0071] F r =Kriging(x i ,y i ,o i ) (8);

[0072] S2.4 constructs the state S, which is represented as a third-order tensor of 224×224×6. The first two dimensions correspond to the longitude and latitude of the ocean environment field, and the third dimension records the current environment field, the past environment field, the environment truth field, and the station information. The above environment field is initialized as the initial environment field, as shown in formula (9):

[0073] S=[F0,F1,F2,F3,F t ,P] (9);

[0074] Where F0 represents the current environment field matrix; F1, F2, and F3 represent the environment field matrices in the past three loop iterations, which are initialized to F0; F t represents the environment truth field matrix; P represents the station information, the matrix element corresponding to the station is recorded as 1, and the other matrix elements are recorded as 0.

[0075] S2.5 Constructing action A represents the movement of the observation station on the two-dimensional plane. Constructing a strategy is used to guide the agent to perform actions according to the current state, as shown in formula (10):

[0076] A=[A 1x ,A 1y ,A 2x ,A 2y ,…,A 80x ,A 80y ,] (10);

[0077] In the formula, A 1x A represents the longitudinal movement of the first station. 1y represents the latitudinal movement of the first station, A 2x Indicates the longitudinal movement of the second station, A 2y Indicates the latitudinal movement of the second station, and so on. 80x represents the longitudinal movement of the 80th station, A 80y Indicates the latitudinal movement of the 80th station. The range of the movement of each station is {-1, 0, 1}, which respectively means moving one unit in the negative direction, staying in place, and moving one unit in the positive direction.

[0078] S2.6 constructs the policy network Policy and the value network Value and initializes them randomly. The policy network obtains action A according to the input state S, and executes action A to move the observation station to a new position. The value network obtains the value evaluation Q of the current state according to the input state S, as shown in formula (11) and formula (12):

[0079] A=Policy(S)=tanh(FC1(Conv1(S))) (11);

[0080] Q=Value(S)=FC2(Conv2(S)) (12);

[0081] In the formula, Conv1 represents the convolutional network in the policy network, FC1 represents the fully connected network in the policy network, tanh represents the hyperbolic tangent activation function of the fully connected network in the policy network, Conv2 represents the convolutional network in the value network, and FC2 represents the fully connected network in the value network.

[0082] S2.7 After the station moves, virtual sampling is performed on the true value field according to the new position to obtain the observation value. The new environmental field is reconstructed using the Kriging interpolation method based on the station position information and the observation value as the current environmental field, and the state S is updated, as shown in formula (13):

[0083] F3←F2,F2←F1,F1←F0,F0←F new ,P←P new (13);

[0084] In the formula, F new The environmental field reconstructed by the current new observation value, P new It is the current new station information matrix.

[0085] S2.8 constructs a reward R, which is obtained by summing the improvement of the current state value and the closeness of the current state to the true value, as shown in formula (14):

[0086]

[0087] Where Improve is the improvement of the current state value evaluation Q0 relative to the previous state value evaluation Q1. If it is positive, it is recorded as +1, if it is negative, it is recorded as -1, otherwise it is recorded as 0. The closeness between the current state and the true value is calculated by calculating the environmental field F0 and the environmental true value field F t The root mean square error (root mean square) is obtained; ω is a weight hyperparameter.

[0088] Training a deep neural network model consists of the following steps:

[0089] S3.1 Observe the current state s t , make decisions based on the policy network a t ~ω(·|s t θ now ), and let the agent perform action a t , where θ now are the policy neural network parameters;

[0090] S3.2 Observe the reward r from the environment t and the new state s t+1 ;

[0091] S3.3 Making decisions based on the policy network But do not let the agent perform actions

[0092] S3.4 Obtain the score of the value network at time t and time t+1 and

[0093]

[0094] In the formula, w now is the value neural network parameter

[0095] S3.5 Calculation target and error δ t :

[0096]

[0097] Where γ is an adjustable hyperparameter, is the output at time t.

[0098] S3.6 Update value network:

[0099]

[0100] Where α is the learning rate of the value network;

[0101] S3.7 Update strategy network:

[0102]

[0103] Where β is the learning rate of the policy network.

[0104] When performing ocean observation station inference, the strategy random sampling method is used. According to the observed state s, the trained strategy network adopts the strategy π in the action space Random sampling is performed to obtain actions a~π(·|s;θ); the process of executing actions, updating states, executing actions, updating states, etc. is repeated until the stopping condition is reached. The stopping condition is set to meet one of the following two conditions: (1) The number of loops n exceeds the number threshold n th , that is, n>n th ; (2) The error D between the reconstructed field and the true value field is lower than the error threshold D th , that is, D <D th .

[0105] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent design of ocean observation stations based on deep reinforcement learning, characterized in that: It includes acquiring multi-source ocean data and producing deep learning sample sets, combining deep reinforcement learning to build and train deep neural network models, and using trained deep neural networks to infer ocean observation station locations.

2. According to claim 1, a method for intelligent design of ocean observation stations based on deep reinforcement learning is characterized in that: Deep reinforcement learning includes constructing an intelligent agent as the subject of deep reinforcement learning, constructing an environment as the interactive object of the intelligent agent, the intelligent agent takes actions based on the current state of the environment, constructing a value to describe the current state and the quality of the action, and constructing a reward as the value returned to the intelligent agent by the environment after the intelligent agent performs an action, so as to maximize the reward and guide the intelligent agent to take actions according to the strategy; Value and strategy are functions that depend on current action, future action, current state, and future state, and are obtained through training and fitting of deep neural networks.

3. According to claim 2, a method for intelligent design of ocean observation stations based on deep reinforcement learning is characterized in that: The deep neural network model includes a policy network and a value network built based on the actor-critic algorithm, which are used to fit the policy function and the value function respectively; The policy network uses multi-source ocean data as environmental input, extracts spatial feature information of the ocean environment field through the convolution layer, and outputs action strategies through the fully connected layer and tanh layer. The value network uses multi-source ocean data as environmental input, extracts spatial feature information of the ocean environment field through the convolution layer, and outputs state-action values ​​through the fully connected layer.

4. According to claim 3, a method for intelligent design of ocean observation stations based on deep reinforcement learning is characterized in that: Training the deep neural network model includes using the gradient ascent method to train the policy network and the gradient descent method to train the value network. Multi-source ocean data is used as the state in the training, and the movement of the observation station on the two-dimensional plane is used as the action. The current reward and the closeness of the current state to the true value are combined to obtain the reward. The deep neural network is trained so that the learning of the deep neural network will maximize the reward and obtain the strategy to move the observation station to the optimal position.

5. According to claim 4, a method for intelligent design of ocean observation stations based on deep reinforcement learning is characterized in that: The ocean observation station reasoning is a process of using a trained deep neural network model, inputting multi-source ocean data, and outputting an intelligent design process for ocean observation station locations.

6. The method for intelligent design of ocean observation stations based on deep reinforcement learning according to claim 5, characterized in that: The actor-critic algorithm includes action, policy network, value network, environment and state. The state and action are input into the policy network and value network. The environment transfers rewards to the value network, and the value network transfers values ​​to the policy network.

7. The method for intelligent design of ocean observation stations based on deep reinforcement learning according to claim 6, characterized in that: The value network consists of a convolutional network and a fully connected network in turn.

8. The method for intelligent design of ocean observation stations based on deep reinforcement learning according to claim 7, characterized in that: The policy network includes a convolutional network, a fully connected network, and a tanh activation function in sequence.

9. The method for intelligent design of ocean observation stations based on deep reinforcement learning according to claim 8, characterized in that: The elements of multi-source ocean data include temperature data, salinity data and density data, and the types of multi-source ocean data include observational data, reanalysis data and numerical simulation data.

10. The method for intelligent design of ocean observation stations based on deep reinforcement learning according to claim 9, characterized in that: The deep learning sample set includes training set, validation set and test set.

Images