A reinforcement learning method for ocean search
Through reinforcement learning methods for sea area search, state space, action space and sparse reward functions are designed, and model training is combined with PPO algorithm, which solves the problem of insufficient applicability and risk hazards of sea area search decision design in the existing technology, realizes an adaptive search strategy in complex maritime environments, and improves search efficiency.
Patent Information
- Application Number
- CN202411625627.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-04-08
AI Technical Summary
The existing sea area search decision design mainly relies on rule algorithms, requires manual participation and prior knowledge, and is difficult to adapt to the complex and changeable maritime environment and escape strategies of intelligent search targets, resulting in poor applicability and unforeseen risks.
The reinforcement learning method for sea area search is adopted to design state space, action space and sparse reward functions, and model training is combined with PPO algorithm to realize adaptive decision-making of the agent.
It improves the applicability of the model and can adaptively adjust search strategies in complex maritime environments, reduce unforeseen risks and improve search efficiency.
Smart Images
Figure SMS_5 
Figure SMS_6 
Figure SMS_16
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of reinforcement learning, and in particular relates to a reinforcement learning method for sea area search. Background Art
[0002] The application scenario of sea area search is generally within a certain limited area. The search subject locates and tracks the hidden search target through sensor exploration and other methods to achieve the search purpose of finding the specific location of the search target. Sea area search has the characteristics of wide range of movement and changeable movement environment. Various factors including climate, hydrology, day and night make sea area search more complicated. At present, the decision-making design of sea area search is generally based on rule algorithms, which can form the movement direction control of the search subject in line with human cognitive conditions, such as continuous motion trajectory design based on basic graphics and oriented to the breadth of search coverage area. The formulation of its rules requires human participation and must have prior knowledge. The degree of detail will also directly affect the actual search effect. At the same time, the marine environment is complex and changeable, and the search subject and the search target may have strong confrontation. The escape strategy of the search target can be intelligent and changeable. The single and prescribed search rules are inevitably considered for all situations. This will lead to unforeseen risks and hidden dangers in the actual application process, and the applicability is not strong. The search rules formulated in a certain scenario may have a good effect, but once the scenario changes, the search rules may lose their effect and basically do not have the conditions for transplantation. Summary of the invention
[0003] In order to solve the technical problems in the prior art, the present invention provides a reinforcement learning method for sea area search, comprising the following steps:
[0004] Step 1: Design the state space for sea area search and set the state space to vector form:
[0005]
[0006] Where n is the search area control coefficient, which can be 1 or 2. If it is 1, the search area is a rectangle; if it is 2, the search area is a circle. They are respectively the horizontal coordinate and the vertical coordinate of the current position of the search subject after normalization; They are the current movement direction of the search subject and the results of the previous three action decisions, arranged in reverse order of the time when the actions occurred; is the speed of the search subject; m is the search range control coefficient, which takes a value of 1 or 2; if it is 1, the search range is a complete circle; if it is 2, the search range is a combination of multiple sectors;
[0007] Step 2: Design of action space for sea area search. Assume that the speed of the search subject remains unchanged during a single round of search. In each step, the search subject only decides the heading change Δα of the current step:
[0008]
[0009] Among them, α max is the maximum heading change, which needs to be set according to the heading performance and time step of the search subject; n is the control parameter of the heading change;
[0010] Step 3: Design of reward function for sea area search and establishment of sparse reward function:
[0011]
[0012] Where d is the Euclidean distance between the search subject and the search target; k is a constant that controls the penalty for unsuccessful decisions during the search process, k∈(0,1). The actual value should be such that k<<1, such as k=0.01, so as to maintain the dominance of the final search result reward;
[0013] Step 4. Design of reinforcement learning algorithm for sea area search. The input of the sea area search agent is a state vector of size 1×25, and the output is a scalar in the range of [-1,1] as the expectation of the action probability distribution;
[0014] Step 5. Based on the established sea area search simulation environment and intelligent agent model, the PPO algorithm is used to implement model training.
[0015] Furthermore,
[0016] When n is 1, They are respectively the horizontal coordinate of the upper left corner position, the vertical coordinate of the upper left corner position, the horizontal coordinate of the lower right corner position, and the vertical coordinate of the lower right corner position of the normalized search area rectangle;
[0017] When n is 2, They are the normalized horizontal coordinate of the center position of the search area circle, the vertical coordinate of the center position, and the radius of the circle. Set to 0.
[0018] Furthermore, each sector of the search range has a common vertex and does not overlap with each other.
[0019] Furthermore, the search scope is:
[0020]
[0021] in is a fan-shaped representation vector, which is expressed as in is the clockwise starting direction of the fan, is the clockwise end direction of the fan, is the sector radius; according to the actual application scenario in the sea area, the maximum number of sectors supported is 4. If it is less than 4, the subsequent m * Fill with [0,0,0].
[0022] Furthermore, in the design of the reinforcement learning algorithm for sea area search, the intelligent agent model structure is proposed to adopt a fully connected neural network, the number of hidden layers is set to 2, the number of hidden layer nodes is set to 128, and the Tanh activation function is used.
[0023] Furthermore, in the design of the reinforcement learning algorithm for sea area search, the output value of the network model is set as μ, and the specific method of obtaining the decision action according to the output value of the network model is:
[0024] Sampling is done according to the normal distribution N(μ,1), and the sampling result x is obtained. The Tanh function is used to map it to the range (-1,1) and written as y.
[0025] According to the action value range, [-1,1] is divided into several intervals, and then the corresponding heading change is obtained according to the interval index where y is located, which is the final decision action a.
[0026] Furthermore, in the design of the reinforcement learning algorithm for sea area search, the corresponding probability density function value of the sampling result is calculated as the action selection probability, as shown in the following formula:
[0027]
[0028] Among them, s is the current state, θ is the current network model parameter, and σ is the distribution standard deviation.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] The present invention improves the applicability of the model at the expense of training time cost. DETAILED DESCRIPTION
[0034] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described below in conjunction with specific implementation methods.
[0035] This application proposes a reinforcement learning method for sea area search, which specifically includes:
[0036] (1) State space design for sea area search
[0037] Set the state space to vector form:
[0038]
[0039] (1-1)n is the search area control coefficient, which takes a value of 1 or 2. If it is 1, the search area is a rectangle; if it is 2, the search area is a circle.
[0040] (1-1-1) When n is 1, They are respectively the normalized horizontal coordinate of the upper left corner position, the vertical coordinate of the upper left corner position, the horizontal coordinate of the lower right corner position, and the vertical coordinate of the lower right corner position of the search area rectangle.
[0041] (1-1-2) When n is 2, They are the normalized horizontal coordinate of the center position of the search area circle, the vertical coordinate of the center position, and the radius of the circle. Set to 0.
[0042] (1-2) They are respectively the horizontal and vertical coordinates of the current position of the search subject after normalization.
[0043] (1-3) They are the current movement direction of the search subject and the results of the previous three action decisions, arranged in reverse order of the time when the actions occurred.
[0044] (1-4) To search for the speed of the subject.
[0045] (1-5) m is the search range control coefficient, which takes a value of 1 or 2. If it is 1, the search range is a complete circle; if it is 2, the search range is a combination of multiple sectors, each sector has a common vertex and does not overlap.
[0046]
[0047] (1-5-1) is a fan-shaped representation vector, which is expressed as in is the clockwise starting direction of the fan, is the clockwise end direction of the fan, is the sector radius. According to the actual application scenario in the sea area, the maximum number of sectors supported is 4. If it is less than 4, the subsequent m * Fill with [0,0,0].
[0048] (2) Action space design for sea area search
[0049] Assume that the speed of the search subject remains unchanged during a single round of search, and the search subject only decides the heading change Δα of the current step in each step:
[0050]
[0051] Among them, α maxis the maximum heading change, which needs to be set according to the heading performance of the search subject and the time step; n is the control parameter of the heading change. The larger n is, the smaller the granularity of heading change control is. It is set by human experience and is generally taken as 10.
[0052] (3) Reward function design for ocean search
[0053] Building a sparse reward function:
[0054]
[0055] Where d is the Euclidean distance between the search subject and the search target. k is a constant that controls the penalty for unsuccessful decisions during the search process, k∈(0,1). The actual value should be such that k<<1, such as k=0.01, so as to maintain the dominance of the final search result reward. Obviously, during the search process, the smaller d is, the larger the reward function value is (less than 0 and the smaller the absolute value is), and the reward function value is maintained in the interval (-k,0) to guide the search subject to get as close to the search target as possible in as few steps as possible.
[0056] (4) Design of reinforcement learning algorithm for ocean search
[0057] The input of the sea area search agent is a state vector of size 1×25, and the output is a scalar in the range of [-1,1] as the expectation of the action probability distribution. The agent model structure is proposed to adopt a fully connected neural network, with the number of hidden layers set to 2, the number of hidden layer nodes set to 128, and the Tanh activation function.
[0058] Suppose the output value of the network model is μ. The specific method to obtain the decision action based on the output value of the network model is: sample according to the normal distribution N(μ,1) to obtain the sampling result x, and use the Tanh function to map it to the range of (-1,1), written as y. Divide [-1,1] into several intervals according to the action value range, and then obtain the corresponding heading change according to the interval index of y, that is, the final decision action a. In addition, calculate the corresponding probability density function value of the sampling result as its action selection probability, as follows:
[0059]
[0060] Among them, s is the current state, θ is the current network model parameter, and σ is the distribution standard deviation.
[0061] Based on the established sea area search simulation environment and intelligent agent model, the PPO algorithm is used to implement model training.
[0062] In summary, the above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited thereto. All equivalent changes and modifications made according to the patent scope of the present invention and the contents of the specification are within the scope covered by the patent of the present invention.
Claims
1. A reinforcement learning method for sea area search, characterized in that: The following steps are involved: Step 1: Design the state space for sea area search and set the state space to vector form: Where n is the search area control coefficient, which can be 1 or 2. If it is 1, the search area is a rectangle; if it is 2, the search area is a circle. They are respectively the horizontal coordinate and the vertical coordinate of the current position of the search subject after normalization; They are the current movement direction of the search subject and the results of the previous three action decisions, arranged in reverse order of the time when the actions occurred; is the speed of the search subject; m is the search range control coefficient, which takes a value of 1 or 2; if it is 1, the search range is a complete circle; if it is 2, the search range is a combination of multiple sectors; Step 2: Design of action space for sea area search. Assume that the speed of the search subject remains unchanged during a single round of search. In each step, the search subject only decides the heading change Δα of the current step: Among them, α max is the maximum heading change, which needs to be set according to the heading performance and time step of the search subject; n is the control parameter of the heading change; Step 3: Design of reward function for sea area search and establishment of sparse reward function: Where d is the Euclidean distance between the search subject and the search target; k is a constant that controls the penalty for unsuccessful decisions during the search process, k∈(0,1). The actual value should be such that k<<1, such as k=0.01, so as to maintain the dominance of the final search result reward; Step 4. Design of reinforcement learning algorithm for sea area search. The input of the sea area search agent is a state vector of size 1×25, and the output is a scalar in the range of [-1,1] as the expectation of the action probability distribution; Step 5. Based on the established sea area search simulation environment and intelligent agent model, the PPO algorithm is used to implement model training.
2. The method according to claim 1, characterized in that: When n is 1, They are respectively the horizontal coordinate of the upper left corner position, the vertical coordinate of the upper left corner position, the horizontal coordinate of the lower right corner position, and the vertical coordinate of the lower right corner position of the normalized search area rectangle; When n is 2, They are the normalized horizontal coordinate of the center position of the search area circle, the vertical coordinate of the center position, and the radius of the circle. Set to 0.
3. The method according to claim 1, characterized in that: Each sector of the search range has a common vertex and does not overlap.
4. The method according to claim 3, characterized in that Search scope: in is a fan-shaped representation vector, which is expressed as in is the clockwise starting direction of the fan, is the clockwise end direction of the fan, is the sector radius; according to the actual application scenario in the sea area, the maximum number of sectors supported is 4. If it is less than 4, the subsequent m * Fill with [0,0,0].
5. The method according to claim 1, characterized in that: In the design of reinforcement learning algorithm for sea area search, the intelligent agent model structure is proposed to adopt a fully connected neural network, the number of hidden layers is set to 2, the number of hidden layer nodes is set to 128, and the Tanh activation function is used.
6. The method according to claim 5, characterized in that In the design of reinforcement learning algorithm for sea area search, the output value of the network model is set as μ. The specific method of obtaining the decision action based on the output value of the network model is as follows: Sampling is done according to the normal distribution N(μ,1), and the sampling result x is obtained. The Tanh function is used to map it to the range (-1,1) and written as y. According to the action value range, [-1,1] is divided into several intervals, and then the corresponding heading change is obtained according to the interval index where y is located, which is the final decision action a.
7. The method according to claim 6, characterized in that In the design of reinforcement learning algorithm for sea area search, the corresponding probability density function value of the sampling result is calculated as the action selection probability, as shown in the following formula: Among them, s is the current state, θ is the current network model parameter, and σ is the distribution standard deviation.