A control method, device and medium for collecting drone information

Abstract

The present invention discloses a control method for drone information collection, comprising: defining the process of drone information collection in a continuous state and action space, taking maximizing the amount of information collected by the drone from multiple nodes within a given time as the optimization goal; converting the optimization goal solution for drone information collection in a continuous state and action space with state estimation errors into the solution of an optimal control strategy in robust reinforcement learning; and utilizing a robust Actor-Critic learning algorithm based on policy gradients to obtain the optimal control strategy in a continuous state and action space with state estimation errors. The present invention, for the first time, proposes how to handle the problem of autonomous control of information collection in a comprehensive situation in a continuous state and action space, where there are estimation errors in the drone state and the information of the collected information nodes is unknown, and converts the problem into the solution of an optimal control strategy in robust reinforcement learning, and finally solves it using a robust Actor-Critic learning algorithm, thereby expanding the application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of drone communication technology, and more specifically, to a control method, device, and medium for collecting drone information. Background Art

[0002] Drones, due to their flexible deployment and low cost, are widely used in military and civilian communications, and hold broad application prospects. Drone communications primarily take the form of coverage communications, relay communications, and information collection. Using drones for information collection offers new technical means and implementation options, with applications in mobile communications, the Internet of Things, and other fields. Furthermore, drones' controllable location and trajectory greatly facilitate information collection.

[0003] Currently, many research results have been achieved in collecting information by utilizing the controllable position / trajectory of drones, mainly including two situations: known information and unknown information.

[0004] When information is known, the drone information collection problem is primarily modeled as an optimization problem (primarily non-convex). Given information such as the drone's position / trajectory and the locations of the nodes being collected, the amount of information collected and energy consumption are used as optimization targets. Non-convex optimization methods are then used to control the drone's position / trajectory and achieve optimal performance. When information is unknown, drone information collection primarily employs methods such as reinforcement learning. Through heuristic learning, the drone's position / trajectory is controlled based on limited environmental and reward information to achieve optimal performance.

[0005] When using drones for information collection, the location of the nodes being collected is often unknown, necessitating the use of methods such as reinforcement learning to adjust their position and trajectory by acquiring limited environmental and reward information. Furthermore, drone information collection applications often face environmental obstructions (such as those caused by tall buildings in urban environments, trees in jungle environments, and buildings indoors or underground). This prevents the drone from obtaining accurate position information from external positioning systems (such as Beidou or GPS), requiring it to rely on its own sensors to estimate its state (including its position). Due to incomplete information and limited algorithm performance, errors are inevitable in the drone's state estimation.

[0006] Existing learning-based methods for controlling drone position and trajectory generally require accurate knowledge of the drone's state, including its position. When the acquired drone position information contains errors, if the position information of the nodes being collected is available, the drone's position error can be modeled and the drone's position and trajectory optimized based on this error model. However, when the collected node position information is unavailable and the drone's position contains errors, controlling the drone's position and trajectory through learning to optimize information collection presents a significant challenge, and currently no publicly available methods exist.

[0007] Based on the aforementioned technical status, the possible methods proposed by conventional optimization ideas are generally to use the estimated position information of the drone as the state space and discretize the state space. The corresponding learning algorithm is generally suitable for situations with a small number of states and cannot be applied to scenarios with a large number of state spaces or continuous state spaces. In addition, the drone motion model proposed by conventional optimization ideas is generally relatively simple and cannot fully reflect the motion model and control model of drones such as fixed-wing drones during flight. Therefore, there is a large room for improvement in how to control the drone position / trajectory through learning to achieve information collection optimization. Summary of the Invention

[0008] In response to at least one defect or improvement need in the prior art, the present invention provides a control method, device, and medium for drone information collection, which are used to provide a solution for the problem of autonomous control of information collection in the case of continuous control variables and state variables, when there is an estimation error in the drone state, and when the information of the node from which the information is collected is unknown.

[0009] To achieve the above objectives, according to a first aspect of the present invention, a method for controlling drone information collection is provided, comprising:

[0010] Define the process of drone information collection in continuous state space and action space, and take maximizing the amount of information collected by drones from multiple nodes within a given time as the optimization goal;

[0011] The optimization objective solution of UAV information collection with state estimation error in continuous state space and action space is transformed into the optimal control strategy solution in robust reinforcement learning;

[0012] The robust Actor-Critic learning algorithm based on policy gradient is used to obtain the optimal control strategy in continuous state space and action space with state estimation errors.

[0013] Furthermore, the use of the robust Actor-Critic learning algorithm based on policy gradient to obtain the optimal control strategy in the continuous state space and action space with state estimation error includes:

[0014] Get an approximation of the robust action value using the robust temporal difference algorithm;

[0015] Based on the obtained approximate value of the robust action value, a robust Actor-Critic learning algorithm is used to obtain a solution to the optimal control strategy function, that is, to obtain the optimal control strategy.

[0016] Furthermore, the expression of the robust action value includes:

[0017]

[0018] in, represents the robust action value corresponding to the robust state value under the initial state s, s∈S, S represents the state space; E P {·} represents the expected cumulative information amount obtained when multiple state transitions are P; min P E P {·} represents the expected cumulative information amount collected in the worst case corresponding to different multiple state transitions P; T max represents the maximum number of decisions made by the drone within the total flight time T; γ represents the discount factor, satisfying γ∈[0,1); c(s i ,a i ) indicates that the drone is in state s i The normalized value of the sum of all node information collected when a i represents the action taken by the drone at the i-th decision, a i ∈A, A represents the action space; a represents the initial action; π θ represents the control strategy determined by the parameter θ;

[0019] The state space expressions include:

[0020]

[0021] Among them, x, y, and z represent the three-dimensional coordinates of the drone’s position at time t; h min and h max They represent the minimum and maximum values of the UAV’s flight altitude respectively; α represents the vertical climb angle of the UAV; β represents the horizontal yaw angle of the UAV; (·) T Represents a matrix transpose operation;

[0022] The expression of the action space includes:

[0023]

[0024] Where v represents the flight speed of the UAV; u1 and u2 represent the angular rate of the yaw angle and the angular rate of the climb angle respectively; c1 and c2 represent the maximum angular rate of the yaw angle and the maximum angular rate of the climb angle respectively; v min and v max Respectively represent the minimum and maximum flight speed of the drone.

[0025] Furthermore, the optimal control strategy function includes:

[0026]

[0027] Among them, θ * represents the optimal control strategy, Θ represents the set of values of parameter θ, E s＜Φ(S) It means to find the expectation when the initial state s is distributed as Φ(S), Φ(S) represents the distribution of the initial position of the UAV, Represents the robust state value under the state space S.

[0028] Furthermore, the expression of the robust state value includes:

[0029]

[0030] in, It represents the robust state value under the initial state s, which is the expected cumulative information that can be collected in the worst case.

[0031] Furthermore, the drone is in state s i The expression for the normalized value of the sum of all node information collected at the time includes:

[0032]

[0033] Among them, ρ n represents the signal-to-noise ratio between the UAV and the nth node at the time of the i-th decision, Indicates that the drone is in state s i The maximum value of the signal-to-noise ratio of N nodes collected at the time; log2(1+ρ n ) represents the amount of information collected by the UAV at the nth node; It represents the sum of the information of N nodes collected by the UAV during the i-th decision.

[0034] According to a second aspect of the present invention, an electronic device is also provided, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit is enabled to perform the steps of any one of the above methods.

[0035] According to a third aspect of the present invention, a storage medium is provided, which stores a computer program executable by an access authentication device. When the computer program runs on the access authentication device, the access authentication device can perform the steps of any of the above methods.

[0036] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0037] (1) The present invention proposes for the first time how to deal with the problem of autonomous control of information collection in a comprehensive situation where there are estimation errors in the drone state and the information of the nodes from which information is collected is unknown under continuous state and action space, and transforms this problem into an optimal control strategy solution in robust reinforcement learning. Finally, the robust actor-critic learning algorithm is used to solve the model in the drone information collection scenario in the field of drone communication technology, thereby expanding the application scenarios of controlling drone information collection and filling the technical gap of how to deal with the problem of autonomous control of information collection in a comprehensive situation where there are estimation errors in the drone state and the information of the nodes from which information is collected is unknown under continuous state and action space.

[0038] (2) The present invention utilizes a robust actor-critic learning algorithm to solve the model in the drone information collection scenario in the field of drone communication technology. In fact, the algorithm can also be applied to other communication application scenarios such as drone coverage communication and relay communication. At the same time, the optimization objective of the present invention can also be changed to an indicator including factors such as communication quality and energy consumption, and the proposed method is still applicable. In addition, the proposed robust actor-critic learning algorithm based on policy gradient can actually be applied to any application scenario where the control strategy is differentiable with respect to parameters, and has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of a method for controlling drone information collection provided by an embodiment of the present invention;

[0041] Figure 2 A block diagram of an electronic device suitable for implementing the above-described method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0043] The terms "including," "having," and any variations thereof in the specification, claims, or drawings of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units but may optionally include steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0044] refer to Figure 1 In one embodiment, a control method for collecting drone information may mainly include the following steps.

[0045] Step 1: Define the process of information collection by the UAV in the continuous state space and action space, and take maximizing the amount of information collected by the UAV from multiple nodes in a given time as the optimization goal.

[0046] Specifically, we first describe the problem of drone information collection in a continuous state space and action space. The optimization objective is to maximize the amount of information a drone can collect from multiple nodes within a given timeframe. This optimization considers the constraints of the drone's actual motion and control models, as well as the constraints on its state variables (state space) and control variables (action space). The drone information collection process can be described as follows: starting from a starting state, taking specific actions, collecting information, and transitioning to the next state based on the motion model; then, taking specific actions in the transitioned state, collecting information, and transitioning to the next state based on the motion model; and repeating this cycle until the maximum information collection time is reached.

[0047] The problem of drone information collection under continuous state variables (state space) and control variables (action space) can be specifically described as follows: Consider a scenario where a drone collects information from N nodes on the ground. The initial position of the drone is represented as S0 = (x0, y0, z0) T ,in(·) T is a matrix transpose operation. The position of the drone at any time t is expressed as S = (x, y, z) T , and h min ≤z≤h max , h min and h maxare the lower and upper limits of the UAV’s flight altitude respectively; the UAV’s flight speed is expressed as v min ≤v≤v max , v min and v max They represent the lower and upper limits of the UAV’s flight speed respectively. The basic motion equation of the UAV is:

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] in, is the derivative of the matrix or variable with respect to time t; α∈[-π / 2,π / 2] is the vertical climb angle of the UAV; β∈[-π,π] is the horizontal yaw angle of the UAV; u1 and u2 are the angular rates of the yaw angle and the climb angle, respectively, and satisfy |u1|≤c1 and |u2|≤c2, where c1 and c2 represent the maximum angular rate of the yaw angle and the maximum angular rate of the climb angle, respectively.

[0054] This invention, based on the actual dynamics of drones, considers continuous control variables and state variables, making it applicable to scenarios with a large or continuous state space. It can fully reflect the motion and control models of drones, such as fixed-wing drones, during flight. Furthermore, this invention proposes, for the first time, a solution for autonomous control of drone information collection in a continuous state and action space, where there are estimation errors in the drone's state and the information of the nodes being collected is unknown.

[0055] Step 2: Convert the optimization objective of drone information collection with state estimation error in continuous state space and action space into the optimal control strategy solution in robust reinforcement learning.

[0056] In the case of drone state estimation errors, considering that the true state is unavailable and the estimated state is a random variable around the true state, the drone's estimated, erroneous state information is used as the state variable, and the drone information collection problem is modeled as an optimization problem based on the estimated state (rather than the actual true state). Furthermore, considering the uncertainty (or randomness) of the estimated state information during state transitions, the drone information collection problem with state estimation errors in continuous state and action spaces is modeled as a robust reinforcement learning problem, and elements such as the state space and action space are defined.

[0057] The state variables of the UAV flight are expressed as X(t) = (x(t), y(t), z(t), β(t), α(t)) T , the control variable is U(t)=(v(t),u1(t),u2(t)) T , the state variables and control variables of the system are all time-related functions (in this invention, all symbols without time variables are the same as symbols with time variables, such as X(t) can be expressed as X). The decision time interval of the drone is expressed as Δt, and this time interval is small enough, then the drone will make T decisions within the total flight time T. max =T / Δt decisions, the time for the i-th decision is i*Δt (i=0,1,…,T max -1). The amount of information collected by the UAV from the nth (n=1,2,…,N) ground node is log2(1+ρ n ), where ρ n is the signal-to-noise ratio between the UAV and the nth node at the time of the i-th decision (corresponding to time i*Δt), which is related to the distance between the UAV and the node and changes with the change of the UAV position / trajectory. In practice, when the UAV is collecting information, it can obtain the signal-to-noise ratio ρ by receiving the signal sent by the node. n Without knowing the location information of the nodes, the drone also controls its flight through autonomous learning based on the signal-to-noise ratio information it obtains. Therefore, the sum of the information collected by the drone from N nodes in the i-th decision is Considering the total flight time of the drone is T, the total amount of information collected by the drone from N nodes within time T is Let Φ(S) represent the distribution of the initial position of the UAV, and the control strategy (or control variable) U(t) of the UAV is random, then the expected value of the amount of information collected by the UAV is Among them, E U {·} is the expected value of the amount of information collected when the control strategy U(t) is adopted, E Φ(S) {·} is the expectation of the initial state variable distribution. Therefore, the UAV information collection problem can be modeled as finding the optimal control strategy U(t) so that the amount of information collected under a given initial state distribution is The maximum can be expressed as the following mathematical problem:

[0058]

[0059]

[0060] α∈[-π / 2,π / 2],β∈[-π,π]

[0061] |u1|≤c1,|u2|≤c2,v min ≤v≤v max (2)

[0062] Where, Ξ=[v min ,v max ]×[-π,π]×[-π / 2,π / 2] is the strategy space of the control variables; α0 and β0 are the UAV’s initial position S0=(x0,y0,z0) T Climb angle and yaw angle at time; X0=(x0,y0,z0,β0,α0) T <Φ(S) means that the initial state X0 obeys the distribution Φ(S); is the vector representation of the UAV motion model, and its expansion is formula (1).

[0063] Note that the state variables in formulas (1) and (2) (including position information and climb angle and yaw angle information) represent the actual position information, climb angle, and yaw angle information of the UAV. When there are estimation errors in the position information, climb angle, and yaw angle information, the position information, climb angle, and yaw angle information in formulas (1) and (2) are unknown, which means that the optimal control strategy in formula (2) cannot be solved. In other words, when there are estimation errors in the state, the current existing methods cannot obtain the optimal solution of formula (2) (or even cannot solve it) because the UAV only knows the state with errors and does not know how the actual state is transferred (i.e., it does not know Φ(S)).

[0064] It is also noted that even when there are estimation errors in the information, the state change of the UAV is still constrained by formula (1) regardless of the control strategy adopted. Since the estimated position, climb angle, and yaw angle information are generally random variables around their true values, this paper uses a robust reinforcement learning method to analyze the optimal control problem of the UAV when there are estimation errors in the state information.

[0065] The problem of collecting information from drones with state estimation errors in continuous state space and action space is modeled as the optimal strategy search problem in robust reinforcement learning. Specifically, since the estimated state information of the drone (including position information, climb angle, and yaw angle information) is a random variable around its true state, when the drone changes from a certain estimated state Si (The corresponding real state is S p ) After a certain control strategy or action U(t) is adopted, its real state is transferred to the actual state S according to formula (1) q (The corresponding estimated state is S j ). Due to S j For around S q random variables, and the drone can only know S j , then S i to S j The state transition of is random. Note that the actual state information of the UAV (including position information and climb angle and yaw angle information) is limited, then S i to S j is also limited, but its true state information S p and S q It may have exceeded the scope of its status information limit.

[0066] The error-free state information obtained by the UAV is used as the state variable X(t) = (x(t), y(t), z(t), β(t), α(t)) T The information collection optimization problem based on the estimated error state information can be modeled as the optimal policy search problem in robust reinforcement learning, as follows.

[0067] Step 2.1, define the state space as S, expressed as:

[0068]

[0069] This state space is a continuous state space and is the estimated state information, not the real state information. In the present invention, the continuous state space is divided into K sufficiently small parts (K is a value approaching infinity), and the state of the drone at the i-th decision (corresponding to time i*Δt) is recorded as s i (s i ∈S). Note that when the continuous state space is not divided, s i It can also be expressed as the state at the corresponding decision time.

[0070] Step 2.2, define the action space as A, expressed as:

[0071]

[0072] The action space is a continuous action space. In the present invention, the continuous action space is divided into W sufficiently small parts (where W is a value approaching infinity), and the action taken by the drone at the i-th decision (corresponding to time i*Δt) is recorded as a i (a i∈A). Note that when the continuous action space is not divided, a i It can also be expressed as the action taken at the corresponding decision time; at the same time, the above control strategy (or control variable) U(t) is the action taken at time t.

[0073] Step 2.3, define the state transition probability variable as and in is a set of probability vectors. When defining state transition, the entire state space is divided into K sufficiently small parts. Since the state space S is a continuous state space, K is a large value. When directly considering the continuous state space without dividing it, the transition probability variable is the conditional probability distribution on the state space S.

[0074] In robust reinforcement learning, when L state transitions are performed, the state transitions are recorded as the L state transition probability matrix The realization of (P1, P2, ..., P L ),in For the convenience of expression, multiple state transitions in robust reinforcement learning are expressed as That is, the multiple transition probability matrix The set of realizations (P1, P2, ...).

[0075] Step 2.4, define the drone in the i-th (i=0,1,…,T max -1) Take action a when making the decision i Transfer to state s i+1 , then the immediate reward is the drone in state s i The normalized value of the sum of all node information collected at the time is:

[0076]

[0077] in, The drone is in state s i The maximum value of the signal-to-noise ratio of N nodes collected at the time, then 0≤c(s i ,a i )≤1. At the same time, define the discount factor γ, satisfying γ∈[0,1). The drone can obtain the signal-to-noise ratio of all N nodes when collecting information, so it can be obtained

[0078] Step 2.5, define the control strategy as π(a i |s i )(corresponding to the control strategy U(t) in formula (2), that is, in state s i The action taken when When the action space is not discretely divided or the number of state spaces after discrete processing is very large, the control strategy is a distribution function, which can be expressed as π θ (a i |s i )(or π θ ), which is the control strategy determined by the parameter θ.

[0079] Step 2.6: Define the robust state value function under the initial state s (s∈S) as the expected cumulative information that can be collected in the worst case, that is:

[0080]

[0081] E P {·} represents the expected cumulative information amount obtained when multiple state transitions are P; min P E P {·} represents the expected cumulative information amount collected in the worst case corresponding to different multiple state transitions P. At the same time, define V π The robust action value function (i.e., Q-value function) corresponding to (s) is:

[0082]

[0083] Therefore, the optimization problem of formula (2) can be further transformed into the problem of finding the optimal control strategy in the robust reinforcement learning problem (S, A, P, c, γ), that is, finding the solution of the following optimal control strategy function:

[0084]

[0085] Among them, E s＜Φ(S) To find the expectation when the initial state s is distributed as Φ(S), Θ is a set, which is the set consisting of the range of values of θ. The solution of formula (6) is the optimal control strategy of the UAV based on the information collection optimization problem with estimated error state information, which is also the optimal control strategy to be found in this invention. Formula (6) represents the solution to make The maximum θ value, θ* represents the optimal control strategy, argmax θ∈Θ It means to find θ in Θ so that maximum.

[0086] Step 3: Use the robust Actor-Critic learning algorithm based on policy gradient to obtain the optimal control strategy in the continuous state space and action space with state estimation error.

[0087] To solve the optimal control strategy for the robust reinforcement learning problem in formula (6), the present invention proposes to use a robust actor-critic learning algorithm based on policy gradient to find the optimal control strategy. To implement the actor-critic learning algorithm, the following steps can be taken.

[0088] Step 3.1: Use the robust time difference algorithm to obtain the approximate value of the robust action value, that is, to obtain the approximate value of the Q value in formula (5). λ (s,a)) are as follows:

[0089] Input: T C , β l And the given control strategy π θ

[0090] Initialization: λ and s0

[0091] 1: Based on the given control strategy π θ (·|s0) selects action a0

[0092] 2: for l=0,1,…,T C -1 do

[0093] 3: Observe the generated c l and s l+1 , where c l Given by formula (3)

[0094] 4: Based on the given strategy π θ (·|s l+1 ) Select action a l+1

[0095] 5: where Q λ (s,a) is an approximation of the Q-value function determined by the parameter λ, Q λ (s,a) can be an approximation of the Q-value function consisting of a linear function or a neural network

[0096] 6:

[0097] 7: in is the gradient of the approximate value function Q with respect to λ

[0098] 8: end for

[0099] Output: λ

[0100] Step 3.2: Based on the obtained approximate Q value, the robust Actor-Critic learning algorithm is used to obtain the solution of formula (6), which is the optimal control strategy. ) are as follows:

[0101] Input: T max , T C , α i ,γ,R,M

[0102] Initialization: Initial parameters θ0 in the control strategy

[0103] 1: for i=0,1,…,T max -1 do

[0104] 2: Run the above robust time difference algorithm T C Second-rate

[0105] 3:

[0106] 4: For all states s∈S, take V i (s)←∑ a∈A π θ (a|s)Q i (s,a)

[0107] 5: for j=1,2,…,M do

[0108] 6: Sample from a geometric distribution with parameter 1-γ+γR to obtain the number of decisions T j

[0109] 7: Get the sample value from the initial state distribution Φ(S)

[0110] 8: Based on strategy Sampling generation state - action trajectory

[0111] 9: in is the control strategy π θ Gradient with respect to θ

[0112] 10:

[0113] 11: Based on strategy Sampling generation state - action trajectory

[0114] 12:

[0115] 13:

[0116] 14: end for

[0117] 15:

[0118] 16: θ i+1 ←∏ Θ (θ i -α i g i )

[0119] 17: end for

[0120] Output:

[0121] It can be seen that when the step size α i Satisfy α i >0, and When T max →∞, the above robust Actor-Critic learning algorithm almost certainly converges to the optimal control policy.

[0122] The present invention proposes for the first time how to deal with the problem of autonomous control of information collection in a comprehensive situation where there are estimation errors in the drone state and the information of the nodes from which information is collected is unknown under continuous state and action space, and transforms this problem into an optimal control strategy solution in robust reinforcement learning. Finally, the robust Actor-Critic learning algorithm is used to solve the model in the drone information collection scenario in the field of drone communication technology, thereby expanding the application scenarios of controlling drone information collection and filling the technical gap in how to deal with the problem of autonomous control of information collection in a comprehensive situation where there are estimation errors in the drone state and the information of the nodes from which information is collected is unknown under continuous state and action space.

[0123] This invention utilizes a robust actor-critic learning algorithm to solve models in drone information collection scenarios within the field of drone communication technology. This algorithm is also applicable to other communication scenarios, such as drone coverage communication and relay communication. Furthermore, the proposed method remains applicable even when the optimization objective includes factors such as communication quality and energy consumption. Furthermore, the proposed robust actor-critic learning algorithm based on policy gradients can be applied to any scenario where the control strategy is differentiable with respect to parameters, demonstrating its scalability.

[0124] Figure 2 The block diagram schematically shows an electronic device suitable for implementing the method described above according to an embodiment of the present invention. Figure 2The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0125] like Figure 2 As shown, the electronic device 1000 described in this embodiment includes: a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. The processor 1001 may, for example, include a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (such as an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0126] Various programs and data required for the operation of the system 1000 are stored in the RAM 1003. The processor 1001, ROM 1002, and RAM 1003 are connected to each other via a bus 1004. The processor 1001 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than the ROM 1002 and RAM 1003. The processor 1001 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0127] According to an embodiment of the present disclosure, electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to bus 1004. System 1000 may also include one or more of the following components connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or a modem. Communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in drive 1010 as needed, so that computer programs read therefrom can be installed into storage section 1008 as needed.

[0128] The method flow according to the embodiment of the present disclosure can be implemented as a computer software program. For example, the embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0129] Embodiments of the present invention further provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the methods according to the embodiments of the present disclosure.

[0130] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In an embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include one or more memories other than the ROM 1002 and / or RAM 1003 described above.

[0131] It should be noted that the functional modules in the various embodiments of the present invention can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product.

[0132] The flowcharts or block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of the present disclosure, and all such combinations and / or couplings fall within the scope of the present disclosure.

[0134] Although the present disclosure has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-described embodiments, but should be determined not only by the appended claims but also by the equivalents of the appended claims.

Claims

1. A control method for collecting drone information, characterized in that: include: Define the process of drone information collection in continuous state space and action space, and take maximizing the amount of information collected by drones from multiple nodes within a given time as the optimization goal; The optimization objective solution of UAV information collection with state estimation error in continuous state space and action space is transformed into the optimal control strategy solution in robust reinforcement learning; Use the robust Actor-Critic learning algorithm based on policy gradient to obtain the optimal control strategy in the continuous state space and action space with state estimation error; The method of using the robust Actor-Critic learning algorithm based on policy gradient to obtain the optimal control strategy in the continuous state space and action space with state estimation error includes: Get an approximation of the robust action value using the robust temporal difference algorithm; Based on the obtained approximate value of the robust action value, a robust Actor-Critic learning algorithm is used to obtain a solution of the optimal control strategy function, that is, to obtain the optimal control strategy; The expression of the robust action value includes: in, Indicates the initial state The robust action value corresponding to the robust state value under , represents the state space; Indicates that multiple state transitions are The expected cumulative information amount obtained under the condition of Indicates multiple different state transitions The expected cumulative amount of information collected in the worst case corresponding to ; Indicates the total flight time of the drone The maximum number of decisions made within a period; represents the discount factor, satisfying ; Indicates that the drone is in status The normalized value of the sum of all node information collected at the time, Indicates that the drone is The action taken at the time of the decision, , represents the action space; Indicates the initial action; Indicated by the parameter Determined control strategy; The state space expressions include: in, Indicates time The three-dimensional coordinates of the drone’s location at that time; and Respectively represent the minimum and maximum values of the drone’s flight altitude; Indicates the climbing angle of the UAV in the vertical direction; Indicates the yaw angle of the drone in the horizontal direction; Represents a matrix transpose operation; The expression of the action space includes: in, Indicates the flight speed of the drone; and denote the angular rate of the yaw angle and the angular rate of the climb angle respectively; and They represent the maximum value of the angular rate of the yaw angle and the maximum value of the angular rate of the climb angle respectively; and Respectively represent the minimum and maximum flight speed of the drone.

2. The control method for collecting drone information according to claim 1, wherein: The optimal control strategy function includes: in, represents the optimal control strategy, Representation parameters The set of values of , Indicates the initial state Distribution When seeking hope, represents the distribution of the initial positions of the UAVs, Representing the state space The robust state value under .

3. The control method for collecting drone information according to claim 2, wherein: The expression of the robust state value includes: in, Indicates the initial state The robust state value under the worst case is the expected cumulative information that can be collected.

4. The control method for collecting drone information according to claim 1, wherein: Drone in status The expression for the normalized value of the sum of all node information collected at the time includes: in, Indicates that the drone is The first decision The signal-to-noise ratio between nodes, Indicates that the drone is in status Collected The maximum value of the signal-to-noise ratio of the nodes; Indicates that the drone is The amount of information collected by each node; Indicates that the drone is The data collected during the decision The sum of the information of the nodes.

5. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit is enabled to perform the steps of the method according to any one of claims 1 to 4.

6. A storage medium, characterized in that It stores a computer program executable by an access authentication device. When the computer program runs on the access authentication device, the access authentication device is enabled to execute the steps of the method according to any one of claims 1 to 4.

Images