Sensor resource scheduling method and system based on combined action space reinforcement learning

Abstract

The invention belongs to the field of sensor resource allocation, and discloses a sensor resource scheduling method and system based on combined action space reinforcement learning. Inputting the system state of the to-be-scheduled sensor networking system into a pre-trained sensor resource allocation model to obtain a combined action, and finally outputting the combined action as a flight target optimal scheduling scheme; according to the method, the combined action space is designed, so that the defect that one discrete action is executed at a time in the discrete action space of a classical single-agent reinforcement learning algorithm can be overcome, one multi-dimensional action combination is executed at a moment, and multiple flight targets are scheduled to track multiple sensors; a sensor networking system is trained by using a reinforcement learning method to obtain a sensor scheduling scheme, so that self-adaptive resource scheduling can be realized, an allocation strategy is dynamically adjusted, and the system performance is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of sensor resource allocation, and in particular relates to a sensor resource scheduling method and system based on combined action space reinforcement learning. Background Art

[0002] The flight target detection and tracking system has gradually entered the rapid development stage from the initial stage. Civil and commercial detection resources continue to emerge, such as sensors, mobile phone base stations and radio and television transmitters. The emergence of these devices provides people with a wealth of detection methods and information sharing. However, it also brings up the problem of how to efficiently integrate detection and communication resources and collaboratively complete perception and communication tasks. Perception resource allocation is to achieve resource scheduling and information sharing between detection equipment and detection targets under different detection models. The solutions to the sensor scheduling problem can be classified into the following types: operations research methods, heuristic scheduling methods, scheduling simulation systems and artificial intelligence methods.

[0003] In recent years, artificial intelligence methods have been widely used in the field of resource allocation. AI-based scheduling methods use artificial intelligence technologies, such as machine learning and deep learning, to intelligently schedule and allocate tasks or resources. This method has the characteristics of adaptability, learning, and efficiency, and can be flexibly adjusted and optimized according to changes in the system and needs. Reinforcement learning is the use of empirical data to map from environmental states to behaviors, and its purpose is to optimize the total return of the environment. However, for most single-agent reinforcement learning algorithms, the action output has only one value, and the output of the neural network selects the action corresponding to the highest action value, and it is impossible to select multiple actions at a time or a multi-dimensional action combination.

[0004] It can be seen from this that the existing single-agent reinforcement learning algorithm can only perform one discrete action at a time in the discrete action space, which cannot meet the scheduling requirements of multiple sensors for multiple flight targets. Summary of the invention

[0005] The purpose of the present invention is to provide a sensor resource scheduling method and system based on combined action space reinforcement learning, so as to solve the technical problem that the existing single-agent reinforcement learning algorithm discrete action space can only execute one discrete action at a time and cannot meet the scheduling needs of multiple sensors for multiple flight targets.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a sensor scheduling method based on combined action space reinforcement learning, comprising: Acquire the system status of the sensor networking system to be scheduled, wherein the system status includes the real-time position of the flight target and the sensor number assigned to the flight target at the current moment; The system state of the sensor networking system to be scheduled is input into the pre-trained sensor resource allocation model to obtain the combined action, and the combined action is output as the optimal scheduling plan for the flight target; The sensor resource allocation model is constructed based on a reinforcement learning algorithm combined with a Markov decision process.

[0007] Furthermore, the training process of the sensor resource allocation model is as follows: Construct a DQN neural network and initialize the network parameters of the DQN neural network. The DQN neural network includes a current network and a target network with the same structure. The initialized DQN neural network is trained using the historical system status of the networked sensor system until the preset training rounds are reached and the training is terminated, and the sensor resource allocation model is output.

[0008] Furthermore, the historical system state of the networked sensor system is expressed as:

[0009] In the formula, Indicates the current position of the flying target. Indicates the sensor number assigned to flight targets a, b, c, d, and e at the current moment.

[0010] Furthermore, the specific steps of using the historical system state of the networked sensor system to train the initialized DQN neural network are as follows: The historical system state of the networked sensor system is input into the initialized DQN neural network, and the mapping combination action is selected based on the greedy algorithm, and the predicted Q value corresponding to the mapping combination action is obtained; According to the selected mapping combination action, the sensor networking system interacts with the environment to obtain the reward value corresponding to the mapping combination action and the system state after executing the mapping combination action; Combine the system state, the mapping combination action, the reward corresponding to the mapping combination action, and the system state after executing the mapping combination action into one experience and store it in the preset experience playback buffer. Repeat the above process until the experience playback buffer reaches the threshold; A preset number of experience values ​​are extracted from the experience replay buffer, and the network parameters of the DQN neural network are updated until the preset number of updates is reached, and the sensor resource allocation model is output.

[0011] Furthermore, the specific steps of implementing environmental interaction with the sensor networking system according to the selected mapping combination action include: According to the current position of the flying target, the Euclidean distance between the flying target and the sensor that dispatches it is calculated; Get the distance reward for each flight target based on Euclidean distance calculation; Determine the relationship between the Euclidean distance and the detection range of the corresponding sensor. If the Euclidean distance is less than the detection range, a positive reward is obtained, otherwise a negative reward is obtained. In each environmental interaction process, a counter is set for each sensor. In each step, if sensor resources are allocated to the flight target, the corresponding sensor count is +1. At the end of the environmental interaction process, it is determined whether the sensor usage time is overloaded. If the sensor is overloaded, a negative reward is obtained, otherwise a positive reward is obtained. In each step, it is determined whether the sensor resources scheduled by the flight target at this moment are consistent with the sensor resources at the previous moment. If they are inconsistent, a sensor resource switch occurs, and the number of times the sensor resource switches are scheduled for each flight target is counted; During the interaction process in this environment, the number of switches is judged. If the number of switches is too many, a negative reward is obtained, otherwise a positive reward is obtained. Among them, the reward for the round is obtained by summing up the switch number reward, the sensor usage time reward and the step length reward.

[0012] Furthermore, the specific steps of extracting a preset number of experience values ​​from the experience playback buffer and updating the network parameters of the DQN neural network include: Extract a preset amount of experience from the experience replay buffer; The extracted experience is input into the DQN neural network as a training sample. The network parameters of the current DQN neural network are updated using the gradient descent algorithm based on the constructed target loss function. The model parameters are updated using the calculated gradients, and the loss function is minimized by adjusting the parameters. According to the set update frequency, the target network parameters are intermittently synchronized with the current network parameters, and finally the sensor resource allocation model is output.

[0013] Furthermore, the loss function of the target is a mean square error loss function:

[0014] In the formula, is the loss function of target a, the total loss is the average of all flying target losses, and N is the number of training samples; is the true Q value of the i-th sample, is the predicted Q value of the i-th sample.

[0015] In a second aspect, the present invention provides a sensor scheduling system based on combined action space reinforcement learning, comprising: A system status acquisition module is used to acquire the system status of the sensor networking system to be scheduled, wherein the system status includes the real-time position of the flight target and the sensor number assigned to the flight target at the current moment; The scheduling scheme output module is used to input the system state of the sensor networking system to be scheduled into the pre-trained sensor resource allocation model to obtain a combined action, and output the combined action as the optimal scheduling scheme for the flight target; wherein the sensor resource allocation model is constructed based on a reinforcement learning algorithm combined with a Markov decision process.

[0016] In a third aspect, the present invention provides a device, comprising: Memory for storing computer programs; A processor is used to implement the steps of the above-mentioned sensor scheduling method based on combined action space reinforcement learning when executing the computer program.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program is used to implement the steps of the above-mentioned sensor scheduling method based on combined action space reinforcement learning.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a sensor scheduling method based on combined action space reinforcement learning. The scheduling method first obtains the system state of the sensor networking system to be scheduled, and then inputs the system state of the sensor networking system to be scheduled into a pre-trained sensor resource allocation model to obtain a combined action, and finally outputs the combined action as the optimal scheduling plan for the flight target; by designing the combined action space, the method can overcome the shortcoming of the discrete action space of the classic single-agent reinforcement learning algorithm that executes one discrete action at a time, and realizes the execution of a multi-dimensional action combination at one time, and the scheduling of multiple sensors for tracking multiple flight targets; the sensor networking system uses the reinforcement learning method to train to obtain a sensor scheduling plan, which can realize adaptive resource scheduling, dynamically adjust the allocation strategy, and optimize the system performance.

[0019] Preferably, in the present invention, the training process of the sensor resource allocation model includes constructing a DQN neural network, initializing network parameters, and using the historical system status of the networked sensor system for training; this method can ensure that the model learns effective sensor resource allocation strategies during the training process, providing reliable model support for subsequent sensor scheduling.

[0020] Preferably, in the present invention, a method for representing the historical system status of the networked sensor system is clarified, including the location of the flight target and the assigned sensor number; this representation method can comprehensively reflect the status information of the sensor networking system and provide accurate data input for model training and scheduling.

[0021] Preferably, in the present invention, the specific steps of using the historical system state of the networked sensor system to train the DQN neural network include inputting the system state, selecting a mapping combination action, interacting with the environment, storing experience, and updating network parameters. This method can ensure that the model continuously learns and optimizes during the training process, thereby improving the generalization ability and scheduling performance of the model.

[0022] Preferably, in the present invention, the specific steps of realizing environmental interaction with the sensor networking system according to the selected mapping combination action include calculating the Euclidean distance, judging the detection range, setting a counter, judging the sensor usage time and the number of sensor resource switching times, etc.; these steps can ensure that the model takes into account multiple factors during the scheduling process, such as the distance between the flight target and the sensor, the detection range of the sensor, the sensor usage time and the number of resource switching times, so as to obtain a more reasonable scheduling solution.

[0023] Preferably, in the present invention, the specific steps of extracting experience values ​​from the experience playback buffer to update the network parameters of the DQN neural network include extracting experience, inputting training samples, updating network parameters, and synchronizing target network parameters. This method can ensure that the model continuously learns and optimizes network parameters during the training process, and improve the scheduling performance and stability of the model.

[0024] Preferably, in the present invention, it is clarified that the target loss function is the mean square error loss function, and a calculation formula for the loss function is given; this loss function can measure the difference between the model predicted Q value and the true Q value, thereby guiding the updating and optimization of the model parameters. By minimizing the loss function, the scheduling performance and accuracy of the model can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a sensor scheduling method based on combined action space reinforcement learning provided by an embodiment of the present invention; Figure 2 A diagram of the structure of a reinforcement learning neural network provided by an embodiment of the present invention; Figure 3 A schematic diagram of a sensor scenario provided by an embodiment of the present invention; Figure 4 A diagram showing the training effect of the DQN algorithm provided in an embodiment of the present invention; Figure 5 A diagram showing a scheduling scheme for a flying target a by a sensor system provided in an embodiment of the present invention; Figure 6 A diagram showing a scheduling scheme for a flying target b by a sensor system provided in an embodiment of the present invention; Figure 7 A flow chart of a sensor scheduling method based on combined action space reinforcement learning provided by the present invention; Figure 8 A schematic diagram of the structure of a sensor scheduling system based on combined action space reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0026] Example 1 As described in the background technology, currently, for most single-agent reinforcement learning algorithms, the action output has only one value, and the output of the neural network selects the action corresponding to the highest action value, and it is impossible to select multiple actions or a multi-dimensional action combination at one time.

[0027] In order to solve the above problems, the present invention provides a sensor scheduling method based on combined action space reinforcement learning. For the scheduling problem of multiple flying targets, to realize the action selection of the intelligent agent at one time, the dimension should be a combination action of the number of flying targets, and a multi-dimensional combination action solution is adopted; first, the action selection in the combined action space is realized in the action selection strategy, and the action selected by the intelligent agent is a multi-dimensional combination action, and the output of the neural network is also changed from the value of an action to the value of the combination action; then the value corresponding to the multi-dimensional action is trained, and the optimal strategy finally obtained is a combination action, which represents the sensor that should be enabled for each flying target.

[0028] The present embodiment is further explained and illustrated below in conjunction with the accompanying drawings: Combination Figure 1 and Figure 2 The sensor scheduling optimization method based on combined action space reinforcement learning in this embodiment is applied to a networked sensor system, which includes parameters such as the number of preset sensors, the sensor detection range, etc., and the detection area of ​​the sensor system includes at least one flying target. In this embodiment, it is assumed that there are Q independent targets q (q=1,2,…,Q) in the scene, and sensors n (n=1,2,…,N) are located in a three-dimensional space, and N is the number of sensors in the sensor system.

[0029] like Figure 1 As shown, this embodiment provides a sensor scheduling method based on combined action space reinforcement learning, including: S1. Initialize the environment state, which includes the sensor position coordinates and detection range, the initial position and motion trajectory of the flight target trajectory, and the sensor number initially corresponding to each flight target.

[0030] In this embodiment, the sensor information includes the detection range and position coordinates of the sensor, etc. The position coordinates of the sensor are represented by three-dimensional space coordinates.

[0031] Specifically, the sensor detection range is set to 100km, and the sensor can detect flying targets less than 100km away from it. Set the sensor usage time to 0. Take the point with an altitude of 0, longitude 88.77902110334314°, and latitude 28.05432313932233° as the origin of the coordinate system (0,0,0) to establish a three-dimensional space coordinate system, with the east longitude direction as the x-axis, the north latitude direction as the y-axis, and the direction perpendicular to the ground upward as the z-axis. The coordinates of the six ground-based sensors are (26, 100,0), (49, 30, 0), (73, 180, 0), (98, 260, 0), (123, 130, 0), and (148, 220, 0); the initial points of the motion trajectories of the three airborne sensors are (30, 90, 45), (100, 110, 45), and (160, 200, 45), and their velocities are (1, -1, 0), (-1, 2, 0), and (-1, 1, 0), respectively, all of which are uniform linear motions.

[0032] Then initialize the positions of the flying targets. Assume that the five flying targets all fly out from the same position (0,0,30), and their speeds and directions are different. The speed of target a is (5,10,0), the speed of target b is (6,9,0), the speed of target c is (5,11,0), the speed of target d is (6,10,0), and the speed of target e is (4,11,0). For each target, the number of sensor switching times is initialized to 0.

[0033] S2. Input the system state into the neural network. The output of the neural network is a multi-dimensional combined action obtained according to the action value function, that is, the scheduling plan of the sensor for the flight target at each moment. The reward function judges the pros and cons of the scheduling plan. Reinforcement learning continuously trains and improves the reward function to optimize the scheduling plan based on the experience value obtained, and finally obtains the optimal scheduling plan.

[0034] In an optional embodiment, step 2 includes: S201. Each flying target corresponds to an initial sensor number, and then in the training of the algorithm, an action is selected for each target through a strategy.

[0035] In this embodiment, the state space includes the position of the sensor and the position of the flying target. The action space is the optional number of sensors, that is, 9. The action combination space is a 5-dimensional combination action realized by 5 layers of nested loops, realizing all possible action combinations.

[0036] The strategy for selecting actions for the target adopts the ε-greedy strategy, and the ε value is set to 0.1, which means that each time an action is selected, there is a 0.1 probability of randomly selecting an action combination, and a 0.9 probability of selecting an action combination with high value based on the experience value.

[0037] The sampling action first takes a random value between 0 and 1. If the value is less than the set ε value, a random action combination is sampled as the action selection of the agent in this step, that is, the sensor number assigned to the target. If the random value is greater than ε, the action combination with high value will be selected according to the action value in the prediction function.

[0038] S202, after obtaining the combined action, the sensor scheduled for the flight target in each step is clear. The reinforcement learning algorithm trains the scene model, setting 300 training rounds, each round contains multiple steps, and the end judgment condition of the step is that the flight target flies out of the detection area of ​​the sensor system.

[0039] The system state is input into the pre-obtained resource allocation model to obtain the resource allocation result of the networked sensor system. The resource allocation result includes the sensor number scheduled for each flight target at each moment and which sensor is enabled to schedule which flight target.

[0040] Among them, the resource allocation model is obtained by using deep reinforcement learning to model the scheduling allocation process of sensors under the flight trajectories of different targets as a Markov decision process.

[0041] A sensor resource allocation scheme can be designed according to the flight trajectory of the flight target, and the scheme can be used to efficiently track the flight target and reduce the energy consumption of the sensor system. Given that the scheduling and allocation of sensors under different flight target trajectories is designed into a Markov decision process (MDP), in order to describe the decision-making process of sensor resource allocation in the sensor system, a state space, action space, reward function and state transition probability matrix can be established to model the sensor resource allocation problem into an MDP model. In this way, the interaction process between the agent and the environment can be described. At time t, the agent is in state s t , and according to the strategy π(a t |s t )Select action a t In reinforcement learning, the policy π(a t |s t ) is a mapping function from state to action probability distribution. According to the transition probability, the state changes from s t Transfer to t+1, and the agent gets the reward value of this action according to the reward function. The goal of the agent is to maximize the long-term expected reward, that is, .

[0042] In this embodiment, the DQN neural network is used to guide the networked sensor system to make the optimal scheduling decision. Specifically, Figure 2 As shown in the figure, the network input layer of DQN receives state information from the environment as the input of the neural network. This state can be pre-set environmental information, such as the starting point of the flight target and the default scheduled sensor. The network contains multiple hidden layers, usually using a deep neural network structure to learn the mapping relationship between complex states and action spaces. The number of nodes and layers in the hidden layer can be designed according to the complexity of the actual problem. The number of nodes in the output layer is equal to the sum of the number of discrete actions that can be executed by different targets in the environment. Each node outputs the predicted Q value of the corresponding mapping combination action. The output layer of the entire network provides the corresponding predicted Q value for each mapping combination action. These Q values ​​can represent the expected rewards that the agent can obtain by performing certain actions in a certain state.

[0043] Furthermore, the acquisition process of the sensor resource allocation model in this embodiment is described in detail, including the following steps: S20201. Build a DQN neural network and initialize network parameters.

[0044] Among them, the DQN neural network includes the current network and the target network with the same structure. The DQN neural network obtains the mapped combination action and the Q value corresponding to the mapped combination action according to the input system state. The mapped combination action is the sensor number assigned to the flight target scheduling at the next moment.

[0045] S20202. Use the system status of the networked sensor system to train the DQN neural network until a preset training round is reached and the training is terminated, and finally a sensor resource allocation plan is obtained.

[0046] In an optional embodiment, the step b comprises: S2020201. Obtain the system status of the sensor network.

[0047] In this embodiment, the system state at time t can be expressed as: ; in Indicates the position of the flying target acquired by the sensor system at the current moment. Indicates the sensor number assigned to the target at the current moment.

[0048] S2020202. Input the system state into the current network, determine the mapping combination action based on the greedy algorithm, and then obtain the predicted Q value corresponding to the mapping combination action.

[0049] In this embodiment, the mapping combination action can be expressed as , Indicates the sensor n that is scheduled to be assigned to the flight target q at the next moment. Here, the ε-greedy strategy is adopted. During the training process, random mapping actions are selected with a probability of ε for exploration, and then the mapping action with the highest predicted Q value is selected with a probability of 1-ε.

[0050] S2020203. Implement environmental interaction with the networked sensor system according to the selected mapping combination action, and obtain a reward value corresponding to the mapping combination action and a system state after executing the mapping combination action.

[0051] In this embodiment, obtaining the reward value corresponding to the mapping combination action includes the following steps: S202020301. According to the current position of the flight target, calculate the Euclidean distance between the flight target and the sensor that dispatches it. The distance formula is: After getting the distance, you can get the corresponding distance reward for each target. Then determine the relationship between the distance and the detection range of the corresponding sensor. If the distance is less than the detection range, a positive reward is obtained. On the contrary, if the distance is greater than the detection range, a negative reward is obtained. In this way, each step will get a corresponding reward value, and the step reward is obtained by setting the corresponding weight for each parameter. .

[0052] S202020302. In each episode, the corresponding final episode reward is obtained. Specifically, a counter is set for each sensor. In each step, sensor resources are allocated to the flight target, and the corresponding sensor count is +1. Finally, the episode determines whether the sensor usage time is overloaded. If the sensor is overloaded, a negative reward is obtained, otherwise a positive reward is obtained. Then, in each step, it is determined whether the sensor resources scheduled by the flight target at this moment are consistent with the sensor resources at the previous moment. If they are inconsistent, a sensor resource switch occurs, and the number of sensor resource switches scheduled for each flight target is counted. At the end of the episode, the number of switches is determined. If the number of switches is too many, a negative reward is obtained, otherwise a positive reward is obtained. These rewards are accumulated to Finally, the reward for this round It is obtained by summing the switching number reward, sensor usage time reward and step reward.

[0053] S2020204. Combine the system state, the mapping combination action, the reward corresponding to the mapping combination action, and the new state after executing the mapping combination action into one experience and store it in the experience playback buffer.

[0054] S2020205. Repeat S2020201- S2020204 until the experience in the experience replay buffer reaches a preset number.

[0055] S2020206. During the training process, a preset number of experience values ​​are extracted from the experience replay buffer, and the network parameters of the DQN neural network are updated until a preset number of updates is reached, and finally a sensor resource allocation plan for the sensor system to the target is obtained.

[0056] In this embodiment, the experience replay mechanism is applied to the DQN neural network training process, and random sampling from the stored historical experience data can be performed for training, which helps to break the correlation between data, improve the efficiency and stability of reinforcement learning training, and make the training process and results more comprehensive.

[0057] Specifically, S2020206 may include: S202020601. Extract a preset amount of experience from the experience replay buffer.

[0058] S202020602. Input the extracted experience as training samples into the DQN neural network, use the gradient descent algorithm to update the network parameters of the current network based on the constructed target loss function, use the calculated gradient to update the model parameters, and minimize the loss function by adjusting the parameters.

[0059] In this embodiment, the loss function of the target is the mean square error loss function (MSE):

[0060] In the formula, is the loss function of target a, the total loss is the average of the five target losses, and N is the number of training samples. is the true Q value of the i-th sample, is the predicted Q value of the i-th sample.

[0061] In this embodiment, the Q-learning algorithm is used to update the network parameters according to the predicted Q value and the actual reward, that is, to find the optimal strategy by learning the Q value function. The update rules are as follows:

[0062] in, Indicates in status The predicted Q value obtained by executing the mapping combination action a under the condition of The reward value obtained after Represents the new state reached after executing the mapping combination action, Represents the learning rate, which is used to control the step size of each update. is a discount factor used to weigh the importance of immediate rewards and future rewards.

[0063] S202020603. According to the set update frequency, intermittently synchronize the target network parameters with the current network parameters.

[0064] In this embodiment, the pseudo code of the specific implementation process of DQN is as follows:

[0065] The sensor resource allocation method based on combined action space reinforcement learning of this embodiment takes the motion trajectory of the flight target as input, calculates the position of each flight target's motion state updated in real time, and inputs the flight target position as the state into the DQN network. The output of the neural network is a combined action, that is, the scheduling plan of the sensor system for each flight target at each moment. Through reinforcement learning training, the optimal scheduling plan of the sensor system for the flight target is finally obtained.

[0066] The sensor resource scheduling optimization method based on combined action space reinforcement learning in the embodiment of the present invention can be applied to a networked sensor system, and the position of the target at each moment is obtained through the trajectory of the flying target. Through reinforcement learning algorithm training, a sensor resource allocation plan for the flying target is obtained, so that sensor resources are used efficiently. At the same time, by establishing a DQN network to solve the optimization problem, the optimization problem existing in the traditional method is avoided through the reinforcement learning algorithm, the system energy consumption is reduced, and the allocation model has a higher generalization ability.

[0067] Furthermore, the effect of the sensor scheduling optimization method based on combined action space reinforcement learning proposed in this embodiment is illustrated by simulation experiments. Assume that the networked sensor system consists of N=9 distributed sensors, of which the coordinates of the six ground-based sensors are (26, 100, 0), (49, 30, 0), (73, 180, 0), (98, 260, 0), (123, 130, 0), (148, 220, 0); the initial points of the motion trajectories of the three air-based sensors are (30, 90, 45), (100, 110, 45), (160, 200, 45), and the speeds are (1, -1, 0), (-1, 2, 0), (-1, 1, 0), all of which are uniform linear motion. Figure 3is the position of each sensor in the sensor system at a certain moment. The sensor detection range is set to 100km, and the sensor can detect flying targets less than 100km away from it. Five flying targets move in a uniform straight line, all flying out from the same position (0,0,30). Their speeds and directions of movement are different. The speed of target a is (5,10,0), the speed of target b is (6,9,0), the speed of target c is (5,11,0), the speed of target d is (6,10,0), and the speed of target e is (4,11,0). The DQN network contains 2 hidden layers, each of which has a size of 128 and uses the ReLU activation function.

[0068] Figure 4 This is a diagram of the training process of the DQN algorithm. It can be found that as the number of training rounds increases, the reward value gradually converges. Figure 5 and Figure 6 The optimal scheduling schemes of the sensor system for flight target a and flight target b are obtained by reinforcement learning algorithm. Compared with the general sensor scheduling method, the method of the present invention takes into account more comprehensive factors, not only the distance affects resource allocation, but also the number of sensor switching times, sensor usage time and other factors, which can improve the overall performance of the system and save system resource consumption to a certain extent.

[0069] The sensor scheduling method based on combined action space reinforcement learning provided in this embodiment realizes resource scheduling for multiple flying targets through combined action space; by analyzing the trajectory, speed and position of the flying target, the system can dynamically adjust the resource allocation plan to more effectively track the target. By using the reinforcement learning algorithm, the networked sensor system can realize adaptive resource scheduling, dynamically adjust the allocation strategy, and achieve optimal system performance.

[0070] Example 2 like Figure 7 As shown, this embodiment provides a sensor scheduling method based on combined action space reinforcement learning, comprising the following steps: Acquire the system status of the sensor networking system to be scheduled, wherein the system status includes the real-time position of the flight target and the sensor number assigned to the flight target at the current moment; The system state of the sensor networking system to be scheduled is input into the pre-trained sensor resource allocation model to obtain the combined action, and the combined action is output as the optimal scheduling plan for the flight target; The sensor resource allocation model is constructed based on a reinforcement learning algorithm combined with a Markov decision process.

[0071] The training process of the sensor resource allocation model is as follows: Construct a DQN neural network and initialize the network parameters of the DQN neural network. The DQN neural network includes a current network and a target network with the same structure. The initialized DQN neural network is trained using the historical system status of the networked sensor system until the preset training rounds are reached and the training is terminated, and the sensor resource allocation model is output.

[0072] Here, the historical system state of the networked sensor system is expressed as:

[0073] In the formula, Indicates the current position of the flying target. Indicates the sensor number assigned to the flight targets a, b, c, d, and e at the current moment; In this embodiment, the specific steps of using the historical system state of the networked sensor system to train the initialized DQN neural network are as follows: The historical system state of the networked sensor system is input into the initialized DQN neural network, and the mapping combination action is selected based on the greedy algorithm, and the predicted Q value corresponding to the mapping combination action is obtained; According to the selected mapping combination action, the sensor networking system interacts with the environment to obtain the reward value corresponding to the mapping combination action and the system state after executing the mapping combination action; Combine the system state, the mapping combination action, the reward corresponding to the mapping combination action, and the system state after executing the mapping combination action into one experience and store it in the preset experience playback buffer. Repeat the above process until the experience playback buffer reaches the threshold; A preset number of experience values ​​are extracted from the experience replay buffer, and the network parameters of the DQN neural network are updated until the preset number of updates is reached, and the sensor resource allocation model is output.

[0074] Specifically, the specific steps of implementing environmental interaction with the sensor networking system according to the selected mapping combination action include: According to the current position of the flying target, the Euclidean distance between the flying target and the sensor that dispatches it is calculated; Get the distance reward for each flight target based on Euclidean distance calculation; Determine the relationship between the Euclidean distance and the detection range of the corresponding sensor. If the Euclidean distance is less than the detection range, a positive reward is obtained, otherwise a negative reward is obtained. In each environmental interaction process, a counter is set for each sensor. In each step, if sensor resources are allocated to the flight target, the corresponding sensor count is +1. At the end of the environmental interaction process, it is determined whether the sensor usage time is overloaded. If the sensor is overloaded, a negative reward is obtained, otherwise a positive reward is obtained. In each step, it is determined whether the sensor resources scheduled by the flight target at this moment are consistent with the sensor resources at the previous moment. If they are inconsistent, a sensor resource switch occurs, and the number of times the sensor resource switches are scheduled for each flight target is counted; During the interaction process in this environment, the number of switches is judged. If the number of switches is too many, a negative reward is obtained, otherwise a positive reward is obtained. Among them, the reward for the round is obtained by summing up the switch number reward, the sensor usage time reward and the step length reward.

[0075] In this embodiment, the specific steps of extracting a preset number of experience values ​​from the experience playback buffer and updating the network parameters of the DQN neural network include: Extract a preset amount of experience from the experience replay buffer; The extracted experience is input into the DQN neural network as a training sample. The network parameters of the current DQN neural network are updated using the gradient descent algorithm based on the constructed target loss function. The model parameters are updated using the calculated gradients, and the loss function is minimized by adjusting the parameters. According to the set update frequency, the target network parameters are intermittently synchronized with the current network parameters, and finally the sensor resource allocation model is output.

[0076] In addition, the loss function of the target is the mean square error loss function:

[0077] In the formula, is the loss function of target a, the total loss is the average of the five target losses, and N is the number of training samples. is the true Q value of the i-th sample, is the predicted Q value of the i-th sample.

[0078] like Figure 8As shown, this embodiment also provides a sensor scheduling system based on combined action space reinforcement learning, including: a system state acquisition module, used to obtain the system state of the sensor networking system to be scheduled, the system state including the real-time position of the flight target and the sensor number assigned to the flight target at the current moment; a scheduling plan output module, used to input the system state of the sensor networking system to be scheduled into a pre-trained sensor resource allocation model to obtain a combined action, and output the combined action as the optimal scheduling plan for the flight target; wherein the sensor resource allocation model is constructed based on a reinforcement learning algorithm combined with a Markov decision process.

[0079] The present invention also provides a device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the sensor scheduling method based on combined action space reinforcement learning when executing the computer program.

[0080] When the processor executes the computer program, the steps of sensor scheduling based on combined action space reinforcement learning are implemented, for example: obtaining the system state of the sensor networking system to be scheduled, the system state including the real-time position of the flight target and the sensor number assigned to the flight target at the current moment; inputting the system state of the sensor networking system to be scheduled into a pre-trained sensor resource allocation model to obtain a combined action, and outputting the combined action as the optimal scheduling plan for the flight target; wherein the sensor resource allocation model is constructed based on a reinforcement learning algorithm combined with a Markov decision process.

[0081] Alternatively, when the processor executes the computer program, it implements the functions of each module in the above system, for example: a system status acquisition module, used to obtain the system status of the sensor networking system to be scheduled, the system status including the real-time position of the flight target and the sensor number assigned to the flight target at the current moment; a scheduling plan output module, used to input the system status of the sensor networking system to be scheduled into a pre-trained sensor resource allocation model to obtain a combined action, and output the combined action as the optimal scheduling plan for the flight target; wherein the sensor resource allocation model is constructed based on a reinforcement learning algorithm combined with a Markov decision process.

[0082] Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete preset functions, and the instruction segments are used to describe the execution process of the computer program in the sensor scheduling device based on combined action space reinforcement learning. For example, the computer program can be divided into a system state acquisition module and a scheduling scheme output module; the specific functions of each module are as follows: a system state acquisition module, which is used to obtain the system state of the sensor networking system to be scheduled, and the system state includes the real-time position of the flight target and the sensor number assigned to the flight target at the current moment; a scheduling scheme output module, which is used to input the system state of the sensor networking system to be scheduled into a pre-trained sensor resource allocation model to obtain a combined action, and output the combined action as the optimal scheduling scheme for the flight target; wherein the sensor resource allocation model is constructed based on a reinforcement learning algorithm combined with a Markov decision process.

[0083] The sensor scheduling device based on combined action space reinforcement learning can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The sensor scheduling device based on combined action space reinforcement learning may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the above is an example of a sensor scheduling device based on combined action space reinforcement learning, and does not constitute a limitation on the sensor scheduling device based on combined action space reinforcement learning. It may include more components than the above, or combine certain components, or different components. For example, the sensor scheduling device based on combined action space reinforcement learning may also include input and output devices, network access devices, buses, etc.

[0084] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the sensor scheduling based on combined action space reinforcement learning, and uses various interfaces and lines to connect the various parts of the entire sensor scheduling device based on combined action space reinforcement learning.

[0085] The memory can be used to store the computer program and / or module, and the processor implements various functions of the sensor scheduling device based on combined action space reinforcement learning by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory.

[0086] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0087] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the sensor scheduling method based on combined action space reinforcement learning are implemented.

[0088] If the module / unit integrated in the sensor scheduling system based on combined action space reinforcement learning is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0089] Based on such understanding, the present invention implements all or part of the processes in the above-mentioned sensor scheduling method based on combined action space reinforcement learning, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned sensor scheduling method based on combined action space reinforcement learning can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or preset intermediate form, etc.

[0090] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0091] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0092] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention.

Claims

1. A sensor scheduling method based on combined action space reinforcement learning, characterized in that: include: Acquire the system status of the sensor networking system to be scheduled, wherein the system status includes the real-time position of the flight target and the sensor number assigned to the flight target at the current moment; The system state of the sensor networking system to be scheduled is input into the pre-trained sensor resource allocation model to obtain the combined action, and the combined action is output as the optimal scheduling plan for the flight target; The sensor resource allocation model is constructed based on a reinforcement learning algorithm combined with a Markov decision process.

2. The sensor scheduling method based on combined action space reinforcement learning according to claim 1 is characterized in that: The training process of the sensor resource allocation model is as follows: Construct a DQN neural network and initialize the network parameters of the DQN neural network. The DQN neural network includes a current network and a target network with the same structure. The initialized DQN neural network is trained using the historical system status of the networked sensor system until the preset training rounds are reached and the training is terminated, and the sensor resource allocation model is output.

3. The sensor scheduling method based on combined action space reinforcement learning according to claim 2 is characterized in that: The historical system status of the networked sensor system is expressed as: In the formula, Indicates the current position of the flying target. Indicates the sensor number assigned to flight targets a, b, c, d, and e at the current moment.

4. The sensor scheduling method based on combined action space reinforcement learning according to claim 2 is characterized in that: The specific steps of using the historical system state of the networked sensor system to train the initialized DQN neural network are as follows: The historical system state of the networked sensor system is input into the initialized DQN neural network, and the mapping combination action is selected based on the greedy algorithm, and the predicted Q value corresponding to the mapping combination action is obtained; According to the selected mapping combination action, the sensor networking system interacts with the environment to obtain the reward value corresponding to the mapping combination action and the system state after executing the mapping combination action; Combine the system state, the mapping combination action, the reward corresponding to the mapping combination action, and the system state after executing the mapping combination action into one experience and store it in the preset experience playback buffer. Repeat the above process until the experience playback buffer reaches the threshold; A preset number of experience values ​​are extracted from the experience replay buffer, and the network parameters of the DQN neural network are updated until the preset number of updates is reached, and the sensor resource allocation model is output.

5. The sensor scheduling method based on combined action space reinforcement learning according to claim 4 is characterized in that: The specific steps of implementing environmental interaction with the sensor networking system according to the selected mapping combination action include: According to the current position of the flying target, the Euclidean distance between the flying target and the sensor that dispatches it is calculated; Get the distance reward for each flight target based on Euclidean distance calculation; Determine the relationship between the Euclidean distance and the detection range of the corresponding sensor. If the Euclidean distance is less than the detection range, a positive reward is obtained, otherwise a negative reward is obtained. In each environmental interaction process, a counter is set for each sensor. In each step, if sensor resources are allocated to the flight target, the corresponding sensor count is +1. At the end of the environmental interaction process, it is determined whether the sensor usage time is overloaded. If the sensor is overloaded, a negative reward is obtained, otherwise a positive reward is obtained. In each step, it is determined whether the sensor resources scheduled by the flight target at this moment are consistent with the sensor resources at the previous moment. If they are inconsistent, a sensor resource switch occurs, and the number of times the sensor resource switches are scheduled for each flight target is counted; During the interaction process in this environment, the number of switches is judged. If the number of switches is too many, a negative reward is obtained, otherwise a positive reward is obtained. Among them, the reward for the round is obtained by summing up the switch number reward, the sensor usage time reward and the step length reward.

6. The sensor scheduling method based on combined action space reinforcement learning according to claim 4 is characterized in that: The specific steps of extracting a preset number of experience values ​​from the experience playback buffer and updating the network parameters of the DQN neural network include: Extract a preset amount of experience from the experience replay buffer; The extracted experience is input into the DQN neural network as a training sample. The network parameters of the current DQN neural network are updated using the gradient descent algorithm based on the constructed target loss function. The model parameters are updated using the calculated gradients, and the loss function is minimized by adjusting the parameters. According to the set update frequency, the target network parameters are intermittently synchronized with the current network parameters, and finally the sensor resource allocation model is output.

7. The sensor scheduling method based on combined action space reinforcement learning according to claim 6 is characterized in that: The loss function of the target is the mean square error loss function: In the formula, is the loss function of target a, the total loss is the average of all flying target losses, and N is the number of training samples; is the true Q value of the i-th sample, is the predicted Q value of the i-th sample.

8. A sensor scheduling system based on combined action space reinforcement learning, characterized in that: include: A system status acquisition module is used to acquire the system status of the sensor networking system to be scheduled, wherein the system status includes the real-time position of the flight target and the sensor number assigned to the flight target at the current moment; The scheduling scheme output module is used to input the system state of the sensor networking system to be scheduled into the pre-trained sensor resource allocation model to obtain a combined action, and output the combined action as the optimal scheduling scheme for the flight target; wherein the sensor resource allocation model is constructed based on a reinforcement learning algorithm combined with a Markov decision process.

9. A device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the sensor scheduling method based on combined action space reinforcement learning as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the steps of the sensor scheduling method based on combined action space reinforcement learning as described in any one of claims 1 to 7.

Images