Wireless sensor network charging path optimization method based on A3C algorithm, base station and medium

Abstract

The invention discloses a wireless sensor network charging path optimization method based on an A3C algorithm, a base station and a medium. The method comprises the following steps: receiving electric quantity data collected by a sensor node; when a charging request is received, performing path planning based on an A3C algorithm to obtain a charging path; controlling the wireless charging vehicle to execute a charging task according to the charging path; the charging task is used for charging the sensor nodes with the residual electric quantity lower than a threshold value; wherein the charging request is sent out when the residual electric quantity of the sensor node is lower than a threshold value; the path planning adopts a reward function to optimize the charging path, and aims at minimizing the moving path, minimizing the number of dead nodes and / or maximizing the energy efficiency. According to the method, the charging path of the wireless rechargeable sensor network is optimized, so that the number of dead nodes is minimized, the charging efficiency is maximized, the service life of the network is prolonged, and the operation life and the energy utilization rate of the wireless sensor network are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology and artificial intelligence application technology, and specifically to a wireless sensor network charging path optimization method, a base station, and a medium based on an A3C algorithm. Background Art

[0002] As a highly maneuverable and flexible communication device, Unmanned Aerial Vehicle (UAV) has been widely used in the fifth-generation wireless network (5G). With its good line-of-sight transmission advantage, UAVs can provide better channel conditions in communication, thereby improving communication quality. However, as application scenarios such as real-time video recognition, augmented reality, and virtual reality put forward higher requirements on computing power and data processing latency, the limited computing resources and energy of traditional UAVs face challenges.

[0003] The emergence of Mobile Edge Computing (MEC) technology provides a solution for 5G communications, allowing drones to offload computing tasks to MEC servers at the edge of the network through wireless links, thereby significantly improving the computing performance and communication efficiency of drones. However, due to the strong line-of-sight characteristics of drone communication links, the risk of eavesdropping increases, resulting in threats to the security of data transmission. Therefore, enhancing the security of drone-assisted communication systems has become an urgent issue to be addressed.

[0004] Existing physical layer security technology (PLS) has been applied in some scenarios, but there are still problems such as complex algorithms and low optimization efficiency. Especially in an environment where multi-task computing offloading and eavesdropping threats coexist, how to maximize the security rate under limited energy and computing resources remains a technical challenge. Summary of the invention

[0005] The purpose of the present invention is to provide a wireless sensor network charging path optimization method, base station, and medium based on the A3C algorithm, to optimize the charging path of the wireless rechargeable sensor network, minimize the number of dead nodes, and maximize the charging efficiency, thereby extending the network life and significantly improving the operating life and energy utilization of the wireless sensor network.

[0006] The present invention provides a wireless sensor network charging path optimization method based on the A3C algorithm, comprising the following steps:

[0007] Receive power data collected by sensor nodes;

[0008] When a charging request is received, path planning is performed based on the A3C algorithm to obtain a charging path;

[0009] Controlling the wireless charging vehicle to perform a charging task according to a charging path; the charging task is to charge sensor nodes whose remaining power is lower than a threshold;

[0010] Wherein, the charging request is issued when the remaining power of the sensor node is lower than a threshold;

[0011] The path planning uses a reward function to optimize the charging path, and aims to minimize the moving path, minimize the number of dead nodes and / or maximize energy efficiency.

[0012] In the above technical solution, the A3C algorithm includes an Actor network and a Critic network; path planning based on the A3C algorithm includes:

[0013] Initialize state space, action space, and network parameters;

[0014] When the wireless charging vehicle completes a charging task, the reward feedback is calculated through the reward function according to the moving path of the wireless charging vehicle and the status of the sensor node;

[0015] Outputting the action and action probability distribution of each optional sensor node according to the current state of the state space through the Actor network, and adjusting the path selection according to the reward feedback;

[0016] The state value function of the current moving path is calculated through the Critic network according to the current state and the action output by the Actor network;

[0017] Update the network parameters of the Actor network and the Critic network to optimize the charging path;

[0018] When the power data received from all sensor nodes reaches the preset power value or reaches the target, the path optimization process ends, the optimal strategy is output, and the path planning is performed according to the optimal strategy.

[0019] In the above technical solution, updating the Actor network includes:

[0020] Taking the advantage function as the optimization target of the policy gradient, the Actor network is updated through the policy gradient method to optimize the charging path selection.

[0021] In the above technical solution, updating the Critic network includes:

[0022] The network parameters of the Critic network are updated by the gradient descent method, with the prediction error of the minimized state value as the error function; the prediction error of the minimized state value is the difference between the predicted reward of the current state value and the actual reward.

[0023] In the above technical solution, the A3C algorithm adopts multi-threaded training, and each thread has an Actor-Critic network for asynchronous training; after updating the network parameters of the Actor network and the Critic network, they are synchronized to the global network.

[0024] In the above technical solution, the advantage function is

[0025] A(s,a)=Q(s,a)-V(s)

[0026] Among them, Q(s,a) is the reward that a may get after selecting action in state s, and V(s) is the expected reward for continuing to act according to the current strategy in state s.

[0027] In the above technical solution, the reward function is

[0028]

[0029] Where K is the length of the charging sequence in the network life cycle, N c is the number of dead nodes, x k and k They are the coordinates of the wireless charging car at the kth sensor node.

[0030] In the above technical solution, the Actor network includes a one-dimensional convolutional network and a decoder. The one-dimensional convolutional network takes the state space as input, and the encoder extracts characteristic elements and decodes them from the state space to output the action and action probability distribution of each optional sensor node;

[0031] The state space is l i is the two-dimensional coordinate of sensor node i, is the energy requirement of the i-th sensor node; when The value of is 0, which means there is no sensor node that needs to be charged in the network.

[0032] The present invention also provides a base station, comprising:

[0033] The invention comprises a memory, a plurality of processors and a program stored in the memory, wherein the program is configured to be executed by the processors, and when the plurality of processors execute the program, the wireless sensor network charging path optimization method based on the A3C algorithm as described in any one of the above items is implemented.

[0034] The present invention also provides a storage medium having a computer program stored thereon, wherein when the computer program is executed, the method for optimizing a charging path of a wireless sensor network based on the A3C algorithm as described above is implemented.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention uses the A3C algorithm to optimize the charging path of the wireless rechargeable sensor network, realizing real-time optimization of the charging path of the wireless charging vehicle. In the case of multiple charging requests, the optimal path can be selected according to the environmental conditions, reducing the moving distance of the wireless charging vehicle and significantly improving the charging efficiency. The present invention has the characteristics of effective node energy management, strong adaptive ability, and high system stability, which improves the operating life and energy utilization of the wireless sensor network. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of a flow chart of an embodiment of the present invention.

[0038] Figure 2 Schematic diagram of a wireless rechargeable sensor network model according to an embodiment of the present invention.

[0039] Figure 3 Schematic diagram of the A3C algorithm network structure according to an embodiment of the present invention.

[0040] Figure 4 FIG. 1 is a schematic diagram for summarizing the A3C network structure according to an embodiment of the present invention.

[0041] Figure 5 Schematic diagram of the Actor network model according to an embodiment of the present invention.

[0042] Figure 6 Schematic diagram of a Critic network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The present invention adopts a unified WRSN model. Figure 2 The randomly deployed n static sensor nodes X = {x1, x2, ..., x n}, a base station (BS) is distributed in the center of the network, acting as a data collection center, responsible for charging and scheduling wireless charging vehicles (WCVs). It is assumed that each sensor node in the sensor network collects data and sends the collected information to the base station through point-to-point for storage and processing. When the capacity of the node in the network is lower than the charging threshold, it will send a charging request to the BS. After receiving a certain number of charging requests, the BS will plan the charging path of the wireless charging vehicle and send the wireless charging vehicle to charge the sensor node.

[0045] In a WRSN network, the time from the start of network deployment to the death of all sensor nodes is called the life cycle. In this application, a sensor node is considered "dead" when its energy consumption reaches 0. Therefore, for each moment, the number of nodes in the network with 0 energy is counted, which corresponds to the number of dead nodes.

[0046] Based on the above background, this embodiment provides a wireless sensor network charging path optimization method based on the A3C algorithm, which optimizes the charging path of the wireless rechargeable sensor network, minimizes the number of dead nodes, and maximizes the charging efficiency, thereby extending the network life. It has superior performance in solving large-scale WRSN problems that are difficult to calculate with traditional heuristic methods, and significantly improves the operating life and energy utilization of the wireless sensor network.

[0047] Specifically, Figure 1 As shown, the method comprises the following steps:

[0048] Step S1, receiving power data collected by the sensor node;

[0049] Step S2: when a charging request is received, path planning is performed based on the A3C algorithm through a path optimization strategy to obtain a charging path; the charging request is issued when the remaining power of the sensor node is lower than a threshold.

[0050] Step S3, controlling the wireless charging vehicle to perform a charging task according to the charging path; the charging task is to charge sensor nodes whose remaining power is lower than a threshold; the path optimization strategy uses a reward function to optimize the charging path, and aims to minimize the moving path, minimize the number of dead nodes and / or maximize energy efficiency.

[0051] The charging scheme problem is abstracted into a multi-objective optimization problem, in which the most important goal is to maximize energy efficiency, which is equivalent to minimizing the charging path length D. In addition, minimizing the number of dead nodes is a secondary goal. i is defined as the current state of the node, where c i =1 means the node is in working state, c i=0 means the node has run out of energy and has died. i The formula is specifically expressed as:

[0052]

[0053] Then the goal of the path optimization strategy can be formulated as:

[0054]

[0055] In this WRSN network model, a tuple {S, A, R, S'} is defined, where S is the current state space; A is the action space; R is the reward after the agent performs the action; and S' is the state of the environment after the action is performed.

[0056] In the network life cycle T, WCV is based on the state S at time k. k Determine an action k , WCV obtains a strategy through continuous exploration, and at the end of this life cycle, the reward R is calculated based on the obtained strategy.

[0057] S: The state space of the system includes the location and energy requirements of all sensor nodes. The state space set of S is expressed as S = {s i}, i = 0, 1, ..., n, where i = 0 represents S. s i is a tuple containing the location and energy requirement, and the formula is l i is the two-dimensional coordinate of sensor node i, For its energy needs. The value of is 0, which means there is no sensor node that needs to be charged in the network.

[0058] A: It is the interaction between WCV and the environment. An action represents the charging node determined by WCV.

[0059] The policy P is described by the expression a=π(s). Where a represents the action, s represents the state, and π represents the policy, which is the mapping from the input state s to the output action a. The goal is to learn an optimal policy to schedule WCV charging.

[0060] R: represents the reward value obtained after the WCV performs an action, which can be used to evaluate the action. Since the goal of this invention is to improve the charging efficiency of WRSN by optimizing the charging path of WCV, while avoiding sensor failure and reducing charging costs, the total trip length and number of dead nodes of WCV are used as reward signals. The reward function is defined as:

[0061]

[0062] Where K is the length of the charging sequence in the network life cycle, N c is the number of dead nodes, i.e. sensor nodes whose energy is exhausted to 0, x k and k They are the coordinates of the wireless charging car at the kth sensor node.

[0063] State space update process:

[0064] 1. S1, a1, R1 (S1, a1);

[0065] 2. S2, a2, R2 (S2, a2);

[0066] k、S k , a k , Rk(S k , a k ).

[0067] The specific process of state update is as follows: Assuming that the WCV is located at the BS at time 0, during the charging process, the WCV selects one or more sensor nodes that have issued charging requests for charging. After visiting sensor node i, the demand of the sensor node and the remaining energy of the WCV are updated as follows:

[0068] Update the energy requirement of the node to be visited at time k to:

[0069]

[0070] The energy requirements of the sensor nodes that were not visited at time k are updated as:

[0071]

[0072] When the energy demand of dead sensor nodes is constrained to 0, we get:

[0073]

[0074] Where Δt is the time from the kth step to the k+1th step of WCV.

[0075] When the WCV visits the BS, the energy of the WCV is replenished according to Formula 7, and the replenishment time is negligible compared to the travel time:

[0076] P m (k+1)=E m ,(visit=BS) (7)

[0077] After the above steps, the energy of WCV after the sensor node is charged is updated as:

[0078]

[0079] Based on the above goals and the energy change of WCV, the A3C algorithm is used in this application to plan the path of wireless charging electric vehicles. The A3C algorithm uses multi-threaded training, and each thread has an Actor-Criti c The network is trained asynchronously; when updating the Actor network and Criti c The network parameters of the network are then synchronized to the global network, and they also share a copy of the global network parameters. Each thread interacts with the environment independently, improving its own performance by sampling actions, updating strategies and value functions. Parallel training can greatly improve training efficiency and sample utilization.

[0080] like Figure 3 As shown in the figure, the A3C algorithm uses a multi-threaded approach. WCVs interact with the environment in multiple threads at the same time. The WCVs in each thread interact with the environment independently and save the learning experience in a global network. The global network and the local network synchronize parameters regularly to guide the subsequent learning interaction between WCV and the environment. The long-term operation of WRSN is achieved by minimizing the number of dead nodes and maximizing energy efficiency.

[0081] Path planning based on the A3C algorithm, including:

[0082] Initialize state space, action space, and network parameters;

[0083] When the wireless charging vehicle completes a charging task, the reward feedback is calculated through the reward function according to the moving path of the wireless charging vehicle and the status of the sensor node;

[0084] Outputting the action and action probability distribution of each optional sensor node according to the current state of the state space through the Actor network, and adjusting the path selection according to the reward feedback;

[0085] The state value function of the current moving path is calculated through the Critic network according to the current state and the action output by the Actor network;

[0086] Update the network parameters of the Actor network and the Critic network to optimize the charging path;

[0087] When the power data received from all sensor nodes reaches the preset power value or reaches the target, the path optimization process ends, the optimal strategy is output, and the path planning is performed according to the optimal strategy.

[0088] Specifically, Figure 4 As shown, in the Actor-Critic network framework, there are two neural networks, with a weight vector θ u and θ QThey are respectively related to the Actor network and the Critic network. The Actor network includes a one-dimensional convolutional network and a decoder. The one-dimensional convolutional network takes the state space as input. The encoder extracts feature elements and decodes from the state space, and outputs the action and action probability distribution of each optional sensor node. Based on the above actions and action probability distributions, the reward value R is calculated from the WCV executing the action sequence a in the WRSN. According to the current statistical data, an expected reward value V is obtained from the Critic network. Finally, the advantage function of the reward value and the expected reward value is used to update the Actor network and the Critic network.

[0089] like Figure 5 As shown in FIG. 1 , the Actor network includes a one-dimensional convolutional network in the first layer and a decoder in the second layer. The one-dimensional convolutional network takes the state space as input, and the encoder extracts feature elements and decodes them from the state space, outputting the action and action probability distribution of each optional sensor node. The one-dimensional convolutional neural network is used to encode structured data input. The encoder network reads a set of inputs: state space S = s i , i = 1, ..., n. Formally, input, s i Represented as a sequence of tuples Among them l i is the two-dimensional coordinate of the fixed sensor node i, is the energy consumption rate corresponding to the sensor node. is the energy requirement of the i-th sensor node; when The value of 0 means that there is no sensor node in the network that needs to be charged. k represents the set of all input states at a fixed time step k.

[0090] The second decoder is implemented as a GRU network. In this application, State is not directly used as the input of the decoder network. The feature elements extracted by the encoder network are used as the input of the decoder network. Then, the Attention mechanism is used to extract relevant information from the input information, and the extracted feature elements and the decoded output are used as the input of the Attention mechanism. The Attention mechanism solves the problem that the GRU network is difficult to obtain the final reasonable vector representation when the input sequence is long, and completes the extraction of important feature information. Finally, the Actor network will output the action and the probability value function P of the action.

[0091] like Figure 6 As shown in the figure, the Critic network consists of two convolutional layers and one fully connected layer, and remains trainable throughout the training process. Updating the Critic network includes:

[0092] The network parameters of the Critic network are updated by the gradient descent method, with the prediction error of the minimized state value as the error function; the prediction error of the minimized state value is the difference between the predicted reward of the current state value and the actual reward.

[0093] When the target life cycle of the sensor network is reached or all sensor nodes have died, the Actor network stops inputting. In this process, a strategy A = {a k}, k = 0, 1, .. K. The parameters of the Actor network and the Critic network are optimized to minimize the total length and the number of dead nodes in the entire life cycle of the sensor network. We use the parameter θ u To parameterize the random policy π, where θ u is a vector of all trainable variables used in the Actor network, including encoders, decoders, and attention mechanisms. Q The Critic network estimates the expected return and iteratively improves the strategy.

[0094] The Actor network first converts the current state sequence s i The input is converted into a fixed-length context vector through a one-dimensional convolutional neural network. The context vector is an internal representation of the input sequence that captures the key features of the input information. One-dimensional convolutional neural networks can process input data in parallel, so they are more efficient when processing long sequences. The elements of the output sequence are generated one by one through the decoder GRU network. At each time step, the GRU receives the input of the current step or the previous time step and the context vector weighted by the attention mechanism, and finally outputs the prediction result of the current time step, that is, the probability density distribution strategy function π of an action.

[0095] In the Actor-Critic framework, the goal of WCV is to maximize the reward R that can be obtained, which we express as the expectation of R. In the probability distribution P, the expectation of X is:

[0096] E P [X] = ∑ i P i X i (9)

[0097] Where X i are all possible values ​​of X, P i is the probability of each value occurring. The expectation can be regarded as the value X i With weight P i The weighted average of .

[0098] Define the state value function V(s) of the random policy π, which can be regarded as the expected discounted return, which can be regarded as the following iterative definition:

[0099] V(s)=E π(s) [r+γV(s)] (10)

[0100] Formula 10 means that the reward that can be obtained in the current state s is the sum of the reward that can be obtained in the next state s′ and the reward r obtained during the state transition process.

[0101] The critic network is parameterized as two convolutional layers and one fully connected layer, which inputs the state sequence s of the current environment. i And the action a taken, the value of the action can be output, that is, the corresponding value function V(s). The loss function calculation formula of the Critic network adopts the mean square error loss function.

[0102] In addition, there is the action value function Q(s, a), which is closely related to the state value function, namely:

[0103] Q(s,a)=r+γV(s′) (11)

[0104] Define a new function A(s, a), called the advantage function:

[0105] A(s,a)=Q(s,a)-V(s) (12)

[0106] Formula 12 expresses the advantage of choosing action a in state s. If action a is better than the average, the advantage function is positively correlated, otherwise, it is negatively correlated.

[0107] Taking the advantage function as the optimization target of the policy gradient, the Actor network is updated through the policy gradient method to optimize the charging path selection.

[0108] Update Criti c The network includes:

[0109] Taking the prediction error of the minimized state value as the error function, update Criti by gradient descent method c The network parameters of the network; the prediction error of the minimized state value is the difference between the predicted reward of the current state value and the actual reward.

[0110] Furthermore, in the Actor-Critic framework, when calculating the Q(s, a), V(s) or A(s, a) functions, only the feedback value of a single step is calculated.

[0111] A(s0)→r0+γV(s1) (13)

[0112] In this case, the immediate reward obtained from the state space (s0, a0, r0, s1) is used, and the function provides an estimate of the next predicted value. To improve the accuracy of the prediction, more steps can be used to provide an n-step estimate:

[0113] V(s0)→r0+γr1+...+γ n V(s n ) (14)

[0114] Among them, r t is the immediate reward obtained at time step t;

[0115] γ is a discount factor, which indicates the importance of future rewards;

[0116] V(s n ) is the state s after the nth step n The estimated value of .

[0117] The advantage of n-step return is that changes in the approximate function propagate faster. Each iteration propagates n steps backward, so the convergence speed will be faster. At the same time, it avoids relying only on the immediate reward of a single time step, providing a better estimate of the future cumulative reward, thereby improving the accuracy of the strategy. By accumulating returns for multiple steps, the n-step estimate can smooth the noise caused by chance or randomness, thereby improving the stability of the reward function and advantage function.

[0118] The following pseudo code explains the entire method:

[0119] First, the network parameters are randomly initialized. Since machine learning models are difficult to abstract, N samples are drawn and a feasible sequence is generated using Monte Carlo simulation. In each cycle, t Start. WCV performs an action a at every t steps t , observe the new state s t+1 ; This process ends until the termination condition is reached. After decoding all N samples, the corresponding rewards and policy gradients for each step are calculated and the Actor network is updated. In this step, V is composed of Criti c The approximate value of the reward calculated by the network, updating Criti c The network is used to reduce the difference between the expected reward and the observed reward during the Monte Carlo rollout. At the same time, tasks are assigned to multiple threads. After learning once, each thread updates its learned parameters to the global network. The next time it learns, it pulls the global parameters to continue learning. T>T max The algorithm ends and outputs the global network parameters θ and θ v , thus obtaining the optimal strategy. The charging path is optimized through the optimal strategy, and the wireless charging vehicle is controlled to perform the charging task according to the charging path.

[0120] The present invention studies and proposes a method for optimizing the charging path of a wireless sensor network based on the A3C algorithm. Since the output under the A3C algorithm framework is a probability distribution of an optional action space rather than a discrete action space, it has a good advantage in solving scenarios with large network scales and large changes in sensor energy states. The A3C algorithm effectively reduces the variance of the policy gradient by introducing the Actor-Critic structure, thereby improving the stability and convergence speed of the algorithm. Furthermore, compared with the classic Actor-Critic algorithm, the A3C algorithm utilizes a multi-threaded parallel exploration environment and updates model parameters, greatly improving the efficiency of training. Therefore, compared with many classic heuristic algorithms, the present invention has better performance when the network scale is increased.

[0121] Furthermore, due to the large difference in the number of sensor nodes in actual applications, this paper considers three networks of different scales, including 50, 100 and 200 sensor nodes, to generate 100,000 training sets, 10,000 validation sets and 10,000 test sets. When the network environment is initialized, the sensor node positions and residual energy are randomly generated. Specifically, the sensor node positions [0,1] are uniformly randomly drawn in the unit square. 2 , the initial residual energy of each node is randomly generated between 10 J and 20 J. The speed of WCV is 0.1 m / s.

[0122] During validation, a sample size of 256 is used, where the sample size depends on the output size. A 1D convolutional layer is used for the encoder, where the input channels are the same as the elements of the state S and the output size is the same as the input size of the decoder. For the decoder, a 1D convolutional layer and a GRU with a hidden size of 128 are used.

[0123] In the Critic network, it has three hidden layers: two convolutional layers with ReLU activation and one linear convolutional layer with a single output. In both networks, there is a dropout layer after the convolution operation with a probability of 0.1. Adam optimizer is used to train our model. The initial learning rate of 0.0005 used in the 50-node and 100-node networks and 0.00005 used in the 200-node network. In the present invention, the search strategy for the training model is sampling decoding, which samples the next step according to its probability. The search strategy of greedy decoding is used during testing.

[0124] The hardware configuration used is NVIDIA GeForce GTX1080, and the software configuration is Python version v3.6.5. The experimental parameters are shown in Table 1.

[0125] Table 1 Experimental parameters

[0126]

[0127]

[0128] The present invention uses the A3C algorithm to optimize the charging path of the wireless rechargeable sensor network, realizing real-time optimization of the charging path of the wireless charging vehicle. In the case of multiple charging requests, the optimal path can be selected according to the environmental conditions, reducing the moving distance of the wireless charging vehicle and significantly improving the charging efficiency. The present invention has the characteristics of effective node energy management, strong adaptive ability, and high system stability, which improves the operating life and energy utilization of the wireless sensor network.

[0129] This method can maximize the energy utilization in wireless sensor networks and is suitable for outdoor and mobile energy storage products. This method optimizes energy storage products to ensure efficient and balanced energy transmission in complex environments with multiple nodes, thereby extending battery life and improving charging efficiency. When smart batteries and solar energy storage products are remote or cannot be frequently maintained manually, the dynamic planning of charging paths and energy scheduling by this method can maximize the utilization of power resources without affecting the operation of the equipment. It provides a more efficient power management solution for portable energy storage devices.

[0130] Based on the same inventive concept, the present application provides a base station, including a memory, multiple processors, and a program stored in the memory, wherein the program is configured to be executed by the processors, and the steps of the above method are implemented when the multiple processors execute the program.

[0131] Based on the same inventive concept, the present invention further provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented.

[0132] Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form, etc. Computer readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in computer readable media can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0133] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0134] The present application also provides an electronic device, comprising a memory, multiple processors, and a program stored in the memory, wherein the program is configured to be executed by the processors, and the steps of the above method are implemented when the multiple processors execute the program.

[0135] In addition, the present invention also provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented. The present invention can be used in many general or special computing system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0136] The device in this embodiment and the method in the aforementioned embodiment are two aspects based on the same inventive concept. The method implementation process has been described in detail above, so those skilled in the art can clearly understand the structure and implementation process of the system in this embodiment based on the aforementioned description. For the sake of brevity of the specification, they will not be repeated here.

[0137] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A wireless sensor network charging path optimization method based on A3C algorithm, characterized in that: The following steps are involved: Receive power data collected by sensor nodes; When a charging request is received, path planning is performed based on the A3C algorithm to obtain a charging path; Controlling the wireless charging vehicle to perform a charging task according to a charging path; the charging task is to charge sensor nodes whose remaining power is lower than a threshold; Wherein, the charging request is issued when the remaining power of the sensor node is lower than a threshold; The path planning uses a reward function to optimize the charging path, and aims to minimize the moving path, minimize the number of dead nodes and / or maximize energy efficiency.

2. The wireless sensor network charging path optimization method based on the A3C algorithm according to claim 1 is characterized in that: The A3C algorithm includes an Actor network and a Critic network. Path planning based on the A3C algorithm includes: Initialize state space, action space, and network parameters; When the wireless charging vehicle completes a charging task, the reward feedback is calculated through the reward function according to the moving path of the wireless charging vehicle and the status of the sensor node; Outputting the action and action probability distribution of each optional sensor node according to the current state of the state space through the Actor network, and adjusting the path selection according to the reward feedback; The state value function of the current moving path is calculated through the Critic network according to the current state and the action output by the Actor network; Update the network parameters of the Actor network and the Critic network to optimize the charging path; When the power data received from all sensor nodes reaches the preset power value or reaches the target, the path optimization process ends, the optimal strategy is output, and the path planning is performed according to the optimal strategy.

3. The wireless sensor network charging path optimization method based on the A3C algorithm according to claim 2 is characterized in that: Updated Actor network, including: Taking the advantage function as the optimization target of the policy gradient, the Actor network is updated through the policy gradient method to optimize the charging path selection.

4. The wireless sensor network charging path optimization method based on the A3C algorithm according to claim 2 is characterized in that: Updated the Critic network, including: The network parameters of the Critic network are updated by the gradient descent method, with the prediction error of the minimized state value as the error function; the prediction error of the minimized state value is the difference between the predicted reward of the current state value and the actual reward.

5. The wireless sensor network charging path optimization method based on the A3C algorithm according to claim 3 is characterized in that: The A3C algorithm adopts multi-threaded training, and each thread has an Actor-Critic network for asynchronous training; after updating the network parameters of the Actor network and the Critic network, they are synchronized to the global network.

6. The wireless sensor network charging path optimization method based on the A3C algorithm according to claim 5 is characterized in that: The advantage function is A(s,a)=Q(s,a)-V(s) Among them, Q(s,a) is the reward that a may get after selecting action in state s, and V(s) is the expected reward for continuing to act according to the current strategy in state s.

7. The wireless sensor network charging path optimization method based on the A3C algorithm according to claim 6 is characterized in that: The reward function is Where K is the length of the charging sequence in the network life cycle, N c is the number of dead nodes, x k and k They are the coordinates of the wireless charging car at the kth sensor node.

8. The wireless sensor network charging path optimization method based on the A3C algorithm according to claim 3 is characterized in that: The Actor network includes a one-dimensional convolutional network and a decoder, wherein the one-dimensional convolutional network takes a state space as input, the encoder extracts characteristic elements from the state space and decodes, and outputs the action and action probability distribution of each optional sensor node; The state space is l i is the two-dimensional coordinate of sensor node i, is the energy requirement of the i-th sensor node; when The value of is 0, which means there is no sensor node that needs to be charged in the network.

9. A base station, characterized in that: include: The invention comprises a memory, a plurality of processors and a program stored in the memory, wherein the program is configured to be executed by the processors, and when the plurality of processors execute the program, the method for optimizing the charging path of a wireless sensor network based on the A3C algorithm as claimed in any one of claims 1 to 8 is implemented.

10. A storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the method for optimizing the charging path of a wireless sensor network based on the A3C algorithm as claimed in any one of claims 1 to 8 is implemented.

Images