Wireless sensor network resource allocation method and device and electronic equipment

Abstract

The invention discloses a wireless sensor network resource allocation method, a wireless sensor network resource allocation device and electronic equipment. The method comprises the following steps: acquiring position information respectively corresponding to a plurality of terminals included in the wireless sensor network; determining data transmission rates between the plurality of terminals and the corresponding remote radio units; acquiring an initial resource allocation strategy of the wireless sensor network; and based on the position information corresponding to the plurality of terminals and the data transmission rates between the plurality of terminals and the corresponding radio remote units, optimizing the initial resource allocation strategy by taking the maximum total throughput of the system in the wireless sensor network as an optimization target to obtain a target resource allocation strategy of the wireless sensor network. According to the invention, the technical problems of low network resource allocation efficiency and low accuracy of a wireless sensor network resource allocation method in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, device and electronic equipment for allocating resources in a wireless sensor network. Background Art

[0002] With the deepening of the construction of the power Internet of Things and the increasing number of wired channels in recent years, the demand for monitoring wired channels has become increasingly urgent. Low-cost and efficient digital operation and maintenance methods are urgently needed to support this demand. Deploying low-cost wireless sensor network (WSN) systems to cover monitoring sensors is one of the important ways to realize channel communication systems in related technologies.

[0003] However, there is no mature resource allocation solution for wireless sensor networks suitable for cable channels in the relevant technology. The resource allocation strategy of wireless sensor networks is mainly determined manually, which is not only highly subjective, but also has problems such as low resource allocation efficiency and poor accuracy.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a wireless sensor network resource allocation method, device and electronic device to at least solve the technical problems of low network resource allocation efficiency and low accuracy existing in the wireless sensor network resource allocation method in the related art.

[0006] According to one aspect of an embodiment of the present invention, a method for allocating resources in a wireless sensor network is provided, comprising: obtaining location information corresponding to each of a plurality of terminals included in the wireless sensor network; determining data transmission rates between each of the plurality of terminals and the corresponding radio remote units; obtaining an initial resource allocation strategy for the wireless sensor network, wherein the initial resource allocation strategy is used to indicate the deployment locations corresponding to each of the plurality of radio remote units included in the wireless sensor network and an initial bandwidth resource allocation strategy; based on the location information corresponding to each of the plurality of terminals and the data transmission rates between each of the plurality of terminals and the corresponding radio remote units, the initial resource allocation strategy is optimized with the maximum total system throughput in the wireless sensor network as the optimization goal, so as to obtain a target resource allocation strategy for the wireless sensor network.

[0007] According to another aspect of an embodiment of the present invention, a wireless sensor network resource allocation device is further provided, comprising: a location acquisition module, used to acquire location information corresponding to a plurality of terminals included in the wireless sensor network; a transmission rate determination module, used to determine the data transmission rates between the plurality of terminals and the corresponding radio remote units; an initial strategy determination module, used to acquire an initial resource allocation strategy of the wireless sensor network, wherein the initial resource allocation strategy is used to indicate the deployment locations corresponding to the plurality of radio remote units included in the wireless sensor network and an initial bandwidth resource allocation strategy; and a strategy optimization module, used to optimize the initial resource allocation strategy based on the location information corresponding to the plurality of terminals and the data transmission rates between the plurality of terminals and the corresponding radio remote units, with the maximum total system throughput in the wireless sensor network as the optimization target, so as to obtain a target resource allocation strategy of the wireless sensor network.

[0008] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the wireless sensor network resource allocation methods.

[0009] In an embodiment of the present invention, by acquiring location information corresponding to a plurality of terminals included in a wireless sensor network, determining data transmission rates between the plurality of terminals and corresponding radio remote units, acquiring an initial resource allocation strategy of the wireless sensor network, wherein the initial resource allocation strategy is used to indicate deployment locations and initial bandwidth resource allocation strategies corresponding to the plurality of radio remote units included in the wireless sensor network, and optimizing the initial resource allocation strategy with the maximum total system throughput in the wireless sensor network as the optimization target based on the location information corresponding to the plurality of terminals and the data transmission rates between the plurality of terminals and corresponding radio remote units, obtaining a target resource allocation strategy of the wireless sensor network, achieving the purpose of optimizing the resource allocation strategy of the wireless sensor network with the maximum total system throughput in the wireless sensor network as the optimization target, thereby achieving the technical effect of improving the resource allocation efficiency and accuracy of the wireless sensor network, and further solving the technical problem of low network resource allocation efficiency and low accuracy in the resource allocation method of the wireless sensor network in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0011] Figure 1 is a flow chart of a wireless sensor network resource allocation method according to an embodiment of the present invention;

[0012] Figure 2 is an optional comparison result diagram of cumulative rewards of an agent during the entire training process according to an embodiment of the present invention;

[0013] Figure 3 is a comparison diagram of changes in the total throughput of the system with the number of remote radio units under different optional algorithms according to an embodiment of the present invention;

[0014] Figure 4 is a schematic diagram of a wireless sensor network resource allocation device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] At present, the power load in large cities is increasing. In order to alleviate the contradiction between the limited capacity of transmission corridors and the growth of load, beautify urban landscapes, ensure urban safety, and reduce the impact of natural disasters such as typhoons on urban power systems, cable channels have been widely promoted and their share has continued to rise. However, the overall construction of cable channels is still in its infancy. The detection and fault diagnosis of cable channels mainly rely on manual labor, and there is a lack of mature, efficient, low-cost, and comprehensive cable channel operation environment and status monitoring and automatic early warning solutions.

[0018] In recent years, with the deepening of the construction of the power Internet of Things and the increasing number of wired channels, the demand for monitoring cable channels has become increasingly urgent. Low-cost and efficient digital operation and maintenance methods are urgently needed to support this demand. Deploying low-cost wireless sensor network (WSN) system coverage monitoring sensors is one of the important ways to realize the channel communication system in related technologies. In addition, with the development of low-power wireless sensor network technology, its requirements for power supply are getting lower and lower, which is one of the most economical and applicable ways to cover cable channels.

[0019] However, there is no mature resource allocation solution for wireless sensor networks suitable for cable channels in the relevant technology. The resource allocation strategy of wireless sensor networks is mainly determined manually, which is not only highly subjective, but also has problems such as low resource allocation efficiency and poor accuracy.

[0020] According to an embodiment of the present invention, a method embodiment of wireless sensor network resource allocation is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] Figure 1 is a flow chart of a wireless sensor network resource allocation method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0022] Step S102, obtaining location information corresponding to a plurality of terminals included in the wireless sensor network;

[0023] Optionally, the wireless sensor network may be a wireless sensor network in a cable tunnel, and the wireless sensor network is composed of a sink node, multiple (e.g., K) remote radio units (RRUs) and multiple (e.g., K×N) terminals, wherein each RRU corresponds to N terminals. For example, the RRU set may be denoted as R={R 1 ,...,R k,...,R K}, let the terminal set served by K RRUs be denoted as D 1 ,...,D k ,...,D K , each RRU serves N terminals, that is D=D 1 ∪D 2 ∪...∪D K represents the total set of terminals. Optionally, the multiple terminals may be multiple sensors. By acquiring the location information corresponding to the multiple terminals respectively, the location distribution of the multiple terminals in the wireless sensor network can be obtained.

[0024] Optionally, a sink node refers to a node in a wireless sensor network that is responsible for collecting and processing data from multiple terminals and multiple radio remote units, and transmitting the obtained data to a server or other upper layer system. The sink node can be implemented by a specially designed hardware device, which may include but is not limited to components such as a microprocessor, a communication module, and a storage device. The sink node is mainly used to coordinate and manage the entire wireless sensor network to ensure timely transmission and processing of data.

[0025] In this implementation, the cable channel refers to the cable line buried underground, which is mainly used for power transmission; the wireless sensor network serves the cable channel operating environment and status monitoring, and monitors the water level, gas, temperature and humidity, partial discharge and other conditions in the channel through various IoT sensors; through the combined use of cable channels and wireless sensor networks, the efficiency of wireless sensor networks in cable channels can be improved, and low-cost, high-efficiency digital cable channel operation and maintenance methods can be developed.

[0026] Step S104, determining data transmission rates between the plurality of terminals and the corresponding radio remote units respectively;

[0027] Optionally, the data transmission rate is used to indicate the rate at which the terminal transmits data to the corresponding remote radio unit. By determining the rate at which each of the multiple terminals transmits data to the corresponding remote radio unit, the efficiency and speed of each terminal transmitting data to the corresponding remote radio unit can be measured.

[0028] Step S106, obtaining an initial resource allocation strategy of the wireless sensor network, wherein the initial resource allocation strategy is used to indicate deployment locations corresponding to a plurality of radio remote units included in the wireless sensor network and an initial bandwidth resource allocation strategy;

[0029] Optionally, a resource allocation strategy can be initialized for the wireless sensor network, where the initial resource allocation strategy includes the deployment locations of multiple radio remote units included in the wireless sensor network and an initial bandwidth resource allocation strategy, wherein the initial bandwidth resource allocation strategy is used to indicate the bandwidth resources initially allocated by the corresponding radio remote unit to the corresponding terminal.

[0030] Step S108, based on the location information corresponding to the multiple terminals, and the data transmission rates between the multiple terminals and the corresponding radio remote units, the initial resource allocation strategy is optimized with the maximum total system throughput in the wireless sensor network as the optimization goal to obtain the target resource allocation strategy of the wireless sensor network.

[0031] It can be understood that by acquiring the location information corresponding to multiple terminals, the location distribution of multiple terminals in the wireless sensor network can be obtained; the efficiency and speed of each terminal transmitting data to the corresponding radio remote unit can be measured through the data transmission rate between the multiple terminals and the corresponding radio remote units; based on the terminal location information and the terminal data transmission rate, and taking the maximum total system throughput in the wireless sensor network as the optimization goal, the wireless sensor network resource allocation is optimized, and the target resource allocation strategy obtained on this basis is the optimal resource allocation strategy that can achieve the maximum system throughput under the conditions of considering the terminal location information and the terminal data transmission rate.

[0032] Optionally, the system throughput can be obtained based on the data transmission rates corresponding to multiple terminals, for example, from the terminal To the remote radio unit R k The data transmission rate TR n,k , the corresponding system throughput of the wireless sensor network can be expressed as

[0033] In an optional embodiment, based on location information corresponding to each of the multiple terminals and data transmission rates between each of the multiple terminals and the corresponding radio remote units, the initial resource allocation strategy is optimized with the maximum total system throughput in the wireless sensor network as the optimization goal, to obtain a target resource allocation strategy for the wireless sensor network, including: constructing a state space based on location information corresponding to each of the multiple terminals and data transmission rates between each of the multiple terminals and the corresponding radio remote units; constructing an action space based on multiple candidate resource allocation strategies; constructing a reward function based on the total system throughput; based on the state space and the action space, with the maximum function value of the reward function as the goal, optimizing parameters of the initial resource allocation model to obtain a target resource allocation model; and using the target resource allocation model to obtain a target resource allocation strategy.

[0034] In the above method, by constructing the state space based on the terminal location information and the data transmission rate, and constructing the action space based on the candidate resource allocation strategy, the various possible states and optional actions in the wireless sensor network system can be clearly described, providing a comprehensive consideration for subsequent resource allocation. By constructing a reward function based on the total system throughput, the advantages and disadvantages of different resource allocation strategies can be quantified, providing a clear goal for optimizing parameters and selecting the best strategy. By optimizing the parameters of the initial resource allocation model and obtaining the target resource allocation model, the performance and efficiency of the system can be improved, and resource allocation can be made more reasonable and effective. The target resource allocation strategy obtained by the target resource allocation model can guide the reasonable allocation of resources in the wireless sensor system and improve the overall performance and user experience of the system.

[0035] Optionally, the location information and data transmission rate of each terminal are represented in the state space S, and the formula is expressed as In the action space A, all possible RRU deployment locations and bandwidth resource allocations are represented by the formula: The reward function r(S,A) can be designed to maximize the total system throughput, as shown in the formula:

[0036] In an optional embodiment, based on the state space and the action space, with the goal of maximizing the function value of the reward function, the parameters of the initial resource allocation model are optimized to obtain a target resource allocation model, including: based on the state space and the action space, with the goal of maximizing the function value of the reward function, a deep deterministic policy gradient algorithm is used to optimize the parameters of the initial resource allocation model to obtain a target resource allocation model.

[0037] It can be understood that the Deep Deterministic Policy Gradient (DDPG) algorithm is a policy optimization algorithm for continuous action space. It combines the deterministic policy gradient method and deep learning technology. It is mainly used to solve the problem of continuous action space in reinforcement learning by learning a deterministic policy function to maximize the cumulative reward. By optimizing the parameters of the initial resource allocation model through the deep deterministic policy gradient algorithm, the model can more effectively allocate resources under a given state space and action space to maximize the function value of the reward function. The target resource allocation model after optimizing the parameters can better adapt to environmental changes and demand changes, improve the resource utilization efficiency of the wireless communication network, and reduce the waste of wireless communication network resources.

[0038] In an optional embodiment, the initial resource allocation model includes a policy network, a critic network, a target policy network and a target critic network. Based on the state space and the action space, with the function value of the reward function as the goal, a deep deterministic policy gradient algorithm is used to optimize the parameters of the initial resource allocation model to obtain a target resource allocation model, including: inputting the state space into the policy network to obtain a first action determined from the action space; using the critic network to evaluate the first action to obtain a first action value function estimate; inputting the state space into the target policy network to obtain a second action determined from the action space; using the target critic network to evaluate the second action to obtain a second action value function estimate; based on the first action value function estimate, a first loss is obtained; based on the second action value function estimate, a second loss is obtained; based on the first loss and the second loss, the policy network, the critic network, the target policy network and the target critic network are optimized parameters; the above operations are repeated until a predetermined termination condition is reached; based on the policy network with updated parameters, the critic network with updated parameters, the target policy network with updated parameters and the target critic network with updated parameters obtained at the predetermined termination condition, a target resource allocation model is constructed.

[0039] Optionally, the target resource allocation model is set to be obtained by optimizing the parameters of four deep neural networks (DNNs): the policy network μ(S, ω), the critic network Q(S, A, λ), the target policy network μ′(S, ω′) and the target critic network Q′(S, A, λ′). ω, λ, ω′, λ′ represent the parameters to be optimized of each DNN. The policy network is the policy network, which is used to output an action according to the current state space, that is, to determine the resource allocation strategy that should be taken in the current state; that is, the policy network is responsible for outputting actions to guide the optimal decision to be made in the current state to achieve the optimal allocation of wireless communication network resources. The critic network is used to evaluate the value of the action output by the policy network, that is, to estimate the value function of the action according to the combination of action and state; that is, the critic network is responsible for providing feedback to the policy network, guiding the policy network to adjust in the direction of higher value actions, thereby improving the efficiency of resource allocation. The target policy network is the "target version" of the policy network used to estimate the target action, which is used to calculate the target action; by setting the target policy network, the instability in training can be reduced, and by updating the target network, the training can be made more stable and converge faster. The target critic network is a "target version" of the critic network used to estimate the target action value function, which is used to calculate the value function of the target action; by setting the target critic network, the instability in training can be reduced, and by updating the target network, the training can be made more stable and converge faster.

[0040] In the above method, through the interactive learning of the policy network and the critic network, the optimization problem of the continuous action space is solved, and the efficiency and accuracy of the resource allocation model are improved; by continuously optimizing the network parameters of each network, the model can more accurately predict the value function and action, and improve the accuracy and efficiency of resource allocation; through the update and training iteration of the target network (i.e., the target policy network and the target critic network), the model is more stable and converges faster, reducing the instability in the training process.

[0041] Optionally, the state space is input into the policy network and the target policy network respectively, and the output results of the policy network and the target policy network, i.e., the first action and the second action, can be obtained; the critic network and the target critic network are further used to evaluate the first action and the second action, respectively, to obtain the output results corresponding to the critic network and the target critic network, i.e., the first action value function estimate and the second action value function estimate. Loss calculation is performed based on the first action value function estimate and the second action value function estimate, and based on the obtained first loss and second loss, the parameters of each network are optimized with the goal of minimizing the loss, and the optimization iteration is continuously performed until the predetermined conditions are met (such as the first loss and the second loss are both minimized).

[0042] In an optional embodiment, a second loss is obtained based on an estimated value of the second action-value function, including: obtaining a target action-value function based on a reward function, a second action, and an estimated value of the second action-value function; and obtaining a second loss based on a mean square error between the target action-value function and an estimated value of the second action-value function.

[0043] Optionally, in the Deep Deterministic Policy Gradient (DDPG) algorithm, the target action value function is a function used to estimate the long-term reward obtained by performing an action in the current state. It can be expressed by the formula Among them, i represents the current round, i+1 represents the next round of the current round, μ′(S i+1 ,ω′) represents the output result of the current round target strategy network (i.e., the second action), Q′(S i+1 ,μ′(S i+1 ,ω′),λ′) indicates that the output of the target strategy network is μ′(S i+1 ,ω′), the output of the target critic network (i.e., the estimated value of the second action value function), S i+1 represents the state of the next round of the current round, and γ represents the discount factor.

[0044] Optionally, the first loss is used to update the parameters of the policy network, and the second loss is used to update the parameters of the critic network. The mean square error between the critic network output Q(S,A,λ) and the loss function is expressed as This loss function can be used to calculate the second loss Loss(λ), where S i Indicates the state of the current round. The above method can adjust the parameters of the critic network to better fit the real action value function, which helps to improve the accuracy of the critic network's estimation of the value of the action, thereby more accurately guiding the decision-making of the intelligent agent. At the same time, optimizing the critic network can improve the training effect of the deep reinforcement learning algorithm and enable the intelligent agent to better learn the optimal strategy.

[0045] Optionally, the loss function of the policy network, that is, the first loss Loss(ω), can be obtained as follows: Minimize the loss function of the policy network and obtain the policy gradient of the sampling strategy in, Represents the gradient value of Loss(ω); Represents the gradient value of the critic network's output Q(S,A,λ) under the influence of the policy network's optimized parameter ω; Represents the gradient value of the critic network's output Q(S,A,λ) under the influence of the action space A; It represents the gradient value of the output result Q(S,A,λ) of the policy network under the influence of the parameter ω to be optimized of the policy network.

[0046] Optionally, the parameter update of the two target networks (i.e., the target policy network and the target critic network) is implemented by soft updating, λ”←τλ+(1-τ)λ′, ω”←τω+(1-τ)ω′, where λ” represents the updated target policy network parameters; ω” represents the updated target critic network parameters, and τ is the soft update coefficient.

[0047] In an optional embodiment, based on the state space and the action space, with the goal of maximizing the function value of the reward function, the parameters of the initial resource allocation model are optimized to obtain a target resource allocation model, including: determining constraints, wherein the constraints include at least bandwidth resource allocation constraints, data transmission power constraints, and data transmission rate constraints: based on the state space, the action space, and the constraints, with the goal of maximizing the function value of the reward function, the parameters of the initial resource allocation model are optimized to obtain a target resource allocation model.

[0048] Optionally, in the process of optimizing the parameters of the initial resource allocation model, by constraining the allocation of bandwidth resources, it is possible to ensure the rational use of resources, avoid excessive allocation or waste of resources, and improve network performance and efficiency; by constraining the data transmission power, it is possible to limit the data transmission power, reduce energy consumption, reduce interference and uneven signal strength, and ensure communication quality and equipment life; by constraining the data transmission rate, it is possible to balance the competition between different terminals or tasks and ensure the fairness and stability of data transmission.

[0049] Optionally, the bandwidth resource allocation constraint is used to indicate that the total amount of bandwidth resources respectively allocated by the multiple radio remote units to the corresponding terminals is less than or equal to a predetermined resource amount, and the bandwidth resource constraint may be set to the following form:

[0050]

[0051] in, Indicates the total amount of bandwidth resources allocated by multiple radio remote units to the corresponding terminal, B T Indicates the total system bandwidth (i.e., the amount of reserved resources), Indicates the bandwidth corresponding to any terminal.

[0052] Optionally, the data transmission power constraint is used to indicate that the data transmission power corresponding to any terminal among the multiple terminals is within a predetermined transmission power range. The data transmission power constraint may be set to the following form:

[0053] 0≤P n,k ≤P max

[0054] Among them, P n,k Represents the data transmission power corresponding to any terminal, P max Indicates the preset maximum transmission power.

[0055] Optionally, the data transmission rate constraint is used to indicate that the data transmission rate between any terminal among the multiple terminals and the corresponding radio remote unit is less than a predetermined transmission rate. The data transmission rate constraint may be set to the following form:

[0056] TR n,k ≤TR * n,k

[0057] Among them, TR * n,k Indicates the predetermined transmission rate corresponding to any terminal.

[0058] In an optional embodiment, the data transmission rate constraint is obtained in the following manner:

[0059]

[0060] Among them, TR * n,k Indicates the predetermined transmission rate corresponding to any terminal; represents the bandwidth resources allocated by any radio remote unit to any terminal; n,k Indicates the data transmission power of any terminal transmitting data to any radio remote unit; h n,k represents the channel gain between any remote radio unit and any terminal; σ 2 Indicates the preset noise power.

[0061] Through the above steps S102 to S108, the purpose of optimizing the resource allocation strategy of the wireless sensor network with the maximum total system throughput in the wireless sensor network as the optimization goal can be achieved, thereby achieving the technical effect of improving the resource allocation efficiency and allocation accuracy of the wireless sensor network, and further solving the technical problems of low network resource allocation efficiency and low accuracy in the wireless sensor network resource allocation method in the related art.

[0062] Based on the above embodiment and optional embodiment, the present invention proposes an optional implementation of a wireless sensor network resource allocation method, the method comprising:

[0063] S1, model establishment, establish a system model of the wireless sensor network in the cable tunnel, and derive the total throughput of the cable tunnel wireless sensor network system, which specifically includes the following steps:

[0064] S11, consider a wireless sensor network in a cable tunnel, which consists of a sink node, K radio remote units RRUs and multiple terminals;

[0065] S12, the RRU set is recorded as R = {R 1 ,...,R k ,...,R K}, let the terminal set served by K RRUs be denoted as D 1 ,...,D k ,...,D K , each RRU serves N terminals, that is D=D 1 ∪D 2 ∪...∪D K Represents the total set of terminals;

[0066] S13, let the horizontal coordinates of RRU and terminal in the channel be q k and in is a known fixed variable, q k is the variable to be determined;

[0067] S14, the channel impulse response (CI) model is used to describe the path loss between the RRU and the terminal, that is, the path loss Where f is the carrier frequency, c is the speed of light, ρ is the path loss exponent, and d 0 =1m is the reference distance, d n,k For Terminal To RRUR k distance;

[0068] S15, and R k The channel gain between

[0069] S16, assuming that to R k The data transmission power is P n,k , Assign to terminal The bandwidth, σ 2 is the noise power, then from the terminal To the remote radio unit R k The data transmission rate TR n,k Constraints should be satisfied

[0070] S17, impose a data transmission power upper limit constraint of the terminal to 0≤P n,k ≤P max , where P max Indicates the preset maximum transmission power;

[0071] S18, assuming the total system bandwidth is B T , the bandwidth resource allocation constraint is imposed as and

[0072] S19, the total system throughput of the cable channel wireless sensor network is expressed as

[0073] S2 uses the DDPG algorithm in deep reinforcement learning to solve the joint optimization problem of radio remote unit deployment and bandwidth resource allocation, which specifically includes the following steps:

[0074] S21, in order to make the system throughput of the wireless sensor network in the cable channel Maximize the horizontal coordinate q of RRU k and bandwidth resources As the optimization variable, Denoted as q, Indicated as B;

[0075] S22, the optimization problem is formulated as That is, the total system throughput is maximized;

[0076] S23, describes the operation from terminal D n k to RRU R k The maximum achievable rate constraint 1:

[0077] S24, describes the upper limit constraint 2 of the terminal transmission power: 0≤P n,k ≤P max ;

[0078] S25, describes the system allocatable bandwidth resource constraint 3: and constraint 4:

[0079] S3 uses the Deep Deterministic Policy Gradient (DDPG) in deep reinforcement learning to solve the joint optimization problem of remote radio unit deployment and bandwidth resource allocation, which specifically includes the following steps:

[0080] S31, model the optimization problem as a Markov process, and then define the state space, action space and reward function;

[0081] S32 represents the location information and data transmission rate of each terminal in the state space S, and the formula is expressed as

[0082] S33 represents the possible deployment positions of the RRUs and the allocation of bandwidth resources in the action space A, and the formula is expressed as

[0083] S34, the reward function r(S,A) can be designed to maximize the total system throughput, and the formula is expressed as:

[0084] S35, define the historical state H t H t ={S 1 ,S 2 ,...,S t}, introduce the discount factor γ, which represents the total reward r t sum For r t sum =r t+1 +γr t+2 +...+γ T-t-1 r T , where S a(a=1,2,…,t) represents any state before the historical moment t, r b (b=1,2…,T) represents the reward corresponding to any moment in the current round, where the discount factor γ corresponding to each moment is different (specifically distinguished by different superscripts);

[0085] S36, which means the expected return of state S is the state value function V π (S) = E[r t sum |S t =S] = E[r t+1 +…+γ T-t-1 r T |S t =S];

[0086] The state value function represents the expected cumulative reward that the agent can obtain in the current state. The state value function measures the expected long-term return of the agent in a certain state, that is, the expected value of the cumulative reward that can be obtained after taking the optimal strategy in the current state.

[0087] S37, which indicates that the expected benefit of taking action A in state S is the action value function Q(S,A) = E[r t sum |S t =S,A t =A] = E[r t+1 +...+γ T-t-1 r T |S t =S,A t =A];

[0088] The action value function represents the expected cumulative reward that the agent can obtain after taking a specific action in the current state. The action value function measures the expected value of the cumulative reward that can be obtained after taking a specific action in a certain state.

[0089] S38, describing the state value function V π The mutual conversion between (S) and the action value function Q(S,A) is expressed by the following formula:

[0090] Among them, the strategy of the agent can be optimized by converting between the state value function and the action value function. The mutual conversion between the state value function and the action value function can help the agent better evaluate the value of different states and actions, thereby guiding the agent to learn the optimal strategy. The setting of the conversion function is to achieve effective conversion between the state value function and the action value function in reinforcement learning, so as to improve the learning efficiency and performance of the agent.

[0091] S39, define four deep neural networks (DNNs): policy network μ(S, ω), critic network Q(S, A, λ), target policy network μ′(S, ω′) and target critic network Q′(S, A, λ′), where ω, λ, ω′, λ′ represent the parameters of each DNN;

[0092] S310, indicating state S t Action A t Can be A t =μ(S t ,ω)+N t , where N t To detect noise, appropriate noise is added to prevent falling into local optimum;

[0093] S311, represents the target action value function for Among them, i represents the current round, i+1 represents the next round of the current round, μ′(S i+1 ,ω′) represents the output of the current round target strategy network, Q′(S i+1 ,μ′(S i+1 ,ω′),λ′) indicates that the output of the target strategy network is μ′(S i+1 ,ω′), the output of the target critic network;

[0094] S312, using gradient descent to minimize the target action-value function The mean square error between the critic network output Q(S,A,λ) and the loss function is expressed as Where M is the sampling batch size, that is, the number of rounds sampled;

[0095] S313, the loss function Loss(ω) of the policy network is expressed as

[0096] S314, minimize the loss function of the policy network and obtain the policy gradient of the sampling strategy

[0097] S315, update the parameters of the two target networks (i.e., the target policy network and the target critic network): λ”←τλ+(1-τ)λ′ and ω”←τω+(1-τ)ω′, where τ is the soft update coefficient.

[0098] It should be noted that the cable tunnel RF remote unit deployment and bandwidth resource allocation optimization method based on deep reinforcement learning method provided by the embodiment of the present invention uses machine learning method to solve the idea of ​​RF remote unit deployment and bandwidth resource allocation. Unit deployment and resource allocation are abstracted into optimization problems through system modeling, and the RF remote unit location deployment and bandwidth resource allocation are regarded as independent variables, and the total system throughput is regarded as a dependent variable. Then the problem is Markov modeled, and then the deep reinforcement learning method can be used to solve it. By defining the observation space, action space and reward function, after a certain period of training, the problem can be directly solved according to the observed data in the execution stage. Therefore, the cable tunnel RF remote unit deployment and bandwidth resource optimization method based on machine learning proposed in the embodiment of the present invention has the advantages of good optimization performance and high reliability. In addition, the deep reinforcement learning method DDPG used in the embodiment of the present invention is a relatively novel and excellent algorithm in machine learning. It not only overcomes the defects of traditional optimization algorithms that optimize variables one by one, but also improves the total system throughput.

[0099] Based on the above embodiment and optional embodiment, the present invention proposes another optional implementation of a wireless sensor network resource allocation method, the method comprising:

[0100] S01: Initialize DNN parameters ω, λ, ω′, λ′, number of rounds ep max , number of steps T, learning rate α, detection noise N t , discount rate γ, soft update coefficient τ, replay buffer capacity M (i.e. sampling batch size), batch size m.

[0101] S02: Initialize channel status and resource requirements.

[0102] S03: Get status S t And choose action A accordingly t =μ(S t ω)+N t .

[0103] S04: According to action A t Determine the RRU placement location v and bandwidth resource allocation B.

[0104] S05: Execute the corresponding strategy and obtain the next state S t+1 and reward r t .

[0105] S06: Store the transformation in the replay buffer (S t ,A t ,r t ,S t+1 ) and randomly sample m batches of transformations, where S t+1Indicates the state at the next moment after time t.

[0106] S07: Determine the target action-value function And minimize the loss function to update the critical network parameter λ.

[0107] S08: Update the actor network parameters ω, and simultaneously update the target actor network parameters ω′ and the target critic network parameters λ′.

[0108] S09: Output the trained neural network parameters ω, λ, ω′, λ′.

[0109] S010: Set simulation parameters. All RRUs and terminals are randomly distributed within a range of 1000m×500m. The number of terminals is 50 and the number of RRUs is 2 to 8. Carrier frequency f = 0.1 GHz (gigahertz), path loss index ρ = 3.27, noise power σ 2 =-122dBm; total system bandwidth resources that can be allocated B T =1MHz; all terminals are randomly generated in the specified area.

[0110] S011: Compare with the benchmark algorithms, including Uniform RRU Deployment (URD): all RRU units are evenly distributed, but bandwidth resources are randomly allocated; Average Bandwidth Resource Allocation (ABRA): bandwidth resources are evenly distributed among terminals, but all RRU units are randomly allocated; Deep Q Network (DQN): Use the Deep Q Network algorithm to optimize location deployment and bandwidth resource allocation.

[0111] Figure 2 This is an optional comparison result diagram of the cumulative rewards of an agent during the entire training process according to an embodiment of the present invention, in which the horizontal axis represents the number of episodes (Episode) and the vertical axis represents the total reward (Sum reward) of each episode. It can be observed that as the training time increases, the total reward gradually increases, indicating that the actions taken by the agent receive higher rewards, and the DDPG method has an 8.45% increase in convergence reward value relative to the DQN method. The convergence time of the DDPG algorithm is about 200 episodes, while the convergence time of the DQN algorithm is about 300 episodes.

[0112] Figure 3This is a comparison chart of the change of the total system throughput (Total system throughput) with the number of RRUs (Number of RRUs) under different optional algorithms according to an embodiment of the present invention. It can be observed that the DDPG and DQN algorithms are significantly better than the ABRA and URD algorithms. When the number of radio remote units is 5, the total system transmission rate of the DDPG method of the embodiment of the present invention is increased by 12.04%, 60.64% and 132.21% compared with DQN, ABRA and URD respectively. As the number of RRUs increases, the total system throughput increases accordingly and eventually converges. This is because, as the number of RRUs increases, there are better deployment locations to serve multiple users. However, when the number of RRUs reaches a certain level, due to the limited total bandwidth resources, the growth rate of the total system throughput will decrease, which limits the bandwidth that can be allocated to users.

[0113] In this embodiment, a wireless sensor network resource allocation device is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module" and "device" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0114] According to an embodiment of the present invention, there is also provided an apparatus embodiment for implementing the above wireless sensor network resource allocation method. Figure 4 is a schematic diagram of the structure of a wireless sensor network resource allocation device according to an embodiment of the present invention. Figure 4 As shown, the wireless sensor network resource allocation device includes: a location acquisition module 400, a transmission rate determination module 402, an initial strategy determination module 404, and a strategy optimization module 406, wherein:

[0115] The location acquisition module 400 is used to acquire location information corresponding to a plurality of terminals included in the wireless sensor network;

[0116] A transmission rate determination module 402, connected to the location acquisition module 400, for determining data transmission rates between the plurality of terminals and the corresponding radio remote units;

[0117] An initial strategy determination module 404, connected to the transmission rate determination module 402, is used to obtain an initial resource allocation strategy of the wireless sensor network, wherein the initial resource allocation strategy is used to indicate the deployment locations and initial bandwidth resource allocation strategies corresponding to the multiple radio remote units included in the wireless sensor network;

[0118] The strategy optimization module 406 is connected to the initial strategy determination module 404, and is used to optimize the initial resource allocation strategy based on the location information corresponding to the multiple terminals, and the data transmission rates between the multiple terminals and the corresponding radio remote units, with the maximum total system throughput in the wireless sensor network as the optimization goal, to obtain the target resource allocation strategy of the wireless sensor network.

[0119] In the embodiment of the present invention, a location acquisition module 400 is provided to acquire location information corresponding to a plurality of terminals included in the wireless sensor network; a transmission rate determination module 402 is connected to the location acquisition module 400 and is used to determine the data transmission rate between the plurality of terminals and the corresponding radio remote units; an initial strategy determination module 404 is connected to the transmission rate determination module 402 and is used to acquire an initial resource allocation strategy of the wireless sensor network, wherein the initial resource allocation strategy is used to indicate the deployment locations corresponding to the plurality of radio remote units included in the wireless sensor network and the initial bandwidth resource allocation strategy; a strategy optimization module 406 is connected to the initial strategy determination module 40 4. It is used to optimize the initial resource allocation strategy based on the location information corresponding to multiple terminals and the data transmission rates between the multiple terminals and the corresponding radio frequency remote units, with the maximum total system throughput in the wireless sensor network as the optimization goal, to obtain the target resource allocation strategy of the wireless sensor network, and achieve the purpose of optimizing the resource allocation strategy of the wireless sensor network with the maximum total system throughput in the wireless sensor network as the optimization goal, thereby achieving the technical effect of improving the resource allocation efficiency and allocation accuracy of the wireless sensor network, and further solving the technical problems of low network resource allocation efficiency and low accuracy in the wireless sensor network resource allocation method in the related art.

[0120] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0121] It should be noted that the above-mentioned location acquisition module 400, transmission rate determination module 402, initial strategy determination module 404, and strategy optimization module 406 correspond to steps S102 to S108 in the embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment. It should be noted that the above-mentioned modules can be run in a computer terminal as part of the device.

[0122] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0123] The above-mentioned wireless sensor network resource allocation device may also include a processor and a memory. The above-mentioned location acquisition module 400, transmission rate determination module 402, initial strategy determination module 404, strategy optimization module 406, etc. are all stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to realize corresponding functions.

[0124] The processor includes a kernel, which retrieves the corresponding program module from the memory. The kernel may be one or more. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory includes at least one memory chip.

[0125] According to an embodiment of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute any of the wireless sensor network resource allocation methods.

[0126] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group, and the non-volatile storage medium includes a stored program.

[0127] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following functions: obtaining location information corresponding to each of the multiple terminals included in the wireless sensor network; determining the data transmission rates between the multiple terminals and the corresponding radio remote units; obtaining an initial resource allocation strategy for the wireless sensor network, wherein the initial resource allocation strategy is used to indicate the deployment locations corresponding to each of the multiple radio remote units included in the wireless sensor network and an initial bandwidth resource allocation strategy; based on the location information corresponding to each of the multiple terminals and the data transmission rates between the multiple terminals and the corresponding radio remote units, the initial resource allocation strategy is optimized with the maximum total system throughput in the wireless sensor network as the optimization goal, so as to obtain a target resource allocation strategy for the wireless sensor network.

[0128] According to an embodiment of the present application, an embodiment of a processor is also provided. Optionally, in this embodiment, the processor is used to run a program, wherein when the program is run, any one of the wireless sensor network resource allocation methods is executed.

[0129] According to an embodiment of the present application, an embodiment of a computer program product is also provided. When executed on a data processing device, it is suitable for executing a program that initializes any one of the steps of the wireless sensor network resource allocation method described above.

[0130] Optionally, the above-mentioned computer program product, when executed on a data processing device, is suitable for executing a program that initializes the following method steps: obtaining location information corresponding to multiple terminals included in the wireless sensor network; determining data transmission rates between the multiple terminals and the corresponding radio remote units; obtaining an initial resource allocation strategy for the wireless sensor network, wherein the initial resource allocation strategy is used to indicate the deployment locations corresponding to the multiple radio remote units included in the wireless sensor network and an initial bandwidth resource allocation strategy; based on the location information corresponding to the multiple terminals, and the data transmission rates between the multiple terminals and the corresponding radio remote units, the initial resource allocation strategy is optimized with the maximum total system throughput in the wireless sensor network as the optimization goal, to obtain a target resource allocation strategy for the wireless sensor network.

[0131] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: obtaining location information corresponding to a plurality of terminals included in a wireless sensor network; determining data transmission rates between the plurality of terminals and corresponding radio remote units; obtaining an initial resource allocation strategy of the wireless sensor network, wherein the initial resource allocation strategy is used to indicate deployment locations corresponding to the plurality of radio remote units included in the wireless sensor network and an initial bandwidth resource allocation strategy; based on the location information corresponding to the plurality of terminals and the data transmission rates between the plurality of terminals and the corresponding radio remote units, the initial resource allocation strategy is optimized with the maximum total system throughput in the wireless sensor network as the optimization target, so as to obtain a target resource allocation strategy of the wireless sensor network.

[0132] The above sequence of the embodiments of the present invention is for description only and does not represent the superiority or inferiority of the embodiments.

[0133] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above modules can be a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, modules or indirect coupling or communication connection of modules, which can be electrical or other forms.

[0135] The modules described above as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0136] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0137] If the above-mentioned integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a non-volatile storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0138] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A wireless sensor network resource allocation method, It is characterized in that include: Acquire location information corresponding to a plurality of terminals included in the wireless sensor network; Determining data transmission rates between the plurality of terminals and the corresponding radio remote units respectively; Acquire an initial resource allocation strategy of the wireless sensor network, wherein the initial resource allocation strategy is used to indicate deployment locations and initial bandwidth resource allocation strategies corresponding to a plurality of radio remote units included in the wireless sensor network; Based on the location information corresponding to the multiple terminals respectively, and the data transmission rates between the multiple terminals and the corresponding radio remote units respectively, and taking the maximum total system throughput in the wireless sensor network as the optimization goal, the initial resource allocation strategy is optimized to obtain the target resource allocation strategy of the wireless sensor network.

2. The method according to claim 1, It is characterized in that The method of optimizing the initial resource allocation strategy based on the location information corresponding to the multiple terminals and the data transmission rates between the multiple terminals and the corresponding radio remote units, taking the maximum total system throughput in the wireless sensor network as the optimization goal, and obtaining the target resource allocation strategy of the wireless sensor network includes: Constructing a state space based on the location information corresponding to the plurality of terminals and the data transmission rates between the plurality of terminals and the corresponding radio remote units; Construct an action space based on multiple candidate resource allocation strategies; Constructing a reward function based on the total throughput of the system; Based on the state space and the action space, with the goal of maximizing the function value of the reward function, optimizing the parameters of the initial resource allocation model to obtain a target resource allocation model; The target resource allocation model is adopted to obtain the target resource allocation strategy.

3. The method according to claim 2, It is characterized in that The method of optimizing the parameters of the initial resource allocation model based on the state space and the action space and taking the maximum function value of the reward function as the goal to obtain the target resource allocation model includes: Based on the state space and the action space, with the goal of maximizing the function value of the reward function, a deep deterministic policy gradient algorithm is used to optimize the parameters of the initial resource allocation model to obtain the target resource allocation model.

4. The method according to claim 3, It is characterized in that The initial resource allocation model includes a policy network, a critic network, a target policy network and a target critic network. Based on the state space and the action space, with the function value of the reward function as the maximum, a deep deterministic policy gradient algorithm is used to optimize the parameters of the initial resource allocation model to obtain the target resource allocation model, including: Inputting the state space into the policy network to obtain a first action determined from the action space; Using the critic network to evaluate the first action, obtaining an estimated value of a first action-value function; Inputting the state space into the target policy network to obtain a second action determined from the action space; Using the target critic network to evaluate the second action, obtaining an estimated value of a second action-value function; Based on the first action-value function estimate, a first loss is obtained; based on the second action-value function estimate, a second loss is obtained; Optimizing parameters of the policy network, the critic network, the target policy network, and the target critic network based on the first loss and the second loss; Repeat the above operations until a predetermined termination condition is reached; The target resource allocation model is constructed based on the policy network with updated parameters, the critic network with updated parameters, the target policy network with updated parameters, and the target critic network with updated parameters obtained when the predetermined termination condition is reached.

5. The method according to claim 4, It is characterized in that The obtaining a second loss based on the second action-value function estimation value includes: Obtaining a target action-value function based on the reward function, the second action, and the second action-value function estimate; The second loss is obtained based on a mean square error between the target action-value function and an estimated value of the second action-value function.

6. The method according to claim 2, It is characterized in that The method of optimizing the parameters of the initial resource allocation model based on the state space and the action space and taking the maximum function value of the reward function as the goal to obtain the target resource allocation model includes: Determine constraints, wherein the constraints include at least bandwidth resource allocation constraints, data transmission power constraints, and data transmission rate constraints: Based on the state space, the action space and the constraint conditions, the parameters of the initial resource allocation model are optimized with the goal of maximizing the function value of the reward function to obtain the target resource allocation model.

7. The method according to claim 6, It is characterized in that The bandwidth resource allocation constraint is used to indicate that the total amount of bandwidth resources respectively allocated by the multiple radio remote units to the corresponding terminals is less than or equal to a predetermined resource amount; The data transmission power constraint is used to indicate that the data transmission power corresponding to any terminal among the multiple terminals is within a predetermined transmission power range; The data transmission rate constraint is used to indicate that a data transmission rate between any terminal among the multiple terminals and a corresponding radio remote unit is less than a predetermined transmission rate.

8. The method according to claim 7, It is characterized in that The data transmission rate constraint is obtained by: Among them, TR* n,k Indicates the predetermined transmission rate corresponding to any terminal; represents the bandwidth resources allocated by any radio remote unit to any terminal; n,k represents the data transmission power of any terminal transmitting data to any radio remote unit; n,k represents the channel gain between any radio remote unit and any terminal; σ 2 Indicates the preset noise power.

9. A wireless sensor network resource allocation device, It is characterized in that include: A location acquisition module, used to acquire location information corresponding to a plurality of terminals included in the wireless sensor network; A transmission rate determination module, used to determine the data transmission rates between the plurality of terminals and the corresponding radio remote units respectively; An initial strategy determination module, used to obtain an initial resource allocation strategy of the wireless sensor network, wherein the initial resource allocation strategy is used to indicate the deployment locations and initial bandwidth resource allocation strategies corresponding to a plurality of radio remote units included in the wireless sensor network; A strategy optimization module is used to optimize the initial resource allocation strategy based on the location information corresponding to the multiple terminals and the data transmission rates between the multiple terminals and the corresponding radio remote units, with the maximum total system throughput in the wireless sensor network as the optimization goal, to obtain the target resource allocation strategy of the wireless sensor network.

10. An electronic device, It is characterized in that It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the wireless sensor network resource allocation method described in any one of claims 1 to 8.

Images