D2D (Device-to-Device) communication resource allocation method and device for electric power emergency communication

By constructing Markov decision-making process and deep reinforcement learning model, optimizing the allocation of D2D communication resources on the ground of UAV, the problem of resource allocation complexity in power emergency communication systems is solved, and channel capacity and connection success rate are improved.

CN118972958APending Publication Date: 2024-11-15STATE GRID HEBEI ELECTRIC POWER CO LTD +2
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411233505.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In the power emergency communication system, the drone relay D2D communication resource allocation algorithm is complex, and traditional methods require fine scenario modeling, making it difficult to optimize resource allocation.

Method used

The resource allocation problem in the drone ground D2D emergency communication system is constructed into a Markov decision-making process, the reward function is designed, and the resource allocation strategy is trained, including channel allocation bandwidth, to optimize resource allocation by building a deep reinforcement learning model.

Benefits of technology

Through the training of deep reinforcement learning models, good power control is achieved, communication interference is reduced, and D2I channel capacity and D2D connection success rate are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972958A_ABST
    Figure CN118972958A_ABST
Patent Text Reader

Abstract

The invention provides a D2D communication resource allocation method and device for electric power emergency communication. The method comprises the following steps: initializing parameters of an unmanned aerial vehicle ground D2D emergency communication system; constructing a resource allocation problem in the unmanned aerial vehicle ground D2D emergency communication system as a Markov decision process to obtain a reward function; constructing a deep reinforcement learning model, taking the environment state of the D2D communication pair as input, taking a resource allocation strategy as output, and training and updating the deep reinforcement learning model according to the reward function; and realizing D2D communication resource allocation by adopting the trained deep reinforcement learning model. By designing the reward function and the training mechanism, good power control can be realized, the communication interference is reduced, and the D2I channel capacity and the D2D connection success rate are effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of low-altitude electric power communication, and in particular to a D2D communication resource allocation device for electric power emergency communication. Background Art

[0002] With the rapid development of drone technology, its application in various fields is becoming more and more extensive, especially in the field of power emergency communication, drone relay emergency communication system is becoming a research hotspot.

[0003] In the power emergency communication system, resource allocation is extremely important for the D2D communication network of drone relay. However, this field currently mainly focuses on the communication between drones and power equipment, and does not fully consider the communication on the ground. In addition, the traditional convex optimization communication resource allocation method requires sophisticated scenario modeling, and it is relatively complicated to find the optimal solution through the modeling system. Summary of the invention

[0004] The embodiment of the present invention provides a D2D communication resource allocation device for power emergency communication to solve the problem of complex D2D communication resource allocation algorithm.

[0005] In a first aspect, an embodiment of the present invention provides a D2D communication resource allocation method for power emergency communication, including:

[0006] Initialize the parameters of the drone-to-ground D2D emergency communication system;

[0007] The resource allocation problem in the UAV-to-ground D2D emergency communication system is constructed as a Markov decision process to obtain a reward function;

[0008] A deep reinforcement learning model is constructed, the environmental state of the D2D communication pair is used as input, the resource allocation strategy is used as output, and the deep reinforcement learning model is trained and updated according to the reward function; the resource allocation strategy includes a channel allocation bandwidth corresponding to the communication pair; the D2D communication pair includes two power devices;

[0009] The trained deep reinforcement learning model is used to realize D2D communication resource allocation.

[0010] In a second aspect, an embodiment of the present invention provides a D2D communication resource allocation device for power emergency communication, including:

[0011] Initialization module, used to initialize the parameters of the drone ground D2D emergency communication system;

[0012] A reward function acquisition module is used to construct the resource allocation problem in the UAV ground D2D emergency communication system as a Markov decision process to obtain a reward function;

[0013] A model training module, for constructing a deep reinforcement learning model, taking the environmental state of the D2D communication pair as input, taking the resource allocation strategy as output, and training and updating the deep reinforcement learning model according to the reward function; the resource allocation strategy includes a channel allocation bandwidth corresponding to the communication pair; the D2D communication pair includes two power devices;

[0014] The communication resource allocation module is used to implement D2D communication resource allocation using a trained deep reinforcement learning model.

[0015] The embodiment of the present invention provides a D2D communication resource allocation method and device for power emergency communication. The method first initializes the parameters of the unmanned aerial vehicle ground D2D emergency communication system; then constructs the resource allocation problem in the unmanned aerial vehicle ground D2D emergency communication system as a Markov decision process to obtain a reward function; constructs a deep reinforcement learning model, takes the environmental state of the D2D communication pair as input, takes the resource allocation strategy as output, and trains and updates the deep reinforcement learning model according to the reward function; the resource allocation strategy includes the channel allocation bandwidth corresponding to the communication pair; the D2D communication pair includes two power equipment; finally, the trained deep reinforcement learning model is used to implement D2D communication resource allocation. The embodiment of the present application can achieve good power control, reduce communication interference, and effectively improve the D2I channel capacity and D2D connection success rate by designing a reward function and a training mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 This is an application scenario diagram of a D2D communication resource allocation method for power emergency communication provided by an embodiment of the present invention;

[0018] Figure 2 It is a flow chart of a D2D communication resource allocation method for power emergency communication provided by an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of the structure of a deep reinforcement learning model provided by an embodiment of the present invention;

[0020] Figure 4 It is a structural schematic diagram of a D2D communication resource allocation device for power emergency communication provided by an embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0024] See also Figure 1 , which shows an application scenario diagram of the D2D communication resource allocation method for power emergency communication provided in this embodiment. The application scenario includes a drone and multiple power equipment located on the ground, and the multiple power equipment can establish communication connections through the drone.

[0025] See also Figure 2 , which shows a flow chart of the implementation of the D2D communication resource allocation method for power emergency communication provided by an embodiment of the present invention, which is described in detail as follows:

[0026] S101: Initialize the parameters of the drone-to-ground D2D emergency communication system.

[0027] In this embodiment, it is first necessary to construct the state space of the power equipment on the ground in the drone relay emergency communication system for training the power equipment. A single power equipment is regarded as an intelligent agent, and the state space of the intelligent agent includes the channel gain observed locally by the single intelligent agent, the interference channel of other power equipment, the interference channel transmitted by the intelligent agent itself to the air base station, and the frequency used by the intelligent agent.

[0028] Secondly, it is necessary to determine the action space of the power device, which is the resource allocation strategy of the power device. The resource allocation strategy may include the transmission power of the power device and the channel bandwidth of the power device. The transmission power is selected from [0-pm].

[0029] In a possible implementation, the specific implementation process of S101 includes:

[0030] Initialize the number of D2D links K = {1, 2, ..., K} and the number of D2I links M = {1, 2, ..., M}, and initialize the acceleration V of the ground mobile power equipment a , Final running speed V f、Actor network parameters θ a and the critic network parameter θ c ;

[0031] Constructing a communication neighbor table corresponding to the electric power equipment, wherein the communication neighbor table includes a plurality of communication pairs, and each communication pair is a communication pair consisting of two nearest neighboring electric power equipment;

[0032] Take the power equipment as an intelligent agent, initialize the environment state S, weight coefficient λ and learning rate γ of the intelligent agent;

[0033] Among them, the environmental state S = {p, w, g, gm, e}; p represents the transmission power of the power equipment, w represents the bandwidth of the channel, g represents the channel interference, gm represents the gain, and e represents the battery capacity of the drone corresponding to the power equipment.

[0034] In this embodiment, D2D (device-to-device) communication technology refers to a communication method in which two peer user nodes communicate directly. In a distributed network composed of D2D communication users, each user node can send and receive signals and has the function of automatic routing (forwarding messages). Participants in the network share some of their hardware resources, including information processing, storage, and network connection capabilities. These shared resources provide services and resources to the network and can be directly accessed by other users without going through an intermediate entity. In a D2D communication network, user nodes play the role of both server and client. Users are aware of each other's existence and self-organize to form a virtual or actual group.

[0035] D2I (device to infrastructure) communication technology refers to a communication method in which user nodes communicate directly with infrastructure such as base stations.

[0036] S102: Constructing the resource allocation problem in the UAV-to-ground D2D emergency communication system as a Markov decision process to obtain a reward function.

[0037] In a possible implementation, the specific implementation process of S102 includes:

[0038] Using the weighted index of D2D communication success rate and D2D channel capacity as the reward function of the Markov decision process;

[0039] The reward function is:

[0040]

[0041] Where R represents the reward function, λ represents the weighted coefficient of air-to-ground channel capacity and D2D connection success rate, and C m [m] represents the capacity of the air-to-ground channel with sequence number m, m represents the sequence number of the power equipment communicating with the air base station, kd represents the sequence number of the power equipment for D2D communication, M represents the number of D2I links, K represents the number of D2D links, L kd [kd] represents the connection success rate of the D2D communication pair with sequence number kd.

[0042] Designing the communication process as a Markov decision process means taking the ground power equipment as the training agent, inputting the state space and action space, and interacting with the environment to train the neural network.

[0043] S103: Construct a deep reinforcement learning model, take the environmental state of the D2D communication pair as input, take the resource allocation strategy as output, and train and update the deep reinforcement learning model according to the reward function; the resource allocation strategy includes the channel allocation bandwidth corresponding to the communication pair; the D2D communication pair includes two power devices.

[0044] In a possible implementation, Figure 3 As shown, the specific implementation process of building a deep reinforcement learning model in S103 includes:

[0045] The dual action decision network and dual delay deep deterministic strategy (DA-DT3) method was adopted, and the KAN network was used as the basic neural network to construct an actor-critic network; the actor-critic network included two Actor networks, two Critic networks and corresponding two target Actor networks and four target Critic networks.

[0046] In this embodiment, the environment state S obtained by the agent is used as an input variable during the training process, and the action state value function Q is output, and a dual actor network is used to increase the training stability. Among them, using a dual actor network to increase the training stability means using dual actors to update the neural network so that its action state value function Q approaches the optimal action state value function Q*.

[0047] In a possible implementation, Figure 3 As shown, in S103, the environmental states of multiple groups of D2D communication pairs are used as input, the resource allocation strategy is used as output, and the specific implementation process of training and updating the deep reinforcement learning model according to the reward function includes:

[0048] S201: Input the current environment state into two Actor networks respectively, obtain two current resource allocation strategies, and determine the next environment state according to the current environment state and the corresponding current resource allocation strategy;

[0049] S202: Input the next environment state into the two target Actor networks respectively to obtain two next resource allocation strategies;

[0050] S203: Input the two current state-action arrays into their respective corresponding Critic networks to obtain corresponding state-action value functions; and select the smaller value of the two state-action value functions as the current Q value; a current state-action array includes a current environment state and a current resource allocation strategy;

[0051] S204: Input two next state-action arrays into four target critic networks respectively to obtain corresponding target state action value functions; and select the minimum value among all target state action value functions as the target Q value; a next state-action array includes a next environment state and a next resource allocation strategy;

[0052] S205: Update the parameters of the Actor network and the Critic network based on the current Q value, the target Q value and the reward function.

[0053] In this embodiment, after the current resource allocation strategy is calculated in S201, the environment is interacted with based on the current resource allocation strategy to reach the next environment state and obtain the corresponding reward function, and the observed environment state, the selected action, the obtained reward function and the next environment state will be stored in the experience replay buffer. In S202, the next environment state is randomly sampled from the experience replay buffer and input into two target Actor networks, two next actions are output, and noise is added to obtain the next resource allocation strategy.

[0054] In this embodiment, the target-critic network generates two state-action value functions and takes the minimum value to train and update the neural network, thereby reducing the overestimation problem of the state value function Q.

[0055] In a possible implementation, training and parameter transfer of the reinforcement learning model refers to transferring directional parameters of the network through the stochastic optimization method adam of adaptive momentum. The specific implementation process of S205 includes:

[0056] Update the parameters of the Actor network and the Critic network based on the loss function;

[0057] The loss function is:

[0058]

[0059] Where k represents the sequence number of the D2D communication link, t represents the current number of iterations, represents the original neural network parameters, represents the target neural network parameters, Represents the current state of the environment of the input Actor network, Represents the next environment state of the input target Actor network, Represents the current resource allocation strategy for the Actor network output, represents the next resource allocation strategy output by the target Actor network, Q'k represents the target Q value, Qk represents the current Q value, represents the reward function at k steps, and γ represents the discount rate.

[0060] In this embodiment, the target neural network parameter update formula also includes:

[0061]

[0062] in, represents the updated target neural network parameters, represents the original neural network parameters, and τ represents the updated weights.

[0063] Among them, the original neural network is the Actor network and Critic network in the deep reinforcement learning model, and the target neural network is the target Actor network and target Critic network in the deep reinforcement learning model.

[0064] In a possible implementation, the further implementation process of S205 includes:

[0065] The parameters of the Critic network are updated based on the loss function, and the parameters of the Actor network are updated after a delay of k steps after the Critic network is updated.

[0066] The method provided in this embodiment adopts a delayed update strategy, first updating the critic network and then updating the actor network, thereby reducing the cumulative error and the variance.

[0067] In a possible implementation manner, the implementation process of S103 further includes:

[0068] The parameters of the target Actor network and the target Critic network are updated based on the current Q value, the target Q value and the reward function.

[0069] In this embodiment, the method provided in this embodiment also includes:

[0070] Get the state-action value function output by the two critic networks, and get the target Q value output by the target critic network;

[0071] The absolute value of the difference between each state-action value function and the target Q value is calculated as the time difference error (TD-Error). The smaller value of the two time difference errors is selected for simulation verification of the deep reinforcement learning model to improve the calculation accuracy of the deep reinforcement learning model.

[0072] S104: Using the trained deep reinforcement learning model to implement D2D communication resource allocation.

[0073] In this embodiment, each D2D power device uses the state obtained according to the environment as input to obtain a resource allocation strategy. The communication resource optimization of the D2D link and the D2I link is completed; it means allocating bandwidth to the communication pair within the communication time slot range, configuring the communication resource allocation method to each communication pair, and completing the resource allocation under emergency communication.

[0074] It can be seen from the above embodiments that the embodiments of the present invention provide a resource allocation method based on improved deep reinforcement learning DA-TD3 to improve the air-to-ground channel capacity and D2D channel connection success rate of medium and low altitude communications, so as to find the transmission power of the optimal power equipment, reduce co-frequency interference, and improve the transmission efficiency and channel capacity between mobile ground power equipment.

[0075] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0076] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0077] Figure 4 The structure diagram of the D2D communication resource allocation device for power emergency communication provided by an embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0078] like Figure 4 As shown, the D2D communication resource allocation device 100 for power emergency communication includes:

[0079] Initialization module 110, used to initialize the parameters of the UAV ground D2D emergency communication system;

[0080] A reward function acquisition module 120 is used to construct the resource allocation problem in the UAV ground D2D emergency communication system as a Markov decision process to obtain a reward function;

[0081] The model training module 130 is used to construct a deep reinforcement learning model, take the environmental state of the D2D communication pair as input, take the resource allocation strategy as output, and train and update the deep reinforcement learning model according to the reward function; the resource allocation strategy includes the channel allocation bandwidth corresponding to the communication pair; the D2D communication pair includes two power devices;

[0082] The communication resource allocation module 140 is used to implement D2D communication resource allocation using a trained deep reinforcement learning model.

[0083] In a possible implementation, the initialization module 110 includes:

[0084] Initialize the number of D2D links and D2I links, and initialize the acceleration V of the ground mobile power equipment a , Final running speed V f 、Actor network parameters θ a and the critic network parameter θ c ;

[0085] Constructing a communication neighbor table corresponding to the electric power equipment, wherein the communication neighbor table includes a plurality of communication pairs, and each communication pair is a communication pair consisting of two nearest neighboring electric power equipment;

[0086] Take the power equipment as an intelligent agent, initialize the environment state S, weight coefficient λ and learning rate γ of the intelligent agent;

[0087] Among them, the environmental state S = {p, w, g, gm, e}; p represents the transmission power of the power equipment, w represents the bandwidth of the channel, g represents the channel interference, gm represents the gain, and e represents the battery capacity of the drone corresponding to the power equipment.

[0088] In one possible implementation, the reward function acquisition module 120 includes:

[0089] Using the weighted index of D2D communication success rate and D2D channel capacity as the reward function of the Markov decision process;

[0090] The reward function is:

[0091]

[0092] Where R represents the reward function, λ represents the weighted coefficient of air-to-ground channel capacity and D2D connection success rate, and C m [m] represents the capacity of the air-to-ground channel with sequence number m, m represents the sequence number of the power equipment communicating with the air base station, kd represents the sequence number of the power equipment for D2D communication, M represents the number of D2I links, K represents the number of D2D links, L kd[kd] represents the connection success rate of the D2D communication pair with sequence number kd.

[0093] In one possible implementation, the model training module 130 includes a model building unit for:

[0094] A dual-action decision network and a dual-delay deep deterministic strategy method are adopted, and a KAN network is used as the basic neural network to construct an actor-critic network; the actor-critic network includes two Actor networks, two Critic networks, and corresponding two target Actor networks and four target Critic networks.

[0095] In a possible implementation, the model training module 130 further includes an original model training unit, which is used to:

[0096] Input the current environment state into two Actor networks respectively to obtain two current resource allocation strategies, and determine the next environment state according to the current environment state and the corresponding current resource allocation strategy;

[0097] Input the next environment state into the two target Actor networks respectively to obtain two next resource allocation strategies;

[0098] Input the two current state-action arrays into their respective corresponding Critic networks to obtain the corresponding state-action value functions; and select the smaller value of the two state-action value functions as the current Q value; a current state-action array includes the current environment state and a current resource allocation strategy;

[0099] Input two next state-action arrays into four target critic networks respectively to obtain the corresponding target state action value functions; and select the minimum value of all target state action value functions as the target Q value; a next state-action array includes the next environment state and a next resource allocation strategy;

[0100] The parameters of the Actor network and the Critic network are updated based on the current Q value, the target Q value and the reward function.

[0101] In a possible implementation, updating the parameters of the Actor network and the Critic network based on the current Q value, the target Q value, and the reward function in the original model training unit includes:

[0102] Update the parameters of the Actor network and the Critic network based on the loss function;

[0103] The loss function is:

[0104]

[0105] Where k represents the sequence number of the D2D communication link, t represents the current number of iterations, represents the original neural network parameters, represents the target neural network parameters, Represents the current state of the environment of the input Actor network, Represents the next environment state of the input target Actor network, Represents the current resource allocation strategy for the Actor network output, represents the next resource allocation strategy output by the target Actor network, Q'k represents the target Q value, Qk represents the current Q value, represents the reward function at k steps, and γ represents the discount rate.

[0106] In a possible implementation, the original model training unit further includes:

[0107] The parameters of the Critic network are updated based on the loss function, and the parameters of the Actor network are updated after a delay of k steps after the Critic network is updated.

[0108] In a possible implementation, the model training module 130 further includes:

[0109] The parameters of the target Actor network and the target Critic network are updated based on the current Q value, the target Q value and the reward function.

[0110] Figure 5 is a schematic diagram of a terminal provided by an embodiment of the present invention. Figure 5 As shown, the terminal 5 of this embodiment includes: a processor 50 and a memory 51. The memory 51 is used to store a computer program 52, and the processor 50 is used to call and run the computer program 52 stored in the memory 51 to perform the steps in the above-mentioned D2D communication resource allocation method for power emergency communication, for example Figure 2 Alternatively, the processor 50 is used to call and run the computer program 52 stored in the memory 51 to implement the functions of each module / unit in the above-mentioned device embodiments, for example Figure 4 The functions of the modules 110 to 140 are shown.

[0111] Exemplarily, the computer program 52 may be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 52 in the terminal 5. For example, the computer program 52 may be divided into Figure 4 Modules 110 to 130 are shown.

[0112] The terminal 5 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal 5 may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5 It is only an example of terminal 5 and does not constitute a limitation on terminal 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal may also include input and output devices, network access devices, buses, etc.

[0113] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0114] The memory 51 may be an internal storage unit of the terminal 5, such as a hard disk or memory of the terminal 5. The memory 51 may also be an external storage device of the terminal 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal 5. Further, the memory 51 may also include both an internal storage unit of the terminal 5 and an external storage device. The memory 51 is used to store the computer program and other programs and data required by the terminal. The memory 51 may also be used to temporarily store data that has been output or is to be output.

[0115] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0116] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0118] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned D2D communication resource allocation method for power emergency communication can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0122] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A D2D communication resource allocation method for power emergency communication. It is characterized in that include: Initialize the parameters of the drone-to-ground D2D emergency communication system; The resource allocation problem in the UAV-to-ground D2D emergency communication system is constructed as a Markov decision process to obtain a reward function; A deep reinforcement learning model is constructed, the environmental state of the D2D communication pair is used as input, the resource allocation strategy is used as output, and the deep reinforcement learning model is trained and updated according to the reward function; the resource allocation strategy includes a channel allocation bandwidth corresponding to the communication pair; the D2D communication pair includes two power devices; The trained deep reinforcement learning model is used to realize D2D communication resource allocation.

2. The D2D communication resource allocation method for power emergency communication according to claim 1, It is characterized in that The parameters of initializing the UAV ground D2D emergency communication system include: Initialize the number of D2D links and D2I links, the acceleration, final running speed, actor network parameters and critic network parameters of the ground mobile power equipment; Constructing a communication neighbor table corresponding to the electric power equipment, wherein the communication neighbor table includes a plurality of communication pairs, and each communication pair is a communication pair consisting of two nearest neighboring electric power equipment; Take the power equipment as an intelligent agent, initialize the environment state S, weight coefficient λ and learning rate γ of the intelligent agent; Among them, the environmental state S = {p, w, g, gm, e}; p represents the transmission power of the power equipment, w represents the bandwidth of the channel, g represents the channel interference, gm represents the gain, and e represents the battery capacity of the drone corresponding to the power equipment.

3. The D2D communication resource allocation method for power emergency communication according to claim 1, It is characterized in that The resource allocation problem in the UAV ground D2D emergency communication system is constructed as a Markov decision process to obtain a reward function, including: Using the weighted index of D2D communication success rate and D2D channel capacity as the reward function of the Markov decision process; The reward function is: Where R represents the reward function, λ represents the weighted coefficient of air-to-ground channel capacity and D2D connection success rate, and C m [m] represents the capacity of the air-to-ground channel with sequence number m, m represents the sequence number of the power equipment communicating with the air base station, kd represents the sequence number of the power equipment for D2D communication, M represents the number of D2I links, K represents the number of D2D links, L kd [kd] represents the connection success rate of the D2D communication pair with sequence number kd.

4. The D2D communication resource allocation method for power emergency communication according to claim 1, It is characterized in that The construction of the deep reinforcement learning model includes: A dual-action decision network and a dual-delay deep deterministic strategy method are adopted, and a KAN network is used as the basic neural network to construct an actor-critic network; the actor-critic network includes two Actor networks, two Critic networks, and corresponding two target Actor networks and four target Critic networks.

5. The D2D communication resource allocation method for power emergency communication according to claim 4, It is characterized in that The method of taking the environmental states of the plurality of D2D communication pairs as input, taking the resource allocation strategy as output, and training and updating the deep reinforcement learning model according to the reward function comprises: Input the current environment state into two Actor networks respectively to obtain two current resource allocation strategies, and determine the next environment state according to the current environment state and the corresponding current resource allocation strategy; Input the next environment state into the two target Actor networks respectively to obtain two next resource allocation strategies; Input the two current state-action arrays into their respective corresponding Critic networks to obtain the corresponding state-action value functions; and select the smaller value of the two state-action value functions as the current Q value; a current state-action array includes the current environment state and a current resource allocation strategy; Input two next state-action arrays into four target critic networks respectively to obtain the corresponding target state action value functions; and select the minimum value of all target state action value functions as the target Q value; a next state-action array includes the next environment state and a next resource allocation strategy; The parameters of the Actor network and the Critic network are updated based on the current Q value, the target Q value and the reward function.

6. The D2D communication resource allocation method for power emergency communication according to claim 5, It is characterized in that The updating of parameters of the Actor network and the Critic network based on the current Q value, the target Q value and the reward function includes: Update the parameters of the Actor network and the Critic network based on the loss function; The loss function is: Where k represents the sequence number of the D2D communication link, t represents the current number of iterations, represents the original neural network parameters, represents the target neural network parameters, Represents the current state of the environment of the input Actor network, Represents the next environment state of the input target Actor network, Represents the current resource allocation strategy for the Actor network output, represents the next resource allocation strategy output by the target Actor network, Q'k represents the target Q value, Qk represents the current Q value, represents the reward function at k steps, and γ represents the discount rate.

7. The D2D communication resource allocation method for power emergency communication according to claim 5, It is characterized in that The updating of the parameters of the Actor network and the Critic network based on the loss function includes: The parameters of the Critic network are updated based on the loss function, and the parameters of the Actor network are updated after a delay of k steps after the Critic network is updated.

8. The D2D communication resource allocation method for power emergency communication according to claim 5, It is characterized in that After updating the parameters of the Actor network and the Critic network based on the current Q value, the target Q value and the reward function, the method further includes: The parameters of the target Actor network and the target Critic network are updated based on the current Q value, the target Q value and the reward function.

9. A D2D communication resource allocation device for power emergency communication, It is characterized in that include: Initialization module, used to initialize the parameters of the drone ground D2D emergency communication system; A reward function acquisition module is used to construct the resource allocation problem in the UAV ground D2D emergency communication system as a Markov decision process to obtain a reward function; A model training module, for constructing a deep reinforcement learning model, taking the environmental state of the D2D communication pair as input, taking the resource allocation strategy as output, and training and updating the deep reinforcement learning model according to the reward function; the resource allocation strategy includes a channel allocation bandwidth corresponding to the communication pair; the D2D communication pair includes two power devices; The communication resource allocation module is used to implement D2D communication resource allocation using a trained deep reinforcement learning model.

10. The D2D communication resource allocation device for power emergency communication according to claim 9, It is characterized in that The initialization module includes: Initialize the number of D2D links and D2I links, and initialize the acceleration V of the ground mobile power equipment a , Final running speed V f 、Actor network parameters θ a and the critic network parameter θ c ; Constructing a communication neighbor table corresponding to the electric power equipment, wherein the communication neighbor table includes a plurality of communication pairs, and each communication pair is a communication pair consisting of two nearest neighboring electric power equipment; Take the power equipment as an intelligent agent, initialize the environment state S, weight coefficient λ and learning rate γ of the intelligent agent; Among them, the environmental state S = {p, w, g, gm, e}; p represents the transmission power of the power equipment, w represents the bandwidth of the channel, g represents the channel interference, gm represents the gain, and e represents the battery capacity of the drone corresponding to the power equipment.

Citation Information

Cited By

  • Cutting process parameter optimization method and system based on deep reinforcement learning

    CN120215452A