Transmission method and system for wireless power and data communication based on power allocation

Abstract

The application provides a transmission method and system of wireless power communication based on power allocation, relates to the transmission technical field of wireless power communication, and comprises the following steps: constructing a DRL agent architecture; inputting a three-dimensional state vector at a current time into the initial DRL agent architecture, generating a corresponding power allocation factor by a policy network, updating a three-dimensional state vector at a next time, and learning the policy network and a value function network by using generated experience data; inputting a real-time three-dimensional state vector into the learned policy network, and receiving an end to split a signal transmitted by a base station according to an optimal power allocation factor to obtain an original radio frequency signal. The policy network is responsible for directly outputting a power allocation factor according to state information such as a current channel gain, a battery residual energy and a noise power, the value function network is responsible for evaluating a long-term value of a current policy, and the two are trained in cooperation so that the agent can learn an optimal policy for long-term cumulative rewards.

Description

Technical Field

[0001] This invention relates to the field of wireless power-carrying communication transmission technology, specifically to a power-allocation-based wireless power-carrying communication transmission method and system. Background Technology

[0002] Wireless power-carrying communication technology enables simultaneous information decoding and energy harvesting on the same radio frequency signal, providing a sustainable power supply solution for energy-constrained wireless communication devices such as IoT sensors and implantable medical devices, and has become a research hotspot in the field of wireless communication. In wireless power-carrying communication systems, the power allocation factor determines the ratio of the received signal split between the energy harvesting branch and the information decoding branch, and is a key parameter affecting system performance. Traditional power allocation methods mainly rely on instantaneous channel state information and employ mathematical methods such as binary search, convex optimization, or dynamic programming for solution. These methods typically require accurate channel models and high online computational complexity. However, real-world wireless communication environments are time-varying and uncertain; channel gain, noise power, and remaining battery power all change dynamically over time, making it difficult for traditional optimization methods to adapt in real time and make optimal decisions in dynamic environments. The patent document with publication number CN110769444A proposes a power-allocation-based wireless energy-carrying communication transmission method. Its core involves first using energy-carrying communication relay nodes to cluster sensing nodes in the Industrial Internet of Things (IIoT), then employing a genetic algorithm to offline optimize the relay nodes' transmission time slots, transmit power, and time and energy allocation ratios. Finally, data is transmitted according to the optimized communication method. However, this scheme has significant shortcomings: First, the genetic algorithm is essentially a static offline optimization method; the optimal communication method it solves is fixed after deployment and cannot cope with real-world wireless scenarios. In communication environments, channel gain, noise power, and battery energy dynamically change over time, lacking online adaptive adjustment capabilities. Secondly, genetic algorithms inherently suffer from premature convergence and slow evolution in later stages, making it difficult to guarantee finding the global optimal solution when dealing with highly nonlinear power allocation optimization problems. More importantly, the objective function of this scheme only optimizes immediate performance indicators such as transmission power consumption and rate at the current moment, lacking consideration for the long-term cumulative performance of the system, such as the comprehensive trade-off between long-term energy efficiency and information rate, and cannot achieve optimal decision-making for long-term returns. Therefore, there is an urgent need for a wireless power-carrying communication transmission method and system based on power allocation.

[0003] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a transmission method and system for wireless power-carrying communication based on power allocation, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: The transmission method for wireless power-carrying communication based on power allocation includes the following steps: S1: Collect communication data consisting of channel gain, remaining battery power, and noise power at the current moment, and perform normalization processing. Construct a three-dimensional state vector from the normalized channel gain, remaining battery power, and noise power in a preset order. At the same time, construct an initial DRL agent architecture. The DRL agent architecture includes a policy network, a value function network, a target policy network with the same structure as the policy network, and a target value function network with the same structure as the value function network. Initially, the parameters of the policy network are copied to the target policy network, and the parameters of the value function network are copied to the target value function network. S2: Input the current three-dimensional state vector into the initial DRL agent architecture, generate the corresponding power allocation factor by the policy network, calculate the information rate and harvested energy based on the power allocation factor and the communication data of the channel at the current time, and further calculate the instant reward based on the information rate and harvested energy. Mark the harvested energy and instant reward as environmental feedback data, and update the three-dimensional state vector of the next time step based on the environmental feedback data and the normalized communication data. S3: Combine the current 3D state vector, power allocation factor, immediate reward, and the 3D state vector of the next moment into a piece of empirical data and store it in the empirical replay pool. When the number of samples in the empirical replay pool reaches a preset threshold, a batch of samples is uniformly drawn from it and combined with the target policy network and the target value function network to learn the policy network and the value function network respectively. S4: Input the real-time three-dimensional state vector into the learned policy network to obtain the corresponding optimal power allocation factor. The receiver segments the signal transmitted by the base station according to the optimal power allocation factor to obtain the original radio frequency signal for energy harvesting and information decoding.

[0006] Furthermore, the communication data is normalized and constructed into a three-dimensional state vector, specifically as follows: Acquire communication data from a preset number of sampling times collected offline, and calculate the mean and standard deviation of channel gain, remaining battery energy, and noise power respectively, as normalization benchmark parameters; Communication data at the current moment is collected at fixed time intervals, and the various types of data collected at the current moment are normalized using the normalization reference parameters. The normalized channel gain, remaining battery energy, and noise power are constructed into a three-dimensional state vector in a fixed order.

[0007] Furthermore, The policy network includes a first hidden layer, a second hidden layer, and an output layer. The first hidden layer contains 64 neurons and uses the ReLU activation function to achieve nonlinear mapping. The second hidden layer contains 32 neurons and also uses the ReLU activation function to further abstract state features. The output layer consists of 1 neuron and uses the Sigmoid activation function. The value function network structure is as follows: the first hidden layer contains 64 neurons for feature extraction, and the first hidden layer uses the ReLU activation function to achieve non-linear transformation; the second hidden layer contains 128 neurons, and also uses the ReLU activation function for high-order feature fusion; the final structure is a linear output layer containing a single neuron.

[0008] Furthermore, the environmental feedback data is calculated, specifically as follows: The computational environment feedback data is as follows: The current three-dimensional state vector is input into the policy network, which generates a corresponding power allocation factor. Based on this power allocation factor and the current channel communication data, the current information rate and the current harvested energy are calculated. The information rate is obtained by calculating the logarithm and adding the signal-to-noise ratio (SNR) of the information decoding branch. This SNR is determined by dividing the product of the base station transmit power, channel gain, and the difference between the base station transmit power and the power allocation factor by the noise power. The harvested energy is determined by multiplying the energy harvesting efficiency, the power allocation factor, the base station transmit power, and the channel gain. Then, the information rate is added to half of the harvested energy to obtain the current instant reward. Finally, the harvested energy and instant reward are labeled as environmental feedback data. Furthermore, the three-dimensional state vector for the next time step is updated based on the harvested energy and the normalized communication data. The three-dimensional state vector for the next time step is updated based on the collected energy and normalized communication data. Specifically, the three-dimensional state vector for the next time step consists of three components: the first component is the channel gain for the next time step, which is determined by multiplying the current channel gain by the channel correlation coefficient, plus the random disturbance of the channel gain multiplied by one minus the square root of the channel correlation coefficient; the second component is the remaining battery energy for the next time step, which is determined by adding the remaining battery energy for the current time step to the collected energy for the current time step, and then subtracting the basic power consumption of the equipment; the third component is the noise power for the next time step, which is determined by multiplying the current noise power by the noise correlation coefficient, plus the random disturbance of the noise power multiplied by one minus the square root of the noise correlation coefficient.

[0009] Furthermore, the policy network and value function network are learned separately, with the following specific steps: An experience replay pool is initialized. At each time step, the policy network in the initial DRL agent architecture outputs the corresponding power allocation factor based on the current three-dimensional state vector. After executing the power allocation factor, an immediate reward is obtained, and the three-dimensional state vector of the next time step is updated. Then, this experience data containing the current three-dimensional state vector, power allocation factor, immediate reward, and the next three-dimensional state vector is stored in the experience replay pool. Once the number of samples in the experience replay pool reaches a preset threshold, a preset number of samples are randomly and uniformly drawn from it. First, for each of these samples, the three-dimensional state vector of the next time step recorded by the sample is input into the target policy network and the target value function network to calculate the target value evaluation value. Then, the target value evaluation value is compared with the actual value evaluation value output by the current value function network to calculate the loss function. The gradient of the loss function with respect to all learnable weights and biases inside the value function network is calculated using the chain rule. The parameters of the value function network are then updated using the gradient. Secondly, for the same batch of samples, the gradient of the policy network output relative to all its internal learnable parameters is calculated simultaneously. The gradient of the value function network output relative to the power allocation factor is multiplied with the gradient of the policy network output relative to its internal learnable parameters to obtain the update direction of the policy network parameters for each sample. Finally, the update directions calculated for all preset number of samples are averaged, and this average value is used to finally update the parameters of the policy network. After each update of the current value function network and the current policy network, the parameters of the target value function network and the target policy network are updated according to the preset soft update coefficient. That is, the parameters of the target network are weighted and averaged with the parameters of the current network, so that the target network slowly tracks the changes in the parameters of the current network. Training is considered complete when the rate of change of the average instantaneous reward over a consecutive preset number of time steps is less than the preset reward range.

[0010] Furthermore, the receiving end segments the signal transmitted by the base station according to the optimal power allocation factor, including: Base station with transmission power Transmit signal symbol After transmission through the channel, the composite radio frequency signal received by the receiving end is: in, This refers to the composite radio frequency signal transmitted by the base station; Indicates the base station's transmission power; Indicates signal symbols; This represents the Gaussian white noise at the receiving end; This represents the channel coefficient at the current moment; The real-time 3D state vector is input into the learned policy network to obtain the corresponding optimal power allocation factor. The receiver then segments the signal transmitted by the base station according to the optimal power allocation factor to obtain the original radio frequency signal. Indicates the energy harvesting flow; This represents the decoded information stream; This represents the optimal power allocation factor.

[0011] The present invention also provides a power-allocation-based wireless power-carrying communication transmission system, wherein the power-allocation-based wireless power-carrying communication transmission system is used to execute the above-described power-allocation-based wireless power-carrying communication transmission method, comprising: The network construction module is used to collect communication data at the current moment, normalize the communication data, construct a three-dimensional state vector, and build an initial DRL agent architecture. The DRL agent architecture includes a policy network, a value function network, a target policy network with the same structure as the policy network, and a target value function network with the same structure as the value function network. Initially, the parameters of the policy network are copied to the target policy network, and the parameters of the value function network are copied to the target value function network. The data update module is used to input the current three-dimensional state vector into the initial DRL agent architecture. The policy network generates the corresponding power allocation factor, calculates the information rate and harvested energy based on the power allocation factor and the current channel communication data, and further calculates the instant reward based on the information rate and harvested energy. The harvested energy and instant reward are marked as environmental feedback data, and the three-dimensional state vector of the next time step is updated based on the environmental feedback data and the normalized communication data. The network learning module is used to combine the current three-dimensional state vector, power allocation factor, immediate reward and the three-dimensional state vector of the next time step into an empirical data, and store it in the empirical replay pool. When the number of samples in the empirical replay pool accumulates to a preset threshold, a batch of samples is uniformly drawn from it, and the policy network and the value function network are learned by combining the target policy network and the target value function network respectively. A wireless transmission module is used to input real-time three-dimensional state vectors into a learned policy network to obtain the corresponding optimal power allocation factor. The receiving end segments the signal transmitted by the base station according to the optimal power allocation factor to obtain the raw radio frequency signal used for energy harvesting and information decoding. Compared with the prior art, the beneficial effects of this invention are: In this scheme, the policy network is responsible for directly outputting the power allocation factor based on current channel gain, remaining battery energy, and noise power, while the value function network is responsible for evaluating the long-term value of the current policy. Their collaborative training enables the agent to learn the optimal policy for long-term cumulative rewards, overcoming the limitation of traditional methods that only optimize immediate performance. The introduction of a target policy network and a target value function network, through a soft update mechanism, provides a stable reference benchmark for calculating the target Q-value, effectively solving the technical problem of constantly changing target values ​​and difficulty in convergence during bootstrapping training in deep reinforcement learning. An experience replay mechanism and a batch gradient update method are employed to improve the efficiency of the algorithm. The reuse of historical experience data and random sampling effectively break the data correlation between consecutive time steps, improve training stability and sample utilization efficiency, and enable the agent to learn the optimal policy through offline training before actual deployment. During the inference phase, only one forward propagation of the policy network is required to output the optimal power allocation factor, resulting in low computational latency and meeting the requirements of real-time communication. The introduction of channel correlation coefficient and noise correlation coefficient in state update and the use of a first-order autoregressive model to describe the time evolution characteristics of channel gain and noise power make the training environment closer to the dynamic characteristics of actual wireless channels, thereby making the learned policy more adaptable to the environment. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Figure 2 This is a graph showing the relationship between information rate and power allocation factor in this invention; Figure 3 This is a schematic diagram of the overall system of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0014] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0015] Example: Please see Figures 1-2 The present invention provides a technical solution: The transmission method for wireless power-carrying communication based on power allocation includes the following steps: S1: Collect communication data consisting of channel gain, remaining battery power, and noise power at the current moment, and perform normalization processing. Construct a three-dimensional state vector from the normalized channel gain, remaining battery power, and noise power in a preset order. At the same time, construct an initial DRL agent architecture. The DRL agent architecture includes a policy network, a value function network, a target policy network with the same structure as the policy network, and a target value function network with the same structure as the value function network. Initially, the parameters of the policy network are copied to the target policy network, and the parameters of the value function network are copied to the target value function network. In this embodiment, the communication data is normalized and constructed into a three-dimensional state vector, specifically as follows: Acquire communication data from a preset number of sampling times collected offline, and calculate the mean and standard deviation of channel gain, remaining battery energy, and noise power respectively, as normalization benchmark parameters; Communication data at the current moment is collected at fixed time intervals, and the various types of data collected at the current moment are normalized using the normalization reference parameters. The normalized channel gain, remaining battery energy, and noise power are constructed into a three-dimensional state vector in a fixed order.

[0016] In this embodiment, the specific method for normalizing the communication data collected at the current moment is as follows: For the three dimensions of data collected at the current moment—channel gain, remaining battery energy, and noise power—a linear transformation is performed using their respective normalization reference parameters. That is, the mean and standard deviation of the corresponding dimensions, which have been calculated in the offline acquisition phase, are used to center and scale the raw data collected at the current moment, so that the normalized data has a statistically zero mean and unit variance. Specifically, for each dimension of data, the mean of the corresponding dimension is first subtracted to center the data to near zero, eliminating the bias caused by the absolute numerical differences between different physical quantities. Then, the data is divided by the standard deviation of the corresponding dimension to scale the dispersion of the data to a uniform scale, so that the data of each dimension are at the same order of magnitude in terms of numerical range. The normalization formula is: the normalized value is equal to the raw value collected at the current moment minus the mean of the corresponding dimension, and then divided by the standard deviation of the corresponding dimension. If the standard deviation is zero, the original value remains unchanged. After the above normalization process, the data of the three dimensions of channel gain, remaining battery energy and noise power are on the same order of magnitude in terms of numerical range. This avoids the bias of input data of different dimensions on the training of policy network and value function network, enabling the deep reinforcement learning agent to perceive the state information of the three dimensions equally, accelerating training convergence and improving the stability of the learned power allocation strategy. The three-dimensional state vector is composed of three physical quantities in a fixed order: channel gain, remaining battery energy, and noise power. Channel gain reflects the attenuation of signal transmission, determining the information rate and the upper limit of energy collection. Remaining battery energy reflects the device's endurance, constraining the urgency of energy collection and the device's sustainable operation. Noise power reflects the environmental interference level at the receiver at the current moment, directly affecting the signal-to-noise ratio of information decoding. Since the dimensions and numerical ranges of these three physical quantities differ significantly, directly combining them into a vector and inputting it into the policy network would cause the larger quantities to dominate gradient updates, making it difficult for the smaller quantities to effectively influence network training. This would affect the convergence speed and final performance of the policy network. Therefore, it is necessary to obtain the mean and standard deviation of each of the three physical quantities over a predetermined number of offline sampling times. Based on this, the data collected at the current moment is normalized to ensure that the numerical ranges of each type of data are on the same order of magnitude. This avoids bias in network training caused by data with different dimensions, allowing the policy network to perceive the state information in all three dimensions equally, thus improving the stability and convergence speed of training. The preset number is determined based on the sampling interval and the coherence time of the wireless channel. The sampling interval is a fixed communication data acquisition cycle, and the coherence time is determined by the Doppler frequency shift. Once the sampling interval is determined, the preset number must ensure that the corresponding total time window covers at least one coherence time length, so that the samples can fully reflect the statistical distribution characteristics of the channel. At the same time, the total time window should not exceed five to ten times the coherence time to avoid introducing outdated channel information due to an excessively long window. Based on this, the preset number is set to 50 to 200 sampling times, preferably 100 sampling times. When the channel changes rapidly, i.e., the coherence time is short, the preset number tends to be 50 to 80 sampling times to prioritize ensuring the response speed of the normalized reference parameter to environmental changes. When the channel changes slowly, i.e., the coherence time is long, the preset number tends to be 150 to 200 sampling times to prioritize ensuring the stability of the statistical estimation.

[0017] The policy network includes a first hidden layer, a second hidden layer, and an output layer. The first hidden layer contains 64 neurons and uses the ReLU activation function to achieve nonlinear mapping. The second hidden layer contains 32 neurons and also uses the ReLU activation function to further abstract state features. The output layer consists of 1 neuron and uses the Sigmoid activation function. The value function network structure is as follows: the first hidden layer contains 64 neurons for feature extraction, and the first hidden layer uses the ReLU activation function to achieve non-linear transformation; the second hidden layer contains 128 neurons, and also uses the ReLU activation function for high-order feature fusion; the final structure is a linear output layer containing a single neuron.

[0018] In the above process, the policy network adopts a dual hidden layer structure with 64 neurons and 32 neurons to extract and abstract features from the three-dimensional state vector step by step from shallow to deep: the first hidden layer has 64 neurons to capture the original interaction relationship between channel gain, remaining battery energy, and noise power, and extract preliminary combined features; the second hidden layer has 32 neurons to further reduce the dimensionality and abstract the features extracted by the first hidden layer, and condense the high-order feature representation most relevant to the power allocation decision, effectively reducing the computational complexity of the subsequent fully connected layers; the output layer of the policy network uses 1 neuron in conjunction with the Sigmoid activation function. Since the physical value range of the power allocation factor is between 0 and 1, the Sigmoid activation function can map the output value of this neuron to this range, directly corresponding to the power allocation factor in the actual physical sense; The value function network employs a dual hidden layer structure with 64 and 128 neurons, more than the policy network. This is because the value function network needs to accurately evaluate the joint input of the current 3D state vector and the power allocation factor. A wider hidden layer retains more high-order feature information, thus improving the accuracy of Q-value estimation. The output layer uses a single neuron with a linear activation function. This is because the value evaluation value theoretically has no fixed upper or lower bound, and the linear activation function can output any real value, avoiding unnecessary constraints on the value evaluation range. Both networks use the ReLU activation function for non-linear mapping, which effectively alleviates the gradient vanishing problem and accelerates training convergence, ensuring the training efficiency and stability of deep reinforcement learning in power allocation scenarios.

[0019] S2: Input the current three-dimensional state vector into the initial DRL agent architecture, generate the corresponding power allocation factor by the policy network, calculate the information rate and harvested energy based on the power allocation factor and the communication data of the channel at the current time, and further calculate the instant reward based on the information rate and harvested energy. Mark the harvested energy and instant reward as environmental feedback data, and update the three-dimensional state vector of the next time step based on the environmental feedback data and the normalized communication data. In this embodiment, the calculation of environmental feedback data specifically includes: The current three-dimensional state vector is input into the policy network, which generates a corresponding power allocation factor. Based on this power allocation factor and the current channel communication data, the current information rate and the current harvested energy are calculated. The information rate is obtained by calculating the logarithmic function plus the signal-to-noise ratio of the information decoding branch. This signal-to-noise ratio is determined by dividing the product of the base station transmit power, the channel gain, and the difference between the base station transmit power and the power allocation factor by the noise power. The harvested energy is determined by multiplying the energy harvesting efficiency, the power allocation factor, the base station transmit power, and the channel gain. Then, the information rate is added to half of the harvested energy to obtain the current instant reward. Finally, the harvested energy and the instant reward are marked as environmental feedback data.

[0020] The formula upon which the above process is based is: in, Indicates the information rate at the current moment; This represents the energy collected at the current moment; Indicates the immediate reward at the current moment; This represents the channel coefficient at the current moment. This represents the channel gain at the current moment; This represents the noise power at the current moment; This represents the power allocation factor at the current moment; Indicates the index at the current time; This indicates energy harvesting efficiency.

[0021] In the above process, information rate This represents the transmission rate that the information decoding branch can achieve at the current moment; a larger value indicates higher information transmission efficiency and better communication quality; energy harvesting. This represents the electrical energy that the energy harvesting branch can collect and store at the current moment. A higher value indicates that the device can replenish more energy and has a longer battery life; instant reward. It characterizes the overall performance evaluation of the system at the current moment in terms of information rate and energy collection. The larger the value, the better the overall performance of the current power allocation decision in terms of both rate and energy objectives. It is used to guide the parameter update of the policy network. In the calculation formula, the molecule This represents the signal power allocated to the information decoding branch. A higher power results in a higher signal-to-noise ratio and a higher information rate; therefore, the power allocation factor... Base station transmission power and channel gain All with There is a positive correlation, while noise power As a denominator within a logarithmic fraction, its larger value indicates stronger interference and a lower signal-to-noise ratio. and There is a negative correlation; it should be noted that, It is not simply a motivating factor; its increase will... Decrease, therefore An increase in power will reduce the proportion of power allocated to information decoding, thus affecting... It produces an inhibitory effect, showing a negative correlation; In the calculation formula, the power allocation factor With energy harvesting efficiency Base station transmission power and channel gain Both show a positive correlation because the proportion of power allocated to the energy harvesting branch is... The larger this ratio, the more radio frequency power can be collected. At the same time, the larger the channel gain, the stronger the signal reaching the receiver and the greater the energy collected. Instant rewards Information rate With energy collection By using a weighted summation method, it is unified into a single scalar, where The weighting coefficient is 1. The weighting coefficient is 0.5. This weighting relationship reflects the optimization orientation of this embodiment, which prioritizes information transmission and supplements it with energy harvesting. and By keeping the values ​​of both factors close to each other, the network avoids one factor dominating the reward, allowing the policy network to make a reasonable trade-off between the two and learn the optimal power allocation strategy for long-term cumulative performance.

[0022] In the above embodiments, 20 sets of data on power allocation factors and corresponding information rates are provided to reflect how the information rate changes with the power allocation factor, as shown in Table 1: Table 1: Relationship between information rate and power allocation factor In Table 1 above, the base station transmit power The value is 1 watt, and the channel gain is... The value is 0.1, noise power. in accordance with Calculate, where the noise power spectral density Values Watts per Hertz, channel bandwidth Values Hertz, thus calculated watt; As can be seen, with the power allocation factor The information rate gradually increases from 0.05 to 1.00. The power allocation factor monotonically decreases from approximately 3.46 bps to approximately 0.86 bps, showing a negative correlation. The physical reason for this is the power allocation factor. This determines the segmentation ratio of the received signal between the energy harvesting branch and the information decoding branch. A larger value indicates a larger proportion of signal power allocated to the energy harvesting branch, and a larger proportion of signal power allocated to the information decoding branch. The smaller the value, the less signal power is used for information decoding, resulting in a lower signal-to-noise ratio and a corresponding decrease in information rate; conversely, the larger the value, the less signal power is used for information decoding. The smaller the value, the more signal power the information decoding branch receives, and the information rate is increased accordingly.

[0023] The three-dimensional state vector for the next time step is updated based on the collected energy and the normalized communication data: The three-dimensional state vector for the next time step is updated based on the collected energy and normalized communication data. Specifically, the three-dimensional state vector for the next time step consists of three components: the first component is the channel gain for the next time step, which is determined by multiplying the current channel gain by the channel correlation coefficient, plus the random disturbance of the channel gain multiplied by one minus the square root of the channel correlation coefficient; the second component is the remaining battery energy for the next time step, which is determined by adding the remaining battery energy for the current time step to the collected energy for the current time step, and then subtracting the basic power consumption of the equipment; the third component is the noise power for the next time step, which is determined by multiplying the current noise power by the noise correlation coefficient, plus the random disturbance of the noise power multiplied by one minus the square root of the noise correlation coefficient.

[0024] The formula upon which the above process is based is as follows: in, This indicates the remaining battery energy at the current moment; Indicates the basic energy consumption of the equipment; Represents the channel correlation coefficient; , This represents the random perturbation of the channel gain at the current moment; This represents the standard deviation of a Gaussian distribution. Represents the three-dimensional state vector at the next moment; Indicates the noise correlation coefficient; This indicates random disturbances in noise power.

[0025] In the above process, the three-dimensional state vector It includes three components: channel gain, remaining battery energy, and noise power, which are updated according to their respective physical evolution laws; the channel gain update adopts a first-order autoregressive model. ,in The historical information of the channel gain from the previous moment is retained. By introducing a random perturbation that follows a Gaussian distribution, both of which together describe the correlation and randomness of the wireless channel in the time dimension, the channel evolution in the training environment is made closer to the continuous change characteristics of channel gain in actual wireless communication scenarios, thereby enabling the power allocation strategy learned by the policy network to have good adaptability under actual dynamic channel conditions. For channel gain components Its difference from the channel gain at the previous time step There is a positive correlation because actual wireless channels exhibit temporal correlation between adjacent moments; the better the channel state at the previous moment, the greater the probability of maintaining a good state at the next moment. This is related to random perturbations in channel gain. The two equations show a positive correlation, indicating that the disturbance reflects the impact of random factors such as multipath fading on channel variations. A larger disturbance indicates a stronger superimposed random variation component at the current moment. The rationale for this updated equation lies in the fact that channel time-selective fading caused by the Doppler effect in wireless communication can be described by a first-order autoregressive model, with coefficients... To ensure that the variance of the channel gain remains constant during time evolution, thus making the generated channel sequence statistically stationary; Battery remaining energy is replaced using This means the sum of the remaining energy at the current moment, the energy collected at the current moment, and the energy consumption of the equipment's basic energy consumption. It reflects the dynamic balance between energy inflow and outflow at the receiving end energy storage device, where the collected energy... With power allocation factor and channel gain The direct correlation enables the policy network to perceive and consider the impact of the current action on the subsequent energy state when making decisions, thereby making a long-term optimal trade-off between maximizing information rate and ensuring device endurance. For the remaining energy component of the battery Its relationship with the current battery energy There is a positive correlation: the more energy remaining at the current moment, the more energy remaining in the next moment; this is related to the energy collected at the current moment. There is a positive correlation: the more energy collected at the current moment, the more fully the battery is replenished, and the higher the remaining energy at the next moment; it is also positively correlated with the equipment's base energy consumption. There is a negative correlation: the greater the energy consumed by the device during operation, the lower the remaining energy at the next moment. The rationale for this updated formula is that the change in the battery's remaining energy strictly follows the law of conservation of energy, and its value is equal to the original remaining energy plus the energy collected through wireless energy-carrying communication, minus the energy consumed by the device during operation. The noise power update adopts Similar to channel gain, it uses the same autoregressive modeling method. Its physical meaning lies in the fact that noise power in actual wireless communication environments is not constant, but rather exhibits time-dependent characteristics due to factors such as environmental electromagnetic interference and temperature changes. This is addressed by introducing a noise correlation coefficient. and random noise disturbance This enables the agent to adapt to the slowly varying characteristics of noise power, avoiding performance degradation of the strategy in real-world environments due to the assumption of constant noise. For noise power components Its noise power compared to the previous moment There is a positive correlation because environmental noise levels are continuous between adjacent moments; the higher the noise power at one moment, the greater the likelihood that the noise power will remain at a high level at the next moment. This is in contrast to random disturbances in noise power. The two sides show a positive correlation, and the disturbance reflects the influence of random factors such as electromagnetic interference on the noise level. The larger the disturbance, the stronger the random change component superimposed at the current moment. The rationale for this updated formula is that the noise power in the actual wireless communication environment is affected by factors such as temperature changes and adjacent channel interference, and exhibits time correlation. Using the same autoregressive model as the channel gain to describe it can make the state transition process closer to the real physical environment change law. The value of is directly related to the product of the maximum Doppler frequency shift and the system sampling interval. In typical wireless channel modeling, the time autocorrelation function of the channel gain follows the zeroth-order Bessel function. The evolutionary pattern, among which For the maximum Doppler frequency shift, This is the time interval between adjacent sampling times; therefore, Based on this autocorrelation function, The value at that location is set, that is... In practical systems, the maximum Doppler shift is determined by the carrier frequency and the relative velocity between the transmitter and receiver. ,in The relative velocity, For carrier frequency, It is the speed of light; when the sampling interval When the relative speed is constant, the greater the relative velocity, the faster the channel changes over time, and the lower the correlation between adjacent moments. The smaller the value, the slower the channel changes; conversely, the smaller the relative speed, the slower the channel changes. The closer the value is to 1, the better. In this embodiment, the system sampling interval is a fixed value. The configuration can be adjusted based on the actual speed of the terminal in the deployment scenario. Typically, the terminal's speed ranges from 0 to 120 kilometers per hour. The value ranges from 0.9 to 0.999; Noise correlation coefficient The time correlation of noise power between adjacent time points is quantified based on the slow variation characteristics of noise power in actual wireless communication environments. Unlike channel gain, the variation of noise power mainly originates from factors such as changes in receiver thermal noise with ambient temperature, fluctuations in adjacent channel interference, and slow changes in the level of ambient electromagnetic interference. The rate of change of these factors is typically much slower than the rate of change of channel fading; therefore, noise power exhibits a strong time correlation between adjacent sampling times. In this embodiment, The settings can be adjusted based on the noise statistics of the receiver's environment in the actual deployment scenario. For scenarios with relatively constant temperatures, such as indoors or underground, the noise power fluctuation is relatively small. The value can be set close to 1, typically between 0.95 and 0.99; for outdoor or industrial environments with significant temperature variations and strong electromagnetic interference, the fluctuation of noise power is relatively noticeable. The value can be appropriately reduced, usually between 0.85 and 0.95; Channel gain random perturbation By using a mean of 0 and a variance of Gaussian distribution In this embodiment, random sampling is performed to determine the method. The value range is usually set to to In between—when the deployment scenario is indoors or a low-speed mobile environment, the random fluctuation of the channel gain is relatively small. Smaller values ​​can be taken, such as to When deployed in outdoor or high-speed mobile environments, the random fluctuations in channel gain are significant. Larger values ​​can be taken, such as to ; Random disturbance of noise power By using a mean of 0 and a variance of Gaussian distribution In this embodiment, random sampling is performed to determine the method. The value range is usually set to to Between these parameters—when deployed in indoor or constant-temperature environments, the noise power fluctuation is relatively small. Smaller values ​​can be taken, such as to When deployed in outdoor or industrial environments with significant temperature variations and strong electromagnetic interference, noise power fluctuations are relatively noticeable. Larger values ​​can be taken, such as to .

[0026] S3: Combine the current 3D state vector, power allocation factor, immediate reward, and the 3D state vector of the next moment into a piece of empirical data and store it in the empirical replay pool. When the number of samples in the empirical replay pool reaches a preset threshold, a batch of samples is uniformly drawn from it and combined with the target policy network and the target value function network to learn the policy network and the value function network respectively. In this embodiment, the policy network and the value function network are learned separately. The specific steps are as follows: An experience replay pool is initialized. At each time step, the policy network in the initial DRL agent architecture outputs the corresponding power allocation factor based on the current three-dimensional state vector. After executing the power allocation factor, an immediate reward is obtained, and the three-dimensional state vector of the next time step is updated. Then, this experience data containing the current three-dimensional state vector, power allocation factor, immediate reward, and the next three-dimensional state vector is stored in the experience replay pool. Once the number of samples in the experience replay pool reaches a preset threshold, a preset number of samples are randomly and uniformly drawn from it. First, for each of these samples, the three-dimensional state vector of the next time step recorded by the sample is input into the target policy network and the target value function network to calculate the target value evaluation value. Then, the target value evaluation value is compared with the actual value evaluation value output by the current value function network to calculate the loss function. The gradient of the loss function with respect to all learnable weights and biases inside the value function network is calculated using the chain rule. The parameters of the value function network are then updated using the gradient. Secondly, for the same batch of samples, the gradient of the policy network output relative to all its internal learnable parameters is calculated simultaneously. The gradient of the value function network output relative to the power allocation factor is multiplied with the gradient of the policy network output relative to its internal learnable parameters to obtain the update direction of the policy network parameters for each sample. Finally, the update directions calculated for all preset number of samples are averaged, and this average value is used to finally update the parameters of the policy network. After each update of the current value function network and the current policy network, the parameters of the target value function network and the target policy network are updated according to the preset soft update coefficient. That is, the parameters of the target network are weighted and averaged with the parameters of the current network, so that the target network slowly tracks the changes in the parameters of the current network. Training is considered complete when the rate of change of the average instantaneous reward over a consecutive preset number of time steps is less than the preset reward range.

[0027] In the above process, the introduction of the experience replay pool is used to solve the problem of training instability caused by the temporal correlation between continuous samples in deep reinforcement learning. That is, the experience data generated by the policy network in actual operation and its interaction with the environment is highly correlated in time. If it is directly used for network updates in chronological order, the gradient estimation will have a large variance and the network will easily overfit to recent experience. By storing the experience data generated at each time step, including the three-dimensional state vector, power allocation factor, immediate reward and the three-dimensional state vector of the next time step, into the experience replay pool, and then randomly and uniformly extracting a preset number of samples for batch updates, the temporal correlation between continuous samples can be broken, so that the training data satisfies the independent and identically distributed assumption, thereby significantly improving the stability and convergence speed of training. The introduction of the target policy network and the target value function network, along with the adoption of a soft update mechanism, is used to address the problem of training divergence caused by the continuous change of the target value during bootstrapping training. If the target value evaluation value is directly calculated using the current value function network, the target value will change with each update of the current network, causing the network to constantly chase a moving target and making convergence difficult. By providing a relatively stable target value evaluation value through the target policy network and the target value function network, and by weighting the parameters of the target network and the current network according to a preset soft update coefficient after each update, the target network slowly tracks the changes of the current network instead of completely replicating them each time. This provides a stable benchmark for the calculation of the target Q value, effectively suppresses oscillations during training, and ensures that the network can stably converge to the optimal policy. The alternating update method of the policy network and the value function network realizes the evaluation-improvement closed-loop learning. That is, the value function network learns accurate value evaluation by minimizing the loss function between the target value evaluation value and the actual value evaluation value, while the policy network uses the value evaluation results output by the value function network to guide the update direction of its own parameters, so that the power allocation factor output by the policy network continuously improves in the direction of obtaining higher value evaluation. When the average instantaneous reward change rate of consecutive preset time steps is less than the preset reward magnitude, it indicates that the performance of the policy network has stabilized and further training will not bring significant performance improvement. At this time, the training is marked as completed to avoid overfitting and waste of computational resources. The preset time step is used to determine whether the training has converged. Its value needs to take into account both the stability of the training process and the reliability of the convergence judgment. If the preset time step is set too small, the average immediate reward may stabilize briefly due to random fluctuations, causing the training to terminate prematurely before it has truly converged. If it is set too large, unnecessary training will continue after the network has converged, increasing computational overhead. In specific implementations, the preset time step can be set according to the batch size of the experience replay pool and the total number of training steps. It is usually set to 1,000 to 5,000 time steps, preferably 2,000 time steps. This value is sufficient to cover dozens of complete batch update cycles, effectively filtering out short-term random fluctuations in immediate rewards, accurately reflecting the long-term trend of policy network performance, and avoiding delays in convergence judgment due to excessively long time windows. It balances training efficiency while ensuring the reliability of the judgment. The soft update coefficient is usually based on empirical values. In this embodiment, the preset soft update coefficient can be set to a value between 0.001 and 0.01. It can be 0.005, which means that the target network can ensure that the response of the target network to the current network changes is smooth enough to avoid drastic oscillations in the target value, and can also enable the target network to gradually track the learning progress of the current network within a reasonable number of training steps, thus achieving a good balance between training stability and convergence speed. In this embodiment, the preset reward level can be determined based on the immediate reward. The typical numerical range is set, due to the information rate. and energy collection Usually in to Within a certain range, instant rewards The typical value range is between 0 and 10, and the preset reward level can be set to... to The values ​​between these two ranges can be taken as follows: This value can effectively filter out the interference of small random fluctuations in immediate rewards during training on the convergence judgment, and can also ensure that the network performance is not misjudged as converged when there is still significant room for improvement, thereby improving training efficiency while ensuring training quality.

[0028] S4: Input the real-time three-dimensional state vector into the learned policy network to obtain the corresponding optimal power allocation factor. The receiver segments the signal transmitted by the base station according to the optimal power allocation factor to obtain the original radio frequency signal for energy harvesting and information decoding.

[0029] In this embodiment, the receiving end segments the signal transmitted by the base station according to the optimal power allocation factor, including: Base station with transmission power Transmit signal symbol After transmission through the channel, the composite radio frequency signal received by the receiving end is: in, This refers to the composite radio frequency signal transmitted by the base station; Indicates the base station's transmission power; Indicates signal symbols; This represents the Gaussian white noise at the receiving end; The real-time 3D state vector is input into the learned policy network to obtain the corresponding optimal power allocation factor. The receiver then segments the signal transmitted by the base station according to the optimal power allocation factor to obtain the original radio frequency signal. Indicates the energy harvesting flow; This represents the decoded information stream; This represents the optimal power allocation factor.

[0030] In the above process, the formula for receiving signals This describes the composite signal that arrives at the receiving end after the radio frequency signal transmitted by the base station has passed through the wireless channel, where the dependent variable is... The amplitude of the composite radio frequency signal actually received by the receiver directly reflects the strength level of the received signal. A larger amplitude indicates that the signal experiences less channel fading during transmission or that the base station has higher transmit power, allowing the receiver to obtain stronger signal energy for subsequent information decoding and energy harvesting; when When the amplitude is small, it means the signal has experienced severe path loss or deep fading, and the signal energy received at the receiver is weak; the rationale for this formula is that the base station uses transmit power... Transmit signal symbol Then, when the signal propagates in free space, it will undergo a process determined by the channel coefficient. The amplitude attenuation and phase rotation are characterized by the superposition of Gaussian white noise introduced by the receiver circuitry and the environment. These three elements together constitute the most basic signal transmission model of the physical layer of a wireless communication system, accurately reflecting the complete physical process of a signal from the transmitter to the receiver. In the formula for receiving signals In the middle, the base station transmit power With received signal There is a positive correlation because the greater the transmit power, the stronger the residual power of the signal after propagation in space, resulting in a higher received signal amplitude under the same channel conditions and noise level; signal symbol With received signal There is a positive correlation. The larger the amplitude, the stronger the signal component that reaches the receiver after power amplification and channel transmission; the channel coefficient With received signal There is a positive correlation. A larger magnitude indicates less signal fading during transmission, allowing the receiver to retain more signal energy from the transmitter; Gaussian white noise With received signal They exhibit a linear superposition relationship. A larger amplitude indicates stronger environmental interference at the receiver, resulting in more noise components mixed in the received signal and thus reducing the signal-to-noise ratio during information decoding. This formula concisely describes the physical relationship between transmit power, channel fading, and noise through linear multiplication and addition. This demonstrates the square root amplification effect of the transmitted power on the signal amplitude. This reflects the multiplicative fading effect of the channel on the signal. This reflects the additive noise interference at the receiving end, and the three factors together determine the actual signal quality that the receiving end can obtain; For energy harvesting flow and information decoding stream , and These respectively represent the signal components input to the energy harvester for RF-DC conversion after power splitting and the signal components input to the information decoder for demodulation and decoding. The larger the amplitude, the more signal energy is allocated to the energy harvesting branch, and the more electrical energy can be converted and stored, but the corresponding signal energy available to the information decoding branch will be reduced. A larger amplitude indicates more signal energy allocated to the information decoding branch, resulting in a higher signal-to-noise ratio for information demodulation and a greater information rate. However, this also reduces the signal energy available to the energy harvesting branch. The rationale for this partitioning method lies in the power allocation factor. It directly determines the energy distribution ratio of the received signal between the two branches, i.e. The larger the energy harvesting flow The greater the amplitude, the more energy is collected; but at the same time The smaller the value, the faster the information decoding stream. The smaller the amplitude, the lower the information rate. This inverse correlation reflects the inherent contradiction between information rate and harvested energy in wireless energy-carrying communication: improving the performance of one comes at the expense of the other. This scheme precisely uses the optimal power allocation factor output by the policy network. This dynamic balance aims to achieve optimal power allocation decisions for the system under different channel conditions and energy states, thus enabling the system to obtain the best overall performance.

[0031] Please see Figure 3 The present invention further provides a power-allocation-based wireless power-carrying communication transmission system, wherein the power-allocation-based wireless power-carrying communication transmission system is used to execute the above-described power-allocation-based wireless power-carrying communication transmission method, comprising: The network construction module is used to collect communication data at the current moment, normalize the communication data, construct a three-dimensional state vector, and build an initial DRL agent architecture. The DRL agent architecture includes a policy network, a value function network, a target policy network with the same structure as the policy network, and a target value function network with the same structure as the value function network. Initially, the parameters of the policy network are copied to the target policy network, and the parameters of the value function network are copied to the target value function network. The data update module is used to input the current three-dimensional state vector into the initial DRL agent architecture, generate the corresponding power allocation factor by the policy network, calculate the environmental feedback data based on the power allocation factor, and update the three-dimensional state vector of the next moment based on the environmental feedback data and the normalized communication data. The network learning module is used to combine the current three-dimensional state vector, power allocation factor, immediate reward and the three-dimensional state vector of the next time step into an empirical data, and store it in the empirical replay pool. When the number of samples in the empirical replay pool accumulates to a preset threshold, a batch of samples is uniformly drawn from it, and the policy network and the value function network are learned by combining the target policy network and the target value function network respectively. The wireless transmission module is used to input the real-time three-dimensional state vector into the learned policy network to obtain the corresponding optimal power allocation factor. The receiving end segments the signal transmitted by the base station according to the optimal power allocation factor to obtain the original radio frequency signal.

[0032] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0033] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0034] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A transmission method for wireless power-carrying communication based on power allocation, characterized in that, include: S1: Collect communication data consisting of channel gain, remaining battery power, and noise power at the current moment, and perform normalization processing. Construct a three-dimensional state vector from the normalized channel gain, remaining battery power, and noise power in a preset order. At the same time, construct an initial DRL agent architecture. The DRL agent architecture includes a policy network, a value function network, a target policy network with the same structure as the policy network, and a target value function network with the same structure as the value function network. Initially, the parameters of the policy network are copied to the target policy network, and the parameters of the value function network are copied to the target value function network. S2: Input the current three-dimensional state vector into the initial DRL agent architecture, generate the corresponding power allocation factor by the policy network, calculate the information rate and harvested energy based on the power allocation factor and the communication data of the channel at the current time, and further calculate the instant reward based on the information rate and harvested energy. Mark the harvested energy and instant reward as environmental feedback data, and update the three-dimensional state vector of the next time step based on the environmental feedback data and the normalized communication data. S3: Combine the current 3D state vector, power allocation factor, immediate reward, and the 3D state vector of the next moment into a piece of empirical data and store it in the empirical replay pool. When the number of samples in the empirical replay pool reaches a preset threshold, a batch of samples is uniformly drawn from it and combined with the target policy network and the target value function network to learn the policy network and the value function network respectively. S4: Input the real-time three-dimensional state vector into the learned policy network to obtain the corresponding optimal power allocation factor. The receiver segments the signal transmitted by the base station according to the optimal power allocation factor to obtain the original radio frequency signal for energy harvesting and information decoding.

2. The transmission method for wireless power-carrying communication based on power allocation according to claim 1, characterized in that, The communication data is normalized and constructed into a three-dimensional state vector, specifically as follows: Acquire communication data from a preset number of sampling times collected offline, and calculate the mean and standard deviation of channel gain, remaining battery energy, and noise power respectively, as normalization benchmark parameters; Communication data at the current moment is collected at fixed time intervals, and the various types of data collected at the current moment are normalized using the normalization reference parameters. The normalized channel gain, remaining battery energy, and noise power are constructed into a three-dimensional state vector in a fixed order.

3. The transmission method for wireless power-carrying communication based on power allocation according to claim 1, characterized in that, The policy network includes a first hidden layer, a second hidden layer, and an output layer. The first hidden layer contains 64 neurons and uses the ReLU activation function to achieve nonlinear mapping. The second hidden layer contains 32 neurons and also uses the ReLU activation function to further abstract state features. The output layer consists of 1 neuron and uses the Sigmoid activation function. The value function network structure is as follows: the first hidden layer contains 64 neurons for feature extraction, and the first hidden layer uses the ReLU activation function to achieve non-linear transformation; the second hidden layer contains 128 neurons, and also uses the ReLU activation function for high-order feature fusion; the final structure is a linear output layer containing a single neuron.

4. The transmission method for wireless power-carrying communication based on power allocation according to claim 1, characterized in that, The computational environment feedback data is as follows: The current three-dimensional state vector is input into the policy network, which generates the corresponding power allocation factor. Based on the power allocation factor and the communication data of the channel at the current time, the information rate and the harvested energy at the current time are calculated respectively. The information rate is obtained by calculating the logarithmic function and adding the signal-to-noise ratio of the information decoding branch. The signal-to-noise ratio is determined by dividing the product of the base station transmit power, the channel gain, and the difference after subtracting the power allocation factor by the noise power. The harvested energy is determined by the product of the energy harvesting efficiency, the power allocation factor, the base station transmit power, and the channel gain. Then, the information rate is added to half of the collected energy to obtain the instant reward for the current moment; Finally, the collected energy and immediate rewards are tagged as environmental feedback data.

5. The transmission method for wireless power-carrying communication based on power allocation according to claim 4, characterized in that, The three-dimensional state vector for the next time step is updated based on the collected energy and the normalized communication data: The three-dimensional state vector for the next time step is updated based on the collected energy and normalized communication data. Specifically, the three-dimensional state vector for the next time step consists of three components: the first component is the channel gain for the next time step, which is determined by multiplying the current channel gain by the channel correlation coefficient, plus the random disturbance of the channel gain multiplied by one minus the square root of the channel correlation coefficient; the second component is the remaining battery energy for the next time step, which is determined by adding the remaining battery energy for the current time step to the collected energy for the current time step, and then subtracting the basic power consumption of the equipment; the third component is the noise power for the next time step, which is determined by multiplying the current noise power by the noise correlation coefficient, plus the random disturbance of the noise power multiplied by one minus the square root of the noise correlation coefficient.

6. The transmission method for wireless power-carrying communication based on power allocation according to claim 5, characterized in that, The policy network and value function network are learned separately, and the specific steps are as follows: An experience replay pool is initialized. At each time step, the policy network in the initial DRL agent architecture outputs the corresponding power allocation factor based on the current three-dimensional state vector. After executing the power allocation factor, an immediate reward is obtained, and the three-dimensional state vector of the next time step is updated. Then, this experience data containing the current three-dimensional state vector, power allocation factor, immediate reward, and the next three-dimensional state vector is stored in the experience replay pool. Once the number of samples in the experience replay pool reaches a preset threshold, a preset number of samples are randomly and uniformly drawn from it. First, for each of these samples, the three-dimensional state vector of the next time step recorded by the sample is input into the target policy network and the target value function network to calculate the target value evaluation value. Then, the target value evaluation value is compared with the actual value evaluation value output by the current value function network to calculate the loss function. The gradient of the loss function with respect to all learnable weights and biases inside the value function network is calculated using the chain rule. The parameters of the value function network are then updated using the gradient. Secondly, for the same batch of samples, the gradient of the policy network output relative to all its internal learnable parameters is calculated simultaneously. The gradient of the value function network output relative to the power allocation factor is multiplied with the gradient of the policy network output relative to its internal learnable parameters to obtain the update direction of the policy network parameters for each sample. Finally, the update directions calculated for all preset number of samples are averaged, and this average value is used to finally update the parameters of the policy network. After each update of the current value function network and the current policy network, the parameters of the target value function network and the target policy network are updated according to the preset soft update coefficient. That is, the parameters of the target network are weighted and averaged with the parameters of the current network, so that the target network slowly tracks the changes in the parameters of the current network. Training is considered complete when the rate of change of the average instantaneous reward over a consecutive preset number of time steps is less than the preset reward range.

7. The transmission method for wireless power-carrying communication based on power allocation according to claim 1, characterized in that, The receiver segments the signal transmitted by the base station according to the optimal power allocation factor, including: Base station with transmission power Transmit signal symbol After transmission through the channel, the composite radio frequency signal received by the receiving end is: in, This refers to the composite radio frequency signal transmitted by the base station; Indicates the base station's transmit power; Indicates signal symbols; This represents the white Gaussian noise at the receiving end; This represents the channel coefficient at the current moment; The real-time 3D state vector is input into the learned policy network to obtain the corresponding optimal power allocation factor. The receiver then segments the signal transmitted by the base station according to the optimal power allocation factor to obtain the original radio frequency signal. Indicates the energy harvesting flow; This represents the decoded information stream; This represents the optimal power allocation factor.

8. A wireless power-carrying communication transmission system based on power allocation, characterized in that: The transmission system is used to perform the transmission method according to any one of claims 1-7, including: The network construction module is used to collect communication data at the current moment, normalize the communication data, construct a three-dimensional state vector, and build an initial DRL agent architecture. The DRL agent architecture includes a policy network, a value function network, a target policy network with the same structure as the policy network, and a target value function network with the same structure as the value function network. Initially, the parameters of the policy network are copied to the target policy network, and the parameters of the value function network are copied to the target value function network. The data update module is used to input the current three-dimensional state vector into the initial DRL agent architecture, generate the corresponding power allocation factor by the policy network, calculate the environmental feedback data based on the power allocation factor, and update the three-dimensional state vector of the next time step based on the environmental feedback data and normalized communication data. The network learning module is used to combine the current three-dimensional state vector, power allocation factor, immediate reward and the three-dimensional state vector of the next time step into an empirical data set, and store it in the empirical replay pool. When the number of samples in the empirical replay pool accumulates to a preset threshold, a batch of samples is uniformly extracted from it, and the policy network and the value function network are learned by combining the target policy network and the target value function network respectively. The wireless transmission module is used to input the real-time three-dimensional state vector into the learned policy network to obtain the corresponding optimal power allocation factor. The receiving end segments the signal transmitted by the base station according to the optimal power allocation factor to obtain the original radio frequency signal.

Images