Parameter optimization method for hybrid energy collection subarray reflector communication system and related equipment

Abstract

The parameter optimization method of the hybrid energy collection subarray reflector communication system and the related equipment provided by the embodiments of the present application can obtain the multi-link channel state information of the communication network and the battery energy state information of the intelligent reflector, and output the optimized radio frequency energy collection time length and a series of reflection adjustment parameters based on the communication parameter optimization model, so that the time frame can be dynamically divided into a downlink energy collection stage and an uplink information transmission stage. Then, the base station sends a radio frequency signal to the intelligent reflector in the downlink stage to supplement energy, and the active / passive mode and the reflection parameters of the subarray are jointly adjusted in the uplink stage to assist the uplink data transmission. Thus, the idle time of the intelligent reflector when the environmental energy supply is insufficient is utilized for radio frequency energy acquisition, so that the long-term average throughput and the communication reliability of the communication system in the energy-limited scenario are improved on the premise of maintaining the low-power operation of the intelligent reflector sub-connection architecture.

Description

Technical Field

[0001] This application relates to the field of wireless communication technology, and in particular to a parameter optimization method and related equipment for a hybrid energy harvesting subarray reflector communication system. Background Technology

[0002] Reconfigurable smart surfaces (RIS), as a technology capable of modulating the electromagnetic wave propagation environment, have attracted widespread attention for improving the performance of wireless communication systems. To alleviate the high power consumption problem caused by active RIS and solve the power supply limitations in remote scenarios, related technologies typically employ a sub-connected hybrid architecture (i.e., multiple reflective units share a power amplifier and support active / passive mode switching) to reduce hardware power consumption. Power is supplied to the RIS via a fixed power source or solar energy collection, thereby enabling the RIS to operate under power.

[0003] However, fixed power supply can restrict the placement of RIS. In addition, reliance on solar energy collection is susceptible to fluctuations in the external environment. When the solar energy supply is insufficient, the effective working time of the RIS in a time slot will be significantly shortened, which will lead to a significant drop in system throughput. As a result, the communication efficiency and reliability of existing communication systems with intelligent reflectors are still relatively low. Summary of the Invention

[0004] This application provides a parameter optimization method and related equipment for a hybrid energy harvesting subarray reflector communication system, which can solve the above-mentioned technical problems.

[0005] To achieve the above objectives, a first aspect of this application proposes a parameter optimization method for a hybrid energy harvesting subarray reflector communication system. The hybrid energy harvesting subarray reflector communication system includes a base station, a smart reflector, and multiple user terminals. The smart reflector is equipped with a hybrid energy harvesting module and multiple reflector subarrays, each of which includes multiple reflector subarrays. The method includes: In the current time frame, obtain the first channel state information between the base station and the smart reflector, the second channel state information between each user terminal and the smart reflector, and the third channel state information between each user terminal and the base station, and obtain the battery energy state information of the smart reflector at the current moment; The first channel state information, the second channel state information, the third channel state information, and the battery energy state information are input into the communication parameter optimization model for data processing to obtain optimized combined parameters. The optimized combined parameters include optimized radio frequency energy harvesting duration, optimized user transmit power, optimized reflector mode switching sequence, optimized reflective phase shift sequence, and optimized reflective amplification coefficient sequence. Based on the optimized radio frequency energy harvesting duration, the current time frame is divided into a downlink energy harvesting phase and an uplink information transmission phase. During the downlink energy harvesting phase, the base station is controlled to send a radio frequency energy signal to the smart reflector, so that the smart reflector can harvest energy according to the radio frequency energy signal. In the current time frame, the smart reflector harvests ambient energy through the hybrid energy harvesting module. During the uplink information transmission phase, the user terminal is controlled to send uplink data signals to the base station according to the optimized user transmit power, and the reflection parameters of each subarray in the smart reflector are adjusted according to the optimized reflection unit mode switching sequence, the optimized reflection phase shift sequence, and the optimized reflection amplification coefficient sequence.

[0006] To achieve the above objectives, a second aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the parameter optimization method for the hybrid energy harvesting subarray reflector communication system as described in the first aspect.

[0007] To achieve the above objectives, a third aspect of this application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the parameter optimization method for the hybrid energy harvesting subarray reflector communication system as described in the second aspect.

[0008] This application proposes a parameter optimization method and related equipment for a hybrid energy harvesting subarray reflector communication system. The hybrid energy harvesting subarray reflector communication system includes a base station, a smart reflector, and multiple user terminals. The smart reflector is equipped with a hybrid energy harvesting module and multiple reflector subarrays, each of which includes multiple reflector subarrays. The method includes: acquiring first channel state information between the base station and the smart reflector, second channel state information between each user terminal and the smart reflector, and third channel state information between each user terminal and the base station in the current time frame; and acquiring the current battery energy state information of the smart reflector; inputting the first channel state information, second channel state information, third channel state information, and battery energy state information into a communication parameter optimization model for data processing to obtain an optimized set. The combined parameters are optimized, including optimizing the radio frequency energy harvesting duration, optimizing the user transmit power, optimizing the reflector mode switching sequence, optimizing the reflection phase shift sequence, and optimizing the reflection amplification coefficient sequence. Based on the optimized radio frequency energy harvesting duration, the current time frame is divided into a downlink energy harvesting phase and an uplink information transmission phase. During the downlink energy harvesting phase, the base station is controlled to send radio frequency energy signals to the intelligent reflector so that the intelligent reflector can harvest energy based on the radio frequency energy signals. In the current time frame, the intelligent reflector collects ambient energy through the hybrid energy harvesting module. During the uplink information transmission phase, based on the optimized user transmit power, the user terminal is controlled to send uplink data signals to the base station, and the reflection parameters of each subarray in the intelligent reflector are adjusted according to the optimized reflector mode switching sequence, optimized reflection phase shift sequence, and optimized reflection amplification coefficient sequence.

[0009] This application embodiment acquires multi-link channel state information of the communication network and battery energy state information of the smart reflector, and outputs optimized radio frequency energy harvesting duration and a series of reflection adjustment parameters based on a communication parameter optimization model. This enables the dynamic division of time frames into downlink energy harvesting phases and uplink information transmission phases. Furthermore, during the downlink phase, the base station sends radio frequency signals to the smart reflector to supplement energy, and environmental energy is harvested throughout the current time frame. During the uplink phase, the active / passive mode of the subarray and reflection parameters are jointly adjusted to assist uplink data transmission. This utilizes the idle time of the smart reflector when environmental energy supply is insufficient for radio frequency energy harvesting. By rationally allocating time resources and coordinating multi-dimensional parameter optimization between energy harvesting and data transmission, the limitations of relying solely on environmental energy harvesting such as solar energy are easily affected by external fluctuations, resulting in a shortened effective working time. While maintaining the low-power operation of the smart reflector sub-connection architecture, the long-term average throughput and communication reliability of the communication system in energy-constrained scenarios are improved.

[0010] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the structure of a hybrid energy harvesting subarray reflector communication system provided in an embodiment of this application.

[0012] Figure 2 This is a schematic diagram of a subarray structure provided in another embodiment of this application.

[0013] Figure 3 This is a schematic diagram of a time frame structure provided in another embodiment of this application.

[0014] Figure 4 This is a flowchart of a parameter optimization method for a hybrid energy harvesting subarray reflector communication system provided in another embodiment of this application.

[0015] Figure 5 This is a schematic diagram of the interaction and training framework structure of a communication parameter optimization model based on hierarchical deep reinforcement learning (HDRL) provided in another embodiment of this application.

[0016] Figure 6 This is a first performance simulation diagram of the parameter optimization method for a hybrid energy harvesting subarray communication system provided in another embodiment of this application.

[0017] Figure 7 This is a second performance simulation diagram of the parameter optimization method for a hybrid energy harvesting subarray communication system provided in another embodiment of this application.

[0018] Figure 8 This is a third performance simulation diagram of the parameter optimization method for a hybrid energy harvesting subarray communication system provided in another embodiment of this application.

[0019] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] In this application, vectors are represented by bold lowercase letters, and matrices are represented by bold uppercase and lowercase letters. The dimension is A complex matrix. and Represents the transpose and conjugate transpose of a vector or matrix. Symbol It represents the statistical expectation. This indicates a diagonalization operation. The dimension is The identity matrix. and These represent the norm of the matrix and the modulus of the complex number, respectively.

[0024] Reconfigurable smart surfaces (RIS), as a technology capable of modulating the electromagnetic wave propagation environment, have attracted widespread attention for improving the performance of wireless communication systems. To alleviate the high power consumption of active RIS and address power supply limitations in remote scenarios, related technologies typically employ a sub-connected hybrid architecture (i.e., multiple reflector units sharing a power amplifier and supporting active / passive mode switching) to reduce hardware power consumption. Power is supplied to the RIS via a fixed power source or environmental energy harvesting (such as solar energy), enabling self-powered operation. However, using a fixed power source requires additional power lines to be laid in the communication system, which can restrict the placement of the RIS and reduce the flexibility and convenience of network deployment. Furthermore, reliance on environmental energy harvesting methods such as solar energy is susceptible to external environmental fluctuations. Insufficient environmental energy supply can significantly shorten the effective operating time of the RIS within a time slot, leading to a substantial decrease in system communication throughput.

[0025] Based on this, the embodiments of this application obtain multi-link channel state information of the communication network and battery energy state information of the smart reflector, and output optimized radio frequency energy harvesting duration and a series of reflection adjustment parameters based on the communication parameter optimization model. This enables the time frame to be dynamically divided into downlink energy harvesting stage and uplink information transmission stage. Furthermore, by using the base station to send radio frequency signals to the smart reflector for energy replenishment during the downlink stage, and by harvesting ambient energy throughout the current time frame, and by jointly adjusting the active / passive mode of the subarray and reflection parameters during the uplink stage to assist uplink data transmission, the idle time of the smart reflector is utilized for radio frequency energy harvesting when ambient energy supply is insufficient. By rationally allocating time resources and coordinating multi-dimensional parameter optimization between energy harvesting and data transmission, the limitations of relying solely on ambient energy harvesting such as solar energy are easily affected by external fluctuations, resulting in a shortened effective working time. Under the premise of maintaining low-power operation of the smart reflector sub-connection architecture, the long-term average throughput and communication reliability of the communication system in energy-constrained scenarios are improved.

[0026] The parameter optimization method and related equipment for the hybrid energy harvesting subarray reflector communication system proposed in the embodiments of this application will be further described below. First, a hybrid energy harvesting subarray reflector communication system will be described. (Refer to...) Figure 1 This is a schematic diagram of a hybrid energy harvesting subarray reflector communication system provided in an embodiment of this application. Figure 1 As shown, this hybrid energy harvesting subarray reflector communication system includes a base station, a smart reflector (i.e., Figure 1 The base station includes a hybrid RIS and multiple user terminals (i.e., the terminals carried by users 1 to J). Root antenna, RIS has There are units, For each user, this solution adopts... Indicates the base station number Root antenna, Indicates RIS One reflective unit, Indicates RIS Each user terminal. In actual communication scenarios, the uplink information signals sent by the user terminal can not only be directly transmitted to the base station through the direct connection channel, but also be auxiliaryly transmitted through the cascaded channel formed by the intelligent reflector (that is, the signal is first sent by the user terminal to the intelligent reflector, and then reflected by the intelligent reflector to the base station), thereby effectively improving the wireless propagation environment and enhancing the received signal strength on the base station side.

[0027] Furthermore, such as Figure 1As shown, the intelligent reflector is equipped with a hybrid energy harvesting module (including energy harvesting circuitry and a battery) and multiple reflector subarrays, each containing multiple reflector units. The hybrid energy harvesting module supports multi-source energy acquisition; it can collect not only environmental energy such as solar and wind power, but also receive radio frequency energy signals from the base station, converting this multi-source energy into electrical energy and storing it in the battery to maintain the intelligent reflector's self-powered operation. Simultaneously, a controller is configured on the intelligent reflector side, connected to the battery and each reflector unit. This controller flexibly controls each reflector unit in the reflector array in active mode based on energy status and communication requirements. Figure 1 (Red tile in the middle) and passive mode ( Figure 1 The system dynamically switches between green blocks to achieve efficient reflection and amplification of incident information signals while reducing system hardware power consumption.

[0028] The intelligent reflector employs a sub-array hardware layout. By dividing all reflective elements into several sub-arrays and configuring a shared power amplifier circuit within each sub-array, the active amplification resources of multiple reflective elements within the sub-array are shared. This effectively reduces the overall static power consumption and implementation cost of the hardware system while providing the necessary signal gain. Simultaneously, to ensure the flexibility of beam reconfiguration, each reflective element within each sub-array is independently and correspondingly equipped with a phase shift circuit and a mode switching switch. This allows each element to independently perform phase offset adjustment based on control commands and dynamically switch between active amplification mode and passive reflection mode using the mode switching switch. This enables the system to finely schedule hardware resources based on real-time channel state information and available battery power, further optimizing the system's self-sustaining operation capability under energy-constrained conditions while maintaining the system's auxiliary transmission performance.

[0029] The RIS employs a hybrid sub-connection architecture. In RIS operating mode, each unit can autonomously switch between active and passive modes. Units within the same group share the same power amplifier circuit. The RIS as a whole can switch between RF energy harvesting mode and RIS operating mode. (Refer to...) Figure 2 This is a schematic diagram of a subarray structure provided in an embodiment of this application. Figure 2 As shown, the array contains multiple (e.g., four as illustrated) reflective units, which are interconnected and share the same power amplifier circuit to form a sub-connection architecture, thereby effectively reducing hardware static power consumption. To switch between RF energy harvesting mode and RIS operating mode, when At that time, the RIS is in radio frequency energy harvesting mode. At this time, the RIS is in working mode. Divide the RIS evenly into... Group, Indicates the first Groups, each group (i.e., subarray) has If there are 1000 reflection units, then the number of RIS groups is 1000. Each group shares a power amplifier, but each reflecting unit uses an independent phase-shift circuit and switch, which controls whether the reflecting unit is in active or passive mode. Definition ,in . This represents the reflective unit. The switch state, when RIS When the switch of a reflective unit is closed, the reflective unit is in active mode; when RIS When the switch of a reflective unit is off, the reflective unit is in passive mode. Definition , represents the phase shift vector of RIS, where It indicates the first The first time slot The phase shift of each reflecting unit, in practical applications, Take discrete values, define If the number of quantization bits for the discrete phase shift of each reflecting unit is given, then the set of discrete phase shifts for each reflecting unit is: ,in .definition ,in It indicates the first The first time slot The magnification factor of each group is defined as follows: .

[0030] This represents the reflection coefficient matrix of the hybrid sub-connected RIS, where This represents a diagonal matrix with the elements in parentheses forming its diagonal elements. The expression is as follows: ; definition This represents the amplified signal and noise figure matrix of the active reflector unit. .

[0031] To achieve coordinated optimization of downlink RF energy harvesting and uplink data transmission, the system employs a time-division multiplexing architecture to divide each time frame. Within a single time frame, time resources are divided into a downlink energy harvesting phase and an uplink data transmission phase. On one hand, the downlink energy harvesting phase provides additional operating power for the RIS (Resource Identifier); on the other hand, the uplink data transmission phase is used to upload user data. By rationally allocating time resources, an effective trade-off can be achieved between RF energy acquisition and data transmission, thereby improving the overall system performance and energy utilization efficiency.

[0032] Reference Figure 3 This is a schematic diagram of a time frame structure provided in an embodiment of this application. Figure 3 As shown, the time frame structure of the communication system includes a downlink radio frequency energy harvesting phase (i.e., downlink energy harvesting phase), an uplink information transmission phase, and an ambient energy harvesting phase. The downlink RF energy harvesting phase lasts for a time frame. At this time, the RIS is in radio frequency energy harvesting mode. The uplink information transmission phase lasts for [duration]. At this time, RIS is in working mode. The environmental energy harvesting phase lasts for [duration]. ,in .exist The average radio frequency energy harvesting time for each time slot is given by the following formula: ; During the downlink energy harvesting phase, the base station to RIS channel is represented as... The channel modeling is shown in the following formula: ; The channel model from the base station to the RIS is a Ricean fading channel. For NLOS components, each element of the vector is modeled as an independent and identically distributed complex Gaussian random variable with a mean of zero and a variance of unit variance, i.e. . The LOS component is modeled as shown in the following formula: ; in and These are the azimuth and elevation angles from the base station to the RIS. It is the departure azimuth angle from the base station to the RIS.

[0033] Based on the hybrid energy harvesting subarray reflector communication system described above, a parameter optimization method for the hybrid energy harvesting subarray reflector communication system provided in this application embodiment will be further described below. The parameter optimization method for the hybrid energy harvesting subarray reflector communication system provided in this application embodiment can be applied to a base station in a communication system or a processor connected to the base station, etc. (Refer to...) Figure 4 This is an optional flowchart of a parameter optimization method for a hybrid energy harvesting subarray reflector communication system provided in this application embodiment. Figure 4 The method may include, but is not limited to, steps 100 to 400. It is also understood that this embodiment... Figure 4 The order of steps 100 to 400 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0034] Step 100: Obtain the first channel state information between the base station and the smart reflector, the second channel state information between each user terminal and the smart reflector, and the third channel state information between each user terminal and the base station in the current time frame, and obtain the battery energy state information of the smart reflector.

[0035] Step 100 is described in detail below.

[0036] In step 100 of some embodiments, the first channel state information This refers to the channel fading and response parameters of the downlink communication link established between the base station and the intelligent reflector, and the second channel state information. This refers to the channel fading and response parameters of the uplink communication link established between each user terminal and the intelligent reflector, and the third channel state information. This refers to the channel fading and response parameters of the direct uplink communication links established between each user terminal and the base station. Battery energy state information refers to the available energy stored in the rechargeable battery equipped on the smart reflector at the initial moment of the current time frame. Because the wireless propagation environment and energy state of the communication system have strong time-varying characteristics, in order to achieve accurate dynamic resource allocation, it is necessary to obtain the physical environment state of the system at the initial stage of each time frame. Therefore, the control system needs to obtain the state information of the direct and cascaded channels between the base station, the smart reflector, and multiple users through methods such as channel estimation, in order to quantitatively assess the signal attenuation under the current environment. Simultaneously, because the smart reflector in this scheme does not rely on a fixed power grid but achieves self-powered operation through multi-source energy harvesting, it is necessary to determine the current available energy reserve of the RIS battery module in real time, thereby facilitating the subsequent calculation of the reflection working time and amplification power within the current time frame.

[0037] Step 200: Input the first channel state information, the second channel state information, the third channel state information, and the battery energy state information into the communication parameter optimization model for data processing to obtain optimized combined parameters. The optimized combined parameters include optimized radio frequency energy harvesting duration, optimized user transmit power, optimized reflector mode switching sequence, optimized reflection phase shift sequence, and optimized reflection amplification coefficient sequence.

[0038] Step 200 is described in detail below.

[0039] In step 200 of some embodiments, the communication parameter optimization model refers to a deep reinforcement learning network structure pre-trained and constructed to output the optimal configuration parameters of the system under complex environmental conditions; the optimized combined parameters refer to the joint decision instruction set output by the model, which covers timing partitioning, terminal control, and reflector hardware allocation, specifically including optimizing the radio frequency energy harvesting duration, optimizing the user transmit power, optimizing the reflector unit mode switching sequence, optimizing the reflection phase shift sequence, and optimizing the reflection amplification coefficient sequence.

[0040] Among them, optimizing the radio frequency energy harvesting duration refers to the length of the time segment allocated to the base station to transmit radio frequency energy signals to the smart reflector within a time frame; optimizing the user transmit power refers to the target transmit power value adopted by each user terminal when transmitting data signals in the uplink; optimizing the reflector mode switch sequence refers to the set of discrete control instructions used to indicate whether each reflector in the smart reflector is in a closed active mode or an open passive mode; optimizing the reflection phase shift sequence refers to the set of phase shift control instructions used to adjust the phase shift of each unit of the smart reflector to the incident signal; and optimizing the reflection amplification coefficient sequence refers to the set of gain control instructions used to adjust the signal amplitude enhancement of the smart reflector group in active mode.

[0041] Traditional numerical calculation methods struggle to effectively solve the throughput maximization problem for systems involving discrete integer variables, continuous variables, and multiple time slot coupling within a short timeframe. Therefore, this application's solution uses acquired complex channel state information and battery energy as input state features, processing them in a communication parameter optimization model. This model extracts the correlation features between the current physical environment and energy state through a forward propagation network, comprehensively considers long-term energy scheduling causal relationships, and outputs the resource allocation action for the current time frame.

[0042] The construction steps of the communication parameter optimization model include steps 210 to 240.

[0043] Step 210: Based on the circuit static power and amplification power of the smart reflector, construct the total power consumption model of the system, and construct the battery energy dispatch constraints based on the radio frequency energy harvesting model and the environmental energy harvesting model.

[0044] Step 210 will be described in detail below.

[0045] In step 210 of some embodiments, the static power of the circuit refers to the inherent power loss generated by the phase shift circuit and power amplifier circuit in the smart reflector when maintaining the basic operating state; the amplification power refers to the dynamic power consumed by the power amplifier circuit to amplify the incident signal when the smart reflector subarray is in active mode. Based on the above two parts of energy consumption, the system adds the power consumption of the passive mode unit and the power consumption of the active mode unit to construct a total system power consumption model for quantifying the overall power consumption of the smart reflector.

[0046] In this application, in the first The static power consumed by the cells of the time-slotted RIS in passive mode is given by the following formula: ; in This represents the static power consumed by a phase shift circuit. Divided into Group, put No. The sum of all elements of a group is expressed as: Define variables , its first The elements are represented by the following formula: ; In the The amplification power consumed by the cells in active mode of the time-slotted RIS is shown in the following formula: ; in, This refers to the quiescent power consumed by the power amplifier circuit. This is the reciprocal of the amplification efficiency. In the... The power consumed by the time-slotted RIS is Therefore, in the first... The model for the total system power consumption corresponding to the RIS consumption during the uplink information transmission phase in a time slot is as follows: Then in the first... The energy consumed by RIS during the uplink information transmission phase in a time slot is: .

[0047] Furthermore, the radio frequency energy harvesting model is used to characterize the process by which the intelligent reflector converts and obtains DC power from the high-frequency electromagnetic energy signal transmitted by the base station during the downlink phase; the environmental energy harvesting model is used to characterize the process by which the system obtains supplementary power from the external natural environment (such as solar energy), and the environmental energy is set to have a time-domain delayed effect. Based on the power consumption model and the multi-source energy harvesting model, battery energy scheduling constraints are constructed to limit the total energy consumed by the intelligent reflector during the uplink information transmission phase to no more than the sum of the battery energy state information at the initial moment of the current time slot and the radio frequency energy collected in the current time slot, as described in detail below.

[0048] The process of constructing battery energy dispatch constraints based on the radio frequency energy harvesting model and the environmental energy harvesting model includes the following steps 211 to 214.

[0049] Step 211: Based on the input signal power, energy conversion efficiency and preset saturation power threshold of the downlink energy harvesting stage, a radio frequency energy harvesting model with piecewise linear characteristics is obtained.

[0050] Step 212: Based on the cross-time domain characteristics of environmental energy harvesting, an environmental energy harvesting model with delayed activation characteristics is obtained; wherein, the environmental energy harvested in the current time frame is constrained to be used only as battery energy increment in subsequent time frames.

[0051] Step 213: Based on the radio frequency harvested energy obtained from the radio frequency energy harvesting model, the initial battery energy of the current time frame, and the power consumption of the reflector determined by the combination of optimized parameters, construct an energy causal constraint. The energy causal constraint is used to limit the energy consumed during the uplink information transmission phase to no more than the sum of the initial battery energy and the radio frequency harvested energy.

[0052] Step 214: Based on the preset maximum battery capacity value, the sum of the remaining energy determined by the energy causal constraint and the energy increment provided by the environmental energy harvesting model is subjected to capacity limiting processing to obtain the initial battery energy of the next time frame, so as to construct the battery energy scheduling constraint conditions.

[0053] Steps 211 to 214 are described in detail below.

[0054] In step 211 of some embodiments, the input signal power refers to the high-frequency radio electromagnetic wave power received from the base station at the smart reflector antenna end during the downlink energy harvesting phase; the energy conversion efficiency refers to the inherent proportionality coefficient by which the rectifier circuit inside the smart reflector converts the received high-frequency radio frequency energy into usable DC power; the preset saturation power threshold refers to the maximum input power threshold that the radio frequency energy harvesting circuit can handle, which is limited by the physical characteristics of nonlinear components; the radio frequency energy harvesting model with piecewise linear characteristics refers to the mathematical relationship used to characterize the proportional conversion of input power into electrical energy when it is below the saturation threshold and when the conversion power remains unchanged when it is above the saturation threshold.

[0055] In practical implementation, since actual RF energy harvesting circuits typically include nonlinear devices such as diodes, traditional linear models cannot accurately reflect energy saturation phenomena under high input power. Therefore, this solution introduces a saturation power threshold. The correction is made so that when the input signal power arriving at the base station does not reach the saturation power threshold, the model output collection power is linearly positively correlated with the input signal power and energy conversion efficiency; however, when the input signal power is strong and exceeds the preset threshold, the circuit reaches saturation, and the instantaneous collection power corresponding to the model no longer increases with the input power, but remains at a constant maximum value.

[0056] In this application, in the first During the downlink energy harvesting phase of the time slot, the base station sends an RF energy signal to the RIS, and the received signal at the RIS is... ,in, It means that in Transmit beamforming vector of time-slot base station, Is The energy signal transmitted from the time-slot base station to the RIS follows a complex Gaussian distribution with a mean of 0 and a variance of 1. .

[0057] exist The power of the signal received by the time-slotted RIS is The expectation operator It is acting on The above can be calculated to obtain Assume the base station's transmit power is... ,but To maximize the power of the received signal, the optimal transmit beamforming vector for the base station is given by the following formula: ; in, It is the first Time slot matrix The eigenvector corresponding to the largest eigenvalue. Then the power of the RF energy harvesting model with piecewise linear characteristics corresponding to the RF signal collected by RIS is given by the following formula: ; in, For radio frequency energy harvesting efficiency, This indicates the saturation power for radio frequency energy harvesting; beyond this power, the harvested power will no longer increase. In the time slot, the radio frequency energy collected by the RIS during the downlink radio frequency energy harvesting phase is: .

[0058] In step 212 of some embodiments, the cross-temporal characteristic of environmental energy harvesting refers to the physical property that the process of capturing and converting electrical energy from the external environment (such as solar energy) has a time lag, causing it to be unable to be immediately invoked within the captured time slot. An environmental energy harvesting model with delayed activation characteristics refers to a numerical allocation model used to calculate the environmental energy collected within the current time frame and to postpone its usability rights sequentially along the time axis, including battery energy increments. This refers to adding the delayed-on ambient energy as additional charge to the initial charge of the smart reflector in subsequent time frames. The values ​​in the text. In actual self-powered systems, in addition to the radio frequency supplemental energy provided by the base station, the intelligent reflector will continuously collect ambient energy. The ambient energy collected in the current time frame cannot be directly consumed during the uplink information transmission phase of the current time frame. Instead, it is temporarily stored in the battery and only used as incremental battery energy in subsequent time frames (i.e., the next time frame and thereafter).

[0059] In step 213 of some embodiments, during the uplink information transmission phase, the smart reflector needs to consume power to maintain the static power consumption of a large number of phase shift circuits and the dynamic amplification power consumption of some subarray power amplifier circuits. In order to ensure that the device can maintain stable operation and not interrupt communication due to power depletion, an energy causal constraint is established to limit the total power consumed during the uplink information transmission phase to not be greater than the sum of the initial battery energy of the current time frame and the radio frequency collection energy just acquired during the downlink phase of the current time frame.

[0060] In this application, the following is defined: For the first At the start of the first time slot, the battery's energy, in the... The energy consumed by RIS during the uplink information transmission phase of each time slot cannot exceed the energy consumed in the first time slot. At the start of the time slot, the battery's energy is added to the energy of the first time slot. The radio frequency signal energy collected by the RIS during the downlink radio frequency energy harvesting phase of each time slot, i.e., the energy causality constraint. .

[0061] In step 214 of some embodiments, The maximum battery capacity refers to the upper limit of electrical energy storage that the energy storage battery equipped with the smart reflector can accommodate based on its physicochemical properties. Remaining energy. This refers to the available battery power remaining at the end of the current time frame, after deducting uplink data transmission consumption from the sum of initial battery energy and RF harvested energy. Capacity limiting processing is a mathematical truncation operation that discards the excess portion when the theoretically calculated accumulated battery power exceeds the battery's physical capacity limit, keeping the result within the maximum capacity range. At the end of each time frame, the battery state needs to be iteratively updated to prepare for and constrain the scheduling strategy for the next time frame, i.e., in the [missing information - likely a specific time frame]. At the start of the time slot, the battery's energy was This refers to the battery energy dispatch constraints, where... For the first The environmental energy collected in one time slot can only be used in the first time slot. And subsequent time slots cannot be used in the current time slot.

[0062] Through steps 211 to 214 above, a piecewise linear radio frequency energy harvesting model with a saturation power threshold is introduced to correct the energy overestimation deviation caused by energy saturation under high input power. At the same time, by combining the delayed effect characteristics of ambient energy and the maximum capacity limiting processing of the battery, the objective causal law of cross-time frame charging and discharging of the energy storage system is restored, so that when dynamically allocating communication resources, the risk of equipment downtime caused by energy overdraft can be effectively avoided, thereby improving the reliability of data communication in the communication system.

[0063] Step 220: Based on the model that maximizes the average throughput of all user terminals within the preset observation period, and based on the average throughput model, battery energy scheduling constraints, and user service quality constraints, the system throughput maximization problem is obtained.

[0064] Step 230: Decompose the system throughput maximization problem into an inner subproblem based on analytical expressions and an outer subproblem based on Markov decision processes.

[0065] Steps 220 to 230 are described in detail below.

[0066] In step 220 of some embodiments, the average throughput model is an averaged calculation expression used to measure the sum of data transmission rates obtained by each user terminal within the observation period of the communication system. To ensure a basic communication experience in multi-user scenarios, this solution introduces user service quality constraints, which are minimum data transmission rate thresholds set to meet the basic communication needs of each mobile terminal. Furthermore, using the average throughput model as the optimization objective function, and combining battery energy scheduling constraints, user service quality constraints, and constraints on variables involving hardware states, an optimization model is established to maximize system throughput, aiming to improve the long-term average data transmission rate and exhibiting highly coupled multivariate characteristics.

[0067] In this application, Time slot, base station receives user terminal The signal is shown in the following formula: ; in, It means Time slot users Normalized transmitted symbols. ,yes A complex vector, representing the state of... Thermal noise at each reflection unit of the time-slotted RIS This represents the variance of the RIS thermal noise, which is also the power of the thermal noise. ,yes A complex vector, representing the state of... Thermal noise at each antenna of the time-slot base station This represents the variance of the base station's thermal noise, and is also the power of the thermal noise. (Definition) ,in Indicates in Time-slot user terminal The transmission power. Indicates in Time slot users The receiving beamforming vector.

[0068] Define cascaded channels for This refers to the direct connection channel between the user and the base station. and the channel via RIS Add them together.

[0069] exist Time slot users The received SINR is given by the following formula: ; In the Time-slot user terminal The rate is Therefore, in the first The sum rate of the time-slot system is given by the following formula: ; Therefore, the system The average throughput model for each time frame is shown in the following formula: ; In this application, the optimization objective of the system throughput maximization problem is to maximize... The average throughput per time slot is optimized by the switching on / off mode of each RIS unit. Amplification factor of RIS active mode Phase shift of RIS Base station receiving beamforming Base station transmit beamforming User's transmit power Downlink RF energy harvesting phase time The system throughput maximization problem is as follows (P4): ; in, The constraint is the battery energy scheduling relationship constraint between each time slot. The constraint is the first Energy consumption constraints of slotted RIS. Constraints represent the mode selection variables for each unit of the RIS. It is a binary variable used to indicate the first... Each RIS unit is in active or passive mode. The constraint represents the phase shift of RIS units. Taken from a predefined set of discrete phase shifts This is to characterize the phase adjustment capability of a real RIS device with limited resolution. The constraint is a constraint on the amplification factor of the RIS active unit. This represents the maximum value of the magnification factor. Constraints for users Constraints on transmit power This represents the maximum transmission power. Constraints Constraints on the range of values. For users QoS constraints, This represents the minimum throughput. For users The unit mode constraint of the beamforming vector received by the base station. This represents the total transmit power constraint of the base station, ensuring that the power of the base station's transmit beamforming vector is equal to... .

[0070] In step 230 of some embodiments, considering that the system throughput maximization problem is a MIP and SP problem, where variables in a single time slot are coupled with each other, and variables related to energy scheduling in different time slots are also coupled with each other, making it difficult to solve directly using conventional numerical algorithms, this application adopts a hierarchical architecture to decompose it into a dimensionality reduction problem. Specifically, the beamforming part of the system throughput maximization problem, which can be solved by rigorous mathematical formula derivation, is stripped to form an inner sub-problem. This inner sub-problem refers to the computational branch that directly obtains the optimal transmit and receive beamforming vectors of the base station using methods such as eigenvector decomposition of the channel covariance matrix and the minimum mean square error criterion. At the same time, the parts of the system throughput maximization problem that require long-term dynamic coordination, such as time allocation, mode switching, and power control, are constructed as an outer sub-problem. This outer sub-problem refers to the resource scheduling branch that requires time-series dynamic decision-making through system state observation. To solve this outer branch, the system transforms it into a Markov decision process, which is a mathematical decision framework that links the system channel with battery state, resource allocation action execution, and system throughput reward feedback in a chain, thereby providing a standard environment interface for the subsequent introduction of intelligent agent learning.

[0071] The inner sub-problem is used to output the optimized transmit beamforming vector and the optimized receive beamforming vector, decomposing the system throughput maximization problem into an inner sub-problem based on analytical expressions, including the following steps 231 to 233.

[0072] Step 231: Obtain the channel covariance matrix based on the first channel state information, and obtain the optimized transmission beamforming vector based on the eigenvector corresponding to the largest eigenvalue in the channel covariance matrix.

[0073] Step 232: Generate a cascaded equivalent channel model based on the first channel state information, the second channel state information, the third channel state information, and the transmit power and reflection parameters output from the outer subproblem.

[0074] Step 233: Based on the minimum mean square error criterion, and according to the cascaded equivalent channel model and system noise power, the inner sub-problem corresponding to the optimization of the received beamforming vector is obtained.

[0075] Steps 231 to 233 are described in detail below.

[0076] In step 231 of some embodiments, the channel covariance matrix refers to the matrix used to characterize the spatial correlation characteristics of the channel, obtained by multiplying the first channel state information by its conjugate transpose. The eigenvector corresponding to the largest eigenvalue refers to the spatial projection vector corresponding to the component with the highest energy density after eigenvalue decomposition of the channel covariance matrix. The optimized transmit beamforming vector refers to the antenna array weighting parameters determined by the base station in the downlink phase to obtain better received power at the smart reflector.

[0077] During the downlink energy harvesting phase, since energy transmission efficiency is highly dependent on the channel spatial characteristics between the base station and the smart reflector, the channel covariance matrix is ​​first calculated using the acquired first channel state information. Specifically, the eigenvector corresponding to the largest eigenvalue in this matrix is ​​extracted using analytical methods and used as the base station's optimized transmission beamforming vector.

[0078] In step 232 of some embodiments, the transmit power and reflection parameters output by the outer subproblem refer to the user transmit energy intensity determined by the decision model in the current time slot, as well as the switching state, phase, and gain performed by the reflector. The cascaded equivalent channel model refers to the comprehensive channel description formed by mathematically merging the direct signal path from the user to the base station with the reflected superimposed signal path after being modulated by the reflector.

[0079] During the uplink information transmission phase, mathematical modeling is performed using the first, second, and third channel state information, combined with the user transmit power provided in real time by the outer subproblem and the reflection coefficient matrix of each unit of the intelligent reflector. By coherently superimposing the channel response of the direct link with the response of the reflected link, which includes the phase offset and amplitude amplification of the reflector, a cascaded equivalent channel model is generated.

[0080] In step 233 of some embodiments, the minimum mean square error (MMSE) criterion refers to a mathematical criterion for determining the receive weights by minimizing the mean square error between the estimated value of the received signal and the original transmitted signal. The optimized receive beamforming vector refers to the receiver antenna weighting vector used by the base station during the uplink phase to suppress interference and enhance the signal gain of the target user.

[0081] After completing the cascaded channel modeling, the noise term is combined with the base station's noise floor to form the system noise power. Combined with the cascaded equivalent channel model, the minimum mean square error criterion is applied for calculation. By constructing an inverse matrix operation relationship that includes the channel matrix product and the noise variance term, the base station can effectively aggregate user signal energy at the receiving end through spatial filtering, while suppressing multiple access interference from other users and the noise floor introduced by active reflection. This yields the inner-layer subproblem for solving the optimized receiving beamforming vector.

[0082] In this application, the system throughput maximization problem (P4) is first transformed into an inner sub-problem (P5), based on... , , , and Maximize average throughput and optimize variables and The inner sub-problem (P5) is modeled as follows: ; Since (P5) is still difficult to solve directly, the proposed solution decomposes it into two inner-level subproblems (P5-a) and (P5-b), as described below.

[0083] Inner sub-problems (P5-a) with respect to variables Optimization is performed to obtain the optimized received beamforming vector. The inner sub-problem (P5-a) is modeled as shown in the following formula: ; Then optimize the received beamforming vector It can be derived from the formula Obtain and satisfy the constraints As shown in the formula below: ; The constraints were satisfied during variable construction. It is handled in the reward function of the outer subproblem.

[0084] The inner sub-problem (P5-b) addresses optimizing the transmitted beamforming vector. After optimization, the inner subproblem (P5-b) is modeled as shown in the following formula: ; Since the MMSE detector is the optimal receiver beamforming scheme, the user terminal The optimized transmit beamforming vector based on MMSE can be expressed as follows: ; The channel covariance matrix is ​​given by the following formula: ; but constraint satisfy, The constraints are addressed in the reward function of the outer subproblem.

[0085] Through steps 231 to 233 above, efficient directional transmission of radio frequency energy was achieved in the downlink phase using eigenvalue decomposition technology, ensuring the power supply of the self-powered system. At the same time, by constructing a cascaded equivalent channel model that integrates the entire link state and using the minimum mean square error criterion, the receiving performance of the base station side was optimized while fully considering the additional amplified noise introduced by active reflection. This reduced the computational load of the deep reinforcement learning model when dealing with high-dimensional continuous variables, not only improving the response speed to dynamic changes in the channel, but also enhancing the system's average data throughput and anti-interference capability in energy-constrained environments through detailed modeling of the physical layer signal transmission process.

[0086] In this application, the system throughput maximization problem (P4) is transformed into an outer subproblem (P6), maximizing... The average throughput of each time slot is calculated. , , , and The outer subproblem is modeled as shown in the following formula (P6): ; Step 240: Construct an initial hierarchical deep reinforcement learning model based on the inner and outer sub-problems, and train the initial hierarchical deep reinforcement learning model in multiple rounds to obtain a communication parameter optimization model.

[0087] Step 240 will be described in detail below.

[0088] The outer subproblem (P6) remains a MIP and SP problem. Since the optimization variables include both discrete and continuous variables, to address this issue, the proposed solution first formulates the outer subproblem (P6) as an MDP, and then uses a PPO-based DRL algorithm to solve the problem (P6).

[0089] In step 240 of some embodiments, the initial hierarchical deep reinforcement learning model refers to an untrained intelligent architecture that is initially built, containing an actor network and a commentator network for handling Markov decision processes, and internally nested with inner-layer analytical computational logic. After the initial model is built, the system performs multiple rounds of training on the model in a set simulation environment, through repeated closed-loop iterative processes such as "state acquisition, action sampling and output, calling inner-layer computation, obtaining rewards and storing them in the experience replay pool," and subsequent parameter gradient updates. Through multiple experience replays and backpropagation adjustments of network weights, the model gradually converges and learns the optimal resource mapping strategy in complex energy environments. The final solidified network model, which can be directly used for online output control commands, is the communication parameter optimization model.

[0090] The initial hierarchical deep reinforcement learning model is constructed based on the inner and outer sub-problems, including the following steps 241 to 244.

[0091] Step 241: Based on the first channel state information, the second channel state information, the third channel state information, and the battery energy state information, obtain the state space.

[0092] Step 242: Based on the RF energy harvesting duration, user transmit power, reflection unit mode switching sequence, reflection phase shift sequence, and reflection amplification coefficient sequence, obtain the action space.

[0093] Step 243: Based on the output results of the inner sub-problems and the instantaneous sum and rate of the action space calculation system, and according to the battery energy scheduling constraints and user service quality constraints, obtain the reward model.

[0094] Step 244: Obtain the initial network parameters of the actor network and the initial comment parameters of the commenter network, and generate an initial hierarchical deep reinforcement learning model based on the state space, action space, and reward model.

[0095] Steps 241 to 244 are described in detail below.

[0096] In step 241 of some embodiments, the state space refers to the set of observed variables used to provide the agent with information about the current physical environment and energy reserves of the system. In specific implementations, since the communication system under study faces dynamically changing wireless channels and random fluctuations in environmental energy, in order to enable the intelligent algorithm to accurately perceive the external environment and make reasonable decisions, the channel fading states of multiple spatial paths are combined with the hardware energy reserves of the intelligent reflector. This allows the channel state information of these three links and the real-time battery energy state information to be used together as input features, providing a complete environmental observation basis for the reinforcement learning model, thereby constituting a state space that characterizes the current operating state of the system.

[0097] In this application, the state space is Defined as .

[0098] exist The state of the time step is The following formula is defined: ; In step 242 of some embodiments, the action space refers to the set of all control commands and resource allocation variables that the agent can output in each decision step. Since the outer subproblem involves multi-dimensional resource allocation and hardware control, these variables to be optimized need to be mapped to actions that the agent can execute. The solution of this application combines the radio frequency energy harvesting duration involving the time layer, the user transmit power involving the power control layer, and the mode switching, phase shift, and amplification coefficient sequences involving the underlying hardware configuration of the reflector to form the action space, enabling the agent to simultaneously complete the joint control of macroscopic protocol timing allocation and microscopic hardware parameters within a single decision step.

[0099] In this application, the action space is Defined as ,exist The state of the time step is Defined as ,action Divided into discrete actions , and continuous action , , .

[0100] In step 243 of some embodiments, the output of the inner subproblem refers to the base station-side transmit and receive beamforming vectors calculated using analytical formulas. System instantaneous sum rate refers to the sum of data transmission rates achieved by all user terminals under the parameter configuration of the current time slot. The reward model refers to the computational function used to evaluate the quality of the agent's actions and provide quantitative numerical feedback.

[0101] In reinforcement learning, the agent needs reward signals to clarify the learning direction of the policy. Therefore, the action space variables output by the agent are first substituted into the physical environment, and combined with the beamforming vector obtained from the inner-layer analytical calculation, the instantaneous time and rate of the system under the current control policy are calculated. To ensure that the policy learned by the agent conforms to the actual physical and business constraints, this instantaneous time and rate are used as the basic reward, and a penalty mechanism is introduced: if the generated action causes the energy consumption of the intelligent reflector to break the battery energy scheduling constraint, or if a user's rate is lower than the user service quality constraint, the system will apply a negative penalty value to the basic reward. By combining throughput reward and limit violation penalty in this way, a reward model that guides the agent to find the optimal solution within the feasible domain is constructed.

[0102] In this application, the inner sub-problem (P5-a) is first solved to obtain the optimized receiving beamforming vector. Solve the inner sub-problem (P5-b) to obtain the optimized transmit beamforming vector. Find and Then process the constraints. The following formula is obtained.

[0103] ; Considering constraints , The minimum value is shown in the following formula: ; therefore, The range of values ​​is .

[0104] From the formula Calculate the first The sum rate of the time slot system Next, handle the constraints. If constraints If not satisfied, then reward. If constraints Satisfaction, reward Therefore, the reward model is as follows: ; In step 244 of some embodiments, the actor network refers to a policy neural network responsible for directly outputting specific control actions based on the observed state. Initial network parameters refer to the neuron connection weights set for the actor network before model training begins. The commentator network refers to a value neural network used to evaluate the long-term value of the actions selected by the actor network. Initial comment parameters refer to the weight values ​​assigned to the commentator network to initiate state value evaluation. The initial hierarchical deep reinforcement learning model refers to the basic algorithmic architecture assembled from the aforementioned state, action, and reward elements, as well as an insufficiently trained dual neural network structure.

[0105] After defining the core elements of the Markov decision process, the system first assigns initial parameter weights to the dual network architecture in the algorithm (i.e., an actor network that handles action branches and a commenter network that is responsible for evaluation). Subsequently, the system establishes the state space as the network's input interface dimension, the action space as the network's output dimension, and the reward model as the feedback evaluation mechanism driving network weight updates. By binding mathematical definitions to the neural network framework, an initial hierarchical deep reinforcement learning model with preliminary environmental interaction capabilities is generated.

[0106] Through steps 241 to 244 above, the complex communication and energy scheduling problem is systematically transformed into a standard Markov decision process, improving the reliability of the hierarchical deep reinforcement learning model. By collecting channel and battery states to construct the state space and integrating multi-dimensional control parameters into the action space, the agent is ensured to have full perception and comprehensive control over the communication environment. At the same time, by adding a penalty mechanism for energy causality and quality of service constraints to the reward model, the agent is effectively guided to explore the optimal solution for system throughput within the legal physical boundaries.

[0107] The actor network includes discrete action networks and continuous action networks, and the initial hierarchical deep reinforcement learning model is trained in multiple rounds, including the following steps 245 to 2410.

[0108] Step 245: In each round of training, the current training state is input into the discrete action network for data processing to obtain the classification logic value. The classification logic value is then normalized to obtain the classification distribution. The updated reflection unit mode switch sequence is then sampled from the classification distribution.

[0109] Step 246: Input the current training state into the output distribution parameters of the continuous action network, construct a beta distribution based on the distribution parameters, sample the original action variables from the beta distribution, and linearly scale the original action variables based on the physical value range constraints to obtain the updated RF energy harvesting duration, updated user transmit power, updated reflection phase shift sequence, and updated reflection amplification coefficient sequence.

[0110] Step 247: Based on the updated reflection unit mode switching sequence, updated RF energy harvesting duration, updated user transmit power, updated reflection phase shift sequence, updated reflection amplification coefficient sequence, and inner layer subproblems, calculate to obtain the updated beamforming vector, and based on the updated beamforming vector and reward model, obtain the current reward value and the updated training state for the next time step.

[0111] Step 248: Combine the current training state, current training action combination, current reward value, and updated training state into training experience information and store it in the experience replay pool. The current training action combination includes updating the reflector mode switching sequence, updating the radio frequency energy harvesting duration, updating the user transmit power, updating the reflection phase shift sequence, and updating the reflection amplification coefficient sequence.

[0112] Step 249: Extract batch training experience information from the experience replay pool, use the reward value in the batch training experience information to calculate the time difference error to update the value function and initial comment parameters of the commenter network, and calculate the probability ratio of the new and old strategies based on the advantage function.

[0113] Step 2410: Truncate the probability ratio based on the preset truncation hyperparameter, and update the initial action parameters of the actor network using gradient descent.

[0114] Steps 245 to 2410 are described in detail below.

[0115] This application specifically employs the PPO algorithm. The deep neural network of PPO consists of an Actor network for outputting the policy and a Critic network for estimating the value function. Since the optimization variables of problem (P6) include both discrete and continuous variables, this paper uses two Actor networks: a discrete-action Actor network and a continuous-action Actor network. Assume the parameters of the Actor networks are... The parameters of the discrete action Actor network are: The parameters of the continuous action Actor network are: The parameters of the Critic network are Actions need to be sampled from the distribution output by the Actor network, as described below.

[0116] In step 245 of some embodiments, the discrete action network refers to a branch neural network in the actor network specifically used to output decision results containing a finite number of mutually exclusive options; the classification logic value refers to the raw predicted real number score of the discrete action network for each option without probability normalization; the classification distribution refers to the discrete probability distribution model representing the probability of occurrence of each discrete action after normalizing the classification logic value; and the updated reflection unit mode switch sequence refers to the discrete control instruction set sampled from the classification distribution in this round for reconfiguring the active / passive state of each unit of the reflector.

[0117] In each round of training, since the mode switching of the intelligent reflector is a discontinuous binary decision (i.e., active or passive), the system inputs the perceived environment and energy state separately into the discrete action network branch of the actor network. The discrete action network outputs a set of classification logic values. Then, the system uses the Softmax function and other methods to normalize and map these logic values, constructing a classification distribution that conforms to probabilistic characteristics. Then, it randomly samples according to the probability of each item in the classification distribution to determine the update sequence of the reflector mode switching for this round, so as to achieve effective optimization exploration of the discrete variable space.

[0118] In this application's scheme, for discrete actions and We use a Categorical distribution for sampling, as shown in the following formula: ; ; in, , These are the logits (i.e., classification logic values) output by the discrete action network, based on constraints. and , , Then, the discrete action probability distribution is obtained by Softmax normalization, and the corresponding actions are sampled from the Categorical distribution to obtain the updated reflection unit mode switch sequence.

[0119] In step 246 of some embodiments, the continuous action network refers to a policy branch neural network in the actor network specifically designed to output values ​​within a continuous real number interval; the distribution parameter refers to a numerical variable output by the continuous action network used to determine the shape characteristics (such as shape factor) of a specific probability distribution; the beta distribution refers to a continuous probability distribution model defined within a finite interval and applicable to bounded continuous action sampling; the original action variable refers to an initial continuous value directly sampled from the beta distribution that has not yet been aligned with the actual physical parameter dimensions; and the physical value range constraint refers to the actual upper and lower boundaries set for each continuous control variable according to the hardware and protocol limitations of the communication system.

[0120] When determining continuous control variables, since physical quantities such as time allocation, transmit power, and phase are all bounded continuous variables, this application's scheme inputs the current training state into the continuous action network to output the distribution parameters required to construct the beta distribution. Specifically, this application's scheme samples original action variables with restricted values ​​from the constructed beta distribution, and then, based on physical value range constraints, performs linear translation and scaling mapping on these original action variables to accurately convert them into updated RF energy harvesting duration, updated user transmit power, updated reflection phase shift sequence, and updated reflection amplification coefficient sequence with real physical meaning.

[0121] For continuous actions , and The Beta distribution is used for sampling to obtain the distribution parameters as shown in the following formula: ; ; ; in, , and , as well as , These are the distribution parameters in the Beta distribution of the output of the continuous action Actor network, i.e. and The parameters, whose validity has been guaranteed through positive value mapping, are shown in the following formula: ; ; ; ; ; ; in, , , , , , , This represents the logits (i.e., classification logic values) of the corresponding actions output by the continuous action network. , and This represents the action before scaling, based on constraints. , The sum of equations (4-39) , and The scaling is performed using the following formula: ; ; ; therefore , , .

[0122] In step 247 of some embodiments, the current reward value refers to the quantitative evaluation value that the environment provides to the agent after implementing a new action combination in this round, which integrates throughput performance and constraint satisfaction. The updated training state refers to the new round of observation characteristics that the system exhibits after executing the selected control parameters, as the physical environment evolves and battery energy is consumed. In order to evaluate whether the newly generated control actions are effective, it is necessary to establish a feedback link between the outer layer decision and the inner layer environment. In this application, all the updated action combinations obtained in the first step, including the switching sequence, are substituted into the inner layer sub-problems (i.e., inner layer sub-problems (P5-a) and inner layer sub-problems (P5-b)) to derive the updated beamforming vector (i.e., the optimized transmission beamforming vector) used by the base station side. Harmonized receive beamforming vector Subsequently, based on all actions and beamforming schemes, the instantaneous communication, rate, and energy changes generated by the environment are calculated and substituted into the reward model to output the current reward value. At the same time, the updated training state for the next time step is calculated based on the physical evolution model, thus completing one training interaction between the agent and the simulated environment.

[0123] In step 248 of some embodiments, training experience information refers to an interaction record formed by sequentially packaging the state, action, reward, and next state according to a specific structure. The experience replay pool refers to a data cache container allocated in the system storage space for caching historical training experience information and supporting random sampling. After completing a single-step interactive calculation with the environment, the present application does not immediately use the single data to update the network. Instead, it concatenates and combines the current training state, the current training action combination including switching sequences, power and phase shift sequences, the current reward value, and the deduced updated training state into a structured training experience record. Subsequently, this training experience record is stored in a pre-initialized experience replay pool, and the state transition tuple is continuously accumulated during the training process.

[0124] In step 249 of some embodiments, batch training experience information refers to a set of multiple independent interaction records randomly batch-sampled from the experience replay pool to support gradient calculation; temporal difference error refers to the prediction deviation between the value target calculated based on actual rewards and the current valuation of the commentator network; value function refers to the fitting function maintained internally by the commentator network to predict the expected cumulative future returns of the system under different states; advantage function refers to a metric used to quantify how much additional return executing the current action in a specific state can bring compared to the average strategy; and probability ratio refers to the ratio of the probability of the new policy network generating a certain action to the probability of the old policy network generating the same action during the policy update process.

[0125] During the parameter learning phase, the system randomly extracts batches of training experience information from the experience replay pool for experience replay learning. The proposed solution uses the real reward values ​​recorded in these batches of data to calculate the temporal difference error, and uses this error as the driving force for backpropagation to update the value function and initial comment parameters of the commenter network. This enables the commenter network to more accurately predict the value of the environmental state. Based on this, the advantage function is calculated in combination with the baseline value, and the probability ratio of selecting the same action under the distribution of parameters of the new and old strategies is calculated based on the advantage function. This provides a mathematical basis for evaluating the direction and rationality of subsequent strategy updates.

[0126] In step 2410 of some embodiments, the truncation hyperparameter refers to a scalar threshold pre-set in the algorithm to limit the policy update magnitude boundary. Truncation processing refers to a limiting operation that forcibly restricts probability ratios exceeding the defined range of the truncation hyperparameter to the boundary of the safe range. Gradient descent is an optimization algorithm that iteratively fine-tunes network parameters based on the negative direction of the gradient of the loss function to find local or global optimal solutions. When attempting to update the agent network's policy, to prevent the learned communication policy from collapsing due to excessively large single-step update magnitude, this application introduces a near-end policy optimization mechanism. It uses a pre-set truncation hyperparameter to truncate the calculated probability ratios, limiting the deviation of the new policy from the old policy. Finally, a target loss function incorporating the truncation ratio index is constructed, and gradient descent is used to steadily update the initial action parameters of the agent network along the direction of loss reduction, prompting the network to continuously converge towards a better policy in multiple iterations.

[0127] In this application, an action combination approach is adopted, and the logarithmic probability of an action is defined as shown in the following formula: ; Indicates the state The agent outputs actions in real time. The logarithmic probability. Indicates the state Time output continuous action The probability of is expressed by the following formula: The formula shown below is the probability density function of the Beta distribution.

[0128] ; Indicates the state Time-discrete action output The probability of is expressed by the following formula: ; The state value function is defined as follows: ; This state-value function measures the change from the current state. Initially, the expected returns that the intelligent agent can obtain in the future. This is the discount factor, which determines the importance of future rewards in the current decision. To obtain the state-value function, a Critic network is used for fitting, i.e. In this application, the optimization objective of PPO is shown in the following formula: ; Among them, new strategies and old strategies The probability ratio is shown in the following formula: ; This represents the old parameters before each training round. To reduce the variance of the advantage estimate and control the estimation bias, the GAE method is used. First, the TD error is defined as follows: ; The advantage function is estimated as shown in the following formula: ; In this application, the PPO truncation mechanism is adopted to constrain the differences between the new and old strategies. Accordingly, its optimization objective function is defined as shown in the following formula: ; in, , indicating that Limited to Within the range, It is a hyperparameter that controls the cutoff range. It is policy entropy, which increases the uncertainty of a policy and encourages the exploration of actions. (The coefficient is missing.) This refers to the degree of exploration and control of actions.

[0129] During neural network training, the action network and comment network update parameters using mini-batch stochastic gradient descent, meaning that parameters are randomly drawn from the experience replay pool for each update. 1 piece of experience information Then, the parameters of the action network and the comment network are updated. For the action network, the parameters are updated using the following formula. .

[0130] ; in This represents the learning rate of the action network. For the comment network, the parameters are updated by minimizing the mean squared error. As shown in the formula below: ; in, This represents the learning rate of the comment network. The objective function is represented by the following formula: ; Through steps 245 to 2410 above, a training mechanism with error correction and robustness was designed by using a discrete and continuous dual-branch network for the hybrid action space and combining it with the Proximal Policy Optimization (PPO) framework, achieving efficient convergence of complex communication parameter models. Specifically, the beta distribution combined with physical value range constraints is used to handle continuous variable sampling, effectively avoiding invalid actions that violate physical norms and improving the effectiveness of exploration. At the same time, the temporal correlation of samples is broken through the empirical replay pool mechanism, and a probability ratio limiting process with truncated hyperparameters is introduced to avoid the risk of policy collapse caused by excessively large single gradient update steps in dynamic channels and complex energy-constrained environments. This ensures stable learning and monotonic improvement of the deep reinforcement learning model when dealing with high-dimensional communication scheduling problems, and improves the reliability and accuracy of subsequent output reflector surface control measurements.

[0131] Through steps 210 to 240 above, by employing a hierarchical decoupling strategy, the strongly coupled mixed integer programming problem, which is difficult to handle directly, is decomposed into an inner-layer analytical model and an outer-layer decision model that are easy to solve. This overcomes the limitations of traditional optimization algorithms in dealing with long-term dynamic energy scheduling, which are characterized by excessive complexity and insufficient foresight. It can guide the network to autonomously learn the optimal cooperative strategies for time partitioning, power control, and reflector state switching, while taking into account both user service quality and energy causal boundaries. This improves the operational stability and throughput of multi-source energy harvesting communication systems in energy-constrained environments from the underlying algorithm architecture.

[0132] In this application, the HDRL algorithm framework employs two action networks and one comment network, the structure and parameters of which are shown below: ; Where fc represents a fully connected layer used to implement linear mapping, and Tanh and ReLU are activation functions. The state dimension STATES_DIM, the number of neurons output by the discrete action Actor network ACTION_DISC, and the number of neurons output by the continuous action Actor network ACTION_CONT are shown in the following formulas: ; ; ; The overall process of the HDRL algorithm is to first use the PPO algorithm to solve the outer subproblem (P6) and obtain the switching switch for the active / passive mode of each RIS unit. Amplification factor of RIS active mode Phase shift of RIS User's transmit power and downlink RF energy harvesting time Then, based on the above variables, solve the inner-layer subproblems (P5-a) and (P5-b) to obtain the receiving beamforming of the base station. and base station transmit beamforming .

[0133] Reference Figure 5 This is a schematic diagram of the interaction and training framework structure of a communication parameter optimization model based on hierarchical deep reinforcement learning (HDRL) provided in an embodiment of this application. Figure 5 As shown, the framework mainly includes neural networks (containing Critic networks, discrete action Actor networks, and continuous action Actor networks), inner-layer solution modules, a physical environment, and an experience replay pool. During the decision-making interaction at each time step, the agent first obtains the current state features from the environment. (This includes multi-link channel state information and remaining battery energy, etc.) This state characteristic is synchronously input into both a discrete action Actor network and a continuous action Actor network, which then process and output the actions of the outer subproblem, respectively. (Including the mode switching sequence, phase shift sequence, amplification factor, user transmit power, and downlink RF collection time of the reflector unit, etc.). Subsequently, the inner layer solver module receives the above-mentioned outer layer actions. And calculate the inner action based on the internal analytical expression. (That is, the transmit and receive beamforming vectors of the base station). Ultimately, all action combinations are formed by merging inner and outer layer actions. It is applied to physical environments (i.e., sub-connected hybrid RIS systems with multi-source energy harvesting) for execution.

[0134] After the environment completes the aforementioned action combination, a quantified reward signal is calculated and fed back based on the current throughput performance and the satisfaction of physical constraints such as energy scheduling. Meanwhile, the environment evolves to a new state at the next time step according to the system's energy transfer model. As shown in the formula below:

[0135] Then, the current state, output action, reward, and new state are packaged and combined into a complete experience information. And store it in the experience replay pool. In the individual model training phase, periodically (such as every...) (Step) from the experience replay pool Experience data from batches is extracted, and this historical interaction data is used to calculate the time difference error to update the parameters of the Critic network's value function. Simultaneously, combined with the advantage function, the discrete action Actor network and the continuous action Actor network are driven to iteratively update their policy parameters. The specific process is as follows: using... Calculation of all empirical information and Then repeat from the experience replay pool. Extraction size is Update parameters based on small batches of experience information. and Finally, empty the experience replay pool. The data in the image. This concludes one episode. The algorithm requires multiple rounds to process the parameters of the discrete action Actor network. Parameters of continuous action Actor network Parameters of the Critic network The process will continue until the end of the update. The specific details are shown below for the HDRL algorithm based on a multi-source energy harvesting scheme, including the initialization process and multiple training steps.

[0136] Initialization: Discrete Action Actor Network Parameters Parameters of a continuous action Actor network Critic network parameters Experience replay pool Discount factor Small batch size Learning rate and .

[0137] 1: For episode do 2: Initialize the environment and obtain the initial state. ; 3: For do 4: State Input discrete action Actor networks and continuous action Actor networks, output action ; 5: Action-based Calculate the solutions to the inner subproblems (P5-a) and (P5-b) to obtain all optimization variables. ; 6: Calculate Rewards And calculate the state of the next time slot. ; 7: Translate a piece of experience information Stored in the experience replay pool middle; 8: End For 9: Use Each piece of empirical information in the calculation and ; 10: For epoch do 11: Disrupt Experience information; 12: For k do 13: From Extraction size is Small batch experience information Update parameters respectively and ; 14: End For 15: End For 16: Clear the experience replay pool Data in the middle; 17: End For The complexity of the HDRL algorithm based on a multi-source energy harvesting scheme mainly comes from the prediction and training phases. Assume the Critic network has... Layered, discrete action actor networks have Layers, continuous action Actor networks have Layers. The number of neurons in each layer of the Critic network is... The number of neurons in each layer of the discrete action Actor network is The number of neurons in each layer of the continuous action Actor network is .

[0138] The parameters are as follows: , , ; , , , ; , , , ; , , , .

[0139] Based on this, the prediction complexity of the HDRL algorithm based on the multi-source energy harvesting scheme is as follows: ; Training complexity arises from updating network parameters via backpropagation during neural network training, which involves different types of neurons, including ReLU layers and tanh layers. Assume the ReLU layer in the Critic network has the following number of neurons: The number of neurons in the tanh layer of the discrete action Actor network is The number of neurons in the ReLU layer of a continuous-action Actor network is Based on this, the training complexity of the HDRL algorithm based on a multi-source energy harvesting scheme can be expressed as follows: ; Step 300: Based on the optimized radio frequency energy collection duration, the current time frame is divided into a downlink energy collection phase and an uplink information transmission phase. During the downlink energy collection phase, the base station is controlled to send radio frequency energy signals to the smart reflector so that the smart reflector can collect energy according to the radio frequency energy signals. In the current time frame, the smart reflector collects ambient energy through the hybrid energy collection module.

[0140] Step 300 is described in detail below.

[0141] In step 300 of some embodiments, the current time frame refers to a periodic time unit in which the system performs a complete round of energy scheduling and data transmission operations, and the radio frequency energy signal refers to a high-frequency radio electromagnetic wave signal emitted by the base station to provide electromagnetic energy for the device to convert into DC power.

[0142] To address the issue of insufficient energy harvesting in average environments such as nighttime or inclement weather, leading to idle smart reflectors, this application employs a time-division multiplexing protocol to segment fixed time frames. Specifically, based on the duration parameters output by the optimization model, this application flexibly allocates RF charging time and communication service time. Within the allocated downlink energy harvesting phase, a mode switch that switches between RF energy harvesting mode and RIS operating mode is activated. Switch to 0, that is All reflective units switch to radio frequency energy harvesting mode. The system scheduler base station stops receiving data and instead focuses on transmitting radio frequency energy signals to the intelligent reflective surface. At this time, the energy harvesting circuit of the intelligent reflective surface receives the high-frequency electromagnetic wave and rectifies it into electrical energy, which is then promptly added to the local battery. This provides sufficient operating energy for subsequent active signal amplification operations, thereby compensating for the instability of a single environmental energy source.

[0143] In addition, throughout the current time frame, the smart reflector continuously captures natural environmental energy such as solar and wind energy from the external natural space to replenish its energy through its own configured hybrid energy harvesting module.

[0144] Step 400: During the uplink information transmission phase, the user terminal sends uplink data signals to the base station according to the optimized user transmit power control, and adjusts the reflection parameters of each subarray in the smart reflector according to the optimized reflection unit mode switching sequence, optimized reflection phase shift sequence, and optimized reflection amplification coefficient sequence.

[0145] Step 400 is described in detail below.

[0146] In step 400 of some embodiments, the uplink data signal refers to the actual service information stream generated by each user terminal that needs to be sent to the base station for demodulation and processing. The reflection parameters refer to the comprehensive physical configuration actually executed by each hardware unit of the intelligent reflector, including active / passive switching state, phase offset, and amplitude amplification factor. Auxiliary transmission refers to the communication enhancement process where the intelligent reflector provides additional coherent superimposed multipath signals to the direct communication link through intelligent control of the incident electromagnetic waves. When the uplink information transmission phase begins, the mode switch between the radio frequency energy harvesting mode and the RIS operating mode will be activated. Switch to 1, that is In the RIS (Reflector System) operating mode, the system controls each user terminal to send data to the base station at an optimized transmit power, thereby reducing interference to other user terminals within the system while ensuring its own transmission rate. Simultaneously, the intelligent reflector receives parameter sequences from the system to finely control the hardware subarray. For units configured as passive mode via a mode switch sequence, phase alignment is performed solely using a phase shift circuit; for units configured as active mode, phase shifting is performed while simultaneously invoking a shared power amplifier circuit to amplify weak signals.

[0147] During this phase, the intelligent reflector follows the energy causality constraint, using the sum of the initial remaining charge and the RF charge just collected in the downlink phase to maintain its phase shift and the operation of the amplification circuit. Thus, through the coordinated reflection and amplification of multiple units, the intelligent reflector can effectively compensate for the double fading of the uplink signal in spatial propagation, thereby improving data transmission efficiency. This will be described in more detail below.

[0148] The process involves adjusting the reflection parameters of each subarray in the smart reflective surface based on the optimized reflection unit mode switching sequence, the optimized reflection phase shift sequence, and the optimized reflection amplification coefficient sequence, including steps 410 to 430.

[0149] Step 410: Based on the optimized reflective unit mode switching sequence, control the mode switching switch of at least one reflective unit subarray to control the reflective unit to switch between active mode and passive mode.

[0150] Step 420: When the reflection unit is in active mode, the corresponding phase shift circuit is controlled to perform phase shift based on the optimized reflection phase shift sequence, and the power amplifier circuit is controlled to amplify the signal based on the optimized reflection amplification coefficient sequence.

[0151] Step 430: When the reflection unit is in passive mode, the phase shift circuit is controlled based on the optimized reflection phase shift sequence to perform phase shifting.

[0152] Steps 410 to 430 are described in detail below.

[0153] In step 410 of some embodiments, the present application scheme performs precise on / off control of the mode switching switch of each reflector on the smart reflector surface according to the discrete switch sequence output by the communication parameter optimization model. The optimized reflector mode switch sequence refers to a set of binary control states indicating whether each reflector is in a closed active mode or an open passive mode in a specific time slot. Specifically, when a specific control bit indicates a closed state, the reflector is connected to the shared amplification path of its subarray to enter active mode; conversely, if the switch is in an open state, the reflector only maintains basic phase adjustment functions. This allows for flexible adjustment of the ratio of active to passive units in different time slots based on real-time channel state information and battery energy state information, thereby achieving a dynamic balance between signal enhancement requirements and hardware power consumption control.

[0154] In step 420 of some embodiments, after the mode switching switch configures the reflection unit to enter active mode, the phase shift circuit adjusts the phase distribution of the reflected signal according to the optimized phase sequence to ensure that the reflected signal can be coherently superimposed with the direct signal or other path signals at the base station side. At the same time, the reflection unit in active mode uses the power amplifier circuit shared by the sub-array to amplify the incident signal according to the optimized output amplification factor, which can significantly alleviate the double fading effect generated by the wireless signal during transmission through the reflector, especially in long-distance or signal-blocked communication environments, and improve the received signal quality at the base station side.

[0155] In step 430 of some embodiments, when the reflector is configured in passive mode, the reflector mainly undertakes the low-power beam steering task. Its power consumption mainly comes from the static power consumption of the phase shift circuit, without having to bear the additional power overhead brought by the amplifier. This effectively extends the continuous working time of the self-powered system in complex energy environments by limiting the intervention of the active amplifier circuit.

[0156] Through steps 410 to 430 above, by jointly controlling the mode switching sequence, phase shift sequence and amplification factor of the reflection unit, the system can allocate active and passive reflection resources in real time according to the battery energy status and wireless environment characteristics. In this way, it can improve the long-term average throughput and service reliability of the self-powered communication system under energy-constrained conditions, while overcoming the dual fading loss of signal propagation in space.

[0157] In addition, the method of this application also includes steps 510 to 540.

[0158] Step 510: Based on the first channel state information, generate the optimized transmission beamforming vector for the base station.

[0159] Step 520: Based on the first channel state information, the second channel state information, the third channel state information, and the reflection parameters adjusted by the intelligent reflector, generate the optimized receiving beamforming vector of the base station.

[0160] Step 530: During the downlink energy harvesting phase, the base station is controlled to adjust the signal transmission direction according to the optimized transmission beamforming vector to transmit radio frequency energy signals to the smart reflector.

[0161] Step 540: During the uplink information transmission phase, the base station receives the uplink data signal sent by the user terminal according to the optimized receive beamforming vector control.

[0162] Steps 510 to 540 are described in detail below.

[0163] In step 510 of some embodiments, during the downlink energy harvesting phase, the primary task of the base station is to provide the smart reflector with as much radio frequency energy as possible. Since the energy transmission efficiency is highly dependent on the spatial channel characteristics between the base station and the smart reflector, the present application scheme utilizes the acquired first channel state information (i.e., the channel from the base station to the smart reflector) to perform analytical calculations using the inner-layer subproblem (P5-b) to obtain the optimized transmit beamforming vector corresponding to the antenna array weighting vector determined by the base station during the downlink energy harvesting phase to ensure that the received signal power at the smart reflector reaches a better level. Specifically, by utilizing the inner-layer subproblem (P5-b), the eigenvector corresponding to the largest eigenvalue of the covariance matrix of the channel matrix is ​​extracted and used as the base station's transmit beamforming vector. This ensures that the transmitted energy can be accurately focused on the aperture of the smart reflector antenna, effectively reducing spatial diffusion loss and maximizing the collection efficiency of radio frequency energy.

[0164] In step 520 of some embodiments, during the uplink information transmission phase, the signal received by the base station is a complex superposition of signals from multiple paths, including the user's direct connection signal, the enhanced signal reflected by the reflector, and accompanying amplified noise. This application's solution comprehensively utilizes the first, second, and third channel state information reflecting the entire link status, and combines this with the reflection matrix parameters currently being executed by the intelligent reflector to construct an equivalent cascaded channel model. Based on this model, the base station uses the minimum mean square error (MMSE) criterion to analytically solve the inner-layer subproblem (P5-a), generating an optimized receive beamforming vector that maximizes the user's received signal-to-interference-plus-noise ratio (SINR). The aim is to accurately separate the service data streams of each user terminal from the complex spatial wave field and suppress mutual interference between multiple users.

[0165] In step 530 of some embodiments, during the downlink energy harvesting phase, the base station invokes an optimized transmit beamforming vector to drive its antenna array to generate a highly directional radio frequency energy beam. By adjusting the signal transmission direction and aligning it with the smart reflector, the base station projects electromagnetic energy as a wireless wave onto the smart reflector, which is then rectified and converted by its energy harvesting circuit and stored in the battery. This provides the necessary power support for the smart reflector to perform active signal amplification in subsequent periods.

[0166] In step 540 of some embodiments, after entering the uplink information transmission stage, the base station switches to receiving mode and loads an optimized receive beamforming vector. This vector is used to weight the weak electromagnetic signals received by the antenna array, enabling the base station to selectively enhance the effective signal components from the intelligent reflector reflection path and the user direct connection path. Simultaneously, it suppresses the amplified noise introduced during active reflection in the spatial domain to improve the signal processing gain on the base station side, ensuring low bit error rate and high throughput performance for uplink data transmission in multi-user concurrent scenarios. It should be noted that step 530 and step 300 above are executed synchronously during the downlink energy harvesting phase; step 540 and step 400 above are executed collaboratively during the uplink information transmission phase, thus forming a complete closed loop for joint control of the space beam and reflector.

[0167] Through steps 510 to 540 above, by introducing bidirectional beamforming mechanisms optimized for energy harvesting and information transmission on the base station side, precise control of wireless space resources is achieved. Specifically, in the downlink phase, the maximum eigenvalue vector technology is used to achieve efficient energy replenishment, effectively alleviating the operational pressure of the self-powered intelligent reflector when the environment is short of energy. In the uplink phase, the MMSE receiving technology, which integrates the full-link channel characteristics and the dynamic parameters of the reflector, effectively cancels the amplified noise and multipath interference caused by active reflection, thereby improving the energy utilization efficiency of the system under complex energy constraints and enhancing the average throughput and link connection reliability of the multi-user communication system.

[0168] Through steps 100 to 400 above, by acquiring the channel fading status of the communication network and the battery energy of the intelligent reflector in real time, and dynamically planning each control variable according to the parameter optimization model, time frame resources can be reasonably allocated to introduce base station radio frequency signals to supplement the energy of the intelligent reflector when the environmental energy supply is scarce. At the same time, combined with the active / passive mode switch and fine control of reflection parameters in the intelligent reflector sub-connection architecture, an effective trade-off between radio frequency energy acquisition time and uplink service transmission time is achieved while maintaining low hardware power consumption. This overcomes the defect of existing communication systems that rely solely on environmental energy collection such as solar energy, which is easily affected by external condition fluctuations and thus shortens the effective working time. Under the premise of ensuring the all-weather self-sustaining operation of the intelligent reflector, the long-term average throughput and uplink data transmission reliability of the communication system in energy-constrained scenarios are improved.

[0169] To further verify the reliability of the parameter optimization method for the hybrid energy harvesting subarray communication system proposed in this application, simulation verification was performed in the embodiments of this application as described below.

[0170] During the simulation, the simulation parameters were set as follows: the base station was located at (0m, 0m, 3m), the RIS was located at (0m, 10m, 6m), and user terminals were randomly distributed on a circle with a radius of 5m centered at (300m, 0m, 0m) in the xy-plane. It was assumed that the RIS collected environmental energy... It conforms to a uniform distribution, that is The mean energy collected by RIS from the environment is Other parameters are shown in the table below: ; This application proposes a sub-connected hybrid RIS-assisted communication system scheme based on multi-source energy harvesting (i.e., a parameter optimization method for a hybrid energy harvesting subarray communication system). The scheme is compared with benchmark schemes. To evaluate the performance of the proposed hybrid communication scheme, computer simulations were performed to verify the performance of each benchmark scheme. All comparative benchmark schemes are shown below: Number of groups The hybrid RIS sub-connection scheme divides the RIS units into 8 groups, with each group sharing the same power amplifier. It adopts a hybrid mode, and all RIS units have a phase shift circuit that can freely switch between active and passive modes.

[0171] Hybrid RIS Fully Connected Solution: The RIS units are not grouped, each unit has a power amplifier and phase shift circuit, adopts a hybrid mode, and each unit can freely switch between active and passive modes.

[0172] Number of groups The active RIS sub-connection scheme divides the RIS units into 8 groups, with each group sharing the same power amplifier. It adopts active mode, and all units of the RIS have a phase shift circuit and cannot be switched to passive mode.

[0173] Active RIS Fully Connected Solution: The RIS units are not grouped, each unit has a power amplifier and phase shift circuit, and only active mode is available. No unit can be switched to passive mode.

[0174] Passive RIS scheme: The RIS unit is not grouped, each unit has a phase shift circuit, there is no power amplifier, and it can only reflect signals.

[0175] Reference Figure 6 This is a performance simulation diagram of the first method for parameter optimization of the hybrid energy harvesting subarray communication system provided in this application embodiment. For example... Figure 6 As shown, this represents the average energy collected from the environment. Base station transmission power and number of groups At that time, the number of different RIS units in the proposed hybrid RIS sub-connection scheme The convergence graph of the corresponding HDRL algorithm shows the relationship between round rewards and round number. Figure 6 The results show that the training algorithm in the parameter optimization method of the hybrid energy harvesting subarray communication system proposed in this application basically converged in the end, and the number of RIS units was also improved. Fewer solutions than RIS units The scheme with more RIS units converges faster because the number of RIS units increases. When there are fewer actions, the number of actions is also less, the algorithm complexity is lower, and reinforcement learning is more likely to find the optimal solution. On the other hand, from Figure 6 It can be observed that the round reward after final convergence increases with the number of RIS units. The increase in the number of RIS units shows a trend of first rising and then falling. from arrive As the number of RIS reflection units increases, the reflected signals can achieve stronger coherent superposition at the receiver, thereby enhancing the effective channel gain and improving the signal-to-interference-plus-noise ratio (SINR). Since throughput increases with increasing SINR, a higher number of RIS elements generally results in greater system throughput and therefore a larger round payoff. When the number of RIS units... from arrive As the number of RIS components increases, according to the RIS architecture proposed in this chapter, the number of phase shift circuits equals the number of RIS components. Therefore, the static power consumption of the RIS increases significantly, and the collected energy is insufficient to meet the normal operation requirements of the RIS. As a result, the average throughput decreases and the round return decreases.

[0176] Reference Figure 7 This is a second performance simulation diagram of the parameter optimization method for the hybrid energy harvesting subarray communication system provided in this application embodiment. For example... Figure 7 The figure shown is when the number of RIS units and number of groups At that time, the average harvesting environment energy in the proposed hybrid RIS sub-connection scheme was different. Corresponding average downlink RF energy harvesting time With base station transmission power The relationship. From Figure 7 The above observation shows that the average environmental energy harvested The more, the longer the average radio frequency energy harvesting time. The smaller the value, the better, because it means that the average amount of energy collected from the environment is reduced. The more [energy / resources] available, the more energy the RIS can run, and therefore the shorter the uplink transmission time. The larger the value, the longer the downlink average RF energy harvesting time. The smaller. On the other hand, it can be observed from the graph that when the average environmental energy collected... as well as At that time, with the base station's transmission power The increase in average radio frequency energy harvesting time Consequently, it decreases, and when the average environmental energy harvested... The rate of decline is faster. This is because when the average amount of energy collected from the environment is lower (…). , When the base station transmits power, the RIS is primarily powered by radio frequency energy harvesting. The amount of radio frequency energy that the RIS can collect is relatively small when the base station's transmit power is low. Limited energy resources make it difficult to meet the energy constraints required for its operation. Therefore, RIS reduces the uplink transmission time per time slot. To meet energy constraints, which means increasing the average downlink radio frequency energy harvesting time. When the base station transmit power When the value is large, the radio frequency energy collected by RIS Compared to base station transmission power More when the data is smaller, therefore RIS increases the uplink data transmission time. To increase communication throughput and average downlink RF energy harvesting time Fewer. Multi-source energy harvesting schemes incorporating radio frequency energy harvesting are compared to single-environment energy harvesting schemes (base station transmit power). Average downlink RF energy harvesting time The smaller size indicates the effectiveness of multi-source energy harvesting schemes. Furthermore, when the average environmental energy harvested is higher ( , Average downlink RF energy harvesting time With the base station's transmission power The changes have remained largely stable. This is because the average energy collected from the environment has remained relatively stable. In this case, the RIS mainly relies on harvesting ambient energy for power, so the base station's transmit power is relatively high. Average downlink RF energy harvesting time The impact is minimal.

[0177] Reference Figure 8 This is a third performance simulation diagram of the parameter optimization method for the hybrid energy harvesting subarray communication system provided in the embodiments of this application. Figure 8 The figure shown is when the number of RIS units and average environmental energy harvesting At that time, the number of groups proposed The hybrid RIS sub-connection scheme and four other benchmark schemes (hybrid RIS fully connected, number of groups) Average throughput and base station transmit power of active RIS sub-connection, active RIS full-connection, and passive RIS The relationship. By Figure 8 It can be seen that, overall, the average throughput increases with the base station's transmit power. The increase is due to the gradual increase in the average energy collected from the environment. At lower power levels, the system primarily relies on downlink power transmission from the base station to provide power support for the RIS. When the power signal transmitted by the base station... As the number of groups increases, the RIS can collect more radio frequency energy, thereby improving its signal amplification capability, enhancing the signal strength of the reflection link, and ultimately increasing the average throughput of the system. Specifically, in the proposed number of groups... In a hybrid RIS sub-connection scheme, when At that time, compared to (i.e., relying solely on ambient energy harvesting) resulted in a 151.8% improvement. This result demonstrates that introducing radio frequency (RF) energy harvesting can significantly improve system performance when ambient energy supply is insufficient. RF energy harvesting, as an effective supplement to ambient energy, can maintain the normal operation of the RIS under poor lighting conditions, thereby ensuring system performance stability. Furthermore, the proposed hybrid RIS sub-connection scheme exhibits superior performance compared to other benchmark schemes under various transmit power conditions. When the base station transmit power... Compared to the hybrid RIS fully connected scheme, the active RIS sub-connected scheme, the passive RIS scheme, and the active RIS fully connected scheme, the average throughput of the proposed hybrid RIS sub-connected scheme is improved by 14.47%, 17.35%, 100.27%, and 298.99%, respectively, demonstrating the superiority of the proposed hybrid RIS sub-connected scheme. Furthermore, it can be observed from the figure that the average throughput of the active RIS fully connected scheme is actually lower than that of the passive RIS scheme. This is because in the active RIS fully connected structure, each unit is equipped with a phase shift circuit and a power amplifier circuit, leading to a significant increase in system power consumption. Under energy constraints, excessive energy consumption will compress the effective energy used for information transmission, shorten the uplink information transmission time, and thus reduce the average throughput of the system. In contrast, the passive RIS can maintain more stable reflection performance under limited energy conditions, and therefore exhibits superior throughput performance in certain scenarios.

[0178] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in a memory, and the processor executes the at least one program to implement the parameter optimization method and transmission parameter optimization method of the hybrid energy harvesting subarray reflector communication system described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), in-vehicle computers, etc.

[0179] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the parameter optimization method and transmission parameter optimization method of the hybrid energy harvesting subarray reflector communication system of the embodiments of this application. The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0180] This application embodiment also provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the parameter optimization method and transmission parameter optimization method of the above-described hybrid energy harvesting subarray reflector communication system.

[0181] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0182] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0183] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0187] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A parameter optimization method for a hybrid energy harvesting subarray reflector communication system, characterized in that, The hybrid energy harvesting subarray reflector communication system includes a base station, a smart reflector, and multiple user terminals. The smart reflector is equipped with a hybrid energy harvesting module and multiple reflector subarrays, each of which includes multiple reflector subarrays. The method includes: In the current time frame, obtain the first channel state information between the base station and the smart reflector, the second channel state information between each user terminal and the smart reflector, and the third channel state information between each user terminal and the base station, and obtain the battery energy state information of the smart reflector. The first channel state information, the second channel state information, the third channel state information, and the battery energy state information are input into the communication parameter optimization model for data processing to obtain optimized combined parameters. The optimized combined parameters include optimized radio frequency energy harvesting duration, optimized user transmit power, optimized reflector mode switching sequence, optimized reflective phase shift sequence, and optimized reflective amplification coefficient sequence. Based on the optimized radio frequency energy harvesting duration, the current time frame is divided into a downlink energy harvesting phase and an uplink information transmission phase. During the downlink energy harvesting phase, the base station is controlled to send a radio frequency energy signal to the smart reflector, so that the smart reflector can harvest energy according to the radio frequency energy signal. In the current time frame, the smart reflector harvests ambient energy through the hybrid energy harvesting module. During the uplink information transmission phase, the user terminal is controlled to send uplink data signals to the base station according to the optimized user transmit power, and the reflection parameters of each subarray in the smart reflector are adjusted according to the optimized reflection unit mode switching sequence, the optimized reflection phase shift sequence, and the optimized reflection amplification coefficient sequence.

2. The parameter optimization method for the hybrid energy harvesting subarray reflector communication system according to claim 1, characterized in that, Each of the reflection unit subarrays in the intelligent reflective surface is provided with a power amplification circuit, and each reflection unit in the reflection unit subarray is correspondingly provided with a phase shift circuit and a mode switching switch. Adjusting the reflection parameters of each subarray in the intelligent reflective surface according to the optimized reflection unit mode switching sequence, the optimized reflection phase shift sequence, and the optimized reflection amplification coefficient sequence includes: Based on the optimized reflective unit mode switching sequence, control the mode switching switch of at least one reflective unit in the reflective unit subarray to control the reflective unit to switch between active mode and passive mode; When the reflection unit is in the active mode, the corresponding phase shift circuit is controlled to perform phase shift based on the optimized reflection phase shift sequence, and the power amplifier circuit is controlled to perform signal amplification based on the optimized reflection amplification coefficient sequence. When the reflection unit is in the passive mode, the phase shift circuit corresponding to the optimized reflection phase shift sequence is controlled to perform phase shifting.

3. The parameter optimization method for the hybrid energy harvesting subarray reflector communication system according to claim 1, characterized in that, The method further includes: Based on the first channel state information, an optimized transmit beamforming vector for the base station is generated; Based on the first channel state information, the second channel state information, the third channel state information, and the reflection parameters adjusted by the smart reflector, an optimized receiving beamforming vector for the base station is generated. During the downlink energy harvesting phase, the base station is controlled to adjust the signal transmission direction according to the optimized transmission beamforming vector in order to transmit the radio frequency energy signal to the smart reflector. During the uplink information transmission phase, the base station is controlled to receive the uplink data signal sent by the user terminal according to the optimized receive beamforming vector.

4. The parameter optimization method for the hybrid energy harvesting subarray reflector communication system according to claim 1, characterized in that, The steps for constructing the communication parameter optimization model include: Based on the static power and amplification power of the circuit of the intelligent reflector, a total power consumption model of the system is constructed, and battery energy dispatch constraints are constructed based on the radio frequency energy harvesting model and the environmental energy harvesting model. Based on the model that maximizes the average throughput of all user terminals within a preset observation period, and based on the average throughput model, the battery energy scheduling constraints, and the user service quality constraints, the system throughput maximization problem is obtained. The system throughput maximization problem is decomposed into an inner subproblem based on analytical expressions and an outer subproblem based on Markov decision processes. An initial hierarchical deep reinforcement learning model is constructed based on the inner and outer sub-problems, and the initial hierarchical deep reinforcement learning model is trained in multiple rounds to obtain the communication parameter optimization model.

5. The parameter optimization method for the hybrid energy harvesting subarray reflector communication system according to claim 4, characterized in that, The battery energy dispatch constraints constructed based on the radio frequency energy harvesting model and the environmental energy harvesting model include: Based on the input signal power, energy conversion efficiency, and preset saturation power threshold of the downlink energy harvesting stage, a radio frequency energy harvesting model with piecewise linear characteristics is obtained. Based on the cross-time domain characteristics of environmental energy harvesting, an environmental energy harvesting model with delayed activation characteristics is obtained; wherein, the environmental energy harvested in the current time frame is constrained to be used as battery energy increment only in subsequent time frames; Based on the radio frequency harvesting energy obtained from the radio frequency energy harvesting model, the initial battery energy in the current time frame, and the reflector operating power consumption determined by the optimized parameter combination, an energy causal constraint is constructed. The energy causal constraint is used to limit the energy consumed in the uplink information transmission phase to not be greater than the sum of the initial battery energy and the radio frequency harvesting energy. Based on the preset maximum battery capacity, the sum of the remaining energy determined by the energy causal constraint and the energy increment provided by the environmental energy harvesting model is subjected to capacity limiting processing to obtain the initial battery energy for the next time frame, so as to construct the battery energy scheduling constraint.

6. The parameter optimization method for the hybrid energy harvesting subarray reflector communication system according to claim 4, characterized in that, The inner sub-problem is used to output optimized transmit beamforming vectors and optimized receive beamforming vectors. The decomposition of the system throughput maximization problem into inner sub-problems based on analytical expressions includes: The channel covariance matrix is ​​obtained based on the first channel state information, and the optimized transmit beamforming vector is obtained based on the eigenvector corresponding to the largest eigenvalue in the channel covariance matrix. Based on the first channel state information, the second channel state information, the third channel state information, and the transmit power and reflection parameters output by the outer subproblem, a cascaded equivalent channel model is generated; Based on the minimum mean square error criterion, and according to the cascaded equivalent channel model and system noise power, the inner sub-problem corresponding to the optimization of the received beamforming vector is obtained.

7. The parameter optimization method for the hybrid energy harvesting subarray reflector communication system according to claim 4, characterized in that, The construction of the initial hierarchical deep reinforcement learning model based on the inner and outer sub-problems includes: Based on the first channel state information, the second channel state information, the third channel state information, and the battery energy state information, a state space is obtained; The action space is obtained based on the radio frequency energy harvesting duration, user transmit power, reflection unit mode switching sequence, reflection phase shift sequence, and reflection amplification coefficient sequence. Based on the output of the inner sub-problem and the instantaneous sum and rate of the action space calculation system, and according to the battery energy scheduling constraints and the user service quality constraints, a reward model is obtained; Obtain the initial network parameters of the actor network and the initial comment parameters of the commenter network, and generate the initial hierarchical deep reinforcement learning model based on the state space, the action space, and the reward model.

8. The parameter optimization method for the hybrid energy harvesting subarray reflector communication system according to claim 7, characterized in that, The actor network includes a discrete action network and a continuous action network, and the multi-round training of the initial hierarchical deep reinforcement learning model includes: In each training round, the current training state is input into the discrete action network for data processing to obtain classification logic values. The classification logic values ​​are then normalized to obtain a classification distribution. Finally, the updated reflection unit mode switch sequence is sampled from the classification distribution. The current training state is input into the output distribution parameters of the continuous action network. A beta distribution is constructed based on the distribution parameters. The original action variables are sampled from the beta distribution. The original action variables are linearly scaled based on the physical value range constraints to obtain the updated radio frequency energy harvesting duration, updated user transmit power, updated reflection phase shift sequence, and updated reflection amplification coefficient sequence. The updated beamforming vector is calculated based on the updated reflection unit mode switching sequence, the updated radio frequency energy harvesting duration, the updated user transmit power, the updated reflection phase shift sequence, the updated reflection amplification coefficient sequence, and the inner layer subproblem. The updated beamforming vector is then used to obtain the current reward value and the updated training state for the next time step, based on the updated beamforming vector and the reward model. The current training state, the current training action combination, the current reward value, and the updated training state are combined into training experience information and stored in the experience replay pool. The current training action combination includes the updated reflector mode switch sequence, the updated radio frequency energy harvesting duration, the updated user transmit power, the updated reflection phase shift sequence, and the updated reflection amplification coefficient sequence. Batch training experience information is extracted from the experience replay pool, and the time difference error is calculated using the reward value in the batch training experience information to update the value function of the commenter network and the initial comment parameters. The probability ratio of the new and old strategies is calculated based on the dominance function. The probability ratio is truncated based on a preset truncation hyperparameter, and the initial action parameters of the actor network are updated using gradient descent.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the parameter optimization method for the hybrid energy harvesting subarray reflector communication system according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the parameter optimization method for the hybrid energy harvesting subarray reflector communication system according to any one of claims 1 to 8.

Images