Energy capture mobile heterogeneous network optimization method based on information age constraint

Abstract

The invention discloses an energy capture mobile heterogeneous network optimization method based on information age constraint, and the method comprises the steps: building a corresponding digital twin layer for a physical entity layer comprising a base station, a hybrid access point, a transmitter and a receiver, and carrying out the control of a time slot in a control stage of the time slot, a transmitter mode selection and channel allocation strategy is obtained through the trained first deep reinforcement learning network, a transmitter power control and energy capture time strategy is obtained through the trained second deep reinforcement learning network, the strategies are sent to the digital twinborn layers, the digital twinborn layers are combined into a complete mode selection and resource allocation strategy, and the mode selection and resource allocation strategy is sent to the digital twinborn layers. And the transmitter works according to the information. According to the method, more efficient state information exchange and acquisition are realized, actions of intelligent agents in deep reinforcement learning are coordinated, competition for priority network resources is reduced, and power control and wireless energy capture time are optimized.

Description

Technical Field

[0001] The present application belongs to the field of wireless transmission technology, and in particular, relates to an energy capture mobile heterogeneous network optimization method based on information age constraints. Background Art

[0002] With the rapid development of the Internet of Things and communication technologies, the number of devices and data traffic are exploding in emerging applications and service scenarios such as smart cities and autonomous driving. The energy limitations of devices cannot be ignored. When a large number of devices are deployed in places that are not suitable for battery charging and replacement, the limited life of the devices will degrade network performance. In order to meet the growing data traffic requirements of emerging wireless communication services while providing high-quality services to devices, new technologies are needed to improve network performance.

[0003] To improve network performance, a heterogeneous network structure can be used to further improve data flow and network performance by deploying multiple small base stations in the coverage area of ​​the macro base station. In addition, in order to increase the energy supply of the device, the devices all use wireless energy transmission technology based on radio frequency signals. The devices in the network can capture the energy of radio frequency signals and store it in batteries. In addition, the device-to-device (D2D) communication mode enables devices with close distances to communicate directly with each other without relays. Due to the short transmission distance, the device can use a lower transmission power to reduce interference with other devices and improve network performance.

[0004] On the other hand, real-time applications and services in the network have strict requirements on data latency. The data arriving at the destination must be as fresh as possible. The age of information (AoI) can measure the freshness of the data and is defined as the time elapsed since the data most recently received by the destination was generated. Therefore, efficient mode selection and resource allocation are crucial to improve the long-term total throughput of the network while meeting the AoI constraints of the device.

[0005] In order to cope with the challenges of long-term performance optimization and device mobility in energy-harvesting mobile heterogeneous networks while meeting the device information age constraint, it is crucial to implement efficient mode selection and resource allocation. In dynamic and complex network scenarios with limited network resources, it is of great significance to implement efficient mode selection and resource allocation to improve the long-term total network throughput while meeting the device AoI constraint. Summary of the invention

[0006] The purpose of this application is to provide an energy capture mobile heterogeneous network optimization method based on information age constraints, so that devices can reasonably select modes and utilize limited network resources in dynamic and complex networks to improve the long-term total throughput of energy capture mobile heterogeneous networks while satisfying the device information age AoI constraints.

[0007] In order to achieve the above purpose, the technical solution of this application is as follows: An energy capture mobile heterogeneous network optimization method based on information age constraint, comprising: Establish a corresponding digital twin layer for the physical entity layer including base stations, hybrid access points, transmitters and receivers; In the control phase of the time slot, the transmitter and receiver move, and the transmitter inputs the local state information into the trained first deep reinforcement learning network to obtain the corresponding mode selection and channel allocation strategy, which is sent to the digital twin layer; The base station inputs the global state information and the transmitter’s mode selection and channel allocation strategy into the trained second deep reinforcement learning network, obtains the transmitter’s power control and energy capture time strategy, and sends it to the digital twin layer; The digital twin layer merges the transmitter’s mode selection and channel allocation strategy with the transmitter’s power control and energy capture time strategy into a complete mode selection and resource allocation strategy, and synchronizes it to the transmitter; In the wireless energy transmission stage, the transmitter performs energy capture according to the energy capture time, and in the wireless data transmission stage, the transmitter performs data transmission according to mode selection, channel allocation and power control.

[0008] Furthermore, the digital twin layer is deployed at the base station.

[0009] Furthermore, the local status information includes: The position coordinates of the transmitter after movement in the current time slot, the position coordinates of the receiver after movement in the current time slot, the ratio of the available energy of the transmitter to the battery capacity at the beginning of the current time slot, the information age of the transmitter in the current time slot, a binary variable indicating whether the transmitter generates data in the current time slot, the channel gain between the transmitter and the associated base station or hybrid access point after movement in the current time slot, the channel gain between the transmitter after movement in the current time slot and the receiver after movement, the throughput of the transmitter in the previous time slot, and the channel allocation of the transmitter in the previous time slot.

[0010] Furthermore, the global status information includes: The ratio of the transmitter's available energy to the battery capacity at the beginning of the current time slot, the transmitter's information age in the current time slot, and the transmitter's throughput in the previous time slot.

[0011] Furthermore, the energy capture mobile heterogeneous network optimization method based on information age constraint further includes: Training the first deep reinforcement learning network and the second deep reinforcement learning network; The training of the first deep reinforcement learning network and the second deep reinforcement learning network comprises: Step F1, initializing the local Q network and the target Q network corresponding to the first deep reinforcement learning network, and initializing the parameters of the current action network, the target action network, the first current value network, the first target value network, the second current value network, and the second target value network corresponding to the second deep reinforcement learning network; Step F2, initializing the hybrid access point, transmitter and receiver; Step F3: The transmitter and the receiver move to update the local state information of the current time slot in the digital twin layer; Step F4: The transmitter sends the local status information to the local Q network, obtains the corresponding mode selection and channel allocation strategy, and sends it to the digital twin layer; Step F5: The base station receives the global state information and the mode selection and channel allocation strategy of the transmitter in the digital twin layer, inputs the current action network, obtains the power control and energy capture time strategy of the transmitter, and sends it to the digital twin layer; Step F6: The digital twin layer combines the transmitter's mode selection and channel allocation strategy with the transmitter's power control and energy capture time strategy into a complete mode selection and resource allocation strategy, and synchronizes it to the transmitter; Step F7, the transmitter executes the complete mode selection and resource allocation strategy; Step F8, the transmitter and the base station calculate corresponding rewards; Step F9: The transmitter and the base station respectively store their own experiences in their own experience playback pools and randomly extract a preset number of experiences; Step F10, the transmitter updates the parameters of the local Q network according to the extracted experience; Step F11: if the result of the current iteration number modulo 2 is not equal to 0, no operation is performed; otherwise, the parameters of the target Q network are updated using the parameters of the local Q network; Step F12: The base station updates the parameters of the first current value network and the second current value network according to the extracted experience; Step F13: If the result of the current iteration number modulo 2 is not equal to 0, no operation is performed; otherwise, the current action network and all target network parameters are updated according to the extracted experience; Step F14: If the current time slot is less than the total number of time slots, the next time slot is used as the current time slot and the process returns to step F3; Step F15: If the current number of iterations is less than the total number of iterations, return to step F2 to continue iterative training; otherwise, end the training.

[0012] Furthermore, the transmitter's own experience includes the transmitter's local state information, mode selection and channel allocation strategy, the reward obtained by the transmitter, and the transmitter's local state information in the next time slot.

[0013] Furthermore, the base station's own experience includes global state information, transmitter mode selection and channel allocation strategy, transmitter power control and energy capture time strategy, rewards obtained by the base station, next time slot global state information and transmitter mode selection and channel allocation strategy.

[0014] The energy capture mobile heterogeneous network optimization method based on information age constraint proposed in this application has the following significant advantages compared with the prior art: This application integrates digital twin technology into the state information exchange and acquisition process in the training of a complex deep reinforcement learning framework. The twin communication between the digital twin layer and the physical entity layer realizes more efficient state information exchange and acquisition, as well as the implementation of mode selection and resource allocation in the physical entity layer. At the same time, in order to further improve the performance of deep reinforcement learning for action output and avoid the problem of algorithm convergence caused by the huge state-action space, this application adopts the first deep reinforcement learning to optimize mode selection and channel allocation. The first deep reinforcement learning will cause the agents to compete with each other for limited network resources. In order to coordinate the actions of the agents, this application incorporates the second deep reinforcement learning, which plays a role in coordinating the actions of the agents to reduce competition through global state information and accepting the output of the first deep reinforcement learning as its own state information, while optimizing power control and wireless energy capture time.

[0015] This application implements reasonable mode selection and resource allocation strategies through deep reinforcement learning, and improves the long-term total network throughput through training while meeting the device AoI constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the energy capture mobile heterogeneous network optimization method based on information age constraints in this application.

[0017] Figure 2 Schematic diagram of the energy capture mobile heterogeneous network structure for this application.

[0018] Figure 3 This is a schematic diagram of the time slot structure of this application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] In one embodiment, Figure 1As shown, an energy-harvesting mobile heterogeneous network optimization method based on information age constraints is provided, which uses deep reinforcement learning to enable devices to reasonably select modes and utilize limited network resources in dynamic and complex networks to maximize the long-term throughput that meets the device AoI constraints.

[0021] The energy capture mobile heterogeneous network optimization method based on information age constraints includes: Step S1: Establish a corresponding digital twin layer for the physical entity layer including base stations, hybrid access points, transmitters and receivers.

[0022] The energy capture mobile heterogeneous network optimization method based on information age constraint provided in this application can be applied to Figure 2 The energy capture mobile heterogeneous network includes a physical layer and a digital twin layer, where the physical layer includes a multi-antenna base station, Multi-antenna hybrid access points and D2D device pairs (D2D pairs for short), each D2D pair includes a D2D transmitter (MDT) and a D2D receiver (MDR), i.e., a transmitter and a receiver, where the base station and the hybrid access point can send RF signals for the transmitter to capture energy, where the transmitter can select the relay mode or the D2D mode to transmit data to the corresponding receiver. The wireless channel can be divided into two modes for relay mode and D2D mode. channels of equal bandwidth.

[0023] The digital twin layer establishes a mapping between the physical entities in the physical entity layer and the corresponding digital twins in the digital twin layer. It is used to store the status information of the physical entities and exchange real-time status information. The digital twin layer is located in the digital twin server and deployed in the base station. Figure 2 Next to the central base station is the digital twin server.

[0024] Each transmitter in the network is equipped with a data buffer and generates data with a certain probability in each time slot. If the transmitter generates data in the current time slot, the data is stored in the data buffer and the data that has not been transmitted before is discarded. Otherwise, the untransmitted data is still stored in the data buffer. When the data in the data buffer is successfully transmitted, the AoI of the transmitter is reduced to the number of time slots that have passed since the successful data transmission was generated, and the throughput achieved by the transmitter is included in the total network throughput of the current time slot. When the data transmission is unsuccessful, the AoI of the transmitter increases by 1, and the throughput achieved by the transmitter is not included in the total network throughput of the current time slot.

[0025] Step S2: In the control phase of the time slot, the transmitter and the receiver move, and the transmitter inputs the local state information into the trained first deep reinforcement learning network to obtain the corresponding mode selection and channel allocation strategy, and sends it to the digital twin layer.

[0026] like Figure 3 As shown, each time slot includes a control phase t0, a wireless energy transmission phase t1(k) and a wireless data transmission phase t2(k).

[0027] In the network, all hybrid access points, transmitters and receivers obey a static Poisson point distribution. The transmitters and receivers move in the control phase according to a Gaussian-Markov random model.

[0028] In time slot ,transmitter Speed and direction Updated to: in It is a constant used to adjust the effect of the previous time slot speed on the current time slot speed. It is a constant used to adjust the effect of the previous time slot direction on the current time slot direction. is the average velocity of the transmitter and receiver, It is a transmitter The average direction of The mean-variance pair is Independent Gaussian distribution, The mean-variance pair is Independent Gaussian distribution of In time slot ,transmitter Location coordinates Updated to: in is the movement time of the transmitter and receiver during the control phase.

[0029] The digital twin layer and the physical entity layer perform twin communication, and the state information stored in the digital twin corresponding to the physical entity in the digital twin layer is updated. Each transmitter obtains the corresponding mode selection and channel allocation strategy by inputting the local state information into the trained first deep reinforcement learning network, and sends it to the digital twin layer through twin communication.

[0030] In time slot ,transmitter The local state information is defined as: in and It is a time slot transmitter The position coordinates after moving; and It is a time slot Receiver The location coordinates of It is a time slot Start transmitter The ratio of available energy to battery capacity, It is a time slot transmitter AoI; It is a time slot transmitter whether to generate binary variables for data, Indicates time slot transmitter Generate data, Indicates time slot transmitter No data is generated; It is a time slot Transmitter after moving The channel gain between the associated base station or hybrid access point, It is a time slot Transmitter after moving After moving the receiver The channel gain between It is a time slot The throughput of the transmitter, It is a time slot Channel assignment for transmitters.

[0031] In time slot ,transmitter The mode selection and channel allocation strategy is defined as: in It is a time slot transmitter Mode selection, Indicates time slot transmitter Select D2D mode, Indicates time slot transmitter Select the relay mode. It is a time slot transmitter The assigned channel number.

[0032] Step S3: The base station inputs the global state information and the mode selection and channel allocation strategy of the transmitter into the trained second deep reinforcement learning network, obtains the power control and energy capture time strategy of the transmitter, and sends it to the digital twin layer.

[0033] The transmitter and receiver remain stationary during the wireless energy transmission phase and the wireless data transmission phase of each time slot.

[0034] The base station sends global status information and The mode selection and channel allocation strategy of each transmitter is input into the trained second deep reinforcement learning network to obtain The power control and energy capture time strategy of each transmitter is determined and sent to the digital twin layer through twin communication.

[0035] In time slot , global status information and The mode selection and channel allocation strategy for each transmitter is defined as: in It is a time slot the channel gain between the transmitter after the move and the receiver determined by the mode selection obtained by the first deep reinforcement learning network (corresponding to the mode selection), It is a time slot According to the channel allocation obtained by the first deep reinforcement learning network (corresponding to the channel allocation strategy), It is a time slot The ratio of the transmitter's available energy to the battery capacity at the start, It is a time slot AoI of the transmitter, It is a time slot The throughput of the transmitter.

[0036] Step S4: The digital twin layer merges the transmitter's mode selection and channel allocation strategy with the transmitter's power control and energy capture time strategy into a complete mode selection and resource allocation strategy, and synchronizes it to the transmitter.

[0037] The digital twin layer will Mode selection and channel allocation strategies for each transmitter and The power control and energy capture timing strategies of each transmitter are combined into a complete mode selection and resource allocation strategy and synchronized to transmitter.

[0038] In time slot , The mode selection and channel allocation strategy for each transmitter is defined as: In time slot , The power control and energy capture time strategy of each transmitter is defined as: in It is a time slot transmitter Energy consumed by data transmission and time slot The wireless data transmission phase begins when the transmitter The ratio of available energy to It is a time slot The ratio of the transmitter energy capture time to the sum of the wireless energy transmission phase and the wireless data transmission phase.

[0039] In time slot , the digital twin layer will Mode selection and channel allocation strategies for each transmitter and The power control and energy capture time strategies of each transmitter are combined into a complete mode selection and resource allocation strategy, which is defined as: .

[0040] Step S5: In the wireless energy transmission stage, the transmitter performs energy capture according to the energy capture time, and in the wireless data transmission stage, the transmitter performs data transmission according to mode selection, channel allocation and power control.

[0041] The transmitter and receiver remain stationary during the wireless energy transmission phase and the wireless data transmission phase of each time slot.

[0042] In the wireless energy transmission stage, Each transmitter performs energy capture according to the energy capture time obtained from the complete mode selection and resource allocation strategy; In the wireless data transmission stage, Each transmitter performs corresponding data transmission according to the mode selection, channel allocation and power control obtained from the analysis of the complete mode selection and resource allocation strategy.

[0043] In another embodiment, a training process of deep reinforcement learning is described, that is, training a first deep reinforcement learning and a second deep reinforcement learning, the training process comprising the following steps: Step F1, initialize the local Q network and target Q network corresponding to the first deep reinforcement learning network, and initialize the parameters of the current action network, target action network, first current value network, first target value network, second current value network and second target value network corresponding to the second deep reinforcement learning network.

[0044] Specifically, the network parameters of each agent in the first deep reinforcement learning network are initialized, including the parameters The local Q network and parameters are The target Q network initializes the parameters of the second deep reinforcement learning network, including the parameters The current action network and parameters are The target action network, parameters are The first current value network, the parameters are The first target value network is The second current value network and parameters are Second target value network, initialize the total training iterations , initialize the total number of time slots per round , set the iteration round .

[0045] Step F2: Initialize the hybrid access point, transmitter and receiver.

[0046] Initialize the hybrid access point, transmitter and mobile receiver distribution in the physical layer, the available energy in the transmitter battery is 0, the throughput is 0, the AoI is 1 and the data is generated according to the probability, setting the time slot in the iteration round .

[0047] Step F3: The transmitter and the receiver move to update the local state information of the current time slot in the digital twin layer.

[0048] The transmitters and receivers move according to the Gauss-Markov random model, the digital twin layer and the physical entity layer perform twin communication, and the state information stored in the digital twin corresponding to the physical entity in the digital twin layer is updated.

[0049] Each transmitter receives the local state information of the current time slot stored by its own digital twin from the digital twin layer .

[0050] Step F4: The transmitter sends the local status information to the local Q network, obtains the corresponding mode selection and channel allocation strategy, and sends it to the digital twin layer.

[0051] Each transmitter will As the input of the local Q network, the corresponding action is output , i.e., mode selection and channel allocation strategy, and sent to the digital twin layer through twin communication, which stores Mode selection and channel allocation strategy for each transmitter.

[0052] Step F5: The base station receives the global state information and the mode selection and channel allocation strategy of the transmitter in the digital twin layer, inputs the current action network, obtains the power control and energy capture time strategy of the transmitter, and sends it to the digital twin layer.

[0053] The base station receives the global state information stored by the corresponding digital twin in the digital twin layer and Mode selection and channel allocation strategy for each transmitter , the base station will As the input of the current action network, the output action , i.e., the power control and energy capture time strategy of the transmitter, and is sent to the digital twin layer through twin communication.

[0054] Step F6: The digital twin layer merges the transmitter's mode selection and channel allocation strategy with the transmitter's power control and energy capture time strategy into a complete mode selection and resource allocation strategy, and synchronizes it to the transmitter.

[0055] The digital twin layer will and Merge into a complete mode selection and resource allocation strategy And synchronize to transmitter.

[0056] Step F7: The transmitter executes the complete mode selection and resource allocation strategy.

[0057] The transmitter receives After that, the data is parsed to obtain the mode selection and resource allocation strategy related to itself. The transmitter is based on The energy capture time in the time slot Energy is captured during the wireless energy transmission phase. In the wireless data transmission phase, data is transmitted based on the strategy of mode selection, channel allocation and power control.

[0058] Step F8: The transmitter and the base station calculate corresponding rewards.

[0059] The transmitter and the base station are in the time slot The control phase obtains the corresponding status information through twin communication, and the transmitter According to the time slot The throughput and AoI achieved are used to calculate the corresponding rewards ,in The calculation formula is: in is the throughput weight, It is a time slot transmitter The throughput achieved, is the AoI weight, It is a time slot transmitter AoI-related penalties. transmitter When the AoI is less than the maximum tolerance threshold, ,otherwise, ,in A positive constant of equilibrium magnitude, Maximum tolerance threshold; base station based on The corresponding reward is calculated based on the reward of each transmitter. ,in The calculation formula is:

[0060] .

[0061] Step F9: The transmitter and the base station respectively store their own experiences in their own experience playback pools and randomly extract a preset number of experiences.

[0062] Each transmitter will time slot Own experience Stored in its own experience replay pool and randomly drawn Experience, including It is a time slot transmitter Local status information, It is a time slot for Mode selection and channel allocation strategy, It is a time slot transmitter The rewards you get, It is a time slot transmitter Local status information.

[0063] The base station sets the time slot Experience Stored in its own experience replay pool and randomly drawn Experience, including It is a time slot Global status information and mode selection and channel allocation strategy for each transmitter, It is a time slot for of Power control and energy capture timing strategies for each transmitter, It is a time slot The base station gets the reward, It is a time slot Global status information and Mode selection and channel allocation strategy for each transmitter.

[0064] Step F10: The transmitter updates the parameters of the local Q network according to the extracted experience.

[0065] Each transmitter sends a randomly selected The local Q network parameters are updated according to formulas (1)-(3) based on the experience.

[0066] in It is a transmitter Experience The target Q value for the corresponding time slot, is the discount factor, It is to make experience Next time slot transmitter corresponding to the time slot The Q network estimates the action with the largest Q value, It is a transmitter Experience The state information of the next time slot corresponding to the time slot and the estimated Q value of the target Q network output corresponding to the action, It is a transmitter The loss function of the Q network is, is the learning rate, express right Seeking guidance, Indicates the transmitter Randomly extract The first experience experience.

[0067] Step F11: If the result of taking the current number of iterations modulo 2 is not equal to 0, no operation is performed; otherwise, the parameters of the target Q network are updated using the parameters of the local Q network.

[0068] If the current number of iterations If the result of modulo 2 is not equal to 0, no operation is performed; otherwise, the transmitter Update the target Q network parameters to .

[0069] Step F12: The base station updates the parameters of the first current value network and the second current value network according to the extracted experience.

[0070] The base station randomly selects The current value network parameters are updated based on the experience according to formulas (4)-(8).

[0071] in Base station experience The smaller value of the time difference error of the first and second current values ​​of the corresponding time slot network, It’s experience The next time slot status information of the corresponding time slot The action value of the target value network (j equals 1 for the first target value network, j equals 2 for the second target value network), and are the loss functions of the first and second current value networks, respectively, It’s experience The state information of the corresponding time slot is the action value of the current value network, It’s experience The state information of the corresponding time slot, the action value of the second current value network, express right Seeking guidance, express right Seeking guidance, Indicates that the base station is randomly selected The first experience experience.

[0072] Step F13: If the result of the current iteration number modulo 2 is not equal to 0, no operation is performed; otherwise, the current action network and all target network parameters are updated according to the extracted experience.

[0073] If the current number of iterations If the result of modulo 2 is not equal to 0, no operation is performed; otherwise, the base station selected The experience is used to update the current action network and all target network parameters according to formulas (9)-(13).

[0074] in is the soft update factor of the action network, It is the soft update factor of the value network.

[0075] Step F14: If the current time slot is less than the total number of time slots, the next time slot is used as the current time slot and the process returns to step F3.

[0076] If the current time slot , K represents the total number of time slots, then , and jump to step F3.

[0077] Step F15: If the current number of iterations is less than the total number of iterations, return to step F2 to continue iterative training; otherwise, end the training.

[0078] like and One iteration is over. , and jump to step F2; if and , the training is over.

[0079] It should be noted that after the training, the parameters of the local Q network obtained are the parameters of the trained first deep reinforcement learning network, and the current action network parameters obtained are the parameters of the second deep reinforcement learning network, which are used for energy capture mobile heterogeneous network optimization based on information age constraints.

[0080] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An energy capture mobile heterogeneous network optimization method based on information age constraints, characterized in that: The energy capture mobile heterogeneous network optimization method based on information age constraints includes: Establish a corresponding digital twin layer for the physical entity layer including base stations, hybrid access points, transmitters and receivers; In the control phase of the time slot, the transmitter and receiver move, and the transmitter inputs the local state information into the trained first deep reinforcement learning network to obtain the corresponding mode selection and channel allocation strategy, which is sent to the digital twin layer; The base station inputs the global state information and the transmitter’s mode selection and channel allocation strategy into the trained second deep reinforcement learning network, obtains the transmitter’s power control and energy capture time strategy, and sends it to the digital twin layer; The digital twin layer merges the transmitter’s mode selection and channel allocation strategy with the transmitter’s power control and energy capture time strategy into a complete mode selection and resource allocation strategy, and synchronizes it to the transmitter; In the wireless energy transmission stage, the transmitter performs energy capture according to the energy capture time, and in the wireless data transmission stage, the transmitter performs data transmission according to mode selection, channel allocation and power control.

2. The method for optimizing mobile heterogeneous networks based on energy capture based on information age constraints according to claim 1 is characterized in that: The digital twin layer is deployed at the base station.

3. The method for optimizing mobile heterogeneous networks based on energy capture based on information age constraints according to claim 1 is characterized in that: The local status information includes: The position coordinates of the transmitter after movement in the current time slot, the position coordinates of the receiver after movement in the current time slot, the ratio of the available energy of the transmitter to the battery capacity at the beginning of the current time slot, the information age of the transmitter in the current time slot, a binary variable indicating whether the transmitter generates data in the current time slot, the channel gain between the transmitter and the associated base station or hybrid access point after movement in the current time slot, the channel gain between the transmitter after movement in the current time slot and the receiver after movement, the throughput of the transmitter in the previous time slot, and the channel allocation of the transmitter in the previous time slot.

4. The method for optimizing mobile heterogeneous networks based on energy capture based on information age constraints according to claim 1 is characterized in that: The global status information includes: The ratio of the transmitter's available energy to the battery capacity at the beginning of the current time slot, the transmitter's information age in the current time slot, and the transmitter's throughput in the previous time slot.

5. The method for optimizing mobile heterogeneous networks based on energy capture based on information age constraints according to claim 1 is characterized in that: The energy capture mobile heterogeneous network optimization method based on information age constraints also includes: Training the first deep reinforcement learning network and the second deep reinforcement learning network; The training of the first deep reinforcement learning network and the second deep reinforcement learning network comprises: Step F1, initializing the local Q network and the target Q network corresponding to the first deep reinforcement learning network, and initializing the parameters of the current action network, the target action network, the first current value network, the first target value network, the second current value network, and the second target value network corresponding to the second deep reinforcement learning network; Step F2, initializing the hybrid access point, transmitter and receiver; Step F3: The transmitter and the receiver move to update the local state information of the current time slot in the digital twin layer; Step F4: The transmitter sends the local status information to the local Q network, obtains the corresponding mode selection and channel allocation strategy, and sends it to the digital twin layer; Step F5: The base station receives the global state information and the mode selection and channel allocation strategy of the transmitter in the digital twin layer, inputs the current action network, obtains the power control and energy capture time strategy of the transmitter, and sends it to the digital twin layer; Step F6: The digital twin layer combines the transmitter's mode selection and channel allocation strategy with the transmitter's power control and energy capture time strategy into a complete mode selection and resource allocation strategy, and synchronizes it to the transmitter; Step F7, the transmitter executes the complete mode selection and resource allocation strategy; Step F8, the transmitter and the base station calculate corresponding rewards; Step F9: The transmitter and the base station respectively store their own experiences in their own experience playback pools and randomly extract a preset number of experiences; Step F10, the transmitter updates the parameters of the local Q network according to the extracted experience; Step F11: if the result of the current iteration number modulo 2 is not equal to 0, no operation is performed; otherwise, the parameters of the target Q network are updated using the parameters of the local Q network; Step F12: The base station updates the parameters of the first current value network and the second current value network according to the extracted experience; Step F13: If the result of the current iteration number modulo 2 is not equal to 0, no operation is performed; otherwise, the current action network and all target network parameters are updated according to the extracted experience; Step F14: If the current time slot is less than the total number of time slots, the next time slot is used as the current time slot and the process returns to step F3; Step F15: If the current number of iterations is less than the total number of iterations, return to step F2 to continue iterative training; otherwise, end the training.

6. The method for optimizing mobile heterogeneous networks based on energy capture based on information age constraints according to claim 5 is characterized in that: The transmitter's own experience includes the transmitter's local state information, mode selection and channel allocation strategy, the reward obtained by the transmitter, and the transmitter's local state information for the next time slot.

7. The method for optimizing mobile heterogeneous networks based on energy capture based on information age constraints according to claim 5 is characterized in that: The base station's own experience includes global state information, transmitter mode selection and channel allocation strategy, transmitter power control and energy capture time strategy, base station rewards, next time slot global state information and transmitter mode selection and channel allocation strategy.

Images