Vertical handover method for sky-air-ground integrated cyber-physical system

Abstract

Disclosed in the present invention is a vertical handover method for a sky-air-ground integrated cyber-physical system. The method comprises: a base station acquiring state spaces sent by mobile devices; in the base station, using a constrained proximal policy optimization algorithm to obtain gradient values of vertical handover policy networks and gradient values of value networks, and sending the gradient values to a central learner; determining in the central learner a first average gradient value and a second average gradient value, determining updated parameters of the vertical handover policy networks and updated parameters of the value networks, and sending the updated parameters to the mobile devices; and in the mobile devices, when the cumulative value of all reward values is greater than or equal to a target value and the cumulative value of the reward values is stable, enabling finally obtained updated vertical handover policy networks to output final policies. The vertical handover method for a sky-air-ground integrated cyber-physical system provided in the present invention can be directly used for the vertical handover of a sky-air-ground integrated cyber-physical system.

Description

A vertical switching method for an integrated air-space-ground cyber-physical system Technical Field

[0001] This invention belongs to the field of vertical switching technology for heterogeneous networks, specifically relating to a vertical switching method for an integrated air-space-ground cyber-physical system. Background Technology

[0002] Cyber-Physical Systems (CPS) tightly connect resources, information, objects, and people in the manufacturing process, transforming the production plant into a smart environment. CPS is a crucial foundation for Industry 4.0. Essentially, a CPS is a networked control system. Its main components include a dynamically controlled object, a sensor for monitoring the system's state, an actuator for changing the controlled object's state, and a controller for generating control decisions. The communication network is responsible for sending the controlled object's state information to the controller and sending control commands from the controller to the actuator. This creates a continuously iterating control loop of "controlled object → sensor → controller → actuator (changing the controlled object's state)," promoting the integration of cyber and physical systems.

[0003] In recent years, with the continuous expansion of manufacturing scale, Cyber-Physical Systems (CPS) for emerging industrial manufacturing have gradually emerged, such as smart factories, forest emergency response, and intelligent maritime transport. The geographical deployment of CPS components has gradually extended from traditional land to forests, oceans, and even space—areas difficult for terrestrial networks to cover. Therefore, it is necessary to introduce an integrated mobile communication network to enable devices to access the network anytime and anywhere across a wide area, ensuring the stable operation of CPS. The Sky-Air-Ground Integrated Network (SAGIN) seamlessly integrates satellite networks, airborne base station networks, and terrestrial base station networks, establishing seamless communication coverage across multiple domains and dimensions, making it a suitable candidate network for carrying emerging CPS services. To use SAGIN as the communication network connecting various CPS components and achieve stable operation of the Sky-Air-Ground Integrated Cyber-Physical System, a key technology—vertical handover methods between heterogeneous air-space-ground networks—is required.

[0004] However, current vertical handover methods for integrated air-space-ground networks cannot be directly used for vertical handover in integrated air-space-ground cyber-physical systems.

[0005] Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a vertical switching method for an integrated air-space-ground cyber-physical system.

[0007] The technical problem to be solved by this invention is achieved through the following technical solution:

[0008] This invention provides a vertical handover method for an integrated air-space-ground cyber-physical system, the vertical handover method comprising:

[0009] Step 1: The base station obtains the state space sent by the mobile device, wherein the state space includes communication state and behavior state, Q randomly deployed mobile devices communicate with one base station respectively, each base station communicates with q mobile devices simultaneously, and each mobile device is equipped with a vertical handover policy network and a value network, 0 < q ≤ Q;

[0010] Step 2: In the base station, the current vertical handover policy network interacts with the environment to gain experience. In the current interaction, when the cumulative value of all reward values ​​in the experience is less than the target value, based on the state space, the gradient value of the vertical handover policy network and the gradient value of the value network are obtained using the constraint and near-end policy optimization algorithm. The gradient value of the vertical handover policy network and the gradient value of the value network are then sent to the central learner.

[0011] Step 3: In the central learner, the average gradient value of all the vertical switching policy networks is calculated to obtain the first average gradient value, and the average gradient value of all the value networks is calculated to obtain the second average gradient value.

[0012] Step 4: Obtain the vertical switching strategy network update parameters using the first gradient average value, obtain the value network update parameters using the second gradient average value, and send the vertical switching strategy network update parameters and the value network update parameters to the mobile device;

[0013] Step 5: In the mobile device, update the vertical switching strategy network and the value network using the vertical switching strategy network update parameters and the value network update parameters respectively to obtain the updated vertical switching strategy network and the updated value network. When the cumulative value of all reward values ​​in Step 2 is greater than or equal to the target value and the cumulative value of all reward values ​​is stable, the finally obtained updated vertical switching strategy network outputs the final strategy.

[0014] Optionally, the state space is represented as:

[0015] s t ={G t M t};

[0016] Among them, s t Let G be the state space of time slot t. t G represents the communication state collected by the mobile device in time slot t.t ={Cap gr (t),Cap air (t),Cap sat (t)},Cap gr (t) represents the channel capacity of the ground base station network in the t-th time slot, Cap air (t) represents the channel capacity of the airborne base station network in the t-th time slot, Cap. sat (t) represents the channel capacity of the satellite network in time slot t, M t Let t represent the behavior state of the mobile device in time slot t.

[0017] Optionally, step 2 includes:

[0018] Step 2.1: Initialize the total number of training rounds to N, the total number of iterations to K, and the total batch size to U;

[0019] Step 2.2: Initialize the learning rate and discount factor of the vertical switching policy network, the learning rate and discount factor of the value network, and the parameters of the vertical switching policy network and the value network, and set the number of training rounds n = 0, where 0 ≤ n ≤ N;

[0020] Step 2.3: Clear the buffer zone;

[0021] Step 2.4: Based on the state space sent by the mobile device to the base station, experience is obtained through interaction between the vertical handover policy network and the environment. This experience is stored in the buffer. In the current interaction, the relationship between the cumulative value of all reward values ​​in the experience and the target value is determined. If the cumulative value of all reward values ​​is greater than or equal to the target value and the cumulative value of all reward values ​​is stable, then the training of the vertical handover policy network and the value network is stopped. If the cumulative value of all reward values ​​is less than or equal to the target value, then step 2.5 is executed. The experience is represented as (s... t ,a t ,s t+1 ,R t ,j mc (a t )), a t For mobile devices to switch to idle state, terrestrial base station network, airborne base station network, or satellite network in time slot t, s t+1 Let R be the state space of time slot t+1. t Let j be the reward function for time slot t. mc (a t This refers to the cost of network usage;

[0022] Step 2.5: Determine the relationship between the capacity of the buffer and the total batch size U. If the capacity of the buffer is less than or equal to the total batch size U, return to step 2.4 to continue obtaining the experience and storing it in the buffer. If the capacity of the buffer is greater than the total batch size U, proceed to step 2.6.

[0023] Step 2.6: Calculate the advantage function based on the experience in the buffer, and store the advantage function in the buffer;

[0024] Step 2.7: Update the parameters of the vertical switching strategy network and the value network, and update the current iteration number to k+1, 0≤k≤K;

[0025] Step 2.8: Determine if the current iteration number is less than or equal to K. If yes, proceed to step 2.7; otherwise, execute step 2.9.

[0026] Step 2.9: Obtain the penalty coefficient for the (n+1)th training round based on the penalty coefficient for the nth training round, and determine whether the current (n+1) is less than or equal to N. If yes, proceed to step 2.3; otherwise, end the process.

[0027] Optionally, the reward function for the t-th time slot is expressed as:

[0028] R t =α1×Γ[Co s (t)] + +α2×Γ[j hc (t)] + ;

[0029] Among them, R t Let be the reward function for time slot t, where α1 and α2 are weighting coefficients satisfying α1 + α2 = 1, α1 > 0, α2 > 0, Γ[·] + To perform an inverse mapping after normalization, Co s (t) represents the covariance of the remote estimation error V in time slot t. t trace, j hc (t) represents the switching overhead for the t-th time slot.

[0030] Optionally, the advantage function is expressed as:

[0031] Among them, A c (t) is the dominance function for time slot t, T is the total number of time slots, λ is the discount factor, and R l Let s be the reward function for the l-th time slot. l+1 For the state space of the (l+1)th time slot, Let s be the state-value function of policy π in the state space of time slot l+1.l Let l be the state space of the l-th time slot. Let ψ be the state-value function of policy π in the state space of time slot l, and let ψ be the penalty coefficient.

[0032] Optionally, the state value function is expressed as:

[0033] in, Let be the state-value function of policy π in the state space at time slot t. Here, θ is the expectation operator, θ is the parameter of the vertical switching policy network, and R is the expectation operator. t+k Let a be the reward function for the (t+k)th time slot. t+k For mobile devices to switch to idle state, terrestrial base station network, airborne base station network, or satellite network in time slot t+k, s t+k Let be the state space of the (t+k)th time slot.

[0034] Optionally, update the parameters of the vertical switching policy network and the value network, including:

[0035] The parameters of the vertical switching policy network are updated using the policy loss function constructed based on the advantage function, and the parameters of the value network are updated using the value loss function constructed based on the state value function.

[0036] Optionally, the update formula for the parameters of the vertical switching strategy network is expressed as:

[0037] The policy loss function is expressed as follows:

[0038] The update formula for the parameters of the value network is expressed as:

[0039] The value loss function is expressed as:

[0040] Where θ(k+1) represents the parameters of the vertical switching policy network after the (k+1)th iteration, and θ(k) represents the parameters of the vertical switching policy network after the kth iteration. These are the parameters of the value network after the (k+1)th iteration. Let η be the parameter of the value network after the k-th iteration. θ The learning rate of the vertical switching policy network. The learning rate of the value network. The operator for differentiating the parameters θ of the vertical switching policy network. For the parameters of the value network The differentiation operator, L A (θ) is the policy loss function. For the value loss function, Here, R is the projection operator, min{·} is the minimum value operator, max{·} is the maximum value operator, ω is a hyperparameter related to the clipping function, and R is the maximum value operator. t Let be the reward function for time slot t. Let rate(t,θ) be the state-value function of policy π in the state space of time slot t+1. t |s t ,θ) / π(a t |s t ,θ old ), π(a t |s t ,θ) is the strategy corresponding to parameter θ, π(a t |s t ,θ old ) is the parameter θ old The corresponding strategy, θ old These are the parameters before the update.

[0041] Optionally, the penalty coefficient for the (n+1)th training round is expressed as:

[0042] ψ(n+1)=[ψ(n)-η ψ (σ-J π )];

[0043] Where ψ(n+1) is the penalty coefficient for the (n+1)th training round, ψ(n) is the penalty coefficient for the nth training round, and η ψ (·) represents the learning rate with penalty coefficient, σ represents the network usage cost budget, and J π The average cost of using strategy π T represents the total time slots. For the expectation operator of strategy π.

[0044] Optionally, the vertical handover strategy network includes a first input layer, several first fully connected hidden layers, and a first output layer connected in sequence. The first output layer includes four neurons, which respectively output the probability of the mobile device switching to an idle state, a ground base station network, an air base station network, and a satellite network.

[0045] The value network includes a second input layer, several second fully connected hidden layers, and a second output layer connected in sequence. The second output layer includes one neuron and outputs an estimated value of the state value function.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] This invention provides a vertical handover method for an integrated air-space-ground cyber-physical system. This method involves acquiring a state space from a mobile device and transmitting it to a base station. At the base station, a vertical handover policy network interacts with the environment to gain experience. Then, a constraint and near-end policy optimization algorithm is used to obtain the gradient values ​​of the vertical handover policy network and the value network. These gradient values ​​are then sent to a central learner. In the central learner, the average of the gradient values ​​of all vertical handover policy networks is calculated to obtain a first average gradient, and the average of the gradient values ​​of all value networks is calculated to obtain a second average gradient. This determines the update parameters for the vertical handover policy network and the value network, updating these parameters on the mobile device. Finally, when the cumulative value of all reward values ​​is greater than or equal to the target value and the cumulative value of all reward values ​​is stable, the final execution policy is output using the last determined vertical handover policy network on the mobile device. Therefore, the vertical handover method for an integrated air-space-ground cyber-physical system provided by this invention can be directly applied to vertical handover within the same system.

[0048] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0049] Figure 1 is a flowchart illustrating a vertical switching method for an integrated air-space-ground cyber-physical system provided in an embodiment of the present invention.

[0050] Figure 2 is a schematic diagram of a distributed learning framework provided in an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram showing the change of the cumulative value of all reward values ​​with the number of training rounds under different penalty coefficients according to an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram illustrating the variation of average usage cost with the number of training rounds under different penalty coefficients according to an embodiment of the present invention;

[0053] Figure 5 is a schematic diagram showing how the cumulative value of all reward values ​​changes with the number of training rounds, according to an embodiment of the present invention.

[0054] Figure 6 is a schematic diagram illustrating how the average usage cost changes with the number of training rounds, according to an embodiment of the present invention;

[0055] Figure 7 is a schematic diagram showing the change of handover frequency with moving speed under different handover strategies according to an embodiment of the present invention;

[0056] Figure 8 is a schematic diagram showing the change of average throughput with moving speed under different handover strategies provided by an embodiment of the present invention. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0058] Example 1

[0059] Currently, there is very little research on handover methods for space-air-ground integrated cyber-physical systems. Most existing handover methods are proposed to improve the service capabilities of space-air-ground integrated networks. The article "A Dynamic Handover Software-Defined Transmission Control Scheme in Space-Air-Ground Integrated Networks" (Guo C, Gong C, Xu H, et al. A Dynamic Handover Software-Defined Transmission Control Scheme in Space-Air-Ground Integrated Networks[J]. IEEE Transactions on Wireless Communications, 2022, 21(8): 6110-6124.) proposes a dynamic handover method for space-air-ground integrated networks, which utilizes software-defined transmission control to solve the bottleneck of information transmission between the controller and the handover node. The article Handoff-Aware Distributed Computing in High Altitude Platform Station (HAPS)-Assisted Vehicular Networks (Ren Q, Abbasi O, Kurt K, et al. Handoff-Aware Distributed Computing in High Altitude Platform Station (HAPS)-Assisted Vehicular Networks[J]. IEEE Transactions on Wireless Communications, 2023, 12(12): 8814-8827.) proposes a handover method for air-ground cooperative computing, in which air-based base stations and ground base stations cooperate to achieve vehicle distributed computing. In addition, considering the complexity of SAGIN network models and the difficulty in obtaining prior knowledge of network topology and modeling assumptions, some scholars have proposed using deep learning methods to formulate network service methods in recent years. The article Space-air-ground integrated multi-domain network resource orchestration based on virtual network architecture (Zhang P, Wang C, Kumar N, et al.)Space-air-ground integrated multi-domain network resource orchestration based on virtual network architecture: A DRL method[J].IEEE Transactions on Intelligent Transportation Systems, 2021, 23(3):2798-2808.) This paper studies a multi-domain resource optimization method that maximizes the allocation efficiency of space-air-ground integrated network resources using deep reinforcement learning; Deep reinforcement learning for delay-oriented IoT task scheduling in SAGIN (Zhou C, Wu W, He H, et al. Deep reinforcement learning for delay-oriented IoT task scheduling in SAGIN[J].IEEE Transactions on Wireless Communications, 2020, 20(2):911-925.) This paper studies a task scheduling method using deep reinforcement learning to minimize the offloading and computational latency of all tasks, addressing the problem of limited space-air-ground storage capacity; Multi-agent Reinforcement Learning-Based Orbital Edge Offloading in SAGIN Supporting Internet of Remote Things (Zhang S, Liu A, Han C, et al. Multi-agent Reinforcement Learning-Based Orbital Edge Offloading in SAGIN Supporting Internet of Remote Things) Orbital Edge Offloading in SAGIN Supporting Internet of Remote Things[J].IEEE Internet of Things Journal,2023,10(23):20472-20483.) This study investigates the optimization objective of multi-UAV cooperative task offloading strategies in SAGIN networks. The objective is for UAVs to select appropriate offloading destinations and task numbers, maximizing the number of tasks that satisfy latency constraints while considering limited UAV energy and satellite processing capabilities.

[0060] However, current vertical handover methods for integrated air-space-ground networks cannot be directly applied to vertical handover in integrated air-space-ground cyber-physical systems. This is because, unlike traditional heterogeneous networks, the primary goal of integrated air-space-ground cyber-physical systems is to ensure stable system operation. The handover criteria need to consider the stability of the control system, rather than simply indicators representing network service capabilities. Furthermore, when using deep reinforcement learning to solve the SAGIN resource management strategy problem, the weights of each attribute are determined before training and cannot be adaptively adjusted based on training results. This fails to meet constraint requirements and results in poor scalability of the network system.

[0061] Based on this, please refer to Figure 1. This invention provides a vertical switching method for an integrated air-space-ground cyber-physical system, which includes:

[0062] Step 1: The base station obtains the state space sent by the mobile device. The state space includes communication state and behavior state. Q randomly deployed mobile devices communicate with one base station, and each base station communicates with q mobile devices simultaneously. Each mobile device is equipped with a vertical handover policy network and a value network, where 0 < q ≤ Q.

[0063] Specifically, as shown in Figure 2, Q mobile devices (MDs) are randomly deployed, and each mobile device communicates with only one base station. Each base station communicates with all q mobile devices simultaneously. Each mobile device is equipped with a CPU (Central Processing Unit), and each base station has a local learner. Therefore, the mobile devices send their collected state space data to the base stations so that the base stations can train the vertical handover policy network and value network. Mobile devices can be, for example, mobile phones, robots, or smart cars. Each mobile device has a vertical handover policy network and a value network to ultimately obtain the policy (i.e., action) to be executed. The base stations can be terrestrial base stations, aerial base stations (such as unmanned aerial vehicles (UAVs), or satellites.

[0064] In this embodiment, the state space is represented as:

[0065] s t ={G t M t};

[0066] Among them, s t Let G be the state space of time slot t. t G represents the communication state collected by the mobile device in time slot t. t ={Cap gr (t),Cap air(t),Cap sat (t)},Cap gr (t) represents the channel capacity of the ground base station network in the t-th time slot, Cap air (t) represents the channel capacity of the airborne base station network in the t-th time slot, Cap. sat (t) represents the channel capacity of the satellite network in time slot t, M t Let t represent the behavioral state of the mobile device in time slot t, which includes the speed and location of the mobile device.

[0067] Optionally, the vertical handover policy network includes a first input layer, several first fully connected hidden layers, and a first output layer connected in sequence. The dimension of the first input layer matches the number of state features; each first fully connected hidden layer includes 512 neurons, using ReLU (Linear Rectification Function) as the activation function; the first output layer includes 4 neurons, corresponding to 4 possible policies, namely the probability of the output mobile device switching to idle state, terrestrial base station network, airborne base station network, and satellite network. The vertical handover policy network uses Softmax (normalized exponential function) as the activation function of the first output layer, and the learning rate of its network parameters is... Set to 0.0001.

[0068] Optionally, the value network comprises a second input layer, several second fully connected hidden layers, and a second output layer connected in sequence. The dimension of the second input layer matches the number of state features; each second fully connected hidden layer includes 512 neurons, using ReLU as the activation function; the second output layer is one-dimensional, comprising one neuron, and outputs an estimate of the state-value function. The value network uses Softmax as the activation function for the second output layer, and the learning rate η of its network parameters... θ Set to 0.0001.

[0069] Step 2: In the base station, the current vertical handover policy network interacts with the environment to gain experience. In the current interaction, when the cumulative value of all reward values ​​in the experience is less than the target value, based on the state space, the gradient values ​​of the vertical handover policy network and the value network are obtained using the Constraint-Proximal Policy Optimization (CPPO) algorithm. The gradient values ​​of the vertical handover policy network and the value network are then sent to the central learner.

[0070] In this embodiment, after the base station obtains the state space sent by the mobile device, it will interact with the environment using the current vertical handover policy network to gain experience. During the interaction, the current vertical handover policy network can interact with the environment many times. Each interaction generates a reward value, and the cumulative value of all reward values ​​can be calculated, i.e., the sum of all reward values. Then, if the cumulative value of all reward values ​​in the experience is less than the target value during this interaction, the vertical handover policy network and the value network need to be trained again. If the cumulative value of all reward values ​​is greater than or equal to the target value and the cumulative value of reward values ​​is stable (i.e., the cumulative value of all reward values ​​obtained in the current interaction is unchanged compared with the cumulative value of all reward values ​​obtained in the previous interaction), the mobile device directly uses the current vertical handover policy network to output the final policy.

[0071] In one specific embodiment, step 2 may include:

[0072] Step 2.1: Initialize the total number of training rounds to N, the total number of iterations to K, and the total batch size to U.

[0073] In this embodiment, when training the vertical switching policy network and value network, Xavier is used to initialize the weights and biases, and the Adam optimizer is used for training. The initial value of the penalty coefficient ψ is set to 0.1, and the learning rate η of the penalty coefficient is... ψ Set to 0.001. Specific parameters are shown in Table 1.

[0074] Table 1 Parameter Settings

[0075] Step 2.2: Initialize the learning rate and discount factor of the vertical switching policy network, the learning rate and discount factor of the value network, and the parameters of the vertical switching policy network and the value network, and set the number of training rounds n = 0, where 0 ≤ n ≤ N.

[0076] Step 2.3: Clear the buffer.

[0077] Step 2.4: Based on the state space sent by the mobile device to the base station, experience is obtained through interaction between the vertical handover policy network and the environment. This experience is stored in a buffer. In the current interaction, the relationship between the cumulative value of all reward values ​​in the experience and the target value is determined. If the cumulative value of all reward values ​​is greater than or equal to the target value and the cumulative value of all reward values ​​is stable, the training of the vertical handover policy network and the value network is stopped. If the cumulative value of all reward values ​​is less than or equal to the target value, step 2.5 is executed. Here, the experience is represented as (s... t ,a t ,s t+1 ,R t ,j mc(a t )), a t For mobile devices to switch to idle state, terrestrial base station network, airborne base station network, or satellite network in time slot t, s t+1 Let R be the state space of time slot t+1. t Let j be the reward function for time slot t. mc (a t Network usage cost refers to the fees charged by network operators to customers for using the network. The specific pricing is based on a comprehensive assessment of factors such as the technical costs, equipment costs, and maintenance costs required for the network.

[0078] In this embodiment, an action space is set. Mobile devices take appropriate actions based on specific environmental conditions. Action space Represented as:

[0079] Among them, a t The value 0, 1, 2, or 3 represents the mobile device switching to an idle state, terrestrial network, space-based network, or satellite network in time slot t, respectively. Here, when a... t =a t-1 This means that the mobile device does not perform a vertical switching action, but maintains its connection with the original base station or performs the corresponding horizontal switching.

[0080] In this embodiment, the reward function for the t-th time slot is expressed as:

[0081] R t =α1×Γ[Co s (t)] + +α2×Γ[j hc (t)] + ;

[0082] Where α1 and α2 are weighting coefficients, satisfying α1 + α2 = 1, α1 > 0, α2 > 0, representing the relative importance of the two indicators in the problem. Different weighting coefficients can be set according to user requirements, Γ[·] + To perform an inverse mapping after normalization, that is, mapping [0,1] to [1,0], the purpose is to unify the dimensions of the trace of the long-range estimation error covariance and the network switching cost, two different metrics. s (t) represents the covariance of the remote estimation error V in time slot t. t traces, that is Tr(·) is the trace of the matrix, j hc (t) represents the switching overhead for the t-th time slot.

[0083] Step 2.5: Determine the relationship between the buffer capacity and the total batch size U. If the buffer capacity is less than or equal to the total batch size U, return to step 2.4 to continue gaining experience and storing it in the buffer. If the buffer capacity is greater than the total batch size U, proceed to step 2.6.

[0084] Step 2.6: Calculate the advantage function based on the experience in the buffer and store the advantage function in the buffer.

[0085] In this embodiment, the dominance function is expressed as:

[0086] Among them, A c (t) is the dominance function for time slot t, T is the total number of time slots, λ is the discount factor, and R l Let s be the reward function for the l-th time slot. l+1 For the state space of the (l+1)th time slot, Let s be the state-value function of policy π in the state space of time slot l+1. l Let l be the state space of the l-th time slot. Let be the state-value function of strategy π in the state space of time slot l.

[0087] In this embodiment, the state value function is expressed as:

[0088] in, Let be the state-value function of policy π in the state space at time slot t. Here, θ is the expectation operator, θ is the parameter of the vertical switching policy network, and R is the expectation operator. t+k Let a be the reward function for the (t+k)th time slot. t+k For mobile devices to switch to idle state, terrestrial base station network, airborne base station network, or satellite network in time slot t+k, s t+k Let be the state space of the (t+k)th time slot.

[0089] Step 2.7: Update the parameters of the vertical switching strategy network and the value network, and update the current iteration number to k+1, 0≤k≤K.

[0090] Specifically, update the parameters of the vertical switching strategy network and the value network, including:

[0091] The policy loss function, constructed based on the advantage function, updates the parameters of the vertical switching policy network, while the value loss function, constructed based on the state value function, updates the parameters of the value network.

[0092] In this embodiment, the update formula for the parameters of the vertical switching policy network is expressed as:

[0093] The policy loss function is expressed as:

[0094] The formula for updating the parameters of the value network is expressed as:

[0095] The value loss function is expressed as:

[0096] Where θ(k+1) represents the parameters of the vertical switching policy network after the (k+1)th iteration, and θ(k) represents the parameters of the vertical switching policy network after the kth iteration. These are the parameters of the value network after the (k+1)th iteration. Let η be the parameter of the value network after the k-th iteration. θ The learning rate of the vertical switching policy network. The learning rate of the value network. The operator for differentiating the parameters θ of the vertical switching policy network. For the parameters of the value network The differentiation operator, L A (θ) is the policy loss function. For the value loss function, As a projection operator, it preserves the iteration point θ by projecting onto a compact convex set. k (θ k To ensure that θ is stable at the k-th iteration, min{·} is the minimum value operator, max{·} is the maximum value operator, ω is a hyperparameter related to the pruning function, and its function is to update the pruning policy loss. R t Let be the reward function for time slot t. Let rate(t,θ) be the state-value function of policy π in the state space at time slot t+1, and let rate(t,θ) represent the old policy π(a t |s t ,θ old ) and the new strategy π (a t |s t The probability ratio between t,θ), rate(t,θ) = π(a t |s t ,θ) / π(a t |s t ,θ old ), π(a t |s t ,θ) is the strategy corresponding to parameter θ, π(a t |s t ,θ old ) is the parameter θ old The corresponding strategy, θ old These are the parameters before the update.

[0097] Step 2.8: Determine if the current iteration number is less than or equal to K. If yes, proceed to step 2.7; otherwise, proceed to step 2.9.

[0098] Step 2.9: Obtain the penalty coefficient for the (n+1)th training round based on the penalty coefficient for the nth training round, and determine whether the current (n+1) is less than or equal to N. If yes, proceed to step 2.3; otherwise, end the training and complete the training of the vertical switching policy network and value network.

[0099] In this embodiment, the penalty coefficient for the (n+1)th training round is expressed as:

[0100] ψ(n+1)=[ψ(n)-η ψ (σ-J π )];

[0101] Where ψ(n+1) is the penalty coefficient for the (n+1)th training round, ψ(n) is the penalty coefficient for the nth training round, and η ψ (·) represents the learning rate with a penalty coefficient, σ is the network usage cost budget, representing the upper limit of the average usage cost, σ>0, J π The average cost of using strategy π This is the expectation operator for policy π, which calculates the expectation under policy π.

[0102] Step 3: In the central learner, calculate the average gradient value of all vertical switching policy networks to obtain the first average gradient value, and calculate the average gradient value of all value networks to obtain the second average gradient value.

[0103] Step 4: Use the first gradient average to obtain the vertical switching strategy network update parameters, use the second gradient average to obtain the value network update parameters, and send the vertical switching strategy network update parameters and the value network update parameters to the mobile device.

[0104] Specifically, the average value of the first gradient is added to the parameters of the previously obtained vertical switching strategy network to obtain the updated parameters of the vertical switching strategy network, and the average value of the second gradient is added to the parameters of the previously obtained value network to obtain the updated parameters of the value network. Then, the obtained updated parameters of the vertical switching strategy network and the updated parameters of the value network are transmitted to each mobile device.

[0105] Step 5: In the mobile device, update the vertical switching strategy network and the value network using the vertical switching strategy network update parameters and the value network update parameters respectively to obtain the updated vertical switching strategy network and the updated value network. When the cumulative value of all reward values ​​in Step 2 is greater than or equal to the target value and the cumulative value of all reward values ​​is stable, the finally obtained updated vertical switching strategy network outputs the final strategy.

[0106] Specifically, in the mobile device, the vertical switching strategy network is updated using the vertical switching strategy network update parameters, and the value network is updated using the value network update parameters. If, during step 2, it is confirmed that the cumulative value of all reward values ​​is greater than or equal to the target value and the cumulative value of all reward values ​​is stable, then the final strategy is output from the last updated vertical switching strategy network obtained in step 5. That is, the updated vertical switching strategy network will output the probabilities corresponding to the four actions respectively. Thus, the action with the highest probability can be selected as the final strategy and executed according to the strategy. Otherwise, it jumps back to step 1 and continues to use the updated vertical switching strategy network to collect the state space, and continues to train the vertical switching strategy network and the value network. The above process is repeated until the training ends. For the specific framework, please refer to Figure 2.

[0107] This invention comprehensively considers the stability, handover overhead, user cost constraints, and scalability of CPS systems, proposing a vertical handover method for integrated air-space-ground cyber-physical systems based on deep reinforcement learning. This method ensures the stability of the cyber-physical system in an integrated air-space-ground network environment, with low handover overhead, and meets user cost constraints. To further improve learning efficiency, this invention also proposes a constraint-proximal policy optimization algorithm and a mechanism for distributed reinforcement learning using ground base stations, airborne base stations, and satellites. The CPPO method is then combined with the distributed reinforcement learning mechanism to form the Distributed Constraint-Proximal Policy Optimization algorithm (DCPPO), which further enhances the stability of the training process and accelerates the convergence speed.

[0108] To verify the superiority of the proposed DCPPO method in solving network handover problems in multi-heterogeneous networks, and to verify the effectiveness of the vertical handover method for integrated air-space-ground cyber-physical systems based on deep reinforcement learning, simulation and comparative experiments were conducted. A two-dimensional simulation environment was set up, along with varying numbers of UAVs, ground base stations, satellites, and mobile devices. The mobile devices were primarily responsible for interacting with the environment and collecting environmental information. Due to the random distribution of UAVs and ground base stations and their limited signal coverage, some areas may have poor communication environments (e.g., only satellite communication is available).

[0109] (I) Experiment 1:

[0110] The simulation environment is a circular area with a radius of 2700 meters, containing 9 ground base stations and 6 rotary-wing UAVs, with full satellite communication coverage within the area. Eight mobile devices are placed in the simulation environment, and their trajectories are generated using a random waypoint model.

[0111] Assume the ground base station is installed atop a signal tower 20 meters high, while the rotorcraft UAV is suspended at a height of 90 meters. Set the bandwidth B of both the ground and airborne base station networks to 20MHz, the signal transmission power E to 200mW, and the noise power P... N The signal strength is -130dBm / Hz, carrier frequency f is 2000MHz, satellite network fixed bandwidth is 150Mbps, and channel capacity threshold φ = 120Mbps. The network usage costs for terrestrial base station network, airborne base station network, and satellite network are respectively j mc (a t ) = 0.3, j mc (a t ) = 0.9 and j mc (a t ) = 2.7, switching overhead j hc (a t =1. The speed of the mobile device is a random value between 5 and 12 m / s. The parameters of the dynamically controlled object are set as follows: state transition matrix A = [1.1, 0; 1, 0.8] and system noise covariance Q. ω = [0.3; 0.3]. Using the state transition matrix, we can obtain ζ≈0.174. λ i Let represent the eigenvalues ​​of the state transition matrix A. To ensure the existence of a feasible solution, considering σ > j mc (a t =3)×ζ=0.4698, set σ=1.20.

[0112] When solving the problem of vertical handover methods for integrated air-space-ground cyber-physical systems, the penalty coefficient ψ of the DCPPO method proposed in this invention can be dynamically updated. The comparison results of its reinforcement learning training process with that of a fixed penalty coefficient ψ are shown in Figures 3 and 4. The fixed penalty coefficient ψ is set to 0.5, 0.6, and 0.7, respectively.

[0113] Figure 3 compares the cumulative values ​​of all rewards (i.e., the cumulative reward values ​​in Figure 3) of reinforcement learning methods with dynamically updated ψ and fixed ψ as the number of training epochs changes. It can be found that the DCPPO method with dynamically updated ψ begins to converge after training up to 200 epochs, and its training is more stable than other methods; other methods have not yet converged and are unstable within 4000 epochs.

[0114] Figure 4 illustrates the change in average usage cost as the number of training epochs increases, mainly including the DCPPO method with dynamically updatable ψ, the DCPPO method with fixed ψ (values ​​of 0.5, 0.6, and 0.7), and the method with a fixed network usage cost budget σ = 1.2 (represented by the black dashed line in Figure 4). It can be seen that the curve of the DCPPO method with dynamically updatable ψ closely follows the baseline, while the average usage cost of the other three lines deviates significantly from the baseline. The method with a fixed ψ of 0.5 has an average usage cost that stabilizes at 2.7, failing to meet the constraint. While the average usage cost is lower than the budget when ψ is 0.6 and 0.7, as shown in Figure 3, its convergence speed and stability are poor. Therefore, the penalty coefficient ψ needs to be chosen appropriately. For example, setting the penalty coefficient to 0.5 too small fails to effectively constrain training, leading to rapid convergence to a strategy that only allows the selection of satellite networks. Although this strategy offers fast convergence and stable performance, it fails to meet the constraint. Alternatively, setting it to 0.6 or 0.7 effectively constrains the network usage cost budget, but performance deteriorates. Therefore, the DCPPO method with dynamically updatable ψ proposed in this invention fine-tunes the penalty coefficient based on the average usage cost calculated in each round. This allows for a reduction in the constraint intensity when it is too high, and vice versa, making the average usage cost curve closer to the baseline and the training process more stable.

[0115] (II) Experiment 2:

[0116] To verify the superiority of the proposed distributed learning framework, this embodiment utilizes the DCPPO method to solve the network handover problem. The results are compared with those of the DPPO (Distributed Proximal Policy Optimization) and CPPO methods in terms of stability, convergence speed, and the effect on average cost of use, as shown in Figures 5 and 6. Here, the CPPO method does not introduce distributed learning into policy training; while the DPPO method uses a distributed learning framework, its local learner is deployed on the mobile device side. Compared to the DCPPO method, which deploys the local learner on the base station side, the DPPO method collects more correlated data but has lower learning efficiency.

[0117] Figure 5 shows the cumulative reward value of the DCPPO, DPPO, and CPPO methods in solving the network handover problem as the number of training rounds changes. It can be seen that the DCPPO method of the distributed framework proposed in this invention converges at approximately 500 rounds, while the CPPO method converges at approximately 2000 rounds, and the DPPO method converges at approximately 800 rounds. This demonstrates that the method proposed in this invention offers the greatest improvement in convergence speed. Figure 6 shows the average cost of use as the number of training rounds changes for the DCPPO, DPPO, and CPPO methods in solving the network handover problem. It can be observed that although all three methods eventually achieve a close fit to the average cost of use constraint as the number of training rounds increases, the DCPPO method proposed in this invention exhibits better convergence speed and stability than the DPPO and CPPO methods. The advantages of the DCPPO method—fast convergence speed, stable convergence, and close fit to the average cost of use constraint—are due to the fact that the distributed framework proposed in this invention fully utilizes the characteristics of the integrated air-space-ground network, moving the local learner up to the ground base station. Ground base stations can collect information from multiple independent mobile devices, which is beneficial for data correlation and improves learning efficiency.

[0118] (III) Experiment 3:

[0119] This embodiment demonstrates a comparison of network performance between the vertical handover method for an integrated air-space-ground cyber-physical system proposed in this invention and a greedy vertical handover method that always selects the lowest usage cost and the most reliable network access at the current moment. First, two metrics are defined: handover frequency and handover efficiency, which are used to measure the performance of the vertical handover method for the integrated air-space-ground cyber-physical system.

[0120] Suppose that a mobile device operates for a total of T time slots in the environment using a specific strategy. Then, the switching frequency and switching efficiency of this strategy are as follows:

[0121] Among them, F h To switch frequencies, N h Let η be the number of handovers and η be the handover efficiency. For effective throughput, Cap(t) is the channel capacity of the t-th time slot, and β c To switch the total time.

[0122] Figures 7 and 8 illustrate the changes in handover frequency and efficiency of the proposed vertical handover method, the greedy vertical handover method (accessing the network with the lowest cost), and the greedy vertical handover method (accessing the most reliable network) as the mobile device moves with increasing speed. Figure 7 shows that the handover frequency of all three methods increases with increasing speed. However, the proposed method exhibits a significantly lower handover frequency at higher speeds compared to the other two greedy methods. This is because the proposed method considers the mobile device's speed and the negative impact of accumulated handover costs on the reward function during training; whereas the greedy method only seeks the currently available, lowest-cost, or most reliable network at each moment. At high speeds, the mobile device rapidly moves from one network domain to another, resulting in frequent handovers. Figure 8 illustrates that the access efficiency of all three methods decreases with increasing mobile device speed. However, the decrease in efficiency of the proposed method is significantly weaker than that of the other two greedy methods. This is because at high speeds, the proposed method has a low handover frequency and maintains network usage costs at the minimum level necessary to ensure system stability.

[0123] To address the network handover problem in an integrated air-space-ground network architecture, traditional handover methods struggle to leverage prior knowledge of the network topology and modeling assumptions inherent in complex SAGIN network models, often leading to local optima. Deep reinforcement learning-based methods, with their attribute weights predetermined before training, cannot adaptively adjust based on training data, failing to meet constraints and exhibiting poor scalability. In contrast, the DCPPO method employed in this invention achieves the smallest convergent optimal solution, fastest convergence speed, and shortest runtime compared to other methods. This invention provides a deep reinforcement learning-based vertical handover method for integrated air-space-ground cyber-physical systems, enhancing system stability and providing beneficial effects.

[0124] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0125] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0126] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0127] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A vertical switching method for an integrated air-space-ground cyber-physical system, characterized in that, The vertical switching method includes: Step 1: The base station obtains the state space sent by the mobile device, wherein the state space includes communication state and behavior state, Q randomly deployed mobile devices communicate with one base station respectively, each base station communicates with q mobile devices simultaneously, and each mobile device is equipped with a vertical handover policy network and a value network, 0 < q ≤ Q; Step 2: In the base station, the current vertical handover policy network interacts with the environment to gain experience. In the current interaction, when the cumulative value of all reward values ​​in the experience is less than the target value, based on the state space, the gradient value of the vertical handover policy network and the gradient value of the value network are obtained using the constraint and near-end policy optimization algorithm. The gradient value of the vertical handover policy network and the gradient value of the value network are then sent to the central learner. Step 3: In the central learner, the average gradient value of all the vertical switching policy networks is calculated to obtain the first average gradient value, and the average gradient value of all the value networks is calculated to obtain the second average gradient value. Step 4: Obtain the vertical switching strategy network update parameters using the first gradient average value, obtain the value network update parameters using the second gradient average value, and send the vertical switching strategy network update parameters and the value network update parameters to the mobile device; Step 5: In the mobile device, update the vertical switching strategy network and the value network using the vertical switching strategy network update parameters and the value network update parameters respectively to obtain the updated vertical switching strategy network and the updated value network. When the cumulative value of all reward values ​​in Step 2 is greater than or equal to the target value and the cumulative value of all reward values ​​is stable, the finally obtained updated vertical switching strategy network outputs the final strategy.

2. The vertical switching method according to claim 1, characterized in that, The state space is represented as: s t ={G t M t }; Among them, s t Let G be the state space of time slot t. t G represents the communication state collected by the mobile device in time slot t. t ={Cap gr (t),Cap air (t),Cap sat (t)},Cap gr (t) represents the channel capacity of the ground base station network in the t-th time slot, Cap air (t) represents the channel capacity of the airborne base station network in the t-th time slot, Cap. sat (t) represents the channel capacity of the satellite network in time slot t, M t Let t represent the behavior state of the mobile device in time slot t.

3. The vertical switching method according to claim 2, characterized in that, Step 2 includes: Step 2.1: Initialize the total number of training rounds to N, the total number of iterations to K, and the total batch size to U; Step 2.2: Initialize the learning rate and discount factor of the vertical switching policy network, the learning rate and discount factor of the value network, and the parameters of the vertical switching policy network and the value network, and set the number of training rounds n = 0, where 0 ≤ n ≤ N; Step 2.3: Clear the buffer zone; Step 2.4: Based on the state space sent by the mobile device to the base station, experience is obtained through interaction between the vertical handover policy network and the environment. This experience is stored in the buffer. In the current interaction, the relationship between the cumulative value of all reward values ​​in the experience and the target value is determined. If the cumulative value of all reward values ​​is greater than or equal to the target value and the cumulative value of all reward values ​​is stable, then the training of the vertical handover policy network and the value network is stopped. If the cumulative value of all reward values ​​is less than or equal to the target value, then step 2.5 is executed. The experience is represented as (s... t ,a t ,s t+1 ,R t ,j mc (a t )), a t For mobile devices to switch to idle state, terrestrial base station network, airborne base station network, or satellite network in time slot t, s t+1 Let R be the state space of time slot t+1. t Let j be the reward function for time slot t. mc (a t This refers to the cost of network usage; Step 2.5: Determine the relationship between the capacity of the buffer and the total batch size U. If the capacity of the buffer is less than or equal to the total batch size U, return to step 2.4 to continue obtaining the experience and storing it in the buffer. If the capacity of the buffer is greater than the total batch size U, proceed to step 2.

6. Step 2.6: Calculate the advantage function based on the experience in the buffer, and store the advantage function in the buffer; Step 2.7: Update the parameters of the vertical switching strategy network and the value network, and update the current iteration number to k+1, 0≤k≤K; Step 2.8: Determine if the current iteration number is less than or equal to K. If yes, proceed to step 2.7; otherwise, execute step 2.

9. Step 2.9: Obtain the penalty coefficient for the (n+1)th training round based on the penalty coefficient for the nth training round, and determine whether the current (n+1) is less than or equal to N. If yes, proceed to step 2.3; otherwise, end the process.

4. The vertical switching method according to claim 3, characterized in that, The reward function for the t-th time slot is expressed as: R t =α1×Γ[Co s (t)] + +α2×Γ[j hc (t)] + ; Among them, R t Let be the reward function for time slot t, where α1 and α2 are weighting coefficients satisfying α1 + α2 = 1, α1 > 0, α2 > 0, Γ[·] + To perform an inverse mapping after normalization, Co s (t) represents the covariance of the remote estimation error V in time slot t. t trace, j hc (t) represents the switching overhead for the t-th time slot.

5. The vertical switching method according to claim 3, characterized in that, The advantage function is expressed as: Among them, A c (t) is the dominance function for time slot t, T is the total number of time slots, λ is the discount factor, and R l Let s be the reward function for the l-th time slot. l+1 For the state space of the (l+1)th time slot, Let s be the state-value function of policy π in the state space of time slot l+1. l Let l be the state space of the l-th time slot. Let ψ be the state-value function of policy π in the state space of time slot l, and let ψ be the penalty coefficient.

6. The vertical switching method according to claim 5, characterized in that, The state value function is expressed as: in, Let be the state-value function of policy π in the state space at time slot t. Here, θ is the expectation operator, θ is the parameter of the vertical switching policy network, and R is the expectation operator. t+k Let a be the reward function for the (t+k)th time slot. t+k For mobile devices to switch to idle state, terrestrial base station network, airborne base station network, or satellite network in time slot t+k, s t+k Let be the state space of the (t+k)th time slot.

7. The vertical switching method according to claim 6, characterized in that, Update the parameters of the vertical switching policy network and the value network, including: The parameters of the vertical switching policy network are updated using the policy loss function constructed based on the advantage function, and the parameters of the value network are updated using the value loss function constructed based on the state value function.

8. The vertical switching method according to claim 7, characterized in that, The update formula for the parameters of the vertical handover strategy network is expressed as follows: The policy loss function is expressed as follows: The update formula for the parameters of the value network is expressed as: The value loss function is expressed as: Where θ(k+1) represents the parameters of the vertical switching policy network after the (k+1)th iteration, and θ(k) represents the parameters of the vertical switching policy network after the kth iteration. These are the parameters of the value network after the (k+1)th iteration. Let η be the parameter of the value network after the k-th iteration. θ The learning rate of the vertical switching policy network. The learning rate of the value network. The operator for differentiating the parameters θ of the vertical switching policy network. For the parameters of the value network The differentiation operator, L A (θ) is the policy loss function. For the value loss function, Here, R is the projection operator, min{·} is the minimum value operator, max{·} is the maximum value operator, ω is a hyperparameter related to the clipping function, and R is the maximum value operator. t Let be the reward function for time slot t. Let rate(t,θ) be the state-value function of policy π in the state space of time slot t+1. t |s t ,θ) / π(a t |s t ,θ old ), π(a t |s t ,θ) is the strategy corresponding to parameter θ, π(a t |s t ,θ old ) is the parameter θ old The corresponding strategy, θ old These are the parameters before the update.

9. The vertical switching method according to claim 3, characterized in that, The penalty coefficient for the (n+1)th training round is expressed as: ψ(n+1)=[ψ(n)-η ψ (σ-J π )]; Where ψ(n+1) is the penalty coefficient for the (n+1)th training round, ψ(n) is the penalty coefficient for the nth training round, and η ψ (·) represents the learning rate with a penalty coefficient, σ represents the network usage cost budget, and J π The average cost of using strategy π T represents the total time slots. For the expectation operator of strategy π.

10. The vertical switching method according to claim 1, characterized in that, The vertical handover strategy network includes a first input layer, several first fully connected hidden layers, and a first output layer connected in sequence. The first output layer includes four neurons, which respectively output the probability of the mobile device switching to an idle state, a ground base station network, an air base station network, and a satellite network. The value network includes a second input layer, several second fully connected hidden layers, and a second output layer connected in sequence. The second output layer includes one neuron and outputs an estimated value of the state value function.

Images