Joint resource allocation method based on cloud edge collaboration
By employing a cloud-edge collaborative resource allocation method, an access point utility model and a multi-agent reinforcement learning framework were constructed. This solved the problems of unstable user association and low power allocation efficiency in the integrated air-space-ground network, achieving efficient and stable user association and power allocation, and improving system performance and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-05
- Publication Date
- 2026-03-27
AI Technical Summary
In an integrated air-space-ground network, how can we ensure the quality of user services while addressing constraints such as highly dynamic networks and limited edge resources, and achieve efficient and stable user association and power allocation? This is especially important given the differences in geographical conditions and uneven user distribution. Traditional methods suffer from long iteration times and poor real-time performance.
A joint resource allocation method based on cloud-edge collaboration is adopted. By constructing an access point utility model and a multi-agent reinforcement learning framework, the policy network is collaboratively trained in the cloud and parameters are periodically distributed to the edge nodes. The edge nodes make real-time decisions based on local channel state information.
It effectively suppresses the phenomenon of users frequently switching access points, improves spectrum utilization and system throughput, enhances the perceived quality for end users, extends device battery life and improves overall energy efficiency, and adapts to large-scale heterogeneous network deployments.
Smart Images
Figure CN121751252A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of communication technology and relates to a joint resource allocation method based on cloud-edge collaboration. Background Technology
[0002] With the large-scale commercialization of fifth-generation (5G) mobile communication technology, the global industry has begun research on the next-generation (6G) mobile communication technology. 6G technology aims to expand beyond human and IoT connectivity to the intelligent interconnection of everything, overcoming terrain limitations and achieving seamless global coverage. Against this backdrop, the Space-Air-Ground Integrated Network (SAGIN), as a heterogeneous network architecture integrating satellites, high-altitude platforms, and ground base stations, has attracted widespread attention. SAGIN can integrate the advantages of multiple access methods, achieving wide-area coverage and ubiquitous access for massive numbers of users, effectively compensating for the shortcomings of traditional terrestrial communication systems in terms of coverage, deployment flexibility, and emergency response capabilities.
[0003] To further improve system capacity and spectrum efficiency, Non-Orthogonal Multiple Access (NOMA) technology has been introduced into SAGIN. This technology allows multiple users to share the same time-frequency resources, improving spectrum utilization through power domain multiplexing. However, compared with traditional terrestrial networks, SAGIN has characteristics such as wide network coverage, strong network dynamism, and uneven user distribution. Due to the high-speed movement of satellites, the dynamic deployment of UAVs, and the uncertainty of user locations, the network topology changes frequently, leading to unstable channel states and access relationships, which brings new challenges to resource allocation. Especially in real-world scenarios, differences in geographical conditions and uneven user distribution result in uneven deployment of terrestrial base stations, while the dual mobility of airborne platforms, satellites, and users further exacerbates the complexity of user association decisions. Under these circumstances, how to achieve stable and efficient user-access point association and combine it with the NOMA mechanism for reasonable power allocation becomes a key factor affecting system performance.
[0004] In recent years, Deep Reinforcement Learning (DRL) has attracted widespread attention in the field of SAGIN resource management due to its superior ability to continuously interact with dynamic environments and its ability to quickly generate decision-making policies after training convergence. This type of method alleviates, to some extent, the problems of long iteration times and poor real-time performance inherent in traditional optimization methods. However, edge nodes in SAGIN are typically limited by computing power, storage space, and energy supply, making it difficult for them to independently run complex neural network models and complete online training and inference tasks.
[0005] Therefore, in NOMA-based integrated air-space-ground networks, how to ensure user service quality while addressing constraints such as highly dynamic networks and limited edge resources, and achieve efficient and stable user association and power allocation, remains a pressing technical challenge. Summary of the Invention
[0006] In view of this, the purpose of this invention is to provide a cloud-edge collaborative resource allocation method to solve technical problems such as unstable user association, low power allocation efficiency, and poor real-time performance caused by highly dynamic network topology and limited edge node resources.
[0007] To achieve the above objectives, the present invention provides the following technical solution: A cloud-edge collaborative joint resource allocation method includes the following steps: S1: Construct a network communication model and an access point utility model that integrates reachable rates, association preferences, and handover penalties; S2: Construct a cloud-edge collaborative multi-agent reinforcement learning framework, setting each access point as an independent agent. The cloud uses historical global experience to conduct collaborative training of the policy network and periodically sends the training parameters to the corresponding edge access points. S3: Each edge access point generates and executes joint decisions on user association and power allocation in real time based on the issued policy network parameters and local channel status information.
[0008] Furthermore, S1 specifically includes the following steps: S11: Based on the access point With users Calculate the historical association frequency and the historical association weight between the two. ,in For time window The inside is the access point. With users The number of associations.
[0009] Social communities are divided based on the degree of communication intimacy between users, and access points are calculated. With the community correlation .
[0010] Based on the historical association weights and the association degree between the access point and the community, the association preference of the access point for the user is calculated, specifically as follows: (1) The numerator represents the degree of relevance between the access point and the user's community. Historical association weights between access points and users The weighted sum, where the denominator represents the access point. The weighted sum of all users within the coverage area; For a binary variable, if the user Does it belong to the community? ,but ,on the contrary .
[0011] S12: Construct an access point utility model based on reachability, association preferences, and handover penalties, specifically: (2) in, For time slots User association vectors; Power control vector for each access point; For users The achievable rate; Access point For users Association preferences; Weights representing association preferences; This is an indicator function, representing the user. Whether the association is switched, if in a time slot If an access point switch occurs, then Conversely ; This is the penalty coefficient; Indicates access point Maximum transmit power; constraints , This means that each user can only connect to one access point, and one access point is associated with... Individual users. Constraints and Indicates access point To its first Transmit power constraints for each associated user. Constraints This means that each user can only belong to one community.
[0012] Furthermore, S2 specifically includes the following steps: S21: Construct a cloud-edge collaborative multi-agent reinforcement learning framework, deploying multiple policy DNNs corresponding to edge access points and a shared value DNN in the cloud.
[0013] S22: Each access point will observe the local state of the previous time slot. Execution of actions Rewards received and the local state of the current time slot Together, they form a marginal experience They also upload their respective edge experiences to the cloud via the backhaul link.
[0014] For each access point, the reward is designed as the access point utility, i.e.: (3) To avoid insufficient social attribute data in the early stages of training, the weights are dynamically adjusted.
[0015] (4) In the early stages of training It mainly relies on physical layer information, in the later stages of training. Approaching The weight gradually increases.
[0016] S23: After receiving the edge experience uploaded by all access points, the cloud integrates it into global experience and stores it in the global experience replay buffer; subsequently, the cloud periodically samples a batch of global experience from the buffer and performs the following training operations: Training the value DNN: With the goal of minimizing the temporal difference error, update the parameters of the value DNN so that it can accurately evaluate the long-term performance of the global policy.
[0017] Training Policy DNN: Using the output of the value DNN as a criterion for evaluating the quality of the policy, the parameters of all policy DNNs are updated through gradient ascent to maximize the long-term utility value of all access points.
[0018] S24: After the policy DNN in the cloud completes a certain number of training updates, it periodically sends the updated network parameters to their respective edge access points via the downlink. Upon receiving the new parameters, the access points update their local edge DNNs, thereby continuously optimizing their decision-making capabilities.
[0019] Furthermore, S3 specifically includes the following steps: S31: At the start of each decision slot, the access point builds local experience and uploads it to the cloud.
[0020] S32: After receiving the local status, the access point outputs user association and power allocation decisions in real time based on the local policy network structure. Each access point first observes the local information. And utilize its edge DNN, based on local information Determine the action space Then, calculate the reward. And collect local experience And transmit it to the cloud. To stabilize training, Gaussian noise is added to the action space, and the access point... Decision It can be represented as: (5) It is Gaussian noise. The noise variance can be expressed as: (6) in and These represent the initial value and the final value, respectively. This represents the attenuation rate.
[0021] Parameters of edge DNN Based on the corresponding cloud-based strategy, the training parameters of the DNN Update. Due to the large number of parameters and the high latency between the cloud and the access point, updating in every time slot is impractical. Therefore, let's assume an update every... Update parameters for each time slot The edge DNN parameter update is represented as: (7) in This refers to the propagation latency from the cloud to the access point.
[0022] S33: The access point executes the decision action generated in S32, namely, allocating the corresponding power to the associated user for communication; at the same time, the access point observes the utility value under this policy as a local reward, and collects the local state of the next time slot, thus forming a new edge experience for subsequent uploading to the cloud, i.e., returning to S22. The entire training phase is repeated continuously until the value network, policy network in the cloud, and edge policy network on the access point are all trained to convergence.
[0023] The beneficial effects of this invention are as follows: (1) This invention effectively suppresses the phenomenon of users frequently switching access points by designing an access point utility model that integrates association preferences and handover penalties. The historical association weight and social community association degree calculation introduced in the technical solution strengthen the long-term association tendency between access points and users, thereby maintaining the continuity and reliability of the connection in a highly dynamic network environment and avoiding communication interruptions caused by topology changes.
[0024] (2) Based on a cloud-edge collaborative multi-agent reinforcement learning framework, this invention achieves joint optimization of user association and power allocation. The cloud utilizes historical global experience to centrally train the policy network and periodically distributes parameters to the edge access points, enabling edge nodes to quickly generate real-time decisions. This method overcomes the limitations of traditional optimization algorithms, such as long iteration time and poor real-time performance, and significantly improves spectrum utilization and system throughput.
[0025] (3) By reducing system overhead and decision latency, this invention directly improves the perceived quality for end users. Edge access points independently perform lightweight decisions based on local channel state information, reducing reliance on global instantaneous information and ensuring timely service response. At the same time, the integration of the association preference mechanism makes users more inclined to connect to access points with frequent historical interactions or high social intimacy, thereby improving service personalization and satisfaction.
[0026] (4) The cloud-edge collaborative architecture fully leverages the advantages of cloud computing and the flexibility of edge nodes, solving the bottleneck of limited resources for edge devices. The cloud is responsible for training complex neural networks, while the edge focuses on real-time inference, which reduces the edge load and ensures decision accuracy. This division of labor and collaboration mode enables the system to adapt to large-scale heterogeneous network deployments, extend device battery life, and improve overall energy efficiency.
[0027] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a network architecture diagram of the present invention; Figure 2 This is a flowchart of a cloud-edge collaborative joint resource allocation method proposed in this invention. Detailed Implementation
[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0030] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0031] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0032] Please see Figure 1 and Figure 2 This invention proposes a joint resource allocation method based on cloud-edge collaboration, which specifically includes the following steps: Step 1: Construct a network communication model and an access point utility model that integrates reachability, association preferences, and handover penalties; Step 1.1: Based on the access point With users Calculate the historical association frequency and the historical association weight between the two. ,in For time window The inside is the access point. With users The number of associations.
[0033] Social communities are divided based on the degree of communication intimacy between users, and access points are calculated. With the community correlation .
[0034] Based on the historical association weights and the association degree between the access point and the community, the association preference of the access point for the user is calculated, specifically as follows: (1) The numerator represents the degree of relevance between the access point and the user's community. Historical association weights between access points and users The weighted sum, where the denominator represents the access point. The weighted sum of all users within the coverage area; For a binary variable, if the user Does it belong to the community? ,but ,on the contrary .
[0035] Step 1.2: Construct an access point utility model based on reachability, association preferences, and handover penalties, specifically as follows: (2) in, For time slots User association vectors; Power control vector for each access point; For users The achievable rate; Access point For users Association preferences; Weights representing association preferences; This is an indicator function, representing the user. Whether the association is switched, if in a time slot If an access point switch occurs, then Conversely ; This is the penalty coefficient; Indicates access point Maximum transmit power; constraints , This means that each user can only connect to one access point, and one access point is associated with... Individual users. Constraints and Indicates access point To its first Transmit power constraints for each associated user. Constraints This means that each user can only belong to one community.
[0036] Step 2: Construct a cloud-edge collaborative multi-agent reinforcement learning framework, setting each access point as an independent agent. The cloud uses historical global experience to conduct collaborative training of the policy network, and periodically sends the training parameters to the corresponding edge access points. Step 2.1: Build a cloud-edge collaboration framework, deploying multiple policy DNNs corresponding to edge access points and a shared value DNN in the cloud.
[0037] Step 2.2: Each access point will observe the local state of the previous time slot. Execution of actions Rewards received and the local state of the current time slot Together, they form a marginal experience They also upload their respective edge experiences to the cloud via the backhaul link.
[0038] For each access point, the reward is designed as the access point utility, i.e.: (3) To avoid insufficient social attribute data in the early stages of training, the weights are dynamically adjusted.
[0039] (4) In the early stages of training It mainly relies on physical layer information, in the later stages of training. Approaching The weight gradually increases.
[0040] Step 2.3: After receiving the edge experiences uploaded by all access points, the cloud integrates them into global experience and stores it in the global experience replay buffer. Subsequently, the cloud periodically samples a batch of global experience from the buffer and performs the following training operations: Training the Value DNN: The goal is to minimize the temporal difference error by updating the parameters of the value DNN so that it can accurately evaluate the long-term performance of the global policy. Let the d-th sample from the D randomly sampled global experiences in the global buffer be... ,Right now The mean squared error loss between the predictions of the value DNN and the target value DNN based on D samples can be calculated as follows: (5) in Indicates the discount factor. Represents the target's global action space.
[0041] To minimize the loss function, gradient descent is used to update the parameter vector. ,Right now: (6) in, This represents the learning rate of the value network. This causes the loss function to gradually decrease, meaning the predicted long-term access point utility of the value network gradually approaches its true value.
[0042] Training the policy DNN: The output of the value DNN is used as a criterion for evaluating the policy's quality. The parameters of all policy DNNs are updated using gradient ascent to maximize the long-term utility of all access points. The objective function of the policy network is set as follows: (7) To maximize the objective function, the parameter vector is updated using gradient ascent, i.e.: (8) in, Let be the learning rate of the policy. This makes the objective function... Gradually increase.
[0043] Step 2.4: After the policy DNN in the cloud completes a certain number of training updates, it periodically sends the updated network parameters to their respective edge access points via the downlink. Upon receiving the new parameters, the access points update their local edge DNNs, thereby continuously optimizing their decision-making capabilities.
[0044] Step 2.5: Parameters of the Target Value DNN and the parameters of the target policy DNN It updates using a soft update method at fixed time slot intervals, that is: (9) (10) Step 3: Each edge access point generates and executes a joint decision on user association and power allocation in real time based on the received policy network parameters and local channel state information.
[0045] Step 3.1: At the beginning of each decision time slot, the access point builds local experience and uploads it to the cloud.
[0046] Step 3.2: After receiving the local status, the access point outputs user association and power allocation decisions in real time based on the local policy network structure. Each access point first observes the local information. And utilize its edge DNN, based on local information Determine the action space Then, calculate the reward. And collect local experience And transmit it to the cloud. To stabilize training, Gaussian noise is added to the action space, and the access point... Decision It can be represented as: (11) It is Gaussian noise. The noise variance can be expressed as: (12) in and These represent the initial value and the final value, respectively. This represents the attenuation rate.
[0047] Parameters of edge DNN Based on the corresponding cloud-based strategy, the training parameters of the DNN Update. Due to the large number of parameters and the high latency between the cloud and the access point, updating in every time slot is impractical. Therefore, let's assume an update every... Update parameters for each time slot The edge DNN parameter update is represented as: (13) in This refers to the propagation latency from the cloud to the access point.
[0048] Step 3.3: The access point executes the decision action generated in Step 3.2, namely, allocating the corresponding power to the associated user for communication; simultaneously, the access point observes the utility value under this policy as a local reward, and collects the local state of the next time slot, thus forming a new edge experience for subsequent uploading to the cloud, i.e., returning to Step 2. The entire training phase is repeated continuously until the value network, policy network in the cloud, and edge policy network on the access point are all trained to convergence.
[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A joint resource allocation method based on cloud-edge collaboration, characterized in that: The method includes the following steps: S1: Construct a network communication model and an access point utility model that integrates reachable rates, association preferences, and handover penalties; S2: Construct a cloud-edge collaborative multi-agent reinforcement learning framework, setting each access point as an independent agent. The cloud uses historical global experience to conduct collaborative training of the policy network and periodically sends the training parameters to the corresponding edge access points. S3: Each edge access point generates and executes joint decisions on user association and power allocation in real time based on the issued policy network parameters and local channel status information.
2. The joint resource allocation method based on cloud-edge collaboration according to claim 1, characterized in that: In step S1, the historical association weight is calculated based on the historical association frequency between the user and the access point. ; Social communities are divided based on the degree of communication intimacy between users, and the correlation between access points and each community is calculated. ; Based on the historical association weights and the association degree between the access point and the community, the association preference of the access point for the user is calculated, specifically as follows: （1） An access point utility model is constructed, with the objective function being to maximize the sum of the utilities of all access points. The utility of each access point is determined by its reachable rate, the associated preferences, and whether an access point handover occurs. A penalty term is introduced for handover behavior to improve connection stability, specifically expressed as follows: （2） in, For time slots User association vectors; Power control vector for each access point; For users The achievable rate; Access point For users Association preferences; The weights for associated preferences; This is an indicator function, representing the user. Whether the association is switched, if in a time slot If an access point switch occurs, then Conversely ; This is the penalty coefficient; Indicates access point Maximum transmit power; constraints , This means that each user can only connect to one access point, and one access point is associated with... Individual users; constraints and Indicates access point To its first Transmit power constraints for each associated user; constraints This means that each user can only belong to one community.
3. The joint resource allocation method based on cloud-edge collaboration according to claim 1, characterized in that: In S2, multiple policy DNNs corresponding to edge access points and a shared value DNN are deployed in the cloud; Each edge access point will observe the local state of the previous time slot. Execution of actions Rewards received and the local state of the current time slot Together, they form a marginal experience And through the backhaul link, they upload their respective edge experiences to the cloud; For each access point, the reward is designed as the access point utility, i.e.: （3） To avoid insufficient social attribute data in the early stages of training, the weights are dynamically adjusted. （4） In the early stages of training Dependent on channel state information, in the later stages of training Approaching The weight gradually increases; The cloud integrates the edge experiences uploaded from all access points into global experiences, stores them in the global experience replay buffer pool, and uses the temporal difference algorithm to sample and train the samples in the buffer pool to update the parameters of the policy DNN and value DNN. The trained and updated policy DNN parameters are periodically distributed to the corresponding edge access points to update their local edge DNNs.
4. The joint resource allocation method based on cloud-edge collaboration according to claim 1, characterized in that: In S3, at the beginning of each decision time slot, the access point builds local experience and uploads it to the cloud; The access point inputs the local state of the current time slot into the updated local DNN, and the local DNN synchronously outputs discrete user association decisions and continuous power allocation values. The user association decisions are generated by calculating the association scores for users within the coverage area and selecting the top K highest scores. The power allocation values are generated by linearly scaling the network output to the actual transmit power constraint range.
5. The joint resource allocation method based on cloud-edge collaboration according to claim 3, characterized in that: In the cloud deployment, the parameters of the target value DNN and the target policy DNN are synchronized from the value DNN and policy DNN during training through soft updates to stabilize the training process.
6. The joint resource allocation method based on cloud-edge collaboration according to claim 4, characterized in that: The parameters of the edge DNN Based on the corresponding cloud-based strategy, the training parameters of the DNN Update; assume every Update parameters for each time slot ; The edge DNN parameter update is represented as: （5） in The propagation latency from the cloud to the access point; The entire training phase is repeated continuously until all networks are trained to convergence.