Resource scheduling method for space-time perception in cellular-free MIMO network

Abstract

The invention belongs to the technical field of mobile communication, and particularly relates to a resource scheduling method for space-time perception in a cellular-free MIMO network, which comprises the following steps: constructing a content cache, user association and resource allocation model; establishing a user association model, an access point sleep model, a communication model, a content request model and a power consumption model; a content caching, user association and resource allocation joint optimization model under a multi-user and multi-content distributed cellular-free MIMO network scene is established by taking maximization of network efficiency as a target; a joint optimization problem is converted into a decentralized partial observable Markov decision process, and a multi-agent deep reinforcement learning algorithm fused with a space-time loop diagram attention mechanism is adopted to solve and obtain an optimal solution. Compared with the existing reference algorithm, the algorithm provided by the invention has effectiveness and superiority.

Description

Technical Field

[0001] This invention belongs to the field of mobile communication technology, and specifically relates to a spatiotemporally aware resource scheduling method for non-cellular MIMO networks. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) and edge computing technologies, global communication traffic has increased dramatically, posing unprecedented challenges to traditional wireless network architectures. Traditional cellular networks, with their rigid structured boundaries and centralized management models, struggle to meet the demands of increasingly complex and high-density communication services. In response, researchers have begun to explore new network architectures, among which cellular-free MIMO is considered a potential solution. Unlike traditional cellular MIMO systems with fixed geographical boundaries, cellular-free MIMO employs a large-scale distributed node network, providing seamless coverage and service to users through collaboration, thereby improving network flexibility and adaptability. This architecture not only ensures more uniform network coverage but also significantly improves performance by dynamically adapting to fluctuating network conditions and user density. Furthermore, cellular-free MIMO can improve signal quality and reduce interference, providing more robust and reliable communication services.

[0003] Beyond flexible network architecture, content caching has become a key technology for improving resource efficiency in non-cellular MIMO networks. By proactively placing content closer to end users, edge caching can reduce transmission latency, optimize communication link usage, and enhance user experience. The inherent scalability and decentralized architecture of non-cellular MIMO systems provide a natural advantage for content caching implementation. In non-cellular MIMO, strategically placing content closer to users by leveraging the widespread distribution of nodes can significantly shorten data transmission distances. This strategic placement not only speeds up access but also reduces the load on the core network, enabling more scalable and efficient network operations. Due to these significant advantages, content caching in non-cellular MIMO has become a focus of extensive research and attention in recent years. Despite these advantages, effectively managing content caching, user association, and multi-dimensional resource allocation in non-cellular MIMO remains a complex challenge, especially in network environments where user needs are highly dynamic and diverse. Summary of the Invention

[0004] To address the highly dynamic and diverse network environments with varying user demands, this invention proposes a spatiotemporally aware resource scheduling method for non-cellular MIMO networks. This method includes: establishing a joint optimization model based on content caching, user association, and resource allocation in a multi-user, multi-content distributed non-cellular MIMO network scenario, with the goal of maximizing network efficiency. The joint optimization problem is transformed into a decentralized partially observable Markov decision process. A multi-agent deep reinforcement learning algorithm incorporating a spatiotemporal cyclic graph attention mechanism is then used to optimize the agents. The optimized agents are deployed at access points, and the optimal content caching, user association, and resource allocation scheme is selected based on the agent's current observed state.

[0005] Furthermore, transforming the joint optimization problem into a decentralized partially observable Markov decision process involves: modeling each access point as an agent; during the training phase, each agent utilizes global state information to optimize its own parameters; during the execution phase, the agent selects actions only based on its local state. In time slot t, the agent's local state includes the access points in the larger time-scale time slot t-1. The cached content set, the access point in the large time scale time slot t-1 Operating status and time slots Location of each mobile device.

[0006] Furthermore, the intelligent agent in time slots The reward function is time slot Total network speed With time slot Total network power consumption The ratio of .

[0007] Furthermore, the actions of agent j include actions in time slot t over a large time scale. Access point cached content set , Indicates time slot The relationship and time slots between access point j and various mobile devices Transmit power allocated to each mobile device by access point j , Indicates access point Large timescale time slots The internal operating status.

[0008] Furthermore, the process of optimizing the agents using a multi-agent deep reinforcement learning algorithm that incorporates a spatiotemporal cyclic graph attention mechanism includes:

[0009] 101. Initialize the agent;

[0010] 102. The agents deployed at each access point select actions based on the value output of the local Q-network according to the state they observe.

[0011] 103. Take the actions selected by each access point as a joint action, execute the joint action, and use the ratio of the total network speed to the total network power consumption after the selected action as the global reward.

[0012] 104. Store the state of an agent, the action performed, the global reward for performing the action, and the state reached after performing the action as a set of experiences in the experience pool. Determine whether the maximum number of explorations has been reached. If not, increment the number of explorations by 1 and return to step 102. If it has been reached, proceed to step 105.

[0013] 105. At fixed time intervals, X samples are sampled from the experience pool. The current agent obtains the local Q-values ​​of all agents in the system and calculates the joint Q-value based on the local Q-values ​​of all agents.

[0014] 106. Update the online network by minimizing the average difference between the target value and the joint Q value of X samples, thus completing the update of the target network parameters based on the online network parameters.

[0015] Furthermore, each agent is configured with a deep Q-network consisting of multiple perceptrons, which is used to obtain local action value estimates in time slots. Agent j receives local observations. As input, and output local action value estimate .

[0016] Furthermore, the process of calculating the joint Q-value includes:

[0017] Get global environment state It is encoded into an embedding vector through a multilayer perceptron. For agent j, a spatiotemporal recursive graph attention network is used to capture the attention features between this agent and other agents. ;

[0018] Estimating the value of local actions and attention characteristics As input to a neural network with monotonicity constraints, the network outputs a joint action value function. .

[0019] Furthermore, the process of capturing the attentional features between agent j and other agents using a spatiotemporal recursive graph attention network includes:

[0020] The multi-agent environment is constructed as a time-varying undirected graph. , A set of nodes composed of intelligent agents. For the connection relationship between nodes, for a given access point, there is an edge relationship between the two access points that are closest to that access point.

[0021] Calculate the attention coefficients between agent j and its neighboring agents, and normalize the attention coefficients using the softmax function;

[0022] Employing a multi-head attention mechanism, including The output feature representation vector of the independent attention mechanism at each node It can be represented as:

[0023]

[0024] in, It is a nonlinear function. This indicates a splicing operation. The sequence number represents the attention mechanism, where L is the number of heads in multi-head attention; For agent j, which outputs the l-th attention head, and its neighboring agents... Attention coefficient For time slots The set of neighboring agents of agent j; Let be the weight matrix that can be learned by the l-th attention head; For time slots intelligent agent The embedding vector;

[0025] To capture temporal features, the output is embedded The input is fused with the output of the previous time slot of a loop unit to obtain the intelligent agent. In the time slot Hidden state at time ;

[0026] Hide state As the input to the multilayer perceptron, the multilayer perceptron provides intelligence to the agent. Local action value function generates adaptive hybrid weights ;

[0027] The local action value function of the agent With adaptive hybrid weights As input to a monotonic neural network, the monotonic neural network outputs a joint action value function. .

[0028] Furthermore, gap intelligent agent Embedded vector The acquisition process includes: obtaining the global environment state Input multilayer perceptron encoding into an intelligent agent The embedding vector.

[0029] This invention proposes a multi-agent deep reinforcement learning algorithm for enhancing spatiotemporal recurrent graph attention neural networks. It can effectively capture the spatiotemporal correlations among content caching demand, content caching, user association, and network interference. The performance of the proposed algorithm was evaluated through extensive simulations, and it was compared with existing benchmark schemes in terms of network efficiency, network speed, power consumption, and content cache hit rate, thus verifying the performance advantages of the proposed algorithm. Attached Figure Description

[0030] Figure 1 This is a flowchart of a preferred embodiment of the spatiotemporally aware resource scheduling method for non-cellular MIMO networks according to the present invention.

[0031] Figure 2 This invention provides a non-cellular MIMO network with multi-user, multi-content caching.

[0032] Figure 3 This invention relates to the relationship between network performance and transmission power at different access points.

[0033] Figure 4 This invention relates to the relationship between network performance and the maximum cache capacity of different access points.

[0034] Figure 5 This invention relates the network performance to the maximum content volume of different networks. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] This invention proposes a spatiotemporally aware resource scheduling method for non-cellular MIMO networks, comprising: establishing a joint optimization model based on content caching, user association, and resource allocation in a multi-user, multi-content distributed non-cellular MIMO network scenario with the goal of maximizing network efficiency; transforming the joint optimization problem into a decentralized partially observable Markov decision process; and using a multi-agent deep reinforcement learning algorithm that integrates a spatiotemporal cyclic graph attention mechanism to optimize the agents; the optimized agents are deployed at the access point, and the optimal content caching, user association, and resource allocation scheme is selected based on the current observed state of the agent.

[0037] Next, this embodiment will use... Figure 1 The present invention will be described in the following process sequence.

[0038] This invention considers a typical multi-user, multi-content distributed non-cellular MIMO network model, which consists of multiple access points and multiple mobile devices, as shown in the network model below. Figure 2 As shown, the network consists of N access points and M mobile devices, represented by sets. and This means that mobile devices dynamically establish connections with access points based on their changing content needs and network conditions. Each access point is equipped with specific content caching capabilities to meet the local content needs of the connected mobile devices. By proactively storing popular content, access points reduce latency and bandwidth consumption through local caching, thereby reducing the need to retrieve content from a remote central processing unit (CPU) via fiber optic connections. Furthermore, to improve energy efficiency, access points dynamically enter sleep mode during periods of low activity to reduce power consumption. (Assuming there are a total of...) A set contains 1 or more distinct contents. To effectively capture the multi-dimensional dynamic characteristics of non-cellular MIMO operation, this invention introduces a discrete-time frame, which is divided into two time scales: a large-scale frame and a small-scale time slot. To model the multi-scale dynamics of buffering, association, and transmission processes in non-cellular MIMO, this invention employs a discrete-time frame with two time scales: a larger time scale handles relatively slow-changing decisions (e.g., content buffering and access point sleep control), and a smaller time scale handles faster-changing decisions (e.g., user association and power control). The large-scale frame is defined as... Indexing is performed, where each frame contains hourly time slot, the first The set of small time-scale time slots within a large time-scale frame is denoted as . ,make Indicates the index of a time slot at a small time scale.

[0039] Based on the multi-user, multi-content distributed non-cellular MIMO network model, user association model, access point sleep model, communication model, content request model, and power consumption model are established. The establishment of each model will be explained in the following sections.

[0040] In each small time scale time slot Each mobile device may dynamically associate with one or more access points based on a variety of factors, including the mobile device's geographical location, the requested content, channel quality, and the spatial density of surrounding connections. This invention enables... Indicates in time slot With mobile devices The associated set of access points, binary variables Indicated for use on mobile devices and access point The relationship between them is defined as follows:

[0041]

[0042] To ensure service availability, each mobile device must be associated with at least one access point in each time slot on every small time scale. Therefore, user associations must meet certain constraints. It's important to note that not all access points are necessarily active or providing service in every time slot. Accordingly, in each time slot... The set of all access points that actively provide services to mobile devices during this period is defined as follows: ,in .

[0043] In resource-intensive non-cellular MIMO networks, effective energy management and interference suppression are crucial for maintaining network performance and sustainability. To improve operational efficiency and reduce unnecessary energy consumption, access points can dynamically switch to sleep mode during periods of low traffic demand or when neighboring access points can adequately meet user service needs.

[0044] make Indicates access point Large timescale time slots The internal operating status, among which Specifically, Indicates access point It is active and ready to provide services to users at any time. Conversely, if If the access point is in sleep mode, it will be temporarily disabled to save energy and reduce network interference.

[0045] To reflect the overall activity of all access points in the network, let Representing a large timescale slot The overall operational status within the system. This binary activation model enables the system to adaptively balance service availability and energy efficiency. By selectively disabling underutilized access points, the network can reduce operating costs, extend infrastructure lifespan, promote environmentally sustainable operation, and maintain the desired quality of service.

[0046] Once a mobile device sends a content request to its associated access point, the content delivery process begins as long as the requested content is available in the access point's local cache. Small timescale slots Inside, from the access point To mobile devices downlink radio channel The model is as follows:

[0047]

[0048] in, Access point With mobile devices The Euclidean distance between them For reference distance, Indicates the path loss index; It represents small-scale fading and is modeled as a circularly symmetric complex Gaussian random variable.

[0049] Time slot Internal access point Transmit to mobile device The symbol is denoted as ,in , Using conjugate beamforming technology based on complete channel state information, the transmitted signal is represented as follows:

[0050]

[0051] in, Access point Assigned to mobile devices The transmission power, Access point Maximum transmission power, collection Includes time slots Internal access point Mobile devices that provide services Indicates access point The power allocation vector.

[0052] mobile device In the time slot The internally received signal can be represented as:

[0053]

[0054] in, Indicates mobile device The noise signal, and It is not compatible with mobile devices The set of associated access points; set Indicates in time slot All mobile devices served during the period.

[0055] mobile device In the time slot The corresponding achievable data rate can be expressed as:

[0056]

[0057] Finally, time slots The total network speed can be obtained from the following formula:

[0058]

[0059] In non-cellular MIMO networks, access points typically have limited storage capacity, allowing them to cache only a portion of the global content, while the CPU has ample storage space to cache all network content. Therefore, each access point must selectively cache content based on the potential needs of its associated mobile devices, within its storage constraints.

[0060] make Indicates access point Maximum amount of content that can be stored Indicates access point Large timescale time slots The actual collection of cached content. And must meet the constraints. In each small timescale slot ,mobile device Content requests Indicates. Access point In the time slot The aggregated collection of content requests from all mobile devices within the service is defined as follows: Content Request It is based on mobile devices The preference vector is determined by which all content is ordered in descending order of popularity. Typically, this ordering is modeled using a Zipf distribution. It's important to note that in practice, content preferences often change over time due to variations in user behavior. To capture this temporal evolution, we assume that each mobile device has... There are 1 content preference level, and each level lasts for 1 hour. ,in .mobile device The complete set of preference vectors is represented as Each of them The specified number The ordering of content within each preference level, and Indicates the first The first of the levels The sorting of content. Accordingly, mobile devices In the time slot Request content The probability can be defined as:

[0061]

[0062] in, Mobile devices The skewness parameter. When At that time, content requests are evenly distributed; as... The more gifts are given, the more likely they will be given to content that ranks higher (is more popular). The value reflects the user's individual preferences and behavioral patterns.

[0063] In a non-cellular MIMO network, if the access point has cached the service content requested by its associated mobile device, it will deliver the content directly via downlink radio transmission. Otherwise, the access point must first obtain the requested content from the CPU via the fronthaul link before delivering it to the mobile device.

[0064] During content caching and distribution, the total energy consumption of the access point mainly includes the following four parts:

[0065] 1) Content transmission power consumption: The transmission power of each access point transmitting content to its associated mobile device varies depending on the specific transmission strategy. At the hourly slot scale... Access point Content transmission power consumption can be defined as:

[0066]

[0067] 2) Content caching power consumption: To adapt to the dynamically changing content request patterns of mobile devices, each access point updates its cached content periodically on each large time-slot scale. Indicates access point The set of new cached or replaced content. The corresponding cache power consumption is:

[0068]

[0069] in, This represents the unit power required to cache a single content item.

[0070] 3) Content forwarding power consumption: If the requested content is not found in the local cache, the access point... The content must first be retrieved from the CPU before it can be delivered to the mobile device. This represents the set of missing content that needs to be retrieved from the CPU. Forwarding power consumption is calculated as follows:

[0071]

[0072] in, This represents the unit power consumption for forwarding content from the CPU to the access point.

[0073] 4) Operating power consumption: The operating power consumption of an access point depends on its activity state. When the access point is in an active state... When in sleep mode, energy is consumed for signal processing, control signaling, and backhaul maintenance. At this time, its operating power consumption is negligible. Therefore, the operating power consumption is expressed as:

[0074]

[0075] in, This is the fixed operating power in active mode. Therefore, the access point In the time slot The total power consumption is:

[0076]

[0077] In summary, time slots The total network power consumption is:

[0078]

[0079] The various models constructed in this invention aim to improve overall network performance by maximizing long-term average energy efficiency, where long-term average energy efficiency is defined as the ratio of achievable total rate of data to total power consumption. This optimization involves four key dimensions: content caching, user association, power allocation, and access point sleep control. This problem is denoted as... The expression is as follows:

[0080]

[0081] in, and These represent time slots on a large time scale. Content caching strategies and small timescale slots The power allocation strategy. Indicates user association strategy, This represents the sleep control decision at the access point. (Constraints) Ensure that the amount of content cached at each access point does not exceed its storage capacity; constraints Ensure that each mobile device is associated with at least one access point; constraints Defines the binary association state between the mobile device and the access point; constraints. and The access point limits the transmission power allocated to mobile devices; finally, the constraints... The operating mode of each access point is specified, and its active or dormant state is determined according to the operating requirements.

[0082] In practical cellular-free MIMO content caching networks, the autonomy and randomness of user access often lead to uneven resource utilization, thereby reducing overall network efficiency and user service quality. This problem is particularly pronounced due to the decentralized architecture and highly dynamic operating environment of cellular-free MIMO systems. In this environment, obtaining accurate global network state information in real time (such as user content requests, channel state conditions, and access policies) becomes increasingly difficult. Furthermore, joint optimization problems... Its mixed-integer, nonlinear, and stochastic characteristics, coupled with multi-timescale decision-making and the partial observability of network states, present significant challenges. System decisions (including content caching, user association, and resource allocation) are based solely on the current system state and have no explicit correlation with historical observations. This characteristic aligns closely with the Markov decision process framework.

[0083] To capture the decentralized and partially observable characteristics of cellular-free MIMO environments, this invention abstracts the original problem into a decentralized partially observable Markov decision process. Furthermore, to effectively learn adaptive policies under dynamic spatiotemporal conditions, this invention develops an enhanced multi-agent deep reinforcement learning framework. This framework integrates a spatiotemporal recurrent graph attention network, supports scalable decentralized policy learning, and possesses temporal state memory and topological feature extraction capabilities. Based on this, this invention proposes a spatiotemporally aware joint optimization method (STA-JOCURA) for content caching, user association, resource allocation, and access point sleep control.

[0084] Optimization problem Converted into a containing The Dec-POMDP is a set of intelligent agents, where each access point is modeled as an intelligent agent using tuples. Description. Here, It is the global environment state space. and They represent access points respectively. The local observation space and action space, It is a reward function. It is a discount factor. In each time slot intelligent agent Receive local observations and select action The joint action of all agents is represented as After executing the joint action, the environment returns a global reward. and transition to the next state. .in:

[0085] 1) Environmental conditions: time The environmental state includes historical content caching decisions, the operational status of access points, the current channel state, content preferences, and the location information of mobile devices. These factors are crucial to the agent's decision-making process. Therefore, the environmental state... It can be defined as follows:

[0086]

[0087] in, It is the access point in the time slot of the previous large time scale. The cached content set Content Is it cached at the access point? middle, Content Cache at access point Middle, otherwise . Access point Channel gain set to each mobile device. Reflects mobile devices In the time slot Content preferences. Mobile devices Location information.

[0088] 2) Local Observation: In partially observable non-cellular MIMO environments, each access point can only observe its own content cache state and user location information for the current time slot. Therefore, access points... Local observation Represented as:

[0089]

[0090] 3) Actions: Based on the question Access point Optimizable variables include content caching decisions, user association, power allocation, and sleep control decisions. Agent access point. action Defined as:

[0091]

[0092] 4) Reward function: All agents perform a joint action. Afterwards, the environment will return a global reward. To evaluate this joint action. Based on the question... The reward function is defined as:

[0093]

[0094] In partially observable environments, each agent Receive local observations Then, based on its local strategy Select Action .make Representing the joint policy of all agents, STA-JOCURA aims to learn a policy that maximizes the cumulative global decay reward by jointly optimizing content caching, user association, resource allocation, and runtime state. Therefore, the value function of joint actions can be defined as:

[0095]

[0096] in, It is the expectation operation, the action value function. Indicates from state Starting point, in joint strategy and actions Under these conditions, the expected cumulative global decay reward is determined. The optimal joint strategy is... Is to make Maximizing the joint strategy, i.e. .

[0097] Traditional multi-agent reinforcement learning (MARL) methods, such as independent Q-learning, can be applied to situations involving... The distributed, partially observable Markov decision process involves multiple agents. However, these methods have some limitations in non-cellular MIMO environments: 1) From each agent's perspective, the environment is non-stationary due to each agent independently updating its content cache, user association, resource allocation, and access point sleep control strategies. This non-stationarity severely hinders convergence and learning stability. 2) Spatiotemporal correlations between agents cannot be effectively utilized. For example, user content requests often exhibit local similarity, and total network throughput is closely related to user association decisions and spatial interference—especially when the Euclidean distance between access points decreases. Furthermore, the popularity of user demands dynamically changes over time and region. To address these limitations and more effectively utilize spatiotemporal correlations, this invention constructs the multi-agent environment as a time-varying undirected graph. ,in For a set of nodes, The connections between nodes are defined. Here, access points, agents, and nodes are conceptually equivalent. Subsequently, we design a value decomposition framework based on spatiotemporal recurrent graph attention, which enables efficient and scalable joint policy learning. Specifically, by leveraging graph attention mechanisms to capture spatial dependencies and recurrent neural networks to capture temporal dynamics, the joint action value function is decomposed into weighted local action value functions.

[0098] The STA-JOCURA algorithm follows a centralized training and distributed execution (CTDE) paradigm. During training, each agent utilizes global state information to optimize its own parameters; while during execution, decisions are made based solely on local observations. The STA-JOCURA framework comprises three core modules:

[0099] 1) Local Action Value Network. Each agent is equipped with a Deep Q-Network (DQN) consisting of Multilayer Perceptrons (MLPs). In time slots... The agent receives local observations. It outputs a local action value estimate. .

[0100] 2) Spatiotemporal recursive graph attention module. Global environment state. First, the nodes are encoded into node embedding vectors using a multilayer perceptron. In this invention, all agents share the global environment state at the same time, so all agents have the same embedding vector in the same time slot. Subsequently, the spatiotemporal recurrent graph attention network (STR-GAT) is used to capture the correlation between agents.

[0101] In a non-cellular MIMO network, each node (e.g., access point) Each has a set of neighbor nodes. The neighbor node set consists of time slots Edge set of time Determined. If an edge exists between two nodes, they are considered neighbors. In a spatiotemporal graph attention network, nodes... Its neighboring nodes The attention coefficient between them can be obtained through The calculation shows that, among which It is a self-attention mechanism. This is the corresponding learnable weight matrix. Attention coefficients. Indicates in time slot At that time, neighboring nodes Features of nodes The importance of each node. To facilitate comparison between different nodes, the attention coefficients need to be normalized using the softmax function:

[0102]

[0103] To stabilize the learning process, we employ a multi-head attention mechanism. Given normalized attention coefficients, we include... The output feature representation vector of the independent attention mechanism at each node It can be represented as:

[0104]

[0105] in, It is a nonlinear function. This indicates a splicing operation. This indicates the sequence number of the attention mechanism. To capture temporal features, the output is embedded... The input is fed into a recurrent unit (e.g., a long short-term memory network):

[0106]

[0107] in, Represents intelligent agents In the time slot The hidden state at that time. This recurrent embedding encodes the spatiotemporal features. Then, the multilayer perceptron network uses... As input, for the intelligent agent Local action value function generates adaptive hybrid weights .

[0108] 3) Hybrid Module. The hybrid module utilizes graph attention features rich in time information. Integrating local action value functions This forms a joint action value function:

[0109]

[0110] in, It is a neural network with monotonicity constraints (e.g., a network similar to QMIX). Finally, the model is trained by minimizing the loss function:

[0111]

[0112] Where θ is the parameter used to evaluate the network. This represents the number of mini-batch samples randomly sampled from the experience replay buffer. Indicates the sample number. Indicates time slot The target value of the x-th sample, Indicates the time slot when the evaluation network is θ. The joint action value function for the x-th sample; the objective value is expressed as... , These are the parameters of the target network.

[0113] The process of the STA-JOCURA algorithm is as follows:

[0114] Algorithm: STA-JOCURA: A spatiotemporal graph attention joint optimization algorithm for cellular-free multiple-input multiple-output (MIMO) systems. enter: ; ; ; ; ; ; ; ; Total training rounds Step size per round Target network update rate Output: The trained local Q-network and United Q Network ;1. Initialize the local Q-network and hybrid networks 2. Initialize the target network And set its weight as , 3. Initialize the experience replay buffer ;4 for episode=1 to do5 Initializes a cellular-free MIMO environment and observes the initial state. ;6 for to do7for each agent do8 observes local state ;9 based on A greedy strategy is used to select actions. ;10 end for11 Execute joint action ;12 Receive global rewards Next state And local observations at the next moment ;13 Transfer the sample Store to 14. If training conditions are met, then 15. From Medium sampling includes Small batch data of each transferred sample; 16 endif17 for each agent do18 Calculates local Q-values ​​using online networks ;19 end for20 Calculating the joint Q-value using a hybrid network based on an attention mechanism 21. Calculate the target value: Update parameters by minimizing the loss. : Regularly update the target network: ;22 end for23end for24 end for

[0115] This embodiment considers a non-cellular MIMO network consisting of 10 access points and 5 mobile devices, covering an area of ​​1000m × 1000m. Each mobile device can access multiple access points simultaneously. The network bandwidth is 40MHz, and the transmission power of each access point is... The power ranges from 30mW to 100mW, with a path loss index of 2 and a Gaussian noise power of [value missing]. W. There are 100 to 300 content requests in the network, and each access point can only cache a portion of the content, with the cache capacity varying between 30 and 100. To simulate dynamic user preferences, assume that users have H = 3 content preference levels, which change periodically over time. Users' content requests are distributed across different geographical regions, and users in neighboring regions exhibit similar preferences.

[0116] For the SJA-CORULA algorithm, each agent's neural network uses a two-layer multilayer perceptron (MLP) structure with 128 and 64 neurons respectively; the MLP layer of the graph attention network (GAT) module has 64 neurons. The algorithm uses the Adam optimizer with a learning rate of 0.0001 and a discount factor of 0.98; the Q-network is updated every 100 steps, the experience pool size is set to 50,000, and the batch size is set to 64. The step size for each training round is 100, and the agent adopts a greedy strategy with the greedy coefficient initially set to 1, which linearly decays to 0.1 after 10,000 iterations.

[0117] To evaluate content caching performance in non-cellular MIMO networks, we define a proportion-based cache hit rate, which reflects partial hit rates in multi-access-point interconnected scenarios. Specifically, for time slots... Mobile devices within Its individual cache hit rate is used The representation is defined as follows:

[0118]

[0119] in, Indicator index: If the requested content is already cached at the access point If the condition is met, return 1; otherwise, return 0.

[0120] The overall network cache hit rate is defined as the average of the individual hit rates of all mobile devices:

[0121]

[0122] Next, we will compare and evaluate the performance of the STA-JOCURA solution in four dimensions: reward, network speed, power consumption, and cache hit rate, with the following benchmark solutions:

[0123] The TM-JOCURA scheme is a joint optimization framework based on traditional multi-agent deep reinforcement learning for content caching, user association, resource allocation, and access point sleep control in non-cellular MIMO networks, but it does not consider the spatiotemporal variation characteristics of user content requests.

[0124] ATM-JOCURA scheme: A benchmark approach where the access point makes joint decisions on content caching, user association, resource allocation, and hibernation control based solely on average content preferences. This approach does not consider users' historical behavior or the temporal variation characteristics of content demand.

[0125] The IA-JOCURA scheme: In this scheme, each access point acts as an agent with limited observability. During training, the agent can only obtain its own cached information and location information, and cannot perceive the state of other agents, thus limiting the effectiveness of collaborative learning.

[0126] ATM-OSC scheme: A variant of the ATM-JOCURA scheme, which omits the access point sleep control optimization step and is used to separately evaluate the impact of sleep control policies on network performance.

[0127] In STA-JOCURA, the agent reward is defined as the average reward across all iterations in a training round. To ensure fairness in the comparison, the observation, action, and reward structures of the TM-JOCURA, ATM-JOCURA, IA-JOCURA, and ATM-OSC schemes are consistent with those of STA-JOCURA, and the neural network depth and neuron configuration are also exactly the same.

[0128] The STA-JOCURA solution is compared with the benchmark solutions mentioned above in terms of four dimensions: average reward, network speed, power consumption, and cache hit rate.

[0129] 1) Performance comparison under varying access point transmit power: Figure 3 This demonstrates the performance comparison between STA-JOCURA and various benchmark solutions when the maximum transmit power of the access point varies from 100 milliwatts to 1000 milliwatts. Figure 3 (a) It is evident that STA-JOCURA consistently achieves the highest average reward across all power levels. TM-JOCURA and ATM-JOCURA follow closely behind, thanks to their observability of the global state and their joint optimization capabilities; while IA-JOCURA and ATM-OSC perform poorly due to limited state information and lack of sleep control, respectively. Figure 5(b) shows that the average network rate of all schemes increases with the increase of transmission power, but STA-JOCURA can make more efficient use of transmission resources due to its spatiotemporal awareness and agent adaptive cooperation mechanism, thus maintaining a clear advantage. Figure 3 In (c), the ATM-OSC scheme has significantly higher power consumption than other schemes due to the lack of sleep control optimization; while STA-JOCURA achieves low and stable power consumption, verifying the effectiveness of its energy-sensing sleep control mechanism. Figure 3 (d) shows the average cache hit rate under different power settings. STA-JOCURA maintains a high cache hit rate by adaptively learning content request patterns and jointly optimizing cache decisions, outperforming all benchmark solutions. In summary, these results demonstrate that STA-JOCURA can effectively improve energy efficiency, throughput, and cache utilization while maintaining low power consumption under a wide range of operating conditions.

[0130] 2) Performance evaluation under changes in access point cache capacity: Figure 4 The performance of the proposed STA-JOCURA scheme and benchmark schemes was evaluated when the maximum cache capacity of each access point varied from 10 to 80 content items. Figure 4 (a) It can be seen that the average reward of all schemes increases with the increase of cache capacity, with STA-JOCURA having the highest reward value. Figure 4 (b) shows that the average network rate for all schemes remained relatively stable as the cache capacity increased. This indicates that while a larger cache capacity can reduce backhaul link usage and energy consumption, it cannot significantly improve network throughput. Figure 4 (c) This highlights the energy-saving advantages of STA-JOCURA: its average power consumption gradually decreases as the buffer capacity increases, a trend stemming from reduced redundant transmissions and efficient utilization of local buffer content. Conversely, the ATM-OSC solution, lacking a sleep control mechanism, consumes significantly more power. Figure 4 As shown in (d), the cache hit rate of all schemes increases with the increase of cache capacity, but STA-JOCURA consistently outperforms other schemes throughout the process, demonstrating its advantage in accurately capturing and adapting to the spatiotemporal characteristics of user content needs. In summary, STA-JOCURA not only achieves the best overall performance in terms of reward, speed, and cache hit rate, but also maintains low power consumption, making it highly practical in cache-constrained edge environments.

[0131] 3) Performance comparison under the maximum variation in the amount of network content: Figure 5 This demonstrates the performance of STA-JOCURA and various benchmark schemes when the maximum number of content items in the network varies from 50 to 100. Figure 5(a) It is evident that the average reward for all solutions decreases as the content library size increases. This is because an increase in content quantity leads to greater content diversity and a lower cache hit probability, thus increasing the difficulty of achieving energy-efficient services. Nevertheless, STA-JOCURA maintains a significant performance advantage, demonstrating its adaptability in dynamic, high-content-density environments. Figure 5 (b) shows that STA-JOCURA maintained the highest average network speed even with the increase in the total amount of content. The performance degradation of other solutions was due to a decrease in cache hit rate, which in turn led to an increase in backhaul link traffic. Figure 5 (c) shows the average power consumption: STA-JOCURA achieves the lowest power consumption across all content scales thanks to its space-time aware caching strategy and access point sleep control mechanism; in contrast, the ATM-OSC scheme has the highest power consumption due to the lack of adaptive sleep control, the access point being constantly active, and redundant transmission. Figure 5 (d) shows that as the amount of available content increases, the average cache hit rate of all schemes decreases—a phenomenon consistent with the objective condition of limited access point storage capacity. However, STA-JOCURA's performance is significantly better than other schemes, verifying the effectiveness of its space-time aware caching strategy. In summary, STA-JOCURA achieves high rewards, high throughput, and high cache hit rate while maintaining low power consumption.

[0132] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for spatial-temporal aware resource scheduling in a cell-free MIMO network, characterized in that, To maximize network efficiency, a joint optimization model based on content caching, user association, and resource allocation is established for multi-user, multi-content distributed non-cellular MIMO network scenarios. The joint optimization problem is transformed into a decentralized partially observable Markov decision process, and a multi-agent deep reinforcement learning algorithm with spatiotemporal cyclic graph attention mechanism is used to optimize the agents. The optimized agents are deployed at the access point, and the optimal content caching, user association, and resource allocation schemes are selected according to the current observed state of the agents. 2.The method of claim 1, wherein, To maximize network efficiency, a joint optimization model based on content caching, user association, and resource allocation is established for multi-user, multi-content distributed non-cellular MIMO network scenarios. This model includes: Constraints: ; ； ； ； ； ； wherein is the set of contents actually cached by all access points in a large time scale slot ; is the set of access points actively serving mobile devices during a time slot ; is the set of power allocation vectors of all access points ; is the set of operational states of all access points in a large time scale slot ; denotes averaging over ; is the number of small time scale slots in a large time scale slot ; is the total network rate in a time slot ; is the total network power consumption in a time slot ; is the set of cached contents of access point ; is the maximum number of contents that can be stored by access point ; is the set of all access points ; is the set of all mobile devices ; is the set of small time scale slots in the th large time scale frame ; denotes the association state between mobile device and access point , if the two are associated , otherwise ; denotes the transmit power allocated by access point to mobile device ; is the maximum transmit power of access point ; is the set of mobile devices served by access point in a time slot ; denotes the operational state of access point in a large time scale slot , if denotes that access point is in sleep state in a large time scale slot denotes the operational state of access point in a large time scale slot The inner is active.

3. The method of claim 2, wherein, Converting the joint optimization problem into a decentralized partially observable Markov decision process includes modeling each access point as an agent, during a training phase, each agent utilizes global state information to optimize its own parameters, during an execution phase, the agents only select actions based on local state, at time slot t, the local state of an agent includes the set of cache contents of the access points in a large time scale time slot t-1, the operating states of the access points in a large time scale time slot t-1, and the locations of the respective mobile devices in time slot t. ​​​ 4. The resource scheduling method for spatiotemporal awareness in non-cellular MIMO networks according to claim 3, characterized in that, intelligent agents in time slots The reward function is time slot Total network speed With time slot Total network power consumption The ratio of .

5. A resource scheduling method for spatiotemporal awareness in non-cellular MIMO networks according to claim 3, characterized in that, Actions of agent j include actions at time scale t Access point Set of cached content of access point j , Denotes time scale Association between access point j and each mobile device, time scale Transmit power assigned by access point j to each mobile device , Denotes operation state of access point At time scale t .

6. The method of claim 1 or 3, wherein, The process of optimizing agents using a multi-agent deep reinforcement learning algorithm that incorporates a spatiotemporal cyclic graph attention mechanism includes:

101. Initialize the agent; 102. The agents deployed at each access point select actions based on the value output of the local Q-network according to the state they observe.

103. Take the actions selected by each access point as a joint action, execute the joint action, and use the ratio of the total network speed to the total network power consumption after the selected action as the global reward.

104. Store the state of an agent, the action performed, the global reward for performing the action, and the state reached after performing the action as a set of experiences in the experience pool. Determine whether the maximum number of explorations has been reached. If not, increment the number of explorations by 1 and return to step 102. If it has been reached, proceed to step 105.

105. At fixed time intervals, X samples are sampled from the experience pool. The current agent obtains the local Q-values ​​of all agents in the system and calculates the joint Q-value based on the local Q-values ​​of all agents.

106. Update the online network by minimizing the average difference between the target value and the joint Q value of X samples, thus completing the update of the target network parameters based on the online network parameters.

7. A spatiotemporally aware resource scheduling method for non-cellular MIMO networks according to claim 6, characterized in that, Each agent is configured with a deep Q-network composed of multiple layers of perceptrons that are used to obtain local action value estimates, at time slot , agent j receives a local observation as input and outputs a local action value estimate .

8. The method of claim 6, wherein, The process of calculating the joint Q-value includes: Obtain global environment state Encode it into an embedding vector through a multi-layer perceptron For agent j, capture attention features between this agent and other agents with a spatio-temporal recurrent graph attention network ; Estimating local action value and attention features as an input to a neural network with monotonicity constraints, which outputs a joint action value function .

9. The method of claim 8, wherein, The process of capturing the attention features between agent j and other agents using a spatiotemporal recursive graph attention network includes: A multi-agent environment is constructed as a time-varying undirected graph , is a node set composed of agents, is a connection relationship between nodes, and for an access point, the two access points closest to the access point have an edge relationship Calculate the attention coefficients between agent j and its neighboring agents, and normalize the attention coefficients using the softmax function; Adopting multi-head attention mechanism, comprising an output feature representation vector of the independent attention mechanism of the node may be represented as: in, It is a nonlinear function. This indicates a splicing operation. The sequence number represents the attention mechanism, where L is the number of heads in multi-head attention; For agent j, which outputs the l-th attention head, and its neighboring agents... Attention coefficient For time slots The set of neighboring agents of agent j; Let be the weight matrix that can be learned by the l-th attention head; For time slots intelligent agent The embedding vector; To capture temporal features, the output embeddings are input to a recurrent unit that fuses the output of the recurrent unit at the previous time step to obtain the hidden state of the agent at time step t ; The hidden state As input to the multi-layer perceptron, the multi-layer perceptron generates adaptive mixing weights for a local action value function of the agent ;​ the agent local action value function with adaptive mixing weights as an input to a monotonicity-constrained neural network that outputs the joint action value function .

10. A spatiotemporally aware resource scheduling method for non-cellular MIMO networks according to claim 9, characterized in that, gap intelligent agent Embedded vector The acquisition process includes: obtaining the global environment state Input multilayer perceptron encoding into an intelligent agent The embedding vector.

Images