Multi-agent reinforcement learning SCMA network resource allocation method and system and medium

Abstract

The invention discloses a multi-agent reinforcement learning SCMA network resource allocation method and device and a medium, and the method comprises the steps: taking a specified machine type communication device as an independent agent, and selecting an SCMA resource block for uplink data transmission according to a local Q table; the base station receives uplink data transmission signals of the multiple agents, determines the number of competitors on each SCMA resource block, and calculates a final reward composed of a basic reward and a direct collision punishment for each agent; the base station feeds back a final reward to the intelligent agent through a downlink channel; and the agent updates a locally maintained Q table according to the received final reward. According to the method, the benefit approaching behavior of the intelligent agent can be effectively inhibited, the problems of resource collision concentration and system performance reduction under high load can be solved, the intelligent agent can be guided to form more balanced global resource occupation distribution with fewer conflicts, and the packet loss rate and collision rate of the system under the scene close to saturation and overload are reduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent wireless communication management technology, specifically to a method, device, and storage medium for resource allocation in a multi-agent reinforcement learning (SCMA) network based on direct collision penalty. Background Technology

[0002] As sixth-generation mobile communication evolves towards an integrated air-space-ground network, the demand for device access in massive machine-type communication scenarios is exploding. SCMA, as an advanced non-orthogonal multiple access (NOMA) technology, allows multiple devices to share the same time-frequency resources by allocating sparse multidimensional multiplexed books to devices, achieving a system overload rate of over 100%. Due to its advantages in high spectral efficiency and high connection density, SCMA is considered an effective way to solve the challenges of massive access in mMTC.

[0003] In mMTC scenarios, to address the random access channel congestion caused by the concurrent access of massive numbers of devices, artificial intelligence technologies are often introduced. Among them, reinforcement learning, especially Q-Learning (QL) algorithms, is widely used in wireless resource management because it does not require an accurate channel model and is easy to implement in a distributed manner. In such schemes, each MTC device can be regarded as an independent agent that autonomously learns the optimal resource selection strategy through interaction with the network environment and trial and error.

[0004] Existing technologies already include solutions that combine SCMA with Q-Learning (SCMA-QL) to leverage the self-learning capabilities of Q-Learning to optimize SCMA resource allocation. However, in practical applications, existing SCMA-QL solutions still have core shortcomings in high-density access scenarios: Since each agent learns based on local information and limited feedback, with the goal of maximizing its own reward (e.g., reward based on signal-to-interference-plus-noise ratio SINR), this "self-interested" learning behavior will lead multiple agents to jointly discover and converge to a few "high-quality" resource blocks with high long-term expected returns in the long-term game. This profit-seeking behavior ultimately leads to the premature solidification of strategies among all agents, resulting in new and more concentrated resource collisions. This phenomenon, known as the "Tragedy of the Commons" in multi-agent learning, causes resource utilization to decrease under high load or saturated competition, while collision rate and packet loss rate rise sharply, ultimately limiting the overall system's speed and throughput.

[0005] To address this, the present invention proposes a multi-agent reinforcement learning SCMA network resource allocation method. By constructing a distributed intelligent resource allocation mechanism and effectively guiding SCMA-QL agents to learn cooperative strategies, the "tragedy of the commons" phenomenon is avoided. By minimizing collisions and maximizing the global performance of the system in saturated competition scenarios, the above-mentioned technical problems are solved. Summary of the Invention

[0006] The main objective of this invention is to provide a resource allocation method for multi-agent reinforcement learning SCMA networks, in order to solve the "tragedy of the commons" problem caused by the "self-interested" learning of agents in the existing SCMA-QL schemes mentioned in the background art, namely the technical problems of resource collision concentration and system performance degradation under high load.

[0007] The present invention solves the above-mentioned technical problems by adopting the following technical solutions: A resource allocation method for multi-agent reinforcement learning SCMA networks includes: S1. Designate multiple active machine-type communication devices as independent Q-learning agents, and each agent selects an SCMA resource block for uplink data transmission based on its locally maintained Q-table; S2. A group of base stations receives uplink data transmission signals from multiple intelligent agents and determines the number of competitors on each SCMA resource block. And calculate a final reward consisting of a basic reward and a direct collision penalty for each agent participating in the transmission; S3. The base station feeds back the corresponding final reward to each agent through the downlink channel; S4. Each agent updates its locally maintained Q-table based on the received final reward until the Q-table converges or reaches a preset number of training frames.

[0008] Preferably, in step S1, all Q-learning agents are located in the same three-dimensional spherical shell space, where the base station is set as the origin of the coordinate system. For randomly and uniformly distributed agents, a channel model is formed by combining the 3D path loss model of the high-rise building center scene with the Rayleigh fading model. Preferably, during the uplink data transmission of the SCMA resource block, for multiple time slots of the transmission frame, the size of the specified time slot group is determined. Divide it into One time slot group; Within each time slot group, by time slot group size and the number of non-zero elements in each codebook Calculate and obtain the number of selectable SCMA codebooks. ,have: ,in For about and The combinational expression; exist Multiplexing on each time slot group A codebook, calculated to obtain Each logical resource block is used to construct the action space of all agents, and each logical resource block is indexed by a unique pair of codebook indices. With time slot group index The combination is determined. ， ; In intelligent agents In the locally maintained Q table, set the size of the Q table to... Used to store each action pair in the codebook With time slot group The expected return Q value, and the initial values ​​of the Q table are all set to 0; During the final transmission, a greedy strategy is used to select the action with the highest current Q value. It then uses the corresponding SCMA resource block to transmit data packets to the base station.

[0009] Preferably, the formula for calculating the direct collision penalty in step S2 is:

[0010] in, As an adjustable penalty factor, The system baseline success reward represents the average reward a single device can receive for successfully transmitting data once under interference-free conditions. It is a dimensionless normalized penalty coefficient.

[0011] Preferably, the normalized penalty coefficient The optimal value is determined through a two-stage search strategy, including: Phase 1: Quickly locate the optimal solution range through large-span coarse-grained scanning; The second stage involves performing a fine-grained search with small steps within the optimal solution range to quickly determine the optimal value.

[0012] Preferably, in step S2, the basic reward is determined based on the signal-to-interference-plus-noise ratio (SINR) of the received signal, wherein: When transmission is successful, the signal-to-interference-plus-noise ratio (SIR) is... A positive reward is given when the threshold is not lowered. ; When transmission fails, i.e., signal-to-interference-plus-noise ratio When the value falls below the preset threshold, a negative reward is obtained. ; As a basic reward.

[0013] Preferably, in step S4, the agent receives the final reward fed back by the base station. Then, the local learning update of table Q is performed as follows: Equipment usage final reward Update the corresponding actions in its Q-table according to the Bellman equation. Q value ,have:

[0014] in, It represents all possible actions in the Q-table. and The learning rate and discount factor are preset.

[0015] A multi-agent reinforcement learning (SCMA) network resource allocation system, comprising the steps of executing any of the aforementioned multi-agent reinforcement learning (SCMA) network resource allocation methods, characterized in that it includes: The decision module is used to select a set of SCMA resource blocks based on the local Q table; The detection module is used to receive uplink signals and determine the number of competitors on each SCMA resource block; A reward calculation module, connected to the detection module, is used to receive the number of competitors and calculate a final reward for each device, including a direct collision penalty, based on the transmission signal quality and the number of competitors. The feedback module is used to broadcast the final reward to the downlink channel; Learning update module: used to update the local Q table based on the received final reward; The detection module, reward calculation module, and feedback module are configured in a unified base station, while the decision-making module and learning update module are configured in each MTC device.

[0016] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the multi-agent reinforcement learning SCMA network resource allocation method described above based on the multi-agent reinforcement learning SCMA network resource allocation system.

[0017] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the above-described multi-agent reinforcement learning SCMA network resource allocation method based on the multi-agent reinforcement learning SCMA network resource allocation system.

[0018] As can be seen from the above technical solution, the present invention provides a resource allocation method for multi-agent reinforcement learning SCMA networks. Compared with the prior art, the present invention has the following advantages: 1. This invention, based on the traditional SINR-based reward (i.e., basic reward), introduces a collision penalty term directly related to the number of competitors on the resource block. This penalty term serves as an explicit cooperation signal, applying direct negative feedback to self-interested behaviors that lead to collisions, effectively suppressing the self-interested behavior of agents, thereby solving the "tragedy of the commons" problem in distributed multi-agent learning.

[0019] 2. This invention constructs a penalty mechanism through collision penalty terms, which can guide the agent to form a more balanced global resource occupancy distribution with fewer conflicts, thereby significantly reducing the packet loss rate and collision rate of the system in near-saturation and overload scenarios, and correspondingly improving the system speed and throughput, enhancing the stability and reliability of the system in high-density scenarios.

[0020] 3. In constructing the penalty mechanism, this invention uses a normalized penalty factor, which makes the penalty intensity easily adjustable according to the average reward level of the current system environment. Combined with the use of a two-stage search strategy to determine the optimal penalty coefficient, this invention can further improve the versatility and stability of the overall operation algorithm under different network parameters.

[0021] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description

[0022] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the operation flow of the method of the present invention; Figure 2 This is a schematic diagram of the overall structural system model of the motion control mechanism of the present invention; Figure 3 A schematic diagram showing the system throughput comparison for verifying the effectiveness of the mechanism of this invention; Figure 4 A schematic diagram showing the comparison of system packet loss rates for verifying the effectiveness of the mechanism of the present invention; Figure 5 This is a schematic diagram of the coarse-grained scanning results for determining the optimal normalization penalty coefficient in this invention; Figure 6This is a schematic diagram illustrating the fine-grained search results for determining the optimal normalization penalty coefficient in this invention. Figure 7 This is a comparison chart of the macroscopic packet loss rate of the system with and without a penalty strategy under the optimal penalty coefficient configuration in this embodiment of the invention. Figure 8 This is a comparison chart of the system and rate macroscopic performance of the present invention with and without a penalty strategy under the optimal penalty coefficient configuration. Figure 9 This is a heatmap comparing the resource usage patterns of the penalty strategy and the non-penalty strategy under stable conditions in an embodiment of the present invention. Figure 10 This is a histogram of resource collisions under a stable state with a penalty strategy in this embodiment of the invention; Figure 11 This is a histogram of resource collisions under a steady state using the no-penalty strategy in this embodiment of the invention. Figure 12 This is a histogram of resource collisions under global statistics with a penalty strategy in this embodiment of the invention; Figure 13 This is a histogram of resource collisions under global statistics for the no-penalty strategy in this embodiment of the invention; Figure 14 This is a comparison chart of the average Q-value convergence during the learning process of the penalized and non-penalized strategies in this embodiment of the invention; Figure 15 This is a comparison diagram of the average Q-table entropy evolution during the learning process of penalized and non-penalized strategies in embodiments of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] For details in the embodiments, please refer to Figures 1 to 15 .

[0025] The multi-agent reinforcement learning SCMA network resource allocation method proposed in this invention improves the traditional reward function, which is based solely on transmission quality, into a composite reward function that simultaneously includes transmission quality (basic reward) and resource contention level (direct collision penalty). This method first constructs a communication system model to describe the operating environment of this invention, referencing relevant 3D SCMA network research.

[0026] In the specific implementation process, consider a 3D single-cell uplink SCMA communication scenario, where the base station is located at the origin (0,0,0). A massive number of MTC devices act as intelligent agents, randomly and uniformly distributed around a base station (BS) with a minimum radius of [missing information]. Maximum radius is Within the three-dimensional spherical shell space.

[0027] equipment European distance to BS The probability density function is calculated as follows: ; At this point, the channel model consists of large-scale fading and small-scale fading. Large-scale fading follows a 3D path loss model suitable for the central scenario of a high-rise building. The path gain is expressed as ,in It is a composite coefficient, derived from the reference distance. Path loss Path loss index and attenuation parameters Jointly decided, specifically Small-scale fading adopts the Rayleigh fading model, and the channel coefficients follow a complex Gaussian distribution with a mean of 0 and a variance of 1.

[0028] The SCMA resource framework is defined as follows: a transport frame contains a total of Each time slot Each time slot ( A time slot group is formed by (the size of the time slot group), totaling [number] slots. There are several time slot groups. Within each time slot group, there are... Several SCMA codebooks are available, among which The value is determined by the size of the time slot group. and the number of non-zero elements in each codebook Joint decision, that is .

[0029] It should also be noted at this point that, through... Multiplexing on each time slot group The system provides a total of codebooks. Each logical resource block (RB) is composed of a unique (codebook index). Time slot group index For the determination, where , .this Each logical resource block constitutes the action space of all MTC device agents.

[0030] Based on this operating environment, such as Figure 1 As shown, the specific operation flow of the method of the present invention includes: Step 1: Perform initialization operations before the algorithm starts. Each active MTC device in the system Each is treated as an independent Q-learning agent, and a local Q-table is initialized, the size of which is [size missing]. Used to store each (codebook) Time slot group The expected reward Q value for the action pair is set to 0 initially.

[0031] At the same time, set the hyperparameters required by the algorithm, including the learning rate. and discount factor .

[0032] In addition, a key parameter is initialized: the penalty factor. To improve the robustness and adaptability of the algorithm, this embodiment preferably uses a normalization method to determine... .

[0033] At this point, the average reward that a device can obtain for successfully transmitting data once is calculated through simulation in an ideal, single-device, interference-free scenario, and this reward is defined as the system baseline success reward. .

[0034] Subsequently, the penalty factor is calculated according to the following normalization formula. :

[0035] in, It is a dimensionless normalized penalty coefficient.

[0036] It can be further explained that... optimal value This can be determined through a two-stage search strategy: the first stage involves a large-scale, coarse-grained scan (e.g., ...). Figure 5 As shown), to quickly locate the approximate range of the optimal solution; the second stage performs a fine-grained search with small steps within the range (e.g., Figure 6(as shown), to accurately determine the optimal value In this embodiment, It was determined to be 0.20.

[0037] Step 2: After initialization, the algorithm performs the main loop, which iterates according to the transmission frame.

[0038] It can be further explained that, during the data transmission process in this embodiment, the following steps are performed sequentially for each frame of the transmission frame: L21, Equipment Decision and Transmission At the beginning of each transmission frame, each active MTC device As an intelligent agent, it independently executes decisions. Device Based on its local Q-table, a greedy strategy is used to select the action with the largest current Q value. After selecting the action, the device... Use the corresponding SCMA resource block (codebook) Time slot group (Transmit data packets to the base station.)

[0039] L22, Base Station Processing and Reward Calculation After receiving uplink signals from all devices in a transmission frame, the base station performs processing and evaluation.

[0040] First, the base station traverses all Each resource block is counted. The number of signals received indicates the number of competitors for that resource block. .

[0041] Secondly, the base station transmits signals to each device. Calculate its signal-to-interference-plus-noise ratio and according to Has the preset successful decoding threshold been reached? To determine the base reward .

[0042] Among them, the basic reward The function is:

[0043] Secondly, the base station is based on the previously counted number of competitors. For equipment Calculate the direct collision penalty .

[0044] The computational logic is as follows Figure 1 As shown in the decision box, its mathematical expression is:

[0045] Finally, the base station adds up the basic reward and penalty to obtain the device's result. The final reward :

[0046] L23, base station feedback The base station transmits the calculated final reward via the downlink channel. Feedback to the corresponding device .

[0047] L24, Equipment Q Table Update Each device Received feedback from base station Then, a local learning update is performed. The device uses the reward value to update the corresponding action in its Q-table according to the Bellman equation. Q value:

[0048] in, It represents all possible actions in the Q table.

[0049] The algorithm repeats the above transmission frame loop, i.e. steps L21 to L24, until the Q-table converges or the preset number of training frames is reached.

[0050] Based on the above embodiments, the parameters used in this invention are subjected to Monte Carlo simulation by a computer to verify the effect. At this time, the simulation parameters all adopt typical values ​​that are recognized in the field.

[0051] In this performance verification comparison, the no-penalty strategy (SCAM-QL / i.e., the benchmark multi-agent Q-Learning algorithm) is equivalent to a special case with a penalty value of 0, where agents only pursue maximizing their individual rates. In contrast, the two-step random access strategy (Two-Step RA / a non-learning benchmark algorithm) involves all active devices accessing the system from... Randomly select one from the available resources.

[0052] Simulations of the conventional strategy (no-penalty strategy) and the present invention were performed, employing a bounded active contention set modeling method to accurately evaluate the resource allocation efficiency of the present invention's method under saturated contention scenarios. This modeling method effectively simulates saturated contention scenarios where the instantaneous number of active devices is controlled near the system capacity under mechanisms such as Access Class Barring (ACCB). This allows the simulation to eliminate interference caused by extreme overload, thus more focusedly verifying the performance advantages of the present invention's DCP mechanism in guiding admitted devices to internal coordination and avoiding the "tragedy of the commons" under continuous, high-density contention. When the total number of potential devices Less than or equal to the system logical resource capacity (in this embodiment, the system logical resource capacity) When ), all All devices are active and participate in resource contention for all transmitted frames.

[0053] When the total number of potential devices Exceeding system capacity (i.e.) At the beginning of each independent simulation round, it will start from... 150 devices are randomly selected from a pool to form a fixed active competition set. In all subsequent transmission frames of this simulation round, these 150 active devices will participate in Q-learning decision-making and resource competition.

[0054] In the specific simulation process: (1) Validation of the DCP mechanism First, select a set of unoptimized, representative penalty values ​​( The present invention is compared with two benchmark algorithms. Figure 3 and Figure 4 The system throughput and packet loss rate were shown as a function of the number of devices. The changes.

[0055] In all load regions, both Q-Learning-based strategies demonstrated superior performance, with throughput far exceeding that of traditional two-step random access.

[0056] In the transition and saturation regions ( The throughput of the method described in this invention begins to exceed that of the no-penalty strategy. This initially demonstrates that the DCP mechanism reduces transmission failures by actively guiding agents to avoid conflicts.

[0057] Packet loss rate best highlights the advantages of the DCP mechanism. Under low load ( Both Q-Learning strategies achieved packet loss rates close to 0. In critical high-load areas (…),… In the no-punishment strategy, collisions intensify due to the agent's "selfish" behavior, and the packet loss rate begins to rise sharply.

[0058] In contrast, the packet loss rate of the method of this invention remains at an extremely low level. In the saturation region ( The method of this invention stabilizes the packet loss rate at 2.5%, while the packet loss rate of the no-penalty strategy is as high as 5.6% (more than twice that of the method of this invention), and the packet loss rate of the two-step random access is as high as 63.1%.

[0059] The above comparison powerfully demonstrates that even with unoptimized penalty values, the DCP mechanism can effectively suppress the "tragedy of the commons" phenomenon, significantly reduce packet loss rate, and greatly enhance system reliability when the network is close to saturation.

[0060] (2) Determination of the optimal normalization penalty coefficient To maximize the performance of the DCP mechanism, it is necessary to determine the optimal penalty factor. This invention further employs a two-stage search strategy in the most representative overload scenario ( Find the optimal value.

[0061] To quickly locate the approximate range of the optimal coefficients, we first perform a logarithmic scale... The values ​​underwent a series of large-span scan tests (0.1, 0.5, 1, 2, 5, 10, 50, 100), such as Figure 5 As shown. The results show that the penalty is too small (e.g. ) or the punishment is too severe (e.g.) Both of these will lead to performance degradation. System and speed are... It reached its peak value and successfully locked the optimal range within [0.1, 1.0].

[0062] Based on the results of the coarse-grained scan, a linear fine-grained search is performed within the located interval [0.1, 1.0], such as... Figure 6 As shown, it can be clearly seen that within this range, the system and speed generally remain at a high level above 3.9 kbit / s / Hz, verifying the accuracy of coarse-grained scanning positioning. After fine searching, the speed and speed are... The global maximum value reached in this experiment was 3.985 kbit / s / Hz.

[0063] The optimal normalization penalty coefficient was finally determined through a two-stage search. All subsequent comprehensive performance analyses will use this optimal configuration.

[0064] (3) Macroscopic performance comparison and verification Figure 7 and Figure 8 This demonstrates the use of the optimal penalty coefficient ( After that, the packet loss rate and speed performance of the method of the present invention are compared with those of the no-penalty strategy.

[0065] In the low load area ( With ample resources, the two algorithms have almost identical performance.

[0066] In high load areas ( As resource competition intensifies, the advantages of the method of this invention become apparent. The packet loss rate of non-penalty strategies begins to rise sharply with increasing load. However, the method of this invention, with its collision avoidance mechanism, achieves a higher packet loss rate even when... Even under intense competition, it can still maintain a very low packet loss rate.

[0067] Because the DCP mechanism effectively suppresses collisions, the system and speed of the method of this invention can be maintained at a higher level (about 4 kbit / s / Hz) under high load, while the performance of the non-penalty strategy stagnates significantly (about 3.8 kbit / s / Hz) due to the increased collisions.

[0068] (4) Microscopic mechanism analysis The improvement in macro performance stems from a fundamental improvement in the efficiency of micro resource allocation.

[0069] like Figure 9 As shown in the heatmap, in Under stable conditions, the method of this invention achieves near-perfect resource allocation. 150 active devices are evenly distributed across 150 independent resource blocks, forming an ideal mode with no conflicts (each device has an occupancy count of "1").

[0070] The no-penalty strategy resulted in obvious "hot spots" (yellow, 2 devices colliding) and "cold spots" (purple, 0 devices occupying), leading to resource waste and performance loss.

[0071] Figure 10 and Figure 11 This difference was statistically quantified. Under steady-state conditions, the method of this invention achieved 150 successful transmissions (100% success rate) with zero collisions and zero wasted resources. In contrast, the no-penalty strategy only achieved 148 successes, resulting in one collision and one wasted resource. Figure 12 and Figure 13 From a global statistical perspective, throughout the entire simulation period, the DCP mechanism strictly controls the resource collision rate to an extremely low level of 0.58%, while the collision rate of the non-penalty strategy is as high as 1.37%, which is more than twice that of the DCP. The method of this invention reduces collision events by more than 59%.

[0072] Figure 14 and Figure 15 The differences in the learning process were revealed. The average Q-value of the method in this invention converges faster and to a higher Q-value level than the no-penalty policy. This indicates that the DCP mechanism guides the agent to find a policy with better long-term rewards. Simultaneously, the Q-table entropy of the method in this invention decreases faster and converges to a lower level, suggesting that the DCP mechanism helps the agent lock in an efficient resource selection policy more quickly and confidently, reducing decision-making uncertainty.

[0073] In summary, simulation experiments demonstrate that the DCP mechanism, by applying precise negative feedback to collisions, effectively guides distributed agents to avoid the "tragedy of the commons," resulting in a social strategy with fewer conflicts and more even distribution. This improvement in efficiency at the microscopic level ultimately translates into a lower packet loss rate and higher system and speed at the macroscopic level for the method of this invention.

[0074] On the other hand, this invention also discloses a multi-agent reinforcement learning SCMA network resource allocation system based on direct collision penalty, which is a hardware implementation of the above method, including: (1) Decision module: Configured in each MTC device. This module is used to store the local Q table and make decisions based on the Q table, select an SCMA resource block, and implement the function of S1 in the above method.

[0075] (2) Detection module: Configured in the base station. This module is used to receive uplink signals and determine the number of competitors on each SCMA resource block. This provides the necessary input for the reward calculation module.

[0076] (3) Reward Calculation Module: Configured in the base station and connected to the detection module. This module is used to calculate rewards based on the quality of the transmitted signal and the number of competitors. Calculate a final reward for each device that includes a direct collision penalty. .

[0077] (4) The detection module and the reward calculation module together implement the function of S2 in the above method.

[0078] (5) Feedback Module: Configured in the base station and connected to the reward calculation module. This module is used to transmit the calculated final reward through the downlink channel. Send to the corresponding MTC device This achieves the function of S3 in the above method.

[0079] (6) Learning and updating module: Configured in each MTC device. This module is used to receive the final reward from the feedback module, execute the Bellman equation based on the reward, update the local Q table, and implement the function of S4 in the above method.

[0080] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor (e.g., a processor of a base station or a processor of an MTC device), causes the processor to perform the steps of the method described above.

[0081] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0082] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the multi-agent reinforcement learning SCMA network resource allocation methods described in the above embodiments.

[0083] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above method.

[0084] This invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. Memory, used to store computer programs; When the processor executes the program stored in memory, it implements the above-mentioned multi-agent reinforcement learning SCMA network resource allocation method.

[0085] The communication bus mentioned in the aforementioned electronic devices can be a peripheral component interconnection standard multi-agent reinforcement learning SCMA network resource allocation method bus or an extended industry standard structure multi-agent reinforcement learning SCMA network resource allocation method bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0086] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0087] The memory may include a random access memory (RAM) multi-agent reinforcement learning (SCMA) network resource allocation method, or a non-volatile memory (NRAM) multi-agent reinforcement learning (SCMA) network resource allocation method, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0088] The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU) multi-agent reinforcement learning SCMA network resource allocation method, a network processor multi-agent reinforcement learning SCMA network resource allocation method, etc.; it can also be a digital signal processor (DSP) multi-agent reinforcement learning SCMA network resource allocation method, an application-specific integrated circuit (ASIC) multi-agent reinforcement learning SCMA network resource allocation method, a field-programmable gate array (FPGA) multi-agent reinforcement learning SCMA network resource allocation method, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0089] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0091] Furthermore, it should be noted that if any directional indication (such as up, down, left, right, front, back, etc.) is involved in the embodiments of the present invention, the directional indication is only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0092] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, in the embodiments of this invention, "multiple" refers to two or more. Moreover, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A resource allocation method for a multi-agent reinforcement learning (SCMA) network, characterized in that, include: S1. Treat the specified machine-type communication device as an independent Q-learning agent, and each of them selects an SCMA resource block for uplink data transmission based on the locally maintained Q table; S2. A group of base stations receives uplink data transmission signals from multiple intelligent agents and determines the number of competitors on each SCMA resource block. And calculate a final reward consisting of a basic reward and a direct collision penalty for each agent participating in the transmission; S3. The base station feeds back the corresponding final reward to each agent through the downlink channel; S4. Each agent updates its locally maintained Q-table based on the received final reward until the Q-table converges or reaches a preset number of training frames.

2. The resource allocation method for multi-agent reinforcement learning SCMA networks as described in claim 1, characterized in that, During the uplink data transmission of the SCMA resource block, for multiple time slots of the transmission frame, the size of the specified time slot group is determined. Divide it into One time slot group; Within each time slot group, by time slot group size and the number of non-zero elements in each codebook Calculate and obtain the number of selectable SCMA codebooks. ,have: ,in For about and The combinational expression; exist Multiplexing on each time slot group A codebook, calculated to obtain Each logical resource block is used to construct the action space of all agents, and each logical resource block is indexed by a unique pair of codebook indices. With time slot group index The combination is determined. ， ; In intelligent agents In the locally maintained Q table, set the size of the Q table to... Used to store each action pair in the codebook With time slot group The expected return Q value is calculated, and the initial values ​​of the Q table are all set to 0; During the final transmission, a greedy strategy is used to select the action with the highest current Q value. It then uses the corresponding SCMA resource block to transmit data packets to the base station.

3. The multi-agent reinforcement learning SCMA network resource allocation method as described in claim 2, characterized in that, The formula for calculating the direct collision penalty in step S2 is as follows: in, As an adjustable penalty factor, The system baseline success reward represents the average reward a single device can receive for successfully transmitting data once under interference-free conditions. It is a dimensionless normalized penalty coefficient.

4. The multi-agent reinforcement learning SCMA network resource allocation method as described in claim 1, characterized in that, In step S2, the basic reward is determined based on the signal-to-interference-plus-noise ratio (SINR) of the received signal, wherein: When transmission is successful, the signal-to-interference-plus-noise ratio (SIR) is... A positive reward is given when the threshold is not lowered. ; When transmission fails, i.e., signal-to-interference-plus-noise ratio When the value falls below the preset threshold, a negative reward is obtained. ; As a basic reward.

5. The multi-agent reinforcement learning SCMA network resource allocation method as described in claim 1, characterized in that, In step S4, the agent receives the final reward from the base station. Then, the local learning update of table Q is performed as follows: Equipment usage final reward Update the corresponding actions in its Q-table according to the Bellman equation. Q value ,have: in, It represents all possible actions in the Q-table. and The learning rate and discount factor are preset.

6. A multi-agent reinforcement learning (SCMA) network resource allocation system, used to perform the steps of the method as described in any one of claims 1 to 5, characterized in that, include: The decision module is used to select a set of SCMA resource blocks based on the local Q table; The detection module is used to receive uplink signals and determine the number of competitors on each SCMA resource block; A reward calculation module, connected to the detection module, is used to receive the number of competitors and calculate a final reward for each device, including a direct collision penalty, based on the transmission signal quality and the number of competitors. The feedback module is used to broadcast the final reward to the downlink channel; Learning update module: used to update the local Q table based on the received final reward; The detection module, reward calculation module, and feedback module are configured in a unified base station, while the decision-making module and learning update module are configured in each MTC device.

7. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 5, based on the system of claim 6.

Images