Dynamic collaborative cluster selection method and device based on multi-agent deep reinforcement learning

Abstract

The disclosed embodiment is about a method and device for dynamic collaborative cluster selection based on multi-agent deep reinforcement learning. The method includes: establishing a system model; decomposing the global state action value Q function; collecting the state, action, reward and next state of the agent as the training data of the network, and storing them in the buffer; training the Q network; using the optimal collaborative cluster selection scheme to serve the user. The disclosed embodiment combines a multi-agent system with a deep reinforcement learning algorithm to solve the combinatorial optimization problem between AP and users in a decellularized large-scale MIMO system. The value function decomposition network is used to improve the environmental non-stationarity problem of the multi-agent system. The non-steady-state environment caused by independent learning of the agent is avoided by value decomposition, the amount of calculation is reduced, and the training process of each agent is more targeted. The best collaborative cluster is selected to serve multiple users in the system at the same time, so that the spectrum efficiency of the system is improved.

Description

Background Art

[0002] Decellularized massive MIMO (Multiple-Input Multiple-Output) technology is one of the key technologies of the sixth-generation mobile communication system. Although its distributed layout of access points (APs) can effectively combat path loss and improve coverage, it comes at the cost of huge backhaul overhead, which not only wastes resources but also increases interference between users.

[0003] To solve this problem, a user-centric approach is introduced into the decellularized massive MIMO system, which alleviates the problem of backhaul link bandwidth usage while providing a higher achievable rate for most users in the system.

[0004] The user-centric concept can be described as a dynamic collaborative cluster selection technology, that is, multiple APs collaborate to form a cluster to serve one user, and adjacent clusters can partially overlap when serving multiple users. Dynamic collaborative cluster selection technology can significantly improve the energy efficiency of the system and reduce the complexity of signal processing, thus becoming a key technology for decellularized massive MIMO systems.

[0005] Existing AP cooperation technologies mainly include AP cooperation methods based on maximum scale fading coefficient and received power. Although these methods improve the energy efficiency of the system, they limit the number of APs that provide services to each user and do not take into account the interference problem between users, thus affecting the further improvement of system performance.

[0006] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0008] The purpose of the embodiments of the present disclosure is to provide a method and device for dynamic collaborative cluster selection based on multi-agent deep reinforcement learning, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.

[0009] According to a first aspect of an embodiment of the present disclosure, a dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning is provided, comprising the following steps:

[0010] Step S100, establishing a system model: establishing a multi-agent deep learning system model according to the decellularized massive MIMO system, and defining the state space, action space and instant reward of the system model;

[0011] In step S100, the action space A of the system model k ={a j |j=1,...ζ k} is the combination of each choice of user k in the access point candidate set, ζ k is the number of actions, a j is an optional action vector of the agent, and each element value in the vector is 0 or 1, wherein 1 represents that the corresponding access point serves user k, and 0 represents that the corresponding access point does not serve user k;

[0012] Step S200, decomposing the global state action value Q function;

[0013] Step S300, collecting the state, action, reward and next state of the agent as training data of the network and storing them in a buffer;

[0014] Step S400, training the Q network to obtain the best collaborative cluster selection solution in the current state;

[0015] Step S500: using the optimal collaborative cluster selection scheme to provide services for users.

[0016] In an exemplary embodiment of the present disclosure, the number of access points selected by the user is regarded as different state values, and the state of the agent is S k ={s l |l=1,...Ω k}, where Ω k is the number of states.

[0017] In an exemplary embodiment of the present disclosure, the reward function of the instant reward is: Where R k is the spectrum efficiency of user k.

[0018] In an exemplary embodiment of the present disclosure, the reward function of the instant reward is: Where B is the bandwidth, P 总 is the total power, R k is the spectrum efficiency of user k.

[0019] In an exemplary embodiment of the present disclosure, in step S200, the global state action value Q function Q is decomposed by a value function decomposition method. total Linear decomposition is performed, and the number of decomposed sub-value functions corresponds to the number of users K in the decellularized massive MIMO system. The sub-value function Q of each agent is k The spectrum efficiency function R corresponding to each user k .

[0020] In an exemplary embodiment of the present disclosure, step S300 includes the following steps:

[0021] Step S301, generating a channel data set by decellularizing a massive MIMO system model;

[0022] Step S302, initializing the state, executing different actions according to the collaborative cluster selection strategy, selecting the action with the highest probability and receiving the immediate reward and transferring to the next state;

[0023] Step S303, storing the state, action, reward and next state of the agent into the experience playback buffer;

[0024] Step S304, determine whether the current state is a terminal state or whether the maximum number of iterations has been reached. If so, execute step S400; if not, use the next state as input data of the neural network and return to step S302.

[0025] In an exemplary embodiment of the present disclosure, step S400 includes the following steps:

[0026] Step S401, uniformly and randomly sampling sample data points from the buffer;

[0027] Step S402, calculating the loss function and transferring the weights to each agent using the gradient descent method;

[0028] Step S403: The agents each update the sub-value Q function to obtain the best collaborative cluster selection solution in the current state.

[0029] In an exemplary embodiment of the present disclosure, in step S403, the experience replay mechanism is used to uniformly randomly sample data points from the buffer to train the value function estimation network, and the predicted Q value is obtained by the online value function estimation network. At the same time, the target network Q value is calculated by the target network, and then the loss function is updated by the gradient descent method, and the weight coefficient of the value function estimation network is updated by back propagation. After repeating step C, the current network parameters are copied to the target network ω - =ω, to enter the next cycle of iterative calculation, where step C is the target network update cycle.

[0030] In an exemplary embodiment of the present disclosure, the network adopts a centralized training and decentralized updating method. During centralized training, the global value function Q is calculated. total The loss function, and then the gradient propagates the error to each sub-value function Q through back propagation k Perform decentralized training to adjust updates.

[0031] According to a second aspect of an embodiment of the present disclosure, a dynamic collaborative cluster selection device based on multi-agent deep reinforcement learning is provided, comprising:

[0032] A system model building unit, used to build a multi-agent deep learning system model according to the decellularized massive MIMO system, and define a state space, an action space, and an immediate reward of the system model;

[0033] A state-action-value Q function decomposition unit, used for decomposing the global state-action-value Q function;

[0034] A training data collection unit is used to collect the state, action, reward and next state of the agent as the training data of the network and store them in a buffer;

[0035] The best collaborative cluster selection acquisition unit is used to train the Q network and obtain the best collaborative cluster selection scheme in the current state;

[0036] The user service unit is used to provide services for users by using the optimal collaborative cluster selection solution.

[0037] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning described in any one of the above embodiments are implemented.

[0038] According to a fourth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0039] Processor; and

[0040] A memory, configured to store executable instructions of the processor;

[0041] In which, the processor is configured to execute the steps of the dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning described in any of the above embodiments by executing the executable instructions.

[0042] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0043] In the embodiments of the present disclosure, the above-mentioned dynamic collaborative cluster selection method and device based on multi-agent deep reinforcement learning solves the problem of system performance degradation caused by interference between collaborative clusters in decellularized large-scale MIMO systems due to the overlapping of multiple collaborative clusters serving several users. On the basis of the deep reinforcement learning algorithm, a multi-agent system is combined to solve the combinatorial optimization problem between APs and users in decellularized large-scale MIMO systems. The value function decomposition network is further used to improve the environmental non-stationarity problem of the multi-agent system. Through value decomposition, not only can the non-steady-state environment caused by independent learning of agents be avoided, but the amount of calculation is greatly reduced, and the training process of each agent is made more targeted. By using this method, the best collaborative cluster can be selected for multiple users in the system at the same time to provide services, thereby further improving the spectrum efficiency of the system.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0046] Figure 1 The following is a diagram showing the architecture of a de-cellularized massive MIMO system in an exemplary embodiment of the present disclosure;

[0047] Figure 2 A flow chart of a dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning in an exemplary embodiment of the present disclosure is shown;

[0048] Figure 3 A flowchart showing the process of collecting the state, action, reward and next state of an intelligent agent as training data of a network and storing them in a buffer in an exemplary embodiment of the present disclosure is shown;

[0049] Figure 4 A flowchart of a scheme for selecting a best collaborative cluster in a current state by training a Q network in an exemplary embodiment of the present disclosure is shown;

[0050] Figure 5 A schematic diagram showing the structure of a dynamic collaborative cluster selection device based on multi-agent deep reinforcement learning in an exemplary embodiment of the present disclosure is shown;

[0051] Figure 6 An algorithm schematic diagram of value function decomposition based on multi-agent deep reinforcement learning in an exemplary embodiment of the present disclosure is shown;

[0052] Figure 7 A schematic diagram of a program product showing a dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning in an exemplary embodiment of the present disclosure;

[0053] Figure 8 A schematic diagram of an electronic device showing a dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0055] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0056] This example implementation first provides a dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning, referring to Figure 1 As shown in , it is assumed that M candidate APs and K users are randomly distributed in an area. In addition, each AP and user is equipped with an antenna, and all APs are connected to the central processor through a backhaul network. In a time division duplex (TDD) system, it is assumed that all M APs simultaneously serve all K users in the same time-frequency resource, and the channel coefficient h mk is modeled as:

[0057]

[0058] Among them, g m,k is the small-scale fading coefficient that follows independent and identical distribution, β m,k is the large-scale fading coefficient, where m=1,...,M,k=1,...K.

[0059] For the decellularized massive MIMO system, the focus is on its downlink data transmission phase. In the downlink, all M APs send data to the user at the same time, so the received signal y of user k in the downlink k for:

[0060]

[0061] in is additive white Gaussian noise. Define a binary matrix X with M×K dimensions as a collaborative cluster selection scheme. m,k ∈X represents the connection relationship between the mth AP and the kth user:

[0062]

[0063] Then we can get the spectrum efficiency R of user k after dynamic cooperative cluster selection. k for:

[0064]

[0065] The overall spectral efficiency is determined by:

[0066]

[0067] The total power consumption is modeled as:

[0068]

[0069] Among them, P m is the power consumption of the amplifier and circuits at the mth AP (including the power consumption of the transceiver chain and the power consumption of signal processing), and P bh,m is the power consumed by the backhaul link connecting the CPU and the mth AP. Power consumption P m Can be modeled as

[0070]

[0071] Among them, 0<α m ≤1 is the power amplifier efficiency, N0 is the noise power, η mk is the power control coefficient, γ m,k is the mean square value of the minimum mean square error channel estimate, P tc,m is the internal power required to operate the circuit components (e.g., switches, mixers, and filters) associated with each antenna of the mth AP.

[0072] The backhaul is used for data transmission between the AP and the CPU, and its power consumption is proportional to the total spectrum efficiency:

[0073] P bh,m =P 0,m+B·R sum ·P bt,m (1.8)

[0074] Among them, P 0,m is the fixed power consumption (power independent of the service) of each backhaul that may depend on the distance between the AP and the CPU and the system topology, P bt,m is the power associated with the service, and B is the system bandwidth.

[0075] Substituting and into:

[0076]

[0077] The overall energy efficiency is defined as the total throughput divided by the total power consumption in the network. It is expressed as:

[0078]

[0079] Therefore, the objective function can be expressed as:

[0080]

[0081] Please refer to Figure 2 , the dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning can specifically include the following steps:

[0082] Step S100, establishing a system model:

[0083] Reinforcement Learning Model Using Triplet<S,A,R> To represent, where:

[0084] A is the action space: Assuming that each user in the system is an intelligent agent, find the AP subset that maximizes the spectrum efficiency. For user k, firstly transform the channel vector {|h 1,k |,...|h M,k |} are sorted in descending order, and then a threshold δ is set to select the top δ% of its channels as the candidate AP set for the user. Each selection combination of the user k in the AP candidate set is an optional action for the agent. Assume that the size of the candidate AP set is Ω k , then the actions of user k are The action combination of user k can be expressed as:

[0085] A k ={a j |j=1,...ζ k} (1.12)

[0086] S is the state space: when the agent performs an action, its state will change. In the initial stage, the agent randomly selects APs. In the final stage, in order to obtain greater performance gain, the user purposefully selects the AP set. Therefore, the number of APs selected by the user can be regarded as different state values. Here, it is assumed that the state of user k can be expressed as:

[0087] S k ={s l |l=1,...Ω k} (1.13)

[0088] R is the reward function: The reward function is set to the objective function in the formula, that is, the total spectrum efficiency of the decellularized massive MIMO system. In each iteration, the value of the reward function is determined by the behavior of the agent. After the agent takes an action in a certain state, it obtains a reward by calculating the spectrum efficiency function:

[0089]

[0090] Step S200, decomposing the global state action value Q function:

[0091] The idea of ​​the value function decomposition algorithm is: if the number of agents is K, the total team reward function is decomposed into K sub-value functions, and each agent in the group optimizes its own sub-value function according to the contribution made to the group by its own learning strategy. Each sub-value function is the main basis for guiding the actions of each agent. This idea corresponds to the decellularized large-scale MIMO multi-user system, that is, the total spectrum efficiency of the system is decomposed into the single spectrum efficiency of each user. Each user selects a cooperative cluster for communication according to its own situation and obtains the corresponding single-user spectrum efficiency. The selection of the cooperative cluster is adjusted according to its contribution to the total spectrum efficiency. Finally, on the basis of maximizing the total spectrum efficiency of the system, the spectrum efficiency of each user can be maximized as much as possible. The specific method of value function decomposition is:

[0092]

[0093] Among them, the number of agents in the environment is K, Q k is the sub-value function of agent k after decomposition. total Corresponding to the total spectral efficiency R in the decellularized massive MIMO system sum Therefore, the sub-value function Q of the decomposed agent k is k (s k ,a k ) corresponds to the downlink spectrum efficiency R of user k in the system k .

[0094] Step S300, collect the state, action, reward and next state of the agent as training data for the network and store them in the buffer:

[0095] Please refer to Figure 3 , step S300 includes steps S301 to S304:

[0096] Step S301, generating a channel data set {h m,k};

[0097] Step S302, initialize the multi-agent state, execute different actions according to the collaborative cluster selection strategy, select the action with the highest probability, receive the immediate reward and transfer to the next state. In order to better balance the relationship between exploration and utilization, the ε-greedy strategy is finally used to select actions:

[0098]

[0099] Among them, rand is a random value between [0,1], and the size of the ε value determines the exploration or utilization frequency of the agent.

[0100] Step S303: The state, action, reward and next state of the agent are t =(s t ,a t ,r t ,s t+1 ) Store in the experience replay buffer;

[0101] Step S304, determine whether the current state is the terminal state or whether the maximum number of iterations has been reached. If so, execute step S400. Otherwise, the next state s t+1 As the input data of the neural network, return to step S302.

[0102] Step S400, training the Q network, please refer to Figure 4 Step S400 specifically includes the following steps:

[0103] Step S401, using the experience playback mechanism to uniformly randomly sample z sample data from the buffer, where

[0104] Step S402, using the target network to update the Q value, that is, after the online estimated network parameters are updated C steps, the network parameters are copied to the target network, and then the updated target network is used to generate the next C steps of parameter updates. The network parameters of the target network are expressed as ω -, the parameters of the online value function estimation network are expressed as ω, the value function estimation network is updated at each step, and the target network is updated synchronously with it every C steps, making the training more stable, where C steps is the target network update cycle. Calculate the target network:

[0105]

[0106] Step S403, calculate the loss function And use the gradient descent method to pass ω to each agent to update their respective sub-value Q function Q k (s k ,a k ).

[0107] Step S404: reset the parameters of the online value function estimation network and the target network every C steps

[0108] Step S500: Serving users using the best collaborative cluster selection solution.

[0109] In this embodiment, the intelligent agent adopts a centralized training and decentralized updating method. After the training is completed, the network outputs the probability value of each action in the current state, and the corresponding Q value is calculated by executing the action with a larger probability. That is, by updating the respective sub-value Q functions, the best collaborative cluster selection scheme in the current state is obtained.

[0110] The present invention solves the problem that in a decellularized large-scale MIMO system, due to the overlapping of multiple cooperative clusters serving several users, the interference between cooperative clusters leads to the degradation of system performance. On the basis of the deep reinforcement learning algorithm, a multi-agent system is combined to solve the combinatorial optimization problem between APs and users in the decellularized large-scale MIMO system. With the introduction of multi-agents, the problem of algorithm convergence difficulty caused by the non-stationary environment follows. To this end, the value function decomposition network is used to improve the environmental non-stationarity problem of the multi-agent system. Through value decomposition, not only the non-steady-state environment caused by independent learning of agents can be avoided, but also the amount of calculation is greatly reduced, and the training process of each agent is made more targeted. By using this method, the best cooperative cluster can be selected for multiple users in the system at the same time to provide services, thereby further improving the spectrum efficiency of the system.

[0111] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc. In addition, it is also easy to understand that these steps may be, for example, executed synchronously or asynchronously in multiple modules / processes / threads.

[0112] In this example implementation, a dynamic collaborative cluster selection device based on multi-agent deep reinforcement learning is also provided. Figure 5 As shown in , the device may include: a system model building unit 100, a Q function decomposition unit 200, a training data collection unit 300, an optimal cooperative cluster selection acquisition unit 400 and a user service unit 500.

[0113] Wherein: the system model building unit 100 is used to build a multi-agent deep learning system model according to the decellularized large-scale MIMO system, and define the state space, action space and instant reward of the system model;

[0114] The Q function decomposition unit 200 is used to decompose the global state action value Q function;

[0115] The training data collection unit 300 is used to collect the state, action, reward and next state of the agent as the training data of the network and store them in the buffer;

[0116] The optimal collaborative cluster selection acquisition unit 400 is used to train the Q network and obtain the optimal collaborative cluster selection scheme in the current state;

[0117] The user service unit 500 is used to provide services for users by using the optimal collaborative cluster selection solution.

[0118] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0119] In the above embodiments, please refer to the algorithm based on multi-agent deep reinforcement learning value function decomposition Figure 6 .

[0120] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units for concretization. The components displayed as modules or units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed scheme. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0121] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, and when the program is executed by, for example, a processor, the steps of the dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning described in any of the above embodiments can be implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above section of the dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning of this specification.

[0122] refer to Figure 7 As shown, a program product 600 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0123] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0124] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0125] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0126] In an exemplary embodiment of the present disclosure, an electronic device is also provided, which may include a processor and a memory for storing executable instructions of the processor. The processor is configured to execute the steps of the dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning described in any of the above embodiments by executing the executable instructions.

[0127] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0128] Refer to the following Figure 8 The electronic device 600 according to this embodiment of the present invention is described. Figure 8 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0129] like Figure 8 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0130] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present invention described in the above-mentioned section of the method for dynamic collaborative cluster selection based on multi-agent deep reinforcement learning. For example, the processing unit 610 can execute the following steps: Figure 2 Follow the steps shown in .

[0131] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0132] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0133] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0134] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0135] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above-mentioned dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning according to the implementation methods of the present disclosure.

[0136] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A dynamic collaborative cluster selection method based on multi-agent deep reinforcement learning, characterized in that: The following steps are involved: Step S100, establishing a system model: establishing a multi-agent deep learning system model according to the decellularized massive MIMO system, and defining the state space, action space and instant reward of the system model; In step S100, the action space A of the system model k ={a j |j=1,...ζ k } is each combination of choices made by user k in the access point candidate set; Step S200, decomposing the global state action value Q function; Step S300, collecting the state, action, reward and next state of the agent as training data of the network and storing them in a buffer; Step S400, training the Q network to obtain the best collaborative cluster selection solution in the current state; Step S500, using the best collaborative cluster selection scheme to provide services for users; Step S100, establishing a system model: Reinforcement Learning Model Using Triplet<S,A,r> To represent; S is the state space, A is the action space, r is the reward function; Each user is an intelligent agent, and the access point AP subset that maximizes the spectrum efficiency is sought. For user k, the channel vector {h 1,k |,...h M,k \} Sort in descending order, then set a threshold δ, select the top δ% of its channels as the candidate AP set of the user; each selection combination of the user k in the candidate AP set is an optional action of the agent; the size of the candidate AP set is Ω k , then the actions of user k are Each combination of user k’s choices in the access point candidate set is represented as the following action space A k : A k ={a j |j=1,...ζ k }(1.12) In the formula, ζ k is the number of actions, a j is an optional action vector of the agent, and each element value in the vector is 0 or 1, wherein 1 represents that the corresponding access point serves user k, and 0 represents that the corresponding access point does not serve user k; When the agent performs an action, the state of the agent will change. In the initial stage, the agent randomly selects APs, while in the final stage, the user will select a set of APs. The number of APs selected by the user is regarded as a different state value; the state space of user k is expressed as: S k ={s l |l=1,...Ω k }(1.13) In the formula, Ω k is the number of states; In each iteration, the value of the reward function is determined by the behavior of the agent. After the agent takes an action in a state, it obtains an immediate reward by calculating the spectrum efficiency function. The reward function r of the immediate reward is as follows: In the formula, R k is the spectrum efficiency of user k; In step S200, the global state action value Q function Q is decomposed by using the value function decomposition method. total Linear decomposition is performed, and the number of decomposed sub-value functions corresponds to the number of users K in the decellularized massive MIMO system. The total team reward function is decomposed into K sub-value functions. Each agent in the group optimizes its own sub-value function according to the contribution of its own learning strategy to the group. The value function decomposition method is: Among them, the number of agents in the environment is K, Q k is the sub-value function of agent k after decomposition, Q total Corresponding to the total spectral efficiency R in the decellularized massive MIMO system sum , the sub-value function Q of the decomposed agent k k (s k ,a k ) corresponds to the downlink spectrum efficiency R of user k in the system k ; Step S300 includes steps S301 to S304: Step S301, by decellularizing the massive MIMO system model Generate channel data set {h m,k }; where g m,k is the small-scale fading coefficient that follows independent and identical distribution, β m,k is the large-scale fading coefficient, where m=1,...,M,k=1,...K; Step S302, initialize the multi-agent state, execute different actions according to the collaborative cluster selection strategy, select the action with the highest probability and receive the immediate reward and transfer to the next state, and select the action using the ε-greedy strategy: Among them, rand is a random value between [0,1], and the value of ε determines the exploration or utilization frequency of the agent; Step S303: The state, action, reward and next state of the agent are t =(s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer; Step S304, determine whether the current state is the terminal state or whether the maximum number of iterations has been reached. If so, execute step S400. Otherwise, the next state s t+1 As the input data of the neural network, return to step S302; The step S400 includes the following steps: Step S401, using the experience playback mechanism to uniformly randomly sample Z sample data from the buffer, where Step S402, using the target network to update the Q value, that is, after the online estimated network parameters are updated C steps, the network parameters are copied to the target network, and then the updated target network is used to generate the next C steps of parameter updates; the network parameters of the target network are expressed as ω - , the parameters of the online value function estimation network are expressed as ω. The value function estimation network is updated at each step, and the target network is updated synchronously with it every C steps, making the training more stable. Among them, C steps is the target network update cycle. The target network is calculated as: Step S403, calculate the loss function And use the gradient descent method to pass ω to each agent to update their respective sub-value Q function Q k (s k ,a k ); Step S404: reset the parameters of the online value function estimation network and the target network every C steps 2. The method for dynamic collaborative cluster selection based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The network adopts the method of centralized training and decentralized updating. During centralized training, the global value function Q is calculated. total The loss function, and then the gradient propagates the error to each sub-value function Q through back propagation k Perform decentralized training to adjust updates.

3. A dynamic collaborative cluster selection device based on multi-agent deep reinforcement learning, characterized in that: The selection method according to claim 1 is used to perform dynamic collaborative cluster selection, wherein the selection device comprises: A system model building unit is used to build a multi-agent deep learning system model according to the decellularized large-scale MIMO system, and define the state space, action space and instant reward of the system model; wherein the action space A of the system model k ={a j |j=1,...ζ k } is the combination of each choice of user k in the access point candidate set, ζ k is the number of actions, a j is an optional action vector of the agent, and each element value in the vector is 0 or 1, wherein 1 represents that the corresponding access point serves user k, and 0 represents that the corresponding access point does not serve user k; A Q function decomposition unit, used to decompose the global state action value Q function; A training data collection unit is used to collect the state, action, reward and next state of the agent as the training data of the network and store them in a buffer; The best collaborative cluster selection acquisition unit is used to train the Q network and obtain the best collaborative cluster selection scheme in the current state; The user service unit is used to provide services for users by using the optimal collaborative cluster selection solution.

Images