A smart activation method for smart reflective surface elements based on deep reinforcement learning

Abstract

This invention claims protection for an intelligent activation method for intelligent reflective surface elements based on deep reinforcement learning, belonging to the fields of deep reinforcement learning and intelligent reflective surface-assisted communication. It includes system model establishment and target problem establishment, Markov decision model element setting, algorithm framework construction, and network structure design. A system model is established based on the considered communication scenario, and a target problem to be solved is proposed. The setting of Markov decision elements mainly defines the states, actions, and reward functions involved in the interaction between the reinforcement learning agent and the environment. A classic speaker-commentator deep reinforcement learning algorithm framework is adopted, where the policy network outputs actions while the evaluation network assists in the gradient update of the policy network. The network structures of the policy network and the evaluation network are adjusted to address the problem of insufficient channel state information extraction caused by the fully connected structure introduced by the intelligent reflective surface. This method reduces the complexity of iterative methods using traditional communication algorithms.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning and intelligent reflective surface assisted communication, and specifically relates to an intelligent activation method for intelligent reflective surface elements based on deep reinforcement learning. Background Art

[0002] Smart reflective surfaces are currently considered to be an important technology that may be used in future 6G wireless communication systems and is expected to surpass massive multi-antenna technology. Generally speaking, smart reflective surfaces are usually composed of a large number of passive reflective elements, each of which can produce controllable amplitude and phase changes to the incident signal. By densely arranging a large number of smart reflective surfaces in wireless networks and cleverly coordinating their reflection angles, the signal transmission between base stations and user equipment can be flexibly reconfigured to achieve the desired goals, which provides a new means to fundamentally solve the channel fading and interference problems under various factors, and may achieve a quantum leap in wireless communication capacity and reliability.

[0003] The deep reinforcement learning algorithm effectively combines the decision-making ability of reinforcement learning with the perception ability of deep learning, and has achieved a series of milestone events in the field of artificial intelligence in recent years. Most of the current mainstream research on smart reflective surfaces focuses on the phase optimization problem of smart reflective surfaces. However, considering that there are relatively many elements on smart reflective surfaces, the problem of smart reflective surface element activation is also a problem worth considering. Under the premise of ensuring the communication rate, ensuring high-quality smart reflective surface element activation can greatly improve the energy efficiency of the smart reflective surface. In order to address this problem, the present invention proposes a smart reflective surface element activation method based on the combination of deep reinforcement learning and traditional communication algorithms, which can be well used to solve continuous integer optimization problems.

[0004] CN112019249A, a method and device for intelligent reflective surface control based on deep reinforcement learning, wherein the method includes: a policy network generates a first action according to a first state; the amplitude is fixed and input into an optimization module, the first action is updated to obtain a second action, and a first target value is obtained at the same time; the second action is applied to a wireless environment to obtain a second state, a new sample is obtained and stored in an experience pool; the policy network and the value network are trained with DDPG according to the sample, and the executor updates its parameters using a policy gradient method; a third target value is determined according to the first target value and the second target value generated by the target Q network, and the DNN of the online Q network is trained according to the third target value and its parameters are updated; the above steps are repeated until the amplitude of the change in the transmit power is less than a preset threshold, and the network parameters that minimize the AP transmit power are obtained and output. The present invention can achieve stable and efficient learning in a shorter time, and can converge to the optimal target faster.

[0005] This patent mainly focuses on the optimization of base station transmission power, phase and amplitude of smart reflective surface elements, etc. under the condition of satisfying the signal-to-noise ratio constraint. The present invention mainly involves the energy loss problem of smart reflective surfaces. Smart reflective surfaces are usually composed of many passive reflective elements. By adjusting the elements of the smart reflective surface, controllable phase and amplitude changes of the incident signal can be achieved. The present invention takes into account the large number of elements of the smart reflective surface. In order to ensure the communication rate between the base station and the user and further reduce energy consumption, it is assumed that only a given number of smart reflective surface elements can be activated. In order to reduce the complexity of traditional communication algorithms, the present invention proposes a method combining deep reinforcement learning with traditional communication algorithms. The traditional communication algorithm solves the phase optimization of the elements of the smart reflective surface to ensure the communication rate. The reinforcement learning strategy network is responsible for outputting the activation strategy and selecting the most appropriate smart reflective surface element for activation. In the feature extraction of the channel state, bidirectional convolution and dense fully connected structure are adopted to accelerate the convergence speed during network training. Summary of the invention

[0006] The present invention aims to solve the above problems of the prior art. A method for intelligent activation of intelligent reflective surface elements based on deep reinforcement learning is proposed. The technical solution of the present invention is as follows:

[0007] A method for intelligent activation of intelligent reflective surface elements based on deep reinforcement learning, comprising the following steps:

[0008] S1: Establish a system model based on the actual communication scenario, and propose the target problem to be solved based on the system model;

[0009] S2: Abstract the goal-solving problem into a Markov decision problem, and define the basic elements involved in the interaction between the reinforcement learning agent and the environment, including the setting of actions, states, and reward functions;

[0010] S3: Build a deep reinforcement learning algorithm based on the classic speaker-critic framework, including a policy network and an evaluation network. The policy network is used to output the action decision of the agent, and the evaluation network is used to evaluate the actions taken by the policy network in the current state and provide the gradient for updating the policy network on this basis.

[0011] S4: Redesign the network structure and use bidirectional convolution and densely connected fully connected structures to extract channel state information. The channel state involves factors such as the location relationship between the base station, smart reflective surface and user, the degree of mutual interference and signal attenuation. Therefore, a neural network is used to fit the mapping relationship between the channel state and the element activation of the smart reflective surface, thereby completing the intelligent activation of the elements of the smart reflective surface.

[0012] Furthermore, the intelligent reflecting surface is a passive intelligent reflecting surface, and the main function of the intelligent reflecting surface is to serve as a relay device between the base station and the user equipment to assist the communication between the two.

[0013] Further, the step S1: according to the actual scenario where there is no line-of-sight communication between the base station and the user due to obstruction, an intelligent reflective surface is deployed between the base station and the user, a system model of intelligent reflective surface assisted communication is established, and a target problem to be solved is proposed according to the system model, specifically including:

[0014] The transmission rate of the Kth user equipment can be expressed as:

[0015]

[0016] in, Φ represents the smart reflection surface coefficient matrix, represents the mean square error between the information received by the receiver and the information sent by the sender, p t is the user transmit power, σ 2 represents the Gaussian noise power, I M represents the standard unit matrix, h d,k 、h r,k and G represent the channel states from the base station to the user equipment, from the smart reflecting surface to the user equipment, and from the smart reflecting surface to the base station, respectively.

[0017] Furthermore, the step S2: abstracts the target solution problem into a Markov decision problem, defines the basic elements involved in the process of interaction between the reinforcement learning agent and the environment, including the setting of actions, states and reward functions, specifically including:

[0018] In the establishment of the Markov decision model, the state is set to the channel state at the current moment. The channel state information includes the real part and the imaginary part. The action of the strategy is a one-dimensional vector v of the number of smart reflective surface elements. Assuming that the number of smart reflective surface elements is N, the action at any time t is expressed as:

[0019] v(t)=[v1(t),v2(t),…v N (t)]

[0020] The reward reflects the performance of the target problem. According to the user's transmission rate formula, the sum of the rates of all user devices is used as the reward function, which can be expressed as:

[0021]

[0022] Furthermore, step S3 constructs a deep reinforcement learning algorithm based on the classic speaker-critic framework, including a policy network and an evaluation network. The policy network is used to output the action decision of the intelligent agent, and the evaluation network is used to evaluate the action taken by the policy network in the current state and provide a gradient for updating the policy network on this basis, specifically including:

[0023] The strategy network as a whole consists of three modules. Module 1 is a bidirectional convolution, which mainly performs horizontal and vertical convolution on the input link channel, and then the outputs in both directions are convolved by a 1X1 convolution kernel, and finally the outputs in both directions are merged into a one-dimensional vector and input into module 2. Module 2 is mainly a two-layer densely connected fully connected structure. The densely connected structure can make the input of the current layer contain the features of all previous layers, further improving the feature extraction and analysis capabilities of the channel. Module 3 uses a 2-layer fully connected adjustment network output. The output of the strategy network is a one-dimensional probability vector of the number of intelligent reflective surface elements. The greater the probability, the greater the possibility of being activated.

[0024] The evaluation network is also composed of three modules. Module 1 also adopts a bidirectional convolution method. Since the input of the evaluation network also includes the output action of the policy network relative to the policy network, the evaluation network will merge the actions of the policy network after 1X1 convolution and then input into module 2. The structure of module 2 is completely consistent with that of the policy network. Module 3 uses a 2-layer full connection mainly to adjust the output of the network. The output of the evaluation network is a Q value, so a value is output in the end.

[0025] Furthermore, the bidirectional convolution module performs bidirectional convolution on the channel status of each link respectively;

[0026] The module 2 is a fully connected structure with two layers of dense connections. The network width of all fully connected structures is limited to 1024. In the use of activation functions, except for the action output of the strategy network using Sigmoid to represent the probability of activation of the intelligent reflection surface element, the remaining activation functions are ReLu, and the size of the experience replay pool is 2 18 , the learning rate of the neural network is 2 -16 , the variance of the exploration noise is 0.0001, the optimizer uses Adam, and the discount factor is set to 0.

[0027] The advantages and beneficial effects of the present invention are as follows:

[0028] 1. In response to the thorny integer optimization problem of traditional communication algorithms, the present invention proposes a joint optimization method that combines traditional communication algorithms with deep reinforcement learning. The fitting ability of neural networks is used to replace the complex iterations of traditional communication algorithms, and only the phase adjustment of the smart reflector elements is calculated by the traditional communication algorithm.

[0029] 2. According to the structure of channel state, a method combining bidirectional convolution and dense fully connected structure is proposed to extract channel features, which greatly improves the ability to extract channel state information and accelerates the convergence speed during network training. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is an algorithm framework diagram of the overall preferred embodiment provided by the present invention;

[0031] Figure 2 A system model diagram for practical considerations of the present invention;

[0032] Figure 3 This is the overall structure diagram of the strategy network and the evaluation network;

[0033] Figure 4 It is the structural diagram of bidirectional convolution;

[0034] Figure 5 It is a fully connected and close-fitting connection structure diagram;

[0035] Figure 6 Comparison chart of training curves of bidirectional convolution and fully connected structure;

[0036] Figure 7 A training curve comparison diagram for setting different total numbers of smart reflective surface elements under a given number of activated smart reflective surface elements; DETAILED DESCRIPTION

[0037] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0038] The technical solution of the present invention to solve the above technical problems is:

[0039] A method for intelligent activation of intelligent reflective surface elements based on deep reinforcement learning, which is characterized by comprising the following steps:

[0040] S1: Establishing the system model and the target problem to be solved: Establishing the system model according to the actual communication scenario and proposing the target problem to be solved based on the system model;

[0041] S2: Reinforcement learning problems are usually abstracted into Markov decision problems, which define the basic elements involved in the process of reinforcement learning agents interacting with the environment, including the setting of actions, states, and reward functions;

[0042] S3: Build a deep reinforcement learning algorithm based on the classic speaker-critic framework. The algorithm mainly consists of two parts: the policy network is used to output the action decision of the agent, and the evaluation network is used to evaluate the actions taken by the policy network in the current state and provide the gradient for the policy network to update based on this;

[0043] S4: The introduction of intelligent reflection surface brings challenges to the channel feature extraction of strategy network and evaluation network. The network structure is redesigned to complete the extraction of channel state information.

[0044] Preferably, the main function of the intelligent reflective surface is to serve as a relay device between the base station and the user equipment to assist the communication between the two. Therefore, in a multi-user uplink transmission scenario, there is a serious obstruction between the base station and the user equipment and there is no line-of-sight channel. The cascade link of the intelligent reflective surface is used to assist in completing the communication task between the base station and the user equipment, and the channel state generally obeys the Rayleigh channel model. Generally, unless otherwise stated, the intelligent reflective surface involved in the present invention refers to a passive intelligent reflective surface.

[0045] Preferably, in S2, according to the interaction model between the agent and the environment in Markov decision making, at any time t, the agent takes action according to the current state of the environment and obtains an immediate reward, and then the state of the environment is updated to the next time. According to this basic interaction model and the target solving problem, the channel state information at each time is set as the state of the environment, the action is a one-dimensional vector of the number of smart reflective surface elements, which is used to indicate the current smart reflective surface activation strategy, and the reward function is usually used to measure the completion of the goal, so the sum of the communication rates of the user devices is used as the reward.

[0046] Preferably, the deep learning algorithm based on the speaker-critic framework is the current mainstream deep reinforcement learning algorithm. The policy network focuses on the policy itself and outputs the activation policy of the intelligent reflective surface elements according to the current state. The update of the policy gradient alone usually requires the agent to complete the complete exploration trajectory before it can be updated, which often cannot meet the requirements in practice. In order to improve the update speed of the policy network, an evaluation network is added to evaluate the performance of the policy network in real time. On the one hand, while evaluating the policy, it can also prompt the policy network to update in the direction guided by the evaluation network. It is worth mentioning that the phase of the intelligent reflective surface is calculated by the traditional communication algorithm. In the environment of interaction with the intelligent agent, the neural network acts as the decision maker for the activation of the intelligent reflective surface and jointly completes the continuous mixed integer optimization problem.

[0047] Preferably, the introduction of intelligent reflective surfaces greatly increases the number and complexity of the overall channel. In order to improve the network's ability to extract channel features, such as Figure 4 With Figure 5As shown in the figure, an extraction structure based on bidirectional convolution and dense connection is used. The convolution kernel in the bidirectional convolution is selected based on the translation invariance of the channel state, and the convolution kernel related to the shape of the link channel state is selected. The densely connected structure can make up for the problem caused by insufficient network capacity on the one hand, and on the other hand, the densely connected structure can integrate all the previous input features to further improve the network's feature extraction ability.

[0048] The algorithm model used in the present invention mainly includes the following steps:

[0049] S1: Establishing the system model and the target problem to be solved: Establishing the system model according to the actual communication scenario and proposing the target problem to be solved based on the system model;

[0050] S2: Reinforcement learning problems are usually abstracted into Markov decision problems, which define the basic elements involved in the process of reinforcement learning agents interacting with the environment, including the setting of actions, states, and reward functions;

[0051] S3: Build a deep reinforcement learning algorithm based on the classic speaker-critic framework. The algorithm mainly consists of two parts: the policy network is used to output the action decision of the agent, and the evaluation network is used to evaluate the actions taken by the policy network in the current state and provide the gradient for the policy network to update based on this;

[0052] S4: The introduction of intelligent reflection surface brings challenges to the channel feature extraction of strategy network and evaluation network. The network structure is redesigned to complete the extraction of channel state information.

[0053] according to Figure 2 The multi-user uplink multi-input single-output system model enhanced by smart reflectors is considered in the paper. It is assumed that two single-antenna user devices transmit information to a remote data center through a base station equipped with two antennas. Considering that there is no line-of-sight channel between the base station and the user device, communication can only be assisted by the smart reflector between the base station and the user device. Since the target problem to be solved is to complete the high-quality activation of a number of smart reflectors while ensuring the communication rate. Only the necessary formulas are given here to illustrate the problem to be solved. The transmission rate of the Kth user device can be expressed as:

[0054]

[0055] in, Φ represents the intelligent reflection surface coefficient matrix, I M represents the standard unit matrix, h d,k 、h r,k and G represent the channel states from the base station to the user equipment, from the smart reflecting surface to the user equipment, and from the smart reflecting surface to the base station, respectively.

[0056] In the establishment of the Markov decision model, the state is set to the channel state at the current moment. The channel state information includes the real part and the imaginary part. The action of the strategy is a one-dimensional vector v of the number of smart reflective surface elements. Assuming that the number of smart reflective surface elements is N, the action at any time t can be expressed as:

[0057] v(t)=[v1(t),v2(t),…v N (t)]

[0058] The reward usually needs to reflect the performance of the target problem. According to the user's transmission rate formula, the sum of the rates of all user devices is used as the reward function, which can be expressed as:

[0059]

[0060] according to Figure 1 In the overall algorithm framework, the policy network outputs the activation decision of high-quality intelligent reflective surface elements, which is the activation action made by the policy network for the current environmental state. The evaluation network is used to evaluate the action of the current policy network, and the evaluation network guides the policy network to update in the direction of increasing the output of the evaluation network during the update of the policy network, that is, to provide the gradient of the policy network update. At the same time, in order to ensure that the strategy of the policy network maintains a certain degree of exploratory nature in the algorithm, an exploratory noise is added in the process of interactive data collection between the intelligent agent and the environment, so as to serve as a trade-off between exploration and utilization of the intelligent agent. The inputs of the policy network and the evaluation network both contain channel state information. In addition to the channel state information, the input of the evaluation network also requires the action output by the policy network, which is an overall evaluation of the current state-action pair.

[0061] according to Figure 3 The overall network framework in the network, module 1 and module 2 are mainly used to complete the new extraction of channel state, which is the main core of the network. Module 3 mainly solves the demand for output dimension, respectively meeting the adjustment of output dimension of different networks, and will not be introduced in detail. Here we focus on the structure of module 1 and module 2. The network structure in module 1 is mainly as follows Figure 4Bidirectional convolution, compared with the fully connected structure, bidirectional convolution has stronger feature extraction capabilities, helping the algorithm to train faster. The use of bidirectional convolution is mainly to consider the actual translation invariance of the channel state. In practice, there are two main ways to implement bidirectional convolution. One is to splice the channel state information of all links into a regular rectangle, which can easily implement convolution. The other is to perform bidirectional convolution on the channel state of each link separately. Considering that the first convolution method will inevitably bring filling elements during the splicing process, which will cause errors, we adopted the latter method. Specifically, the total number of smart reflective surface elements is set to 24. Taking the channel G from the smart reflective surface to the base station as an example, the entire network process of bidirectional convolution is analyzed in detail. As the name implies, bidirectional convolution mainly performs convolution operations on G in the horizontal and vertical directions using convolution kernels of (24X1) and (1X2) respectively, and then performs a convolution operation with a convolution kernel size of (1X1) on the outputs in both directions respectively, with the number of channels being 512 and 256 respectively, and then the two outputs are merged according to a certain output dimension and sent to module 2.

[0062] Module 2 adopts the following Figure 5 The 2-layer densely connected fully connected structure shown in the figure can avoid the problem of insufficient network capacity on the one hand, and on the other hand, because the current layer in the dense connection comprehensively includes the features of all previous layers, the ability to extract channel states is further improved. It is worth mentioning that the network width of all fully connected structures is limited to 1024. In the use of activation functions, except for the action output of the strategy network, which uses Sigmoid to represent the probability of activation of the intelligent reflection surface element, the remaining activation functions are ReLu, and the size of the experience replay pool is 2 18 , the learning rate of the neural network is 2 -16 , the variance of the exploration noise is 0.0001, the optimizer uses Adam, and the discount factor is set to 0.

[0063] Figure 6 The training curves of the actual algorithm training process using bidirectional convolution and fully connected structures are compared. Figure 7 The sum of the user equipment rates that can be achieved when the number of activated smart reflective surface elements is 18, and when the total number of smart reflective surface elements is 24 and 32 respectively

[0064] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0065] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0066] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0067] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A method for intelligent activation of intelligent reflective surface elements based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Establish a system model based on the actual communication scenario, and propose the target problem to be solved based on the system model; S2: Abstract the goal-solving problem into a Markov decision problem, and define the basic elements involved in the interaction between the reinforcement learning agent and the environment, including the setting of actions, states, and reward functions; S3: Build a deep reinforcement learning algorithm based on the classic speaker-critic framework, including a policy network and an evaluation network. The policy network is used to output the action decision of the agent, and the evaluation network is used to evaluate the actions taken by the policy network in the current state and provide the gradient for updating the policy network on this basis. S4: Redesign the network structure, i.e., adopt a bidirectional convolution and densely connected fully connected structure to extract the channel state information; the channel state involves the position relationship, mutual interference degree and signal attenuation factor of the base station, smart reflective surface and user, and use the neural network to fit the mapping relationship between the channel state and the element activation of the smart reflective surface, thereby completing the intelligent activation of the elements of the smart reflective surface; The step S3 constructs a deep reinforcement learning algorithm based on the classic speaker-critic framework, including a policy network and an evaluation network. The policy network is used to output the action decision of the intelligent agent, and the evaluation network is used to evaluate the action taken by the policy network in the current state and provide a gradient for updating the policy network on this basis. Specifically, it includes: The strategy network as a whole consists of three modules. Module 1 is a bidirectional convolution, which mainly performs horizontal and vertical convolution on the input link channel. The outputs in both directions are then convolved with a 1X1 convolution kernel, and finally the outputs in both directions are merged into a one-dimensional vector and input into module 2. Module 2 is mainly a two-layer densely connected fully connected structure. The densely connected structure can make the input of the current layer contain the features of all previous layers, further improving the feature extraction and analysis capabilities of the channel. Module 3 uses a 2-layer fully connected adjustment network output. The output of the strategy network is a one-dimensional probability vector of the number of intelligent reflective surface elements. The greater the probability, the greater the possibility of being activated. The evaluation network is also composed of three modules. Module 1 also adopts the bidirectional convolution method. Since the input of the evaluation network also includes the output action of the policy network relative to the policy network, the evaluation network will merge the action of the policy network after the 1X1 convolution and then input into module 2. The structure of module 2 is completely consistent with that of the policy network. Module 3 adopts 2-layer full connection mainly to adjust the output of the network. The output of the evaluation network is a Q value, so the final output is a value. The bidirectional convolution module performs bidirectional convolution on the channel status of each link respectively; The module 2 is a fully connected structure with two layers of dense connections. The network width of all fully connected structures is limited to 1024. In the use of activation functions, except for the action output of the strategy network using Sigmoid to represent the probability of activation of the intelligent reflection surface element, the remaining activation functions are ReLu, and the size of the experience replay pool is 2 18 , the learning rate of the neural network is 2 -16 , the variance of the exploration noise is 0.0001, the optimizer uses Adam, and the discount factor is set to 0.

2. According to claim 1, a method for intelligent activation of intelligent reflective surface elements based on deep reinforcement learning is characterized in that: The smart reflective surface is a passive smart reflective surface, and the main function of the smart reflective surface is to serve as a relay device between the base station and the user equipment to assist the communication between the two.

3. According to claim 1, a method for intelligent activation of intelligent reflective surface elements based on deep reinforcement learning is characterized in that: The step S1: according to the actual scenario where there is no line-of-sight communication between the base station and the user due to occlusion, an intelligent reflective surface is deployed between the base station and the user, a system model of intelligent reflective surface assisted communication is established, and a target problem to be solved is proposed according to the system model, specifically including: The transmission rate of the Kth user equipment can be expressed as: in, Φ represents the smart reflection surface coefficient matrix, represents the mean square error between the information received by the receiver and the information sent by the sender, p t is the user transmit power, σ 2 represents the Gaussian noise power, I M represents the standard unit matrix, h d,k 、h r,k and G represent the channel states from the base station to the user equipment, from the smart reflecting surface to the user equipment, and from the smart reflecting surface to the base station, respectively.

4. According to claim 3, a method for intelligent activation of intelligent reflective surface elements based on deep reinforcement learning is characterized in that: The step S2: abstracts the solution of the target problem into a Markov decision problem, defines the basic elements involved in the process of interaction between the reinforcement learning agent and the environment, including the setting of actions, states and reward functions, specifically including: In the establishment of the Markov decision model, the state is set to the channel state at the current moment. The channel state information includes the real part and the imaginary part. The action of the strategy is a one-dimensional vector v of the number of smart reflective surface elements. Assuming that the number of smart reflective surface elements is N, the action at any time t is expressed as: v(t)=[v1(t),v2(t),…v N (t)] The reward reflects the performance of the target problem. According to the user's transmission rate formula, the sum of the rates of all user devices is used as the reward function, which can be expressed as:

Images