A power internet of things wireless network access method based on RIS technology

Abstract

The application relates to a power internet of things wireless network access method based on RIS technology, which comprises the following steps: building a power internet of things communication architecture based on RIS technology in a city scene; according to the built power internet of things communication architecture, a corresponding target optimization function is designed, the phase shift of RIS, the base station beam and user selection are optimized, the service buffer queue of the user is ensured to be not more than a threshold value, the service quality of the user is ensured, and stable connection of the power internet of things user is realized; based on the proposed target optimization function, in combination with corresponding constraint conditions and the service buffer queue of the user, the target optimization function is optimized through Lyapunov, is established as a distributed Markov decision process, and then deep reinforcement learning is used to realize optimization solving. The application adopts wireless connection to ensure the connection of the power system, avoids the influence of a large number of obstacles in the city scene on wireless access, and provides a more intelligent optimization method for future power internet of things.

Description

Technical Field

[0001] This application belongs to the field of power Internet of Things (IoT) technology, and in particular relates to a power IoT wireless network access method based on RIS technology. Background Technology

[0002] In recent years, with the development of power systems, various systems and equipment operation status measurement systems, including data acquisition and monitoring control systems and synchronous phasor measurement devices, have been installed on high-voltage transmission lines and other equipment to ensure system safety and stability. These systems are also equipped with fiber optic communication, achieving reliable power and information interconnection. However, for distribution networks with lower voltage levels, fiber optic coverage has not been implemented simultaneously due to cost and other factors. Furthermore, compared to transmission networks, distribution networks connect far more devices, reaching every user and even every household's electrical equipment. These massive numbers of devices are connected only through power lines of different voltage levels. However, the massive number of devices on the distribution side have not yet achieved complete information interconnection, and the "last mile" challenge of the power system remains and urgently needs to be addressed. Traditional wired network access methods are costly and have poor applicability for the Internet of Things (IoT), so IoT generally connects via wireless networks, such as 4G mobile networks, Wi-Fi hotspots, and Zigbee. In recent years, 5G communication technologies, including massive MIMO, have developed rapidly, offering significant leaps in bandwidth, latency, and other aspects compared to 4G / 3G / 2G communication. 5G communication attempts to realize the Internet of Things by utilizing communication networks, and is a core technology for realizing and building the power Internet of Things.

[0003] However, in urban scenarios, numerous obstacles significantly impact wireless network transmission. Therefore, a power internet wireless access network architecture based on Reconfigurable Intelligent Surface (RIS) technology has been proposed. RIS is a revolutionary new technology that intelligently reconfigures the wireless propagation environment by integrating a large number of low-cost passive reflective elements on a plane, thereby significantly improving the performance of wireless communication networks. Different elements of the RIS can independently reflect incident signals by controlling their amplitude and phase, thus collaboratively achieving precise passive beamforming for directional signal modification. In RIS-based urban 5G communication scenarios, the power internet of things (IoT) enables wireless access for a large number of terminal users through massive MIMO antennas, reducing the impact of urban obstacles on user access. In the aforementioned wireless communication architecture, due to the large number of power IoT users and complex channels, optimizing parameters such as base station beamforming, RIS phase shift, and user selection can ensure stable user connections and increase system throughput to guarantee quality of service. However, traditional optimization algorithms struggle to solve such high-dimensional mixed-integer programming problems. Summary of the Invention

[0004] The purpose of this application is to provide a power Internet of Things (IoT) wireless network access method based on RIS technology, to realize IoT connectivity in urban scenarios, and to apply massive MIMO technology to ensure user connection reliability and service quality.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] This application provides a method for accessing a power Internet of Things wireless network based on RIS technology, characterized by the following steps:

[0007] Step 1: Establish a power IoT communication architecture based on RIS technology in an urban scenario. The network architecture includes: 5G base stations and Each RIS serves a 5G base station. Each power IoT user and equipped with One antenna, each RIS is equipped with Each reflector element and its corresponding controller; the RIS controller can control the phase shift of the reflector element.

[0008] Step 2: Based on the power IoT communication architecture built above, design the corresponding objective optimization function. By optimizing the phase shift of RIS, base station beam and user selection, ensure that the user's service buffer queue does not exceed the threshold and guarantee the user's service quality, realize the stable connection of power IoT users, and reduce the impact of obstacles on user connection in urban scenarios.

[0009] Step 3: Based on the objective optimization function proposed in Step 2, combined with the corresponding constraints and the user's service buffer queue, the objective optimization function is optimized by Lyapunov to establish a distributed Markov decision process. Then, deep reinforcement learning is used to achieve the optimization solution, and the power Internet of Things wireless network access based on RIS technology is completed.

[0010] Step 1 is described in detail as follows:

[0011] Step 1.1: First, within the network framework of the power IoT wireless connection, the entire transmission process is divided into T time slots, represented as follows: Define all 5G base stations as Each base station serves K m There are n users, and the k-th user served by the m-th base station is represented as... Then the set of all users is represented as Due to the large number of end-users in the power IoT, this architecture employs massive MIMO technology to improve system capacity, with each base station equipped with N...m Each antenna can serve a maximum of one base station user.

[0012] Step 1.2: Define the RIS set as Each RIS is equipped with L reflective elements, denoted as... Each reflective element on a RIS is controlled by a RIS controller. The l-th reflective element of the s-th RIS is represented as (s, l), and the diagonal matrix of the reflective element can be represented as follows:

[0013]

[0014] In the above formula Indicates amplitude, This represents the phase in time slot t, and the amplitude and phase satisfy...

[0015] Step 1.3: For base station user k m Channel from base station to user The channel for the base station user is determined by the direct channel from the base station to the user and the indirect channel via RIS, and is given by the following formula:

[0016]

[0017] in Indicates the distance from the base station to user k m direct channel, This represents the channel from the base station through all RIS reflective elements to the base station user, where Θ s (t), f m，s (t) represent the channel from RIS to the base station user, the diagonal matrix of the reflection coefficients of RIS, and the channel from the base station to RIS, respectively;

[0018] User k of the power Internet of Things m The received signal consists of the useful signal, intra-cell interference, inter-cell interference, and noise, specifically represented as follows:

[0019]

[0020] in and These represent the beamforming vector and transmitted signal of the base station user, respectively. This represents the Gaussian random noise received by the user, while These represent the useful signal received by the user, the signal received by other users within the base station, and the signal received by user k, respectively. m This causes signal interference within the cell and inter-cell interference from other users of other base stations.

[0021] Based on the signals received by the user, the capacity calculation formula for base station m can be obtained as follows:

[0022]

[0023] variable User k m Whether user k is selected. m hour, otherwise Signal-to-interference-plus-noise ratio (SINR) It can be represented as follows

[0024]

[0025] Step 1.4: To ensure service quality and fairness among users, it is assumed that the power IoT wireless network based on RIS technology reserves a service buffer queue for each IoT user. The queue length is updated at the beginning of each time slot t according to the following formula:

[0026]

[0027] In the above formula, [x] + =max{x, 0} ensures that the queue length is a positive number. It is the amount of communication data arriving at time slot t, which follows a Poisson distribution.

[0028] Queue length threshold User stability can be expressed as...

[0029]

[0030] Step 2 is described in detail below:

[0031] By using a fixed online policy π and observing the environmental state of each agent, the user selection, RIS phase shift, and base station beamforming in time slot t are jointly optimized. Under the system model described in step 1, the joint online policy can both reduce the impact of obstacles on wireless network connections in urban scenarios and maximize system throughput. The constrained optimization problem is expressed as:

[0032]

[0033] The above constraints represent the limitations on base station beam power, the limitations on RIS phase values, the binary constraints on user selection, the maximum number of antennas, and the user queue stability constraints, respectively.

[0034] Step 3 is described in detail below:

[0035] Step 3.1: First, create a virtual queue, defined as...

[0036]

[0037] Based on the virtual queue, the Lyapunov function in time slot t is defined as follows to ensure user stability.

[0038]

[0039] Then, the Lyapunov drift function is defined as follows:

[0040]

[0041] The Lyapunov drift penalty function is defined by selecting control actions in each time slot to minimize the upper bound of the drift penalty expression, expressed as:

[0042]

[0043] It is a non-negative constant that affects the trade-off between queue length and system throughput, and the upper bound of the Lyapunov drift penalty function is expressed as follows:

[0044]

[0045] in

[0046] Based on the upper bound of the Lyapunov drift penalty function, the constrained optimization problem can be transformed into minimizing the upper bound of the Lyapunov drift penalty function. The objective constrained optimization function can be rewritten as maximizing the following objective function:

[0047]

[0048] Step 3.2: Model the constrained optimization problem as a distributed partially observable Markov decision process, Dec-POMDP. The main problem of Dec-POMDP is described as follows:

[0049] State: The 5G base station and RIS are considered as two intelligent agents, represented as follows: Define the global state as

[0050]

[0051] in and These are the local observations of the base station and the RIS agent, respectively. The local observations of different agents are defined as follows:

[0052]

[0053]

[0054] Action: The action of the t-th time slot is defined as follows:

[0055]

[0056] in The beam representing the intelligent agent of the base station, The set selected for base station users The phase shift variable represents the RIS agent, and the actions include discrete and continuous actions, which we will represent respectively. and To represent their set,

[0057] Global Reward: Define the global reward as:

[0058]

[0059] Where λ is the penalty factor for the minimum rate limit.

[0060] Step 3.3: In step 3.2, the constrained Dec-POMDP problem is transformed into a regular Dec-POMDP problem, which is then solved using a reinforcement learning algorithm.

[0061] Compared with existing technologies, the beneficial effects of this application are as follows: This application addresses the access needs of power IoT networks in densely populated urban scenarios by constructing a power IoT communication architecture based on RIS technology in urban environments, thereby meeting the access requirements of massive numbers of devices. This application optimizes the phase shift, base station beamforming, and user selection of the RIS to ensure that the user's service buffer queue does not exceed a threshold and guarantees the user's service quality, achieving stable connections for power IoT users. Furthermore, this application proposes a power wireless network access method based on RIS. Based on the proposed objective optimization function, combined with corresponding constraints and the user's service buffer queue, the objective optimization function is optimized through Lyapunov optimization, establishing a distributed Markov decision process, and then using deep reinforcement learning to achieve the optimization solution. The method proposed in this application can effectively solve joint optimization problems in complex access networks, achieving highly customized communication. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below. It should be understood that the following drawings only show certain aspects of this application.

[0063] These embodiments are therefore not intended to limit the scope. For those skilled in the art, other related drawings can be obtained from these drawings without any inventive effort.

[0064] Figure 1 This is a flowchart illustrating a method according to an embodiment of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined in subsequent drawings.

[0066] Meaning and explanation.

[0067] The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements, but also...

[0068] It also includes other elements not explicitly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, the phrase "including a..."

[0069] The definition of a specific element does not preclude the presence of other identical elements in a process, method, article, or apparatus that includes the element.

[0070] Please see Figure 1 This application provides a method for accessing a power Internet of Things wireless network based on RIS technology, including the following steps.

[0071] Step 1: Establish a power IoT communication architecture based on RIS technology in an urban scenario.

[0072] The network architecture includes: 5G base stations and Each RIS serves a 5G base station. Each power IoT user and equipped with One antenna, each RIS is equipped with Each reflector element and its corresponding controller; the RIS controller can control the phase shift of the reflector element.

[0073] Step 2: Based on the power IoT communication architecture built above, design the corresponding target optimization function. By optimizing the phase shift of RIS, base station beam and user selection, ensure that the user's service buffer queue does not exceed the threshold and guarantee the user's service quality, realize the stable connection of power IoT users, and reduce the impact of obstacles on user connection in urban scenarios.

[0074] Step 3: Based on the objective optimization function proposed in Step 2, combined with the corresponding constraints and the user's business buffer queue, the objective function is optimized by Lyapunov and established as a distributed Markov process. Then, deep reinforcement learning is used to achieve the optimization solution.

[0075] Preferably, step 1 is as follows:

[0076] Step 1.1: In urban scenarios, numerous obstacles significantly reduce base station signal strength, impacting the wireless connectivity of the power IoT. RIS (Radio Reflector System) can reflect electromagnetic wave beams, altering the channel in space and mitigating the impact of obstacles on wireless connectivity. Firstly, within the network framework of the power IoT wireless connection, the entire transmission process is divided into T time slots, denoted as...

[0077] Define all 5G base stations as Each base station serves K m There are n users, and the k-th user served by the m-th base station is represented as... Then the set of all users is represented as Due to the large number of power Internet of Things (IoT) end users, massive MIMO technology is adopted in this architecture to improve system capacity. Each base station is equipped with one antenna, and each antenna can serve at most one base station user.

[0078] Step 1.2: Define the RIS set as Each RIS is equipped with A reflective element, denoted as Each reflector on the RIS is controlled by a RIS controller. To achieve more precise phase control, the RIS reflector can perform continuous phase shifts, thereby altering the spatial channel, reducing the impact of obstacles on wireless connectivity, and enhancing the performance of base station users. We will... The first RIS Each reflective element is represented as The diagonal matrix of the reflecting element can be represented as follows:

[0079]

[0080] In the above formula Indicates amplitude, This represents the phase within a time slot, and the amplitude and phase satisfy...

[0081] Step 1.3: The addition of RIS makes the wireless network channel more complex. For base station user k m Channel from base station to user The channel is determined by the direct channel from the base station to the user and the indirect channel via the RIS. Therefore, the channel for the base station user is given by the following formula:

[0082]

[0083] in Indicates the distance from the base station to user k m direct channel, This represents the channel from the base station through all RIS reflective elements to the base station user, where Θ s (t), f m，s (t) represents the channel from RIS to the base station user, the diagonal matrix of the reflection coefficients of RIS, and the channel from the base station to RIS, respectively.

[0084] Therefore, the user k of the power Internet of Things m The received signal consists of the useful signal, intra-cell interference, inter-cell interference, and noise, specifically represented as follows:

[0085]

[0086] in and These represent the beamforming vector and transmitted signal of the base station user, respectively. This represents the Gaussian random noise received by the user, while These represent the useful signal received by the user, the signal received by other users within the base station, and the signal received by user k, respectively. m This causes signal interference within the cell and inter-cell interference from other users of other base stations.

[0087] Based on the signals received by the user, the capacity calculation formula for base station m can be obtained as follows:

[0088]

[0089] variable User k m Whether user k is selected. m hour, otherwise Signal-to-interference-plus-noise ratio (SINR) It can be represented as follows

[0090]

[0091] Step 1.4: To ensure service quality and fairness among users, it is assumed that the power IoT wireless network based on RIS technology reserves a service buffer queue for each IoT user. The queue length is updated at the beginning of each time slot t according to the following formula:

[0092]

[0093] In the above formula, [x] + =max{x, 0} ensures that the queue length is a positive number. It is the amount of communication data arriving at time slot t, which follows a Poisson distribution.

[0094] In this network framework, it's necessary to consider user selection while maximizing the data rate for all users under the base station. To ensure fairness in communication among users, it's essential to prevent the base station from consistently selecting users with better communication conditions while neglecting the communication needs of other users. To ensure the quality of service for users, we need to keep the queue length below a threshold. User stability can be expressed as...

[0095]

[0096] Preferably, step 2 is as follows:

[0097] Step 2.1: In this network, our goal is to jointly optimize user selection, RIS phase shift, and base station beamforming in time slot t by observing the environmental state of each agent through a fixed online policy π. Under the system model described in Step 1 above, the joint online policy can both reduce the impact of obstacles on wireless network connectivity in urban scenarios and maximize system throughput. Therefore, the constrained optimization problem can be expressed as:

[0098]

[0099]

[0100] The above constraints represent the limitations on base station beam power, the limitations on RIS phase values, the binary constraints on user selection, the maximum number of antennas, and the user queue stability constraints, respectively.

[0101] Preferably, step 3 is as follows:

[0102] Step 3.1: In the optimization problem described above, the constraint on user stability is a long-run average constraint. To satisfy this constraint, we need prior information about the distribution of data arrivals and the historical queue lengths. We can use the Lyapunov drift penalty algorithm to solve this problem. This algorithm can transform the long-run constraint into a constraint that only requires information about the current queue length. Its main process is to first establish a queue, and then ensure the stability of the queue based on specific circumstances. In this process, we need to establish a Lyapunov drift function and find its upper bound. Then, by minimizing the upper bound of the Lyapunov drift penalty function, we minimize the objective function, thus ensuring the stability of the user queue.

[0103] When establishing a business buffer queue, it is necessary to protect the stability of the queue. Therefore, a virtual queue is first established, defined as follows:

[0104]

[0105] Based on the virtual queue, we can define the Lyapunov function in time slot t as follows to ensure user stability.

[0106]

[0107] Then, the Lyapunov drift function can be defined as

[0108]

[0109] Therefore, the Lyapunov drift penalty function can be defined by selecting the control action in each time slot to minimize the upper bound of the drift penalty expression, expressed as:

[0110]

[0111] It is a non-negative constant that influences the trade-off between queue length and system throughput. The upper bound of the Lyapunov drift penalty function is expressed as:

[0112]

[0113] in

[0114] Based on the upper bound of the Lyapunov drift penalty function, the constrained optimization problem can be transformed into minimizing the upper bound of the Lyapunov drift penalty function. Therefore, the objective constrained optimization function is rewritten as maximizing the following objective function:

[0115]

[0116] The constraints remain unchanged. Joint optimization can maximize the objective function while ensuring the stability of the business buffer queue. In practical applications, ζ needs to be adjusted to a reasonable value to achieve the optimal balance between the objective function and connection stability.

[0117] Step 3.2: This constrained problem can be modeled as a Markov Decision Process (MDP) and solved centrally through reinforcement learning. However, this requires extensive information exchange between multiple agents and a central network, and the central network needs to handle a high-dimensional state-action space, leading to inefficiency. To solve the problem more efficiently with lower complexity, we model it as a decentralized partially-observable Markov decision process (Dec-POMDP).

[0118] Dec-POMDP is a coordination and decision-making model among multiple agents. It is a generalization of POMDP, allowing distributed control by agents who may not be able to observe the exact state. In this network, the base station and RIS are distributed. Therefore, the main problems of Dec-POMDP are described as follows:

[0119] State: The 5G base station and RIS are considered as two intelligent agents, represented as follows: Therefore, the global state is defined as

[0120]

[0121] in and These are the local observations of the base station and the RIS agent, respectively. Local observations of different agents are defined as follows:

[0122]

[0123]

[0124] Action: The action of the t-th time slot is defined as follows:

[0125]

[0126] in The beam representing the intelligent agent of the base station, The set selected for base station users The phase shift variable represents the RIS agent. Actions include discrete actions and continuous actions, which we represent respectively using... and To represent their set.

[0127] Global Reward: Our goal is to maximize base station throughput while ensuring user service quality and connection stability. Therefore, we define the global reward as:

[0128]

[0129] Where λ is the penalty factor for the minimum rate limit.

[0130] Step 3.3: In Step 3.2, the constrained Dec-POMDP problem is transformed into a regular Dec-POMDP problem. However, the optimization variables now include discrete user choices as well as continuous base station beam and RIS reflector phase shifts. The action space contains both discrete and continuous elements, and it is a multi-agent problem. Therefore, reinforcement learning algorithms can be used to solve it. By introducing appropriate deep learning algorithms, the multi-agent problem with a mixed action space can be solved.

[0131] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A power Internet of Things (IoT) wireless network access method based on RIS technology, characterized in that, Includes the following steps: Step 1: Establish a power IoT communication architecture based on RIS technology in an urban scenario. The power IoT communication architecture includes: 5G base stations and Each RIS serves a 5G base station. Each power IoT user and equipped with One antenna, each RIS is equipped with Each reflector element and its corresponding controller; the RIS controller can control the phase shift of the reflector element. Step 2: Based on the power IoT communication architecture built above, design the corresponding objective optimization function. By optimizing the phase shift of RIS, base station beam and user selection, ensure that the user's service buffer queue does not exceed the threshold and guarantee the user's service quality, realize the stable connection of power IoT users, and reduce the impact of obstacles on user connection in urban scenarios. Step 3: Based on the objective optimization function proposed in Step 2, combined with the corresponding constraints and the user's service buffer queue, the objective optimization function is optimized by Lyapunov and established as a distributed Markov decision process. Then, deep reinforcement learning is used to achieve the optimization solution and complete the power Internet of Things wireless network access based on RIS technology. Step 1 is described in detail as follows: Step 1.1: First, within the network framework of the power Internet of Things wireless connection, the entire transmission process is divided into... Each time slot is represented as Define all 5G base stations as Each base station serves The user, the first The first base station service Each user is represented as Then the set of all users is represented as Due to the large number of power IoT end users, massive MIMO technology is adopted in this architecture to improve system capacity, and each base station is equipped with Each antenna can serve a maximum of one base station user. Step 1.2: Define the RIS set as Each RIS is equipped with A reflective element, denoted as Each reflective element on the RIS is controlled by a RIS controller, which will... The first RIS Each reflective element is represented as The diagonal matrix of the reflecting element is represented as follows: ， In the above formula Indicates amplitude, Indicates in time slot The phase, and the amplitude and phase satisfy , Step 1.3: For base station users Channel from base station to user The channel for the base station user is determined by the direct channel from the base station to the user and the indirect channel via RIS, and is given by the following formula: ， in Indicates the distance from the base station to the user. direct channel, This represents the channel from the base station through all RIS reflective elements to the base station user, where , , These represent the channel from the RIS to the base station user, the diagonal matrix of the reflection coefficients of the RIS, and the channel from the base station to the RIS, respectively. Users of the power Internet of Things The received signal consists of the useful signal, intra-cell interference, inter-cell interference, and noise, specifically represented as follows: ， in and These represent the beamforming vector and transmitted signal of the base station user, respectively. This represents the Gaussian random noise received by the user, while , , These respectively represent the useful signal received by the user, and the signal received by other users within the base station. This results in signal interference within the cell and inter-cell interference from other users of other base stations. Based on the signals received by the user, the base station is obtained. The capacity calculation formula is as follows: ， variable Indicates user Whether to be selected, when selecting a user hour, ,otherwise And signal-to-interference-plus-noise ratio It is expressed as follows ， Step 1.4: To ensure service quality and fairness among users, it is assumed that the power IoT wireless network based on RIS technology reserves a service buffer queue for each IoT user. And update the queue length at the beginning of each time slot t according to the following formula: ， In the above formula Ensure the queue length is a positive number. Time slots that follow a Poisson distribution The amount of communication data arriving at the location, Queue length threshold User stability is represented as ； Step 2 is described in detail below: Through a fixed online strategy By observing the environmental status of each agent, time slots can be jointly optimized. In the user selection, RIS phase shift, and base station beaming framework described in step 1, the joint online strategy can both reduce the impact of obstacles on wireless network connectivity in urban scenarios and maximize system throughput. The constrained optimization problem is expressed as: ， ， ， The above constraints represent the limitations on base station beam power, the limitations on RIS phase values, the binary constraints on user selection, the maximum number of antennas, and the user queue stability constraints, respectively.

2. The power IoT wireless network access method based on RIS technology according to claim 1, characterized in that, Step 3 is described in detail below: Step 3.1: First, create a virtual queue, defined as... ， Define time slots based on virtual queues. The Lyapunov function in the code is as follows, to ensure user stability. ， Then, the Lyapunov drift function is defined as follows: ， The Lyapunov drift penalty function is defined by selecting control actions in each time slot to minimize the upper bound of the drift penalty expression, expressed as: ， It is a non-negative constant that affects the trade-off between queue length and system throughput, and the upper bound of the Lyapunov drift penalty function is expressed as follows: ， in , , , Based on the upper bound of the Lyapunov drift penalty function, the constrained optimization problem is transformed into minimizing the upper bound of the Lyapunov drift penalty function. The objective constrained optimization function is then rewritten as maximizing the following objective function: ， Step 3.2: Model the constrained optimization problem as a distributed partially observable Markov decision process, Dec-POMDP. The description of the Dec-POMDP problem is as follows: State: The 5G base station and RIS are considered as two intelligent agents, represented as follows: Define the global state as ， in and These are the local observations of the base station and the RIS agent, respectively. The local observations of different agents are defined as follows: ， Action: First The action of each time slot is defined as ， in The beam representing the intelligent agent of the base station, The set selected for base station users The phase shift variable represents the RIS agent, and the actions include discrete and continuous actions, which we will represent respectively. and To represent their set, Global Reward: Define the global reward as: ， in, It is the penalty factor for the minimum rate limit. Step 3.3: In step 3.2, the constrained Dec-POMDP problem is transformed into a regular Dec-POMDP problem, which is then solved using a reinforcement learning algorithm.

Images