Power configuration method and device, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses a power configuration method and device, electronic equipment and a nonvolatile storage medium. The method comprises the steps that the system state of a communication system is determined, the communication system comprises a plurality of user pairs, each user pair comprises a signal transmitting end and a signal receiving end, and the system state comprises the total system rate of the communication system; an intelligent agent corresponding to the signal transmitting end is used for determining an expected return value corresponding to each candidate action in an execution action space in a system state, a target action is determined in the candidate actions according to the expected return value, and the expected return value is determined according to the change condition of the system total rate of the communication system after the candidate actions are executed; and performing power configuration on the signal transmitting end in the communication system by executing the target action. According to the method and the device, the technical problems of high complexity, high calculation amount and poor feasibility of a channel interference processing scheme in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of mobile communications and artificial intelligence technology, and more specifically, to a power configuration method, device, electronic device and non-volatile storage medium. Background Art

[0002] When multiple wireless communication links operate on the same common medium (such as time, spectrum, etc.), channel interference is inevitable. Channel interference is the most common problem in the field of mobile communications. With the excessive use of spectrum resources and the rapid growth of the number of communication devices, channel interference will become a bottleneck restricting the development of wireless communications.

[0003] In wireless communication networks, excessive interference to non-target users often occurs due to unreasonable signal transmitter power configuration. However, most of the channel interference processing solutions in related technologies have problems such as high complexity, high computational cost, and difficulty in practical application.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a power configuration method, device, electronic device and non-volatile storage medium to at least solve the technical problems of high complexity, high calculation amount and poor feasibility of channel interference processing solutions in related technologies.

[0006] According to one aspect of an embodiment of the present application, a power configuration method is provided, comprising: determining a system state of a communication system, wherein the communication system includes multiple user pairs, each user pair includes: a signal transmitting end and a signal receiving end, and the system state includes: a total system rate of the communication system; using an intelligent agent corresponding to the signal transmitting end, determining an expected reward value corresponding to executing each candidate action in an action space under the system state, and determining a target action among the candidate actions based on the expected reward value, wherein the candidate action is used to configure a signal transmission power for the signal transmitting end, different candidate actions correspond to different signal transmission powers, and the expected reward value is determined based on the change in the total system rate of the communication system after executing the candidate action; by executing the target action, the signal transmitting end in the communication system is power configured.

[0007] Optionally, determining the system state of the communication system includes: determining a first signal coefficient of a target channel between a signal transmitting end and a signal receiving end belonging to the same user pair in the communication system, and a second signal coefficient of an interference channel between a signal transmitting end and a signal receiving end that do not belong to the same user pair, wherein the first signal coefficient / the second signal coefficient is used to characterize the attenuation degree of the signal in the channel; determining a signal-to-noise ratio corresponding to each signal receiving end in the communication system based on the first signal coefficient, the second signal coefficient, and the signal transmission power of the signal transmitting end; determining a channel capacity corresponding to the signal receiving end based on the signal-to-noise ratio, and determining a total system rate corresponding to the communication system based on the channel capacity corresponding to each signal receiving end, to obtain the system state.

[0008] Optionally, determining the expected reward value corresponding to executing each candidate action in the state space under the system state includes: obtaining an expected reward table, wherein the expected reward table contains expected reward values ​​corresponding to executing different candidate actions under different system states, and each intelligent agent in the communication system shares the same expected reward table; based on the expected reward table, determining the expected reward value corresponding to each candidate action that can be executed under the current system state of the communication system.

[0009] Optionally, during the training process, after power configuration is performed on the signal transmitting end in the communication system, the method further includes: determining a reward value based on a system state corresponding to a first time step and a system state corresponding to a second time step, wherein the first time step is a time step for executing a target action, and the second time step is a time step immediately following the first time step, and the reward value is used to characterize a change in the total system rate of the communication system after executing the target action, and when the total system rate increases, the reward value is a positive value, and when the total system rate decreases, the reward value is a negative value; based on the reward value, the expected reward value corresponding to executing each candidate action under the system state in the expected reward table is updated.

[0010] Optionally, based on the reward value, updating the expected reward value corresponding to executing each candidate action under the system state in the expected reward table includes: determining a learning rate, wherein the learning rate is used to indicate the speed at which new information covers original information; determining the maximum expected reward value that can be obtained by executing the candidate action in the state space under the system state corresponding to the second time step; and updating the expected reward value corresponding to executing the target action under the system state corresponding to the first time step in the expected reward table based on the learning rate, the expected reward value corresponding to the first time step, and the maximum expected reward value.

[0011] Optionally, determining the target action among candidate actions based on the expected return value includes: determining a preset probability, and selecting a target action determination strategy according to the preset probability, wherein the target action determination strategy includes: a first action determination strategy and a second action determination strategy; when the selected target action determination strategy is the first action determination strategy, determining the candidate action with the largest corresponding expected return value in the action space as the target action; when the selected target action determination strategy is the second action determination strategy, randomly selecting a candidate action in the action space, and determining the candidate action as the target action.

[0012] Optionally, the method also includes: determining a maximum signal transmission power; determining a signal transmission power for each candidate action based on the maximum signal transmission power and the number of candidate actions planned in the action space to obtain an action space, wherein each intelligent agent in the communication system shares the same action space.

[0013] According to another aspect of an embodiment of the present application, a power configuration device is also provided, including: a state determination module, used to determine the system state of a communication system, wherein the communication system includes multiple user pairs, each user pair includes: a signal transmitting end and a signal receiving end, and the system state includes: the total system rate of the communication system; a power determination module, used to use the intelligent agent corresponding to the signal transmitting end to determine the expected reward value corresponding to the execution of each candidate action in the action space under the system state, and determine the target action among the candidate actions based on the expected reward value, wherein the candidate action is used to configure a signal transmission power for the signal transmitting end, different candidate actions correspond to different signal transmission powers, and the expected reward value is determined based on the change in the total system rate of the communication system after the candidate action is executed; a power setting module, used to perform power configuration on the signal transmitting end in the communication system by executing the target action.

[0014] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the power configuration method is executed when the program is run.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the power configuration method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the power configuration method are implemented.

[0017] In an embodiment of the present application, a system state of a communication system is determined, wherein the communication system includes multiple user pairs, each user pair includes: a signal transmitter and a signal receiver, and the system state includes: a total system rate of the communication system; using the intelligent agent corresponding to the signal transmitter, the expected reward value corresponding to the execution of each candidate action in the action space under the system state is determined, and based on the expected reward value, a target action is determined among the candidate actions, wherein the candidate action is used to configure a signal transmission power for the signal transmitter, and different candidate actions correspond to different signal transmission powers, and the expected reward value is determined based on the change in the total system rate of the communication system after the candidate action is executed; by executing the target action, the signal transmitter in the communication system is configured for power, and power configuration is performed based on multi-agent reinforcement learning, so as to achieve the purpose of reasonably configuring the transmission power, maximizing the total system rate, and effectively reducing channel interference, which has the characteristics of low complexity, small computational complexity, and strong feasibility, thereby solving the technical problems of high complexity, high computational complexity, and poor feasibility of the channel interference processing scheme in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for power configuration provided in an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a method flow of power configuration provided according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of the architecture of a communication system provided according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a multi-agent Q learning algorithm provided according to an embodiment of the present application;

[0023] Figure 5 It is a structural schematic diagram of a power configuration device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] When multiple wireless communication links operate on the same common medium (such as time, spectrum, etc.), channel interference is inevitable. Channel interference is the most common problem in the field of mobile communications. Due to the existence of interference, network capacity does not necessarily increase with the increase of network scale. With the development of 5G (5th Generation Mobile Communication Technology), the number of various communication devices in the world has grown rapidly, and mobile data traffic has also exploded. The amount of spectrum required far exceeds the sum of previous generations of mobile communication technology. It can be foreseen that with the excessive use of spectrum resources and the rapid growth of the number of communication devices, the channel interference problem will become a bottleneck restricting the development of wireless communications.

[0027] In wireless communication networks, excessive interference to non-target users often occurs due to unreasonable signal transmitter power configuration. The channel interference processing scheme in related technologies uses numerical optimization algorithms (such as weighted minimum mean square error algorithm, iterative water filling algorithm, etc.), takes a given set of real-time problem parameters, such as channel implementation or signal-to-noise ratio specifications in wireless beamformer design as input, runs a large number of iterations, and outputs an optimization scheme. However, such optimization algorithms are often quite complex, and the large number of iterations leads to high computational costs, resulting in a serious gap between theoretical design, analysis, and practical application.

[0028] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, and a power configuration scheme based on multi-agent reinforcement learning is proposed. Reinforcement learning uses the agent to continuously interact with the unknown environment to obtain effective data to find the optimal strategy. Compared with the traditional numerical optimization algorithm, the advantages of the reinforcement learning algorithm lie in the following two aspects: First, the final decision of reinforcement learning comes only from the reward of environmental feedback and will not be interfered by too many external factors. When the channel state changes, the reward of environmental feedback will also change. Therefore, the reinforcement learning algorithm has strong adaptability; secondly, reinforcement learning learns the optimal strategy based on maximizing the cumulative reward. Therefore, different reward values ​​can be designed for different communication standards to obtain the optimal strategy. Without a large number of complex formula derivations and calculations, a relatively ideal performance effect can be achieved. The algorithm design is simple and universal.

[0029] The present application scheme performs power configuration based on multi-agent reinforcement learning, and maximizes the total system rate by performing reasonable power configuration on each signal transmitter, thereby achieving the goal of reducing channel interference and improving user communication quality. The following is a detailed description.

[0030] According to an embodiment of the present application, a method embodiment of power configuration is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing a power configuration method. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0032] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the power configuration method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned power configuration method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0036] In the above operating environment, the embodiment of the present application provides a power configuration method. Figure 2 is a schematic diagram of a power configuration method flow according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0037] Step S202, determining a system state of the communication system, wherein the communication system includes a plurality of user pairs, each user pair includes: a signal transmitting end and a signal receiving end, and the system state includes: a total system rate of the communication system;

[0038] Step S204, using the agent corresponding to the signal transmitting end, determining the expected reward value corresponding to executing each candidate action in the action space under the system state, and determining the target action from the candidate actions according to the expected reward value, wherein the candidate action is used to configure a signal transmission power for the signal transmitting end, and different candidate actions correspond to different signal transmission powers, and the expected reward value is determined according to the change of the total system rate of the communication system after executing the candidate action;

[0039] Step S206: power configuration is performed on the signal transmitting end in the communication system by executing the target action.

[0040] Through the above steps, by performing power configuration based on multi-agent reinforcement learning, the purpose of reasonably configuring the transmission power, maximizing the total system rate, and effectively reducing channel interference is achieved. It has the characteristics of low complexity, small calculation amount, and strong feasibility, thereby solving the technical problems of high complexity, high calculation amount, and poor feasibility of channel interference processing solutions in related technologies.

[0041] The power configuration method in steps S202 to S206 of the embodiment of the present application is further introduced below.

[0042] The embodiments of the present application can be applied to a communication system where channel interference exists. For example, in Figure 3 In the communication scenario shown, N source nodes (ie, signal transmitters) send messages to N destination nodes (ie, signal receivers), forming N user pairs.

[0043] In the embodiment of the present application, each signal transmitting end is equipped with an intelligent agent, such as Figure 4 As shown in the figure, the agent is used to configure a reasonable transmission power for each signal transmitter according to the system state of the communication system, so as to reduce the signal interference to non-target users and improve the user communication quality under the premise of maximizing the total rate of the system. The specific target optimization problem can be expressed as:

[0044] max sum-rate

[0045] st0<p n ≤p max

[0046] Where sum-rate is the total rate of the communication system, p n The signal transmission power configured for the signal transmitter, pmax The maximum signal transmission power is set.

[0047] For the above optimization problems, the related technologies usually use numerical optimization algorithms to deal with them. For example, when dealing with wireless interference problems, the weighted minimum mean square error algorithm (WMMSE) algorithm performs complex operations such as matrix inversion and equal division in each iteration; the iterative water injection algorithm (IWFA) and the interference pricing algorithm involve singular value decomposition in each iteration; the semidefinite relaxation (SDR)-based solution needs to first promote the original vector-based optimization problem to a matrix-based optimization problem, and then solve one or several semi-definite equations, each of which is solved in turn by a special iterative algorithm. These algorithms in the related technologies are quite complex, and the large number of iterations leads to high computational costs, resulting in a serious gap between theoretical design, analysis and practical application.

[0048] In the embodiment of the present application, the reinforcement learning mechanism is introduced into the channel interference network. By equipping each signal transmitter with an agent, the agent is used to continuously interact with the unknown environment to obtain effective data to find the optimal strategy. The final decision is only derived from the reward of environmental feedback. When the channel state changes, the reward of environmental feedback will also change, which can adapt to the environment to the greatest extent. Moreover, the iterative rules of reinforcement learning are relatively simple. Only by learning the optimal strategy based on maximizing the cumulative reward, a relatively ideal performance effect can be achieved without a large number of complex formula derivations and calculations. Specifically, the embodiment of the present application can use the Multi-Agent Q-Learning (MAQL) algorithm to solve the above-mentioned target problem.

[0049] The algorithm flow in the embodiments of the present application is further introduced below.

[0050] First, combined with the above-mentioned objective problem (maximizing the total system rate and thus improving the user communication quality by reasonably configuring the signal transmission power), the parameters such as the state set, action set (action space), and reward function in reinforcement learning are defined.

[0051] In this embodiment, the environment state (i.e., the system state of the communication system) in the state set S is defined by the total system rate of the communication system, which is used to characterize the performance of the current network (communication system). Therefore, the definition of the state set S is as follows:

[0052] S = {sum-rate t ,sum-rate t+1 ....}

[0053] The action set (action space) A of the agent is defined as the configuration action for the signal transmission power of the signal transmitter, as follows.

[0054] In some embodiments of the present application, the method also includes the following steps: determining the maximum signal transmission power; determining a signal transmission power for each candidate action based on the maximum signal transmission power and the number of candidate actions set in the action space plan, to obtain an action space, wherein each intelligent agent in the communication system shares the same action space.

[0055] Specifically, the candidate actions in the action set (action space) A are the signal transmission power configured by the signal transmitter, which can be expressed as a set of discrete power levels, as shown below:

[0056]

[0057] Among them, P max is the maximum signal transmission power, D is the set length of A, that is, the number of candidate actions set in the action space plan. In this embodiment, all agents in the communication system share the same action set (action space).

[0058] For the reward function r t , can be defined based on the current total system rate. When the total system rate at a certain moment is higher than that at the previous moment, the agent is rewarded, otherwise it is punished.

[0059] After defining the state set, action set (action space), reward function and other parameters in reinforcement learning, you can start training the agent. During the training process, each agent continuously interacts with the environment, receives the environment state and selects the optimal power configuration based on the current state. The agent optimizes its strategy by accumulating rewards and eventually finds the optimal power configuration solution under different environmental conditions. After the training cycle is over, each agent can jointly decide the final power configuration strategy based on the training results. The following is a detailed introduction to the process, and its corresponding pseudo code process is as follows:

[0060]

[0061]

[0062] Specifically, first determine the current system state s of the communication system t , the specific steps are as follows.

[0063] In some embodiments of the present application, determining the system state of a communication system includes the following steps: determining a first signal coefficient of a target channel between a signal transmitting end and a signal receiving end belonging to the same user pair in the communication system, and a second signal coefficient of an interference channel between a signal transmitting end and a signal receiving end that do not belong to the same user pair, wherein the first signal coefficient / the second signal coefficient is used to characterize the attenuation degree of the signal in the channel; determining a signal-to-noise ratio corresponding to each signal receiving end in the communication system based on the first signal coefficient, the second signal coefficient, and the signal transmission power of the signal transmitting end; determining a channel capacity corresponding to the signal receiving end based on the signal-to-noise ratio, and determining a total system rate corresponding to the communication system based on the channel capacity corresponding to each signal receiving end, to obtain a system state.

[0064] Specifically, Figure 3 As shown, in this embodiment, the target user has n -R n The target channel coefficient between the signal transmitting end and the signal receiving end belonging to the same user pair (i.e., the first signal coefficient of the target channel between the signal transmitting end and the signal receiving end belonging to the same user pair) is h nn , n∈[1,N]; non-target users T m -R n The interference channel coefficient between the signal transmitting end and the signal receiving end that do not belong to the same user pair is h mn , m∈[1,N];

[0065] Then, according to the first signal coefficient, the second signal coefficient, and the signal transmission power of the signal transmitting end, the signal receiving end R in the communication system is determined. n The received signal-to-noise ratio SINR n (Signal to Interference plus Noise Ratio), as shown in the following formula:

[0066]

[0067] Among them, p n ∈[0,p max ] is the signal transmitting end T n The signal transmission power, p m ∈[0,p max ] is the signal transmitting end T m The signal transmission power, For R n The noise power at .

[0068] Then, according to the signal receiving end R n Signal-to-noise ratio SINR n , calculate R n The channel capacity C n(ChannelCapacity), as shown in the following formula:

[0069]

[0070] Finally, according to the channel capacity corresponding to each signal receiving end, the total system rate sum-rate corresponding to the communication system is determined, that is, the system state is determined, as shown in the following formula:

[0071]

[0072] After determining the system state s at the current time (for example, time t) t Afterwards, the agent can obtain the Q-value function Q(s) calculated based on MAQL stored in the expected return table (Q-table). t ,a t ) (i.e. expected return value), as follows.

[0073] In some embodiments of the present application, determining the expected reward value corresponding to executing each candidate action in the state space under the system state includes the following steps: obtaining an expected reward table, wherein the expected reward table contains the expected reward values ​​corresponding to executing different candidate actions under different system states, and each intelligent agent in the communication system shares the same expected reward table; based on the expected reward table, determining the expected reward value corresponding to each candidate action that can be executed under the current system state of the communication system.

[0074] After that, the target action a to be executed under the system state can be determined based on the expected return value. t , the specific steps are as follows.

[0075] In some embodiments of the present application, determining a target action from candidate actions based on an expected return value includes the following steps: determining a preset probability, and selecting a target action determination strategy according to the preset probability, wherein the target action determination strategy includes: a first action determination strategy and a second action determination strategy; when the selected target action determination strategy is the first action determination strategy, determining the candidate action with the largest corresponding expected return value in the action space as the target action; when the selected target action determination strategy is the second action determination strategy, randomly selecting a candidate action in the action space, and determining the candidate action as the target action.

[0076] Specifically, in this embodiment, in order to improve the efficiency of reinforcement learning and balance the channel interference network model between "exploration" and "utilization", the embodiment of the present application adopts an ε-greedy selection strategy to compromise between "exploration" and "utilization" based on a preset probability ε, as shown in the following formula:

[0077]

[0078] At the same time, the agent has a probability of 1-ε to choose the action with the highest current reward value (the first action determination strategy), that is, to determine the candidate action with the largest expected return value in the action space as the target action; it also has a probability of ε to choose a random action (the second action determination strategy), that is, to randomly select a candidate action in the action space and determine the candidate action as the target action.

[0079] In addition, during the process of agent reinforcement learning training, when the agent determines the target action a t After feedback to the environment, that is, after the power of the signal transmitter in the communication system is configured by executing the target action, the stimulated environment will continue to output the next state s after executing the target action to the agent. t+1 and the reward value obtained after executing the target action, and use the reward value to calculate Q(s t ,a t ) to update. The specific steps are as follows.

[0080] In some embodiments of the present application, during the training process, after the power configuration of the signal transmitting end in the communication system is performed, the method further includes the following steps: determining a reward value based on a system state corresponding to a first time step and a system state corresponding to a second time step, wherein the first time step is a time step for executing a target action, and the second time step is a time step immediately following the first time step, and the reward value is used to characterize the change in the total system rate of the communication system after executing the target action, and when the total system rate increases, the reward value is positive, and when the total system rate decreases, the reward value is negative; based on the reward value, the expected reward value corresponding to executing each candidate action under the system state in the expected reward table is updated.

[0081] Specifically, first, according to the reward function r t Calculate the reward value obtained after executing the target action. For example, when time t passes through a t The total system rate of stimulus generation sum-rate t+1 Greater than sum-rate t , then rewards are implemented, otherwise penalties are imposed. Reward function r t It can be defined as:

[0082]

[0083] The above reward mechanism can drive the agent to continuously adjust the power configuration to maximize the total rate of the system. The design of the reward function is simple and direct, which is easy to implement in practical applications.

[0084] After calculating the reward value, the agent calculates the Q(s) stored in the expected return table Q-table based on the reward value. t ,a t ) to update. The specific steps are as follows.

[0085] In some embodiments of the present application, updating the expected reward value corresponding to executing each candidate action under the system state in the expected reward table based on the reward value includes the following steps: determining a learning rate, wherein the learning rate is used to indicate the speed at which new information covers original information; determining the maximum expected reward value that can be obtained by executing the candidate action in the state space under the system state corresponding to the second time step; updating the expected reward value corresponding to executing the target action under the system state corresponding to the first time step in the expected reward table based on the learning rate, the expected reward value corresponding to the first time step, and the maximum expected reward value.

[0086] Q(s t ,a t ) is updated as follows:

[0087]

[0088] in, represents the maximum Q value that can be obtained by all possible candidate actions in the new state (i.e., the system state corresponding to the second time step), λ is the reward discount value, which determines the current value of future rewards, α=1 / (1+visit(s,a)), α∈(0,1] is the learning rate, and visit(s,a) is the MAQL visit state-action pair (s t ,a t ). As the number of visits t→∞, the state-action pair can be updated countless times and α will approach zero. At this time, Q(s t ,a t ) will converge to the optimal Q * (s t ,a t ), after the expected return table converges, the agent can extract the strategy according to The optimal strategy can be obtained, that is, for each system state, the candidate action with the highest expected return value is selected as the optimal action in that state.

[0089] It should be noted that in the embodiment of the present application, all agents share the same action set and Q table. This design not only reduces the consumption of computing resources, but also ensures that each agent can quickly converge to the global optimal solution under different circumstances. By sharing the strategy table, the agents can work better together and improve the performance of the overall system.

[0090] Through the above process, the Q value is continuously updated iteratively, so that the agent can learn which actions to take in each state to maximize its expected cumulative return. As the number of iterations increases, the Q value will gradually approach the optimal value, and the behavior of the agent will become more reasonable and efficient.

[0091] The present application scheme introduces reinforcement learning into the power configuration of the channel interference network based on the power configuration method of multi-agent reinforcement learning, and adopts the framework of multi-agent reinforcement learning. Each signal transmitter is equipped with an agent. By interacting with the environment, the agent can dynamically learn and optimize its power configuration strategy. It can achieve the optimal solution for power configuration through simple iterative rules, and the computational complexity is significantly reduced. By reasonably configuring the transmission power, the present invention can reduce interference to non-target users and improve user communication quality while ensuring the maximization of the total system rate. This not only improves the overall efficiency of the network, but also improves the user's communication experience, especially in scenarios where spectrum resources are tight.

[0092] This application scheme can dynamically adjust the power configuration according to environmental changes by applying a reinforcement learning algorithm. When the channel conditions or network environment change, the agent quickly adjusts the power configuration strategy through reward feedback to ensure continuous optimization of the total system rate and dynamically adapt to changes in channel conditions and network environment;

[0093] The present application scheme does not require complex formula derivation or parameter adjustment for different scenarios, is applicable to many different types of communication networks, and has strong versatility; compared with the numerical optimization algorithm in the related art, the scheme of automatically learning the optimal strategy through reinforcement learning is easier to implement, suitable for deployment and application in actual large-scale communication networks, and has strong versatility and scalability. Whether under different communication standards or in different network topologies, the power configuration scheme of the present application scheme can be effectively applied. Especially in scenarios that require real-time adjustment and rapid response, the performance is particularly outstanding. Specifically, the application scenarios of the present application scheme include but are not limited to: 1) High-density communication environments: such as 5G base stations, cellular networks, Internet of Things device networks, etc. In these environments, the density of communication equipment is high and the interference problem is particularly prominent. 2) Scenarios with limited spectrum resources: such as urban areas with tight spectrum. In this environment, reasonable power configuration is particularly important for improving communication quality. 3) Communication networks that require dynamic adaptation: such as frequently changing mobile communication environments, requiring the system to quickly adapt to environmental changes and adjust power configuration.

[0094] According to an embodiment of the present application, an embodiment of a power configuration device is also provided. Figure 5 is a schematic diagram of a power configuration device provided according to an embodiment of the present application. Figure 5 As shown, the device comprises:

[0095] The state determination module 50 is used to determine the system state of the communication system, wherein the communication system includes a plurality of user pairs, each user pair includes: a signal transmitting end and a signal receiving end, and the system state includes: a total system rate of the communication system;

[0096] The power determination module 52 is used to determine the expected reward value corresponding to executing each candidate action in the action space under the system state by using the intelligent agent corresponding to the signal transmitting end, and determine the target action from the candidate actions according to the expected reward value, wherein the candidate action is used to configure a signal transmission power for the signal transmitting end, and different candidate actions correspond to different signal transmission powers, and the expected reward value is determined according to the change of the total system rate of the communication system after executing the candidate action;

[0097] The power setting module 54 is used to perform power configuration on the signal transmitting end in the communication system by executing the target action.

[0098] Optionally, determining the system state of the communication system includes: determining a first signal coefficient of a target channel between a signal transmitting end and a signal receiving end belonging to the same user pair in the communication system, and a second signal coefficient of an interference channel between a signal transmitting end and a signal receiving end that do not belong to the same user pair, wherein the first signal coefficient / the second signal coefficient is used to characterize the attenuation degree of the signal in the channel; determining a signal-to-noise ratio corresponding to each signal receiving end in the communication system based on the first signal coefficient, the second signal coefficient, and the signal transmission power of the signal transmitting end; determining a channel capacity corresponding to the signal receiving end based on the signal-to-noise ratio, and determining a total system rate corresponding to the communication system based on the channel capacity corresponding to each signal receiving end, to obtain the system state.

[0099] Optionally, determining the expected reward value corresponding to executing each candidate action in the state space under the system state includes: obtaining an expected reward table, wherein the expected reward table contains expected reward values ​​corresponding to executing different candidate actions under different system states, and each intelligent agent in the communication system shares the same expected reward table; based on the expected reward table, determining the expected reward value corresponding to each candidate action that can be executed under the current system state of the communication system.

[0100] Optionally, during the training process, after power configuration is performed on the signal transmitting end in the communication system, the power configuration device is further used to: determine a reward value based on a system state corresponding to a first time step and a system state corresponding to a second time step, wherein the first time step is a time step for executing a target action, and the second time step is a time step immediately following the first time step, and the reward value is used to characterize a change in the total system rate of the communication system after executing the target action, and when the total system rate increases, the reward value is positive, and when the total system rate decreases, the reward value is negative; based on the reward value, the expected reward value corresponding to executing each candidate action under the system state in the expected reward table is updated.

[0101] Optionally, based on the reward value, updating the expected reward value corresponding to executing each candidate action under the system state in the expected reward table includes: determining a learning rate, wherein the learning rate is used to indicate the speed at which new information covers original information; determining the maximum expected reward value that can be obtained by executing the candidate action in the state space under the system state corresponding to the second time step; and updating the expected reward value corresponding to executing the target action under the system state corresponding to the first time step in the expected reward table based on the learning rate, the expected reward value corresponding to the first time step, and the maximum expected reward value.

[0102] Optionally, determining the target action among candidate actions based on the expected return value includes: determining a preset probability, and selecting a target action determination strategy according to the preset probability, wherein the target action determination strategy includes: a first action determination strategy and a second action determination strategy; when the selected target action determination strategy is the first action determination strategy, determining the candidate action with the largest corresponding expected return value in the action space as the target action; when the selected target action determination strategy is the second action determination strategy, randomly selecting a candidate action in the action space, and determining the candidate action as the target action.

[0103] Optionally, the power configuration device is also used to: determine the maximum signal transmission power; determine a signal transmission power for each candidate action based on the maximum signal transmission power and the number of candidate actions set in the action space plan, to obtain an action space, wherein each intelligent agent in the communication system shares the same action space.

[0104] It should be noted that the various modules in the above-mentioned power configuration device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0105] It should be noted that the power configuration device provided in this embodiment can be used to perform Figure 2 The power configuration method shown, therefore, the relevant explanations and descriptions of the above power configuration method are also applicable to the embodiments of the present application and will not be repeated here.

[0106] An embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following power configuration method by running the computer program: determining the system state of the communication system, wherein the communication system includes multiple user pairs, each user pair includes: a signal transmitter and a signal receiver, and the system state includes: the total system rate of the communication system; using the intelligent agent corresponding to the signal transmitter, determining the expected reward value corresponding to executing each candidate action in the action space under the system state, and determining the target action among the candidate actions based on the expected reward value, wherein the candidate action is used to configure a signal transmission power for the signal transmitter, different candidate actions correspond to different signal transmission powers, and the expected reward value is determined based on the change in the total system rate of the communication system after executing the candidate action; by executing the target action, the signal transmitter in the communication system is power configured.

[0107] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the power configuration method described in each embodiment of the present application: determining a system state of a communication system, wherein the communication system includes multiple user pairs, each user pair includes: a signal transmitter and a signal receiver, and the system state includes: a total system rate of the communication system; using an intelligent agent corresponding to the signal transmitter, determining an expected reward value corresponding to executing each candidate action in the action space under the system state, and determining a target action from the candidate actions based on the expected reward value, wherein the candidate action is used to configure a signal transmission power for the signal transmitter, different candidate actions correspond to different signal transmission powers, and the expected reward value is determined based on the change in the total system rate of the communication system after executing the candidate action; by executing the target action, the signal transmitter in the communication system is power configured.

[0108] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0109] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0111] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0114] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A power configuration method, characterized in that: include: Determine a system state of a communication system, wherein the communication system includes a plurality of user pairs, each of the user pairs includes: a signal transmitting end and a signal receiving end, and the system state includes: a total system rate of the communication system; Utilizing the intelligent agent corresponding to the signal transmitting end, determine the expected reward value corresponding to executing each candidate action in the action space under the system state, and determine the target action among the candidate actions based on the expected reward value, wherein the candidate action is used to configure a signal transmission power for the signal transmitting end, and different candidate actions correspond to different signal transmission powers, and the expected reward value is determined based on the change in the total system rate of the communication system after executing the candidate action; by executing the target action, the power of the signal transmitting end in the communication system is configured.

2. The power configuration method according to claim 1, characterized in that: Determining the system status of a communication system includes: Determine a first signal coefficient of a target channel between the signal transmitting end and the signal receiving end belonging to the same user pair in the communication system, and a second signal coefficient of an interference channel between the signal transmitting end and the signal receiving end not belonging to the same user pair, wherein the first signal coefficient / the second signal coefficient is used to characterize the attenuation degree of the signal in the channel; Determining a signal-to-noise ratio corresponding to each of the signal receiving ends in the communication system according to the first signal coefficient, the second signal coefficient, and the signal transmission power of the signal transmitting end; According to the signal-to-noise ratio, the channel capacity corresponding to the signal receiving end is determined, and according to the channel capacity corresponding to each of the signal receiving ends, the total system rate corresponding to the communication system is determined to obtain the system state.

3. The power configuration method according to claim 1, characterized in that: Determining the expected reward value corresponding to executing each candidate action in the state space under the system state includes: Obtaining an expected reward table, wherein the expected reward table contains the expected reward values ​​corresponding to executing different candidate actions under different system states, and each of the intelligent agents in the communication system shares the same expected reward table; The expected reward value corresponding to each of the candidate actions that can be executed under the current system state of the communication system is determined according to the expected reward table.

4. The power configuration method according to claim 3, characterized in that: During the training process, after power configuration is performed on the signal transmitting end in the communication system, the method further includes: Determine a reward value according to the system state corresponding to a first time step and the system state corresponding to a second time step, wherein the first time step is a time step for executing the target action, and the second time step is a time step immediately following the first time step, and the reward value is used to characterize the change of the total system rate of the communication system after executing the target action, and when the total system rate increases, the reward value is a positive value, and when the total system rate decreases, the reward value is a negative value; According to the reward value, the expected reward value corresponding to executing each candidate action in the system state in the expected reward table is updated.

5. The power configuration method according to claim 4, characterized in that: According to the reward value, updating the expected reward value corresponding to executing each candidate action in the system state in the expected reward table includes: Determining a learning rate, wherein the learning rate is used to indicate the speed at which new information overwrites old information; Determine a maximum expected reward value that can be obtained by executing the candidate action in the state space under the system state corresponding to the second time step; The expected reward value corresponding to executing the target action in the system state corresponding to the first time step in the expected reward table is updated according to the learning rate, the expected reward value corresponding to the first time step, and the maximum expected reward value.

6. The power configuration method according to claim 1, characterized in that: Determining a target action from the candidate actions according to the expected reward value includes: Determine a preset probability, and select a target action determination strategy according to the preset probability, wherein the target action determination strategy includes: a first action determination strategy and a second action determination strategy; When the selected target action determination strategy is the first action determination strategy, determining the candidate action with the largest corresponding expected reward value in the action space as the target action; In a case where the selected target action determination strategy is the second action determination strategy, a candidate action in the action space is randomly selected, and the candidate action is determined as the target action.

7. The power configuration method according to claim 1, characterized in that: The method further comprises: Determine the maximum signal transmission power; According to the maximum signal transmission power and the number of candidate actions planned to be set in the action space, a signal transmission power is determined for each candidate action to obtain the action space, wherein each of the intelligent agents in the communication system shares the same action space.

8. A power configuration device, characterized in that: include: A state determination module is used to determine the system state of the communication system, wherein the communication system includes a plurality of user pairs, each of the user pairs includes: a signal transmitting end and a signal receiving end, and the system state includes: a total system rate of the communication system; A power determination module, used to determine, using the agent corresponding to the signal transmitting end, the expected reward value corresponding to executing each candidate action in the action space under the system state, and determine the target action from the candidate actions according to the expected reward value, wherein the candidate action is used to configure a signal transmission power for the signal transmitting end, and different candidate actions correspond to different signal transmission powers, and the expected reward value is determined according to the change of the total system rate of the communication system after executing the candidate action; A power setting module is used to perform power configuration on the signal transmitting end in the communication system by executing the target action.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the power configuration method according to any one of claims 1 to 7 is executed when the program is run.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the power configuration method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the power configuration method according to any one of claims 1 to 7 are implemented.

Images