Multi-cell network security energy efficiency optimization method, computer device and storage medium

Abstract

The application relates to a multi-cell network security energy efficiency optimization method, computer equipment and a storage medium. The method comprises the following steps: constructing a target communication network, the target communication network covering a plurality of target cells, each target cell being configured with a corresponding base station; acquiring a local observation space representation of the target communication network; inputting the local observation space representation into a preset meta-reinforcement learning framework, the preset meta-reinforcement learning framework comprising a multi-agent proximal policy optimization module; outputting an action space representation based on the local observation space representation by using the multi-agent proximal policy optimization module; calculating a target security energy efficiency and a tail enhanced reward value based on the action space representation; calculating a target reward value based on the target security energy efficiency and the tail enhanced reward value; and optimizing the multi-agent proximal policy optimization module based on the target reward value. The application can improve the security energy efficiency of the multi-cell network.

Description

Technical Field

[0001] This application relates to the field of communication and network technology, and in particular to a method for optimizing network security energy efficiency in multi-cell environments, computer equipment, and storage media. Background Technology

[0002] With the continuous evolution of 6G communication technology, networks are increasingly carrying more critical services, such as industrial control, vehicle networking, and intelligent manufacturing, placing higher demands on communication security and reliability. In multi-cell Rate Splitting Multiple Access (RSMA) downlink networks, the open nature of the wireless channel makes the system vulnerable to eavesdropping attacks, leading to information leakage risks. Simultaneously, strong interference coupling between multiple cells and random channel fluctuations result in significant uncertainty in system performance under different environments. To achieve reliable communication, multi-cell RSMA networks need to consider both communication efficiency and security constraints during resource allocation.

[0003] Existing deep reinforcement learning-based optimization methods typically only optimize average performance metrics, neglecting performance degradation under extreme conditions. For example, under extreme conditions such as severe channel degradation or drastic increase in interference, the system is highly susceptible to a small number of but extremely destructive low-performance events. Since these tail-risk events account for a small proportion of the overall sampling, their negative impact is difficult to reflect in the average metric.

[0004] However, tail risk events have a significant impact on the overall security and stability of the system, and can easily lead to low security energy efficiency in multi-cell RSMA networks. Summary of the Invention

[0005] This application provides a method, computer device, and storage medium for optimizing the network security energy efficiency of multi-cell networks, aiming to solve the problem that traditional optimization of only average performance indicators leads to low security energy efficiency in multi-cell RSMA networks.

[0006] In a first aspect, embodiments of this application provide a method for optimizing network security energy efficiency in multiple cells, the method comprising: Construct a target communication network, wherein the target communication network covers multiple target cells, and each target cell is configured with a corresponding base station; Obtain the local observation space representation of the target communication network, wherein the local observation space is represented as follows: , The action space representation for each agent, For each user's signal-to-interference-plus-noise ratio information. For the safety and energy efficiency corresponding to the previous time slot, For instant rewards, , For the signal-to-interference-plus-noise ratio of user public streams, For user private streams, signal-to-interference-plus-noise ratio, For the eavesdropper to monitor the signal-to-interference-plus-noise ratio of the public stream, The signal-to-interference-plus-noise ratio (SIR) of the private stream being monitored by the eavesdropper; The local observation space representation is input into a preset meta-reinforcement learning framework, which includes a multi-agent proximal policy optimization module. The multi-agent proximal policy optimization module outputs an action space representation based on the local observation space representation. Based on the action space representation, the target safety efficiency and tail enhancement reward value are calculated; Calculate the target reward value based on the target safety efficiency and the tail enhancement reward value; The multi-agent proximal policy optimization module is optimized based on the target reward value.

[0007] A further technical solution is that the calculation of the target safety energy efficiency and tail enhancement reward value includes: Obtain the target total energy consumption and target secure communication rate of the target communication network; The target security energy efficiency is obtained based on the ratio between the target secure communication rate and the target total energy consumption; Based on the target safety efficiency and the preset historical safety efficiency, the tail enhancement reward value is obtained.

[0008] A further technical solution is that obtaining the target total energy consumption of the target communication network includes: Obtain the public stream beamforming vector, the private stream beamforming vector, and the artificial noise vector; Based on the public stream beamforming vector, the private stream beamforming vector, and the artificial noise vector, the total transmit power of the corresponding base station in each target cell is obtained; Based on the total transmit power, the preset circuit fixed loss, and the preset power amplifier factor, the total energy consumption of each target cell is obtained. The total energy consumption of all the target cells is summed to obtain the target total energy consumption of the target communication network.

[0009] A further technical solution is that each target cell contains multiple users, and obtaining the target secure communication rate of the target communication network includes: Obtain the public stream reception rate, public stream eavesdropping rate, private stream reception rate, and private stream eavesdropping rate; Based on the public stream reception rate and the public stream eavesdropping rate, determine the target public stream allocation rate for each user; Based on the target public stream allocation rate, the private stream reception rate, and the private stream eavesdropping rate, calculate the security rate for each user; The target secure communication rate of the target communication network is obtained by summing the secure rates of all users in all target cells.

[0010] A further technical solution is that obtaining the public stream reception rate, public stream eavesdropping rate, private stream reception rate, and private stream eavesdropping rate includes: Obtain the legitimate user channel vectors and eavesdropper channel vectors of the target communication network; Based on the legitimate user channel vector, calculate the intra-area interference value of the user's public stream, the intra-area interference value of the user's private stream, and the inter-area interference value of the user's decoding; and based on the eavesdropper channel vector, calculate the intra-area interference value of the eavesdropper listening to the public stream, the intra-area interference value of the eavesdropper listening to the private stream, and the inter-area interference value of the eavesdropper listening. Based on the legitimate user channel vector, the eavesdropper channel vector, the intra-area interference value of the user's public stream, the intra-area interference value of the user's private stream, the inter-area interference value of the user's decoded data, the intra-area interference value of the eavesdropper listening to the public stream, the intra-area interference value of the eavesdropper listening to the private stream, and the inter-area interference value of the eavesdropper, the public stream reception rate, the public stream eavesdropping rate, the private stream reception rate, and the private stream eavesdropping rate are calculated respectively.

[0011] A further technical solution is that the calculation of the public stream reception rate, public stream eavesdropping rate, private stream reception rate, and private stream eavesdropping rate includes: Based on the legitimate user channel vector, the public stream beamforming vector, the intra-area interference value of the user public stream, the inter-area interference value of the user decoding, and the preset noise power, the public stream receiving rate is obtained. Based on the eavesdropper channel vector, the public stream beamforming vector, the intra-area interference value of the public stream monitored by the eavesdropper, the inter-area interference value of the public stream monitored by the eavesdropper, and the preset noise power, the public stream eavesdropping rate is obtained. Based on the legitimate user channel vector, the private stream beamforming vector, the intra-area interference value of the user's private stream, the inter-area interference value of the user's decoded data, and the preset noise power, the private stream receiving rate is obtained. Based on the eavesdropper channel vector, the private stream beamforming vector, the intra-area interference value of the private stream monitored by the eavesdropper, the inter-area interference value monitored by the eavesdropper, and the preset noise power, the private stream eavesdropping rate is obtained.

[0012] A further technical solution is that, based on the target safety efficiency and the tail enhancement bonus value, the calculation of the target bonus value includes: Obtain the power penalty value, rate penalty value, and decoding penalty value respectively; The target reward value is obtained based on the target safety energy efficiency, the tail enhancement reward value, the power penalty value, the rate penalty value, and the decoding penalty value.

[0013] A further technical solution is that the preset meta-reinforcement learning framework further includes a meta-learning module, which is used to perform iterative training based on multiple tasks with different distributions to obtain the target meta-policy parameters, and send the target meta-policy parameters to the multi-agent proximal policy optimization module to initialize the policy parameters of the multi-agent proximal policy optimization module.

[0014] Secondly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0015] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0016] This application provides a method, computer device, and storage medium for optimizing network security efficiency in multiple cells. The method includes: constructing a target communication network covering multiple target cells, each target cell having a corresponding base station; obtaining a local observation space representation of the target communication network; inputting the local observation space representation into a preset meta-reinforcement learning framework, the preset meta-reinforcement learning framework including a multi-agent near-end policy optimization module; using the multi-agent near-end policy optimization module to output an action space representation based on the local observation space representation; calculating a target security efficiency and a tail-boosting reward value based on the action space representation; calculating a target reward value based on the target security efficiency and the tail-boosting reward value; and optimizing the multi-agent near-end policy optimization module based on the target reward value.

[0017] This application embodiment introduces a tail-enhancing reward value as a risk metric. This tail-enhancing reward value is used to characterize the tail average value below a specific confidence level in the reward distribution. Based on this, the target reward value calculated from the tail-enhancing reward value, combined with a risk-sensitive mechanism, is more in line with actual communication scenarios. On this basis, the multi-agent near-end policy optimization module is optimized using this target reward value. This allows the multi-agent near-end policy optimization module to not only focus on average performance during the optimization process, but also effectively avoid extreme low-performance situations, thereby improving the overall security and stability of the system and improving the security energy efficiency of multi-cell networks. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0021] Figure 1 A flowchart illustrating the first embodiment of a multi-cell network security energy efficiency optimization method provided in this application; Figure 2 A schematic diagram of the target communication network provided in this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0024] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0027] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0028] To address the aforementioned issues, this application provides a multi-cell network security energy efficiency optimization method that enables the target communication network to simultaneously achieve security, robustness, and rapid adaptability.

[0029] See Figure 1 , Figure 1 A flowchart illustrating a first embodiment of a multi-cell network security energy efficiency optimization method provided in this application is shown. The method may include the following steps: Step 110: Construct the target communication network.

[0030] The target communication network covers multiple target cells, and each target cell is configured with a corresponding base station. For example, such as... Figure 2 As shown, the target communication network can be a multi-cell RSMA network, which contains N cells, each with K users and M eavesdroppers. Each base station sends public and private streams to the users and sends artificial noise to the eavesdroppers to interfere with their decoding.

[0031] Step 120: Obtain the local observation space representation of the target communication network.

[0032] Step 130: Input the local observation space representation into a preset meta-reinforcement learning framework, wherein the preset meta-reinforcement learning framework includes a multi-agent proximal policy optimization module.

[0033] Step 140: Utilize the multi-agent proximal policy optimization module to output the action space representation based on the local observation space representation.

[0034] Step 150: Calculate the target safety efficiency and tail enhancement reward value based on the action space representation.

[0035] Step 160: Calculate the target reward value based on the target safety efficiency and the tail enhancement reward value.

[0036] Step 170: Optimize the multi-agent proximal policy optimization module based on the target reward value.

[0037] This embodiment introduces a tail-enhancing reward value as a risk metric. This tail-enhancing reward value is used to characterize the tail average value below a specific confidence level in the reward distribution. Based on this, the target reward value calculated from the tail-enhancing reward value, combined with a risk-sensitive mechanism, is more in line with actual communication scenarios. On this basis, the multi-agent near-end policy optimization module is optimized using this target reward value. This allows the multi-agent near-end policy optimization module to not only focus on average performance during the optimization process, but also effectively avoid extreme low-performance situations, thereby improving the overall security and stability of the system and improving the security energy efficiency of multi-cell networks.

[0038] In some embodiments, the calculation of the target safety energy efficiency and tail-enhancement bonus value in step 150 includes steps 151-153: Step 151: Obtain the target total energy consumption and target secure communication rate of the target communication network.

[0039] In step 151, the total target energy consumption of the target communication network can be obtained by referring to steps 11-14, and the target secure communication rate can be obtained by referring to steps 15-18. Step 11: Obtain the public stream beamforming vector, the private stream beamforming vector, and the artificial noise vector.

[0040] Step 11 will be explained using an RSMA downlink communication network as an example, with the target communication network being an RSMA downlink communication network. The constructed multi-cell RSMA downlink communication system may include... Target communities ( (a set of target cells), each target cell is configured with a... Base station with root transmitting antenna and users ( , (for user sets), and exist simultaneously One listener ( , (A collection of eavesdroppers). Each base station sends three types of signals: a public stream sent to users. Private stream Artificial noise sent to the listener The information sent by the base station to each user It is divided into two parts, namely the public part. and private parts The common areas for each user within the community are merged into one. Public information With private information They were respectively encoded as public flows and private stream The total transmitted signal of the base station is ; in, Shape vectors for public stream beams, Shape vectors for private flow beams. This is the vector of artificial noise.

[0041] Step 12: Based on the public stream beamforming vector, the private stream beamforming vector, and the artificial noise vector, obtain the total transmit power of the corresponding base station in each target cell.

[0042] Specifically, step 12 can be achieved using the following formula: ; in, Let be the total transmit power of the nth base station.

[0043] Step 13: Based on the total transmit power, the preset circuit fixed loss, and the preset power amplifier factor, obtain the total energy consumption of each target cell.

[0044] Specifically, the total energy consumption of the target community The calculation can be performed using the following formula: ; in, The preset power amplifier factor, This refers to the preset fixed circuit losses.

[0045] Step 14: Sum the total energy consumption of all the target cells to obtain the target total energy consumption of the target communication network.

[0046] In some embodiments, obtaining the target secure communication rate of the target communication network includes: Step 15: Obtain the public stream reception rate, public stream eavesdropping rate, private stream reception rate, and private stream eavesdropping rate.

[0047] Because the system transmits within the same frequency band, cross-cell interference exists between cells, and multi-user interference exists between different users within the same cell. The base station makes resource allocation decisions based on the current channel state information. For the first base stations ( ) to the The first base station For each user's channel, the interference in the signal-to-interference-plus-noise ratio (SINR) of the public flow of each cell user comes from the interference of the private flow of its own cell, the interference of artificial noise, and the interference of public and private flows and artificial noise from other cells. The interference in the SINR of the private flow comes from the interference of other private flows and artificial noise from its own cell, and the interference of public and private flows and artificial noise from other cells.

[0048] Based on this, in some embodiments, step 15 may include: obtaining the legitimate user channel vector and the eavesdropper channel vector of the target communication network; calculating the intra-area interference value of the user's public stream, the intra-area interference value of the user's private stream, and the inter-area interference value of the user's decoding based on the legitimate user channel vector; calculating the intra-area interference value of the eavesdropper listening to the public stream, the intra-area interference value of the eavesdropper listening to the private stream, and the inter-area interference value of the eavesdropper based on the eavesdropper channel vector; and calculating the public stream reception rate, the public stream eavesdropping rate, the private stream reception rate, and the private stream eavesdropping rate, respectively, based on the legitimate user channel vector, the eavesdropper channel vector, the intra-area interference value of the user's public stream, the intra-area interference value of the user's private stream, the inter-area interference value of the user's decoding, the intra-area interference value of the eavesdropper listening to the public stream, the intra-area interference value of the eavesdropper listening to the private stream, and the inter-area interference value of the eavesdropper listening to the public stream.

[0049] Specifically, the intra-area interference value of user public flow The interfering value within the area monitored by the eavesdropper on the public stream. The following formulas can be used respectively:

[0050] in, For legitimate user channel vectors, This is the channel vector for the listener.

[0051] Similarly, the intra-area interference value of user private streams The interference value within the area monitored by the eavesdropper for the private stream The following formulas are used respectively: User-decoded interval interference value The calculation can be performed using the following formula: Interference values ​​monitored by the eavesdropper The calculation can be performed using the following formula: in, This refers to other target communities besides this one; This refers to users other than this user.

[0052] In some embodiments, the public stream receiving rate can be obtained based on the legitimate user channel vector, the public stream beamforming vector, the intra-area interference value of the user public stream, the inter-area interference value of the user decoding, and a preset noise power. For details, please refer to the following formula: ; in, Beamforming, , For public stream reception rate, Shape vectors for public stream beams, For the intra-area interference value of the user's public flow, For the user, the interval interference value is decoded. This is the preset noise power.

[0053] Similarly, in some embodiments, the public stream eavesdropping rate can be obtained based on the eavesdropper channel vector, the public stream beamforming vector, the intra-area interference value of the public stream being eavesdropped by the eavesdropper, the inter-area interference value being eavesdropped by the eavesdropper, and a preset noise power. For details, please refer to the following formula: ; in, For public stream eavesdropping rate, For the eavesdropper to monitor the interfering values ​​within the public stream area, Interference values ​​in the interval monitored by the eavesdropper.

[0054] In some embodiments, the private stream receiving rate can be obtained based on the legitimate user channel vector, the private stream beamforming vector, the intra-area interference value of the user private stream, the inter-area interference value of the user decoding, and a preset noise power. For details, please refer to the following formula: ; in, For the user's private stream receiving rate, Shape vectors for private flow beams. This refers to the interference value within the user's private stream area.

[0055] In some embodiments, the private stream eavesdropping rate can be obtained based on the eavesdropper channel vector, the private stream beamforming vector, the intra-area interference value of the private stream being monitored by the eavesdropper, the inter-area interference value being monitored by the eavesdropper, and a preset noise power. For details, please refer to the following formula: ; in, For private stream eavesdropping rate, To monitor the interference values ​​within the private stream for the listener.

[0056] Step 16: Based on the public stream reception rate and the public stream eavesdropping rate, determine the target public stream allocation rate for each user.

[0057] In step 16, to ensure that the public stream can be decoded by all users, the following should be selected: This represents the public stream decoding rate. (Note: The last part is a typo and can be left as is.) The public stream reception rate for each user. This must meet the following requirements. Furthermore, to ensure that the public stream is not decoded by eavesdroppers, the following conditions must be met. Therefore, the safe public flow rate allocated to the user, i.e., the target public flow allocation rate, is... , must meet .in, , This is the rate threshold.

[0058] Step 17: Calculate the security rate for each user based on the target public stream allocation rate, the private stream reception rate, and the private stream eavesdropping rate.

[0059] In this step, the user's secure rate The following formula can be used: in, , For the previous text abbreviation, For the previous text Abbreviation.

[0060] Step 18: Sum the security rates of all users in all target cells to obtain the target secure communication rate of the target communication network.

[0061] Step 152: Based on the ratio between the target secure communication rate and the target total energy consumption, obtain the target secure energy efficiency.

[0062] Specifically, step 152 can be performed using the following formula: ; in, For the target secure communication rate, The target is total energy consumption.

[0063] Step 153: Based on the target safety efficiency and the preset historical safety efficiency, obtain the tail enhancement reward value.

[0064] Among them, the multi-agent near-end policy optimization module stores historical safety and energy efficiency information during the interaction with new tasks, and calculates the tail enhancement reward value through preset historical safety and energy efficiency. Using this tail enhancement reward value to assist policy updates can ensure the stability and security of the network in complex and ever-changing environments.

[0065] In some embodiments, in terms of reward function design, the system's secure energy efficiency can be used as the primary optimization objective, and a constraint penalty term and CVaR are introduced to constitute the final reward. The penalty term includes: penalties for transmit power exceeding a preset threshold, penalties for user rates falling below the minimum security requirement, and penalties for users failing to successfully decode secure public streams, thereby ensuring that the learning process meets the system's physical and communication constraints. The CVaR term is used to characterize the tail-average performance below a given confidence level in the reward distribution. By focusing on optimizing low-performance samples, the strategy can maintain stable secure energy efficiency even under adverse channel conditions or strong interference scenarios. Specifically, step 160 in the first embodiment, namely, calculating the target reward value based on the target secure energy efficiency and the tail-enhancing reward value, includes: Step 161: Obtain the power penalty value, rate penalty value, and decoding penalty value respectively.

[0066] Specifically, power penalty value The penalty value generated when the transmission power exceeds the transmission power threshold can be calculated using the following formula: ; in, For the first The actual transmission power of each base station For the transmit power threshold, This represents the maximum actual transmit power among all base stations at the current moment.

[0067] Specifically, the rate penalty value The penalty value for a user's rate failing to reach the minimum rate threshold can be calculated using the following formula: ; in, For users' index, The minimum rate threshold, For the first The first base station The security rate for each user The minimum safe rate among all current users.

[0068] Specifically, decoding penalty value The penalty value for a user's inability to decode a secure public stream can be calculated using the following formula: ; Step 162: Based on the target safety energy efficiency, the tail enhancement reward value, the power penalty value, the rate penalty value, and the decoding penalty value, obtain the target reward value.

[0069] In step 162, the target reward value The following formula can be used: ; in, For the goal of safety and energy efficiency, The reward for CVaR enhancement, namely the tail enhancement reward value, can be derived by calculating the historical safety energy efficiency during training. Specifically... , Represents the mathematical expectation. for abbreviation, For example, regarding the risk value of safety and energy efficiency, , This indicates the confidence level, typically ranging from 0.9 to 0.99. Let SEE represent the cumulative distribution function.

[0070] In some embodiments, the multi-agent proximal policy optimization module can employ a multi-agent variant of the Proximal Policy Optimization (PPO) algorithm, namely the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm. The MAPPO module can comprise multiple PPO agents. Since each PPO agent can only obtain local channel state information and user-related information within its own cell and cannot obtain the global network state, the entire system can be modeled as a Partially Observable Markov Decision Process (POMDP). Under this model, each agent makes decisions based on its own local observations, outputting corresponding local resource allocation actions, including parameters such as beamforming vectors, safe common flow rates, and artificial noise beams.

[0071] Because traditional mathematical optimization methods rely on training on fixed tasks, they usually require multiple iterations to solve problems when faced with changes in user location, channel statistics, and the location of the eavesdropper. This makes it difficult to adapt quickly to new environments and fails to meet real-time requirements.

[0072] Based on this, this application introduces a meta-learning module into the meta-reinforcement learning framework. Specifically, the meta-learning module employs First-Order Model-Agnostic Meta-Learning (FOMAML) to train the meta-policy. This involves training a meta-policy across multiple tasks. For example, a multi-cell RSMA network environment with different user and eavesdropper location distributions can be modeled as multiple tasks, and a parameter-sharing meta-MAPPO policy can be trained on each task distribution. This approach enables the learned policy to possess excellent cross-task transfer capabilities, allowing for rapid policy adjustment with only a few iterations when facing new channel conditions or network topology changes. This achieves high security and energy efficiency in dynamic environments. Therefore, in the application phase, the meta-policy parameters trained by the meta-learning module can be directly deployed to the multi-agent near-end policy optimization module to initialize its policy parameters.

[0073] Referring to the second embodiment, the training phase of the preset meta-reinforcement learning framework is described in detail. The training phase may include the following process: 1) Construct a multi-cell RSMA downlink communication system with eavesdroppers present; In this system, to achieve safe and efficient resource scheduling, the system's resource allocation strategy can be abstracted into a mathematical optimization problem. During the training phase, this resource optimization problem can be modeled as follows: ; ; ; ; ; ; in, User security service quality constraints. The threshold for user service quality requirements; To constrain the transmit power of each base station, This represents the maximum transmit power of the base station. and It ensures that the secure public stream allocated to each user is non-negative and that the secure public stream can be decoded by all users.

[0074] 2) Construct a Partially Observable Markov Decision Process (POMDP) ​​model.

[0075] Based on the aforementioned multi-cell RSMA downlink communication system model, the resource optimization problem is modeled as a POMDP model. The POMDP model includes the action space, local observation space representation, and global state space for each agent, i.e. .in, Let be the set of target cells, which corresponds to the set of the number of agents in this model. For the global state space, For local observation space, For the action space, For the reward function, This is the discount factor.

[0076] The action space is defined as various resource allocation variables, including the transmission power allocation of public flows, private flows, and artificial noise, as well as the safe public flow rate allocation parameters for each user within the cell. All these actions are output by the policy network. The action space representation for each agent is as follows: It can be as follows: ; The local observation space is used to characterize the local information that the agent can acquire during the execution phase. Specifically, it includes the cell's action information in the previous time slot, the signal-to-interference-plus-noise ratio (SIR) of each user, and the security efficiency and reward value corresponding to the previous time slot. The local observation space represents... It can be as follows: ; in, , For the signal-to-interference-plus-noise ratio of user public streams, For user private streams, signal-to-interference-plus-noise ratio, For the eavesdropper to monitor the signal-to-interference-plus-noise ratio of the public stream, The signal-to-interference-plus-noise ratio (SIR) of the private stream being monitored by the eavesdropper. For instant rewards.

[0077] The global state space is used for value evaluation during the policy training phase. It includes the action information of all cells in the previous time slot, the signal-to-interference-plus-noise ratio information of all users, and the overall security efficiency and reward information of the system. This provides a global reference for multi-agent collaborative optimization. The global state space is represented as follows: It can be as follows: ; in, For the joint action of all intelligent agents, This is the reward value.

[0078] 3) Construct a multi-agent proximal policy optimization model Specifically, the multi-agent proximal policy optimization module can contain multiple PPO agents. Each PPO agent has a Critic neural network and an Actor neural network. The Critic network outputs the value of the current state, i.e., the quality of the current action; the Actor neural network outputs the action based on the current local observations. Both the Critic and Actor neural networks are constructed using multilayer perceptrons (MLPs). For example, both the Critic and Actor neural networks can contain two MLP layers, each with 512 neurons, and use the Tanh function as the activation function.

[0079] Specifically, Critic neural networks are used to estimate and output state values. The reward value r is the state value. The fundamental principle of the function is that rewards from multiple moments are accumulated after discounts to form the actual return. Critic neural network learns state value Approximates the expected return; Actor neural networks, on the other hand, utilize the actual return. The state value (i.e., the estimated value) output by the Critic neural network. The difference between them yields the dominance function. and utilize the advantage function Update strategy.

[0080] For example, the advantage function and actual returns The following formula can be used: ; ; Where t is the current time and T is the end time. This represents the immediate reward value obtained by the agent at the current moment. This is the discount factor.

[0081] Based on this, we can use the advantage function The probability of adjusting the sign and magnitude of the action. For example, when When the value is greater than 0, it indicates that the action is better than the average expectation of the current strategy, and the probability of the action is increased; when... When the value is less than 0, it indicates that the action is below expectations, and the probability of the action should be reduced.

[0082] 4) Constructing the meta-training process First, construct the meta-learning task distribution. Each task The user locations are distributed differently. At the same time, a strategy parameter is created. meta-strategy This meta-strategy aims to rapidly adapt and instantiate specific MAPPO agents for subsequent meta-tasks. In each training iteration, several tasks are randomly sampled from the task distribution, and the following process is performed on each task: 4-1) For a certain task First, use the meta-policy parameters MAPPO strategy Perform initialization (i.e., let) = Subsequently, several time steps are executed within this task environment to collect the interactive dataset. Gradient updates are performed based on the update rules of the MAPPO algorithm to obtain task-specific parameters. , It can be as follows: ; in, For the purpose of the task The policy parameters obtained during training, This is the learning rate for the inner loop. This process, called the inner loop update, is used to adapt the policy to the current task. The objective function representing the strategy is essentially an estimate of the cumulative return, designed to quantitatively evaluate the performance of the current strategy.

[0083] 4-2) In the outer loop, each strategy Experience was obtained by resampling during the task. The update results from multiple tasks are aggregated, and the initial parameters are updated by backpropagation by minimizing the policy loss of all tasks after the update. This can be represented as follows: ; in, For the outer loop learning rate, These are the parameters of the meta-policy after iteration.

[0084] 4-3) By repeatedly performing the inner and outer loop processes described above, a meta-policy parameter with good cross-task generalization ability is finally obtained, i.e., the iterated meta-policy parameter. The meta-policy parameter It can serve as a policy initialization benchmark, enabling agents to converge quickly in new environments.

[0085] Specifically, after meta-training is completed, the trained meta-policy parameters are saved. Based on this, the pre-defined meta-reinforcement learning framework can directly load the meta-policy parameters obtained during the training phase during the application phase. This makes the initial performance of the multi-agent proximal policy optimization model significantly better than that of random initialization.

[0086] Thus, when faced with new task scenarios in the application phase, the pre-defined meta-reinforcement learning framework allows the agent to quickly adapt with only a few iterations. Specifically, the agent can rapidly optimize beamforming. Assigning rate with target public flow This optimizes network security energy efficiency. During the iteration process, the intelligent agent can utilize the latest... Safety and energy efficiency data are recorded in a sliding window, and the risk value of safety and energy efficiency is calculated based on this data. Tail Enhancement Bonus Value And based on the risk value of safety and energy efficiency Tail Enhancement Bonus Value Calculate the final target reward value Utilize target reward value Incentivize agents to focus on the tail performance of the network.

[0087] Thus, this application introduces the tail augmentation reward value CVaR as a risk metric into the meta-reinforcement learning framework, reconstructs the reward function, and guides the agent to focus on tail performance. At the same time, combined with the meta-learning mechanism, a meta-policy with cross-task generalization ability is trained in multiple tasks, and the policy adjustment is completed quickly in unknown new environments through a small number of iterations, thereby achieving optimization of safe energy efficiency in dynamic channels and potential eavesdropping environments.

[0088] Corresponding to the above-mentioned multi-cell network security energy efficiency optimization method, this application also provides a multi-cell network security energy efficiency optimization device. This multi-cell network security energy efficiency optimization device includes a unit for executing the above-mentioned multi-cell network security energy efficiency optimization method, and can be configured in terminals such as desktop computers, tablet computers, and laptops.

[0089] like Figure 3 As shown in the figure, this application provides a computer device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. Memory 113 is used to store computer programs; In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the multi-cell network security energy efficiency optimization method provided in any of the foregoing method embodiments, including: Construct a target communication network, wherein the target communication network covers multiple target cells, and each target cell is configured with a corresponding base station; Obtain the local observation space representation of the target communication network; The local observation space representation is input into a preset meta-reinforcement learning framework, which includes a multi-agent proximal policy optimization module. The multi-agent proximal policy optimization module outputs an action space representation based on the local observation space representation. Based on the action space representation, the target safety efficiency and tail enhancement reward value are calculated; Calculate the target reward value based on the target safety efficiency and the tail enhancement reward value; The multi-agent proximal policy optimization module is optimized based on the target reward value.

[0090] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0091] Therefore, this application embodiment also provides a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the steps of the multi-cell network security energy efficiency optimization method provided in any of the foregoing method embodiments, including: Construct a target communication network, wherein the target communication network covers multiple target cells, and each target cell is configured with a corresponding base station; Obtain the local observation space representation of the target communication network; The local observation space representation is input into a preset meta-reinforcement learning framework, which includes a multi-agent proximal policy optimization module. The multi-agent proximal policy optimization module outputs an action space representation based on the local observation space representation. Based on the action space representation, the target safety efficiency and tail enhancement reward value are calculated; Calculate the target reward value based on the target safety efficiency and the tail enhancement reward value; The multi-agent proximal policy optimization module is optimized based on the target reward value.

[0092] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0093] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0095] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0097] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0098] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Since these modifications and variations fall within the scope of the claims and their equivalents, this application also intends to include these modifications and variations.

[0099] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for optimizing network security energy efficiency in multi-cell environments, characterized in that, The method includes: Construct a target communication network, wherein the target communication network covers multiple target cells, and each target cell is configured with a corresponding base station; Obtain the local observation space representation of the target communication network, wherein the local observation space is represented as follows: , This refers to the action information from the previous time slot. The signal-to-interference-plus-noise ratio (SIR) information for each user in the previous time slot. For the safety and energy efficiency corresponding to the previous time slot, As an instant reward for the previous time slot, , For the signal-to-interference-plus-noise ratio of user public streams, For user private streams, signal-to-interference-plus-noise ratio, For the eavesdropper to monitor the signal-to-interference-plus-noise ratio of the public stream, The signal-to-interference-plus-noise ratio (SIR) of the private stream being monitored by the eavesdropper; The local observation space representation is input into a preset meta-reinforcement learning framework, which includes a multi-agent proximal policy optimization module. The multi-agent proximal policy optimization module outputs an action space representation based on the local observation space representation. Based on the action space representation, the target safety efficiency and tail enhancement reward value are calculated; Calculate the target reward value based on the target safety efficiency and the tail enhancement reward value; The multi-agent proximal policy optimization module is optimized based on the target reward value.

2. The method according to claim 1, characterized in that, The calculation of the target safety efficiency and tail enhancement bonus value includes: Obtain the target total energy consumption and target secure communication rate of the target communication network; The target security energy efficiency is obtained based on the ratio between the target secure communication rate and the target total energy consumption; Based on the target safety efficiency and the preset historical safety efficiency, the tail enhancement reward value is obtained.

3. The method according to claim 2, characterized in that, The step of obtaining the target total energy consumption of the target communication network includes: Obtain the public stream beamforming vector, the private stream beamforming vector, and the artificial noise vector; Based on the public stream beamforming vector, the private stream beamforming vector, and the artificial noise vector, the total transmit power of the corresponding base station in each target cell is obtained; Based on the total transmit power, the preset circuit fixed loss, and the preset power amplifier factor, the total energy consumption of each target cell is obtained. The total energy consumption of all the target cells is summed to obtain the target total energy consumption of the target communication network.

4. The method according to claim 3, characterized in that, Each target cell contains multiple users, and obtaining the target secure communication rate of the target communication network includes: Obtain the public stream reception rate, public stream eavesdropping rate, private stream reception rate, and private stream eavesdropping rate; Based on the public stream reception rate and the public stream eavesdropping rate, determine the target public stream allocation rate for each user; Based on the target public stream allocation rate, the private stream reception rate, and the private stream eavesdropping rate, calculate the security rate for each user; The target secure communication rate of the target communication network is obtained by summing the secure rates of all users in all target cells.

5. The method according to claim 4, characterized in that, The acquisition of the public stream reception rate, public stream eavesdropping rate, private stream reception rate, and private stream eavesdropping rate includes: Obtain the legitimate user channel vectors and eavesdropper channel vectors of the target communication network; Based on the legitimate user channel vector, calculate the intra-area interference value of the user's public stream, the intra-area interference value of the user's private stream, and the inter-area interference value of the user's decoding; and based on the eavesdropper channel vector, calculate the intra-area interference value of the eavesdropper listening to the public stream, the intra-area interference value of the eavesdropper listening to the private stream, and the inter-area interference value of the eavesdropper listening. Based on the legitimate user channel vector, the eavesdropper channel vector, the intra-area interference value of the user's public stream, the intra-area interference value of the user's private stream, the inter-area interference value of the user's decoded data, the intra-area interference value of the eavesdropper listening to the public stream, the intra-area interference value of the eavesdropper listening to the private stream, and the inter-area interference value of the eavesdropper, the public stream reception rate, the public stream eavesdropping rate, the private stream reception rate, and the private stream eavesdropping rate are calculated respectively.

6. The method according to claim 5, characterized in that, The calculation of the public stream reception rate, public stream eavesdropping rate, private stream reception rate, and private stream eavesdropping rate includes: Based on the legitimate user channel vector, the public stream beamforming vector, the intra-area interference value of the user public stream, the inter-area interference value of the user decoding, and the preset noise power, the public stream receiving rate is obtained. Based on the eavesdropper channel vector, the public stream beamforming vector, the intra-area interference value of the public stream monitored by the eavesdropper, the inter-area interference value of the public stream monitored by the eavesdropper, and the preset noise power, the public stream eavesdropping rate is obtained. Based on the legitimate user channel vector, the private stream beamforming vector, the intra-area interference value of the user's private stream, the inter-area interference value of the user's decoded data, and the preset noise power, the private stream receiving rate is obtained. Based on the eavesdropper channel vector, the private stream beamforming vector, the intra-area interference value of the private stream monitored by the eavesdropper, the inter-area interference value monitored by the eavesdropper, and the preset noise power, the private stream eavesdropping rate is obtained.

7. The method according to claim 1, characterized in that, The calculation of the target reward value based on the target safety efficiency and the tail enhancement reward value includes: Obtain the power penalty value, rate penalty value, and decoding penalty value respectively; The target reward value is obtained based on the target safety energy efficiency, the tail enhancement reward value, the power penalty value, the rate penalty value, and the decoding penalty value.

8. The method according to claim 1, characterized in that, The preset meta-reinforcement learning framework also includes a meta-learning module, which is used to perform iterative training based on multiple tasks with different distributions to obtain target meta-policy parameters, and send the target meta-policy parameters to the multi-agent proximal policy optimization module to initialize the policy parameters of the multi-agent proximal policy optimization module.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images