Method and device for resource allocation of common sense fusion network based on maximum entropy reinforcement learning

Abstract

The application provides a kind of based on maximum entropy reinforcement learning's through sensing fusion network resource allocation method and device, comprising: exploring flexible antenna assisted through sensing fusion network architecture, the network architecture considers the scene of multiple pinching antennas on the same waveguide, and the communication data rate and the sensing demand are considered comprehensively, to establish flexible antenna assisted through sensing fusion network The problem model of maximum communication data rate while meeting the sensing demand is considered;Reconstruct the objective function of problem model, including designing the state space, action space and reward function of optimization target reinforcement learning network;With the criterion of maximizing cumulative discount reward while maximizing the entropy of strategy, train optimization target reinforcement learning network, and update the evaluation network and policy network of optimization target reinforcement learning network based on experience replay mechanism;After training, the solution of problem model is obtained, including the optimal antenna position and power allocation set.The application can allocate resources to through sensing fusion network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless communication, in particular to a method and device for resource allocation of integrated sensing and communication network based on maximum entropy reinforcement learning. BACKGROUND

[0002] Integrated Sensing and Communication (ISAC) technology integrates wireless communication and radar sensing functions into the same platform, uses shared resources such as hardware, spectrum and energy, improves operational efficiency, reduces costs and promotes sustainable development. Integrated sensing and communication networks use wireless communication signals to achieve identification, positioning, imaging and other sensing functions, and use sensing information to further enhance and exploit potential communication capabilities, so that wireless signals not only achieve the transmission of effective communication information, but also can sense, detect and characterize the physical world. Integrated sensing and communication gives wireless communication stronger sensing capabilities and gives birth to more abundant application scenarios. The utilization efficiency of shared resources and the adaptability of dynamic environment bring more challenges to integrated sensing and communication systems, especially their antenna systems. On the one hand, the resource sharing between sensing and communication functions requires the use of antennas with high directivity and flexibility to reduce signal interference and improve resource utilization efficiency. On the other hand, in a complex time-varying environment, the antenna must have the ability to dynamically adapt to environmental changes to ensure communication quality and sensing accuracy. However, traditional fixed-position antenna designs often fail to meet the requirements of shared resource utilization efficiency and dynamic environment adaptability, so it is necessary to build an integrated sensing and communication network architecture for flexible antenna systems.

[0003] In recent years, flexible antenna systems such as pinching antennas have received widespread attention. Pinching antenna systems create new line-of-sight links and / or enhance existing transceiver channels by applying low-cost dielectric materials at arbitrary positions on the dielectric waveguide. Unlike traditional antennas, pinching antennas can be deployed flexibly, and increasing their number will not incur additional costs. Pinching antennas avoid the high cost and difficulty of combating large-scale path loss of other flexible antennas, and their flexible radiation patterns and strong adaptability of layout capabilities show broad application prospects. The flexible deployment of pinching antennas and the dynamic changes in user mobility pose deeper challenges to resource scheduling of integrated networks. On the one hand, flexible antenna deployment can change the network topology, increasing the complexity of resource allocation; on the other hand, user mobility causes communication channels and sensing environments to change constantly, and resource demand fluctuates dynamically. How to adjust resources such as spectrum, power and time in real time to meet communication and sensing needs, and how to maintain service continuity during user mobility, are difficult problems that need to be solved in resource allocation research of integrated sensing and communication networks. SUMMARY

[0004] To solve the above technical problems existing in the prior art, the present application provides a maximum entropy reinforcement learning-based common sense fusion network resource allocation method and device, and the technical solution is as follows:

[0005] In one aspect, a maximum entropy reinforcement learning-based common sense fusion network resource allocation method is provided, which comprises:

[0006] S1, a flexible antenna-assisted common sense fusion network architecture is explored, the network architecture considers the scenario of multiple pinching antennas on the same waveguide, and comprehensively considers the communication data rate and the sensing demand, to establish a problem model of maximizing the communication data rate while meeting the sensing demand in the flexible antenna-assisted common sense fusion network;

[0007] S2, the objective function of the problem model is reconstructed, including designing the state space, action space and reward function of the optimization target reinforcement learning network;

[0008] S3, the optimization target reinforcement learning network is trained according to the criterion of maximizing the cumulative discounted reward while maximizing the entropy of the strategy, and the evaluation network and the policy network of the optimization target reinforcement learning network are updated based on the experience replay mechanism;

[0009] S4, after the training is completed, the solution of the problem model is obtained, and the solution of the problem model includes an optimal antenna position and power allocation set.

[0010] Optionally, the S1 specifically comprises:

[0011] The communication data rate maximization of user m is represented as:

[0012]

[0013] wherein M represents the total number of users, N represents the total number of pinching antennas, represents the position set of the antenna, p=(p m ) m∈M represents the power allocation set, and the power of user m is p m , represents the position of the antenna, d represents the height of the antenna, ψ m =(x m ,y m ,0) represents the position of user m, θ n is the phase shift of the transmission signal through antenna n, α represents the spherical wave parameter, e is the natural base, j represents the imaginary unit, π is an irrational number, λ represents the wavelength, σ 2 represents the additive white Gaussian noise;

[0014] The signal-to-interference-plus-noise ratio Γ of the sensing target k is consideredk , set a threshold value of the sensing accuracy sen , the sensing needs to meet k -Γ sen ≥0;

[0015] The problem model of maximizing the communication data rate while meeting the sensing needs is as follows:

[0016]

[0017] Optionally, the state space, action space and reward function of the optimization target reinforcement learning network designed in S2 specifically include:

[0018] The state space s t ∈S, Wherein, ψ t , respectively represent the pinching antenna position set, the user position set and the sensing target position set at time slot t, E t represents the remaining energy of the system;

[0019] The action space a t ∈A, Wherein, Δψ t , respectively represent the pinching antenna position change and the moving user position change, p t represents the power allocation vector at time slot t, ΔE t represents the energy consumed by the system at time slot t;

[0020] The reward function r t ∈R, Wherein τ is a weight factor.

[0021] Optionally, the criterion function of the optimization target reinforcement learning network is trained according to the criterion of maximizing the cumulative discounted reward while maximizing the entropy of the strategy in S3, and the criterion function is:

[0022]

[0023] Wherein, E(·) represents the expectation calculation, γ∈(0,1) represents the discount factor, T is the total time slot of training, represents the strategy network, and the network parameter of the strategy network is is the entropy of the strategy, and ρ represents the temperature factor, which is used to balance the importance of the reward and the entropy.

[0024] Optionally, the training process of the optimization target reinforcement learning network is:

[0025] First stage: initialize the strategy network and the evaluation network Initialize experience replay buffer B;

[0026] Second stage:

[0027] In the environment iteration, perform action update With probability P, perform state update s t+1 = P(s t+1 |s t ,a t ), generate a new experience tuple and store it into the replay buffer B←B∪{(s t ,a t ,r t (s t ,a t ),s t+1 )};

[0028] In the gradient iteration, sample a mini-batch of experience tuples B from the experience replay buffer B, where the number of experience tuples can be denoted as |B|, and then update the policy network and the evaluation network with gradient descent, where the policy network The update formula is:

[0029]

[0030] The evaluation network The update formula is:

[0031]

[0032] If the termination condition is met, the training is terminated, otherwise loop the second stage.

[0033] In another aspect, a flexible antenna assisted common sense fusion network resource allocation device based on maximum entropy reinforcement learning is provided, and the device comprises:

[0034] A building module is configured to explore a flexible antenna assisted common sense fusion network architecture, wherein the network architecture considers the scenario of multiple pinching antennas on the same waveguide, and comprehensively considers the communication data rate and the sensing demand, and builds a problem model of maximizing the communication data rate while meeting the sensing demand in the flexible antenna assisted common sense fusion network;

[0035] A reconstruction module is configured to reconstruct the objective function of the problem model, including designing the state space, the action space and the reward function of the optimization target reinforcement learning network;

[0036] A training module is configured to train the optimization target reinforcement learning network based on the criterion of maximizing the cumulative discounted reward while maximizing the entropy of the policy, and update the evaluation network and the policy network of the optimization target reinforcement learning network based on the experience replay mechanism;

[0037] An obtaining module is configured to obtain a solution of the problem model after the training is completed, the solution of the problem model including an optimal set of antenna positions and power allocation.

[0038] Optionally, the establishing module is specifically configured to:

[0039] The maximum communication data rate of the user m is represented as:

[0040]

[0041] wherein M represents a total number of users, N represents a total number of pinching antennas, represents a set of positions of the antennas, p=(p m ) m∈M represents a set of power allocations, the power of the user m being p m , represents a position of the antenna, d represents a height of the antenna, ψ m =(x m ,y m ,0) represents a position of the user m, θ n represents a phase shift of a transmission signal passing through the antenna n, a represents a spherical wave parameter, e is a natural base, j represents an imaginary unit, π is an irrational number, λ represents a wavelength, σ 2 represents an additive white Gaussian noise;

[0042] Consider a signal-to-interference-plus-noise ratio Γ k of the sensing target k, set a threshold Γ sen of the sensing accuracy, the sensing needs to satisfy Γ k -Γ sen ≥0;

[0043] The problem model of maximizing the communication data rate while meeting the sensing requirement is as follows:

[0044]

[0045] Optionally, the state space, the action space and the reward function of the optimization target reinforcement learning network designed in the reconstructing module specifically include:

[0046] The state space s t ∈S, wherein ψ t , respectively represent a set of pinching antenna positions, a set of user positions and a set of sensing target positions at the t time slot, E t represents a remaining energy of the system;

[0047] The action space a t ∈A, wherein Δψ t , p represents the change in pinching antenna position and the change in mobile user position, respectively. t Denotes the power allocation vector in time slot t, ΔE t This represents the energy consumed by the system in time slot t;

[0048] reward function r t ∈R, Where τ is the weighting factor.

[0049] Optionally, the training module uses the criterion of maximizing both the cumulative discount reward and the policy entropy to train the optimization objective reinforcement learning network. The criterion function is as follows:

[0050]

[0051] Where Ε(·) represents the expectation calculation, γ∈(0,1) represents the discount factor, and T is the total training time slot. The policy network is represented by the following network parameters: Let ρ be the entropy of the strategy, and let ρ be the temperature factor used to balance the importance of reward and entropy.

[0052] Optionally, the training process of the optimization target reinforcement learning network is as follows:

[0053] Phase 1: Initializing the Policy Network and evaluation network Initialize experience replay pool B;

[0054] Phase Two:

[0055] During environment iteration, action updates are performed. Perform state update s with probability P t+1 =P(s) t+1 |s t ,a t ), generate new experience tuples and store them in the replay pool B←B∪{(s t ,a t ,r t (s t ,a t ),s t+1 )};

[0056] During gradient iteration, a mini-batch of empirical tuples B is drawn from the empirical replay pool B, where the number of empirical tuples can be represented as |B|. Subsequently, the policy network and evaluation network are updated using gradient descent. The updated formula is:

[0057]

[0058] Evaluation Network The update formula is:

[0059]

[0060] If the termination condition is met, the training is terminated, otherwise the second stage is looped.

[0061] The technical scheme provided by the present application has at least the following beneficial effects:

[0062] The present application solves the problem of flexible antenna assisted sensing and communication fusion network resource allocation, and is particularly suitable for dynamic time-varying environment, such as mobile users and flexible antenna system caused by channel state time-varying scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] Figure 1 It is a flow chart of a sensing and communication fusion network resource allocation method based on maximum entropy reinforcement learning provided by the embodiment of the present application;

[0065] Figure 2 It is a sensing and communication fusion network resource allocation device block diagram based on maximum entropy reinforcement learning provided by the embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the technical problems, technical schemes and advantages of the present application more clear, the following will be described in detail in combination with the drawings and specific embodiments.

[0067] The embodiment of the present application provides a sensing and communication fusion network resource allocation method based on maximum entropy reinforcement learning, which can be realized by an electronic device, which can be a terminal or a server. Figure 1 The method flow chart is shown, and the processing flow can include the following steps:

[0068] S1, explore the flexible antenna assisted sensing and communication fusion network architecture, the network architecture considers the scene of multiple pinching antennas on the same waveguide, and comprehensively considers the communication data rate and sensing demand, and establishes the problem model of maximizing the communication data rate while meeting the sensing demand in the flexible antenna assisted sensing and communication fusion network;

[0069] Optionally, the S1 specifically includes:

[0070] The maximum communication data rate of user m is represented as:

[0071]

[0072] where M represents the total number of users, N represents the total number of pinching antennas, represents the position set of the antenna, p = (p m ) m∈M represents the power allocation set, the power of user m is p m , represents the position of the antenna, d represents the height of the antenna, ψ m = (x m , y m , 0) represents the position of user m, θ n is the phase shift of the transmission signal through antenna n, a represents the spherical wave parameter, e is the natural base, j represents the imaginary unit, π is an irrational number, λ represents the wavelength, σ 2 represents additive white Gaussian noise;

[0073] Consider the signal-to-interference-plus-noise ratio Γ k of the sensing target k, set the threshold of the sensing accuracy Γ sen , and the sensing needs to satisfy Γ k - Γ sen ≥ 0;

[0074] The problem model of maximizing the communication data rate while meeting the sensing needs is as follows:

[0075]

[0076] S2, reconstruct the objective function of the problem model, including designing the state space, action space and reward function of the optimization target reinforcement learning network;

[0077] Reinforcement learning learns the optimal policy through the interaction between the agent and the environment, and is particularly suitable for solving dynamic optimization problems in sensory fusion. Its advantages include the ability to adapt to environmental changes and uncertainties, the ability to optimize multiple goals simultaneously through the design of appropriate reward functions, and the ability to achieve real-time decision-making based on online learning or offline pre-training. Reinforcement learning algorithms in sensory fusion provide powerful tools for solving complex problems such as resource allocation, beamforming and environmental perception. With the continuous development of 6G technology, reinforcement learning will play an increasingly important role in sensory fusion, promoting the deep integration of communication and sensing.

[0078] The state space, action space and reward function of the optimization target reinforcement learning network designed in the embodiment of the application are as follows: according to the optimization problem model, the state space includes the positions of pinching antennas, the positions of communication users, the positions of sensing targets and the total residual energy of the system; the action space includes the position amount of pinching antennas, the position change amount of mobile users, a power distribution vector and energy consumption; and the reward function is related to communication data rate and sensing accuracy, and is specifically as follows:

[0079] The state space, action space and reward function of the optimization target reinforcement learning network designed in the S2 are specifically as follows:

[0080] The state space s t ∈ S, wherein ψ t , respectively represent the pinching antenna position set, the user position set and the sensing target position set at the t time slot, E t represents the residual energy of the system;

[0081] The action space a t ∈ A, wherein Δψ t , respectively represent the pinching antenna position change amount and the mobile user position change amount, p t represents the power distribution vector at the t time slot, ΔE t represents the energy consumed by the system at the t time slot;

[0082] The reward function r t ∈ R, wherein τ is a weight factor.

[0083] S3, the optimization target reinforcement learning network is trained according to the criterion of maximizing the cumulative discounted reward while maximizing the entropy of the strategy (the design idea aims to improve the exploration ability and robustness of the reinforcement learning algorithm), and the evaluation network and the strategy network of the optimization target reinforcement learning network are updated based on the experience replay mechanism;

[0084] Optionally, the criterion function for training the optimization target reinforcement learning network according to the criterion of maximizing the cumulative discounted reward while maximizing the entropy of the strategy in the S3 is as follows:

[0085]

[0086] wherein E(·) represents an expectation calculation, γ∈(0, 1) represents a discount factor, T is the total time slot for training, represents the strategy network, and the network parameters of the strategy network are is the entropy of the strategy, and ρ represents a temperature factor for balancing the importance of the reward and the entropy.

[0087] Experience replay mechanism updates network parameters by storing and reusing past experiences. During the algorithm training process, the data (s t ,a t ,r t ,s t+1 ) generated by the interaction between the agent and the environment is stored in a fixed-size experience pool B, and the algorithm updates the network parameters by randomly sampling a batch of data B from the buffer B, rather than directly using the latest data. The main advantage of the experience replay mechanism is to reuse historical data, reducing the demand for environmental interaction; random sampling reduces data correlation and reduces training variance; offline learning can be combined with other strategy-generated data.

[0088] Optionally, the training process of the optimization target reinforcement learning network is:

[0089] First stage: initialize the policy network and the evaluation network Initialize the experience replay pool B;

[0090] Second stage:

[0091] In the environment iteration, the action update is performed with a probability P to update the state s t+1 =P(s t+1 |s t ,a t ), generate new experience tuples and store them in the replay pool B←B∪{(s t ,a t ,r t (s t ,a t ),s t+1 )};

[0092] In the gradient iteration, a small batch of experience tuples B is extracted from the experience replay pool B, where the number of experience tuples can be represented as |B|, and then the policy network and the evaluation network are updated by gradient descent, where the policy network The update formula is:

[0093]

[0094] The update formula of the evaluation network is:

[0095]

[0096] If the termination condition (such as relative convergence) is met, the training is terminated, otherwise the second stage is looped.

[0097] S4, after the training is completed, an optimal antenna position and power allocation set are obtained.

[0098] After the training of the optimization target reinforcement learning network is completed, the problem is solved, and the policy network outputs an optimal antenna position and power allocation set.

[0099] As shown in Figure 2 The embodiment of the application also provides a maximum entropy reinforcement learning-based network resource allocation device for C2BET, which comprises:

[0100] The establishing module 210 is configured to explore a C2BET network architecture assisted by flexible antennas, the network architecture considers the scenario of multiple pinching antennas on the same waveguide, and comprehensively considers communication data rate and sensing demand, and establishes a problem model for maximizing the communication data rate while meeting the sensing demand in the C2BET network assisted by flexible antennas.

[0101] The reconstructing module 220 is configured to reconstruct the objective function of the problem model, including designing the state space, action space and reward function of the optimization target reinforcement learning network.

[0102] The training module 230 is configured to train the optimization target reinforcement learning network according to the criterion of maximizing the cumulative discounted reward while maximizing the entropy of the policy, and update the evaluation network and the policy network of the optimization target reinforcement learning network based on an experience replay mechanism.

[0103] The obtaining module 240 is configured to obtain the solution of the problem model after the training is completed, the solution of the problem model including an optimal antenna position and power allocation set.

[0104] Optionally, the establishing module is specifically configured to:

[0105] The maximization of the communication data rate of the user m is represented as:

[0106]

[0107] Wherein, M represents the total number of users, N represents the total number of pinching antennas, represents the position set of the antenna, p = (p m ) m∈M represents the power allocation set, and the power of the user m is p m , represents the position of the antenna, d represents the height of the antenna, ψ m = (x m , y m , 0) represents the position of the user m, and θ nFor the phase shift of the transmitted signal through the antenna n, a represents a spherical wave parameter, e is the natural base, j represents the imaginary unit, π is an irrational number, λ represents the wavelength, σ 2 represents an additive white Gaussian noise;

[0108] Considering the signal-to-interference-plus-noise ratio Γ of the sensing target k k , the threshold Γ of the sensing accuracy is set sen , the sensing needs to satisfy Γ k -Γ sen ≥0;

[0109] The problem model of maximizing the communication data rate while meeting the sensing requirement is as follows:

[0110]

[0111] Optionally, the state space, action space and reward function of the optimization target reinforcement learning network designed in the reconstruction module specifically include:

[0112] The state space s t ∈S, wherein ψ t , respectively represent the pinching antenna position set, the user position set and the sensing target position set at the t time slot, E t represents the remaining energy of the system;

[0113] The action space a t ∈A, wherein Δψ t , respectively represent the pinching antenna position change and the moving user position change, p t represents the power allocation vector at the t time slot, ΔE t represents the energy consumed by the system at the t time slot;

[0114] The reward function r t ∈R, wherein τ is a weight factor.

[0115] Optionally, the criterion function of the optimization target reinforcement learning network is trained in the training module according to the criterion of maximizing the cumulative discounted reward while maximizing the entropy of the strategy:

[0116]

[0117] wherein E(·) represents the expectation calculation, γ∈(0,1) represents the discount factor, T is the total training time slot, represents the strategy network, and the network parameter thereof is For the policy's entropy, p represents a temperature factor, used to balance the importance of rewards and entropy.

[0118] Optionally, the training process of the optimization target reinforcement learning network is as follows:

[0119] First stage: initialize the policy network and the evaluation network Initialize the experience replay pool B;

[0120] Second stage:

[0121] In the environment iteration, perform the action update Perform the state update s with a probability P t+1 = P(s t+1 |s t ,a t ), generate a new experience tuple and store it in the replay pool B <- B U {(s t ,a t ,r t (s t ,a t ), s t+1 )};

[0122] In the gradient iteration, draw a small batch of experience tuples B from the experience replay pool B, where the number of experience tuples can be represented as |B|, and then update the policy network and the evaluation network with gradient descent, where the policy network The update formula is as follows:

[0123]

[0124] The evaluation network The update formula is as follows:

[0125]

[0126] If the termination condition is met, the training is terminated, otherwise the second stage is looped.

[0127] The function structure of the network resource allocation device based on the maximum entropy reinforcement learning provided by the embodiment of the present application corresponds to the network resource allocation method based on the maximum entropy reinforcement learning provided by the embodiment of the present application, and will not be described here.

[0128] The above only describes the preferred embodiments of the present application, and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for resource allocation of a common sense fusion network based on maximum entropy reinforcement learning, characterized in that, The method comprises: S1, exploring a flexible antenna assisted sensing and communication network architecture, the network architecture considering a scenario of multiple pinching antennas on the same waveguide, and comprehensively considering a communication data rate and sensing requirements, establishing a problem model of maximizing the communication data rate while meeting the sensing requirements in the flexible antenna assisted sensing and communication network; S2, reconstructing an objective function of the problem model, including designing a state space, an action space and a reward function of an optimization target reinforcement learning network; S3, training the optimization target reinforcement learning network according to a criterion of maximizing cumulative discounted rewards while maximizing the entropy of a strategy, and updating an evaluation network and a policy network of the optimization target reinforcement learning network based on an experience replay mechanism; S4, after the training is completed, obtaining a solution of the problem model, the solution of the problem model including an optimal antenna position and power allocation set; The S1 specifically comprises: The maximization of the communication data rate of the user m is represented as: where M represents the total number of users, N represents the total number of pinching antennas, represents the position set of antennas, p = (p m ) m∈M represents the power allocation set, the power of user m is p m , represents the position of antennas, d represents the height of antennas, ψ m = (x m , y m , 0) represents the position of user m, θ n is the phase shift of the transmission signal through antenna n, α represents the spherical wave parameter, e is the natural base, j represents the imaginary unit, π is an irrational number, λ represents the wavelength, σ 2 represents the additive white Gaussian noise; considering a signal-to-interference-plus-noise ratio Γ of the perception target k k setting a threshold value Γ of the perception accuracy sen the perception needs to satisfy Γ k - Γ sen ≥ 0; The problem model of maximizing the communication data rate while meeting the sensing requirements is as follows: The state space, the action space and the reward function of the optimization target reinforcement learning network designed in the S2 specifically comprise: State space s t ∈ S, where ψ t , respectively represent the set of pinching antenna positions, the set of user positions and the set of perception target positions at time slot t, E t represents the remaining energy of the system; Action space a t ∈ A, where Δψ t , respectively represent the pinching antenna position variation and the moving user position variation, p t represents the power allocation vector at time slot t, ΔE t represents the energy consumed by the system at time slot t; reward function r t ∈ R, where τ is a weight factor.

2. The method of claim 1, wherein, The criterion function for training the optimization target reinforcement learning network in the S3 according to the criterion of maximizing cumulative discounted rewards while maximizing the entropy of the strategy is: where Ε(·) denotes the expected computation, γ ∈ (0, 1) denotes a discount factor, and T is the total time slots for training, denotes the policy network, whose network parameters are is the entropy of the policy, and ρ denotes a temperature factor to balance the importance of rewards and entropy.

3. The method of claim 2, wherein, The training process of the optimization target reinforcement learning network is as follows: First stage: initialize policy network and evaluation network Initialize experience replay buffer B; The second stage: During environment iteration, action updates are performed. Perform state update s with probability P t+1 =P(s) t+1 |s t ,a t ), generate new experience tuples and store them in the replay pool B←B∪{(s t ,a t ,r t (s t ,a t ),s t+1 )}; In the gradient iteration, a small batch of experience tuples B is drawn from the experience replay buffer B, where the number of experience tuples can be denoted as |B|, and then the policy network and the critic network are updated with gradient descent, where the policy network The update formula is: Evaluation network The update formula is: If the termination condition is met, the training is terminated, otherwise the second stage is looped.

4. A network resource allocation apparatus based on maximum entropy reinforcement learning, comprising: The device comprises: A establishing module configured to explore a flexible antenna assisted sensing and communication network architecture, the network architecture considering a scenario of multiple pinching antennas on the same waveguide, and comprehensively considering a communication data rate and sensing requirements, establishing a problem model of maximizing the communication data rate while meeting the sensing requirements in the flexible antenna assisted sensing and communication network; A reconstructing module configured to reconstruct an objective function of the problem model, including designing a state space, an action space and a reward function of an optimization target reinforcement learning network; A training module configured to train the optimization target reinforcement learning network according to a criterion of maximizing cumulative discounted rewards while maximizing the entropy of a strategy, and update an evaluation network and a policy network of the optimization target reinforcement learning network based on an experience replay mechanism; An obtaining module configured to obtain a solution of the problem model after the training is completed, the solution of the problem model including an optimal antenna position and power allocation set; The establishing module is specifically configured to: The maximization of the communication data rate of the user m is represented as: where M represents the total number of users, N represents the total number of pinching antennas, represents the position set of antennas, p = (p m ) m∈M represents the power allocation set, the power of user m is p m , represents the position of antennas, d represents the height of antennas, ψ m = (x m , y m , 0) represents the position of user m, θ n is the phase shift of the transmission signal through antenna n, α represents the spherical wave parameter, e is the natural base, j represents the imaginary unit, π is an irrational number, λ represents the wavelength, σ 2 represents the additive white Gaussian noise; Consider the signal-to-interference-plus-noise ratio Γ of the perceived target k k , set a threshold Γ of the perception accuracy sen , the perception needs to satisfy Γ k -Γ sen ≥ 0; The problem model of maximizing the communication data rate while meeting the sensing requirements is as follows: The state space, the action space and the reward function of the optimization target reinforcement learning network designed in the reconstructing module specifically comprise: State space s t ∈ S, where ψ t , denote the set of pinching antenna locations, the set of user locations and the set of sensing target locations at time slot t, respectively, E t denotes the remaining energy of the system; Action space a t ∈ A, where Δψ t , respectively represent the pinching antenna position variation and the moving user position variation, p t represents the power allocation vector at time slot t, ΔE t represents the energy consumed by the system at time slot t; reward function r t ∈ R, where τ is a weight factor.

5. The apparatus of claim 4, wherein, The criterion function for training the optimization target reinforcement learning network in the training module according to the criterion of maximizing cumulative discounted rewards while maximizing the entropy of the strategy is: where Ε(·) denotes the expected computation, γ ∈ (0, 1) denotes a discount factor, and T is the total time slots for training, denotes the policy network, whose network parameters are is the entropy of the policy, and ρ denotes a temperature factor to balance the importance of rewards and entropy.

6. The apparatus of claim 5, wherein, The training process of the optimization target reinforcement learning network is as follows: First stage: initialize policy network and evaluation network Initialize experience replay buffer B; The second stage: During environment iteration, action updates are performed. Perform state update s with probability P t+1 =P(s) t+1 |s t ,a t ), generate new experience tuples and store them in the replay pool B←B∪{(s t ,a t ,r t (s t ,a t ),s t+1 )}; In the gradient iteration, a small batch of experience tuples B is drawn from the experience replay buffer B, where the number of experience tuples can be denoted as |B|, and then the policy network and the critic network are updated with gradient descent, where the policy network The update formula is: Evaluation network The update formula is: If the termination condition is met, the training is terminated, otherwise the second stage is looped.

Images