A communication perception joint optimization method and system based on constrained reinforcement learning

Abstract

The present invention discloses a communication perception joint optimization method and system based on constrained reinforcement learning. The method includes: modeling the communication perception system as a constrained Markov decision process and determining the optimization objectives and constraints; constructing a policy network, a value network, and a Lagrange multiplier and collecting sample data; updating and training the policy network, the value network, and the Lagrange multiplier based on the sample data based on the optimization objectives and constraints; and the communication perception system performing information exchange based on the trained policy network, the trained value network, and the trained Lagrange multiplier. By using the present invention, the time allocation problem in the communication perception dual-function scenario can be solved when the environmental model is not known in advance. As a communication perception joint optimization method and system based on constrained reinforcement learning, the present invention can be widely used in the field of perception and communication joint technology.

Description

Technical Field

[0001] The present invention relates to the field of perception-communication joint technology, and in particular to a communication-perception joint optimization method and system based on constrained reinforcement learning. Background Art

[0002] In recent years, joint perception and communication has attracted increasing attention due to its significant advantages. First, joint perception and communication allows radar perception systems and communication systems to share frequency bands with each other, thereby reducing spectrum scarcity caused by the gradual overlap of the two spectrums. Second, joint perception and communication enables a single hardware platform to perform perception and communication functions simultaneously, thereby reducing system size and hardware costs, while meeting the needs of a large number of emerging applications for high-quality wireless connectivity and high-precision perception capabilities. However, the combination of radar perception systems and communication systems will also lead to competition for limited resources among systems. Therefore, further research is needed on the joint optimization problem of communication and perception. In fact, many works have been carried out in this regard. However, existing works do not jointly consider the indicators of timeliness and effectiveness in the optimization problem, and current works do not take into account the fact that the environment in which the system is located is often highly dynamic and uncertain, making it difficult to model when solving the optimization problem, which makes them unable to meet the needs of various application scenarios well. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a communication perception joint optimization method and system based on constrained reinforcement learning, which can solve the time allocation problem in the communication perception dual function scenario when the environmental model is unknown in advance.

[0004] The first technical solution adopted by the present invention is: a communication perception joint optimization method based on constrained reinforcement learning, comprising the following steps:

[0005] Model the communication perception system as a constrained Markov decision process and determine the optimization objectives and constraints;

[0006] Construct policy network, value network and Lagrange multiplier and collect sample data;

[0007] Based on the optimization objectives and constraints, the policy network, value network and Lagrange multiplier are updated and trained through sample data to obtain the trained policy network, trained value network and trained Lagrange multiplier;

[0008] The communication perception system performs information interaction based on the trained policy network, the trained value network and the trained Lagrange multipliers.

[0009] Furthermore, the constrained Markov decision process specifically includes the system state space, the system action space and the reward function, and its expression is specifically as follows:

[0010] (S,A,Pr(·|·),μ,R,C,γ)

[0011] In the above formula, S represents the system state space, and the state at the beginning of time slot t is defined as s t , A represents the action space, action a t represents the time allocation executed in time slot t, Pr(·|·) represents the state transition probability after executing the action, μ represents the initial distribution of the state, R represents the reward function, R t In state s t Next, perform action a t The immediate reward obtained by the agent, C represents the cost function, C t In state s t Next, perform action a t The immediate cost consumed by the agent, γ represents the discount factor.

[0012] Furthermore, the optimization objective is the long-term average perceptual mutual information in the communication perception system, and the constraint condition is the long-term average age penalty in the communication perception system, wherein:

[0013]

[0014]

[0015] In the above formula, s1 represents the initial state, δ max represents the preset constraint value of the average age penalty, represents the expectation operator, π θ represents a policy based on a parameterized deep neural network, the corresponding parameter is θ, and T represents the total number of steps to perform the task in the environment;

[0016] The expression of the constraint condition is:

[0017]

[0018] In the above formula, I rad (U(t)) represents the size of the latest data packet received by the communication receiver, represents the preset upper limit of the slope, and δ(t) represents the information age of the communication receiver.

[0019] Furthermore, the step of constructing the policy network, the value network and the Lagrange multiplier specifically includes:

[0020] The policy network includes a mean network and a standard deviation network. The policy network is used to input the system state into the policy network, construct a Gaussian distribution based on the obtained action mean and action standard deviation, sample the obtained Gaussian distribution to obtain the pre-output action, and the pre-output action is clipped to the interval [0,1] to obtain the output action;

[0021] The value network includes an input layer, a fully connected layer, an activation layer, and an output layer. The value network is used to update the objective function of the policy network to see if there is an advantage function, and to provide an estimate of the value function, which is then used to calculate the advantage function estimate.

[0022] The Lagrangian multiplier is used for Lagrangian relaxation and absorbs the cost into the reward.

[0023] Furthermore, the step of collecting sample data specifically includes:

[0024] According to the initial distribution of states in the constrained Markov decision process, the communication perception system is initialized to obtain the initialized system state;

[0025] Input the initialized system state into the policy network for action sampling processing to obtain the sampled action;

[0026] Inputting the sampled action into the communication perception system for execution to obtain preliminary experience trajectory data, wherein the preliminary experience trajectory data includes an immediate reward, an immediate cost, and a system state at the next moment;

[0027] The above-mentioned motion sampling and processing steps and the communication perception system execution steps are repeated until the preliminary experience trajectory data meets the preset data length, and the experience trajectory data is output;

[0028] The above-mentioned step of obtaining the empirical trajectory data is repeated until the empirical trajectory data meets the preset data quantity, and the sample data is constructed.

[0029] Furthermore, the step of updating and training the policy network, the value network, and the Lagrange multiplier based on the optimization objective and the constraints through sample data to obtain the trained policy network, the trained value network, and the trained Lagrange multiplier specifically includes:

[0030] Absorb the cost of the sample data into the reward through Lagrange multipliers and calculate the corresponding generalized advantage estimate;

[0031] Based on the ADAM optimizer, the Lagrange multiplier and value network parameters are updated by the stochastic gradient descent algorithm according to the corresponding generalized advantage estimate, and the policy network parameters are updated by the stochastic gradient ascent algorithm to obtain the updated policy network, updated value network and updated Lagrange multiplier.

[0032] The sample data collection step and the sample data are used to update the policy network, value network and Lagrange multiplier in a cyclic iterative manner until the preset number of iterations is met, and the trained policy network, trained value network and trained Lagrange multiplier are output.

[0033] Furthermore, the objective function of updating the Lagrange multiplier is specifically as follows:

[0034]

[0035] In the above formula, O P (β) represents the objective function of the Lagrange multiplier, and β represents the Lagrange multiplier.

[0036] Furthermore, the objective function of the updated policy network is as follows:

[0037]

[0038] In the above formula, O CLIP (θ) represents the objective function of the policy network, θ represents the parameters of the policy network, π θ (a t |s t ) means following the strategy π θ When in state s t Next select action a t The probability of represents the generalized odds estimate, θ odd represents the policy network parameters before updating, ∈ represents the pruning parameters, Indicates empirical expectations.

[0039] Furthermore, the objective function of the updated value network is as follows:

[0040]

[0041] In the above formula, O V (Θ) represents the value network objective function, Θ represents the value network parameters, V(s t |Θ) indicates that the input is s t The network output value at time γ k represents the kth power of the discount factor γ, It represents the Lagrangian penalty reward signal obtained by absorbing the cost signal C into the reward signal R through the Lagrangian multiplier β.

[0042] The second technical solution adopted by the present invention is: a communication perception joint optimization system based on constrained reinforcement learning, comprising:

[0043] A decision module is used to model the communication perception system as a constrained Markov decision process and determine the optimization objectives and constraints;

[0044] The acquisition module is used to construct the policy network, value network and Lagrange multiplier and collect sample data;

[0045] The training module updates and trains the policy network, value network, and Lagrange multiplier based on the optimization objectives and constraints using sample data to obtain the trained policy network, trained value network, and trained Lagrange multiplier.

[0046] The interaction module is used for the communication perception system to perform information interaction based on the trained policy network, the trained value network and the trained Lagrange multiplier.

[0047] The beneficial effects of the method and system of the present invention are as follows: the present invention adopts perceptual mutual information as the target, age penalty as the constraint, and uses the proximal strategy optimization-Lagrangian algorithm to solve the corresponding constraint problem, so that the dual-function node can dynamically adjust the time allocation between perception and communication according to the system state, thereby improving the perception performance and meeting the timeliness requirements of communication at the same time. It is used to solve the time allocation problem in the dual-function scenario of communication and perception, so as to cope with the environment that is highly dynamic and uncertain and difficult to model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flowchart of the steps of a communication perception joint optimization method based on constrained reinforcement learning of the present invention;

[0049] Figure 2 This is a structural block diagram of a communication perception joint optimization system based on constrained reinforcement learning of the present invention;

[0050] Figure 3 This is a flowchart of the steps of a communication perception joint optimization method based on constrained reinforcement learning according to a specific embodiment of the present invention;

[0051] Figure 4 is the average perceptual mutual information of the proposed joint optimization method under different discount factors;

[0052] Figure 5 is the average age penalty of the joint optimization method proposed in this invention under different discount factors. DETAILED DESCRIPTION

[0053] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0054] This specific embodiment considers a simple system consisting of a dual-function node, a communication receiver and a sensing target. The following are the corresponding system parameter settings: B = 100MHz, N0 = -174dBm / Hz, λ = 3 / 35m, d rad =d com =200m, P rad =P com =1W,σ RCS =1m 2 , as well as In addition, this embodiment considers that small-scale fading obeys the Rice distribution with a K factor of 5 dB. Furthermore, this embodiment takes perceptual mutual information as the goal and age penalty as the constraint to study the time allocation problem in this simple system.

[0055] Reference Figure 1 and Figure 3 The present invention provides a communication perception joint optimization method based on constrained reinforcement learning, which includes the following steps:

[0056] S1. Model the communication-perception dual-function system as a constrained Markov decision process and determine the optimization objectives and corresponding constraints.

[0057] S11. Model the above system as a constrained Markov decision process defined by the tuple (S, A, Pr(·|·), μ, R, C, γ);

[0058] Specifically, the dual-function system is modeled as a constrained Markov decision process defined by the tuple (S, A, Pr(·|·), μ, R, C, γ);

[0059] Where S is the state space, where the state s at the beginning of time slot t is t Defined as (δ(t),I rad (U(t)),min(t-Nt,δ,IradNt,δ(t)=t-Ut represents the information age of the communication receiver, Ut represents the generation time slot of the latest received data packet, I rdd (U(t)) represents the size of the latest data packet received by the communication receiver, represents the information age of the dual-function node, N(t) represents the time slot in which the data packet was generated recently, Indicates the preset upper age limit of information, I rad (N(t)) represents the size of the latest generated data packet;

[0060] A is the action space, action a t represents the time allocation executed in time slot t;

[0061] In particular, the packet size is measured using the perceptual mutual information, and the mutual information sensed within time slot t is obtained as follows:

[0062]

[0063] In the above formula, τ is the duration of a time slot, g rad represents the path loss of the sensing signal, h rad (t) represents the small-scale fading from the dual-function node to the sensing target, P rad is the transmit power of the sensing signal, B is the system bandwidth, and N0 is the power spectral density of the background noise;

[0064] g rad Specifically defined as:

[0065]

[0066] In the above formula, is the transmit antenna gain of the dual-function node, is the receiving antenna gain of the dual-function node, σ RCS is the radar cross section of the target, λ is the wavelength, d rad is the distance between the dual-function node and the target;

[0067] Considering that the content obtained by perception is the content to be sent in subsequent communication, the success of data packet transmission depends on the relationship between the perceived mutual information and the amount of transmitted data. The amount of transmitted data is obtained by the following formula:

[0068]

[0069] In the above formula, g com Indicates the path loss of the communication signal, h com (t) represents the small-scale fading from the dual-function node to the communication receiver, P com is the transmission power of the communication signal;

[0070] g com Specifically defined as:

[0071]

[0072] In the above formula, is the receiving antenna gain of the communication receiver, d com is the distance between the dual-function node and the communication receiver, and Pr(·|·) represents the state transition probability after executing the action;

[0073] In particular, the evolution of information age δ(t) can be given by the following formula,

[0074]

[0075] In the above formula, μ is the initial distribution of states;

[0076] R is the reward function, R t In state s t Next, perform action a t The immediate reward obtained by the agent is defined as:

[0077] R t =I rad (t)

[0078] C is the cost function, C t In state s t Next, perform action a t The immediate cost incurred by the post-agent is defined as:

[0079] C t =f(t)

[0080] Among them, the age penalty f(t) is obtained by combining the perceptual mutual information and the information age to make up for the defects of the single information age. A simple age penalty function is given as follows:

[0081]

[0082] In the above formula, is the preset upper limit of the slope, and γ is the discount factor used to control the balance between short-term rewards and long-term rewards.

[0083] S12. Determine the long-term average perceptual mutual information as the optimization target and the long-term average age penalty as the constraint, and construct a constrained optimization problem to be solved.

[0084] Specifically, we determine the long-term average perceptual mutual information as the optimization target and the long-term average age penalty as the constraint, and obtain the following constrained optimization problem:

[0085]

[0086]

[0087] In the above formula, s1 is the initial state, δ max is the preset constraint value of the average age penalty.

[0088] S2, constructing strategy network, value network and Lagrange multiplier;

[0089] S21. Construct a policy network, where the hidden layer of the mean network is set to 2 fully connected layers with 256 neurons, and ReLU is used as the hidden layer activation function, while the parameters of the standard deviation network are initialized to 1;

[0090] Specifically, a policy network is constructed, which is further divided into a mean network and a standard deviation network. The basic structure of a mean network is: input layer - fully connected layer - activation layer - output layer - activation layer. The policy network takes the state as input and outputs the time allocation action. Therefore, the number of units in the input layer should be the same as the number of state elements, and the number of units in the output layer should be the same as the number of action elements. The number of neurons in the fully connected layer-activation layer module and its components can be increased or decreased as appropriate based on the specific situation. Since the time allocation action is a time allocation ratio with a value range of [0, 1], the activation layer after the output layer uses a sigmoid activation function to adjust the output action mean to between 0 and 1. The standard deviation network is a list of trainable parameters with the same number of action elements. That is, a state-independent standard deviation is trained using the nn.Parameter method. This is often better than having the neural network output both the mean and standard deviation simultaneously. The strategy adopts Gaussian strategy, that is, the process of outputting action is as follows: the state inputs the strategy network, and a Gaussian distribution is constructed based on the obtained action mean and action standard deviation. The Gaussian distribution obtained by sampling is the pre-output action, and the pre-output action is clipped to the interval [0,1] to obtain the output action. The parameters of the strategy network are represented by θ, and the strategy π is followed. θ When in state s t Next select action a t The probability is expressed as π θ (a t |s t ).

[0091] S22. Build a value network, where the hidden layer of the value network is also set to 2 fully connected layers with 256 neurons, and ReLU is also used as the hidden layer activation function;

[0092] Specifically, we construct a value network. The basic structure of the value network is similar to that of the mean network, which is: input layer - fully connected layer - activation layer - output layer. The difference is that there is no activation layer after the output layer, and the number of units in the output layer of the value network should be 1. The parameters of the value network are represented by Θ, and the input is s t The network output is expressed as V(s t |Θ).

[0093] S23. Construct a Lagrange multiplier and initialize it to 1.

[0094] Specifically, a Lagrange multiplier is constructed using trainable parameters for subsequent Lagrange relaxation to absorb the cost into the reward; the Lagrange multiplier is represented by β;

[0095] In particular, both the policy network and the value network use orthogonal initialization. Specifically, the weight matrix is ​​initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 1. This weight matrix is ​​then subjected to singular value decomposition to obtain two orthogonal matrices, one of which is used as the weight matrix for the neural network layer. The Lagrange multipliers are initialized based on preset initial values.

[0096] S3, the agent interacts with the environment to collect and store sample data;

[0097] Specifically, the agent interacts with the environment to collect N = 10 trajectories, each of length T = 1000.

[0098] S31, initialize the system state according to the initial state distribution μ;

[0099] S32, inputting the system state into the policy network to sample actions;

[0100] S33: Execute the sampled action and receive the immediate reward, immediate cost, and new system state;

[0101] S34, after executing S32 and S33 for T times, the obtained experience trajectory (s t ,a t ,R t ,C t ,s t+1 ), t=1,…,T is stored in the pre-built sample buffer, where T is the preset trajectory length;

[0102] S35 , loop through S31 - S34 to collect N trajectories. Trajectories may also be collected in parallel, that is, directly initialize N environments in step S31 to collect data.

[0103] S4, training the policy network, the value network and the Lagrange multiplier through the collected data;

[0104] Specifically, in this embodiment, the learning rate of the policy network and the value network is set to 0.0003, and the learning rate of the Lagrange multiplier is set to 0.05.

[0105] S41, pre-processing the data in the sample buffer, specifically including: using Lagrange multipliers to absorb the cost into the reward, that is, calculating The generalized odds estimate is calculated using the following formula,

[0106]

[0107] In the above formula, ζ is a parameter that controls the trade-off between bias and variance; the long-term discounted penalty reward is calculated, that is,

[0108] S42. Use the ADAM optimizer to update the Lagrange multiplier β by stochastic gradient descent. The update is performed once. The objective function used for the update is as follows:

[0109]

[0110] S43, using the ADAM optimizer to update the policy network parameters θ through stochastic gradient ascent, the update is performed M times, and the objective function used for the update is as follows,

[0111]

[0112] Among them, ∈ is the cropping parameter, θ odd are the policy network parameters before updating.

[0113] S44, using the ADAM optimizer to update the network parameter Θ by stochastic gradient descent, the update is performed M times, and the objective function used for the update is as follows,

[0114]

[0115] In particular, the policy network parameters, value network parameters and the learning rate of the Lagrange multiplier, θ lr 、Θ lr and θ lr , can be set according to actual conditions. Generally speaking, the settings of learning rate and M need to make the time scale of network update greater than the time scale of multiplier update.

[0116] S5, loop through S3 and S4 until the exit condition is met;

[0117] Specifically, the loop executes S3 and S4 a certain number of times L and then exits; the number of loop iterations L is based on the preset total number of steps, the number of trajectories N required for each iteration, and the number of steps T required for each trajectory. The corresponding calculation formula is as follows

[0118] In particular, the total number of training steps is preset to 10,000,000, so the loop iteration will be Second-rate.

[0119] S6. Output and save the trained policy network, value network, and Lagrange multiplier.

[0120] This embodiment describes the communication perception joint optimization method based on constrained reinforcement learning provided by the present invention. In fact, the method provided by the present invention uses the proximal policy optimization-Lagrangian algorithm to approximately solve the constrained optimization problem with the average reward objective function. Therefore, it is necessary to explore the performance under different discount factors γ = {0.9, 0.95, 0.99, 0.999}. In order to obtain the optimal performance under different discount factors, this embodiment tunes the hyperparameters ζ = {0.9, 0.95, 0.99} and M = {10, 40, 80}. Figure 4 and Figure 5 The joint optimization method provided by the present invention can meet the preset average age penalty constraint under different discount factors and improve the obtained average perceptual mutual information as much as possible; in addition, in the simulation setting of this embodiment, the joint optimization method provided by the present invention has the best performance when the discount factor γ = 0.95.

[0121] Reference Figure 2 , a communication-aware joint optimization system based on constrained reinforcement learning, including:

[0122] A decision module is used to model the communication perception system as a constrained Markov decision process and determine the optimization objectives and constraints;

[0123] The acquisition module is used to construct the policy network, value network and Lagrange multiplier and collect sample data;

[0124] The training module updates and trains the policy network, value network, and Lagrange multiplier based on the optimization objectives and constraints using sample data to obtain the trained policy network, trained value network, and trained Lagrange multiplier.

[0125] The interaction module is used for the communication perception system to perform information interaction based on the trained policy network, the trained value network and the trained Lagrange multiplier.

[0126] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0127] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A communication perception joint optimization method based on constrained reinforcement learning, characterized in that: The following steps are involved: Model the communication perception system as a constrained Markov decision process and determine the optimization objectives and constraints; Construct policy network, value network and Lagrange multiplier and collect sample data; Based on the optimization objectives and constraints, the policy network, value network and Lagrange multiplier are updated and trained through sample data to obtain the trained policy network, trained value network and trained Lagrange multiplier; The communication perception system performs information interaction based on the trained policy network, the trained value network and the trained Lagrange multiplier; The optimization objective is the long-term average perceptual mutual information in the communication perception system, and the constraint condition is the long-term average age penalty in the communication perception system, where: The expression of the optimization objective is: In the above formula, s1 represents the initial state, δ max represents the preset constraint value of the average age penalty, represents the expectation operator, π θ represents a policy based on a parameterized deep neural network, the corresponding parameter is θ, and T represents the total number of steps to perform the task in the environment; The expression of the constraint condition is: In the above formula, I rad (U(t)) represents the size of the latest data packet received by the communication receiver, represents the preset upper limit of the slope, and δ(t) represents the information age of the communication receiver; Based on the optimization objectives and constraints, the policy network, value network and Lagrange multipliers are updated and trained through sample data to obtain the trained policy network, trained value network and trained Lagrange multipliers, specifically including: Absorb the cost of the sample data into the reward through Lagrange multipliers and calculate the corresponding generalized advantage estimate; Based on the ADAM optimizer, the Lagrange multiplier and value network parameters are updated by the stochastic gradient descent algorithm according to the corresponding generalized advantage estimate, and the policy network parameters are updated by the stochastic gradient ascent algorithm to obtain the updated policy network, updated value network and updated Lagrange multiplier. The sample data collection step and the sample data are used to update the policy network, value network and Lagrange multiplier in a cyclic iterative manner until the preset number of iterations is met, and the trained policy network, trained value network and trained Lagrange multiplier are output.

2. The communication perception joint optimization method based on constrained reinforcement learning according to claim 1, characterized in that: The constrained Markov decision process specifically includes the system state space, the system action space and the reward function, and its expression is as follows: (S,A,Pr(·|·),μ,R,C,γ) In the above formula, S represents the system state space, and the state at the beginning of time slot t is defined as s t , A represents the action space, action a t represents the time allocation executed in time slot t, Pr(·|·) represents the state transition probability after executing the action, μ represents the initial distribution of the state, R represents the reward function, R t In state s t Next, perform action a t The immediate reward obtained by the agent, C represents the cost function, C t In state s t Next, perform action a t The immediate cost consumed by the agent, γ represents the discount factor.

3. The communication perception joint optimization method based on constrained reinforcement learning according to claim 2, characterized in that: The step of constructing the policy network, the value network and the Lagrange multiplier specifically includes: The policy network includes a mean network and a standard deviation network. The policy network is used to input the system state into the policy network, construct a Gaussian distribution based on the obtained action mean and action standard deviation, sample the obtained Gaussian distribution to obtain the pre-output action, and the pre-output action is clipped to the interval [0,1] to obtain the output action; The value network includes an input layer, a fully connected layer, an activation layer, and an output layer. The value network is used to update the objective function of the policy network to see if there is an advantage function, and to provide an estimate of the value function, which is then used to calculate the advantage function estimate. The Lagrangian multiplier is used for Lagrangian relaxation and absorbs the cost into the reward.

4. The communication perception joint optimization method based on constrained reinforcement learning according to claim 3 is characterized in that: The step of collecting sample data specifically includes: According to the initial distribution of states in the constrained Markov decision process, the communication perception system is initialized to obtain the initialized system state; Input the initialized system state into the policy network for action sampling processing to obtain the sampled action; Inputting the sampled action into the communication perception system for execution to obtain preliminary experience trajectory data, wherein the preliminary experience trajectory data includes an immediate reward, an immediate cost, and a system state at the next moment; The above-mentioned motion sampling and processing steps and the communication perception system execution steps are repeated until the preliminary experience trajectory data meets the preset data length, and the experience trajectory data is output; The above-mentioned step of obtaining the empirical trajectory data is repeated until the empirical trajectory data meets the preset data quantity, and the sample data is constructed.

5. The communication perception joint optimization method based on constrained reinforcement learning according to claim 4 is characterized in that: The objective function for updating the Lagrange multiplier is as follows: In the above formula, O P (β) represents the objective function of the Lagrange multiplier, and β represents the Lagrange multiplier.

6. The communication perception joint optimization method based on constrained reinforcement learning according to claim 5, characterized in that: The objective function of the updated policy network is as follows: In the above formula, O CLIP (θ) represents the objective function of the policy network, θ represents the parameters of the policy network, π θ (a t ∣s t ) means following the strategy π θ When in state s t Next select action a t The probability of represents the generalized odds estimate, θ odd represents the policy network parameters before updating, ∈ represents the pruning parameters, Indicates empirical expectations.

7. The communication perception joint optimization method based on constrained reinforcement learning according to claim 6, characterized in that: The objective function of the updated value network is as follows: In the above formula, O V (Θ) represents the value network objective function, Θ represents the value network parameters, V(S t |Θ) indicates that the input is s t The network output value at time γ k represents the kth power of the discount factor γ, It represents the Lagrangian penalty reward signal obtained by absorbing the cost signal C into the reward signal R through the Lagrangian multiplier β.

8. A communication perception joint optimization system based on constrained reinforcement learning, characterized in that: The method for executing the communication-aware joint optimization method based on constrained reinforcement learning according to claim 1 comprises the following modules: A decision module is used to model the communication perception system as a constrained Markov decision process and determine the optimization objectives and constraints; The acquisition module is used to construct the policy network, value network and Lagrange multiplier and collect sample data; The training module updates and trains the policy network, value network, and Lagrange multiplier based on the optimization objectives and constraints using sample data to obtain the trained policy network, trained value network, and trained Lagrange multiplier. The interaction module is used for the communication perception system to perform information interaction based on the trained policy network, the trained value network and the trained Lagrange multiplier.

Images