A hierarchical decision-making and conflict self-resolution method for multi-dimensional resource scheduling in communication radar

Abstract

This invention provides a method for hierarchical decision-making and conflict self-resolution in multi-dimensional resource scheduling for communication radars. The method includes: Markov modeling of the problems of base station-user association scheduling, communication frequency band selection, and detection frequency band selection for unmanned aerial vehicle radars in dynamic environments; calculating approximate constraint function values ​​based on the communication and detection frequency band selections and spectrum utilization status; and performing robust action correction on frequency band selections that may lead to frequency band conflicts. Furthermore, through information interaction with the environment, the heterogeneous communication radar system center learns the association scheduling of base stations and user clusters, and the base stations and unmanned aerial vehicle radars learn the optimal strategies for communication frequency band and airborne radar frequency band selection. This method effectively solves the complex resource scheduling problem of multiple devices and multiple time granularities in heterogeneous communication radar systems.

Description

Technical Field

[0001] The present invention relates to the technical field of wireless communications, and in particular to a method for hierarchical decision-making and conflict self-resolution of multi-dimensional resource scheduling of a communication radar. Background Art

[0002] The RadChat system demonstrates the effectiveness of integrated communication and radar technology operating in the 77 GHz frequency band, particularly in leveraging existing vehicular radar technology to reduce radar interference through coordinated communication between vehicles. The successful application of this technology demonstrates the feasibility of communication radar systems in practical applications. However, as system scale expands, the number of heterogeneous devices—base stations, users, and drone-mounted radars—increases. Simultaneously, the number of base station-user communication and drone-mounted radar detection tasks increases dramatically. This significantly increases the probability of mutual interference caused by beam sidelobes when base station-user communication and drone-mounted radar detection occupy the same frequency band. This, in turn, leads to bandwidth accumulation between base station and user communication and a decrease in the probability of drone-mounted radar detection. Therefore, how to make multidimensional resource scheduling decisions in heterogeneous communication radar systems has become a hot topic of research. Traditional constrained Markov process (CMDP) methods may still violate constraints in certain time slots, failing to address the robustness of frequency band decisions made by various users in complex dynamic environments, specifically the technical issue of avoiding potential frequency band conflicts within each time slot. Summary of the Invention

[0003] The purpose of the present invention is to propose a communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method for a communication radar heterogeneous system with associated communication between base stations and user clusters and detection by multiple unmanned aerial vehicle radars, which effectively solves the complex resource scheduling problem of multiple devices and multiple time granularities in the communication radar heterogeneous system.

[0004] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0005] A communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method, the communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method comprising the following steps:

[0006] A neural network-based hierarchical reinforcement learning method is used to perform Markov modeling on the problems of base station and user association scheduling, communication frequency band selection, and UAV-mounted radar detection frequency band selection in dynamic environments. Approximate constraint function values ​​are calculated based on the communication and detection frequency band selections and spectrum utilization status to determine whether frequency band conflicts will occur. Robust action correction is performed on frequency band selections that may cause frequency band conflicts, thus constructing a multi-dimensional resource scheduling decision problem for communication radar based on multiple time granularities.

[0007] The multi-dimensional resource scheduling decision problem for communication radars based on multiple time granularities is modeled as a top-down, two-layer coupled structure. The top layer is used to determine the association between base stations and user clusters, spanning multiple fine-grained time periods. The bottom layer is divided into two different decision-making processes based on the different targets of heterogeneous devices: the communication frequency band decision between base stations and user devices, and the airborne radar frequency band decision.

[0008] By interacting with the environment, with the goal of maximizing the weighted sum of system throughput and the target detection success rate of the UAV-mounted radar, the communication radar heterogeneous system center learns the associated scheduling of base stations and user clusters, and the base stations and UAV-mounted radars learn the optimal strategy for selecting communication frequency bands and airborne radar frequency bands.

[0009] Furthermore, based on the neural network hierarchical reinforcement learning method, the Markov modeling process for base station and user association scheduling and communication frequency band selection in a dynamic environment and UAV-borne radar detection frequency band selection includes the following steps:

[0010] At the beginning of each Δ time slot, the communication radar heterogeneous system center obtains the association relationship between base stations and user clusters in the previous t-Δ time slot, the data backlog matrix of all base station task queues, and the data backlog value of all base station task queues as the current system center state, and selects the base station and user cluster association scheme;

[0011] In the next Δ time slots, after the base station and the user cluster are associated, the base station's position, the position of the target detected by the UAV radar, the signal strength of each frequency band observed by the user associated with the base station at the previous time slot t-1, the backlog of data values ​​in the task queue transmitted by the base station to all users in the user cluster and the backlog of task data of all base stations, and the communication frequency band selected by the base station and all users in the associated user cluster are collected as the status of the base station. The communication frequency band is selected for data transmission with the associated user cluster; the UAV radar obtains the position of all base stations, the signal strength of each frequency band in the previous time slot t-1 and the detection frequency band selected by all UAV radars as the status of the UAV radar, selects the detection frequency band, and detects the target;

[0012] The approximate constraint function value is calculated based on the communication frequency band selection, detection frequency band selection and spectrum utilization status to determine whether a frequency band conflict will occur, and robust action correction is performed on the frequency band selection that may cause a frequency band conflict.

[0013] Furthermore, the process of robust action correction for the frequency band selection that may lead to frequency band conflict includes the following steps: at time slot t, the communication radar heterogeneous system center observes the spectrum utilization state s t And the action set a of base station communication frequency band selection and UAV radar detection frequency band selection t; Use the constraint function of the spectrum utilization state and frequency band selection at time slot t to approximate the constraint function of the spectrum utilization state at the next time slot t+1: u n )s t ,a t (is a linear approximation, is the original value; using neural network b n (s t ;w n ) performs a parameterized linear approximation on the constraint function Interact with the dynamic environment to obtain the dataset D = {s t ,a t ,s t+1}, calculate the corresponding constraint function value b n (s t ,w n ) T a t,n and a t,n Indicates the action set for base station communication frequency band selection and UAV radar detection frequency band selection in the nth frequency band; n represents the nth frequency band;

[0014] In the offline training phase, the neural network b is trained using the dataset. n (s t ,w n ) for training, by solving the optimization problem Get the neural network parameters w n ;

[0015] In the online reasoning phase, the communication radar heterogeneous system center calculates the approximate constraint function value based on the spectrum utilization status and frequency band selection Determine whether the approximate constraint function value is within the constraint range. If This indicates that the approximate constraint function value is within the constraint range, the frequency band selection will not conflict, and there is no need for robust motion correction; if This indicates that the approximate constraint function value exceeds the constraint range. The frequency band selection will lead to frequency band conflict, and robust action correction is required. The correction process is as follows:

[0016] In Euclidean space, a correction action that is closest to the original action and does not violate the constraints is solved. By constructing the Lagrangian form of the action correction problem and analyzing the Carlo-Kuhn-Tucker conditions, the closed-form solution of the corrected action is obtained.

[0017] Furthermore, the correction process is:

[0018] The optimal band selection correction action is expressed as:

[0019]

[0020] in, Indicates the corrected action Whether the device z occupies the nth frequency band; Relax the constraints so that they become

[0021] Construct the Lagrangian form of the motion correction problem:

[0022]

[0023] where λ1=[λ 1,n ,...,λ 1,N ] and λ2=[λ 2,n ,...,λ 2,N ] T is the Lagrange multiplier, and according to the Carlo-Kuhn-Tucker condition, we get:

[0024]

[0025] Where, represents the optimal action set for base station communication frequency band selection and UAV radar detection frequency band selection after correction; λ 1,n * and represents the Lagrange multiplier corresponding to the optimal solution in the nth constraint; It represents the set of actions for selecting the optimal base station communication frequency band and UAV-borne radar detection frequency band in the nth frequency band after correction;

[0026] Solved The closed-form optimal solution of :

[0027]

[0028] Calculate it according to the following formula

[0029]

[0030] Calculate it according to the following formula

[0031]

[0032] Will Projection to The optimized frequency band selection action is obtained within the original feasible domain of

[0033] Furthermore, the communication radar heterogeneous system center learns the association scheduling between the base station and the user cluster, and the base station and the unmanned aerial vehicle radar learn the optimal strategy for selecting the communication frequency band and the airborne radar frequency band. The process includes the following steps:

[0034] After the base station and user cluster are associated, within the next Δ time slots, after the communication frequency band selection and detection frequency band selection are completed in each time slot, the base station and the UAV radar obtain the corresponding reward value and train the actor neural network and the judge neural network of the base station and the UAV radar.

[0035] When Δ time slots end, at time slot t+Δ, the communication radar heterogeneous system center obtains the corresponding reward value and trains the actor neural network and the judge neural network of the communication radar heterogeneous system center.

[0036] Furthermore, the process of training the actor neural network and the judge neural network of the base station and the UAV-mounted radar includes the following steps:

[0037] At the beginning of training, the actor neural network and judge neural network of the base station and UAV-mounted radar are initialized respectively. After the base station and UAV-mounted radar receive rewards in each time slot, the TD mean square error and loss function are calculated respectively. The base station and UAV-mounted radar use the calculated TD mean square error and loss function to update their respective actor neural network parameters and judge neural network parameters.

[0038] Furthermore, the joint base station and user cluster association scheduling and frequency band decision of the communication radar heterogeneous system is expressed as The optimization problem of the joint base station and user cluster association scheduling and frequency band scheduling strategy is expressed as:

[0039]

[0040] The system's instant reward in time slot t Among them, w d and w F Expressed as the system's preference between different goals, F k,t is the indicator variable of the successful detection of the target by the UAV-borne radar k in time slot t; Indicates that base station j sends the mth i The cumulative amount of local task queues transmitted by user devices; J and M i are the total number of base stations and the total number of user devices in the user cluster respectively; γ represents the discount factor, γ∈(0,1], γ t-1 represents the t-1 power of the discount factor;

[0041] Define the policy network θ of the system center agent for deciding the association between base stations and user clusters C and the state-value network wC , in time slot t, the macro action of the base station and user cluster association is: It indicates that the status The macro action of selecting a base station and associating a user cluster with a certain probability And the sum of the selection probabilities of all actions is equal to 1,

[0042] The association between the base station and the user cluster lasts for Δ time slots, according to The TD error calculated for value network update is:

[0043]

[0044] Where, represents the reward obtained by the agent after the base station and the user cluster associate macro action to execute Δ time slots; represents the state value function of the intelligent agent at the center of the system used to make decisions on the association between base stations and user clusters;

[0045] Use TD mean square error as the loss function:

[0046]

[0047] The parameter update method of the value network is:

[0048]

[0049] Where α represents the learning rate;

[0050] The loss function of the policy network is:

[0051]

[0052] The parameter update method of the policy network is:

[0053]

[0054] Furthermore, the process of training the actor neural network and the judge neural network of the communication radar heterogeneous system center includes the following steps:

[0055] At the beginning of training, the actor neural network and the judge neural network of the communication radar heterogeneous system center are initialized; after the communication radar heterogeneous system center obtains the reward in each Δ time slot, the TD mean square error and the loss function are calculated; the communication radar heterogeneous system center uses the calculated TD mean square error and loss function to update its own actor neural network parameters and the judge neural network parameters.

[0056] Furthermore, define the UAV radar strategy network and value network The frequency band decision action in time slot t is expressed as:

[0057]

[0058] UAV-mounted radar detection spectrum decision-making makes decisions in each small time slot and receives rewards after the action is executed The obtained TD error is expressed as:

[0059]

[0060] Where, Represents the state value function of the UAV-borne radar detection spectrum decision agent; Represents the state value function of the spectrum decision agent for communication between the base station and the user;

[0061] Define each base station j and user equipment m i Policy Network for Spectrum Decision Making and value network Base station j and user equipment m i The spectrum decision action of communication is expressed as:

[0062]

[0063] Define each base station j and user equipment m i Get rewards after data transmission The obtained TD error is expressed as:

[0064]

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method of the present invention effectively solves the complex resource scheduling problem of multiple devices and multiple time granularities in the communication radar heterogeneous system, thereby ensuring the robustness of frequency band decisions of each frequency-using device in a complex dynamic environment and avoiding potential frequency band conflicts in each time slot. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flow chart of the main steps of the communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method of the present invention.

[0068] Figure 2 This is a system model diagram for hierarchical decision-making of multi-dimensional resource scheduling for communication radars of the present invention.

[0069] Figure 3 This is a hierarchical decision structure diagram for multi-dimensional resource scheduling of communication radars of the present invention.

[0070] Figure 4 This is a specific operational flow chart of the communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method of the present invention. DETAILED DESCRIPTION

[0071] The embodiments of the present invention are described in further detail below with reference to the accompanying drawings.

[0072] This invention addresses heterogeneous communication radar scenarios, where base stations associate with user clusters for data transmission and multiple unmanned aerial vehicle (UAV) radars perform target detection, with potential frequency conflicts between base station-user communication and UAV radar detection. To address the complex resource scheduling challenges of multiple devices and multiple time granularities in heterogeneous communication radar systems, this invention proposes a hierarchical decision-making and conflict resolution method for multi-dimensional resource scheduling. At the beginning of each Δ time slot, the heterogeneous communication radar system center selects a base station-user cluster association scheme. After the base station completes the association with the user cluster, in the next Δ time slots, the base station selects a certain frequency band for communication in each time slot, and the unmanned aerial vehicle radar selects a certain frequency band for target detection. In order to obtain the optimal strategy for the system center learning association scheduling, the base station and unmanned aerial vehicle radar learn the frequency band selection, which is modeled as a semi-Markov decision process. A hierarchical structure-based robust decision algorithm for association scheduling and frequency band selection is proposed. This algorithm uses a hierarchical reinforcement learning method based on a neural network to perform Markov modeling on the base station and user association scheduling and communication frequency band selection, as well as the unmanned aerial vehicle radar detection frequency band selection problem in a dynamic environment. A robust action correction method is also used to centrally correct the communication frequency band selection and detection frequency band selection that may cause frequency band conflicts. Figure 1 The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method of the present invention includes the following steps:

[0073] Step 1: The system center obtains the association relationship between the base station and the user cluster in the last t-Δ time slot, the data backlog matrix of all base station task queues, and the data backlog value of all base station task queues as the current state of the system center.

[0074] Step 2: The system center selects an association scheme from a preset association scheme set and distributes it to each base station and user cluster.

[0075] Step 3: In the next Δ time slots, after the base station and the user cluster are associated, the base station's location is collected, the location of the target detected by the drone's radar, the signal strength of each frequency band observed by the user associated with the base station at the previous time slot t-1, the backlog of data in the task queue transmitted by the base station to all users in the user cluster and the backlog of task data of all base stations, and the communication frequency band selected by the base station and all users in the associated user cluster as the base station status.

[0076] Step 4: The base station selects a communication frequency band and transmits data with the associated user cluster. The communication frequency band selection will be performed in the next time slot t+1.

[0077] Step 5: The UAV-borne radar obtains the locations of all base stations, the signal strength of each frequency band in the previous time slot t-1, and the detection frequency band selection of all UAV-borne radars as the status of the UAV-borne radar.

[0078] Step 6: The UAV-borne radar selects the detection frequency band and detects the target. The detection frequency band selection will be executed in the next time slot t+1.

[0079] Step 7: Calculate the approximate constraint function value based on the communication band selection and the detection band selection to determine whether a band conflict will occur, and perform robust action correction on the band selection that may cause a band conflict.

[0080] Step 8: After the communication frequency band selection and detection frequency band selection are completed in the next time slot t+1, the base station and the UAV-mounted radar obtain the corresponding reward values ​​and train the neural network.

[0081] Step 9: Repeat steps 3 to 8 in each time slot until the time slot t+Δ.

[0082] Step 10: After the association scheme is executed at time slot t+Δ, the system center obtains the corresponding reward value and trains the neural network.

[0083] Step 11: Repeat steps 1 to 10 above until the algorithm converges. At this point, the action selection decision is the optimal decision.

[0084] In the communication radar heterogeneous system studied, such as Figure 2 As shown in the figure, it includes K drone-borne radars flying at a constant altitude H and a ground communication system. The ground communication system consists of J base stations and I user clusters, where each base station j (satisfying the relationship j∈{1,...,J}) only communicates with M users in user cluster i (satisfying the relationship i∈{1,...,I}, and I=J) i A set of user devices in user cluster i is represented as (satisfy relationship), and there is no intersection of user devices in each user cluster, that is, The set of all user devices in the system can be expressed as The position of the UAV-mounted radar k in the system is q UAV,k =[x k ,y k ,H], the location of base station j is q BS,j =[x j ,y j,0], user device m i The location is

[0085] Considering that when base station j is associated with user equipment m in user cluster i i When communicating, its beam may affect the user equipment m in the user cluster i′ associated with the base station j′. i′ The channel of base station j′ and user equipment m i′ Data transmission fails. According to the beam angle φ of base station j j,t and base station j and user equipment m i′ Horizontal azimuth The relationship between When the beam of base station j is i′ At the same time, when base station j and user equipment m in user cluster i associated with it i During communication, its beam may also interfere with the echo signal of the drone-mounted radar k, resulting in a decrease in the probability of the drone-mounted radar detecting the target. j,t and the horizontal azimuth angle φ between base station j and UAV-mounted radar k j,k,t The relationship between them, when |φ j,k,t -φ j,t When |≤90°, the beam of base station j will interfere with the echo signal of drone-borne radar k; otherwise, no interference will occur.

[0086] When multiple UAV radars select the same frequency band at the same time, there will be a problem of frequency interference caused by the echo signals received by the UAV radars from other UAV radars, thereby reducing the probability of the UAV radars detecting the target. When the airborne radar k detects the target, its beam may affect the user device m in the user cluster i associated with the base station j. i The channel of base station j and user equipment m causes interference. i Data transmission failed. According to the beam angle φ of the airborne radar k k,t and airborne radar k and user equipment m i Horizontal azimuth relationship, when When the beam of airborne radar k is directed to user equipment m associated with base station j, i Interference will occur on the channel with the same channel; otherwise, no interference will occur.

[0087] Based on the above-mentioned interference problem, since each base station / UAV airborne radar does not know the frequency band selected by other intelligent agents in the current t time slot when deciding the communication / detection frequency band of the current device, and the frequency band selection action can only be observed in the next t+1 time slot. Based on this, in the t time slot, it is necessary to consider that there must be no conflict in the frequency band selection between Z, Z=J+K devices. Since each device cannot observe the frequency band decision action of other devices when making a frequency band decision, in order to ensure that the frequency band decision action of Z devices with potential frequency conflicts is a t =[a 1,t ,...,a z,t ,...,a Z,t ] T , where a z,t =[a z,t,1 ,...,a z,t,n ,...,a z,t,N ] T , a z,t,n =1 means that the zth device selects the nth frequency band at time slot t, otherwise it does not select the frequency band. t =[a t,1 ,...,a t,n ,...,a t,N ] T , where a t,n =[a 1,t,n ,...,a z,t,n ,...,a Z,t,n ] T The frequency band decision action at time slot t is actually executed at time slot t+1. Therefore, to ensure that there is no frequency conflict between the Z devices at time slot t+1, the following conditions must be met:

[0088]

[0089] The 3D gain model is used to model the signals of base stations and drone-mounted radars. The antenna gain can be divided into the horizontal antenna gain G h (φ) and vertical antenna gain G v (θ), where φ is the horizontal azimuth angle and θ is the elevation angle. The antenna gain in the horizontal direction is:

[0090]

[0091] where φ h is the pointing angle of the beam, φ 3dB is the half-power beamwidth in the horizontal dimension, FBR h is the power ratio before and after passing through the antenna.

[0092] The vertical antenna gain is calculated as follows:

[0093]

[0094] where θ tilt is the downtilt angle of the beam, θ 3dB is the half-power beamwidth in the vertical dimension, and SLL is the sidelobe level. Finally, the 3D antenna gain is:

[0095] G(θ,φ)=-min{-[G h (φ)+G v (θ)],AG m}

[0096] Among them AG m Represents the antenna element gain.

[0097] According to the radar equation, the received echo signal power of the UAV-mounted radar k is:

[0098]

[0099] Among them, P R , G Rr and G Rt are radar transmit power, radar transmit antenna gain and radar receive antenna gain respectively; λ k,t is the signal wavelength of airborne radar k in time slot t; d k,tar,t is the distance between the airborne radar k and the target at time slot t.

[0100] The 3D antenna gain of the UAV-mounted radar k is:

[0101]

[0102] in, is the ground user m associated with base station j i The elevation angle between the beam of the UAV-mounted radar k, is the horizontal azimuth.

[0103] User equipment m associated with base station j i The interference signal power received from the UAV-borne radar k is:

[0104]

[0105] The power of the echo signal from the UAV-borne radar k′ that is reflected by the target and received by the UAV-borne radar k is:

[0106]

[0107] Among them, d k′,tar,t represents the distance between the UAV-borne radar k′ and the target at time slot t.

[0108] When the base station transmits data to the user equipment, since there is a LoS channel between the base station and the user equipment, at time slot t, the transmission distance from base station j to the UE m in its associated user cluster i is i The channel power gain can be calculated as:

[0109]

[0110] Among them, the variable and β represent the large-scale fading of the ground channel with a reference distance of 1m, BSj to UEm i distance, and path loss exponent.

[0111] The 3D antenna gain of base station j is:

[0112] G j,k,t (θ j,k,t ,φ j,k,t )=-min{-[G h (φ j,k,t )+G v (θ j,k,t )],AG m}

[0113] Among them, θ j,k,t is the elevation angle between the UAV radar k and the base station j beam, φ j,k,t is the horizontal azimuth.

[0114] Therefore, the power of the interference signal received by the UAV-borne radar k from the base station j is:

[0115]

[0116] Among them, P C is the transmission power of the base station; j,t is the wavelength of base station j in time slot t, d j,k,t is the physical distance between base station j and the UAV radar at time slot t, B C and B R They represent the bandwidth of the base station and the bandwidth of the airborne radar respectively.

[0117] The 3D antenna gains of other base stations j′ are:

[0118]

[0119] in, User equipment m associated with base station j j The horizontal azimuth angle between the beams of other base stations j′.

[0120] So the user equipment m associated with base station j iThe interference signal power received from other base stations j′ is

[0121]

[0122] Ground user m associated with base station j i The received signal-to-interference-and-noise ratio (SINR) at time slot t can be modeled as:

[0123]

[0124] in is the average power of the Gaussian white noise signal received at the user.

[0125] The SINR of the drone-borne radar k at time slot t is:

[0126]

[0127] in is the average power of the Gaussian white noise signal received by the UAV-borne radar.

[0128] Since the bandwidths occupied by communication equipment and drone-mounted radar equipment are different, the average power of the Gaussian white noise signals received by the two is different and can be calculated as and PSD n is the power spectral density of the Gaussian white noise in the environment.

[0129] Base station j generates a signal to user equipment m in user cluster i in time slot t-1. i The data transmitted can be represented as Task bit. Definition is the signal sent to user equipment m in user cluster i observed by base station j at the end of time slot t-1 i The task bits accumulated in the local task queue for transmission include the task bits generated but not processed before time slot t-1. In the present invention, the normalized throughput is used to measure the data transmission efficiency. Define the number of task bits between base station j and user equipment m in time slot t i The normalized throughput is Given a SINR threshold Γ0, if Then the transmitted signal can be decoded, otherwise, Therefore, base station j sends the mth user in the cluster to the user in time slot t. i The cumulative amount of local task queues transmitted by a user device can be calculated as:

[0130]

[0131] The amount of tasks accumulated in the local task queue of base station j at time slot t can be calculated as:

[0132]

[0133] Therefore, the accumulated task amount of the local task queues of all base stations in the system at time slot t can be calculated as:

[0134]

[0135] The detection capability of the UAV-borne radar for the target can be calculated by the detection probability. The detection probability P of the UAV-borne radar k in the time slot t is k,t is modeled as:

[0136]

[0137] Among them, P fa is the false alarm probability, erfc(·) is the complementary error function, that is:

[0138]

[0139] At time slot t, the drone-mounted radar k calculates the signal-to-interference-to-noise ratio (SINR) to obtain the detection probability of the target. First, a random number υ~U(0,1) that follows a (0,1) uniform distribution is generated. Then, this random number is compared with the calculated detection probability P k,t Compare, if υ≤P k,t , it means that the UAV-borne radar k successfully detects the target in time slot t, so the variable F is set k,t =1 is used to indicate the detection result of the radar in this time slot, which reflects the ability of the UAV radar to correctly identify the target; otherwise, the UAV radar cannot detect the target, and accordingly, F k,t =0.

[0140] like Figure 3 As shown in the figure, the multi-dimensional resource scheduling decision problem of communication radar based on multiple time granularities is modeled as a top-down two-layer mutually coupled structure. The top layer is used to decide the association relationship between base stations and user clusters and spans multiple fine-grained time. The bottom layer is divided into two different decisions according to the different targets of heterogeneous devices: base station and user equipment communication frequency band decision and airborne radar frequency band decision. The semi-Markov decision process has a macro action strategy and macro-action internal strategy π(a BS |s BS ) / π(a UAV |s UAV ) two strategies. The intelligent agent located in the center of the communication radar heterogeneous system selects an association relationship between a base station and a user cluster according to the macro action strategy each time. C , and then execute macro actions in parallel C The corresponding macro-action internal strategy π(a BS |sBS ) / π(a UAV |s UAV ), the internal strategy is responsible for selecting the frequency band for communication between the base station and the user equipment and the frequency band of the airborne radar until the macro action o C termination, Indicates macro action o C The termination condition (state s C There is β C (s C ) to determine the association relationship between the base station and the user cluster. C The execution step length of the top-level decision macro action The underlying agent makes decisions on frequency band selection in a fine-grained time. After Δ time slots, it decides on the new association between base stations and user clusters.

[0141] See also Figure 4 , with the goal of maximizing the weighted sum of system throughput and target detection success rate of UAV-mounted radar to solve the multi-dimensional resource scheduling problem. Therefore, the instantaneous reward of the system in time slot t is:

[0142]

[0143] Among them, w d and w F Expressed as the system's preference between different goals, F k,t is the indicator variable of whether the UAV-borne radar k successfully detects the target in time slot t.

[0144] The joint base station and user cluster association scheduling and frequency band decision of the communication radar heterogeneous system can be expressed as At the same time, since there is a need to ensure conflict-free frequency band selection constraints, the optimization problem of the joint base station and user cluster association scheduling and frequency band scheduling strategy is expressed as:

[0145]

[0146] Define the policy network θ of the system center agent for deciding the association between base stations and user clusters C and the state-value network w C , then in time slot t, the macro action of associating the base station with the user cluster can be obtained as:

[0147]

[0148] It indicates that the status The macro action of selecting a base station and associating a user cluster with a certain probability At the same time, meet:

[0149]

[0150] That is, the sum of the selection probabilities of all actions is equal to 1.

[0151] Since the association between the base station and the user cluster lasts for Δ time slots, the TD error can be calculated based on is calculated as:

[0152]

[0153] The TD error is used to update the value network in the same way as the DNN update, using the TD mean square error as the loss function:

[0154]

[0155] Similarly, the parameter update method of the value network is:

[0156]

[0157] The loss function of the policy network is:

[0158]

[0159] The parameter update method of the policy network is:

[0160]

[0161] Similarly, define the UAV radar strategy network and value network The frequency band decision action in time slot t can be expressed as:

[0162]

[0163] Different from the decision of association between base stations and user clusters, the UAV radar detection spectrum decision needs to be made in each small slot and rewarded after the action is executed. The obtained TD error can be expressed as:

[0164]

[0165] Define each base station j and user equipment m i Policy Network for Spectrum Decision Making and value network Then base station j and user equipment m i The spectrum decision action of communication can be expressed as:

[0166]

[0167] Define each base station j and user equipment m iGet rewards after data transmission The obtained TD error can be expressed as:

[0168]

[0169] UAV-borne radar k and base stations j and user equipment m i The parameter update method of the value network and policy network is the same as that of the above system center.

[0170] In order to ensure that the frequency band selection of UAV radar detection and the frequency band selection of communication between base station and user do not cause frequency band conflicts due to violation of constraints in each time slot, the present invention proposes a robust frequency band decision correction mechanism. In the offline training phase, after executing the frequency band decision action of a time slot, the spectrum utilization state s can be obtained. t , frequency band decision action a t and the state s of the next time slot t+1 So we can calculate b n (s t ,w n ) T a t,n and In order to minimize the linear approximation and the original value The gap between them, parameter w n It can be obtained through the following optimization solution:

[0171]

[0172] In the online reasoning stage, the spectrum utilization state s output by the hierarchical structure is t and frequency band decision action a t , we can get the constraint function If Explain the frequency band decision action a of the hierarchical structure output t The constraint will not be violated, that is, multiple devices with potential frequency conflicts in the next time slot will not conflict, and there is no need to make a frequency band decision action a t Otherwise, In this case, it means that the frequency band decision action a of the hierarchical structure output t The constraint is violated, that is, there may be frequency conflicts between multiple devices in the next time slot, affecting the data transmission between the base station and the user equipment or the detection of the target by the drone-mounted radar. Therefore, it is necessary to correct the frequency band decision action and replace the original a with a near-optimal but safe frequency band selection action. t .

[0173] In order to ensure that the corrected action does not violate the constraints while being as close as possible to the original strategy to obtain the maximum long-term average reward, the optimal band selection correction action can be expressed as:

[0174]

[0175] in, It is the corrected action The nth element of the device z band selection action in . This problem becomes a highly non-convex integer programming problem. Intuitively, we can Perform relaxation so that the constraint becomes

[0176] The Lagrange equation for the above constrained optimization problem is:

[0177]

[0178] where λ1=[λ 1,n ,...,λ 1,N ] and λ2=[λ 2,n ,...,λ 2,N ] T is the Lagrange multiplier, according to the Carlo-Kuhn-Tucker condition, we can get:

[0179]

[0180]

[0181] By solving the above equations, we can get The closed-form optimal solution of :

[0182]

[0183] It can be obtained by the following formula:

[0184]

[0185] It can be obtained by the following formula:

[0186]

[0187] because Each element in is a continuous variable, so it needs to be projected onto The optimized frequency band selection action is obtained within the original feasible domain of

[0188] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.

[0189] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0190] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions for executing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0192] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0193] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method, characterized by: The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method comprises the following steps: A neural network-based hierarchical reinforcement learning method is used to perform Markov modeling on the problems of base station and user association scheduling, communication frequency band selection, and UAV-mounted radar detection frequency band selection in dynamic environments. Approximate constraint function values ​​are calculated based on the communication and detection frequency band selections and spectrum utilization status to determine whether frequency band conflicts will occur. Robust action correction is performed on frequency band selections that may cause frequency band conflicts, thus constructing a multi-dimensional resource scheduling decision problem for communication radar based on multiple time granularities. The multi-dimensional resource scheduling decision problem for communication radars based on multiple time granularities is modeled as a top-down, two-layer coupled structure. The top layer is used to determine the association between base stations and user clusters, spanning multiple fine-grained time periods. The bottom layer is divided into two different decision-making processes based on the different targets of heterogeneous devices: the communication frequency band decision between base stations and user devices, and the airborne radar frequency band decision. By interacting with the environment, with the goal of maximizing the weighted sum of system throughput and the target detection success rate of the UAV-mounted radar, the communication radar heterogeneous system center learns the associated scheduling of base stations and user clusters, and the base stations and UAV-mounted radars learn the optimal strategy for selecting communication frequency bands and airborne radar frequency bands.

2. The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method according to claim 1 is characterized in that: The Markov modeling process for base station-user association scheduling and communication frequency band selection in a dynamic environment, as well as UAV-borne radar detection frequency band selection, based on a neural network-based hierarchical reinforcement learning method, includes the following steps: At the beginning of each Δ time slot, the communication radar heterogeneous system center obtains the association relationship between base stations and user clusters in the previous t-Δ time slot, the data backlog matrix of all base station task queues, and the data backlog value of all base station task queues as the current system center state, and selects the base station and user cluster association scheme; In the next Δ time slots, after the base station and the user cluster are associated, the base station's position, the position of the target detected by the UAV radar, the signal strength of each frequency band observed by the user associated with the base station at the previous time slot t-1, the backlog of data values ​​in the task queue transmitted by the base station to all users in the user cluster and the backlog of task data of all base stations, and the communication frequency band selected by the base station and all users in the associated user cluster are collected as the status of the base station. The communication frequency band is selected for data transmission with the associated user cluster; the UAV radar obtains the position of all base stations, the signal strength of each frequency band in the previous time slot t-1 and the detection frequency band selected by all UAV radars as the status of the UAV radar, selects the detection frequency band, and detects the target; The approximate constraint function value is calculated based on the communication frequency band selection, detection frequency band selection and spectrum utilization status to determine whether a frequency band conflict will occur, and robust action correction is performed on the frequency band selection that may cause a frequency band conflict.

3. The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method according to claim 2 is characterized in that: The process of robust action correction for band selection that may lead to band conflicts consists of the following steps: The spectrum utilization state s observed by the center of the communication radar heterogeneous system at time slot t t And the action set a of base station communication frequency band selection and UAV radar detection frequency band selection t ; Use the constraint function of the spectrum utilization state and frequency band selection in time slot t to approximate the constraint function of the spectrum utilization state in the next time slot t+1: u n (s t ,a t ) is a linear approximation, is the original value; using neural network b n (s t ;w n ) performs a parameterized linear approximation on the constraint function Interact with the dynamic environment to obtain the dataset D = {s t ,a t ,s t+1 }, calculate the corresponding constraint function value b n (s t ;w n ) T a t,n and a t,n Indicates the action set for base station communication frequency band selection and UAV radar detection frequency band selection in the nth frequency band; n represents the nth frequency band; In the offline training phase, the neural network b is trained using the dataset. n (s t ;w n ) for training, by solving the optimization problem Get the neural network parameters w n ; In the online reasoning phase, the communication radar heterogeneous system center calculates the approximate constraint function value based on the spectrum utilization status and frequency band selection Determine whether the approximate constraint function value is within the constraint range. If This indicates that the approximate constraint function value is within the constraint range, the frequency band selection will not conflict, and there is no need for robust motion correction; if This indicates that the approximate constraint function value exceeds the constraint range. The frequency band selection will lead to frequency band conflict, and robust action correction is required. The correction process is as follows: In Euclidean space, a correction action that is closest to the original action and does not violate the constraints is solved. By constructing the Lagrangian form of the action correction problem and analyzing the Carlo-Kuhn-Tucker conditions, the closed-form solution of the corrected action is obtained.

4. The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method according to claim 3 is characterized in that: The calibration process is: The optimal band selection correction action is expressed as: in, Indicates the corrected action Whether the device z occupies the nth frequency band; Perform relaxation so that the constraint becomes Construct the Lagrangian form of the motion correction problem: where λ1=[λ 1,n ,...,λ 1,N ] and λ2=[λ 2,n ,...,λ 2,N ] T is the Lagrange multiplier, and according to the Carlo-Kuhn-Tucker condition, we get: Where, represents the optimal action set for base station communication frequency band selection and UAV radar detection frequency band selection after correction; λ 1,n * and represents the Lagrange multiplier corresponding to the optimal solution in the nth constraint; It represents the set of actions for selecting the optimal base station communication frequency band and UAV-borne radar detection frequency band in the nth frequency band after correction; Solved The closed-form optimal solution of : Calculate it according to the following formula Calculate it according to the following formula Will Projection to The optimized frequency band selection action is obtained within the original feasible domain of 5. The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method according to claim 2 is characterized in that: The communication radar heterogeneous system center learns the association scheduling between base stations and user clusters, and the base stations and unmanned aerial vehicle radars learn the optimal strategy for communication frequency band and airborne radar frequency band selection. The process includes the following steps: After the base station and user cluster are associated, within the next Δ time slots, after the communication frequency band selection and detection frequency band selection are completed in each time slot, the base station and the UAV radar obtain the corresponding reward value and train the actor neural network and the judge neural network of the base station and the UAV radar. When Δ time slots end, at time slot t+Δ, the communication radar heterogeneous system center obtains the corresponding reward value and trains the actor neural network and the judge neural network of the communication radar heterogeneous system center.

6. The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method according to claim 5 is characterized in that: The process of training the Actor and Criterion neural networks for the base station and drone-mounted radars involves the following steps: At the beginning of training, the actor neural network and judge neural network of the base station and UAV-mounted radar are initialized respectively. After the base station and UAV-mounted radar receive rewards in each time slot, the TD error and loss function are calculated respectively. The base station and UAV-mounted radar use the calculated TD error and loss function to update their respective actor neural network parameters and judge neural network parameters.

7. The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method according to claim 6 is characterized in that: The joint base station and user cluster associated scheduling and frequency band decision of the communication radar heterogeneous system is expressed as π: The optimization problem of the joint base station and user cluster association scheduling and frequency band scheduling strategy is expressed as: The system's instant reward in time slot t Among them, w d and w F Expressed as the system's preference between different goals, F k,t is the indicator variable of the successful detection of the target by the UAV-borne radar k in time slot t; d j,mi,t Indicates that base station j sends the mth i The cumulative amount of local task queues transmitted by user devices; J and M i are the total number of base stations and the total number of user devices in the user cluster respectively; γ represents the discount factor, γ∈(0,1], γ t-1 represents the t-1 power of the discount factor; Define the policy network θ of the system center agent for deciding the association between base stations and user clusters C and the state-value network w C , in time slot t, the macro action of the base station and user cluster association is: It indicates that the status The macro action of selecting a base station and associating a user cluster with a certain probability And the sum of the selection probabilities of all actions is equal to 1, The association between the base station and the user cluster lasts for Δ time slots, according to The TD error calculated for value network update is: Where, represents the reward obtained by the agent after the base station and the user cluster associate macro action to execute Δ time slots; represents the state value function of the intelligent agent at the center of the system used to make decisions on the association between base stations and user clusters; Use TD error as the loss function: The parameter update method of the value network is: Where α represents the learning rate; The loss function of the policy network is: The parameter update method of the policy network is:

8. The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method according to claim 5 is characterized in that: The process of training the actor and judge neural networks at the heart of the communication radar heterogeneous system includes the following steps: At the beginning of training, the actor neural network and the judge neural network of the communication radar heterogeneous system center are initialized; after the communication radar heterogeneous system center obtains the reward in each Δ time slot, the TD error and loss function are calculated; the communication radar heterogeneous system center uses the calculated TD error and loss function to update its own actor neural network parameters and the judge neural network parameters.

9. The communication radar multi-dimensional resource scheduling hierarchical decision-making and conflict self-resolution method according to claim 4 is characterized in that: Defining UAV-borne Radar Strategy Networks and value network The frequency band decision action in time slot t is expressed as: UAV-mounted radar detection spectrum decision-making makes decisions in each small time slot and receives rewards after the action is executed The obtained TD error is expressed as: Where, Represents the state value function of the UAV-borne radar detection spectrum decision agent; Define each base station j and user equipment m i Policy Network for Spectrum Decision Making and value network Base station j and user equipment m i The spectrum decision action of communication is expressed as: Define each base station j and user equipment m i Get rewards after data transmission The obtained TD error is expressed as: Where, Represents the state-value function of the spectrum decision-making agent in communication between the base station and the user.

Images