Multi-terminal gateway resource scheduling method and system based on deep reinforcement learning

Abstract

The present invention discloses a multi-terminal gateway resource scheduling method and system based on deep reinforcement learning, the method comprising: initializing the terminal gateway and obtaining the current environmental state; extracting and evaluating the current environmental state based on a policy execution network and a policy evaluation network to obtain a recommended spread spectrum code scheme; scheduling the terminal gateway resources according to the recommended spread spectrum code scheme to obtain a scheduling result; updating the policy execution network and the policy evaluation network according to the scheduling result, and scheduling the terminal gateway resources based on the updated policy execution network. By using the present invention, the terminal gateways can efficiently share limited spread spectrum communication resources while reducing the complexity of the multi-terminal gateway system. As a multi-terminal gateway resource scheduling method and system based on deep reinforcement learning, the present invention can be widely used in the field of synchronous satellite Internet of Things spread spectrum communication technology.

Description

Technical Field

[0001] The present invention relates to the field of synchronous satellite Internet of Things spread spectrum communication technology, and in particular to a multi-terminal gateway resource scheduling method and system based on deep reinforcement learning. Background Art

[0002] Based on geosynchronous orbit satellites, a new generation of satellite networks for the Internet of Things is formed. The system usually relies on satellite gateways distributed in various places to complete the data reception and processing of terminal sensor devices, and realizes communication connection with satellite gateways through satellite forwarding. Different satellite gateways may be in different communication scenarios or responsible for collecting information of terminal devices of different importance levels. The available resources of satellite Internet of Things systems are usually limited, such as the number of allocable spread spectrum codes in the spread spectrum system. Since the communication method based on transparent forwarding of geosynchronous orbit satellites will introduce an inherent transmission delay of 250ms-270ms, the commonly used resource scheduling method currently adopts a centralized scheduling method. After the needs of all terminals are aggregated together, they are ranked according to the importance of each need and then the resources are uniformly scheduled and allocated. This resource scheduling method that relies on the decision-making center will bring large scheduling delays for each resource scheduling, and cannot meet the special requirements of the multi-terminal scenario of the Internet of Things. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a multi-terminal gateway resource scheduling method and system based on deep reinforcement learning, which can enable terminal gateways to efficiently share limited spread spectrum communication resources while reducing the complexity of the multi-terminal gateway system.

[0004] The first technical solution adopted by the present invention is: a multi-terminal gateway resource scheduling method based on deep reinforcement learning, comprising the following steps:

[0005] Initialize the terminal gateway and obtain the current environment status;

[0006] The current environment state is extracted and processed by the policy execution network and the extraction result of the policy execution network is evaluated and processed by the policy evaluation network to obtain the recommended spread spectrum code scheme;

[0007] Perform resource scheduling training of the terminal gateway according to the recommended spreading code scheme and obtain the scheduling result;

[0008] The policy execution network and the policy evaluation network are updated according to the scheduling result, and the resource scheduling of the terminal gateway is performed based on the updated policy execution network.

[0009] Further, the step of initializing the terminal gateway and obtaining the current environment state specifically includes:

[0010] Initializing a multi-terminal gateway system and marking the terminal gateway, wherein the multi-terminal gateway system includes a plurality of terminal gateways;

[0011] The current environment state of the terminal gateway is obtained, where the current environment state includes system available spread spectrum code information and its own service state.

[0012] Furthermore, the step of extracting and processing the current environment state through the policy execution network and evaluating and processing the extraction result of the policy execution network in combination with the policy evaluation network to obtain a recommended spreading code scheme specifically includes:

[0013] The current environment state is extracted and processed through the policy execution network to obtain a preliminary recommended spreading code scheme;

[0014] The preliminary recommended spreading code scheme is evaluated and processed through the strategy evaluation network and the current environmental status to obtain the recommended spreading code scheme.

[0015] Furthermore, the step of extracting and processing the current environmental state through the policy execution network to obtain a preliminary recommended spreading code scheme specifically includes:

[0016] Performing feature extraction processing on the current environment state based on the policy execution network input layer to obtain first feature information;

[0017] Based on the policy execution network convergence layer, data dimension extraction and processing are performed on the current environment state to obtain first high-dimensional information;

[0018] Based on the output layer of the strategy execution network, the first feature information and the first high-dimensional information are combined to output the value corresponding to the environmental state at the current time and perform a maximum value operation to obtain a preliminary recommended spread spectrum code scheme.

[0019] Furthermore, the step of evaluating the preliminary recommended spreading code scheme by evaluating the network and the current environment state through the strategy to obtain the recommended spreading code scheme specifically includes:

[0020] Based on the input layer of the strategy evaluation network, feature extraction is performed on the current environmental state and the preliminary recommended decision plan to obtain the second feature information;

[0021] Based on the policy evaluation network aggregation layer, the data dimension extraction and processing of the current environmental status and the preliminary recommended decision plan are performed to obtain the second highest dimensional information;

[0022] Based on the policy execution network output layer, the recommended spreading code scheme is output in combination with the second feature information and the second high-dimensional information.

[0023] Further, the step of performing resource scheduling training of the terminal gateway according to the recommended spreading code scheme to obtain a scheduling result specifically includes:

[0024] Uniformly distribute the recommended spreading code scheme to obtain random numbers;

[0025] Judge the random number;

[0026] If the random number is greater than the exploration coefficient, the greedy strategy is executed to select the recommended spread spectrum code scheme;

[0027] If the random number is less than or equal to the exploration coefficient, the exploration strategy is executed and a random spread spectrum code scheme is selected;

[0028] The resource scheduling training of the terminal gateway is performed according to the selected scheme to obtain the scheduling result.

[0029] Further, the step of updating the policy execution network and the policy evaluation network according to the scheduling result, and performing resource scheduling of the terminal gateway based on the updated policy execution network specifically includes:

[0030] Randomly select the scheduling results and update the strategy execution network and strategy evaluation network through the multi-agent deep reinforcement algorithm to obtain the training difference;

[0031] Make a judgment based on the training difference, judge until the training cutoff condition is met, and output the updated strategy execution network and the updated strategy evaluation network;

[0032] The resource scheduling of the terminal gateway is performed based on the updated policy execution network.

[0033] Furthermore, the update formula of the strategy evaluation network is as follows:

[0034]

[0035]

[0036] In the above formula, L n (ω) represents the error function of the calculation, α represents the learning rate, ω represents the parameters of the policy evaluation network, B represents the batch size of each calculation, S represents the current environment, π(S) represents the current decision action, S,A,R,S′ represents the scheduling experience extracted from this training, R represents the timely reward of the environment feedback after the action is executed, S′ represents the state of the environment after the action is executed, γ represents the decay rate, Q t , Q both represent the strategy evaluation network, π t , π both represent the policy execution network.

[0037] Furthermore, the update formula of the policy execution network is as follows:

[0038]

[0039]

[0040] In the above formula, J n (θ) represents the calculated error function, α represents the learning rate, and θ represents the parameters of the policy execution network.

[0041] The second technical solution adopted by the present invention is: a multi-terminal gateway resource scheduling system based on deep reinforcement learning, comprising:

[0042] The acquisition module is used to initialize the terminal gateway and obtain the current environment status;

[0043] An evaluation module is used to extract and process the current environmental state through the policy execution network and evaluate the extraction result of the policy execution network in combination with the policy evaluation network to obtain a recommended spread spectrum code scheme;

[0044] The scheduling module is used to perform resource scheduling training of the terminal gateway according to the recommended spreading code scheme and obtain the scheduling result;

[0045] The updating module is used to update the policy execution network and the policy evaluation network according to the scheduling result, and perform resource scheduling of the terminal gateway based on the updated policy execution network.

[0046] The beneficial effects of the method and system of the present invention are as follows: the present invention performs feature extraction processing on the time point status of the terminal gateway through the policy execution network, obtains the difference in the resource performance requirements of each terminal gateway and the difference between the available resources and the business status of different terminal gateways themselves, evaluates and processes the environmental status at the current time through the policy evaluation network, reduces the situation of cache full data overflow and discard, and then updates the policy execution network and uses it for resource scheduling of the terminal gateway, thereby reducing the number of communication failures of the multi-terminal gateway system due to the simultaneous use of the same spread spectrum code, enabling the terminal gateways to efficiently share limited spread spectrum communication resources while reducing the complexity of the multi-terminal gateway system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the steps of the multi-terminal gateway resource scheduling method based on deep reinforcement learning of the present invention;

[0048] Figure 2 It is a structural block diagram of a multi-terminal gateway resource scheduling system based on deep reinforcement learning of the present invention;

[0049] Figure 3 It is a system framework diagram of the multi-terminal gateway system resource scheduling method of the present invention;

[0050] Figure 4 It is a schematic diagram of the structure of the strategy evaluation network of the present invention;

[0051] Figure 5 It is a schematic diagram of the structure of the strategy execution network of the present invention;

[0052] Figure 6 It is a comparative data analysis diagram of the number of conflicts using the resource scheduling method of the present invention compared with the commonly used scheduling algorithm under different numbers of terminal gateways;

[0053] Figure 7 It is a comparative data analysis diagram of the amount of discarded data using the resource scheduling method of the present invention compared with the commonly used scheduling algorithm under different numbers of terminal gateways;

[0054] Figure 8 It is a comparative data analysis chart of the amount of data successfully transmitted using the resource scheduling method of the present invention compared with the commonly used scheduling algorithm under different numbers of terminal gateways. DETAILED DESCRIPTION

[0055] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] Reference Figure 1 and Figure 3 The present invention provides a multi-terminal gateway resource scheduling method based on deep reinforcement learning, which includes the following steps:

[0057] S1. Initialize the terminal gateway and obtain the current environment status;

[0058] S11, initializing a multi-terminal gateway system and marking the terminal gateway, wherein the multi-terminal gateway system includes a plurality of terminal gateways;

[0059] Specifically, the parameters of the multi-terminal gateway system are initialized to determine the multi-terminal gateway satellite Internet of Things system, which includes N satellite terminal gateways with data collection and transmission functions, namely, terminal gateways, represented as a set Each terminal gateway is marked as n, The terminal gateway is marked to facilitate the description of the current environment state obtained by the policy evaluation network, and to which terminal gateway it corresponds;

[0060] Further consider a satellite IoT system with N (N = 6, 8, 10, 12) terminal gateways working together. The terminal gateway makes a decision on the use of spread spectrum code resources at the beginning of each time slot. The number of the selected spread spectrum code is determined based on the observed state of the available spread spectrum codes in the system and its own demand for resource performance, communication conditions, cache data volume, and the amount of cache overflow data in the previous time slot. At the same time, it is also necessary to estimate the decisions of other terminal gateways to reduce system conflicts. In the satellite spread spectrum communication scenario, the success of spread spectrum communication depends on whether multiple terminals use the same set of spread spectrum codes for communication at the same time, and the communication between different spread spectrum codes is independent of each other.

[0061] S12: Acquire the current environment state of the terminal gateway, where the current environment state includes the system available spread spectrum code information and its own service state.

[0062] Specifically, the current environmental state can also be expressed as the environmental state. In the centralized training stage, the current environmental state can be generated through simulation experiments or sampled and recorded in the actual system to obtain the state information used in training. In order to fully simulate the state in the actual system to pursue better scheduling effects, during the entire spread spectrum code resource scheduling process, at each communication time point, the gateway n will obtain the current environmental state, which includes all available spread spectrum code information in the current system C(t) = {c1(t),…c M (t)} and its own business status S n (t) = {q n (t),e n (t),δ n (t),Δ n (t)}, wherein the business status includes the demand for resource performance q n (t), the communication condition e at the current communication time point n (t), the amount of data currently cached δ n (t), the amount of old data discarded due to cache fullness at the previous time point Δ n (t) and is input into the policy execution network and generates the corresponding solution action a n (t);

[0063] The available spreading code information {c1(t), ...c M (t)} is specifically expressed as: M represents the total number of available spreading codes, c m (t)∈{1,2} represents the anti-interference performance of the corresponding spreading code. The gateway can obtain greater transmission rewards by using spreading codes with higher performance;

[0064] The requirement for resource performance q n(t) is specifically expressed as: corresponding to the two-level anti-interference performance of the spread spectrum code, when the performance of the spread spectrum code selected by the gateway is not lower than its own requirements, that is, c m (t)≥q n (t), it means that the gateway communicates as expected and can obtain the full transmission reward; when the spreading code resources used do not meet the gateway performance requirements, that is, c m (t) n (t), indicating that the gateway transmission anti-interference performance is poor and can only obtain transmission rewards with attenuation, where q n (t)∈{1,2};

[0065] The communication condition at the current communication time point n (t) is specifically expressed as follows: the terminal gateway is in different physical scenarios, and may be affected by sudden strong interference, busy data processing and other factors, and may not support transmission communication at all communication time points. n (t) = 0, the corresponding gateway does not try to select the spread spectrum code communication at the current communication time point; when e n When (t) = 1, the gateway will select the spreading code resource according to the strategy execution network recommendation, where e n (t)∈{0,1};

[0066] The current cached data volume δ n (t) is specifically expressed as: Due to the design of the frame structure of the multi-terminal gateway system, the maximum amount of data that can be sent during each communication is determined, expressed as 1, so the amount of data that the gateway can cache is δ n (t)∈[0,1], in addition, at each communication time point, the terminal gateway n will collect the terminal sensor data that it is responsible for, and the data will arrive according to a certain Poisson distribution probability. The amount and probability of data arriving at different gateways at a time are different. After the terminal gateway receives and processes the data, it will store the data in the cache. After the spread spectrum code is selected to modulate the cache and send it out, the cache will be cleared, that is, δ n (t+1)=0;

[0067] The amount of old data Δ discarded due to cache fullness at the previous time point n (t) is specifically expressed as: The specific meaning is to measure the waiting time between two choices of sending by the terminal gateway. If the gateway stops sending for too long, the amount of data arriving will overflow because it exceeds the cache. In order to ensure the timely validity of the data, it is necessary to discard the earliest arriving data to reserve space for new data. When δ n When (t)>1, the output is Δ n (t) = 1-δ n (t), and at the same time n (t) is set to 1, otherwise the output is Δ n ​(t) = 0;

[0068] The corresponding scheme action a n (t) is specifically expressed as: It is recommended that the terminal gateway n select the number of the spreading code resource, where a n (t) = 0 means that it is recommended not to use resources for sending at the next communication time point, so that resources can be given to other terminal gateways to reduce conflicts caused by selecting duplicate resources; a n (t) = 1,…,M represents the number of available spreading codes {c1(t),…c M (t)}, select the corresponding number of spread spectrum code communication, where a n (t)∈{0,1,…,M};

[0069] Further consider that there are M = 2 sets of available spreading codes in the system, and each time the spreading code is updated c m (t) Select the corresponding anti-interference level from {1,2} with a probability of [0.5,0.5], and update the resource performance requirements q of N terminal gateways at the same time n (t), select the corresponding demand in {1,2} with probability [0.6,0.4].

[0070] S2, extracting and processing the current environmental state through the policy execution network and evaluating and processing the extraction result of the policy execution network in combination with the policy evaluation network to obtain a recommended spreading code scheme;

[0071] Specifically, each terminal gateway will calculate the current environment state {C(t), S n (t)} is input into the policy execution network for feature extraction, and a preliminary recommended spreading code scheme is obtained. At the same time, the environmental status of all terminal gateways at the current time {C(t), S1(t), S2(t), …S N (t)} and the corresponding action decision {a1(t),a2(t),…a N (t)}, input into the strategy evaluation network for feature extraction, evaluate the preliminary recommended spreading code scheme, and obtain the recommended spreading code scheme.

[0072] S21, extracting and processing the current environment state through the policy execution network to obtain a preliminary recommended spread spectrum code scheme;

[0073] S211, performing feature extraction processing on the current environment state based on the policy execution network input layer to obtain first feature information;

[0074] S212, performing data dimension extraction processing on the current environment state based on the policy execution network convergence layer to obtain first high-dimensional information;

[0075] S213. Based on the strategy execution network output layer, the first feature information and the first high-dimensional information are combined to output the value corresponding to the environmental state at the current time and perform a maximum value operation to obtain a preliminary recommended spread spectrum code scheme.

[0076] Specifically, refer to Figure 5 The policy execution network includes an input layer, a convergence layer and an output layer, all of which are fully connected layers. Based on the policy execution network input layer, feature extraction processing is performed on the available spread spectrum code information, the demand for resource performance, the communication conditions at the current communication time point, the current cached data volume and the old data volume discarded due to the cache being full at the previous time point. The high-dimensional information of the above data is further extracted based on the policy execution network convergence layer. The extraction process adopts a linear rectification function, and then the estimated value Q of the optional action corresponding to each data is output through the policy execution network output layer. m (t), and perform the maximum value operation to obtain the preliminary recommended spreading code scheme.

[0077] S22. Evaluate the preliminary recommended spreading code scheme by using the strategy evaluation network and the current environment status to obtain a recommended spreading code scheme.

[0078] S221, performing feature extraction processing on the current environmental state and the preliminary recommended decision-making scheme based on the strategy evaluation network input layer to obtain second feature information;

[0079] S222, based on the strategy evaluation network convergence layer, extract and process the data dimension of the current environmental state and the preliminary recommended decision plan to obtain the second highest dimensional information;

[0080] S223. Based on the policy execution network output layer, the recommended spreading code scheme is output in combination with the second feature information and the second high-dimensional information.

[0081] Specifically, refer to Figure 4 The strategy evaluation network includes an input layer, a convergence layer and an output layer, all of which are fully connected layers. The input layer of the strategy evaluation network performs feature extraction processing on the current environmental state and the preliminary recommended decision-making scheme. The convergence layer of the strategy evaluation network extracts high-dimensional information corresponding to the data. The extraction process uses a linear rectification function. The output layer of the policy execution network evaluates the value of the terminal gateway executing the preliminary recommended spread spectrum code scheme under the current environmental state at the current time to obtain the recommended spread spectrum code scheme.

[0082] S3, performing resource scheduling training of the terminal gateway according to the recommended spreading code scheme to obtain a scheduling result;

[0083] Specifically, the resource scheduling process is defined as an optimization problem, that is, finding a solution strategy that minimizes scheduling conflicts and data discards while maximizing communication efficiency within time T, where:

[0084] Reference Figure 6 As the number of terminal gateways increases, in addition to the agreed polling scheduling strategy in advance, the number of transmission conflicts of the random scheduling strategy and the cache data volume strategy will also increase. The resource scheduling method of the present invention enables the terminal gateway to effectively predict the use of the spread spectrum code of other gateways, greatly reducing the probability of conflicts caused by repeated use of spread spectrum codes;

[0085] Reference Figure 7 The resource scheduling method of the present invention can ensure the fairness of the use of spread spectrum codes by the terminal gateways as much as possible, and reduce the situation where a gateway does not use the spread spectrum code for communication for a long time due to fear of conflicts, and at the same time, too much data arrives, resulting in cache overflow and having to choose to discard part of the data. The scheduling algorithm of the present invention maintains the multi-terminal gateway system in long-term operation, and the amount of data discarded is more than 30% less than the best polling scheduling strategy in the commonly used scheduling algorithm;

[0086] The scheduling conflict is specifically expressed as follows: within the current communication time, if more than one terminal gateway selects a repeated spread spectrum code for communication, it is considered that mutual interference will occur at the receiving end, causing the terminals that select the spread spectrum code to fail to communicate, and the transmitted data will be lost. At the same time, the number of conflicts is recorded. num =conflict num +1, to ensure the minimum number of conflicts, i.e. Furthermore, the multi-terminal gateway system has a long delay characteristic and cannot support the error retransmission function. Therefore, the number of discarded packets is used to describe the amount of data loss caused by the system due to conflict or long waiting time, where 0(·) means that when the conflict definition is met, the value in the bracket is taken, otherwise it is taken as 0. To ensure the minimum amount of data discarded, the calculation formula is as follows:

[0087]

[0088] In the above formula, Δ n (t) represents the amount of old data discarded due to cache fullness at the previous time point, δ n (t) represents the current cached data volume, N represents the number of terminal gateways, and T represents the preset time.

[0089] Reference Figure 8The resource scheduling method of the present invention can also ensure that the gateway can use the limited spread spectrum code resources more effectively. In the same transmission time slot, a larger amount of data can be transmitted. By reducing the frequency of the terminal gateway with a small amount of data occupying the spread spectrum resources, the gateway with a large amount of data can use the resources more frequently for communication. In the long-term operation of the system maintained by the scheduling algorithm of the present invention, the amount of data successfully transmitted basically remains unchanged, depending on the condition of the number of available spread spectrum code resources. In the system maintained by the commonly used scheduling algorithm, the amount of data successfully transmitted will decrease as the number of users increases.

[0090] The maximum communication efficiency is specifically expressed as follows: if there is only one terminal gateway that selects a certain spreading code for communication, it is considered that the receiving end can correctly receive the amount of data δ buffered by the terminal gateway before sending n (t), in order to ensure the maximum amount of successfully transmitted data, where 1(·) means that when the above non-conflict definition is met, the value in the bracket is taken, otherwise it is taken as 0. The calculation formula is as follows:

[0091]

[0092] S31, uniformly distribute the recommended spreading code scheme to obtain a random number;

[0093] S32, judging the random number;

[0094] S33, judging that the random number is greater than the exploration coefficient, executing the greedy strategy and selecting the recommended spread spectrum code scheme;

[0095] S34, judging that the random number is less than or equal to the exploration coefficient, executing the exploration strategy and selecting a random spread spectrum code scheme;

[0096] S35. Perform resource scheduling training for the terminal gateway according to the selection result to obtain a scheduling result.

[0097] Specifically, according to the strategy evaluation network recommended spread spectrum code scheme, the terminal gateway will combine the "exploration-greedy" strategy to select action update, uniformly distribute the recommended spread spectrum code scheme to obtain a random number, if the random number at this time is greater than the exploration coefficient ∈, then execute the greedy action, that is, recommend the spread spectrum code scheme, otherwise, execute the random spread spectrum code scheme, the exploration coefficient decreases from 1→0.1 as the number of training times increases in the centralized training phase (1-0.9*)[current training times*(1 / maximum training times)], indicating that there is a greater probability of executing the exploration strategy in the early stage of training, and a greater probability of executing the greedy strategy in the later stage of training, ∈ will gradually decrease with the number of training times, and the terminal gateway will gradually tend to use the recommended spread spectrum code scheme. Assuming that the system has M available spread spectrum codes {c1(t),…c M (t)}, the optional action a n(t)∈{0,1,…,M}, 0 means no spread spectrum code is used, 1,…,M means using the recommended number spread spectrum code, the random spread spectrum code scheme is to use a random number function to select a number in {0,1,…,M} with the same probability as the action selection output, so after each action is executed, a part of the state transition history is randomly extracted from the experience pool for training and parameter update of the policy execution network and the policy evaluation network, and the scheduling result is output, which includes the immediate reward R of the environment feedback after the action is executed n (t+1) and the next state S of the transition n (t+1), where the scheduling result includes the immediate reward R of the environment feedback after executing the action n (t+1)=1(tr n (t))-[0(δ n (t)+Δ n (t))]*q n (t), and Each time an action is selected and executed and the scheduling result is obtained, the "current observation states of all terminal gateways {C(t), S1(t), S2(t), ... S N (t)}, the action generated according to this observation state {a1(t), a2(t),…a N (t)}, output instant rewards {R1(t+1), R2(t+1),…R N (t+1)} and the next state of the transition {C(t+1), S1(t+1), S2(t+1),…S N (t+1)}" is saved in the cache experience pool, which is called a scheduling experience. During training, several scheduling experiences (32 in this example) are extracted from the experience pool, and the error function L is calculated using the update formula n (ω) and J n (θ) guides the two networks to converge.

[0098] S4. The policy execution network and the policy evaluation network are updated according to the scheduling result, and the resource scheduling of the terminal gateway is performed based on the updated policy execution network.

[0099] S41, randomly selecting the stored scheduling results, updating and training the strategy execution network and the strategy evaluation network through the multi-agent deep reinforcement algorithm, and obtaining the training difference;

[0100] Specifically, for the policy evaluation network parameter update of terminal gateway n, the parameters are first defined as the current environment Current Action Next time point environment action Policy enforcement for all gateways in Network π ′According to the environment The recommended strategy output And the next state value of the strategy evaluation network output Where R n (t+1) is the immediate return of environmental feedback after adopting the strategy. The strategy execution network and strategy evaluation network are used to calculate the strategy execution value and estimate the error respectively. n Output estimated next state value, Q n Use as Y n The error estimate of

[0101] The strategy evaluation network parameter update method uses the mean square error to calculate the loss, and uses the back propagation method to update the parameters. The update formula of the strategy evaluation network is as follows:

[0102]

[0103]

[0104] In the above formula, L n (ω) represents the error function of the calculation, α represents the learning rate, ω represents the parameters of the policy evaluation network, B represents the batch size of each calculation, S represents the current environment, π(S) represents the current decision action, S,A,R,S′ represents the scheduling experience extracted from this training, R represents the timely reward of the environment feedback after the action is executed, S′ represents the state of the environment after the action is executed, γ represents the decay rate, Q t , Q both represent the strategy evaluation network, π t ,π both represent the policy execution network;

[0105] The decay rate is set to 0.95 in this example, and the network parameters of Q and π are updated every time training. t , π t The network parameters of Q are defined. After training 30 batches of historical experience, the network parameters of Q are copied to Q t , and similarly copy the network parameters of π to π t ;

[0106] The update formula of the policy execution network is as follows:

[0107]

[0108]

[0109] In the above formula, J n (θ) represents the calculated error function, α represents the learning rate, and θ represents the parameters of the policy execution network;

[0110] Further, at the beginning of each transmission time slot, the communication conditions of N terminal gateways in the time slot are updated simultaneously. n (t) Select from {0,1} with probability [0.5,0.5] to determine whether the corresponding gateway in this time slot can use the spread spectrum code for communication. At the same time, update the cache data volume of each gateway δ n (t), where the data arrival amount of the current time slot is generated according to the Poisson distribution probability. Different gateways have different means. According to the gateway number n, the latest data arrival amount is generated by selecting the mean from {0.4, 0.4, 0.015, 0.015, 0.01, 0.01, 0.01, 0.015, 0.015, 0.01, 0.01}. If the cache of gateway n overflows δ after the new data arrives, n (t)>1, then the corresponding discarded old data volume Δ is output n (t) = 1-δ n (t), and at the same time n (t) is set to 1, the learning rate α is 0.001; the exploration rate ∈ decays from 1 to 0.1, and the decay rate γ is set to 0.95; the strategy execution target network π ′ and strategy evaluation target network Q ′ The update frequency is 30; the number of training times is 300, and each training time is 1000 scheduling time slots; the experience pool size is 10000.

[0111] S42, judging based on the training difference, judging that the training cutoff condition is met, and outputting an updated strategy execution network and an updated strategy evaluation network;

[0112] S43. Perform resource scheduling of the terminal gateway based on the updated policy execution network.

[0113] Specifically, the judgment is made based on the training difference, which is the error function L completed based on the scheduling experience extracted this time. n (ω) and J n The calculation result of (θ) is defined as the error function difference between two adjacent training calculations is less than 10 -4 , if the training cutoff condition is met, the updated policy execution network and the updated policy evaluation network are output, otherwise, the process returns to step S2; the resource scheduling of the terminal gateway is performed based on the updated policy execution network, and the resource scheduling process no longer requires the intervention of the policy evaluation network. Each gateway only needs to update the state {C(t), S n (t)} is input into each strategy execution deep neural network, and there is no need for negotiation between decision networks. The action a is output in a tacit way agreed upon. n (t) To achieve the purpose of sharing resources.

[0114] Reference Figure 2 , a multi-terminal gateway resource scheduling system based on deep reinforcement learning, including:

[0115] The acquisition module is used to initialize the terminal gateway and obtain the current environment status;

[0116] An evaluation module is used to extract and process the current environmental state through the policy execution network and evaluate the extraction result of the policy execution network in combination with the policy evaluation network to obtain a recommended spread spectrum code scheme;

[0117] The scheduling module is used to schedule the resources of the terminal gateway according to the recommended spreading code scheme and obtain the scheduling result;

[0118] The updating module is used to update the policy execution network and the policy evaluation network according to the scheduling result, and perform resource scheduling of the terminal gateway based on the updated policy execution network.

[0119] like Figures 6 to 8 As shown, in order to compare the performance of the scheduling algorithm proposed in the present invention in a multi-sensor network, three commonly used scheduling algorithms are considered under the same environmental conditions, including a round-robin scheduling strategy (RoundRobin), a random scheduling strategy (Random) and a cache data volume-based strategy (Cache_Based).

[0120] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0121] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A multi-terminal gateway resource scheduling method based on deep reinforcement learning, characterized in that: The following steps are involved: Initialize the terminal gateway and obtain the current environment status; The current environment state is extracted and processed by the policy execution network and the extraction result of the policy execution network is evaluated and processed by the policy evaluation network to obtain the recommended spread spectrum code scheme; Perform resource scheduling training of the terminal gateway according to the recommended spreading code scheme and obtain the scheduling result; The policy execution network and the policy evaluation network are updated according to the scheduling results, and the resource scheduling of the terminal gateway is performed based on the updated policy execution network; The step of extracting and processing the current environment state through the policy execution network and evaluating and processing the extraction result of the policy execution network in combination with the policy evaluation network to obtain a recommended spreading code scheme specifically includes: The current environment state is extracted and processed through the policy execution network to obtain a preliminary recommended spreading code scheme; The preliminary recommended spreading code scheme is evaluated and processed through the strategy evaluation network and the current environment status to obtain the recommended spreading code scheme; The step of extracting and processing the current environment state through the policy execution network to obtain a preliminary recommended spread spectrum code scheme specifically includes: Performing feature extraction processing on the current environment state based on the policy execution network input layer to obtain first feature information; Based on the policy execution network convergence layer, data dimension extraction and processing are performed on the current environment state to obtain first high-dimensional information; Based on the output layer of the strategy execution network, the first feature information and the first high-dimensional information are combined to output the value corresponding to the environmental state at the current time and perform a maximum value operation to obtain a preliminary recommended spread spectrum code scheme; The step of evaluating the preliminary recommended spreading code scheme by evaluating the network and the current environment state through the strategy to obtain the recommended spreading code scheme specifically includes: Based on the input layer of the strategy evaluation network, feature extraction is performed on the current environmental state and the preliminary recommended decision plan to obtain the second feature information; Based on the policy evaluation network aggregation layer, the data dimension extraction and processing of the current environmental status and the preliminary recommended decision plan are performed to obtain the second highest dimensional information; Based on the output layer of the strategy execution network, combining the second feature information and the second high-dimensional information, outputting a recommended spreading code scheme; The step of updating the policy execution network and the policy evaluation network according to the scheduling result, and performing resource scheduling of the terminal gateway based on the updated policy execution network specifically includes: Randomly select the scheduling results and update the strategy execution network and strategy evaluation network through the multi-agent deep reinforcement algorithm to obtain the training difference; Make a judgment based on the training difference, judge until the training cutoff condition is met, and output the updated strategy execution network and the updated strategy evaluation network; Perform resource scheduling of terminal gateways based on the updated policy execution network; The update formula of the strategy evaluation network is as follows: In the above formula, L n (ω) represents the error function of the calculation, α represents the learning rate, ω represents the parameters of the policy evaluation network, B represents the batch size of each calculation, S represents the current environment, π(S) represents the current decision action, S,A,R,S' represents the scheduling experience extracted from this training, R represents the timely reward of the environment feedback after the action is executed, S' represents the state of the environment after the action is executed, γ represents the decay rate, Q t , Q both represent the strategy evaluation network, π t ,π both represent the policy execution network; The update formula of the policy execution network is as follows: In the above formula, J n (θ) represents the calculated error function, α represents the learning rate, and θ represents the parameters of the policy execution network.

2. According to claim 1, the multi-terminal gateway resource scheduling method based on deep reinforcement learning is characterized in that: The step of initializing the terminal gateway and obtaining the current environment state specifically includes: Initializing a multi-terminal gateway system and marking the terminal gateway, wherein the multi-terminal gateway system includes a plurality of terminal gateways; The current environment state of the terminal gateway is obtained, where the current environment state includes system available spread spectrum code information and its own service state.

3. According to claim 1, the multi-terminal gateway resource scheduling method based on deep reinforcement learning is characterized in that: The step of performing resource scheduling training of the terminal gateway according to the recommended spreading code scheme to obtain a scheduling result specifically includes: Uniformly distribute the recommended spreading code scheme to obtain random numbers; Judge the random number; If the random number is greater than the exploration coefficient, the greedy strategy is executed to select the recommended spread spectrum code scheme; If the random number is less than or equal to the exploration coefficient, the exploration strategy is executed and a random spread spectrum code scheme is selected; The resource scheduling training of the terminal gateway is performed according to the selected scheme to obtain the scheduling result.

4. A multi-terminal gateway resource scheduling system based on deep reinforcement learning, characterized in that: The method for executing multi-terminal gateway resource scheduling based on deep reinforcement learning as claimed in claim 1 comprises the following modules: The acquisition module is used to initialize the terminal gateway and obtain the current environment status; An evaluation module is used to extract and process the current environmental state through the policy execution network and evaluate the extraction result of the policy execution network in combination with the policy evaluation network to obtain a recommended spread spectrum code scheme; The scheduling module is used to perform resource scheduling training of the terminal gateway according to the recommended spreading code scheme and obtain the scheduling result; The updating module is used to update the policy execution network and the policy evaluation network according to the scheduling result, and perform resource scheduling of the terminal gateway based on the updated policy execution network.

Images