Downlink resource scheduling method and device for 5G differentiated scene, and storage medium

Abstract

The invention discloses a downlink resource scheduling method and device for a 5G differentiated scene and a storage medium. The method comprises the following steps: acquiring information, wherein the information comprises time slot network state information of each time slot; converting the information into historical state track information; the historical state track information serves as input vectors to be input into a DQN neural network and a TD3 neural network respectively, and core parameters are output; respectively training a DQN neural network and a TD3 neural network according to the historical state trajectory information and the core parameters to obtain a trained DQN neural network and a trained TD3 neural network; the time slot network state information of the current time slot serves as a current input vector to be input into the trained DQN neural network and the trained TD3 neural network, and a current core parameter is output; calculating the priority of each user for each resource block based on a priority formula of a heuristic algorithm according to the current core parameter; and performing resource scheduling on the resource blocks based on a greedy strategy according to the priority. According to the method, the defects of a traditional priority-based scheduling algorithm in a complex network environment are overcome.

Description

Technical Field

[0001] The present invention relates to the field of wireless network technology, and in particular to a downlink resource scheduling method, device and storage medium for a 5G differentiated scenario. Background Art

[0002] With the rapid development of the fifth generation mobile communication technology (5G), efficient scheduling of network resources has become one of the key technologies to meet the needs of diversified services and improve user experience. 5G networks have the characteristics of ultra-high speed, ultra-low latency and massive device connections, which puts higher requirements on network resource scheduling. In the 5G NR (New Radio) downlink, resource allocation faces many challenges, especially in how to dynamically adjust the resource allocation strategy according to the real-time network status to maximize the system throughput and meet the user's quality of service (QoS) requirements.

[0003] Traditional resource scheduling methods usually rely on fixed priority strategies for resource allocation. The Proportional Fair (PF) algorithm is a classic resource scheduling method that balances network throughput and user fairness. It dynamically allocates resources to maximize the overall network throughput while maintaining a certain degree of fairness. This algorithm is widely used in wireless networks. However, the scheduling decision of the PF algorithm mainly considers the user's throughput requirements, but fails to fully consider the latency indicator. This defect leads to insufficient performance in latency-sensitive 5G application scenarios, and it cannot effectively meet the low latency requirements, affecting the optimization of the overall network performance.

[0004] To solve this problem, some existing technologies have begun to design priority strategies that include delay indicators to improve the scheduling performance of delay-sensitive services. The priority formula designed is: , in is the throughput term, is the delay term, is the priority item. However, the design of the general formula does not perform well in the face of complex real-world network environments and changing traffic demands, and cannot strictly meet the user's QoS delay requirements. In addition, the priority formula with fixed parameters is also difficult to dynamically adapt to changes in network conditions, resulting in system performance degradation when the network environment and business load fluctuate. Summary of the invention

[0005] The present invention proposes a downlink resource scheduling method, device and storage medium for 5G differentiated scenarios to solve the problem of system performance degradation when the network environment and business load fluctuate.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] The present invention provides a downlink resource scheduling method for a 5G differentiated scenario, comprising:

[0008] Acquiring information, the information including time slot network status information of each time slot;

[0009] Converting the information into historical state trajectory information;

[0010] The historical state trajectory information is input as an input vector into the DQN and TD3 neural networks respectively, and the core parameters are output;

[0011] According to the historical state trajectory information and the core parameters, the DQN and TD3 neural networks are trained respectively to obtain a trained DQN neural network and a TD3 neural network;

[0012] Input the time slot network state information of the current time slot as the current input vector into the trained DQN and TD3 neural networks, and output the current core parameters;

[0013] Calculating the priority of each user for each resource block based on a priority formula of a heuristic algorithm according to the current core parameters;

[0014] According to the priority level, resource scheduling is performed on the resource blocks based on a greedy strategy.

[0015] Specifically, converting the information into historical state trajectory information includes:

[0016] A state vector is formed based on the data rate, buffer size, head-of-line delay and channel quality of each user in each time slot;

[0017] The time slot network state information of the corresponding time slot is formed according to the state vectors of all the users in each time slot;

[0018] The historical status track information is composed according to the time slot network status information of multiple time slots.

[0019] Specifically, before converting the information into historical state trajectory information, the information is preprocessed, and the preprocessing step includes:

[0020] The data rate is preprocessed, and the preprocessing formula is: in is the data rate after preprocessing, is the data rate, The maximum rate calculated when using the maximum modulation and coding scheme for all resource blocks in each time slot;

[0021] Preprocess the buffer size, the preprocessing formula is: , is the buffer size after preprocessing, is the buffer size, N represents the discretized value space and is a natural number set. The buffer size is discretized to indicate how many time slots are needed to transmit all the data in the buffer;

[0022] Preprocess the head-of-line delay, and the preprocessing formula is:

[0023] , is the head-of-line delay after preprocessing, Indicates the head-of-line delay, It represents the delay target. The head-of-line delay is divided by the delay target as a proportional term, which represents the user's timeout situation. If it is in the range [0,1], it means that the current head-of-line delay does not exceed the delay target.

[0024] The channel quality is preprocessed, and the preprocessing formula is:

[0025] , Indicates the channel quality flag after preprocessing. Indicates the channel quality flag.

[0026] in It is the maximum value of the user channel quality indicator.

[0027] Specifically, the historical state trajectory information is used as an input vector to be input into the DQN and TD3 neural networks respectively, and the core parameters are output, including:

[0028] Using a gated recurrent unit to capture the time-dependent features in the input vector to obtain a feature vector;

[0029] Input the feature vector into a DQN neural network and output a discrete random variable, wherein the discrete random variable includes an initial weight of a control delay term factor and a growth trend of the control delay term factor;

[0030] The feature vector is input into the TD3 neural network, and a continuous random variable is output, wherein the continuous random variable includes a weight ratio for controlling a delay term factor and a throughput term factor in a heuristic algorithm.

[0031] Specifically, the DQN and TD3 neural networks are trained respectively according to the historical time slot network state information and the historical core parameters, including:

[0032] Construct a Markov decision process quadruple;

[0033] Setting the input vector of the current time slot as the state of the Markov decision process quadruple;

[0034] The core parameters of the outputs of the two neural networks DQN and TD3 in the current time slot are set as the actions of the Markov decision process quadruple;

[0035] Combining the throughput index and the delay index of the current time slot as the reward function of the Markov decision process quadruple;

[0036] According to the time slot network state information of the next time slot, the reward of the current time slot is calculated based on the reward function and the input vector of the next time slot is constructed;

[0037] storing the Markov decision process quadruple of each time slot into a buffer area respectively;

[0038] For each of the historical time slots, quadruple groups are sampled in batches from the experience pool, and the DQN neural network and the TD3 neural network are trained respectively by the quadruple groups obtained by the batch sampling.

[0039] Specifically, for each of the historical time slots, quadruple groups are sampled from the experience pool in batches, and the DQN neural network and the TD3 neural network are trained respectively by the quadruple groups sampled in batches, including:

[0040] Input the state in the quadruple into the DQN neural network and output the DQN state value;

[0041] Input the state in the quadruple into the TD3 neural network and output the TD3 state value;

[0042] Calculating a DQN differential timing and a TD3 differential timing according to the DQN state value and the TD3 state value;

[0043] According to the DQN differential time series and the TD3 differential time series, the policy network of the DQN neural network and the critic network of the TD3 neural network are updated based on the gradient descent method;

[0044] According to the negative value of the TD3 state value output by the Critic network as the loss value of the Actor network, the Actor network is updated using the gradient descent method.

[0045] Specifically, the reward function is:

[0046]

[0047] in is the flow priority, and n is the number of users.

[0048] Specifically, the calculation formula of the priority w of the heuristic algorithm is:

[0049]

[0050] in is the QCI priority factor, is the throughput factor, is the delay factor, m and k respectively control the initial weight of the delay factor and the growth trend of the delay factor when the user exceeds the delay budget. The ratio of the delay factor and the throughput factor is controlled, and d represents the queuing delay of the first data unit in the radio link control layer buffer in the 5G NR protocol stack.

[0051] The present invention also provides a downlink resource scheduling device for a 5G differentiated scenario, comprising:

[0052] An acquisition module, the acquisition module is used to acquire information, the information including time slot network status information of each time slot;

[0053] A conversion module, the conversion module is used to convert the information into historical state trajectory information;

[0054] A first input module, the first input module is used to input the historical state trajectory information as an input vector into the DQN and TD3 neural networks respectively, and output core parameters;

[0055] A training module, wherein the training module is used to train the DQN and TD3 neural networks respectively according to the historical state trajectory information and the core parameters to obtain a trained DQN neural network and a trained TD3 neural network;

[0056] A second input module, the second input module is used to input the time slot network state information of the current time slot as the current input vector into the trained DQN and TD3 neural networks, and output the current core parameters;

[0057] A calculation module, the calculation module is used to calculate the priority of each user for each resource block based on the priority formula of the heuristic algorithm according to the current core parameters;

[0058] A scheduling module is used to schedule resources for resource blocks based on a greedy strategy according to the priority level.

[0059] The present invention also provides a storage medium, which is a readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the downlink resource scheduling method for a 5G differentiated scenario are implemented.

[0060] The beneficial effects of the present invention are:

[0061] The present invention provides a downlink resource scheduling method, device and storage medium for 5G differentiated scenarios, aiming to optimize resource allocation through intelligent scheduling strategies to meet the quality of service (QoS) requirements of different users. The present invention considers the design of a new segmented priority formula, considers the delay index and combines deep reinforcement learning technology, and overcomes the shortcomings of traditional priority-based scheduling algorithms in complex network environments by dynamically adjusting the core parameters of the priority formula of the heuristic method. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of a single base station downlink transmission scenario in a 5G differentiated scenario;

[0063] Figure 2 Traffic curve diagram for 5G differentiated scenarios;

[0064] Figure 3 This is a flow chart of a 5G downlink resource scheduling system in an embodiment of the present invention;

[0065] Figure 4 A neural network architecture diagram in an embodiment of the present invention;

[0066] Figure 5 is an update flow chart of the TD3 algorithm in an embodiment of the present invention;

[0067] Figure 6 is an update flow chart of the DQN algorithm in an embodiment of the present invention;

[0068] Figure 7 This is a performance comparison chart of the 5G differentiated scenario algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0070] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0072] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use, or are directions or positional relationships commonly understood by those skilled in the art. These directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0073] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0074] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0075] In this paper, the term "DQN" means deep Q network, "TD3" means double-delayed deep deterministic policy gradient, "RLC layer" means radio link control layer, "GRU" means gated recurrent unit; "QCI" means quality of service level identifier.

[0076] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0077] Figure 1 The present invention shows a 5G downlink single base station transmission network scenario in which multiple users are randomly distributed around the base station, and each user has a different type of traffic.

[0078] Figure 2 The traffic load variation curves of four types of flows are shown, each type of traffic has different quality of service (QoS) requirements.

[0079] Based on the above scenario system, the present invention proposes a downlink resource scheduling method for 5G differentiated scenarios, such as Figure 3The present invention mainly includes three stages, namely, the state collection stage, the base station is responsible for collecting network information, performing preprocessing operations, and finally combining historical trajectory information as the input vector of the neural network; the parameter settlement stage, the intelligent agent is responsible for receiving the historical state trajectory information output by the state collection module, and after the GRU gating unit processes the timing information, it is input into the DQN network and the TD3 network respectively to combine and output the core parameters of the heuristic algorithm; the resource allocation stage, after the base station receives the core parameters output by the current time slot parameter calculation stage, the priority calculation formula of the combined heuristic algorithm is used to calculate the priority of each user on each resource block, and the resource blocks are greedily allocated.

[0080] Specifically, the state collection phase includes the following steps:

[0081] Step 1.1: Collect the RLC layer buffer information, QCI information and user information of each user in each time slot, including the buffer size. , Head of Line Delay , flow priority , Delay Target , Packet Error Rate Target , the user's CQI size and data rate ;

[0082] Step 1.2: Preprocess the collected data. The state vector of each user is , the network state of each time slot is , and combine the state vectors of multiple time slots into a historical trajectory information ,... ] as the input vector of the neural network;

[0083] The specific operations of step 1.2 for preprocessing the collected data are as follows:

[0084] (1) Data rate: in The maximum rate calculated using the maximum modulation and coding scheme (MCS) for all resource blocks in each time slot;

[0085] (2) Buffer size: N, discretizes the buffer size, indicating how many time slots are needed to transmit all the data in the buffer;

[0086] (3) Head-of-line delay: , the head-of-line delay is divided by the delay target as a proportional term, which represents the user's timeout situation. If it is in [0,1], it means that the current head-of-line delay does not exceed the delay target;

[0087] (4) Channel quality: ,in is the maximum value of the user channel quality indicator;

[0088] The parameter calculation phase includes the following steps:

[0089] Step 2.1: Use Gated Recurrent Unit (GRU) to process timing information ,... ], to capture the time-dependent features in the input vector;

[0090] Step 2.2, input the output vector processed by the GRU unit into the neural network of the deep Q network (DQN) and the double-delayed deep deterministic policy gradient (TD3) algorithm to output the core parameters of the heuristic algorithm;

[0091] Step 2.3, training the neural network: take the historical state trajectory information constructed by the state collection module as the input state, the core parameters output by the DQN algorithm and the TD3 algorithm as the output, combine the throughput index and the delay index as the reward function, store the Markov decision process quadruple of each time slot in the buffer, and train the neural network;

[0092] like Figure 4 The neural network architecture diagram designed by the present invention is shown in the figure. It contains a layer of GRU gated unit, two layers of fully connected layers and a layer of batch normalization layer. The state collection module will construct the historical trajectory information ,... ] is input into DQN and TD3, and after a layer of GRU gating unit processes the timing information, it passes through two layers of fully connected networks to output the core parameters of the heuristic algorithm.

[0093] The priority formula in the heuristic algorithm in step 2.2 is:

[0094]

[0095] in is the QCI priority factor, is the throughput factor, is the delay factor, m and k respectively control the initial weight of the delay factor and the growth trend of the delay factor when the user exceeds the delay budget. The ratio of the delay factor and the throughput factor is controlled.

[0096] The specific steps of outputting the core parameters of the heuristic algorithm in step 2.2 are:

[0097] Step 2.2.1, the DQN algorithm outputs discrete random variables m and k, where m controls the initial weight of the delay factor and k controls the growth trend of the delay factor;

[0098] Step 2.2.2, the TD3 algorithm outputs a continuous random variable β, which is used to control the weight ratio of the delay factor and the throughput factor in the heuristic algorithm;

[0099] The specific steps of training the neural network in step 2.3 are:

[0100] Step 2.3.1. Calculate the reward of the current time slot and construct the input vector of the next time slot based on the network status information and reward function collected in the next time slot. Stored in the buffer, the reward function is:

[0101]

[0102] Step 2.3.2: In each time slot, the newly stored batch of quads is sampled from the buffer according to the new sampling rules, and the state vector Input to the Policy network of the DQN algorithm and the Critic network of the TD3 algorithm to obtain the state value and , calculate the differential timing difference and , and use the gradient descent method to update the policy network of the DQN algorithm and the Critic network of the TD3 network

[0103] Step 2.3.3: Based on the output of the TD3 network Critic network The negative value of is used as the loss value of the Actor network, and the gradient descent method is used to update the Actor network.

[0104] Figure 5 and Figure 6 The process of training neural networks using TD3 and DQN algorithms is given as follows:

[0105] DQN algorithm: Get quadruplets from the experience pool Data, respectively Input to the policy network, Input to the target network, output corresponding state-action value function , then according to the reward Calculate the time difference TD-Error:

[0106]

[0107] Then the gradient descent algorithm is used to update the policy network, and the target network periodically copies the parameters of the policy network. They represent the state and action at time t respectively, Respectively represent the state and action at time t+1, represents the reward at time t.

[0108] TD3 algorithm: Get quadruplets from the experience pool Data will Input to Actor Network Output Action After adding random noise, it is input into the Critic network to output the action state value ,Will Input to the target Actor network, output action Then random noise is added and input into the target Critic network, which outputs a smaller value, and then according to The temporal difference TD-Error corresponding to the two Critic networks is calculated, and the gradient descent strategy is used to update the neural network. Then, the negative value of the state action value output by the Critic network is used as the loss value of the Actor network. The Actor network is updated using the gradient descent method, and then the parameters of the target network are periodically updated.

[0109] The specific steps of the resource allocation phase are:

[0110] Step 3.1, in each time slot, use the priority formula of the heuristic algorithm combined with the core parameters output in the parameter calculation phase to calculate the priority of each user on each resource block;

[0111] Step 3.2: According to the calculated priority, greedily allocate resource blocks to ensure that the quality of service (QoS) requirements of high-priority users are met first and the overall performance and efficiency of the network are maximized.

[0112] In order to verify the effect of the method of the present invention, simulation experiments were carried out on the NS3 simulation platform, and compared with the traditional heuristic method PF, the traditional heuristic method native to NS3, and the new priority formula designed by the present invention with fixed parameters. The traditional heuristic method greedily allocates resource blocks through different priority formulas, while the present invention considers adaptively adjusting the core parameters of the priority formula through deep reinforcement learning to adapt to different network environments and traffic loads.

[0113] The parameters of the NS3 simulation platform are shown in Table 1 below: Table 1 Parameter Type Parameter Value Channel Model InH_OfficeOpen Numerology 0 centralFrequency 4e9 bandwidth 2e7 totalTxPower 43dbm

[0114] The reinforcement learning parameters are shown in Table 2 below: Table 2

[0115] The service quality requirements for different user traffic are shown in Table 3: Table 3 Traffic Type Delay Budget GBR_CONV_VOICE 100ms NGBR_LOW_LAT_EMBB 10ms GBR_CONV_VIDEO 150ms NGBR_VIDEO_TCP_DEFAULT 300ms

[0116] like Figure 7 As shown, in the scenario of traffic burst in downlink transmission of a single 5G base station, the present invention can ensure the quality of service (QoS) of different users, while the baseline algorithm for comparison cannot meet the constraints of the delay indicator because it does not take into account the traffic load and service quality requirements of different users.

[0117] In summary, the present invention proposes a downlink resource scheduling method for 5G differentiated scenarios. In the data collection stage, network information is collected and preprocessed to form historical trajectory information as the input vector of the neural network. In the parameter calculation stage, the historical trajectory information is accepted, and after a layer of GRU gating unit processes the timing information, the core parameters of the corresponding heuristic algorithm are output through two layers of fully connected layers, and the corresponding neural network is trained using the DQN algorithm and the TD3 algorithm. Finally, in the resource allocation stage, the core parameters of the priority formula output by the receiving parameter calculation stage are used to calculate the priority of each user on each resource block, and the resource block is allocated using a greedy strategy, and finally the resource allocation result is output. The present invention successfully solves the limitations of traditional scheduling algorithms in complex network environments through an intelligent resource allocation mechanism. In this method, the optimal utilization of network resources is achieved through accurate data acquisition and preprocessing in the state collection stage, the combination of deep learning and reinforcement learning in the parameter calculation stage, and the efficient allocation strategy in the resource allocation stage. Experimental results show that on the NS3 simulation platform, the method of the present invention can meet the quality of service (QoS) of different users for different user traffic requirements and burst scenarios. Compared with traditional heuristic methods, the present invention exhibits better delay control capability and resource utilization efficiency.

[0118] The present invention also provides a downlink resource scheduling device for a 5G differentiated scenario, comprising:

[0119] An acquisition module, the acquisition module is used to acquire information, the information including time slot network status information of each time slot;

[0120] A conversion module, the conversion module is used to convert the information into historical state trajectory information;

[0121] A first input module, the first input module is used to input the historical state trajectory information as an input vector into the DQN and TD3 neural networks respectively, and output core parameters;

[0122] A training module, wherein the training module is used to train the DQN and TD3 neural networks respectively according to the historical state trajectory information and the core parameters to obtain a trained DQN neural network and a trained TD3 neural network;

[0123] A second input module, the second input module is used to input the time slot network state information of the current time slot as the current input vector into the trained DQN and TD3 neural networks, and output the current core parameters;

[0124] A calculation module, the calculation module is used to calculate the priority of each user for each resource block based on the priority formula of the heuristic algorithm according to the current core parameters;

[0125] A scheduling module is used to schedule resources for resource blocks based on a greedy strategy according to the priority level.

[0126] The present invention also provides a storage medium, which is a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the downlink resource scheduling method for a 5G differentiated scenario are implemented. The readable medium may be included in the electronic device described in the above embodiment; or it may exist alone without being assembled into the electronic device.

[0127] The computer-readable medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments.

[0128] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0129] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0130] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0131] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A downlink resource scheduling method for a 5G differentiated scenario, characterized in that: include: Acquiring information, the information including time slot network status information of each time slot; Converting the information into historical state trajectory information; The historical state trajectory information is input as an input vector into the DQN and TD3 neural networks respectively, and the core parameters are output; According to the historical state trajectory information and the core parameters, the DQN and TD3 neural networks are trained respectively to obtain a trained DQN neural network and a TD3 neural network; Input the time slot network state information of the current time slot as the current input vector into the trained DQN and TD3 neural networks, and output the current core parameters; Calculating the priority of each user for each resource block based on a priority formula of a heuristic algorithm according to the current core parameters; According to the priority level, resource scheduling is performed on the resource blocks based on a greedy strategy.

2. According to claim 1, a downlink resource scheduling method for a 5G differentiated scenario is characterized in that: Convert the information into historical state trajectory information, including: A state vector is formed based on the data rate, buffer size, head-of-line delay and channel quality of each user in each time slot; The time slot network state information of the corresponding time slot is formed according to the state vectors of all the users in each time slot; The historical status track information is composed according to the time slot network status information of multiple time slots.

3. According to claim 2, a downlink resource scheduling method for a 5G differentiated scenario is characterized in that: Before converting the information into historical state trajectory information, the information is preprocessed, and the preprocessing step includes: The data rate is preprocessed, and the preprocessing formula is: in is the data rate after preprocessing, is the data rate, The maximum rate calculated when using the maximum modulation and coding scheme for all resource blocks in each time slot; Preprocess the buffer size, the preprocessing formula is: N, is the buffer size after preprocessing, is the buffer size, N represents the discretized value space and is a natural number set. The buffer size is discretized to indicate how many time slots are needed to transmit all the data in the buffer; Preprocess the head-of-line delay, and the preprocessing formula is: , is the head-of-line delay after preprocessing, Indicates the head-of-line delay, It represents the delay target. The head-of-line delay is divided by the delay target as a proportional term, which represents the user's timeout situation. If it is in the range [0,1], it means that the current head-of-line delay does not exceed the delay target. The channel quality is preprocessed, and the preprocessing formula is: , Indicates the channel quality flag after preprocessing. Indicates the channel quality flag. in It is the maximum value of the user channel quality indicator.

4. According to claim 2, a downlink resource scheduling method for a 5G differentiated scenario is characterized in that: The historical state trajectory information is used as an input vector to input into the DQN and TD3 neural networks respectively, and the core parameters are output, including: Using a gated recurrent unit to capture the time-dependent features in the input vector to obtain a feature vector; Input the feature vector into a DQN neural network and output a discrete random variable, wherein the discrete random variable includes an initial weight of a control delay term factor and a growth trend of the control delay term factor; The feature vector is input into the TD3 neural network, and a continuous random variable is output, wherein the continuous random variable includes a weight ratio for controlling a delay term factor and a throughput term factor in a heuristic algorithm.

5. According to claim 2, a downlink resource scheduling method for a 5G differentiated scenario is characterized in that: The DQN and TD3 neural networks are trained respectively according to the historical time slot network state information and the historical core parameters, including: Construct a Markov decision process quadruple; Setting the input vector of the current time slot as the state of the Markov decision process quadruple; The core parameters of the outputs of the two neural networks DQN and TD3 in the current time slot are set as the actions of the Markov decision process quadruple; Combining the throughput index and the delay index of the current time slot as the reward function of the Markov decision process quadruple; According to the time slot network state information of the next time slot, the reward of the current time slot is calculated based on the reward function and the input vector of the next time slot is constructed; Storing the Markov decision process quadruple of each time slot in a buffer area respectively; For each of the historical time slots, quadruple groups are sampled in batches from the experience pool, and the DQN neural network and the TD3 neural network are trained respectively by the quadruple groups obtained by the batch sampling.

6. The downlink resource scheduling method for a 5G differentiated scenario according to claim 5, characterized in that: Sampling quadruple groups from the experience pool in batches for each of the historical time slots, and respectively training the DQN neural network and the TD3 neural network with the quadruple groups obtained by the batch sampling, including: Input the state in the quadruple into the DQN neural network and output the DQN state value; Input the state in the quadruple into the TD3 neural network and output the TD3 state value; Calculating a DQN differential timing and a TD3 differential timing according to the DQN state value and the TD3 state value; According to the DQN differential time series and the TD3 differential time series, the policy network of the DQN neural network and the critic network of the TD3 neural network are updated based on the gradient descent method; According to the negative value of the TD3 state value output by the Critic network as the loss value of the Actor network, the Actor network is updated using the gradient descent method.

7. The downlink resource scheduling method for a 5G differentiated scenario according to claim 5, characterized in that: The reward function is: ， in is the flow priority, and n is the number of users.

8. The downlink resource scheduling method for a 5G differentiated scenario according to claim 1, characterized in that: The calculation formula of the priority w of the heuristic algorithm is: ， in is the QCI priority factor, is the throughput factor, is the delay factor, m and k respectively control the initial weight of the delay factor and the growth trend of the delay factor when the user exceeds the delay budget. Then control the ratio of delay factor and throughput factor, Represents the queuing delay of the first data unit in the radio link control layer buffer in the 5G NR protocol stack.

9. A downlink resource scheduling device for a 5G differentiated scenario, characterized in that: include: An acquisition module, the acquisition module is used to acquire information, the information including time slot network status information of each time slot; A conversion module, the conversion module is used to convert the information into historical state trajectory information; A first input module, the first input module is used to input the historical state trajectory information as an input vector into the DQN and TD3 neural networks respectively, and output core parameters; A training module, wherein the training module is used to train the DQN and TD3 neural networks respectively according to the historical state trajectory information and the core parameters to obtain a trained DQN neural network and a trained TD3 neural network; A second input module, the second input module is used to input the time slot network state information of the current time slot as the current input vector into the trained DQN and TD3 neural networks, and output the current core parameters; A calculation module, the calculation module is used to calculate the priority of each user for each resource block based on the priority formula of the heuristic algorithm according to the current core parameters; A scheduling module is used to schedule resources for resource blocks based on a greedy strategy according to the priority level.

10. A storage medium, characterized in that: The storage medium is a readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of a downlink resource scheduling method for a 5G differentiated scenario as described in any one of claims 1 to 8 are implemented.

Images