Network slice selection method based on deep reinforcement learning of gated loop unit

Abstract

The invention provides a network slice selection method and system based on deep reinforcement learning of a gated cycle unit, and relates to the technical field of wireless communication. The method comprises the following steps: constructing a simulation vehicle-mounted network scene and modeling the simulation vehicle-mounted network scene into a Markov decision process; a dynamic attenuation action selection strategy is adopted for action selection, a time sequence state, actions, rewards and an updated state form a tetrad to be stored in a time sequence experience pool, and the experience pool is managed through a first-in first-out and stratified sampling mechanism; and training a neural network containing a GRU layer based on a sample in the time sequence experience pool, taking a mean square error of a target Q value and a predicted Q value as a loss function optimization parameter, and outputting an optimal slice selection result by utilizing the trained network. According to the invention, through dynamic attenuation strategy balance exploration and utilization, the unbalance problem of a traditional fixed probability mechanism is solved; state time sequence association is reserved by means of a time sequence experience pool, and time sequence breakage caused by isolated samples is avoided; state long-term dependence is effectively captured by using the GRU network, the Q value estimation accuracy is improved, and an efficient solution is provided for slice selection in a vehicle-mounted network environment.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication. It utilizes network slicing technology to slice communication networks, thereby providing targeted services to users. The combination of gated recurrent units and deep Q-learning improves network operating efficiency. Background Technology

[0002] With the rapid development of 5G and subsequent next-generation communication technologies, vehicular networks are experiencing explosive growth and have gradually become a crucial application scenario in next-generation communication systems. The intelligence and connectivity of vehicles have spurred a massive number of vehicle-to-everything (V2X) applications, from basic real-time navigation and collision warning to advanced autonomous driving collaboration and vehicle-to-infrastructure (V2I) scheduling. These applications, leveraging the full coverage of wireless networks, not only significantly improve traffic efficiency but also fundamentally enhance road safety. However, different V2X services often have heterogeneous Quality of Service (QoS) requirements. To meet these diverse QoS needs, network slicing technology has been widely adopted in vehicular networks. Network slicing technology combines Software-Defined Networking (SDN) with Network Functions Virtualization (NFV) to achieve efficient network slice management. Using network slicing technology, physical infrastructure can be divided into multiple virtual networks, providing customized services for different user needs and ensuring that heterogeneous service requirements in vehicular networks are met.

[0003] While network slicing technology provides an effective means of resource management for vehicular networks, the highly random and unpredictable nature of mobile devices and user behavior within vehicular networks presents significant challenges in developing efficient network slicing selection schemes. Specifically, different types of devices vary significantly in terms of computing power, communication requirements, and protocol support, while user mobility leads to constantly changing network slicing needs. Furthermore, the structure of vehicular networks is influenced by factors such as traffic density and time of day, making them prone to traffic surges. Traditional static network slicing resource allocation schemes struggle to cope with these dynamic changes.

[0004] Currently, researchers are turning to deep reinforcement learning methods to address the problems of network slice selection and resource allocation. These methods aim to enable agents to autonomously learn optimal network slice selection strategies through continuous interaction with the vehicular network environment. However, existing deep reinforcement learning-based algorithms still have significant shortcomings: First, most algorithms use fully connected neural networks as value function approximators. When processing continuous time-series data in vehicular networks (such as continuous vehicle position changes and network load fluctuation curves), these networks cannot effectively capture long-term temporal dependencies between states. This leads to biased estimations of Q-values ​​by the agent, resulting in low learning efficiency and difficulty in quickly adapting to the dynamic environment of the vehicular network, thus affecting the real-time performance and accuracy of network slice selection. Second, there are significant flaws in action selection strategies: greedy strategies that rely solely on maximizing Q-values ​​are prone to getting trapped in local optima, while exploration strategies with fixed probabilities struggle to achieve a dynamic balance between exploring new slices and utilizing known optimal slices. In the early stages of training, insufficient exploration may lead to missed potential optimal strategies, while in the later stages, over-exploration may cause policy fluctuations, making it impossible to consistently output high-quality network slice selection results.

[0005] Disadvantages of existing technology

[0006] 1. Existing deep reinforcement learning-based network slice selection algorithms employ a fixed-probability exploration mechanism for action selection, failing to dynamically adjust exploration probabilities based on training progress. This can lead to missed potential optimal strategies in the early stages of training due to insufficient exploration, resulting in local optima; conversely, overexploration in the later stages can cause policy fluctuations, hindering stable high-quality output and slow policy convergence, thus impacting the long-term stability of resource allocation. 2. Existing algorithms use fully connected neural networks to process vehicular network time-series data. These networks cannot effectively capture long-term temporal dependencies between states, leading to biased Q-value estimations by the agent, low learning efficiency, and difficulty adapting to the dynamic nature of vehicular networks due to high vehicle mobility and large traffic density variations. This affects the real-time performance and accuracy of network slice selection, making it difficult to meet the QoS requirements of heterogeneous services. 3. Existing algorithms use isolated single-step samples in their experience pools, disrupting temporal correlations. The causal relationships of resource allocation decisions in consecutive time steps are severed, causing training data to fail to reflect the dynamic changes in resource allocation in real-world scenarios. This fails to provide effective learning basis for the model, affecting training stability and making the model prone to convergence oscillations during training.

[0007] The main objective of this invention is to address the problems of existing deep reinforcement learning-based network slice selection algorithms when dealing with vehicular network issues, such as insufficient capture of temporal features, difficulty in adapting to network dynamics, imbalance between exploration and utilization, and temporal discontinuities in experience replay. To this end, this invention proposes a network slice selection method based on gated recurrent unit (GRU) deep reinforcement learning. By constructing a GRU-based neural network structure, designing a dynamically decaying action selection strategy, and proposing a temporal experience pool mechanism, this method improves the accuracy of Q-value estimation, balances exploration and utilization, and ensures training stability. Ultimately, it enhances the accuracy and resource utilization of network slice selection, meets the heterogeneous service requirements of vehicular networks while maintaining convergence speed, and improves overall network performance. Summary of the Invention

[0008] To address the aforementioned problems, the present invention aims to provide a network slice selection method based on deep reinforcement learning with gated recurrent units. This method solves the problems of imbalance between action selection strategy exploration and utilization, inability of fully connected neural networks to effectively capture long-term temporal correlations, inefficiency in training due to the destruction of temporal features by experience pools, and low performance and efficiency of slice selection in dynamic environments of vehicular networks. It improves the accuracy of Q-value estimation, balances exploration and utilization, ensures training stability, and enhances the accuracy and resource utilization efficiency of network slice selection.

[0009] The technical solution of this invention is:

[0010] A network slice selection method based on gated recurrent unit deep reinforcement learning, characterized by the following steps:

[0011] S01: Construct a simulated vehicular network scenario. Construct a simulated vehicular network scenario for a specific area, including base stations, user equipment, and various network slice types. Define the attribute parameters and interaction rules of each core element in the scenario, and clarify the specific boundaries and optimization indicators of the network slice selection problem.

[0012] S02: Model the network slice selection problem as a Markov decision process. Define the temporal state space, action space, reward function, and state transition probabilities.

[0013] S03: Modify the action selection strategy and experience pool mechanism in the deep Q-network algorithm. A dynamically decaying action selection strategy is adopted, adjusting the exploration probability as training progresses. New actions are attempted with higher probability in the early stages of training, while optimal actions are selected later. After the agent executes an action, its state is updated and the reward is calculated. A four-tuple consisting of the temporal state, action, reward, and updated state is stored as temporal experience in the temporal experience pool. The experience pool uses a first-in, first-out (FIFO) mechanism to maintain dynamic balance, removing the oldest sample when adding new experience to ensure data timeliness. A hierarchical sampling mechanism is also employed to provide effective data for training.

[0014] S04: Train and optimize the GRU neural network, outputting the optimal slice selection result. Construct a neural network architecture containing an input layer, a GRU layer, and a fully connected layer. The GRU layer captures long-term temporal dependencies of states through a gating mechanism. When the sample size in the experience pool reaches a threshold, a hierarchical sampling mechanism is used to extract samples, input them into the network to calculate the predicted Q-value, and use the mean squared error between the target Q-value and the predicted Q-value as the loss function, iteratively updating the parameters through gradient descent. After each training round, use the current neural network to calculate the Q-value of all actions for the updated state, select the action with the largest Q-value as the optimal slice selection result, and record it. Repeat the experience generation, network training, and slice selection process until the termination condition is met.

[0015] Preferably, the step of building the vehicle network simulation environment in step S01 includes:

[0016] S11: Construct a network slicing simulation model based on a specific area of ​​the vehicular network. This area is defined as having one base station (BS), whose core function is to allocate heterogeneous network resources to diverse user groups within the area. Since the resource needs of different user groups vary significantly, the model introduces network slicing technology to achieve differentiated services. Simultaneously, the base station is required to monitor network status and user request dynamics in real time, adjusting resource allocation strategies accordingly to ensure efficient utilization of network resources and optimal service quality.

[0017] S12: Set the maximum network bandwidth to B max This general-purpose frequency band can carry service requests for |N| types of network slice resources, and these requests follow the same SLA protocol. A network slice can be represented as N = {n1, n2, n3, ..., n1}. Each network slice contains two key attribute parameters: Quality of Service (QoS) ρ and resource utilization μ. QoS refers to the performance level provided by the network service within a certain time; resource utilization refers to the proportion of resources used by the network slice service to obtain QoS. Within this area, users send service requests for different network slices to the system according to their needs. These requests are queued into corresponding waiting queues according to the first-in, first-out principle. The waiting queues can be represented as K = {k1, k2, k3, ..., kk1}. |N| Whenever a request receives a response, the corresponding service request is removed from the waiting queue and added to the queue of responded service requests. The number of responded service requests is denoted as O, which can be represented as O = {o1, o2, o3, ..., o}. |N|}

[0018] S13: The core issue in network slice selection is maximizing network utility U while satisfying system resource constraints. tSince resource utilization is related to the number of requests responded to, it is necessary to keep resource utilization at a low level while ensuring high QoS, so as to achieve optimal overall service quality under limited resource conditions.

[0019] Preferably, the construction step of Markov decision modeling and experience generation storage in step S02 includes:

[0020] S21: Define the core components of the Markov decision process. The temporal state space S contains the slice request queue K of the current time slot, the number of responded requests O, and the historical states of the previous T time slots. The action space A represents the network slice selection scheme, and the total resource allocation does not exceed the maximum bandwidth of the base station. The reward function R is calculated as follows:

[0021]

[0022] The state transition probability P is fixed in this invention.

[0023] Preferably, the construction steps of the dynamic decay action selection strategy and time-series experience pool management in step S03 include:

[0024] S31: Initialize the temporal experience pool containing temporal features. Set the maximum capacity of the experience pool to M, and the storage unit to temporal experience. Each segment contains associated data for T consecutive time slots. The specific information includes the state sequence of the T consecutive time slots, the corresponding action sequence, the immediate reward sequence, and the next state sequence of the last time slot of the segment.

[0025] S32: In the early stages of model training, the temporal experience pool is filled. The agent interacts with the vehicle network environment and continuously collects state S, action A, and reward R step by step. In each time slot, the continuous data of the previous T time window is packaged into a temporal experience segment and stored in the temporal experience pool. This process is repeated until the storage capacity of the experience pool reaches the minimum sampling capacity.

[0026] S33: Set the core parameters of the dynamically decaying action selection strategy. Determine the initial exploration probability ε0 and the minimum exploration probability ε0. min And the decay coefficient λ, when calculating the decay action factor ε, the exponent λ of the decay coefficient with respect to the number of training steps is used. t The action factor is the maximum of the initial exploration probability and the minimum exploration probability. When selecting an action, the agent makes a random or greedy selection based on the probability of the current decaying action factor ε, then updates the state and action reward and stores them in the experience pool as a quadruple.

[0027] S34: Generate and store time-series quadruples. After selecting action a based on the states of T time windows, the agent updates the state S→S′, and incorporates the states of the T time windows, the current action, the immediate reward, and the next state into the quadruple (S,a,S′,R), storing it as an experience in the time-series experience pool.

[0028] S35: Once the experience pool meets the training requirements, sampling training is performed. Each time, a batch of data is sampled and replayed from the time-series experience pool for model training. The experience pool adopts a first-in, first-out (FIFO) mechanism. When the storage capacity reaches the maximum capacity M, the oldest stored experience is removed to maintain dynamic balance.

[0029] Preferably, the GRU neural network training and parameter optimization in step S04 includes the following steps:

[0030] S41: Training Data Preparation and Input. When the sample size in the experience pool reaches a set threshold, a stratified sampling mechanism is used to extract a batch of samples from the experience pool. The samples contain a temporal state sequence, corresponding action, reward, and next state information. These samples are input into the constructed GRU neural network, where the input layer receives a state matrix containing a temporal window of T, providing a data foundation for subsequent training.

[0031] S42: Input data is processed by the GRU layer, dynamically filtering historical and current state association information through reset and update gates, and outputting a hidden state to the fully connected layer. The state input is first processed by the GRU unit, then outputs a hidden state h containing historical information. t It reflects the status information of the current and past time slots:

[0032] h t =GRU(S) t ,h t-1 )

[0033] Among them, h t This represents the output of the GRU network in time slot t. Specifically, it represents the output hidden state h. t for:

[0034]

[0035] Among them, h t-1 This is the hidden state from the previous time slot. As a candidate state for the current time slot, z t To update the door, it was decided that h t-1 and The ratio between them. Therefore, when using a GRU-based neural network for Q-value estimation, the network's Q-value output is:

[0036] Q(s t ,at ;θ)=FC(h t )

[0037] Here, FC stands for fully connected output layer. This means that the network input includes not only the current state information but also historical state information. The fully connected layer uses an activation function for non-linear mapping, and finally, the output layer outputs the Q-value prediction results corresponding to the action selection of each network slice, completing the forward propagation process.

[0038] S43: Loss Function Calculation and Backpropagation. The mean squared error between the target Q-value and the predicted Q-value is used as the loss function. The gradient of the loss with respect to the parameters of each layer is calculated using the gradient descent algorithm. The method for calculating the Q-value is as follows:

[0039] Q(s,a)←Q(s,a)+α(R(s,a)+γmaxQ(s′,a)-Q(s,a))

[0040] The gradient is backpropagated starting from the output layer, and the weights and biases of the fully connected layer and GRU layer are updated sequentially to continuously optimize the network parameters and reduce the loss.

[0041] S44: Iterative Model Training and Slice Selection Validation. After each round of parameter updates, the Q-values ​​of all actions are calculated using the currently trained model for the updated state. The action with the largest Q-value is selected as the optimal slice resource allocation result and recorded. The data sampling, network training, and action selection process is repeated until the network converges and the termination condition is met.

[0042] This application has achieved the following beneficial effects:

[0043] (1) In the deep reinforcement learning training process for selecting vehicular network slices, the traditional fixed-probability exploration mechanism has significant shortcomings. This mechanism fixes the exploration probability at a certain value and cannot dynamically adjust it according to the progress of the training phase and changes in the network state. In the early stage of training, because the exploration probability is fixed and may be at a low level, the agent is unable to fully explore the complex and ever-changing state space in the vehicular network and is prone to getting stuck in local optima due to insufficient exploration, resulting in the learned resource allocation strategy being only applicable to limited scenarios. In the later stage of training, as the agent accumulates a certain number of effective strategies, the fixed exploration probability will trigger too many unnecessary exploration behaviors, causing fluctuations in the strategy iteration process and a significant decrease in stability. Especially in the vehicular network environment, scenarios such as sudden traffic surges and rapid changes in vehicle density occur frequently. The fixed exploration probability cannot quickly adapt to these dynamic changes, which not only prolongs the convergence time of the resource allocation strategy but also seriously affects the long-term stability of resource allocation, making it difficult to continuously guarantee the service quality of network slices.

[0044] (2) Existing deep reinforcement learning methods based on fully connected neural networks have significant shortcomings in capturing temporal features when dealing with the vehicular network slice selection problem. Fully connected neural networks can only extract static features from the network state at the current moment and cannot effectively model the long-term dependencies of the vehicular network state over time. This deficiency directly leads to a large deviation in Q-value estimation, making it impossible for the agent to accurately evaluate the long-term benefits of different resource allocation actions during the learning process, resulting in a significant reduction in learning efficiency. Ultimately, the generated resource allocation strategy is difficult to adapt to the characteristics of high user mobility and rapid state changes in vehicular networks, often resulting in unreasonable resource allocation and excessive service latency in practical applications.

[0045] (3) Traditional experience replay mechanisms suffer from a serious problem in deep reinforcement learning training for vehicular network resource allocation: they disrupt the temporal correlation of experience. This mechanism typically stores isolated samples of <state, action, reward, next state> for a single time step, severing the inherent causal relationship between resource allocation decisions across consecutive time steps. This fragmented training data fails to provide the model with a learning basis that aligns with the actual dynamic changes in vehicular networks, leading to frequent convergence oscillations and low training efficiency. Furthermore, because the optimal strategy pattern in a dynamic environment cannot be accurately extracted from historical experience, the model often makes unreasonable resource allocation decisions when facing new network states, further limiting its application effectiveness in real-world vehicular network scenarios.

[0046] This invention proposes a network slice selection method based on deep reinforcement learning using gated recurrent units (GRUs). This method optimizes the slice selection process by improving the deep Q-network architecture, specifically including the following steps: First, based on the dynamic characteristics of the training process, the exploration probability of action selection is adaptively adjusted, while the decay coefficient dynamically changes with the number of training steps. In the early stages of training, this ensures the agent fully explores the vehicular network state space, and as training progresses, the utilization of existing knowledge is gradually strengthened, thus balancing exploration needs and utilization efficiency. Second, a neural network structure is constructed using GRUs, leveraging their advantage in processing temporal data to enhance the ability to capture long-term temporal dependencies of network states, solving the problem of Q-value estimation bias in traditional fully connected networks. Third, a temporal experience pool mechanism incorporating temporal features is adopted, storing the state sequence, action sequence, and reward sequence of consecutive time steps as a whole sample, preserving the temporal causal relationship of state information, and completing effective training through the state sequence within the temporal window. Through these steps, the problems of insufficient temporal feature capture, imbalance between exploration and utilization, and temporal discontinuity in experience replay during traditional deep reinforcement learning in vehicular network slice selection are effectively solved, improving the accuracy and stability of network slice selection and ensuring the effective guarantee of vehicular network service quality.

[0047] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0048] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0050] Figure 1 This is a schematic diagram of the network slice selection algorithm in vehicular networks.

[0051] Figure 2 This is a diagram illustrating the application scenario of the algorithm. In this area, network slicing technology divides network resources into several different services (slice 1, slice 2, slice 3, slice 4), and the system allocates different network resources to user devices based on user needs using an algorithm.

[0052] Figure 3 This diagram illustrates the algorithm's execution flow. The algorithm consists of several time steps, each of which executes the process shown in the diagram above, thereby generating several allocation strategies for different states. Figure 4 This is a schematic diagram of the network training process. During operation, the algorithm continuously updates the training neural network to build a real-time updated deep Q-learning model to solve for the Q-value. The process shown in the diagram above is executed during each network training session. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0054] Example:

[0055] like Figure 1 As shown, the process of a network slice selection method based on gated loop units in an in-vehicle network is as follows:

[0056] S01: Construct a specific area simulation vehicle network scenario including base stations, user equipment, and network slice types, define the core elements, and clarify the network slice selection issue.

[0057] S02: Model the scenario problem as a Markov decision process, defining the temporal state space, action space, reward function, and state transition probability.

[0058] S03: A dynamic decay strategy is adopted for action selection. The time series state, action, reward and updated state are combined into a quadruple and stored in the time series experience pool. The experience pool is managed through first-in-first-out and hierarchical sampling mechanisms.

[0059] S04: Construct a neural network with a GRU layer, train and optimize parameters based on samples sampled from the experience pool, and iteratively execute experience generation, training and slice selection until termination.

[0060] like Figure 2 As shown, the vehicle network scenario construction steps of the present invention are as follows:

[0061] (1) First, a network slicing simulation model based on a specific area of ​​the vehicular network is built. A base station (BS) is deployed in this specific area. The core function of the base station is to allocate heterogeneous network resources to different types of user groups in the area. Considering that different user groups have different needs for network resources, network slicing technology is introduced into the model to achieve differentiated services for various types of users. At the same time, the base station is set to be able to monitor the network operation status and user request dynamics in real time, and adjust the resource allocation strategy in a timely manner based on this real-time information, so as to ensure that network resources are used efficiently and that the quality of service provided to users is optimal.

[0062] (2) Next, the relevant network parameters are set. The maximum bandwidth of the network is set to a specific value. This general frequency band has the ability to carry multiple network slice resource service requests, and all these service requests follow the same SLA protocol. The network slice can be specifically represented by a corresponding symbolic form. For each network slice, there are two key attribute parameters: Quality of Service (QoS) and resource utilization. Among them, Quality of Service (QoS) refers to the performance level provided by the network service within a certain period of time; resource utilization is the proportion of resources used by the network slice service to achieve the corresponding QoS. Within this area, users will send service requests for different network slices to the system according to their actual needs. These requests enter the corresponding waiting queue according to the first-in-first-out principle, and the waiting queue can be represented by a specific symbol. When a request is responded to, the service request will be removed from the waiting queue and added to the queue of responded service requests. The number of responded service requests is recorded as O, which can also be represented by a specific symbol.

[0063] (3) Clarify the core issues of network slice selection. That is, to maximize network utility under the premise of system resource constraints. Since resource utilization is related to the number of requests responded to, it is necessary to control resource utilization at a low level while ensuring high quality of service (QoS), so as to achieve the best overall service quality under limited resource conditions.

[0064] like Figure 3 As shown, in the neural network training process of this invention, the dynamically decaying action selection strategy and the temporal experience pool mechanism work together, and the specific implementation is as follows:

[0065] (1) There are N types of network slice services available in the system. Before selecting an action, the specific value of the decay coefficient is calculated based on the current training process T. This value is then multiplied by the initial exploration probability to obtain the actual exploration probability required for the current training step. This value is then compared with the minimum exploration probability, and the larger value is selected as the final exploration probability.

[0066] (2) A random number `random` is then generated. When `random` is greater than or equal to the exploration probability, service resource allocation is performed on the network slice type corresponding to the largest Q value; when `random` is less than the exploration probability, a random action is selected. After selecting the network slice type, the current time series state is updated, and the reward value R for the action is calculated according to the reward calculation formula, which is:

[0067]

[0068] Simultaneously, a temporal experience pool containing temporal features is initialized, with its maximum capacity set to a temporal experience fragment of size D. Each experience corresponds to state information over a continuous time T, specifically including a state sequence with 2N features over a continuous time slot T, as well as the action sequence, reward sequence, and state sequence of the current state over the next continuous time slot T.

[0069] (3) In the initial stage of model training, the agent interacts with the vehicle network environment with an initial exploration rate, continuously collects data for T time slots, encapsulates it into a time-series experience segment, and stores it in the experience pool. When the amount of experience in the experience pool reaches the amount required for training, the model starts the training process. After the capacity of the experience pool meets the minimum training requirements, the agent collects the state data of the most recent T time slots each time it performs a network slice selection action, supplements the current action, immediate reward, and next state data to form a new time-series segment and stores it in the experience pool; if the experience pool is full, the oldest segment is deleted according to the first-in-first-out principle. During each training, the required experience is selected from the time-series experience pool using a hierarchical sampling and playback method, and the neural network parameters are updated according to the difference between the calculated Q value and the estimated Q value.

[0070] like Figure 4As shown, the training process of the improved deep Q-learning of the present invention includes:

[0071] (1) The default pre-training yields the initial update neural network, followed by a retraining process.

[0072] (2) Initialize the request queue for N network slice types as Queue, and the number of allocated requests as Φ. When the system executes to meet the minimum timing requirements, the current timing state S is used as a reference, where S is the set of the current service request queue Queue and the number of allocated requests Φ in the T time slot window, with a size of 2N*T. Subsequently, the corresponding action is selected according to the action selection mechanism based on the decay of the training process. After the action is selected, the state S→S′ is updated, that is, the selected slice type in Queue is reduced by 1, the selected slice type in Φ is increased by 1, and the reward value R of the selected action is calculated. The quadruple consisting of the T time slot window state, action, updated state, and reward value is stored as an experience in the timing experience pool D.

[0073] (3) After the experience pool D is stored into the batch, the time-series experience pool is sampled according to the hierarchical sampling and playback mechanism to obtain the experience required for training the batch network. For each sampled experience, it is input into the neural network, and the loss is calculated according to the corresponding action and reward. The neural network is trained using gradient descent.

[0074] (4) In the updated state, the reward Q value of action N is calculated using the trained neural network. The action selected by the action selection mechanism is used as the network slice type and saved. The saved action set a is the optimal solution for network slice selection. Repeat step (2) in a loop, adding the obtained experience to the experience pool and exiting the first experience to enter, satisfying the first-in-first-out rule. Continue with step (3) to train the neural network. Finally, repeat step (4) until the allocation of network resources reaches the limit condition, that is, when the network resources allocated to the selected network slice are greater than the maximum bandwidth, the loop ends. Finally, update the parameter slice request queue of the current time step to the Queue of the next time step. t+1 =Q t .

[0075] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented, in whole or in part, as a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A network slice selection method based on gated recurrent unit deep reinforcement learning, characterized in that, Includes the following steps: S01: Construct a simulated vehicle network scenario; Construct a simulated vehicle network scenario for a specific area that includes base stations, user equipment and various network slice types, define the attribute parameters and interaction rules of each core element in the scenario, and clarify the specific boundaries and indicators to be optimized for the network slice selection problem; S02: Model the network slice selection problem as a Markov decision process; Model the network slice selection problem as a Markov decision process, and define the temporal state space, action space, reward function, and state transition probability; S03: Modify the action selection strategy and experience pool mechanism in the deep Q network algorithm; adopt a dynamically decaying action selection strategy, adjust the exploration probability as the training process progresses, try new actions with a higher probability in the early stage of training, and focus on selecting better actions in the later stage; after the agent executes an action, update the state and calculate the reward, and form a quadruple of temporal state, action, reward and updated state as temporal experience and store it in the temporal experience pool. The experience pool uses a first-in-first-out (FIFO) mechanism to maintain dynamic balance. When adding new experience, the oldest sample is removed to ensure data timeliness. At the same time, a stratified sampling mechanism is used to provide effective data for training. S04: Perform GRU neural network training and parameter optimization, and output the optimal network slice selection result; construct a neural network architecture containing an input layer, a GRU layer, and a fully connected layer. The GRU layer captures long-term temporal dependencies of the state through a gating mechanism; when the sample size in the experience pool reaches a threshold, a hierarchical sampling mechanism is used to extract samples, input them into the network to calculate the predicted Q value, and the mean square error between the target Q value and the predicted Q value is used as the loss function to iteratively update the parameters through gradient descent; after each round of training, the current neural network is used to calculate the Q value of all actions for the updated state, and the action with the largest Q value is selected as the optimal slice selection result and recorded; repeat the experience generation, network training, and slice selection process until the termination condition is reached.

2. The network slice selection method based on gated recurrent unit deep reinforcement learning according to claim 1, characterized in that, The steps S1 for building the in-vehicle network simulation environment include: S11: Construct a network slicing simulation model based on a specific area of ​​the vehicular network; clarify that there is one base station (BS) in this area, whose core function is to allocate heterogeneous network resources to diverse user groups in the area; since the resource needs of different user groups are significantly different, the model introduces network slicing technology to achieve differentiated services; at the same time, it is set that the base station needs to monitor the network status and user request dynamics in real time, and adjust the resource allocation strategy accordingly to ensure that network resources are used efficiently and the service quality reaches the optimal level; S12: Set the maximum network bandwidth to B max This general-purpose frequency band can carry service requests for |N| types of network slice resources, and these requests follow the same SLA protocol. A network slice can be represented as N = {n1, n2, n3, ..., n1}. Each network slice contains two key attribute parameters: Quality of Service (QoS) ρ and resource utilization μ. QoS refers to the performance level provided by the network service within a certain time period; resource utilization refers to the proportion of resources used by the network slice service to obtain QoS. Within this area, users send service requests for different network slices to the system according to their needs. These requests are queued into corresponding waiting queues according to the first-in, first-out principle. The waiting queues can be represented as K = {k1, k2, k3, ..., kk1}. |N| }; Whenever a request receives a response, the corresponding service request is removed from the waiting queue and added to the queue of responded service requests. The number of responded service requests is denoted as O, which can be represented as O = {o1, o2, o3, ..., o}. |N| }; S13: The core issue in network slice selection is maximizing network utility U while satisfying system resource constraints. t Since resource utilization is related to the number of requests responded to, it is necessary to keep resource utilization at a low level while ensuring high QoS, so as to achieve the best overall service quality under limited resource conditions.

3. The network slice selection method based on gated recurrent unit deep reinforcement learning according to claim 1, characterized in that, The construction steps of Markov decision modeling and experience generation storage in step S02 include: S21: Define the core components of the Markov decision process; the temporal state space S includes the slice request queue K of the current time slot, the number of responded requests O, and the historical states of the previous T time slots; the action space A is the network slice selection scheme, and the total resource allocation does not exceed the maximum bandwidth of the base station; the reward function R is calculated as follows: The state transition probability P is fixed in this invention.

4. The network slice selection method based on gated recurrent unit deep reinforcement learning according to claim 1, characterized in that, By employing a dynamic decay action selection strategy, we can balance the exploratory needs in the early stages of training with the convergence stability in the later stages, thereby improving training efficiency.

5. The network slice selection method based on gated recurrent unit deep reinforcement learning according to claim 1, characterized in that, By using a temporal experience pool containing temporal features, the efficiency of experience utilization is improved through temporal correlation, providing matching training samples for temporal dependency modeling of gated recurrent units.

6. A network slice selection method based on gated recurrent unit deep reinforcement learning according to claims 1, 4, and 5, characterized in that, The construction steps of the dynamic decay action selection strategy and time-series experience pool management in step S03 include: S31: Initialize the time series experience pool containing time series features; set the maximum capacity of the experience pool to M, the storage unit to time series experience, and each segment to contain the associated data of T consecutive time slots. The specific information includes the state sequence of the T consecutive time slots, the corresponding action sequence, the instant reward sequence, and the next state sequence of the last time slot of the segment. S32: In the early stage of model training, fill the time-series experience pool; the agent interacts with the vehicle network environment and continuously collects state S, action A, and reward R according to time steps. In each time slot, the continuous data of the previous T time window is packaged into a time-series experience segment and stored in the time-series experience pool. This process is repeated until the storage capacity of the experience pool reaches the minimum sampling capacity. S33: Set the core parameters of the dynamic decay action selection strategy; determine the initial exploration probability ε0 and the minimum exploration probability ε0. min And the decay coefficient λ, when calculating the decay action factor ε, the exponent λ of the decay coefficient with respect to the number of training steps is used. t Multiply the initial exploration probability by the maximum value between the initial exploration probability and the minimum exploration probability as the decay action factor; when selecting an action, the agent makes a random or greedy selection based on the probability of the current decay action factor ε, and then updates the state and action reward and forms a quadruple to store in the experience pool. S34: Generate and store time-series quadruples; After the agent selects action a based on the state of T time windows, it updates the state S→S′, and incorporates the state of T time windows, the current action, the immediate reward and the next state into the quadruple (S,a,S′,R), and stores it as an experience in the time-series experience pool. S35: Once the experience pool meets the training requirements, sampling training is performed. Each time, a batch of data is sampled and replayed from the time-series experience pool for model training. The experience pool adopts a first-in-first-out mechanism. When the storage capacity reaches the maximum capacity M, the earliest stored experience is removed to maintain dynamic balance.

7. The network slice selection method based on gated recurrent unit deep reinforcement learning according to claim 1, characterized in that, Gated recurrent unit neural networks extract historical information from the state, increasing the accuracy of Q-value estimation and thus improving the algorithm's performance.

8. A network slice selection method based on gated recurrent unit deep reinforcement learning according to claims 1 and 7, characterized in that, The GRU neural network training and parameter optimization in step S04 includes the following steps: S41: Training data preparation and input; When the sample size of the experience pool reaches a set threshold, a stratified sampling mechanism is used to extract a batch of samples from the experience pool; The samples contain a temporal state sequence, corresponding action, reward and next state information. These samples are input into the constructed GRU neural network, where the input layer receives a state matrix containing a temporal window of T, providing a data foundation for subsequent training; S42: Input data is processed by the GRU layer, and historical and current state association information is dynamically filtered through reset and update gates. The hidden state is then output to the fully connected layer. The state input is first processed by the GRU unit, and then a hidden state h containing historical information is output. t It reflects the status information of the current and past time slots: h t =GRU(S t ,h t-1 ) Among them, h t The output of the GRU network in time slot t; specifically, the output hidden state h. t for: Among them, h t-1 This is the hidden state from the previous time slot. As a candidate state for the current time slot, z t To update the door, it was decided that h t-1 and The ratio between; therefore, when using a GRU-based neural network for Q-value estimation, the network Q-value output is: Q(s t ,and t ;θ)=FC(h t ) Here, FC stands for fully connected output layer; this means that the network input includes not only the current state information, but also the historical state information; the fully connected layer uses an activation function for nonlinear mapping, and finally the output layer outputs the Q-value prediction results corresponding to the action selection of each network slice, thus completing the forward propagation process; S43: Loss Function Calculation and Backpropagation; using the mean squared error between the target Q-value and the predicted Q-value as the loss function, the gradient of the loss with respect to the parameters of each layer is calculated using the gradient descent algorithm; where... The Q value is calculated as follows: Q(s,a)←Q(s,a)+α(R(s,a)+γmaxQ(s′,a)-Q(s,a)) The gradient is backpropagated from the output layer, and the weights and biases of the fully connected layer and the GRU layer are updated sequentially to continuously optimize the network parameters and reduce the loss. S44: Model iterative training and slice selection verification; after each round of parameter update, use the currently trained model to calculate the Q value of all actions in the updated state, select the action with the largest Q value as the optimal slice resource allocation result and record it; repeat the data sampling, network training and action selection process until the network converges and the termination condition is met.

9. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores an executable computer program, and when the processor runs the computer program stored in the memory, the processor executes the network slice selection method based on gated recurrent unit deep reinforcement learning as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to perform the network slice selection method based on gated recurrent unit deep reinforcement learning as described in any one of claims 1 to 8.

Images