An intelligent network slice selection method for mobile services

Through the intelligent slice switching method of deep reinforcement learning, combined with base station and user information, optimize slice switching, the problem of single slice switching performance indicators in the existing technology is solved, and the overall performance optimization and cost saving of the mobile communication system is achieved.

CN115834415BActive Publication Date: 2024-11-26THE 54TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORPORATION
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202211314643.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-26
Publication Date
2024-11-26
Estimated Expiration
2042-10-26

AI Technical Summary

Technical Problem

The prior art ignores the consideration of base station coverage and network resources in the slice switching problem in mobile communication systems, and the performance indicators are single, so the overall performance of the system cannot be effectively optimized to ensure user QoS.

Method used

Using an intelligent slice switching method based on deep reinforcement learning, the agent is trained through a dual-deep Q network, combining base station, slice, and user information to design the status and reward functions in Markov's decision-making process, considering service rewards, switching costs and delays At the cost, realize intelligent slicing selection.

Benefits of technology

In high-dimensional state space, consider user type and task information, optimize the slicing switching process, improve system performance, reduce computing costs, and achieve overall performance optimization in multiple business scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115834415B_ABST
    Figure CN115834415B_ABST
Patent Text Reader

Abstract

The present invention relates to an intelligent network slice selection method for mobile services, and belongs to the field of mobile communications. The present invention first establishes a slice selection agent, and trains the agent based on a double deep Q network; when the training result converges, a trained agent is obtained, and the trained agent is used to select intelligent network slices. The present invention expands the measurement content, adds a measurement function that supports the measurement of corresponding measurement quantities, and on this basis proposes an implementation process of an intelligent network slice selection method for mobile services based on Double DQN, providing an intelligent solution to the slice selection problem.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of mobile communications, and specifically relates to an intelligent network slice selection method for mobile services. Background Art

[0002] The development of mobile communication technology has promoted the emergence of a large number of new businesses in various vertical industries. The network requirements of various new businesses are significantly different, and it is impossible to meet the network requirements of all businesses through one network. To this end, a physical network can be logically isolated to form multiple sub-networks, and network slicing technology that provides customized services for various services is applied to 5G mobile communication systems. When network slicing technology is applied to mobile communication systems, users can access specific network slices to obtain customized services. However, the coverage of network slices is limited. In order to ensure that users obtain high-quality and continuous services, it is necessary to select suitable network slices for users to access. Therefore, in network slicing technology, the switching problem is an issue that cannot be ignored. Due to the continuous emergence of new businesses, the system state dimension increases rapidly. Solving the slice switching problem through traditional methods such as mathematical models will consume a lot of time and computing costs. In order to save costs and optimize the overall performance of the system while ensuring user QoS, deep reinforcement learning theory has become an effective method to solve the slice switching problem.

[0003] However, there is still room for improvement in the consideration of slice switching in the prior art. First, some studies only consider the association between users and slices, ignoring the base stations that provide network resources for the slices. However, the service scope of a network slice depends on the coverage of the base stations that provide network resources for it. To ensure the feasibility of the switching method, the coverage of the base stations must be considered. In addition, the modeling of network status in existing studies is not sufficient. Many studies only consider radio access network resources or core network resources separately, without modeling both at the same time. Or both radio access network and core network resources are considered at the same time, but only stay at the number of resources, without considering more complex network information such as service type, user status, and user tasks. Secondly, the performance indicators considered in most studies are relatively single, such as only considering switching cost, service rate, system throughput, etc. When the network provides services for different services, the service requirements are often diverse and are not limited to a single performance. Moreover, the performance indicators may affect each other or even conflict with each other. Therefore, in order to optimize the overall performance of the system while ensuring the QoS of users in the network, it is necessary to comprehensively consider a variety of performance indicators that affect network performance, and the overall performance of the system cannot be characterized by a single performance indicator. Summary of the invention

[0004] In view of this, the present invention discloses a slice switching method for mobile services. The present invention is oriented to mobile services and implements intelligent slice switching based on deep reinforcement learning.

[0005] The technical solution adopted by the present invention is:

[0006] An intelligent network slice selection method for mobile services includes the following steps:

[0007] In the training phase, a slice selection agent is established and trained based on a dual-depth Q network;

[0008] In the usage phase, when the training results converge, a trained intelligent agent is obtained, and the trained intelligent agent is used to select intelligent network slices.

[0009] Furthermore, the specific method of the training session is:

[0010] Step T1, check the arrival of user tasks, initialize data transmission information according to the number of tasks, task delay, and data transmission status, update the user task list, update the switching decision queue, and go to step T2;

[0011] Step T2, collect base station, slice inherent information, resource status information, user inherent information, user task information, data transmission information, user service demand information, user connection information, and model the above information as state s in the Markov decision process j , and the state s j Delivered to step T3 and step T7;

[0012] The status j The specific implementation is as follows:

[0013] For user u j For , the corresponding environmental state is s j =[s s ,s u ,s w ],in:

[0014]

[0015] In the formula, K and N represent the number of base stations and the number of slices in the network respectively. Indicates the service type of the lth base station slice pair, which is the inherent information of the base station and slice; α l represents the real-time available radio access network resources for the lth base station slice, β l Indicates the core network resources available in real time for the lth base station slice, which is resource status information;

[0016]

[0017] In the formula, α j,l , β j,l Represents user u j Obtain the number of radio access network resources and core network resources required for the service on base station slice pair l, which is user service demand information; C j,l Represents user u j The delay cost of accessing the base station slice l is the switching information; p j,m , d j,m Represents user u j The delay cost of the mth data packet to be transmitted in the task queue and the remaining time before interruption, D j Represents user u j The number of packets to be served, which is data transmission information; k l Indicates whether the base station slice pair l can be used by user u j Provide service, if the service is available, its value is 1, otherwise it is 0, I j Represents user u j The service type is the user's inherent information; A j Represents user u j The base station slice pair accessed at the previous time step is the user connection information;

[0018] s w is an array containing only elements 0 or 1, where the number of 1s represents the number of users remaining to be decided;

[0019] Step T3, the state s obtained in step T2 j Input the dual-depth Q network, calculate and output the Q values ​​of all base station slice pairs, and go to step T4;

[0020] Step T4: All actions, i.e., all base station slice pairs, are passed through the action filter to filter out actions that cannot provide services to users, and then action a is selected according to the ε-greedy strategy. j , and go to step T5; in addition, the action information, that is, the user connection information, is sent to step T2;

[0021] The action filter is used to judge each constraint condition in turn, eliminating the insufficient resource quantity, service rate or delay that does not meet the user u j The base station slice pairs with service quality requirements only leave the available base station slice pairs for users to filter;

[0022] Step T5, follow the action a obtained in step T4 j Execute slice switching, the user occupies the network resources of the corresponding base station slice pair, go to step T6, and perform action a j The information is sent to step T7;

[0023] Step T6, design a reward function to calculate the reward r according to the switching result in step T5 j , the environmental state is given by s j Transfer to s' j , go to step T7;

[0024] The specific implementation of the reward function is as follows:

[0025] The reward function is designed as the sum of service reward, switching cost and delay cost; at each time step, user u j The reward is r j Indicates that its value takes into account three utility values, namely service reward C1, switching cost C2, and delay cost C3; the reward value C1 is divided into 5 levels according to the service type, and the user switching is divided into three cases: switching only slices, switching only base stations, and switching slices and base stations at the same time; according to the different switching types, the switching cost C2 is also divided into 3 levels; each data packet has the same total penalty until the interruption occurs, but because the sensitivity of each type of task to delay information is different, the delay cost of each type of task at each time step is also different; the parameter table of each utility value is as follows:

[0026]

[0027]

[0028] r j Calculated by the following formula:

[0029]

[0030] Step T7, the (s j ,a j ,r j ,s' j ) is added as a training sample to the experience playback, and then goes to step T8;

[0031] Step T8, if the number of samples in the experience playback in step T7 is greater than or equal to 32, go to step T9;

[0032] Step T9, to maximize the r j To optimize the target, perform dual-depth Q network training and go to step T10;

[0033] Step T10, after all users have completed their decision-making, all user task information is read in the order of the switching decision queue. If a user has a task that has been completed or has timed out, the corresponding data packet is deleted and the corresponding delay information is deleted. If there is no task to be served in the task list of a user, the user is deleted from the switching decision queue, the corresponding occupied network resources are released, the task information and resource information are updated, and the information is sent to step T2.

[0034] Furthermore, the specific method of the use link is:

[0035] Step U1, check the arrival of user tasks, initialize data transmission information according to the number of tasks, task delay, and data transmission status, update the user task list, update the switching decision queue, and go to step U2;

[0036] Step U2 collects base station and slice inherent information, as well as resource status information, user inherent information, user task information, data transmission information, user service demand information, and user connection information, and models the above information as state s in the Markov decision process j , the state s j Delivered to step U3 and step U7;

[0037] Step U3, the state s obtained in step U2 j Input the dual-depth Q network, calculate and output the Q values ​​of all base station slice pairs, and go to step U4;

[0038] Step U4: All actions, i.e., all base station slice pairs, are filtered through the action filter to filter out actions that cannot provide services to users, and then action a is selected according to the greedy strategy. j , go to step U5 and change action a j The information, i.e. the user connection information, is sent to step U2;

[0039] Step U5, follow the action a obtained in step U4 j Execute slice switching, the user occupies the network resources of the corresponding base station slice pair, and the action a j The information is sent to step U7 and step U6;

[0040] Step U6, calculate the reward r according to the switching result in step U5 j , the environmental state is given by s j Transfer to s' j , go to step U7;

[0041] Step U7, after all users have completed their decision, read all user task information in the order of the switching decision queue. If a user has a task that has been completed or has timed out, delete the corresponding data packet and the corresponding delay information. If there is no task to be served in the task list of a user, delete the user from the switching decision queue, release the corresponding occupied network resources, update the task and resource information and transfer the information to step U2. If all users' tasks have been completed, the entire switching process ends, otherwise, start the cycle again from step U1.

[0042] The advantages of the present invention compared with the prior art are:

[0043] 1. The present invention is aimed at scenarios with high-dimensional state space and coexistence of multiple services, and considers the problem of slice switching in heterogeneous networks.

[0044] 2. The present invention incorporates consideration of user type and task information into the system status; expands the measurement content and designs a slice switching protocol that supports information measurement in the proposed model.

[0045] 3. The present invention extracts and integrates the information obtained from the measurement, models the slice switching process as a Markov decision process, takes into account resource information, user information, and task information in the state modeling, and designs a reward function that takes into account service rewards, switching costs, and delay costs, thereby realizing an intelligent slice switching solution based on deep reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a framework diagram of slice switching according to an embodiment of the present invention.

[0047] Figure 2 Schematic diagram of a network according to an embodiment of the present invention.

[0048] Figure 3 It is an implementation flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0050] An intelligent network slice selection method for mobile services includes the following steps:

[0051] In the training phase, a slice selection agent is established and trained based on a dual-depth Q network;

[0052] In the usage phase, when the training results converge, a trained intelligent agent is obtained, and the trained intelligent agent is used to select intelligent network slices.

[0053] Furthermore, the specific method of the training session is:

[0054] Step T1, check the arrival of user tasks, initialize data transmission information according to the number of tasks, task delay, and data transmission status, update the user task list, update the switching decision queue, and go to step T2;

[0055] Step T2, collect base station, slice inherent information, resource status information, user inherent information, user task information, data transmission information, user service demand information, user connection information, and model the above information as state s in the Markov decision process j , and the state s j Delivered to step T3 and step T7;

[0056] The status j The specific implementation is as follows:

[0057] For user u j For , the corresponding environmental state is s j =[s s ,s u ,s w ],in:

[0058]

[0059] In the formula, K and N represent the number of base stations and the number of slices in the network respectively. Indicates the service type of the lth base station slice pair, which is the inherent information of the base station and slice; α l represents the real-time available radio access network resources for the lth base station slice, β l Indicates the core network resources available in real time for the lth base station slice, which is resource status information;

[0060]

[0061] In the formula, α j,l , β j,l Represents user u j Obtain the number of radio access network resources and core network resources required for the service on base station slice pair l, which is user service demand information; C j,l Represents user u j The delay cost of accessing the base station slice l is the switching information; p j,m , d j,m Represents user uj The delay cost of the mth data packet to be transmitted in the task queue and the remaining time before interruption, D j Represents user u j The number of packets to be served, which is data transmission information; k l Indicates whether the base station slice pair l can be used by user u j Provide service, if the service is available, its value is 1, otherwise it is 0, I j Represents user u j The service type is the user's inherent information; A j Represents user u j The base station slice pair accessed at the previous time step is the user connection information;

[0062] s w is an array containing only elements 0 or 1, where the number of 1s represents the number of users remaining to be decided;

[0063] Step T3, the state s obtained in step T2 j Input the double deep Q network (i.e., Double DQN), calculate and output the Q values ​​of all base station slice pairs, and then go to step T4;

[0064] Step T4: All actions, i.e., all base station slice pairs, are passed through the action filter to filter out actions that cannot provide services to users, and then action a is selected according to the ε-greedy strategy. j , and go to step T5; in addition, the action information, that is, the user connection information, is sent to step T2;

[0065] The action filter is used to judge each constraint condition in turn, eliminating the insufficient resource quantity, service rate or delay that does not meet the user u j The base station slice pairs with service quality requirements only leave the available base station slice pairs for users to filter;

[0066] Step T5, follow the action a obtained in step T4 j Execute slice switching, the user occupies the network resources of the corresponding base station slice pair, go to step T6, and perform action a j The information is sent to step T7;

[0067] Step T6, design a reward function to calculate the reward r according to the switching result in step T5 j , the environmental state is given by s j Transfer to s' j , go to step T7;

[0068] The specific implementation of the reward function is as follows:

[0069] The reward function is designed as the sum of service reward, switching cost and delay cost; at each time step, user uj The reward is r j Indicates that its value takes into account three utility values, namely service reward C1, switching cost C2, and delay cost C3; the reward value C1 is divided into 5 levels according to the service type, and the user switching is divided into three cases: switching only slices, switching only base stations, and switching slices and base stations at the same time; according to the different switching types, the switching cost C2 is also divided into 3 levels; each data packet has the same total penalty until the interruption occurs, but because the sensitivity of each type of task to delay information is different, the delay cost of each type of task at each time step is also different; the parameter table of each utility value is as follows:

[0070]

[0071]

[0072] r j Calculated by the following formula:

[0073]

[0074] Step T7, the (s j ,a j ,r j ,s' j ) is added as a training sample to the experience playback, and then goes to step T8;

[0075] Step T8, if the number of samples in the experience playback in step T7 is greater than or equal to 32, go to step T9;

[0076] Step T9, to maximize the r j To optimize the target, perform dual-depth Q network training and go to step T10;

[0077] Step T10, after all users have completed their decision-making, all user task information is read in the order of the switching decision queue. If a user has a task that has been completed or has timed out, the corresponding data packet is deleted and the corresponding delay information is deleted. If there is no task to be served in the task list of a user, the user is deleted from the switching decision queue, the corresponding occupied network resources are released, the task information and resource information are updated, and the information is sent to step T2.

[0078] Furthermore, the specific method of the use link is:

[0079] Step U1, check the arrival of user tasks, initialize data transmission information according to the number of tasks, task delay, and data transmission status, update the user task list, update the switching decision queue, and go to step U2;

[0080] Step U2 collects base station and slice inherent information, as well as resource status information, user inherent information, user task information, data transmission information, user service demand information, and user connection information, and models the above information as state s in the Markov decision process j , the state s j Delivered to step U3 and step U7;

[0081] Step U3, the state s obtained in step U2 j Input the dual-depth Q network, calculate and output the Q values ​​of all base station slice pairs, and go to step U4;

[0082] Step U4: All actions, i.e., all base station slice pairs, are filtered through the action filter to filter out actions that cannot provide services to users, and then action a is selected according to the greedy strategy. j , go to step U5 and change action a j The information, i.e. the user connection information, is sent to step U2;

[0083] Step U5, follow the action a obtained in step U4 j Execute slice switching, the user occupies the network resources of the corresponding base station slice pair, and the action a j The information is sent to step U7 and step U6;

[0084] Step U6, calculate the reward r according to the switching result in step U5 j , the environmental state is given by s j Transfer to s' j , go to step U7;

[0085] Step U7, after all users have completed their decision, read all user task information in the order of the switching decision queue. If a user has a task that has been completed or has timed out, delete the corresponding data packet and the corresponding delay information. If there is no task to be served in the task list of a user, delete the user from the switching decision queue, release the corresponding occupied network resources, update the task and resource information and transfer the information to step U2. If all users' tasks have been completed, the entire switching process ends, otherwise, start the cycle again from step U1.

[0086] Aiming at the information needs of the mobile service slice selection process, this method expands the measurement content, designs a slice switching protocol, and designs a specific switching process.

[0087] This method is oriented to the task information of mobile service users. Each user corresponds to a task flow that conforms to the Poisson distribution, which is used to characterize the task arrival time and task type information. In the service model, it is necessary not only to consider the inherent attribute information of the user itself, but also to consider the type information, delay information, quantity information, transmission information, etc. of the data packet. In order to obtain the above status information, the support of the corresponding measurement function is required. To this end, this application designs the following Figure 1 The slice switching protocol shown.

[0088] The handover protocol of this method includes two parts: a state information measurement layer and a handover decision layer. The state information measurement layer and the handover decision layer continuously exchange information. The state information measurement layer transmits the measured information to the handover decision layer for the handover decision layer to extract and integrate the information and use it for handover decision. The decision in the handover decision layer affects the state information, and the information will also be transmitted to the state information measurement layer in real time to update the state measurement and ensure the real-time nature of the state information.

[0089] The following is a detailed description of the two layers of the switching protocol:

[0090] The state information measurement layer includes three parts: link quality measurement function, resource state measurement function and user information measurement function. The link quality measurement function is used to measure the channel quality information between the user and each base station, and to provide the access decision layer with signal strength information of different channels. The resource state measurement function is used to measure the available resource information of each base station slice pair. The user information measurement function is used to measure the service type, resource quantity, user task status and other information required by the user service.

[0091] The handover decision layer includes two parts: the state information table and the intelligent handover decision algorithm. The state information table includes the base station inherent information table, the slice inherent information table, the base station slice pair information table and the user state information table. The information in the state information table includes two types of information: inherent information and information measured by the state information measurement layer.

[0092] In the base station inherent information table, the number is used to distinguish base stations, the power is used to calculate the channel quality between each user and each base station, the inherent resource information is the total number of resources of the base station, and the deployed slice information indicates the slice number deployed by the base station. These four types of information are all inherent information.

[0093] In the slice inherent information table, the number is used to distinguish each slice, the service type indicates the type of service that the slice can provide, the inherent resource information indicates the total amount of resources allocated to the user from the base station that provides physical resources for it, and the coverage base station information indicates the base station number covered by the slice. The information in the slice inherent information table is all inherent information.

[0094] In the base station slice pair information table, the service type is inherent information, where the service type indicates the type of service that the base station slice pair can provide, which is consistent with the type of service that the slices in the base station slice pair can provide; the switching cost between other base station slice pairs needs to be calculated according to the different switching methods, and the different switching methods will be described in detail below. The real-time available resource information indicates the number of networks available in real time for each base station slice pair, including core network resources and wireless access network resources. When a user obtains service to occupy resources or completes service to release resources, the switching decision layer will interact with the status information measurement layer to recalculate the real-time available resources and update the status information table.

[0095] The user status information table contains mostly information that needs to be measured. The service transmission quality requirements include the user service type and the number of resources required for data transmission. The user service type is fixed information, and the number of resources required for data transmission requires the status information measurement layer to measure and calculate in combination with the link quality between the user and each base station. Task quantity information, task delay information, and data transmission status are all related information about the data packets that the user needs to transmit. Whether each user has data packets to transmit, how many data packets need to be transmitted, how long the delay of the data packets to be transmitted is, and how much data each data packet has to wait for transmission all need to be measured and recorded by the status information measurement layer, and passed to the user status information table in the switching decision layer during information exchange.

[0096] After the information in the status information table is collected, the status information is integrated and extracted according to the switching needs of the user to be decided, and a switching decision is made for the user through an intelligent switching decision algorithm.

[0097] Here is a more specific example:

[0098] Figure 1 This is a framework diagram of slice switching. The switching protocol includes two parts: the state information measurement layer and the switching decision layer. The state information measurement layer and the switching decision layer exchange information in real time. The state information measurement layer measures information and transmits it to the switching decision layer. The switching decision layer uses the obtained information to make intelligent switching decisions for users. The impact of the switching decision on the state information will also be fed back to the state information measurement layer in real time to facilitate the update of the state information.

[0099] Figure 2 This is a network diagram. B base stations are deployed in the network, including B M macro base stations, B P Small base stations, B FThere are N home base stations with N slices deployed. One base station can deploy multiple slices, and one slice can also cover multiple base stations. This deployment and coverage relationship constitutes L base station slice pairs. The sequence number of the base station slice pair corresponding to the nth slice and the bth base station is l = n × B + b. U users move randomly in the network. Each slice and user has a corresponding service type. Only when the types match can the slice provide services to the user.

[0100] Figure 3 It is a flow chart of the switching scheme, which describes the specific workflow of the slice switching method proposed in this scheme. In order to describe this scheme more clearly, the implementation method of this scheme is introduced from two parts: MDP modeling and switching method process.

[0101] 1. MDP model

[0102] Considering that the switching decision process is a continuous process in time, this scheme models the slice switching decision process of T time steps as a complete plot of an MDP. MDP can be described by (S, A, P, R). When the decision starts, based on the current state S, the agent selects action A for the user. The user performs action A and obtains a reward value R. Then the state is transferred from S to S', and the above process is repeated until the terminal state is reached. The following is a detailed introduction to each element of MDP.

[0103] (1) State: State design includes system state and user state. j In terms of the corresponding environmental state s j =[s s ,s u ,s w ],in:

[0104]

[0105] Among them, α l represents the real-time available radio access network resources for the lth base station slice, β l Represents the core network resources available in real time for the lth base station slice. l Indicates the service type of the lth base station slice pair.

[0106]

[0107] where α j,l , β j,l Represents user u j The number of radio access network resources and core network resources required to obtain services on base station slice pair l. l Indicates whether the base station slice pair l can be used by user u j Provide service, if the service is available, its value is 1, otherwise it is 0.j,l Represents user u j The latency cost incurred by accessing the base station slice l. j,m , d j,m Represents user u j The delay cost of the mth data packet to be transmitted in the task queue and the remaining time before interruption. j Represents user u j A j Represents user u j The base station slice pair accessed in the previous time step. D j Represents user u j The number of packets to be serviced.

[0108] s w is an array containing only elements 0 or 1, where the number of 1s represents the number of users remaining to be decided.

[0109] (2) Action: All base station slice pairs formed by the coverage deployment relationship between base stations and slices.

[0110] (3) Reward: The reward function is designed as the sum of service reward, switching cost and delay cost. j The reward is r j Indicates that its value takes into account three utility values, namely service reward C1, switching cost C2, and delay cost C3. The reward value C1 is divided into 5 levels according to the type of service. User switching is divided into three cases: switching only slices, switching only base stations, and switching slices and base stations at the same time. According to the type of switching, the switching cost C2 is also divided into 3 levels. Each data packet has the same total penalty until the interruption occurs, but because the sensitivity of each type of task to delay information is different, the delay cost of each time step generated by different types of tasks is also different. j It can be calculated by the following formula:

[0111]

[0112] (4) Optimization goal: maximize the total reward obtained by all users within T time steps, that is, maximize

[0113] 2. Switching scheme workflow

[0114] Here, we take a task process as an example to introduce the workflow of the switching solution. In each task process, users with tasks arriving or unfinished tasks need to make switching decisions. The central controller selects the appropriate base station slice pair to provide services for users based on the number of network resources, user task information and other status information.

[0115] At the start of the task, all users move randomly. The change of user position will cause changes in path loss and shadow fading. It is necessary to recalculate the signal-to-interference-noise ratio and measure the channel quality information to obtain the channel quality information between the user and each base station. Subsequently, all user task information is read, and users with new tasks arriving but not in the handover decision queue are added to the handover decision queue, and the data transmission information and delay information of the newly arrived tasks are initialized.

[0116] Then, according to the order of the handover decision queue, handover decisions are made for the users one by one. The decision process requires selecting a base station slice for the user to access. Here, user u j Take this as an example to illustrate. In this step, first obtain the system state s j , then, the Q network is based on the state s j , calculate and output the Q value corresponding to each base station slice pair. After outputting the Q value, we added an action filter to filter out the Q value of illegal actions that cannot provide services to users, and only retain actions that can provide services to users for users to choose, in order to accelerate convergence. Then, the ε-greedy strategy is used as u j Select a base station slice pair.

[0117] After selecting a base station slice pair, u j Release the network resources occupied by the previous time step and try to access the selected base station slice pair. If the remaining network resources of the selected base station slice pair are sufficient for u j Provide services, and the selected slice service type can be u j Provide services, then u j Access, consume network resources to transmit data packets. The transmission process follows the first-come-first-served principle, and only the first data packet in the current user task queue is transmitted. After the transmission is completed, the remaining number of data packets to be transmitted and the delay information of all data packets are updated. In order to ensure the fairness of the service, the user who successfully switches will be moved to the end of the switching decision list.

[0118] After the decision is made, according to the switching situation and resource consumption, u j Get reward value r j . Then, the environment state is transferred to s' j , will (s j ,a j ,r j ,s' j ) is stored as a training sample in the experience replay. If there are enough training samples stored in the experience replay, training begins.

[0119] After the task is completed and all user decisions are made, all user task information is read in the order of the switching decision queue. If a user has a task that has been completed or has timed out, the corresponding data packet is deleted and the corresponding delay information is deleted. If there is no task to be served in the task list of a user, the user is deleted from the switching decision queue.

[0120] The present invention expands the measurement content and can support the need for switching information under the mobile business model. The present invention designs a slice switching protocol that supports the measurement of information required by the switching method, extracts and integrates the switching information obtained by the measurement, models the slice switching process as a Markov decision process, considers resource information, user information, and task information in state modeling, and designs a reward function that considers service rewards, switching costs, and delay costs at the same time, realizing an intelligent slice switching solution based on deep reinforcement learning.

Claims

1. An intelligent network slice selection method for mobile services, It is characterized in that The following steps are involved: In the training phase, a slice selection agent is established and trained based on a dual-depth Q network. The specific method of the training phase is as follows: Step T1, check the arrival of user tasks, initialize data transmission information according to the number of tasks, task delay, and data transmission status, update the user task list, update the switching decision queue, and go to step T2; Step T2, collect base station, slice inherent information, resource status information, user inherent information, user task information, data transmission information, user service demand information, user connection information, and model the above information as state s in the Markov decision process j , and the state s j Delivered to step T3 and step T7; The status j The specific implementation is as follows: For user u j For , the corresponding environmental state is s j =[s s ,s u ,s w ],in: In the formula, K and N represent the number of base stations and the number of slices in the network respectively. Indicates the service type of the lth base station slice pair, which is the inherent information of the base station and slice; α l represents the real-time available radio access network resources for the lth base station slice, β l Indicates the core network resources available in real time for the lth base station slice, which is resource status information; In the formula, α j,l , β j,l Represents user u j Obtain the number of radio access network resources and core network resources required for the service on base station slice pair l, which is user service demand information; C j,l Represents user u j The delay cost of accessing the base station slice l is the switching information; p j,m , d j,m Represents user u j The delay cost of the mth data packet to be transmitted in the task queue and the remaining time before interruption, D j Represents user u j The number of packets to be served, which is data transmission information; k l Indicates whether the base station slice pair l can be used by user u j Provide service, if the service is available, its value is 1, otherwise it is 0, I j Represents user u j The service type is the user's inherent information; A j Represents user u j The base station slice pair accessed at the previous time step is the user connection information; s w is an array containing only elements 0 or 1, where the number of 1s represents the number of users remaining to be decided; Step T3, the state s obtained in step T2 j Input the dual-depth Q network, calculate and output the Q values ​​of all base station slice pairs, and go to step T4; Step T4: All actions, i.e., all base station slice pairs, are passed through the action filter to filter out actions that cannot provide services to users, and then action a is selected according to the e-greedy strategy. j , and go to step T5; in addition, the action information, that is, the user connection information, is sent to step T2; The action filter is used to judge each constraint condition in turn, eliminating the insufficient resource quantity, service rate or delay that does not meet the user u j The base station slice pairs with service quality requirements only leave the available base station slice pairs for users to filter; Step T5, follow the action a obtained in step T4 j Execute slice switching, the user occupies the network resources of the corresponding base station slice pair, go to step T6, and perform action a j The information is sent to step T7; Step T6, design a reward function and calculate the reward r according to the switching result in step T5 j , the environmental state is given by s j Transfer to s' j , go to step T7; The specific implementation of the reward function is as follows: The reward function is designed as the sum of service reward, switching cost and delay cost; at each time step, user u j The reward is r j Indicates that its value takes into account three utility values, namely service reward C 1 , switching cost C 2 , delay cost C 3 ; Reward value C 1 There are five levels according to different service types. User switching is divided into three situations: switching slices only, switching base stations only, and switching slices and base stations at the same time. According to different switching types, the switching cost C 2 It is also divided into 3 levels; each data packet has the same total penalty until the interruption occurs, but because each type of task has different sensitivities to delay information, the delay costs incurred by different types of tasks per time step are also different; the parameter table of each utility value is as follows: r j Calculated by the following formula: Step T7, the (s j ,a j ,r j ,s' j ) is added as a training sample to the experience playback, and then goes to step T8; Step T8, if the number of samples in the experience playback in step T7 is greater than or equal to 32, go to step T9; Step T9, to maximize the r j To optimize the target, perform dual-depth Q network training and go to step T10; Step T10, after all users have completed their decision, all user task information is read in the order of the switching decision queue. If a user has a task that has been completed or has timed out, the corresponding data packet is deleted and the corresponding delay information is deleted; if there is no task to be served in the task list of a user, the user is deleted from the switching decision queue, the corresponding occupied network resources are released, the task information and resource information are updated, and the information is sent to step T2; In the use phase, when the training results converge, a trained intelligent agent is obtained, and the trained intelligent agent is used to select intelligent network slices. The specific method of the use phase is as follows: Step U1, check the arrival of user tasks, initialize data transmission information according to the number of tasks, task delay, and data transmission status, update the user task list, update the switching decision queue, and go to step U2; Step U2 collects base station and slice inherent information, as well as resource status information, user inherent information, user task information, data transmission information, user service demand information, and user connection information, and models the above information as state s in the Markov decision process j , the state s j Delivered to step U3 and step U7; Step U3, the state s obtained in step U2 j Input the dual-depth Q network, calculate and output the Q values ​​of all base station slice pairs, and go to step U4; Step U4: All actions, i.e., all base station slice pairs, are filtered through the action filter to filter out actions that cannot provide services to users, and then action a is selected according to the greedy strategy. j , go to step U5 and change action a j The information, i.e. the user connection information, is sent to step U2; Step U5, follow the action a obtained in step U4 j Execute slice switching, the user occupies the network resources of the corresponding base station slice pair, and the action a j The information is sent to step U7 and step U6; Step U6, calculate the reward r according to the switching result in step U5 j , the environmental state is given by s j Transfer to s' j , go to step U7; Step U7, after all users have completed their decision, read all user task information in the order of the switching decision queue. If a user has a task that has been completed or has timed out, delete the corresponding data packet and the corresponding delay information. If there is no task to be served in the task list of a user, delete the user from the switching decision queue, release the corresponding occupied network resources, update the task and resource information and transfer the information to step U2. If all users' tasks have been completed, the entire switching process ends, otherwise, start the cycle again from step U1.

Citation Information

Patent Citations

  • End-to-end network slice resource allocation algorithm based on deep reinforcement learning

    CN111683381A

  • D2D communication network slice allocation method based on deep reinforcement learning

    CN113163451A

  • End-to-end network slice switching method and system

    CN113965945A

  • Service orchestration system and method based on intent driving in intelligent fusion identification network

    WO2022193534A1