Resource allocation model training method, resource allocation method and device

Abstract

The present application provides a resource allocation model training method, a resource allocation method and a device, which relate to the field of mobile communication technology, including: obtaining the current state parameters of a simulated core network and a simulated access network; adjusting the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network; obtaining the updated state parameters of the simulated core network and the simulated access network and calculating the reward of the simulated core network and the reward of the simulated access network based on the updated state parameters; calculating the first value corresponding to the current state parameters and the target action set through the verification network, and calculating the second value corresponding to the updated state parameters through the reward of the simulated core network, the reward of the simulated access network and the target network; calculating the loss function value based on the first value and the second value, training the verification network and the target network, and using the trained target network as a resource allocation model. The network service quality can be improved.

Description

Technical Field

[0001] The present application relates to the field of mobile communication technology, and in particular to a resource allocation model training method, a resource allocation method and a device. Background Art

[0002] Network slicing technology is a solution based on public physical infrastructure, which can logically or physically divide the network into different network slices according to different attributes (such as bandwidth requirements, delay tolerance, user density, etc.), so as to achieve scheduling optimization under the constraints of limited resources and meet the needs of different scenarios. In order to meet the many differentiated needs of IoT services, network slicing technology can also be applied to the sixth generation mobile communication technology (6G) network.

[0003] Network slices can be assigned to specific use cases, for example, a network slice can be assigned to a class of services, or to a class of customers, or to specific mobile network operators, or to a network type, such as wireless communication type or wired communication type.

[0004] At present, the network slice corresponding to each use case is determined. If the user demand corresponding to a certain network slice increases or decreases significantly, or an idle network slice is configured for a new use case, it will affect all components in the 6G network, resulting in unreasonable amount of resources occupied by each network slice, and then leading to a decline in network service quality. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a resource allocation model training method, a resource allocation method and a device to improve the quality of network services. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a resource allocation model training method, characterized by comprising:

[0007] Obtain current status parameters of the simulated core network and simulated access network;

[0008] Adjusting the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network according to the target action set;

[0009] Obtain updated state parameters of the simulated core network and the simulated access network, and calculate a reward of the simulated core network and a reward of the simulated access network based on the updated state parameters, wherein the reward of the simulated core network is used to represent a client delay rate, and the reward of the simulated access network is used to represent a client blocking rate;

[0010] Calculate the first value corresponding to the current state parameter and the target action set through the verification network, and calculate the second value corresponding to the updated state parameter through the reward of the simulated core network, the reward of the simulated access network and the target network;

[0011] A loss function value is calculated based on the first value and the second value, the verification network and the target network are trained based on the loss function value, and the trained target network is used as a resource allocation model, and the resource allocation model is used to adjust the computing resource ratio of the network slice of the actual core network and the bandwidth ratio of the network slice of the actual access network.

[0012] In a possible implementation manner, adjusting the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network according to the target action set includes:

[0013] Generate a random number between 0 and 1;

[0014] If the random number is greater than or equal to the preset greed coefficient, the verification network is run based on the current state parameters, and based on the output result of the verification network, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are determined; if the random number is less than the preset greed coefficient, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are randomly generated;

[0015] Obtain an action value for each network slice corresponding to the first action index value from a preset action space, and obtain an action value for each network slice corresponding to the second action index value from the preset action space;

[0016] The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the simulated core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the simulated access network;

[0017] According to the computing resource adjustment ratio of each network slice in the simulated core network, the computing resource ratio of each network slice in the simulated core network is adjusted; and according to the bandwidth adjustment ratio of each network slice in the simulated access network, the bandwidth ratio of each network slice in the simulated access network is adjusted.

[0018] In a possible implementation, the current state parameters include a state matrix of the simulated core network and a state matrix of the simulated access network; and the step of running the verification network based on the current state parameters and determining a first action index value corresponding to the simulated core network and a second action index value corresponding to the simulated access network based on an output result of the verification network includes:

[0019] Converting the state matrix of the simulated core network into a state vector, and converting the state matrix of the simulated access network into a state vector;

[0020] Splicing the state vector of the simulated core network and the state vector of the simulated access network to obtain a spliced ​​vector;

[0021] Input the splicing vector into the input layer of the verification network, and process the splicing vector through the hidden layer of the verification network to obtain the action vector output by the output layer of the verification network;

[0022] Splitting the action vector into a first action vector of the simulated core network and a second action vector of the simulated access network;

[0023] The index value of the maximum value of each element included in the first action vector in the first action vector is used as the first action index value, and the index value of the maximum value of each element included in the second action vector in the second action vector is used as the second action index value.

[0024] In a possible implementation, the state matrix of the simulated access network includes instantaneous bandwidth usage, client blocking rate and bandwidth ratio of each network slice;

[0025] The state matrix of the simulated core network includes the client quantized delay, client delay rate and resource utilization rate of each network slice.

[0026] In a possible implementation, after obtaining updated state parameters of the simulated core network and the simulated access network, and calculating a reward of the simulated core network and a reward of the simulated access network based on the updated state parameters, the method further includes:

[0027] The current state parameters, the target action set, the updated state parameters, the reward of the simulated core network and the reward of the simulated access network are stored in an experience replay pool as a set of experience data, and the step of obtaining the current state parameters of the simulated core network and the simulated access network is returned until the training of the verification network and the target network is completed; wherein the target action set includes the action value of each network slice corresponding to the first action index value and the action value of each network slice corresponding to the second action index value;

[0028] The first value corresponding to the current state parameter and the target action set is calculated through the verification network, and the second value corresponding to the updated state parameter is calculated through the reward of the simulated core network, the reward of the simulated access network and the target network, including:

[0029] Acquire a preset number of groups of experience data from the experience replay pool;

[0030] Calculate the value corresponding to the current state parameter and the target action set in each set of experience data through the verification network, and use the average value of all the calculated values ​​as the first value;

[0031] Based on the updated state parameters in each set of experience data, the rewards of the simulated core network and the rewards of the simulated access network, the target network calculates the value corresponding to the updated state parameters in each set of experience values, and takes the average of all the calculated values ​​as the second value.

[0032] In a possible implementation, the training of the verification network and the target network based on the loss function value, and using the trained target network as a resource allocation model, includes:

[0033] Adjusting parameters of the verification network based on the loss function value;

[0034] Each time the parameters of the verification network are adjusted a preset number of times, the parameters of the verification network are copied to the target network;

[0035] When the training termination condition is met, the target network is used as the resource allocation model.

[0036] In a possible implementation, the calculating the reward of the simulated core network and the reward of the simulated access network based on the updated state parameters includes:

[0037] Calculate the product of the client delay rate of each network slice of the simulated core network and the first reward weight of the network slice, and use the sum of the products corresponding to each network slice as the reward of the simulated core network, where the sum of the first reward weights of each network slice is 1;

[0038] Calculate the product of the client blocking rate of each network slice of the simulated access network and the second reward weight of the network slice, and take the sum of the products corresponding to each network slice as the reward of the simulated access network. The sum of the second reward weights of each network slice is 1.

[0039] In a possible implementation, the simulated core network includes a plurality of simulation servers, each simulation server is used to simulate a computing network element in the core network, and each simulation server is used to process data packets of a service type.

[0040] In a second aspect, an embodiment of the present application provides a resource allocation method, comprising:

[0041] Obtain status parameters of the core network and access network;

[0042] Inputting the state parameters of the core network and the access network into a resource allocation model, and obtaining an action vector output by the resource allocation model; the resource allocation model is a model trained by the method described in the first aspect above;

[0043] Determine, based on the action vector, a computing resource adjustment ratio of each network slice in the core network and a bandwidth adjustment ratio of each network slice in the access network;

[0044] The computing resource ratio of each network slice in the core network is adjusted according to the computing resource adjustment ratio of each network slice in the core network, and the bandwidth ratio of each network slice in the access network is adjusted according to the bandwidth adjustment ratio of each network slice in the access network.

[0045] In a possible implementation manner, determining, based on the action vector, a computing resource adjustment ratio of each network slice in the core network and a bandwidth adjustment ratio of each network slice in the access network includes:

[0046] Splitting the action vector into a first action vector of the simulated core network and a second action vector of the simulated access network;

[0047] Using the index value of the maximum value of each element included in the first motion vector in the first motion vector as the first motion index value, and using the index value of the maximum value of each element included in the second motion vector in the second motion vector as the index value in the second motion vector;

[0048] Obtain an action value for each network slice corresponding to the first action index value from a preset action space, and obtain an action value for each network slice corresponding to the second action index value from the preset action space;

[0049] The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the access network.

[0050] In a third aspect, an embodiment of the present application provides a resource allocation model training device, comprising:

[0051] An acquisition module is used to obtain current status parameters of the simulated core network and the simulated access network;

[0052] An adjustment module, used to adjust the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network according to the target action set;

[0053] A calculation module, used to obtain updated state parameters of the simulated core network and the simulated access network, and calculate a reward of the simulated core network and a reward of the simulated access network based on the updated state parameters, wherein the reward of the simulated core network is used to represent a client delay rate, and the reward of the simulated access network is used to represent a client blocking rate;

[0054] The calculation module is further used to calculate the first value corresponding to the current state parameter and the target action set through the verification network, and calculate the second value corresponding to the updated state parameter through the reward of the simulated core network, the reward of the simulated access network and the target network;

[0055] A training module is used to calculate a loss function value based on the first value and the second value, train the verification network and the target network based on the loss function value, and use the trained target network as a resource allocation model, wherein the resource allocation model is used to adjust the computing resource ratio of the network slice of the actual core network and the bandwidth ratio of the network slice of the actual access network.

[0056] In a possible implementation, the adjustment module is specifically used to:

[0057] Generate a random number between 0 and 1;

[0058] If the random number is greater than or equal to the preset greed coefficient, the verification network is run based on the current state parameters, and based on the output result of the verification network, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are determined; if the random number is less than the preset greed coefficient, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are randomly generated;

[0059] Obtain an action value for each network slice corresponding to the first action index value from a preset action space, and obtain an action value for each network slice corresponding to the second action index value from the preset action space;

[0060] The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the simulated core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the simulated access network;

[0061] According to the computing resource adjustment ratio of each network slice in the simulated core network, the computing resource ratio of each network slice in the simulated core network is adjusted; and according to the bandwidth adjustment ratio of each network slice in the simulated access network, the bandwidth ratio of each network slice in the simulated access network is adjusted.

[0062] In a possible implementation, the current state parameters include a state matrix of the simulated core network and a state matrix of the simulated access network; and the adjustment module is specifically configured to:

[0063] Converting the state matrix of the simulated core network into a state vector, and converting the state matrix of the simulated access network into a state vector;

[0064] Splicing the state vector of the simulated core network and the state vector of the simulated access network to obtain a spliced ​​vector;

[0065] Input the splicing vector into the input layer of the verification network, and process the splicing vector through the hidden layer of the verification network to obtain the action vector output by the output layer of the verification network;

[0066] Splitting the action vector into a first action vector of the simulated core network and a second action vector of the simulated access network;

[0067] The index value of the maximum value of each element included in the first action vector in the first action vector is used as the first action index value, and the index value of the maximum value of each element included in the second action vector in the second action vector is used as the second action index value.

[0068] In a possible implementation manner, the device further includes:

[0069] A storage module, used to store the current state parameters, the target action set, the updated state parameters, the reward of the simulated core network and the reward of the simulated access network as a set of experience data into an experience replay pool, and trigger the acquisition module to execute the step of acquiring the current state parameters of the simulated core network and the simulated access network until the training module completes the training of the verification network and the target network; wherein the target action set includes the action value of each network slice corresponding to the first action index value and the action value of each network slice corresponding to the second action index value;

[0070] The adjustment module is specifically used for:

[0071] Acquire a preset number of groups of experience data from the experience replay pool;

[0072] Calculate the value corresponding to the current state parameter and the target action set in each set of experience data through the verification network, and use the average value of all the calculated values ​​as the first value;

[0073] Based on the updated state parameters in each set of experience data, the rewards of the simulated core network and the rewards of the simulated access network, the target network calculates the value corresponding to the updated state parameters in each set of experience values, and takes the average of all the calculated values ​​as the second value.

[0074] In a fourth aspect, an embodiment of the present application provides a resource allocation device, comprising:

[0075] An acquisition module, used to acquire status parameters of the core network and the access network;

[0076] The acquisition module is further used to input the state parameters of the core network and the access network into a resource allocation model to obtain the action vector output by the resource allocation model; the resource allocation model is a model trained by the device of the third aspect;

[0077] A determination module, configured to determine, based on the action vector, a computing resource adjustment ratio of each network slice in the core network and a bandwidth adjustment ratio of each network slice in the access network;

[0078] An adjustment module is used to adjust the computing resource ratio of each network slice in the core network according to the computing resource adjustment ratio of each network slice in the core network, and to adjust the bandwidth ratio of each network slice in the access network according to the bandwidth adjustment ratio of each network slice in the access network.

[0079] In a possible implementation, the determining module is specifically configured to:

[0080] Splitting the action vector into a first action vector of the core network and a second action vector of the access network;

[0081] Using the index value of the maximum value of each element included in the first motion vector in the first motion vector as the first motion index value, and using the index value of the maximum value of each element included in the second motion vector in the second motion vector as the index value in the second motion vector;

[0082] Obtain an action value for each network slice corresponding to the first action index value from a preset action space, and obtain an action value for each network slice corresponding to the second action index value from the preset action space;

[0083] The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the access network.

[0084] In a fifth aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0085] Memory, used to store computer programs;

[0086] The processor is used to execute the method steps described in the first aspect or the second aspect when executing the program stored in the memory.

[0087] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in the first aspect or the second aspect are implemented.

[0088] In a seventh aspect, an embodiment of the present application provides a computer program product, which prompts the processor to implement the method steps described in the first aspect or the second aspect above.

[0089] By adopting the embodiment of the present application, the current state parameters of the simulated core network and the simulated access network can be obtained, as well as the updated state parameters generated after adjusting the computing resource ratio and bandwidth ratio of each network slice according to the target action set, and the first value corresponding to the current state parameters and the target action set is calculated by the verification network, so that the first value can reflect the value generated by executing the target action set based on the current state parameters predicted by the verification network. The second value corresponding to the updated state parameters is calculated by the reward of the simulated core network, the reward of the simulated access network and the target network, so that the second value can represent the target value expected to be achieved under the updated state parameters. Then, the loss function value is calculated based on the first value and the second value, and the verification network and the target network are trained based on the loss function value, so that the resource allocation model obtained by training can adjust the computing resource ratio of each network slice of the core network and the bandwidth ratio of each network slice of the access network with the goal of improving value, thereby improving the quality of network service.

[0090] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0092] Figure 1 A flowchart of a resource allocation model training method provided in an embodiment of the present application;

[0093] Figure 2 A schematic diagram of the relationship between the 6G network environment and the resource allocation model provided in an embodiment of the present application;

[0094] Figure 3 An exemplary schematic diagram of a 6G network architecture provided in an embodiment of the present application;

[0095] Figure 4 An exemplary schematic diagram of a core network simulation solution provided in an embodiment of the present application;

[0096] Figure 5 An exemplary schematic diagram of a process for generating a data packet delay provided in an embodiment of the present application;

[0097] Figure 6 An exemplary schematic diagram of the working principle of the verification network provided in the embodiment of the present application;

[0098] Figure 7 An exemplary schematic diagram of a resource allocation model training process provided in an embodiment of the present application;

[0099] Figure 8 A schematic diagram of a flow chart of a simulated access network provided in an embodiment of the present application;

[0100] Fig. 9 A flowchart of a resource allocation method provided in an embodiment of the present application;

[0101] Fig.10 A schematic diagram of the structure of a resource allocation model training device provided in an embodiment of the present application;

[0102] Fig.11 A schematic diagram of the structure of a resource allocation device provided in an embodiment of the present application;

[0103] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0105] In order to improve the quality of network services, the present application embodiment provides a resource allocation model training method, such as Figure 1 As shown, the method includes:

[0106] S101. Obtain current status parameters of a simulated core network and a simulated access network.

[0107] Among them, the current state parameter of the simulated core network is used to represent the network state of the core network, and the current state parameter of the simulated access network is used to represent the network state of the access network.

[0108] S102. Adjust the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network according to the target action set.

[0109] Among them, the computing resource ratio of each network slice is the ratio between the computing resource amount of the network slice and the total computing resource amount of all network slices in the simulated core network. The computing resources of the network slice are the resources used to process data packets in the core network.

[0110] The bandwidth ratio of each network slice is the ratio between the bandwidth of the network slice and the total bandwidth of all network slices in the simulated access network.

[0111] S103. Obtain updated state parameters of the simulated core network and the simulated access network, and calculate a reward for the simulated core network and a reward for the simulated access network based on the updated state parameters.

[0112] Among them, the reward of the simulated core network is used to represent the client delay rate, and the reward of the simulated access network is used to represent the client blocking rate.

[0113] It can be understood that after adjusting the computing resource ratio of each network slice of the simulated core network and the bandwidth ratio of each network slice of the simulated access network, the network status of the simulated core network and the simulated access network will change, and the updated status parameters can be obtained at this time.

[0114] S104. Calculate the first value corresponding to the current state parameters and the target action set through the verification network, and calculate the second value corresponding to the updated state parameters through the reward of the simulated core network, the reward of the simulated access network and the target network.

[0115] S105. Calculate the loss function value based on the first value and the second value, train the verification network and the target network based on the loss function value, and use the trained target network as a resource allocation model. The resource allocation model is used to adjust the computing resource ratio of the network slice of the actual core network and the bandwidth ratio of the network slice of the actual access network.

[0116] By adopting the embodiment of the present application, the current state parameters of the simulated core network and the simulated access network can be obtained, as well as the updated state parameters generated after adjusting the computing resource ratio and bandwidth ratio of each network slice according to the target action set, and the first value corresponding to the current state parameters and the target action set is calculated by the verification network, so that the first value can reflect the value generated by executing the target action set based on the current state parameters predicted by the verification network. The second value corresponding to the updated state parameters is calculated by the reward of the simulated core network, the reward of the simulated access network and the target network, so that the second value can represent the target value expected to be achieved under the updated state parameters. Then, the loss function value is calculated based on the first value and the second value, and the verification network and the target network are trained based on the loss function value, so that the resource allocation model obtained by training can adjust the computing resource ratio of each network slice of the core network and the bandwidth ratio of each network slice of the access network with the goal of improving value, thereby improving the quality of network service.

[0117] The embodiment of the present application can simulate a 6G network environment, that is, simulate a 6G core network and an access network, and use a reinforcement learning model to allocate resources for the 6G network environment, such as Figure 2 As shown, Figure 2 A simulated 6G network environment including a simulated core network and a simulated access network, as well as an intelligent agent are shown. The intelligent agent is specifically a group deep Q network (DQN), and the group DQN includes the verification network and the target network in the above-mentioned embodiment.

[0118] After the state parameters and rewards corresponding to the simulated 6G network environment are input into the group DQN, the group DQN can calculate the action set to be executed in the simulated 6G network environment and feed the action set back to the simulated 6G network environment. After the simulated 6G network environment executes the action set, the updated state parameters and the rewards generated by executing the action set will be fed back to the group DQN again. In this way, the group DQN can be trained based on the state parameters and rewards of the simulated 6G network environment and the action set calculated by the group DON, so that the trained group DQN can accurately calculate the action set.

[0119] The 6G network structure in the embodiment of the present application is as follows Figure 3 As shown, the network includes a base station subsystem, a network subsystem, and a system support part, such as a security management device.

[0120] Figure 3 Two base stations are exemplarily shown, namely a base station in city A and a base station in city B. A terminal in city A can access the base station in city A, and a terminal in city B can access the base station in city B. Figure 3 Both base stations in the figure belong to the base station subsystem, and the base station subsystem belongs to the access network.

[0121] The base station can transmit the terminal's request to the core network through the bearer network. The core network belongs to the network subsystem. The core network is used to forward the call request or data request received from the access network to other networks through the backbone network. The core network and the backbone network communicate through the bearer network. The functional equipment of the core network can be deployed in the telecommunications room. Assume that there are 3 servers as computing resources in each core network functional equipment to process the channel allocation and calculation of uplink and downlink data.

[0122] The simulated core network in the embodiment of the present application includes multiple simulation servers, each simulation server is used to simulate a network element in the core network, and each simulation server is used to process data packets of a service type.

[0123] Taking three servers as computing resources of the core network as an example, the core network simulation solution provided by the embodiment of the present application is shown in Figure 4. Figure 4 The right side shows three servers, namely SERVER_0, SERVER_1 and SERVER_2. Each server has 8 CPUs, namely CPU0, CPU1...CPU7. The processing rate of each CPU is 25Mbps / s.

[0124] Figure 4 The top of the figure shows a processing cycle of the uplink and downlink bandwidths. A processing cycle includes 5 time slots, which can be functionally divided into uplink time slots and downlink time slots. In a processing cycle simulated in the embodiment of the present application, there are 3 uplink time slots, namely slot0, slot1 and slot2, and 2 downlink time slots, namely slot3 and slot4. The duration of each time slot is defined as 0.5ms. Therefore, within 1s, there are 2000 time slot units, totaling 400 uplink and downlink bandwidth processing cycles.

[0125] Figure 4 The figure below shows the processing process of the core network computing resources in a time slot. In a 0.5ms time slot, it can be seen from the total computing rate of the CPU that the data processing capacity of the CPU in a time slot is 12KBit / 0.5ms.

[0126] The embodiment of the present application simulates three types of services, namely Enhanced Mobile Broadband (eMBB), Massive Machine Type Communication (mMTC) and Ultra Reliable Low Latency Communication (URLLC). The data packets of each service type have different properties, such as the arrival rate of data packets (packets / slot), the size of data packets (Bit / packet) and the guaranteed processing rate of data packets (Bit / 5us).

[0127] Among them, the data packet arrival rate of the eMBB service type is 60 packets / slot, the data packet size is 5000 Bit / packet, and the guaranteed processing rate of the data packet is 50 Bit / 5us.

[0128] The data packet arrival rate of the mMTC service type is 120 packets / slot, the data packet size is 4000 Bit / packet, and the guaranteed data packet processing rate is 40 Bit / 5us.

[0129] The data packet arrival rate of the URLLC service type is 70 packets / slot, the data packet size is 4000 bits / packet, and the guaranteed data packet processing rate is 280 bits / 5us.

[0130] During the simulation, it is assumed that the computing resources of each business are independent and do not affect each other.

[0131] Figure 4 The three data packets (Packet_0, Packet_1 and Packet_2) of the eMBB service type arrive in sequence, that is, the computing resources of the eMBB type just continue to process the next data packet after processing one data packet.

[0132] Three data packets (Packet_0, Packet_1, and Packet_2) of the mMTC service type arrive at the same time, that is, the computing resources of the mMTC service type can process these three data packets within 30us.

[0133] The three data packets (Packet_0, Packet_1, and Packet_2) of the URLLC service type arrive out of order, that is, when the computing resources of the URLLC service type are processing one data packet, other data packets arrive.

[0134] The arrival rate of data packets of each service type is determined by the number of clients of that service type that are active (ie, connected). The more active clients there are, the faster the packet arrival rate is, and vice versa.

[0135] During the simulation, the size of the data packet is set based on experience, with a constant value as the size of each data packet. The data packet sizes of different services are different. Similarly, the guaranteed processing rate of the data packet is also set as a constant and is related to the specific service type. The guaranteed processing rate of the data packet limits the number of data packets that the computing resources of a service type can process simultaneously at one time. The specific calculation formula is:

[0136]

[0137] Wherein, N represents the number of data packets that can be processed simultaneously by the computing resources of this service type at one time, V represents the computing resource rate of this service type, and G represents the guaranteed processing rate of the data packets.

[0138] When there are enough computing resources for this service type to process all arriving data packets at the same time, the network is unobstructed; when the number of data packets to be processed is greater than N, the later arriving data packets will have a processing delay and can only be processed when sufficient computing resources are released.

[0139] Packet processing delay T d The calculation formula is as follows:

[0140] T d =T p -T a

[0141] Where T p Indicates the time when the data packet actually starts to be processed, T a Indicates the arrival time of the data packet.

[0142] The processing delay T of a data packet of a service type within a period of time db It can be obtained by the following formula:

[0143]

[0144] Where j represents the index of the data packet, n represents the number of data packets of this service type, and T dj represents the processing delay of data packet j.

[0145] The generation process of packet delay is as follows: Figure 5 As shown, Figure 5 The horizontal axis is time, in us, and the vertical axis is resource utilization, in %. Figure 5As can be seen from the figure, the arrival time of packet Packet_j is T aj , the processing time is T pj , then the processing delay T dj T pj With T aj The difference.

[0146] Based on the calculated processing delay of each data packet, the processing delay of all data packets in the current simulated network is T dt It can be obtained by the following formula:

[0147]

[0148] Where i represents the index of the business type into which the computing resources are divided, m represents the number of business types, and T dji Indicates the processing delay of the jth data packet of the i-th service type.

[0149] Based on the simulated core network and simulated access network introduced in the above embodiment, the above S103 can be specifically implemented as follows:

[0150] Calculate the product of the client latency rate of each network slice of the simulated core network and the first reward weight of the network slice, and use the sum of the products corresponding to each network slice as the reward of the simulated core network. The sum of the first reward weights of each network slice is 1;

[0151] Calculate the product of the client blocking rate of each network slice of the simulated access network and the second reward weight of the network slice, and take the sum of the products corresponding to each network slice as the reward of the simulated access network. The sum of the second reward weights of each network slice is 1.

[0152] The calculation formula for the reward of the simulated core network is:

[0153]

[0154] st.

[0155]

[0156] Where N is the number of network slices, i is the network slice index, β i is the first reward weight of network slice i, DR i is the client latency rate of network slice i.

[0157] The reward calculation formula for the simulated access network is:

[0158]

[0159] st.

[0160]

[0161] Where N is the number of network slices, i is the index of the network slice, and γ i is the second reward weight of network slice i, BR i is the client blocking rate of network slice i.

[0162] By using the embodiment of the present application, the reward of the simulated core network is calculated by the client delay rate, so that the calculated reward of the simulated core network can reflect the impact of adjusting the computing resource ratio of each network slice in the simulated core network on the network service quality. Similarly, by calculating the reward of the simulated access network by the client blocking rate, the calculated reward can accurately reflect the impact of adjusting the bandwidth ratio of each network slice in the simulated access network on the network service quality. Furthermore, the resource allocation model subsequently trained based on the rewards of the simulated core network and the rewards of the simulated access network can more accurately and dynamically adjust the computing resource ratio for each network slice of the actual core network and the bandwidth ratio for each network slice of the actual access network.

[0163] Based on the simulated core network and simulated access network introduced in the above embodiment, current state parameters of the simulated core network and the simulated access network can be obtained. The current state parameters include a state matrix of the simulated core network and a state matrix of the simulated access network.

[0164] The state matrix of the core network includes 3 groups of data, each group of data includes 3 elements, that is, the state matrix of the core network includes 9 variables in total. The state matrix of the access network has the same structure as the state matrix of the core network.

[0165] The first set of data in the core network state matrix is ​​C i , the second set of data is B i , the third set of data is R i , i represents the service type index of the network slice. i The sum of the processing delays of the data packets of network slice i divided by the total number of data packets, then divided by the average processing time of the data packets, and then quantized, that is, the client quantized delay. i R represents the proportion of delayed data packets processed by network slice i to the total data packets. i Indicates the resource utilization rate of network slice i. C1, C2 and C3 in the core network state matrix represent the client quantized delay of network slice 1, network slice 2 and network slice 3 respectively, B1, B2 and B3 represent the client delay rate of network slice 1, network slice 2 and network slice 3 respectively, and R1, R2 and R3 represent the resource utilization rate of network slice 1, network slice 2 and network slice 3 respectively.

[0166] The first set of data in the access network state matrix is ​​Ai , the second set of data is D i , the third set of data is T i , i represents the service type index of the network slice. i represents the sum of the processing delays of the data packets of network slice i, D i represents the proportion of delayed data packets processed by network slice i to the total data packets, T i Indicates the resource utilization rate of network slice i. A1, A2 and A3 in the access network state matrix represent the instantaneous bandwidth utilization rates of network slice 1, network slice 2 and network slice 3 respectively, D1, D2 and D3 represent the client blocking rates of network slice 1, network slice 2 and network slice 3 respectively, and T1, T2 and T3 represent the bandwidth ratios of network slice 1, network slice 2 and network slice 3 respectively.

[0167] Furthermore, in some embodiments of the present application, the above S102, adjusting the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network according to the target action set, can be specifically implemented as follows:

[0168] Step 1: Generate a random number between 0 and 1.

[0169] Step 2. If the random number is greater than or equal to the preset greedy coefficient, run the verification network based on the current state parameters, and determine the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network based on the output result of the verification network; if the random number is less than the preset greedy coefficient, randomly generate the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network.

[0170] The preset greed coefficient may be 0.9, that is, there is a 90% probability of randomly generating the first action index value and the second action index value, so that all action index values ​​have a chance to be used, so as to improve the performance of the resource allocation model obtained by training.

[0171] Step 3: Obtain the action value of each network slice corresponding to the first action index value from the preset action space, and obtain the action value of each network slice corresponding to the second action index value from the preset action space.

[0172] Among them, the set consisting of the action value of each network slice corresponding to the first action index value and the action value of each network slice corresponding to the second action index value is the target action set in the above embodiment.

[0173] The preset action space includes multiple groups of action values. Taking the simulated core network and the simulated access network each including 3 network slices as an example, the preset action space is shown in Table 1. Since the total amount of computing resources that can be allocated to each network slice by the simulated core network and the total bandwidth that can be allocated to each network slice by the simulated access network are fixed, the sum of each group of action values ​​is 0.

[0174] Table 1

[0175] Network Slicing 1 Network Slicing 2 Network Slicing 3 -1 2 -1 2 -1 -1 -2 1 1 0 0 0 1 -2 1 -1 -1 2 1 1 -2

[0176] Step 4: multiply the action value of each network slice corresponding to the first action index value by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the simulated core network, and multiply the action value of each network slice corresponding to the second action index value by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the simulated access network.

[0177] The action amplitude is a preset value, for example, the action amplitude may be 0.01.

[0178] Assuming that the first action index value is 0, the first set of action values ​​can be obtained from Table 1, namely -1, 2, -1. By multiplying by the action amplitude, the computing resource adjustment ratio of each network slice in the simulated core network can be obtained as -1%, 2%, and -1%.

[0179] Assuming that the second action index value is 1, the second set of action values ​​can be obtained from Table 1, namely 2, -1, -1. By multiplying by the action amplitude, the bandwidth adjustment ratio of each network slice in the simulated access network can be obtained as 2%, -1%, -1%.

[0180] Step 5: According to the bandwidth adjustment ratio of each network slice in the simulated core network, adjust the computing resource ratio of each network slice in the simulated core network, and according to the bandwidth adjustment ratio of each network slice in the simulated access network, adjust the bandwidth ratio of each network slice in the simulated access network.

[0181] Continuing with the example in the previous step, assuming that the computing resource adjustment ratio of each network slice in the simulated core network is -1%, 2%, and -1%, the computing resource ratio of network slice 1 in the simulated core network can be reduced by 1%, the computing resource ratio of network slice 2 can be increased by 2%, and the computing resource ratio of network slice 3 can be reduced by 1%.

[0182] Assuming that the bandwidth adjustment ratio of each network slice in the simulated access network is 2%, -1%, and -1%, the bandwidth ratio of network slice 1 in the simulated access network can be increased by 2%, the bandwidth ratio of network slice 2 can be reduced by 1%, and the bandwidth ratio of network slice 3 can be reduced by 1%.

[0183] It can be understood that after adjusting the computing resource ratio of each network slice of the simulated core network and the simulated access network, the network status of the simulated core network and the simulated access network will change.

[0184] Based on the above embodiments, the following Figure 6 The process of running the verification network based on the current state parameters and determining the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network based on the output result of the verification network is described. The verification network and the target network in the embodiment of the present application both belong to the grouped DQN, and the network structure of the verification network and the target network is the same, that is, the workflow of the target network can also refer to Figure 6 .

[0185] After obtaining the state matrix of the simulated core network and the state matrix of the simulated access network, the state matrix of the simulated core network is converted into a state vector, and the state matrix of the simulated access network is converted into a state vector. Then, the state vector of the simulated core network and the state vector of the simulated access network are spliced ​​to obtain a spliced ​​vector.

[0186] in, Figure 6 Taking the simulated core network and the simulated access network both including 3 network slices as an example, the state vector of the simulated core network and the state vector of the simulated access network are both one-dimensional vectors. The state vector of the simulated core network includes 9 elements, namely C1, C2, C3, B1, B2, B3, R1, R2, R3, and the state vector of the simulated access network also includes 9 elements, namely A1, A2, A3, D1, D2, D3, T1, T2, T3. After vertical splicing, a splicing vector including 18 elements can be obtained.

[0187] The concatenated vector can then be input into the input layer of the verification network, and the concatenated vector can be processed by the hidden layer of the verification network to obtain the action vector input into the output layer of the verification network.

[0188] Among them, the verification network includes 1 input layer, 2 hidden layers and 1 output layer.

[0189] The input layer includes 18 neurons, corresponding to 9 elements in the state vector of the simulated core network and 9 elements in the state vector of the simulated access network. Each hidden layer includes 18 neurons.

[0190] The output layer includes 14 neurons, corresponding to the action space of the core network and the access network, that is, the number of actions in the action space of the core network and the access network is 7 each.

[0191] Figure 6 This is only an example. In actual implementation, the number of neurons included in each layer of the verification network can be set based on actual network requirements.

[0192] The verification network has two goals. The goal for the simulated core network is to minimize the delay rate of data packets. The goal for the simulated access network is to adjust the bandwidth ratio for different network slices to maximize the bandwidth utilization rate of each network slice and maximize the SSR, which can be called the Quality of Experience (QoE). SSR is the ratio of the number of customers who meet the bandwidth, reliability and delay requirements to the number of all clients that need to be served.

[0193] Based on this, the objective function of the verification network for the simulation core network can be expressed as:

[0194]

[0195] st.

[0196]

[0197] CW i，t ≤CW i

[0198]

[0199] Where δ and ε are the weights of computing resources and packet delay rate in the objective function respectively. i,t is the computing resource used by network slice i at time t, CW i is the computing resource size of network slice i (computing resource size of network slice i = computing resource ratio of network slice i * total computing resources), and the sum of computing resource ratios of all network slices is 1; CR i is the packet delay rate of network slice i (packet delay rate of network slice i = number of packets with delay greater than 0 in network slice i / total number of packets arriving in network slice i), i is the network slice index, and N is the number of network slices; CW c The total amount of computing resources.

[0200] The objective function of the verification network for the simulated access network can be expressed as:

[0201]

[0202] st.

[0203]

[0204] CW i，t ≤BW i

[0205]

[0206] Where α and β are the weights of bandwidth and client blocking rate in the objective function. i,t is the bandwidth used by network slice i at time t, BW i is the bandwidth size of network slice i (bandwidth size of slice i = bandwidth ratio of slice i * total bandwidth resources), and the sum of bandwidth ratios of all network slices is 1; BR i is the blocking rate of network slice i (blocking rate of network slice i = number of clients that failed to connect in network slice i / number of clients that tried to connect to network slice i), i is the network slice index, and N is the number of network slices; BW a is the total amount of bandwidth resources.

[0207] Then, the objective function of the verification network can be expressed as:

[0208]

[0209] After obtaining the action vector output by the verification network, the action vector can be split into a first action vector of the simulated core network and a second action vector of the simulated access network, and then the index value of the maximum value of each element included in the first action vector in the first action vector is used as the first action index value, and the index value of the maximum value of each element included in the second action vector in the second action vector is used as the second action index value.

[0210] exist Figure 6 In , the 14-dimensional action vector output by the verification network includes P1-P14, which can be split into two 7-dimensional action vectors, that is, P1-P7 is used as the first action vector, and P8-P14 is used as the second action vector. Assuming that the maximum value in the first action vector is P3 and the maximum value in the second action vector is P14, it can be determined that the index value of P3 in P1-P7 is 2, that is, the first action index value corresponding to P3 is 2, and the index value of P14 in P8-P14 is 6, that is, the second action index value corresponding to P14 is 6. Among them, the index values ​​corresponding to the elements in each action vector start from 0, that is, the index value corresponding to the first element in the action vector is 0, the index value corresponding to the second element is 1, and so on.

[0211] It can be seen that the embodiments of the present application can adjust the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network based on the output results of the verification network, and can subsequently use the updated state parameters of the simulated core network and the simulated access network and the generated rewards to train the verification network and the target network, that is, the target action set obtained based on the verification network during the training process will act on the simulated core network and the simulated access network, and the data for training the verification network and the target network comes from the simulated core network and the simulated access network. In this way, the resource allocation model finally trained can accurately and reasonably adjust resources for each network slice of the actual core network and the actual access network, thereby improving the quality of network services.

[0212] After adjusting the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network, and calculating the reward of the simulated core network and the reward of the simulated access network based on the updated state parameters through the above S103, S101-S103 can be repeatedly executed until the training of the verification network and the target network is completed.

[0213] Each time S103 is executed, the current state parameters, the target action set, the bandwidth adjustment ratio of each network slice, the updated state parameters, the reward of the simulated core network and the reward of the simulated access network can be stored in the experience replay pool as a set of experience data. The target action set includes the action value of each network slice corresponding to the first action index value and the action value of each network slice corresponding to the second action index value.

[0214] Optionally, each set of experience data may also include state boundary conditions, that is, each set of experience data includes [s, a, s_, r, t], s is the current state parameter, a is the target action set, s_ is the updated state parameter, r is the reward of the simulated core network and the reward of the simulated access network, and t is the state boundary condition. When the simulated core network and the simulated access network reach the state boundary conditions, it is necessary to reinitialize the simulated core network and the simulated access network, and continue to execute S101-S103 after completing the initialization to generate experience data. The state boundary conditions may include the termination conditions of the simulated core network and the termination conditions of the simulated access network. The termination condition of the simulated core network is that the data packet delay rate of all network slices in the simulated core network is greater than a preset ratio, and the termination condition of the simulated access network is that the client blocking rate of all network slices in the simulated access network is greater than a preset ratio. As an example, the preset ratio can be 50%.

[0215] Furthermore, the above S104 can be specifically implemented as follows:

[0216] A preset number of groups of experience data are obtained from the experience replay pool, and the values ​​corresponding to the current state parameters and the target action set in each group of experience data are calculated through the verification network, and the average value of all the calculated values ​​is used as the first value; based on the updated state parameters in each group of experience data, the rewards of the simulated core network, and the rewards of the simulated access network, the value corresponding to the updated state parameters in each group of experience values ​​is calculated through the target network, and the average value of all the calculated values ​​is used as the second value.

[0217] As an example, the preset number may be 16.

[0218] The first value can be expressed as Q eval , the second value can be expressed as Q next , and then the loss function formula used in calculating the loss function value in the embodiment of the present application is:

[0219]

[0220] Among them, based on the empirical data [s, a, s_, r, t] generated in the embodiment of the present application, Q eval and Q next The specific calculation method of and the specific calculation method of the loss function can refer to the DQN algorithm in the related technology, which is not limited in the embodiments of the present application.

[0221] In some embodiments of the present application, each time the loss function value is calculated, the parameters of the verification network can be adjusted based on the calculated loss function value. After each preset number of parameter adjustments of the verification network, the parameters of the verification network can be copied to the target network. When the training termination condition is met, the target network is used as a resource allocation model.

[0222] As an example, the preset number of times can be 10 times. Specifically, each time the parameters of the verification network are adjusted, the number of iterations can be divided by 10 and the remainder can be taken. If the remainder is not 0, the target network is not updated; if the remainder is 0, the parameters of the verification network are copied to the target network.

[0223] The training termination condition may be that the number of iterations of the verification network reaches a preset number. Optionally, the training process may also be terminated manually.

[0224] For ease of understanding, combined Figure 7 The resource allocation model training method of the embodiment of the present application is introduced. Figure 7 The leftmost part is the simulation environment, which includes the simulated core network and the simulated access network. The verification network obtains s from the simulated core network and the simulated access network. c and a , and generate an action set a a and a c , action set aa and a c After being applied to the simulated core network and simulated access network, the simulated core network data (s c , a c , r c , s_ c ) and simulated access network data (s a , a a , r a , s_ a ) is stored in the experience replay pool as a set of experience data.

[0225] Among them, a a is the action value of each network slice of the simulated access network, a c is the action value of each network slice of the simulated core network. a Execute action a for the simulated access network a Previous state data, s_ a Execute action a for the simulated access network a The updated state data, r a Execute action a for the simulated core network a The reward generated later. c Execute action a for the simulated core network c Previous state data, s_ c Execute action a for the simulated core network c The updated state data, r c Execute action a for the simulated core network c The rewards generated later.

[0226] During the training process, small batches of data can be obtained from the experience playback pool, that is, a preset number of sets of experience data, and then the Q value (Q eval ), calculate the Q value of the target network through the target network (Q next ), then calculate the group DQN loss function based on the two Q values, update the parameters of the verification network based on the calculated loss function value, and copy the parameters of the verification network to the target network every time the parameters of the verification network are updated for a preset number of times. In this way, a resource allocation model that can accurately generate action values ​​for the actual core network and the actual access network can be trained.

[0227] It should also be noted that in the embodiments of the present application, any access network simulation method in the relevant technology can be used to implement the simulated access network in the embodiments of the present application. As an example, the simulated access network in the embodiments of the present application can be implemented using the simulation platform using the discrete event simulation library simpy and the KDTree search method.

[0228] Among them, simpy is a discrete event simulation framework based on standard python and process.

[0229] The access network can be simulated based on a two-dimensional plane, and the attributes of the simulated base station include the coordinates on the two-dimensional plane, the initial bandwidth, and the initial bandwidth ratio allocated to different network slices. The network slice type can be the same as the network slice type in the simulated core network introduced in the above embodiment. Different network slice types correspond to different bandwidth capacities, service quality levels, delay tolerances, uplink and downlink bandwidth rates, etc., so as to serve different services, such as enhanced mobile broadband including ultra-high-definition holographic video communications, enhanced cloud immersive VR online games, etc.

[0230] The attributes of the simulated client include coordinates on the two-dimensional plane and other special attributes, such as the movement mode, which means the movement state of the client, and the movement mode ratio, which means the proportion of different movement modes among all users. For example, if the number of clients is set to 100, 50 of these 100 clients are walking, which means the walking movement mode accounts for 50%.

[0231] In walking movement mode, two values ​​in the x and y directions are randomly generated for the client every second and applied to the client's two-dimensional coordinates to simulate the user's movement. The user's two-dimensional coordinates are updated at a frequency of seconds.

[0232] The simulation process of the simulated access network is as follows: Figure 8 As shown, the following steps are included:

[0233] S801. Read the YMAL configuration file. The YMAL configuration file includes configuration parameters such as base stations, clients, network slices, and mobility modes. YMAL is a data format.

[0234] S802. Use the python dependent library simpy to initialize the simulation environment.

[0235] S803, load the setting parameters, client flow mode, initialize the base station, client, and network slice. After the initialization is completed, start the simulation.

[0236] S804. Refresh the client status at every integer second.

[0237] The client has a usage frequency attribute, which is a value between the interval [0,1] and is used to control whether the client makes a connection request at the current moment.

[0238] S805: Determine whether the client requests to connect to the nearest base station. If yes, execute S806; if no, execute S813.

[0239] Specifically, the client's request status at the current moment can be determined. If the client is in the state of requesting a connection, the KDTree search method is used to find the five base stations closest to the client and within its coverage range, with the client's coordinates as the center of the circle. The base stations are sorted in ascending order of distance, and a request is made to connect to the base station closest to the client. If the client is in the state of not requesting a connection, it enters the non-connected state.

[0240] If the Euclidean distance between the client coordinates and the base station coordinates is less than the coverage radius of the base station, it can be determined that the client is within the coverage range of the base station.

[0241] S806: Determine whether the base station approves the connection request.

[0242] The conditions for the base station to determine whether to pass the connection request include:

[0243] (1) The services used by the client have corresponding network slices on the base station;

[0244] (2) The bandwidth of the network slice corresponding to the service used by the client on the base station is greater than the bandwidth required by the client.

[0245] If (1) and (2) are both true, it can be determined that the base station has passed the connection request, and then S807 is executed; otherwise, the client requests to connect to other base stations found in S805, that is, S816 is executed.

[0246] S807. The connection is successful, occupying the bandwidth of one network slice on the base station.

[0247] S808: The client enters the connection state.

[0248] S809. Refresh the status of all clients in the next integer second.

[0249] At this time, after refreshing the status of the client in the connected, unconnected and blocked states, S804 ( Figure 8 not shown).

[0250] In addition, for the client that is already in the connected state, the current connection state is kept unchanged, and S810 is continued to be executed.

[0251] S810. Determine whether the bandwidth of the network slice of the base station is fully used by the client.

[0252] If yes, execute S812; if no, execute S811, continue to occupy the bandwidth, and return to S808.

[0253] S812. Disconnect and release bandwidth.

[0254] S813: The client enters a disconnected state.

[0255] S814. In the next integer second, refresh the status of all clients again.

[0256] S815: Determine whether the client attempts to connect to the base station again.

[0257] If yes, return to S806; if no, return to S813.

[0258] S816. Initialize count value N, N=0.

[0259] S817. Find the base station closest to the other four base stations and set N=N+1.

[0260] S818. Determine whether N is less than or equal to 4.

[0261] If yes, execute S819; if no, execute S821.

[0262] S819: Determine whether the base station approves the connection request.

[0263] If yes, execute S820, the switch is successful, and execute S807; if no, execute S817.

[0264] S821. The client enters a blocking state.

[0265] S822. In the next integer second, refresh the status of all clients again.

[0266] S823: Determine whether the client attempts to connect to the base station again.

[0267] If yes, return to S806; if no, return to S821.

[0268] Among them, the base station in the simulated access network can communicate with the simulated core network, that is, the embodiment of the present application can simulate a simulated 6G network environment including a simulated access network and a simulated core network.

[0269] Based on the same inventive concept, the embodiment of the present application also provides a resource allocation method, such as Fig. 9 As shown, the method includes:

[0270] S901. Obtain status parameters of the core network and the access network.

[0271] Among them, the status parameters of the core network and access network include the core network status matrix and the access network status matrix. The core network status matrix includes the client quantized delay, client delay rate and resource utilization rate of each network slice in the core network; the access network status matrix includes the bandwidth utilization rate, client blocking rate and bandwidth ratio of each network slice in the access network.

[0272] S902: Input the status parameters of the core network and the access network into the resource allocation model, and obtain the action vector output by the resource allocation model.

[0273] The resource allocation model is a model trained by the resource allocation model training method introduced in the above embodiment.

[0274] S903. Determine the computing resource adjustment ratio of each network slice in the core network and the bandwidth adjustment ratio of each network slice in the access network based on the action vector.

[0275] S904. Adjust the computing resource ratio of each network slice in the core network according to the computing resource adjustment ratio of each network slice in the core network, and adjust the bandwidth ratio of each network slice in the access network according to the bandwidth adjustment ratio of each network slice in the access network.

[0276] By adopting this method, the current state parameters of the simulated core network and the simulated access network can be obtained, as well as the updated state parameters generated after adjusting the computing resource ratio and bandwidth ratio of each network slice according to the target action set, and the first value corresponding to the current state parameter and the target action set is calculated by the verification network, so that the first value can represent the value generated by executing the target action set based on the current state parameter predicted by the verification network. The second value corresponding to the updated state parameter is calculated by the reward of the simulated core network, the reward of the simulated access network and the target network, so that the second value can represent the target value expected to be achieved under the updated state parameter. Then, the loss function value is calculated based on the first value and the second value, and the verification network and the target network are trained based on the loss function value to obtain a resource allocation model. When applying the resource allocation model, the state parameters of the core network and the access network are input into the resource allocation model, so that the resource allocation model can output an action vector with the goal of improving the value, and then the computing resource ratio of each network slice of the core network and the bandwidth ratio of each network slice of the access network are adjusted based on the adjustment method indicated by the action vector, so as to improve the network service quality.

[0277] In the embodiment of the present application, S903, determining the computing resource adjustment ratio of each network slice in the core network and the bandwidth adjustment ratio of each network slice in the access network based on the action vector, can be specifically implemented as follows:

[0278] The action vector is split into a first action vector of the core network and a second action vector of the access network; the index value of the maximum value of each element included in the first action vector in the first action vector is used as the first action index value, and the index value of the maximum value of each element included in the second action vector in the second action vector is used as the index value in the second action vector.

[0279] Then, the action value of each network slice corresponding to the first action index value is obtained from the preset action space, and the action value of each network slice corresponding to the second action index value is obtained from the preset action space. The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the access network.

[0280] Among them, the preset action space includes the action values ​​of each network slice in the core network and the access network. The method of obtaining the action value from the preset action space can refer to the relevant description in the above embodiment, and will not be repeated here.

[0281] Corresponding to the above method embodiment, the present application embodiment also provides a resource allocation model training device, such as Fig.10 Said device comprises:

[0282] An acquisition module 1001 is used to acquire current status parameters of a simulated core network and a simulated access network;

[0283] An adjustment module 1002, configured to adjust the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network according to the target action set;

[0284] A calculation module 1003 is used to obtain updated state parameters of the simulated core network and the simulated access network, and calculate a reward of the simulated core network and a reward of the simulated access network based on the updated state parameters, wherein the reward of the simulated core network is used to represent a client delay rate, and the reward of the simulated access network is used to represent a client blocking rate;

[0285] The calculation module 1003 is further used to calculate the first value corresponding to the current state parameter and the target action set through the verification network, and calculate the second value corresponding to the updated state parameter through the reward of the simulated core network, the reward of the simulated access network and the target network;

[0286] The training module 1004 is used to calculate the loss function value based on the first value and the second value, train the verification network and the target network based on the loss function value, and use the trained target network as a resource allocation model. The resource allocation model is used to adjust the computing resource ratio of the network slice of the actual core network and the bandwidth ratio of the network slice of the actual access network.

[0287] Optionally, the adjustment module 1002 is specifically configured to:

[0288] Generate a random number between 0 and 1;

[0289] If the random number is greater than or equal to the preset greed coefficient, the verification network is run based on the current state parameters, and based on the output result of the verification network, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are determined; if the random number is less than the preset greed coefficient, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are randomly generated;

[0290] Obtain an action value for each network slice corresponding to a first action index value from a preset action space, and obtain an action value for each network slice corresponding to a second action index value from a preset action space;

[0291] The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the simulated core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the simulated access network;

[0292] According to the computing resource adjustment ratio of each network slice in the simulated core network, the computing resource ratio of each network slice in the simulated core network is adjusted, and according to the bandwidth adjustment ratio of each network slice in the simulated access network, the bandwidth ratio of each network slice in the simulated access network is adjusted.

[0293] Optionally, the current state parameters include a state matrix of a simulated core network and a state matrix of a simulated access network; the adjustment module 1002 is specifically configured to:

[0294] Converting the state matrix of the simulated core network into a state vector, and converting the state matrix of the simulated access network into a state vector;

[0295] Splicing the state vector of the simulated core network and the state vector of the simulated access network to obtain a spliced ​​vector;

[0296] The splicing vector is input into the input layer of the verification network, and the splicing vector is processed by the hidden layer of the verification network to obtain the action vector output by the output layer of the verification network;

[0297] Splitting the action vector into a first action vector for simulating a core network and a second action vector for simulating an access network;

[0298] The index value of the maximum value of each element included in the first action vector in the first action vector is used as the first action index value, and the index value of the maximum value of each element included in the second action vector in the second action vector is used as the second action index value.

[0299] Among them, the state matrix of the simulated access network includes the instantaneous bandwidth utilization rate, client blocking rate and bandwidth ratio of each network slice;

[0300] The state matrix of the simulated core network includes the client quantized delay, client delay rate and resource utilization of each network slice.

[0301] Optionally, the device further comprises:

[0302] A storage module is used to store the current state parameters, the target action set, the updated state parameters, the rewards of the simulated core network and the rewards of the simulated access network as a set of experience data into the experience replay pool, and trigger the acquisition module 1001 to execute the step of acquiring the current state parameters of the simulated core network and the simulated access network until the training module 1004 completes the training of the verification network and the target network; wherein the target action set includes the action value of each network slice corresponding to the first action index value and the action value of each network slice corresponding to the second action index value;

[0303] The adjustment module 1002 is specifically used for:

[0304] Get a preset number of sets of experience data from the experience replay pool;

[0305] The verification network calculates the value corresponding to the current state parameter and the target action set in each set of experience data, and takes the average value of all the calculated values ​​as the first value;

[0306] Based on the updated state parameters in each set of experience data, the rewards of the simulated core network and the rewards of the simulated access network, the target network calculates the value corresponding to the updated state parameters in each set of experience values, and takes the average of all the calculated values ​​as the second value.

[0307] Optionally, the training module 1004 is specifically configured to:

[0308] Adjust the parameters of the verification network based on the loss function value;

[0309] The parameters of each pair of verification networks are adjusted a preset number of times, and the parameters of the verification network are copied to the target network;

[0310] When the training termination condition is met, the target network is used as the resource allocation model.

[0311] Optionally, the calculation module 1003 is specifically configured to:

[0312] Calculate the product of the client latency rate of each network slice of the simulated core network and the first reward weight of the network slice, and use the sum of the products corresponding to each network slice as the reward of the simulated core network. The sum of the first reward weights of each network slice is 1;

[0313] Calculate the product of the client blocking rate of each network slice of the simulated access network and the second reward weight of the network slice, and take the sum of the products corresponding to each network slice as the reward of the simulated access network. The sum of the second reward weights of each network slice is 1.

[0314] In an embodiment of the present application, the simulated core network includes multiple simulation servers, each simulation server is used to simulate a computing network element in the simulated core network, and each simulation server is used to process data packets of a service type.

[0315] Corresponding to the above method embodiment, the present application embodiment also provides a resource allocation device, such as Fig.11 As shown, the device comprises:

[0316] An acquisition module 1101 is used to acquire status parameters of a core network and an access network;

[0317] The acquisition module 1101 is also used to input the state parameters of the core network and the access network into the resource allocation model to obtain the action vector output by the resource allocation model; the resource allocation model is Fig.10 The model obtained by training the device;

[0318] A determination module 1102, configured to determine a computing resource adjustment ratio of each network slice in the core network and a bandwidth adjustment ratio of each network slice in the access network based on the action vector;

[0319] The adjustment module 1103 is used to adjust the computing resource ratio of each network slice in the core network according to the computing resource adjustment ratio of each network slice in the core network, and adjust the bandwidth ratio of each network slice in the access network according to the bandwidth adjustment ratio of each network slice in the access network.

[0320] Optionally, the determination module 1102 is specifically configured to:

[0321] Splitting the action vector into a first action vector of the core network and a second action vector of the access network;

[0322] The index value of the maximum value of each element included in the first motion vector in the first motion vector is used as the first motion index value, and the index value of the maximum value of each element included in the second motion vector in the second motion vector is used as the index value in the second motion vector;

[0323] Obtain an action value for each network slice corresponding to a first action index value from a preset action space, and obtain an action value for each network slice corresponding to a second action index value from a preset action space;

[0324] The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the core network; the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the access network.

[0325] The present application also provides an electronic device, such as Fig.12 As shown, it includes a processor 1201, a communication interface 1202, a memory 1203 and a communication bus 1204, wherein the processor 1201, the communication interface 1202, and the memory 1203 communicate with each other through the communication bus 1204.

[0326] Memory 1203, used for storing computer programs;

[0327] The processor 1201 is used to implement the method steps of the resource allocation model training method or the method steps of the resource allocation method in the above method embodiment when executing the program stored in the memory 1203.

[0328] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0329] The communication interface is used for communication between the above electronic device and other devices.

[0330] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0331] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0332] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned resource allocation model training methods or resource allocation methods are implemented.

[0333] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any resource allocation model training method or resource allocation method in the above-mentioned embodiments.

[0334] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0335] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0336] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0337] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A resource allocation model training method, characterized in that: include: Obtain current status parameters of the simulated core network and simulated access network; Adjusting the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network according to the target action set; Obtain updated state parameters of the simulated core network and the simulated access network, and calculate a reward of the simulated core network and a reward of the simulated access network based on the updated state parameters, wherein the reward of the simulated core network is used to represent a client delay rate, and the reward of the simulated access network is used to represent a client blocking rate; Calculate the first value corresponding to the current state parameter and the target action set through the verification network, and calculate the second value corresponding to the updated state parameter through the reward of the simulated core network, the reward of the simulated access network and the target network; A loss function value is calculated based on the first value and the second value, the verification network and the target network are trained based on the loss function value, and the trained target network is used as a resource allocation model, and the resource allocation model is used to adjust the computing resource ratio of the network slice of the actual core network and the bandwidth ratio of the network slice of the actual access network.

2. The method according to claim 1, characterized in that The adjusting, according to the target action set, the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network includes: Generate a random number between 0 and 1; If the random number is greater than or equal to the preset greed coefficient, the verification network is run based on the current state parameters, and based on the output result of the verification network, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are determined; if the random number is less than the preset greed coefficient, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are randomly generated; Obtain an action value for each network slice corresponding to the first action index value from a preset action space, and obtain an action value for each network slice corresponding to the second action index value from the preset action space; The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the simulated core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the simulated access network; According to the computing resource adjustment ratio of each network slice in the simulated core network, the computing resource ratio of each network slice in the simulated core network is adjusted; and according to the bandwidth adjustment ratio of each network slice in the simulated access network, the bandwidth ratio of each network slice in the simulated access network is adjusted.

3. The method according to claim 2, characterized in that The current state parameters include a state matrix of the simulated core network and a state matrix of the simulated access network; the step of running the verification network based on the current state parameters and determining a first action index value corresponding to the simulated core network and a second action index value corresponding to the simulated access network based on an output result of the verification network includes: Converting the state matrix of the simulated core network into a state vector, and converting the state matrix of the simulated access network into a state vector; Splicing the state vector of the simulated core network and the state vector of the simulated access network to obtain a spliced ​​vector; Input the splicing vector into the input layer of the verification network, and process the splicing vector through the hidden layer of the verification network to obtain the action vector output by the output layer of the verification network; Splitting the action vector into a first action vector of the simulated core network and a second action vector of the simulated access network; The index value of the maximum value of each element included in the first action vector in the first action vector is used as the first action index value, and the index value of the maximum value of each element included in the second action vector in the second action vector is used as the second action index value.

4. The method according to claim 3, characterized in that The state matrix of the simulated access network includes the instantaneous bandwidth usage, client blocking rate and bandwidth ratio of each network slice; The state matrix of the simulated core network includes the client quantized delay, client delay rate and resource utilization rate of each network slice.

5. The method according to claim 2 or 3, characterized in that: After acquiring updated state parameters of the simulated core network and the simulated access network, and calculating the reward of the simulated core network and the reward of the simulated access network based on the updated state parameters, the method further includes: The current state parameters, the target action set, the updated state parameters, the reward of the simulated core network and the reward of the simulated access network are stored in an experience replay pool as a set of experience data, and the step of obtaining the current state parameters of the simulated core network and the simulated access network is returned until the training of the verification network and the target network is completed; wherein the target action set includes the action value of each network slice corresponding to the first action index value and the action value of each network slice corresponding to the second action index value; The first value corresponding to the current state parameter and the target action set is calculated through the verification network, and the second value corresponding to the updated state parameter is calculated through the reward of the simulated core network, the reward of the simulated access network and the target network, including: Acquire a preset number of groups of experience data from the experience replay pool; Calculate the value corresponding to the current state parameter and the target action set in each set of experience data through the verification network, and use the average value of all the calculated values ​​as the first value; Based on the updated state parameters in each set of experience data, the rewards of the simulated core network and the rewards of the simulated access network, the target network calculates the value corresponding to the updated state parameters in each set of experience values, and takes the average of all the calculated values ​​as the second value.

6. The method according to claim 5, characterized in that The step of training the verification network and the target network based on the loss function value and using the trained target network as a resource allocation model includes: Adjusting parameters of the verification network based on the loss function value; After each preset number of parameter adjustments to the verification network, the parameters of the verification network are copied to the target network; When the training termination condition is met, the target network is used as the resource allocation model.

7. The method according to claim 1, characterized in that The calculating the reward of the simulated core network and the reward of the simulated access network based on the updated state parameters includes: Calculate the product of the client delay rate of each network slice of the simulated core network and the first reward weight of the network slice, and use the sum of the products corresponding to each network slice as the reward of the simulated core network, where the sum of the first reward weights of each network slice is 1; Calculate the product of the client blocking rate of each network slice of the simulated access network and the second reward weight of the network slice, and take the sum of the products corresponding to each network slice as the reward of the simulated access network. The sum of the second reward weights of each network slice is 1.

8. The method according to claim 1, characterized in that The simulated core network includes a plurality of simulated servers, each of which is used to simulate a computing network element in the core network, and each of which is used to process data packets of a service type.

9. A resource allocation method, characterized in that: include: Obtain status parameters of the core network and access network; Inputting the state parameters of the core network and the access network into a resource allocation model, and obtaining an action vector output by the resource allocation model; the resource allocation model is a model trained by the method according to any one of claims 1 to 8; Determine, based on the action vector, a computing resource adjustment ratio of each network slice in the core network and a bandwidth adjustment ratio of each network slice in the access network; The computing resource ratio of each network slice in the core network is adjusted according to the computing resource adjustment ratio of each network slice in the core network, and the bandwidth ratio of each network slice in the access network is adjusted according to the bandwidth adjustment ratio of each network slice in the access network.

10. The method according to claim 9, characterized in that The determining, based on the action vector, a computing resource adjustment ratio of each network slice in the core network and a bandwidth adjustment ratio of each network slice in the access network includes: Splitting the action vector into a first action vector of the core network and a second action vector of the access network; Using the index value of the maximum value of each element included in the first motion vector in the first motion vector as the first motion index value, and using the index value of the maximum value of each element included in the second motion vector in the second motion vector as the index value in the second motion vector; Obtain an action value for each network slice corresponding to the first action index value from a preset action space, and obtain an action value for each network slice corresponding to the second action index value from the preset action space; The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the access network.

11. A resource allocation model training device, characterized in that: include: An acquisition module is used to obtain current status parameters of the simulated core network and the simulated access network; An adjustment module, used to adjust the computing resource ratio of each network slice in the simulated core network and the bandwidth ratio of each network slice in the simulated access network according to the target action set; A calculation module, used to obtain updated state parameters of the simulated core network and the simulated access network, and calculate a reward of the simulated core network and a reward of the simulated access network based on the updated state parameters, wherein the reward of the simulated core network is used to represent a client delay rate, and the reward of the simulated access network is used to represent a client blocking rate; The calculation module is further used to calculate the first value corresponding to the current state parameter and the target action set through the verification network, and calculate the second value corresponding to the updated state parameter through the reward of the simulated core network, the reward of the simulated access network and the target network; A training module is used to calculate a loss function value based on the first value and the second value, train the verification network and the target network based on the loss function value, and use the trained target network as a resource allocation model, wherein the resource allocation model is used to adjust the computing resource ratio of the network slice of the actual core network and the bandwidth ratio of the network slice of the actual access network.

12. The device according to claim 11, characterized in that The adjustment module is specifically used for: Generate a random number between 0 and 1; If the random number is greater than or equal to the preset greed coefficient, the verification network is run based on the current state parameters, and based on the output result of the verification network, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are determined; if the random number is less than the preset greed coefficient, the first action index value corresponding to the simulated core network and the second action index value corresponding to the simulated access network are randomly generated; Obtain an action value for each network slice corresponding to the first action index value from a preset action space, and obtain an action value for each network slice corresponding to the second action index value from the preset action space; The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the simulated core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the simulated access network; According to the computing resource adjustment ratio of each network slice in the simulated core network, the computing resource ratio of each network slice in the simulated core network is adjusted; and according to the bandwidth adjustment ratio of each network slice in the simulated access network, the bandwidth ratio of each network slice in the simulated access network is adjusted.

13. The device according to claim 12, characterized in that The current state parameters include a state matrix of the simulated core network and a state matrix of the simulated access network; the adjustment module is specifically used to: Converting the state matrix of the simulated core network into a state vector, and converting the state matrix of the simulated access network into a state vector; Splicing the state vector of the simulated core network and the state vector of the simulated access network to obtain a spliced ​​vector; Input the splicing vector into the input layer of the verification network, and process the splicing vector through the hidden layer of the verification network to obtain the action vector output by the output layer of the verification network; Splitting the action vector into a first action vector of the simulated core network and a second action vector of the simulated access network; The index value of the maximum value of each element included in the first action vector in the first action vector is used as the first action index value, and the index value of the maximum value of each element included in the second action vector in the second action vector is used as the second action index value.

14. The device according to claim 13, characterized in that The device also includes: A storage module, used to store the current state parameters, the target action set, the updated state parameters, the reward of the simulated core network and the reward of the simulated access network as a set of experience data into an experience replay pool, and trigger the acquisition module to execute the step of acquiring the current state parameters of the simulated core network and the simulated access network until the training module completes the training of the verification network and the target network; wherein the target action set includes the action value of each network slice corresponding to the first action index value and the action value of each network slice corresponding to the second action index value; The adjustment module is specifically used for: Acquire a preset number of groups of experience data from the experience replay pool; Calculate the value corresponding to the current state parameter and the target action set in each set of experience data through the verification network, and use the average value of all the calculated values ​​as the first value; Based on the updated state parameters in each set of experience data, the rewards of the simulated core network and the rewards of the simulated access network, the target network calculates the value corresponding to the updated state parameters in each set of experience values, and takes the average of all the calculated values ​​as the second value.

15. A resource allocation device, characterized in that: include: An acquisition module, used to acquire status parameters of the core network and the access network; The acquisition module is further used to input the state parameters of the core network and the access network into a resource allocation model to obtain an action vector output by the resource allocation model; the resource allocation model is a model trained by the device according to any one of claims 11 to 14; A determination module, configured to determine, based on the action vector, a computing resource adjustment ratio of each network slice in the core network and a bandwidth adjustment ratio of each network slice in the access network; An adjustment module is used to adjust the computing resource ratio of each network slice in the core network according to the computing resource adjustment ratio of each network slice in the core network, and to adjust the bandwidth ratio of each network slice in the access network according to the bandwidth adjustment ratio of each network slice in the access network.

16. The device according to claim 15, characterized in that The determination module is specifically used for: Splitting the action vector into a first action vector of the core network and a second action vector of the access network; Using the index value of the maximum value of each element included in the first motion vector in the first motion vector as the first motion index value, and using the index value of the maximum value of each element included in the second motion vector in the second motion vector as the index value in the second motion vector; Obtain an action value for each network slice corresponding to the first action index value from a preset action space, and obtain an action value for each network slice corresponding to the second action index value from the preset action space; The action value of each network slice corresponding to the first action index value is multiplied by the action amplitude to obtain the computing resource adjustment ratio of each network slice in the core network, and the action value of each network slice corresponding to the second action index value is multiplied by the action amplitude to obtain the bandwidth adjustment ratio of each network slice in the access network.

Images