Traffic migration-based model training method and device, equipment and medium

Abstract

The application discloses a model training method and device based on traffic migration, equipment and medium. The method comprises the following steps: acquiring a user traffic set composed of multiple user traffics; traversing each user traffic in the user traffic set, randomly sampling multiple traffic migration strategies based on a first load state of a virtual network function (VNF) instance in a current network, obtaining a target traffic migration strategy corresponding to each user traffic; calculating a reward value corresponding to the execution of the target traffic migration strategy, and determining a second load state corresponding to the VNF instance; updating model parameters of an initial traffic migration model based on the first load state, the second load state, the target traffic migration strategy and the reward value, and obtaining a target traffic migration model. The scheme provided by the application can reduce the migration cost of traffic migration while ensuring the network performance.

Description

Technical Field

[0001] This application belongs to the field of network function virtualization technology, and in particular relates to a model training method, apparatus, device and medium based on traffic migration. Background Technology

[0002] Typically, user traffic in a network is dynamic. To ensure network performance, in practical applications, it is necessary to perform traffic migration for user traffic in the network.

[0003] In related technologies, a fixed threshold method is typically used to determine when to perform traffic migration operations. However, the fixed threshold method requires manually setting a reasonable threshold range. If the threshold range is set too tightly, traffic migration will be triggered even with slight changes in user traffic, and frequent traffic migration operations will undoubtedly incur significant migration costs. If the threshold range is set too loosely, changes in user traffic cannot be detected in a timely manner, resulting in delayed traffic migration and reduced network performance. Summary of the Invention

[0004] This application provides a model training method, apparatus, device, and medium based on traffic migration, which can reduce the migration cost of traffic migration while ensuring network performance.

[0005] In a first aspect, embodiments of this application provide a model training method based on traffic migration. The method includes: acquiring a set of user traffic consisting of multiple user traffic flows; traversing each user traffic flow in the set, and randomly sampling multiple traffic migration strategies based on a first load state of a Virtual Network Function (VNF) instance in the current network to obtain a target traffic migration strategy corresponding to each user traffic flow, wherein the first load state is the load state of the VNF instance before executing the target traffic migration strategy; calculating a reward value corresponding to executing the target traffic migration strategy, and determining a second load state corresponding to the VNF instance, wherein the second load state is the load state of the VNF instance after executing the target traffic migration strategy; the reward value is used to characterize the VNF instance state penalty cost and traffic migration cost corresponding to executing the target traffic migration strategy; and updating the model parameters of an initial traffic migration model based on the first load state, the second load state, the target traffic migration strategy, and the reward value to obtain a target traffic migration model.

[0006] Secondly, embodiments of this application provide a model training apparatus based on traffic migration. The apparatus includes: a traffic acquisition module for acquiring a set of user traffic consisting of multiple user traffic flows; a policy determination module for traversing each user traffic flow in the set and randomly sampling multiple traffic migration policies based on a first load state of a Virtual Network Function (VNF) instance in the current network to obtain a target traffic migration policy corresponding to each user traffic flow, wherein the first load state is the load state of the VNF instance before executing the target traffic migration policy; a state determination module for calculating the reward value corresponding to executing the target traffic migration policy and determining a second load state corresponding to the VNF instance, wherein the second load state is the load state of the VNF instance after executing the target traffic migration policy; the reward value is used to characterize the VNF instance state penalty cost and traffic migration cost corresponding to executing the target traffic migration policy; and a parameter update module for updating the model parameters of an initial traffic migration model based on the first load state, the second load state, the target traffic migration policy, and the reward value to obtain a target traffic migration model.

[0007] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the model training method based on traffic migration as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the model training method based on traffic migration as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the traffic migration-based model training method as described in the first aspect.

[0010] As can be seen from the above, this application constructs a target traffic migration model. This target traffic migration model can determine the traffic migration strategy at the current moment based solely on the load status of VNF instances and user traffic in the current network, without setting a threshold. This avoids the problems of reduced network performance and increased migration costs caused by unreasonable threshold settings.

[0011] Furthermore, in this application, the target traffic migration model is determined based on the load state before and after migration, the migration strategy, and the reward value representing the migration cost. The load state after migration represents the migration effect corresponding to the execution of the migration strategy. Therefore, this application comprehensively considers both the migration cost and the migration effect during the training of the target traffic migration model. Thus, when using the trained target traffic migration model for traffic migration, the model can determine the target traffic migration strategy to be executed at the current moment based on the reward value corresponding to different traffic migration strategies. This effectively avoids the unreasonable phenomenon in related technologies where the benefits of network performance improvement are lower than the overhead of traffic migration operations, achieving the goal of reducing traffic migration costs while ensuring migration effectiveness. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic flowchart of a model training method based on traffic migration provided in one embodiment of this application;

[0014] Figure 2 This is a schematic diagram of the training framework of a target traffic migration model provided in one embodiment of this application;

[0015] Figure 3(a) is a schematic diagram of the maximum VNF instance load ratio optimization result in an Abilene topology provided by an embodiment of this application;

[0016] Figure 3(b) is a schematic diagram of the maximum VNF instance load ratio optimization results in a Geant topology provided by an embodiment of this application;

[0017] Figure 3(c) is a schematic diagram of the minimum VNF instance load ratio optimization result in an Abilene topology provided by an embodiment of this application;

[0018] Figure 3(d) is a schematic diagram of the minimum VNF instance load ratio optimization result in a Geant topology provided by an embodiment of this application;

[0019] Figure 4(a) is a schematic diagram of the cumulative probability distribution of the VNF instance load variance in an Abilene topology provided in an embodiment of this application;

[0020] Figure 4(b) is a schematic diagram of the cumulative probability distribution of the VNF instance load variance in a Geant topology provided in an embodiment of this application;

[0021] Figure 5(a) is a schematic diagram of the cumulative traffic migration cost in an Abilene topology provided in an embodiment of this application;

[0022] Figure 5(b) is a schematic diagram of the cumulative traffic migration cost in a Geant topology provided in an embodiment of this application;

[0023] Figure 6(a) is a schematic diagram illustrating the number of times the traffic migration strategy is executed in an Abilene topology provided in an embodiment of this application;

[0024] Figure 6(b) is a schematic diagram illustrating the number of times the traffic migration strategy is executed in a Geant topology provided in an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of the structure of a model training device based on traffic migration provided in another embodiment of this application;

[0026] Figure 8 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0027] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0028] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0029] The solution provided in this application can be applied to the technical field of Network Function Virtualization (NFV). Network service providers can use NFV technology to implement network functions in a software-defined manner, thereby replacing network functions implemented based on dedicated hardware devices. The following explains the relevant terminology in the field of Network Function Virtualization technology.

[0030] (1) VNF (Virtual Network Function) is a network function implemented based on NFV technology. Among them, the virtual network function implemented in software can run on general x86 architecture devices.

[0031] (2) VNFI (Virtual Network Function Instance) is a container used to host VNFs. Typically, VNFs run in different containers (e.g., virtual machines or Docker containers) on the physical device to avoid interference between different VNFs on the same device.

[0032] (3) SFC (Service Function Chain) is a set of chain sequences of VNFs. With the widespread application of NFV technology, the provision of network services has become more flexible, and the deployment and operation costs of network services have been greatly reduced. Network services are usually composed of several different VNFs. User traffic needs to pass through the VNFs that constitute the network service in a specified order to obtain a specific service, and the set of chain sequences of these VNFs is the service chain.

[0033] (4) SDN (Software Defined Network) is a routing method that allows network operators to flexibly manage user traffic in the network. By issuing specified routing rules to routing and forwarding devices in the network, specific routing objectives can be achieved, thereby optimizing the overall network performance (e.g., reducing traffic transmission latency, avoiding network congestion, etc.).

[0034] Before explaining the solution provided in this application, the application background of the solution provided in this application will be explained first.

[0035] Because user traffic in a network is dynamic, the load on network function instances (NFV) also changes in response to these changes. Typically, NFV instances have an ideal operating load range. If the actual load on an NFV instance exceeds the upper limit of this ideal range, issues such as packet loss or excessive processing latency may occur, leading to a degraded quality of service (QoS) for users. Conversely, if the actual load on an NFV instance is below the lower limit of its ideal operating load range, the network computing resources it utilizes are not fully utilized, increasing operational costs.

[0036] Considering the dynamic changes in network traffic, networks supporting NFV technology need to take corresponding adjustments to maintain network performance. In related technologies, traffic migration strategies are commonly used to mitigate the network performance degradation caused by dynamic traffic changes. For example, when increased traffic causes some VNF instances in the service chain to become overloaded, some traffic passing through these VNF instances needs to be migrated to other unloaded VNF instances, or some traffic can be directly migrated to newly started VNF instances. The operations corresponding to traffic migration strategies typically include VNF state transitions, packet redirection, and traffic routing rule updates.

[0037] In related technologies, a fixed threshold method is typically used to determine when to execute traffic migration strategies. Specifically, a fixed threshold range is set for the workload of a VNF instance. If the cumulative traffic passing through a VNF instance changes, causing its load to exceed the upper limit of this threshold range, the VNF instance is considered overloaded. In this case, a scaling-up migration strategy is executed to migrate some traffic from the overloaded VNF instance to other VNF instances until the load of the current VNF ​​instance returns to the normal threshold range. Conversely, if the amount of traffic flowing through a VNF instance is too small, its load may fall below the lower limit of the threshold range, indicating a light load. In this case, a scaling-down migration strategy is executed to migrate all traffic from the lightly loaded VNF instance to other VNF instances and shut down the lightly loaded VNF instance. Furthermore, if the traffic distribution in the network causes the load variance among VNF instances to exceed a pre-set threshold, the VNF instance load is considered unbalanced. In this case, user traffic needs to be migrated from high-load VNF instances to low-load VNF instances until the load among VNF instances returns to equilibrium.

[0038] While the fixed threshold method described above can determine when to execute traffic migration policies, and is simple and intuitive, setting this threshold range requires the network service provider to understand the network status. If the threshold range is set too tightly, the traffic migration policy will be triggered even with minor changes in user traffic, leading to frequent triggering of the traffic migration policy. Although this method can improve network performance, frequent triggering of the traffic migration policy will also increase the migration cost. Conversely, if the threshold range is set too loosely, it may not be able to meet the load requirements of VNF instances in a timely manner. Furthermore, even if a suitable threshold range can be determined for traffic changes within a network, the previously set threshold range will no longer be applicable when the traffic change pattern within the network changes significantly. For example, if the traffic rate fluctuates greatly in the changed traffic change pattern, the load of the VNF instance will jump back and forth between the upper and lower limits of the previously set threshold range, resulting in repeated triggering of the traffic migration policy.

[0039] In related technologies, a cooling-off period can be used to avoid frequent triggering of traffic migration strategies, but setting an appropriate cooling-off period is also a challenge. Furthermore, schemes based on fixed thresholds to determine the timing of traffic migration strategy execution often ignore the costs incurred by the traffic migration strategy. This results in the network performance improvement benefits after executing the traffic migration strategy being lower than the overhead, meaning that related technologies cannot balance migration costs and migration effects.

[0040] As can be seen from the above, in related technologies, when using the fixed threshold method for traffic migration, it is impossible to balance the cost of traffic migration and the network performance after migration.

[0041] To address the problems existing in the aforementioned related technologies, embodiments of this application provide a model training method, apparatus, device, and medium based on traffic migration. The model training method based on traffic migration provided in this application embodiment will be described first below.

[0042] Figure 1 A schematic flowchart of a model training method based on traffic migration provided in one embodiment of this application is shown. Figure 1 As shown, the method includes the following steps:

[0043] Step S101: Obtain a user traffic set consisting of multiple user traffic streams.

[0044] In step S101, user traffic may include, but is not limited to, personal traffic and enterprise traffic. Each piece of user traffic has its corresponding routing path. For example, personal traffic accessing a webpage may have a routing path that passes through network functions such as an address mapping unit, a firewall, and an anomaly detection unit. Each of these network functions is a network function, and the three network functions are connected in series to form a network service, i.e., a service chain.

[0045] Step S102: Traverse each user traffic in the user traffic set, and randomly sample multiple traffic migration strategies based on the first load state of the Virtual Network Function (VNF) instance in the current network to obtain the target traffic migration strategy corresponding to each user traffic.

[0046] In step S102, the first load state is the load state of the VNF instance before the target traffic migration strategy is executed; that is, the first load state is the load state of the VNF instance before traffic migration. The load state of the VNF instance includes, but is not limited to, normal state, overload state, light load state, and unbalanced load state.

[0047] In addition, in step S102, multiple traffic migration strategies include, but are not limited to, no-operation, expansion migration, shrinkage migration, and load balancing migration. Specifically, no-operation means no operation is performed, i.e., no traffic migration occurs; the expansion migration strategy is used to migrate a suitable amount of traffic from overloaded VNF instances to other VNF instances when some VNF instances in the SFC are overloaded, until the load demand of the original overloaded VNF instances is met; the shrinkage migration strategy is used to migrate all traffic flowing through lightly loaded VNF instances to other VNF instances and shut down the original lightly loaded VNF instances when some VNF instances in the SFC are lightly loaded; the load balancing migration strategy is used to migrate a suitable amount of traffic from VNF instances with high loads to VNF instances with low loads when the load of VNF instances in different SFCs is unbalanced, until the load imbalance between VNF instances is effectively alleviated.

[0048] In one example, for each user traffic flow, the load state of the VNF instances in the network is first determined. Then, multiple traffic migration strategies are randomly sampled based on the probability distribution corresponding to the load state of the VNF instances to obtain the target traffic migration strategy for that user traffic. That is, during the training phase of the traffic migration model, random sampling is used to select the traffic migration strategy to avoid the problem of targeted selection leading to directional training results and affecting the training accuracy of the traffic migration model. Furthermore, during the application phase, when performing traffic migration, the traffic migration model can select the target traffic migration strategy based on the degree of matching between each traffic migration strategy and the load state of the VNF instances in the network, thereby minimizing traffic migration costs while ensuring effective traffic migration.

[0049] Step S103: Calculate the reward value corresponding to the execution of the target traffic migration strategy and determine the second load state corresponding to the VNF instance.

[0050] In step S103, the second load state is the load state of the VNF instance after the target traffic migration policy is executed, that is, the second load state is the load state of the VNF instance after traffic migration.

[0051] Furthermore, in step S103, the reward value is used to characterize the VNF instance state penalty cost and traffic migration cost corresponding to the execution of the target traffic migration strategy. That is, the reward value of the target traffic migration strategy consists of the VNF instance state penalty cost and the traffic migration cost. The VNF instance state penalty cost characterizes the cost incurred when the load of the VNF instance is outside the reasonable target load range. Depending on the load state of the VNF instance, the VNF instance state penalty cost can be composed of three items: instance overload penalty cost, instance light load penalty cost, and instance variance penalty cost. The traffic migration cost characterizes the cost incurred during the traffic migration process.

[0052] As an example, after determining the target traffic migration strategy, it is executed. At this point, the instance overload penalty cost, instance light load penalty cost, instance variance penalty cost, and traffic migration cost can be calculated. The sum of these four items yields the reward value corresponding to the execution of the target traffic migration strategy. After the target traffic migration strategy is completed, the load state of the VNF instance can be determined for use in the subsequent training of the target traffic migration model.

[0053] Step S104: Based on the first load state, the second load state, the target traffic migration strategy, and the reward value, update the model parameters of the initial traffic migration model to obtain the target traffic migration model.

[0054] In step S104, the first load state, the second load state, the target traffic migration strategy, and the reward value can form a quadruple, which represents the state transition information of the VNF instance corresponding to the target traffic migration strategy.

[0055] As an example, in each training round of the initial traffic migration model, the state transition information corresponding to all user traffic in the user traffic set can be obtained. Then, based on this state transition information, the initial traffic migration model is updated to obtain the model parameters of the initial traffic migration model in the current training round. The model parameters obtained in this round are then used as the model parameters of the initial traffic migration model in the next training round. After the initial traffic migration model has reached the target number of training rounds, training ends, and the target traffic migration model is obtained.

[0056] Based on the scheme defined in steps S101 to S104 above, it can be seen that this application constructs a target traffic migration model. This target traffic migration model can determine the traffic migration strategy at the current moment based solely on the load status of VNF instances and user traffic in the current network, without setting a threshold. This avoids the problems of reduced network performance and increased migration costs caused by unreasonable threshold settings.

[0057] Furthermore, in this application, the target traffic migration model is determined based on the load state before and after migration, the migration strategy, and the reward value representing the migration cost. The load state after migration represents the migration effect corresponding to the execution of the migration strategy. Therefore, this application comprehensively considers both the migration cost and the migration effect during the training of the target traffic migration model. Thus, when using the trained target traffic migration model for traffic migration, the model can determine the target traffic migration strategy to be executed at the current moment based on the reward value corresponding to different traffic migration strategies. This effectively avoids the unreasonable phenomenon in related technologies where the benefits of network performance improvement are lower than the overhead incurred by the traffic migration strategy, achieving the goal of reducing traffic migration costs while ensuring migration effectiveness.

[0058] Based on the scheme defined in steps S101 to S104 above, it can be understood that the traffic migration model trained using the scheme provided in this application can select an appropriate traffic migration strategy according to the load status of VNF instances in the network, so as to realize the migration of the load of VNF instances, thereby improving network performance while reducing the cost of traffic migration. Therefore, the scheme provided in this application can be simplified to an optimization problem of how to select and execute an appropriate traffic migration strategy at the right time within a time period T to minimize the reward value of VNF instances. This optimization problem can be transformed into the following equation:

[0059]

[0060] In formula (1), P O (t) represents the instance overload penalty cost at time t; P U (t) represents the instance light load penalty cost at time t; P V (t) represents the instance variance penalty cost at time t; a(o, t) represents the target traffic migration strategy with identifier 'o' adopted at time t; C M (a(o,t)) represents the traffic migration cost corresponding to the target traffic migration strategy.

[0061] The constraints corresponding to formula (1) are as follows:

[0062]

[0063]

[0064]

[0065] In formulas (2)-(3), DT is the maximum migration cost that can be tolerated for a single migration; L i (t) represents the load of VNF instance i at time t; T represents the system's operating cycle. Formula (2) limits the time cost of executing the traffic migration strategy to avoid selecting a high-time-cost traffic migration strategy; Formula (3) ensures the load of the VNF instance is within an acceptable range; Formula (2) limits that only one traffic migration strategy can be executed at each time t, and the identifier o of the selected traffic migration strategy is within a preset range. As an example, in this application, there can be four traffic migration strategies, with corresponding identifiers o of 0, 1, 2, and 3, where 0 represents no operation, 1 represents a scaling-up migration strategy, 2 represents a scaling-down migration strategy, and 3 represents a load balancing migration strategy.

[0066] It should be noted that the optimization problem shown in equations (1)-(4) can be transformed into a Markov Decision Process (MDP) and the decision-making for traffic migration operations can be completed based on the Proximal Policy Optimization (PPO) algorithm. The training framework of the target traffic migration model provided in this application is as follows: Figure 2 As shown, in Figure 2In this process, the PPO agent can observe the load status of each VNF instance in the cloud network, select a traffic migration strategy from the action space based on the policy, and execute the selected traffic migration strategy. After the selected traffic migration strategy is executed, the load of the VNF instances in the current network will change and enter the next state. Then, the reward value of the selected action is calculated based on the change in the VNF instance load and the reward function.

[0067] In one example, the state space is used to monitor load state changes of VNF instances in the network, where the state space of the MDP at time t can be represented as s(t)=[L1(t),L2(t),…,L… I The load on a VNF instance changes dynamically with traffic fluctuations and the execution of traffic migration strategies.

[0068] In one example, the action space contains policies or rules for adjusting the network environment. In this application, the action space of MDP at time t contains four types of traffic migration policies: expansion migration policy, shrinkage migration policy, load balancing migration policy, and no-operation migration policy.

[0069] It should be noted that in this application, the action space is discrete and can be represented as a(t) = [no operation, scaling-up migration strategy, scaling-down migration strategy, load balancing migration strategy]. The PPO agent only needs to determine the operation type of the traffic migration strategy to be executed. The execution process of the traffic migration strategy can be implemented using traffic migration optimization algorithms. These algorithms can provide specific traffic migration solutions and evaluate the related costs incurred by traffic migration. The aforementioned migration optimization algorithms can employ relevant algorithms in the prior art, and further examples are not provided here.

[0070] In one example, the reward function is determined by the network's specified optimization objective. For instance, in this application, the reward function is determined by the traffic migration cost and VNF instance load state penalty incurred at each time t.

[0071] As described above, the PPO agent can train the target traffic migration model. Specifically, the PPO agent determines the corresponding migration state information based on the reward value of the target traffic migration strategy for each user traffic item in the traffic set, and constructs the target traffic migration model based on this state transition information. The PPO agent can determine the reward value based on instance overload penalty cost, instance light load penalty cost, instance variance penalty cost, and traffic migration cost.

[0072] Specifically, the PPO agent needs to obtain the maximum load of VNF instances, the minimum load of VNF instances, and the load variance of the VNF instances included in the current network.

[0073] As an example, the maximum load of a VNF instance is the maximum of the load limits corresponding to all VNF instances in the current network. That is, the maximum load of a VNF instance can be expressed by the following formula:

[0074]

[0075] In formula (5), L max (f) represents the maximum load of the VNF instance; I f For the set of instances of VNF f.

[0076] Additionally, the load limit corresponding to the aforementioned VNF instance can be represented as L. upper (f), which is the upper limit of the ideal load threshold of VNF f.

[0077] As an example, the minimum load of a VNF instance is the minimum of the load lower limits corresponding to all VNF instances in the current network. That is, the minimum load of a VNF instance can be expressed by the following formula:

[0078]

[0079] In formula (6), L min (f) represents the minimum load of the VNF instance.

[0080] Additionally, the load floor corresponding to the aforementioned VNF instance can be expressed as L. lower (f), which is the lower limit of the ideal load threshold of VNF f.

[0081] As an example, the load variance of a VNF instance can be represented by the following formula:

[0082]

[0083] In formula (7), L var (f) represents the load variance among instances belonging to VNF f at time t.

[0084] After obtaining the maximum load value, minimum load value, and load variance value of the VNF instance, the instance overload penalty cost, instance light load penalty cost, and instance variance penalty cost can be calculated based on the above three values.

[0085] In one example, the instance overload penalty cost corresponding to the current VNF ​​instance can be determined based on the maximum load of the VNF instance and the load limit corresponding to the current VNF ​​instance, where the current VNF ​​instance is any one of the VNF instances contained in the current network.

[0086] The instance overload penalty cost can be expressed by the following formula:

[0087]

[0088] In formula (8), P O (t) represents the instance overload penalty cost of the VNF instance at time t; λ1 and Δ1 are both constants. λ1 is used to adjust the rate of increase of the instance overload penalty cost, and Δ1 is used to control the range of the instance overload penalty cost to avoid the instance overload penalty cost being too large.

[0089] It should be noted that the exponential function in formula (8) makes the distribution of instance overload penalty costs more reasonable, that is, the maximum load of the VNF instance exceeds L. upper (f) The more instances there are, the faster the instance overload penalty cost increases.

[0090] In one example, the instance light load penalty cost for the current VNF ​​instance can be determined based on the minimum load of the VNF instance and the lower limit of the load corresponding to the current VNF ​​instance. The instance light load penalty cost can be expressed by the following formula:

[0091]

[0092] In formula (9), P U (t) represents the instance light load penalty cost of the VNF instance at time t; λ2 and Δ2 are both constants. λ2 is used to adjust the increase rate of the instance light load penalty cost, and Δ2 is used to control the range of the instance light load penalty cost to avoid the instance light load penalty cost being too large.

[0093] It should be noted that, similar to the instance overload penalty cost, the calculation of the instance light load penalty cost only considers that the minimum load of the VNF instance is lower than L. lower (f) is the case.

[0094] In one example, the instance variance penalty cost corresponding to the current VNF ​​instance can be determined based on the load variance value. The instance variance penalty cost can be expressed by the following formula:

[0095]

[0096] In formula (10), P V (t) represents the instance variance penalty cost of the VNF instance at time t; λ3 and Δ3 are both constants. λ3 is used to adjust the rate of increase of the instance variance penalty cost, and Δ3 is used to control the range of the instance variance penalty cost to avoid the instance variance penalty cost being too large.

[0097] In one example, the traffic migration cost corresponding to the execution of the target traffic migration strategy can be calculated based on a preset traffic migration algorithm. The preset traffic migration algorithm can be a commonly used algorithm in related technologies, and no specific limitation is made here.

[0098] After obtaining the instance overload penalty cost, instance light load penalty cost, instance variance penalty cost, and traffic migration cost, the reward value can be determined. The reward value can be determined according to the following formula:

[0099] r(t) = P O (t)+P U (t)+P V (t)+ω1C M (a(o,t)) (11)

[0100] In formula (11), ω1 is a weighting coefficient used to balance the traffic migration cost and load optimization result of the target traffic migration model. The load optimization result is the result obtained by optimizing the load status of VNF instances in the current network.

[0101] It should be noted that, as shown in formula (11), ω1 is the weighting coefficient that balances traffic migration cost and VNF instance state penalty cost. If the value of ω1 is set high, it indicates that the optimization of migration cost is given more attention, and the trained target traffic model will tend to execute traffic migration strategies with lower migration costs, or even not execute any migration strategy at all. If the value of ω1 is set low, it indicates that the optimization of VNF instance state penalty cost is given more attention, and the trained target traffic model will more actively execute traffic migration strategies to improve the load performance of VNF instances. Therefore, the setting of the weighting coefficient ω1 is of great significance.

[0102] In one example, the weight range of the weight coefficients can be determined based on the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic flow, the maximum number of migrating traffic flows from VNF instances currently included in the network, and the minimum value of the VNF instance state penalty cost. The average time cost of migrating each user traffic flow can be determined through empirical measurement; the maximum number of migrating traffic flows from VNF instances currently included in the network can be determined by the model trainer training the target traffic migration model.

[0103] Specifically, after obtaining the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic, the maximum number of migrated traffic from the VNF instances currently included in the network, and the minimum value of the VNF instance state penalty cost, the first constraint condition for the weighting coefficients is constructed based on the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic, and the maximum number of migrated traffic. That is, the first constraint condition can be expressed by the following formula:

[0104] ω1×n1×t1<p1 (12)

[0105] In formula (12), p1 is the maximum value of the VNF instance state penalty cost; t1 is the average time cost of migrating each user traffic; and n1 is the maximum number of migrated traffic.

[0106] Furthermore, a second constraint on the weighting coefficients can be constructed based on the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic, and the minimum value of the VNF instance state penalty cost. That is, the second constraint can be expressed by the following formula:

[0107] ω1×n1×t1>p2 (13)

[0108] In formula (13), p2 is the minimum value of the VNF instance state penalty cost.

[0109] After determining the first and second constraints, the weight range corresponding to the weight coefficients can be determined based on the first and second constraints.

[0110] In one example, the weight coefficient ω1 is set to ensure that the traffic migration cost is lower than the maximum VNF instance state penalty cost, so that when VNF instances are overloaded, lightly loaded, or unbalanced in the network, the traffic migration policy corresponding to the VNF instance state can be executed.

[0111] As an example, if the maximum value of the VNF instance state penalty cost p1 is 1200, the average time cost of each user traffic migration t1 is 10, and the maximum acceptable number of migration traffic n1 is 300, then ω1×300×10<1200 can be obtained from formula (12).

[0112] When the minimum value p2 of the VNF instance state penalty cost is 30, according to formula (13), we can obtain ω1×300×10>30.

[0113] Therefore, it can be determined that the range of values ​​for the weight coefficient ω1 (i.e., the weight range) is 0.01 to 0.4.

[0114] After determining the weight range of the weight coefficients, model trainers can choose the specific value of the weight coefficient ω1 within the aforementioned weight range according to actual needs. For example, if the value of ω1 is closer to 0.4, the model trainer is more inclined to optimize traffic migration costs. Conversely, if the value of ω1 is set closer to 0.01, the model trainer is more inclined to optimize the VNF instance load status.

[0115] It should be noted that, generally speaking, once the target traffic migration model has been trained, the weight coefficients will not change; if the model trainer needs to adjust the weight coefficients of the target traffic migration model, a new traffic migration model needs to be trained.

[0116] Furthermore, after determining the weight coefficients, the reward value can be calculated according to formula (11). Then, the PPO agent can update the model parameters of the initial traffic migration model based on the first load state, the second load state, the target traffic migration strategy and the reward value to obtain the target traffic migration model.

[0117] Specifically, the PPO agent constructs a state transition tuple for each user traffic flow based on its first load state, second load state, target traffic migration strategy, and reward value, and stores this state transition tuple in the target cache area. Then, the state transition tuples in the target cache area are partitioned into multiple datasets. Based on the state transition tuples stored in the target cache area, the dominant function corresponding to the target traffic migration strategy used in the first load state of each VNF instance is determined. Finally, based on the multiple datasets and the dominant function, the objective function of the initial traffic migration model is constructed, and the model parameters of the initial traffic migration model are optimized based on the objective function to obtain the target traffic migration model.

[0118] It should be noted that the state transition tuple is used to represent the state transition information of the VNF instance under the target traffic migration strategy.

[0119] As an example, the pseudocode for offline training of the target traffic migration model is as follows:

[0120] Input: a traffic set [R(1), R(2), ..., R(T)], the computational resource capacity of each VNF instance i∈I, the initial routing path of each traffic, the migration cost of a single traffic, the number of training rounds M, and the size of the mini-batch dataset σ.

[0121] Output: PPO proxy neural network parameters θ

[0122]

[0123] The following is an explanation of the pseudocode above:

[0124] Before offline training begins, it is necessary to collect relevant information about user traffic (e.g., the amount of traffic at each moment, the initial routing path of each traffic instance, etc.), the computational resource capacity of VNF instances in the network, the total number of training rounds R, and the model learning period T. learnThe size of the small batch dataset σ is used as the input parameters for model training of the PPO agent. Before model training, the neural network parameters θ and buffer M of the PPO agent need to be initialized. Then, the PPO agent performs R rounds of training. In each round, T training steps are executed, where the number of training steps T also represents the size of the traffic set. In each training step t, the PPO agent interacts with the network environment. First, the PPO agent calculates the VNF instance load based on the current network state and inputs it into the PPO agent's neural network as the load state s(t) at time t. Then, the PPO agent calculates the load based on the probability distribution π. θ (·|s(t)) randomly samples from multiple traffic migration strategies to obtain the traffic migration strategy a(t) to be executed at the current time. Next, the PPO agent executes the traffic migration strategy (t) in the network and calculates the VNF instance load state s(t+1) and the corresponding reward value r(t) after executing the traffic migration strategy. In each training step, the PPO agent stores the state transition tuple in the buffer M. If the time corresponding to the current training step t reaches the learning period T... learn The PPO agent then calculates an estimate of the dominance function based on the state transition data stored in buffer M. The state transition data in the buffer M is divided into several mini-batches (i.e., multiple data sets) of size σ, and the objective function L(Θ) is optimized using each mini-batch. After optimization, the PPO agent updates the old neural network parameters θ based on the current neural network parameters θ. old Simultaneously, the buffer M is cleared. After completing training for all rounds, the parameters θ of the PPO proxy neural network can be obtained, which are the model parameters corresponding to the target traffic migration model.

[0125] In one example, the objective function can be constructed according to the following formula:

[0126]

[0127] In formula (14), L(Θ) is the objective function; θ is the model parameter of the initial flow migration model; Let be the expected value of sampling each dataset at time t; is the dominant function at time t; ∈ is the preset parameter; clip(·) is the clipping function used to limit the output range of the objective function; s(t) represents the load state of the VNF instance at time t, a(t) represents the traffic migration policy executed at time t, and π θ (a(t), s(t)) represents the probability of performing a(t) on s(t) at time t in the first flow migration model. Let a(t) be the probability of performing a(t) on s(t) at time t in the second traffic migration model. The first traffic migration model is the model before the model parameters are updated, and the second traffic migration model is the model after the model parameters are updated.

[0128] As an example, the objective function of a PPO agent can be represented by the following formula:

[0129]

[0130] In this application, the objective function shown in formula (15) is improved by introducing a clip function to control the policy update, wherein the clip(·) function is used to limit the probability value to the range [1-∈, 1+∈].

[0131] It should be noted that after the target traffic migration model has been trained, it can be applied to traffic migration in the network.

[0132] As an example, in practical applications, a traffic migration system can acquire the target user traffic in real time and input it into the target traffic migration model. After acquiring the target user traffic, the target traffic migration model can determine the execution probability of each traffic migration strategy based on the target user traffic, the load status information of VNF instances in the current network, and the reward value corresponding to each traffic migration strategy. It then selects the target traffic migration strategy with the highest execution probability from multiple strategies and outputs it. After acquiring the target traffic migration strategy output by the target traffic migration model, the traffic migration system executes the target traffic migration strategy at the current moment.

[0133] As an example, this application uses the following experiment to verify the traffic migration effect of the target traffic migration model trained using the scheme provided in this application.

[0134] Before conducting the verification experiment, the experimental dataset is first set up, including topology settings, VNF and SFC settings, and traffic data settings.

[0135] For topology setup, this verification experiment uses two topologies: Abilene (containing 11 nodes and 14 links) and Geant (containing 23 nodes and 37 links) to test the performance of different traffic migration models. In each topology, several nodes are randomly selected as cloud nodes, and the remaining nodes are treated as switching nodes. Cloud nodes are divided into two categories: resource-sufficient nodes and resource-constrained nodes. For resource-sufficient nodes, VNF instances deployed on these nodes can be allocated more computing resources; while resource-constrained nodes can only allocate fewer computing resources to their VNF instances.

[0136] It should be noted that the difference in computing resources allocated to the two types of cloud nodes will affect the maximum traffic capacity that VNF ​​instances running on different types of cloud nodes can handle. Additionally, the forwarding latency of the links in both topologies is set within the range of [3, 70] milliseconds, and the bandwidth capacity is set within the range of [1, 10] Gb / s.

[0137] For VNF and SFC configuration, this verification experiment sets up six different types of VNFs, such as firewalls, traffic intrusion detection, and network address mapping. For each VNF, the migration time cost of a single traffic flow through the VNF can be set within the range of [1, 50]. Regarding the maximum processing capacity of a VNF instance, if the current VNF ​​instance is running on a resource-sufficient cloud node, the processing capacity range is [0.9, 1.2] GB / s; if the current VNF ​​instance is running on a resource-constrained cloud node, the processing capacity range is [0.4, 0.8] GB / s. In this verification experiment, a certain number of VNF instances of each type can be randomly allocated on each cloud node, with the number of VNF instances set within the range of [1, 5]. Most VNF ​​instances are set to active status to ensure that most active VNF instances can process traffic at any time, while the remaining VNF instances remain shut down and are activated when existing VNF instances cannot handle all traffic to supplement the processing capacity of the SFC in the network.

[0138] It should be noted that in this verification experiment, any 2 to 5 VNFs can be set to form an SFC, and 5 different SFCs can be set.

[0139] For traffic data setup, this validation experiment will use two sets of traffic matrix data from the Abilene and Geant topologies, respectively, to set user traffic and their SFC requirements. The time span for both sets of traffic matrices is one week, with the traffic data from the first five days used to train the PPO model, and the traffic data from the last two days used to test the specific effects of different traffic migration models.

[0140] It should be noted that in this verification experiment, the traffic volume between each node pair in the traffic matrix was normalized, and then the normalized traffic data was scaled up proportionally (1 unit of traffic is 1000MB). When dividing the traffic into individual streams, the distribution of traffic size can follow the 80 / 20 rule, that is, 20% of all traffic has a larger load length (labeled as large flow), while the remaining 80% of traffic has a smaller load length (labeled as small flow). The load length of small flow ranges from 10KB to 50KB, and the load length of large flow ranges from 500KB to 5MB. Finally, it is necessary to ensure that the sum of the load lengths of all traffic between each node pair is close to the original traffic size between that node pair.

[0141] In addition, it should be noted that in this verification experiment, the SFC required for all traffic can be randomly set, and the initial routing path of the traffic is determined according to Dijkstra's algorithm. This initial routing path must ensure that the traffic can fully obtain the service of the SFC.

[0142] After setting up the experimental dataset, it is also necessary to set up the comparison algorithm, which includes the fixed threshold method and the automatic traffic migration decision algorithm. The automatic traffic migration decision algorithm is the algorithm used by this application to train the target traffic migration model.

[0143] For the fixed threshold method, this verification experiment sets a fixed threshold range for VNF instances and monitors the load of VNF instances. If the load of a VNF instance exceeds the set threshold range, a scaling-up or scaling-down migration strategy is executed. The load variance between VNF instances can also be used to determine when to execute traffic migration; if the load variance is too high (exceeding the set threshold), a load balancing migration strategy is executed.

[0144] In this verification experiment, the automatic traffic migration decision algorithm proposed in this application is mainly compared with two types of fixed threshold methods:

[0145] (a) Considering only the scaling up and scaling down migration strategies, the upper limit of the VNF instance load threshold is L. upper =80%, lower limit is L lower =20%, this method is identified as th_soi;

[0146] (b) Both scaling up and scaling down migration strategies were executed, as well as load balancing migration strategies. During the experiment, three different load variance thresholds were set to control the timing of the load balancing migration strategy: 0.01, 0.03, and 0.06 (in this verification experiment, the load variance of the VNF instance fluctuated in the range of 0 to 0.08).

[0147] It's important to note that a high load variance threshold means that a higher VNF instance load variance can be tolerated, resulting in fewer executions of the load balancing migration strategy. Conversely, a low load variance threshold means that the load balancing migration strategy will be executed frequently to maintain a low load variance state. These methods with different load variance thresholds will be labeled as: th_all:0.01, th_all:0.03, and th_all:0.06, respectively.

[0148] In this verification experiment, two automatic traffic migration decision algorithms were compared. One algorithm used a larger reward function ω1 during training; in this experiment, ω1 = 0.3. Therefore, this algorithm tends to perform fewer traffic migrations to optimize traffic migration costs, and is labeled AFMD (for mig). The other algorithm used a smaller reward function ω1 during training; in this experiment, ω1 = 0.003. Therefore, this algorithm tends to perform more traffic migration operations to optimize the load on the VNF instance, and is labeled AFMD (for load).

[0149] After determining the experimental dataset and the comparison function, the experiment can be conducted. This validation experiment includes the following four optimization results: optimization results for the maximum and minimum VNF instance load ratio, optimization results for VNF instance load variance, optimization results for traffic migration cost, and results for the number of executions of different traffic migration operations. The following explains these four optimization results.

[0150] Regarding the optimization results of the maximum and minimum VNF instance load ratio, the cumulative probability distribution (CDF) results of the maximum and minimum VNF instance load ratios achieved by different algorithms in the Abilene and Geant topologies in this verification experiment are shown in Figures 3(a), 3(b), 3(c), and 3(d). From Figures 3(a) and 3(b), it can be seen that the algorithms that achieve a smaller maximum VNF instance load with a higher probability are ranked as follows: th_all:0.01 > AFMD(for load) > th_all:0.03 > AFMD(for mig) > th_all:0.06 ≈ th_soi. Wherein, the defined VNF instance load upper limit L... upperUsing 80% as a threshold, the th_all:0.01 algorithm achieves the aforementioned high load ratio with a 94% probability in the Abilene topology and a 98% probability in the Geant topology. The th_all:0.03 algorithm achieves the same high load ratio with 81% and 82% probabilities in the Abilene and Geant topologies, respectively. The th_all:0.06 algorithm achieves the same high load ratio with 62% and 73% probabilities in the Abilene and Geant topologies, respectively. The th_soi algorithm achieves an 80% load ratio with probabilities of 97% (Abilene topology) and 92% (Geant topology), the AFMD (for load) algorithm with probabilities of 96% (Abilene topology) and 95% (Geant topology), and the AFMD (for mig) algorithm with probabilities of 93% (Abilene topology) and 93% (Geant topology).

[0151] The experimental results above show that, since these comparison algorithms can perform scaling migration and load balancing migration operations in a timely manner to reduce the maximum load of VNF instances, th_all:0.01, th_soi, AFMD(for load), and AFMD(formig) can control the maximum load of VNF instances below the threshold limit in most cases.

[0152] However, within a continuous time period, dynamic changes in traffic and traffic migration operations performed in the previous period may affect traffic migration operations performed at a later time. For example, due to time constraints of migration operations, the load of a VNF instance may not be optimized to below the upper limit of the load threshold. According to the experimental results shown in Figures 3(a) and 3(b), both th_all:0.03 and th_all:0.06 have a certain probability of causing the VNF instance load to exceed the upper limit of the ideal load threshold. However, all algorithms can guarantee that the total traffic flowing through each VNF instance will not exceed its capacity limit, that is, the load of the VNF instance will not exceed 100%.

[0153] Regarding the optimization effect on the minimum VNF instance load, according to Figure 3(c) and 3(d) The results shown indicate that despite implementing both the scaling-down migration strategy and the load balancing migration strategy, the load on some VNF instances remains below the lower limit of the ideal load threshold (L). lowerThe situation (20%) occurs. The main reason for this is that during traffic migration, at least one instance of each type of VNF must always be active, and these VNF instances cannot be shut down to prevent the SFC from failing to provide complete service to user traffic. In addition, the more times the load balancing migration strategy is executed, the more likely it is to cause low-load VNF instances to appear. This situation arises mainly because only one traffic migration strategy can be executed at each time t. If the load balancing migration strategy is executed, the scaling-down traffic migration strategy cannot be executed to shut down low-load VNF instances (the load balancing migration strategy does not consider shutting down VNF instances). Another reason is that load balancing migration cannot guarantee that after migrating traffic from high-load VNF instances to low-load VNF instances, the load of the low-load VNF instances will necessarily be higher than the ideal load threshold L. lower Using the lower limit of the ideal VNF instance load threshold as the boundary, the th_all:0.01 method will generate low-load VNF instances with the highest probability, 48% in the Abilene topology and 45% in the Geant topology.

[0154] Regarding the VNF instance load variance optimization results, in this verification experiment, the cumulative probability distribution of VNF instance load variance achieved by different algorithms in the Abilene and Geant topologies is shown in Figures 4(a) and 4(b). In most cases, the algorithms that achieve smaller VNF instance load variance with higher probability are ranked as follows: th_all:0.01>AFMD(forload)>th_all:0.03>AFMD(for mig)>th_all:0.06≈th_soi. Using a load variance of 0.04 as a boundary, as shown in Figure 4(a), in the Abilene topology, methods such as th_all:0.01, th_all:0.03, th_all:0.06, th_soi, AFMD(forload), and AFMD(for mig) can achieve a load variance of 0.04 with probabilities of 93%, 78%, 49%, 54%, 87%, and 70%, respectively. In the Geant topology, as shown in Figure 4(b), the above methods can control the load variance below 0.04 with probabilities of 97%, 79%, 57%, 56%, 90%, and 75%, respectively. If we take a cumulative probability of 90% as the threshold, `th_all: 0.01` can control the maximum VNF instance load variance below 0.035 (Abilene topology) and 0.022 (Geant topology); `th_all: 0.03` can control the maximum VNF instance load variance below 0.061 (Abilene topology) and 0.052 (Geant topology); `th_all: 0.06` can control the maximum VNF instance load variance below 0.071 (Abilene topology) and 0.057 (Geant topology); `th_soi` can control the maximum VNF instance load variance below 0.061 (Abilene topology) and 0.054 (Geant topology); `AFMD(for load)` can control the maximum VNF instance load variance below 0.048 (Abilene topology) and 0.042 (Geant topology); `AFMD(for load)` can control the maximum VNF instance load variance below 0.048 (Abilene topology) and 0.042 (Geant topology). The mig can control the maximum VNF instance load variance below 0.054 (Abilene topology) and 0.047 (Geant topology).

[0155] As shown in Figures 4(a) and 4(b), the optimization effect of the load variance of VNF instances is affected by the number of times the load balancing migration strategy is executed. That is, the more times the load balancing migration strategy is executed, the easier it is to control the load variance between VNF instances at a low level (e.g., th_all: 0.01). The timing of the execution of the load balancing migration strategy is also important. The automatic traffic migration decision algorithm proposed in this application can achieve similar or even better load variance optimization effects with fewer load balancing migration strategies.

[0156] Regarding the optimization results of traffic migration costs, in this verification experiment, the cumulative traffic migration costs generated within the time period of the test traffic matrix were statistically analyzed, with each unit of time (i.e., 1 Unit) being 25 minutes. Figures 5(a) and 5(b) show the cumulative traffic migration costs under different topologies, respectively. As can be seen from Figures 5(a) and 5(b), the cumulative traffic migration costs generated by different algorithms are arranged in descending order as follows: th_all:0.01>th_all:0.03>AFMD(for load)>AFMD(for mig)>th_all:0.06>th_soi. It can be seen that whether the fixed threshold method or the automatic traffic migration decision algorithm is used to determine the execution timing of the traffic migration strategy, both can optimize the load of VNF instances. Therefore, in order to better optimize the load of VNF instances, the traffic migration strategy may be triggered frequently, resulting in higher traffic migration costs, as shown by th_all:0.01, th_all:0.03, and AFMD(for load) in Figures 5(a) and 5(b).

[0157] In addition, Table 1 shows the optimization of VNF instance load and traffic migration cost by different algorithms. As shown in Table 1, the optimization of traffic migration cost is mainly based on the traffic migration cost generated by th_all:0.01, and the reduction in traffic migration cost by other algorithms is statistically analyzed. The results in Table 1 show that although th_all:0.01 achieves the best VNF ​​instance load optimization effect (including maximum VNF instance load and VNF instance load variance), it also generates the highest traffic migration cost. While th_all:0.06 and th_soi methods generate the lowest traffic migration cost, their optimization effect on VNF instance load is not ideal. Specifically, th_soi is particularly poor in optimizing VNF instance load variance. Compared to th_all:0.03, whether it is more inclined to optimize VNF instance load (AFMD(for load)) or more inclined to optimize traffic migration cost (AFMD(for mig)), it can achieve similar or even better VNF instance load optimization results with lower traffic migration cost. Moreover, AFMD(for mig) can simultaneously ensure good optimization results for both VNF instance load and traffic migration cost.

[0158] Table 1

[0159]

[0160] In this verification experiment, the execution counts of different traffic migration strategies in different topologies were statistically analyzed, resulting in the bar charts shown in Figures 6(a) and 6(b). As shown in Figures 6(a) and 6(b), th_all:0.01 executes the traffic migration strategy the most times. Because it sets a lower load variance threshold, th_all:0.01 can replace scaling up and scaling down strategies by executing the load balancing migration strategy multiple times, thus incurring higher traffic migration costs. In contrast, the proposed algorithms AFMD(for load) and AFMD(for mig) execute the traffic migration strategy fewer times. AFMD(for mig) executes the traffic migration strategy the fewest times, and the number of times AFMD(for load) and AFMD(for mig) select no operation is 82%–157% higher than th_all:0.01. Moreover, in practical applications, fewer execution counts of the traffic migration strategy help maintain network stability and reduce traffic migration costs.

[0161] The above four optimization results show that the fixed threshold method can only complete the traffic migration strategy based on the set threshold, but it does not consider the related costs incurred in optimizing the migration strategy. This results in achieving less network performance optimization with higher traffic migration costs. Furthermore, in real-world network environments, setting an appropriate threshold range for the timing of traffic migration strategy decisions is very difficult, especially setting an appropriate load variance threshold for load balancing migration strategies.

[0162] In contrast, the automatic traffic migration decision algorithm proposed in this application comprehensively considers the VNF instance load and traffic migration-related costs during model training, and selects an appropriate traffic migration algorithm to execute at a suitable time after training. Therefore, the automatic traffic migration decision algorithm proposed in this application can better balance the optimization effects between VNF instance load status and traffic migration costs. Furthermore, the automatic traffic migration decision algorithm proposed in this application also allows for adjusting the emphasis on migration costs and load optimization effects during training (i.e., adjusting the weight coefficients in the reward function) to achieve different optimization effects.

[0163] This application also provides a model training device based on traffic migration, such as... Figure 7 As shown, the device includes: a traffic acquisition module 701, a policy determination module 702, a status determination module 703, and a parameter update module 704.

[0164] The traffic acquisition module 701 is used to acquire a set of user traffic consisting of multiple user traffic streams;

[0165] The strategy determination module 702 is used to traverse each user traffic in the user traffic set, and randomly sample multiple traffic migration strategies based on the first load state of the virtual network function (VNF) instance in the current network to obtain the target traffic migration strategy corresponding to each user traffic. The first load state is the load state of the VNF instance before executing the target traffic migration strategy.

[0166] The state determination module 703 is used to calculate the reward value corresponding to the execution of the target traffic migration strategy and determine the second load state corresponding to the VNF instance. The second load state is the load state of the VNF instance after the execution of the target traffic migration strategy. The reward value is used to characterize the state penalty cost and traffic migration cost of the VNF instance corresponding to the execution of the target traffic migration strategy.

[0167] The parameter update module 704 is used to update the model parameters of the initial traffic migration model based on the first load state, the second load state, the target traffic migration strategy, and the reward value, so as to obtain the target traffic migration model.

[0168] As can be seen from the above, this application constructs a target traffic migration model. This target traffic migration model can determine the traffic migration strategy at the current moment based solely on the load status of VNF instances and user traffic in the current network, without setting a threshold. This avoids the problems of reduced network performance and increased migration costs caused by unreasonable threshold settings.

[0169] Furthermore, in this application, the target traffic migration model is determined based on the load state before and after migration, the migration strategy, and the reward value representing the migration cost. The load state after migration represents the migration effect corresponding to the execution of the migration strategy. Therefore, this application comprehensively considers both the migration cost and the migration effect during the training of the target traffic migration model. Thus, when using the trained target traffic migration model for traffic migration, the model can determine the target traffic migration strategy to be executed at the current moment based on the reward value corresponding to different traffic migration strategies. This effectively avoids the unreasonable phenomenon in related technologies where the benefits of network performance improvement are lower than the overhead of traffic migration operations, achieving the goal of reducing traffic migration costs while ensuring migration effectiveness.

[0170] In one example, the state determination module includes: a load state acquisition module, a first cost determination module, a second cost determination module, a third cost determination module, a fourth cost determination module, and a reward calculation module. The load state acquisition module acquires the maximum VNF instance load, the minimum VNF instance load, and the load variance of the VNF instances in the current network. The maximum VNF instance load is the maximum of the upper load limits for all VNF instances in the current network, and the minimum VNF instance load is the minimum of the lower load limits for all VNF instances in the current network. The first cost determination module determines the instance overload penalty cost for the current VNF ​​instance based on the maximum VNF instance load and the upper load limit for the current VNF ​​instance. The current VNF ​​instance is the upper load limit for the current network. The system includes: a first module for determining the instance light load penalty cost of the current VNF ​​instance based on the minimum load of the VNF instance and the lower limit of the load corresponding to the current VNF ​​instance; a second module for determining the instance variance penalty cost of the current VNF ​​instance based on the load variance value; a third module for determining the instance variance penalty cost of the current VNF ​​instance based on the load variance value; a fourth module for determining the traffic migration cost corresponding to the execution of the target traffic migration strategy based on a preset traffic migration algorithm; and a reward calculation module for determining the reward value based on the instance overload penalty cost, instance light load penalty cost, instance variance penalty cost, and traffic migration cost.

[0171] In one example, the reward calculation module is specifically used to determine the reward value according to the following formula:

[0172] r(t) = P O (t)+P U (t)+P V (t)+ω1C M (a(o,t))

[0173] Where r(t) is the reward value at time t; P O (t) represents the instance overload penalty cost at time t; P U (t) represents the instance light load penalty cost at time t; P V (t) represents the instance variance penalty cost at time t; a(o, t) represents the target traffic migration strategy with identifier 'o' adopted at time t; C M (a(o,t)) represents the traffic migration cost corresponding to the target traffic migration strategy; ω1 is a weighting coefficient used to balance the traffic migration cost and load optimization result of the target traffic migration model. The load optimization result is the result obtained by optimizing the load status of VNF instances in the current network.

[0174] In one example, the model training device based on traffic migration further includes: a cost data acquisition module, a first condition construction module, a second condition construction module, and a weight determination module. The cost data acquisition module is used to acquire the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic flow, the maximum number of migrated traffic flows from VNF instances currently included in the network, and the minimum value of the VNF instance state penalty cost. The first condition construction module is used to construct a first constraint condition for the weight coefficients based on the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic flow, and the maximum number of migrated traffic flows. The second condition construction module is used to construct a second constraint condition for the weight coefficients based on the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic flow, and the minimum value of the VNF instance state penalty cost. The weight determination module is used to determine the weight range corresponding to the weight coefficients based on the first and second constraint conditions.

[0175] In one example, the first constraint is:

[0176] ω1×n1×t1<p1

[0177] Where p1 is the maximum value of the VNF instance state penalty cost; t1 is the average time cost of migrating each user traffic; and n1 is the maximum number of migrated traffic.

[0178] The second constraint is:

[0179] ω1×n1×t1>p2

[0180] Where p2 is the minimum value of the state penalty cost of the VNF instance.

[0181] In one example, the parameter update module includes: a transfer data construction module, a data grouping module, a function determination module, an optimization function construction module, and a parameter optimization module. Specifically, the transfer data construction module constructs a state transition tuple for each user traffic flow based on the first load state, second load state, target traffic migration strategy, and reward value, and stores this state transition tuple in the target cache area. The state transition tuple represents the state transition information of the VNF instance under the target traffic migration strategy. The data grouping module partitions the state transition tuples in the target cache area to obtain multiple data sets. The function determination module determines the dominant function corresponding to the target traffic migration strategy used in the first load state of each VNF instance based on the state transition tuples stored in the target cache area. The optimization function construction module constructs the objective function of the initial traffic migration model based on the multiple data sets and the dominant function. The parameter optimization module optimizes the model parameters of the initial traffic migration model based on the objective function to obtain the target traffic migration model.

[0182] In one example, the objective function is constructed according to the following formula:

[0183]

[0184] Where L(Θ) is the objective function; θ is the model parameter of the initial traffic migration model; Let be the expected value of sampling each dataset at time t; is the dominant function at time t; ∈ is the preset parameter; clip(·) is the clipping function used to limit the output range of the objective function; s(t) represents the load state of the VNF instance at time t, a(t) represents the traffic migration policy executed at time t, and π θ (a(t), s(t)) represents the probability of performing a(t) on s(t) at time t in the first flow migration model. Let a(t) be the probability of performing a(t) on s(t) at time t in the second traffic migration model. The first traffic migration model is the model before the model parameters are updated, and the second traffic migration model is the model after the model parameters are updated.

[0185] In one example, the traffic migration-based model training device further includes: a user traffic acquisition module, a target policy determination module, and a traffic migration module. The user traffic acquisition module acquires the target user traffic at the current moment; the target policy determination module inputs the target user traffic into the target traffic migration model, enabling the model to determine the execution probability of each traffic migration policy based on the target user traffic, the load status information of VNF instances in the current network, and the reward value corresponding to each traffic migration policy, and to determine the target traffic migration policy with the highest execution probability from multiple traffic migration policies, outputting the target traffic migration policy; the traffic migration module executes the target traffic migration policy at the current moment.

[0186] The traffic migration-based model training apparatus provided in this application embodiment can implement all the processes implemented in the aforementioned method embodiments. To avoid repetition, it will not be described again here.

[0187] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0188] Figure 8 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0189] Electronic devices may include a processor 801 and a memory 802 storing computer program instructions.

[0190] Specifically, the processor 801 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0191] Memory 802 may include mass storage for data or instructions. For example, and not limitingly, memory 802 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 802 may include removable or non-removable (or fixed) media. Where appropriate, memory 802 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 802 is non-volatile solid-state memory.

[0192] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0193] The processor 801 reads and executes computer program instructions stored in the memory 802 to implement any of the traffic migration-based model training methods in the above embodiments.

[0194] In one example, the electronic device may also include a communication interface 803 and a bus 810. Wherein, as... Figure 8 As shown, the processor 801, memory 802, and communication interface 803 are connected through bus 810 and complete communication with each other.

[0195] The communication interface 803 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0196] Bus 810 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 810 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0197] Furthermore, in conjunction with the traffic migration-based model training method in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the traffic migration-based model training methods in the above embodiments.

[0198] Furthermore, in conjunction with the traffic migration-based model training method in the above embodiments, this application embodiment can provide a computer program product for implementation. When the instructions in this computer program product are executed by the processor of an electronic device, the electronic device performs any of the traffic migration-based model training methods described in the above embodiments.

[0199] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0200] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0201] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0202] The foregoing flowcharts and / or block diagrams describing the methods, apparatus, devices, and media for model training based on traffic migration according to embodiments of this disclosure have described various aspects of the present disclosure. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0203] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A model training method based on traffic migration, characterized in that, include: Retrieve a collection of user traffic streams consisting of multiple user traffic streams; Traverse each user traffic in the user traffic set, and randomly sample multiple traffic migration strategies based on the first load state of the Virtual Network Function (VNF) instance in the current network to obtain the target traffic migration strategy corresponding to each user traffic. The first load state is the load state of the VNF instance before executing the target traffic migration strategy. Calculate the reward value corresponding to the execution of the target traffic migration strategy, and determine the second load state corresponding to the VNF instance, wherein the second load state is the load state of the VNF instance after the execution of the target traffic migration strategy; the reward value is used to characterize the VNF instance state penalty cost and traffic migration cost corresponding to the execution of the target traffic migration strategy. Based on the first load state, the second load state, the target traffic migration strategy, and the reward value, the model parameters of the initial traffic migration model are updated to obtain the target traffic migration model. Obtain the target user traffic at the current moment; input the target user traffic into the target traffic migration model, so that the target traffic migration model determines the execution probability of each traffic migration strategy based on the target user traffic, the load status information of the VNF instances in the current network at the current moment, and the reward value corresponding to each traffic migration strategy, and determines the target traffic migration strategy with the highest execution probability from the multiple traffic migration strategies, and outputs the target traffic migration strategy; execute the target traffic migration strategy at the current moment.

2. The method according to claim 1, characterized in that, Calculating the reward value corresponding to the execution of the target traffic migration strategy includes: Obtain the maximum VNF instance load, the minimum VNF instance load, and the load variance of the VNF instances included in the current network. The maximum VNF instance load is the maximum value of the upper limit of the load corresponding to all VNF instances in the current network, and the minimum VNF instance load is the minimum value of the lower limit of the load corresponding to all VNF instances in the current network. Based on the maximum load of the VNF instance and the load limit corresponding to the current VNF ​​instance, the instance overload penalty cost corresponding to the current VNF ​​instance is determined, wherein the current VNF ​​instance is any one of the VNF instances included in the current network; Based on the minimum load of the VNF instance and the lower limit of the load corresponding to the current VNF ​​instance, determine the instance light load penalty cost corresponding to the current VNF ​​instance; The instance variance penalty cost corresponding to the current VNF ​​instance is determined based on the load variance value; The traffic migration cost corresponding to the execution of the target traffic migration strategy is calculated based on a preset traffic migration algorithm. The reward value is determined based on the instance overload penalty cost, the instance light load penalty cost, the instance variance penalty cost, and the traffic migration cost.

3. The method according to claim 2, characterized in that, The reward value is determined based on the instance overload penalty cost, the instance light load penalty cost, the instance variance penalty cost, and the traffic migration cost, including: The reward value is determined according to the following formula: in, Let be the reward value at time t; Let t be the instance overload penalty cost; The light-load penalty cost for the instance at time t; Let $t$ be the instance variance penalty cost at time $t$. The identifier used at time t is The target traffic migration strategy; The traffic migration cost corresponding to the target traffic migration strategy; The weighting coefficient is used to balance the traffic migration cost and load optimization result of the target traffic migration model. The load optimization result is the result obtained by optimizing the load status of VNF instances in the current network.

4. The method according to claim 3, characterized in that, The method further includes: Obtain the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic, the maximum number of migrated traffic from the VNF instances included in the current network, and the minimum value of the VNF instance state penalty cost; Based on the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic, and the maximum number of migrated traffic, the first constraint condition for the weight coefficient is constructed. Based on the maximum value of the VNF instance state penalty cost, the average time cost of migrating each user traffic, and the minimum value of the VNF instance state penalty cost, a second constraint condition for the weight coefficient is constructed. The weight range corresponding to the weight coefficient is determined based on the first constraint and the second constraint.

5. The method according to claim 4, characterized in that, The method further includes: The first constraint is: in, This represents the maximum value of the penalty cost for the VNF instance state. The average time cost for migrating each user traffic item; This refers to the maximum number of migration flows; The second constraint is: in, This represents the minimum state penalty cost of the VNF instance.

6. The method according to claim 1, characterized in that, Based on the first load state, the second load state, the target traffic migration strategy, and the reward value, the model parameters of the initial traffic migration model are updated to obtain the target traffic migration model, including: Based on the first load state, the second load state, the target traffic migration strategy, and the reward value corresponding to each user traffic, a state transition tuple corresponding to each user traffic is constructed, and the state transition tuple corresponding to each user traffic is stored in the target cache area. The state transition tuple is used to characterize the state transition information of the VNF instance under the target traffic migration strategy. The state transition tuples in the target cache region are divided to obtain multiple data sets; Based on the state transition tuples stored in the target cache area, determine the advantage function corresponding to the target traffic migration strategy adopted in the first load state of each VNF instance; The objective function of the initial traffic migration model is constructed based on the multiple datasets and the advantage function. The model parameters of the initial traffic migration model are optimized based on the objective function to obtain the target traffic migration model.

7. The method according to claim 6, characterized in that, The objective function of the initial traffic migration model is constructed based on the multiple datasets and the advantage function, including: The objective function is constructed according to the following formula: in, The objective function is... These are the model parameters for the initial traffic migration model; Let be the expected value of sampling each dataset at time t; The dominant function at time t; These are preset parameters; This is a pruning function used to limit the output range of the objective function; , Let t represent the load status of the VNF instance. This represents the traffic migration strategy executed at time t. In the first traffic migration model, at time t, for s implement The probability, In the second flow migration model, at time t, for s implement The probability is given by the first traffic migration model, which is the model before the model parameters are updated, and the second traffic migration model, which is the model after the model parameters are updated.

8. A model training device based on traffic migration, characterized in that, include: The traffic acquisition module is used to acquire a set of user traffic consisting of multiple user traffic streams; The strategy determination module is used to traverse each user traffic in the user traffic set, and randomly sample multiple traffic migration strategies based on the first load state of the virtual network function (VNF) instance in the current network to obtain the target traffic migration strategy corresponding to each user traffic. The first load state is the load state of the VNF instance before executing the target traffic migration strategy. The state determination module is used to calculate the reward value corresponding to the execution of the target traffic migration strategy and determine the second load state corresponding to the VNF instance, wherein the second load state is the load state of the VNF instance after the execution of the target traffic migration strategy; the reward value is used to characterize the VNF instance state penalty cost and traffic migration cost corresponding to the execution of the target traffic migration strategy. The parameter update module is used to update the model parameters of the initial traffic migration model based on the first load state, the second load state, the target traffic migration strategy, and the reward value, so as to obtain the target traffic migration model. The user traffic acquisition module is used to acquire the target user traffic at the current moment; the target policy determination module is used to input the target user traffic into the target traffic migration model, so that the target traffic migration model determines the execution probability of each traffic migration policy based on the target user traffic, the load status information of the VNF instances in the current network at the current moment, and the reward value corresponding to each traffic migration policy, and determines the target traffic migration policy with the highest execution probability from the multiple traffic migration policies, and outputs the target traffic migration policy; the traffic migration module is used to execute the target traffic migration policy at the current moment.

9. An electronic device, characterized in that, Electronic devices include: processors and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the model training method based on traffic migration as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the model training method based on traffic migration as described in any one of claims 1-7.

11. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the model training method based on traffic migration as described in any one of claims 1-7.

Images