User experience perceived service function chain intelligent arrangement method

Abstract

The invention discloses a user experience perceived service function chain intelligent arrangement method. The method comprises the following steps: constructing a Markov decision model for a service function chain arrangement process; acquiring network bandwidth, time delay information and user request data, encoding the network bandwidth, the time delay information and the user request data into a feature vector, inputting the feature vector into a double-layer convolutional neural network as a representation of a state, and acquiring an action strategy representing an action; in each time step, selecting and executing an action, and obtaining a new state and an observation reward; the state, the executed action and the observation reward are put back into a memory bank, samples are randomly extracted from the memory bank for learning, neural network parameters are updated by using a gradient descent method, the gradient of each function is evaluated according to a loss function, and the parameters are updated based on a dynamically adjusted learning rate. According to the user experience perception-based service function chain intelligent arrangement method provided by the invention, efficient utilization of network resources and improvement of user experience can be realized.

Description

Technical Field

[0001] The present application relates to the field of Internet of Things technology, and more specifically, to a user experience-aware service function chain intelligent orchestration method. Background Art

[0002] With the diversified application of IoT and network devices, software-defined networking (SDN) and network function virtualization (NFV) are regarded as the basic key technologies for optimizing networks. Network services are built by orchestrating different network functions (such as firewalls, deep packet inspection, WAN optimizers, agents, etc.). However, the current indicator for measuring network service performance is the quality of service (QoS) at the data level, and the QoS at the data level does not necessarily correspond to a good user experience. Therefore, flexible network service performance should be judged by the intuitive user experience quality (QoE) and meet the network's service performance quality requirements for users.

[0003] The deployment of the service function chain is to select and arrange various network function instances to the corresponding network resource nodes. When the network node resources meet the resource requirements of the network function instance, the chain can be successfully established. There are many commonly used heuristic algorithms, such as genetic algorithms, ant colony algorithms, greedy algorithms, etc. These algorithms have the defects of high computational complexity, poor scalability and easy to fall into local optimality, which cannot meet the technical requirements of the growing Internet of Things. Therefore, a new service function chain intelligent arrangement method is urgently needed. Summary of the invention

[0004] In response to at least one defect or improvement need in the prior art, the present invention provides a user experience-aware service function chain intelligent orchestration method, which can solve at least one of the problems existing in the above-mentioned background technology.

[0005] To achieve the above object, according to a first aspect of the present invention, a user experience-aware service function chain intelligent orchestration method is provided, comprising:

[0006] Construct a Markov decision model for the service function chain orchestration process, define the state space, action space, transition probability and reward function, initialize the two-layer convolutional neural network, and initialize the experience replay memory library;

[0007] Obtaining network bandwidth, latency information and user request data, encoding them into feature vectors as representations of states and inputting them into the two-layer convolutional neural network to obtain action strategies including actions;

[0008] At each time step, an action is selected and executed based on the current state and action strategy to obtain a new state and observation reward;

[0009] The state, executed actions, and observed rewards are replayed into the memory bank, from which samples are randomly drawn for learning, and the neural network parameters are updated using the gradient descent method. The gradient of each function is evaluated according to the loss function, and the parameters are updated based on the dynamically adjusted learning rate.

[0010] Furthermore, in the above-mentioned user experience-aware service function chain intelligent orchestration method, the state space, action space, transition probability and reward function specifically include:

[0011] The state space includes network bandwidth, latency information and user request data, the action space includes a set of all executable operations, the transition probability is the probability of a state executing an action and transitioning to another state, and the reward function is a quantified result of the user experience quality.

[0012] Furthermore, in the above-mentioned user experience-aware service function chain intelligent orchestration method, the Markov decision model is constructed for the service function chain orchestration process, specifically including:

[0013] At each time step, for the state information S t ∈S, action A executed t ∈A, the reward value R obtained after executing the action t+1 ∈R, the new state S after executing the action t+1 , under the conditions of certain states and actions, the probability of the action occurring at time t is

[0014] p{s * ,r|s,a}=Pr{S t =s * ,R t =r|S t-1 =s,A t-1 =a}

[0015] Among them, s * ∈S is the state at time t, s is the state of the previous jump, and a is the previous action.

[0016] Furthermore, in the above-mentioned user experience-aware service function chain intelligent orchestration method, the quantified result of the user experience quality specifically includes:

[0017] Based on the WFL law and IQX hypothesis, data analysis and fitting of user service performance quality are performed to obtain quantitative results of user experience quality.

[0018] Furthermore, the user experience-aware service function chain intelligent orchestration method specifically includes:

[0019] For the t-th user service performance quality indicator qos of service function chain c, we have

[0020]

[0021] Among them, α p , β p , γ p and θ p is a constant parameter;

[0022] Combining the above formulas, we can obtain the user experience quality qoe as the reward function in reinforcement learning:

[0023]

[0024] Furthermore, in the above-mentioned user experience-aware service function chain intelligent orchestration method, during the service function chain intelligent orchestration process, the operating cost opex of loading the remote virtual machine vm stored in the image repository and instantiating the virtual network function vnf is

[0025]

[0026] Where j represents the j-th virtual machine vm, i represents the i-th virtual network function vnf enabled on the j-th virtual machine, Indicates the opex of the virtual machine corresponding to the i-th function startup. Indicates the opex of starting the corresponding vnf instance. If a potential vnf instance is selected for instantiation, its opex is opex normal ;

[0027] Then the opex of service function chain c is the sum of all vnf instances, that is,

[0028]

[0029] Define the immediate reward associated with service function chain c as

[0030]

[0031] For vnf instances participating in the construction of service function chain c, the rewards are evenly distributed, that is,

[0032]

[0033] According to a second aspect of the present invention, there is also provided a user experience-aware service function chain intelligent arrangement device, comprising:

[0034] The model building module is used to build a Markov decision model for the service function chain orchestration process, define the state space, action space, transition probability and reward function, initialize the two-layer convolutional neural network, and initialize the experience replay memory library;

[0035] A strategy acquisition module is used to acquire network bandwidth, latency information and user request data, encode them into feature vectors as state representations and input them into the two-layer convolutional neural network to acquire action strategies including actions;

[0036] The execution module is used to select and execute actions according to the current state and action strategy at each time step to obtain new states and observed rewards;

[0037] The update module is used to replay the state, executed actions and observed rewards into the memory bank, randomly extract samples from it for learning, update the neural network parameters using the gradient descent method, evaluate the gradient of each function according to the loss function, and update the parameters based on the dynamically adjusted learning rate.

[0038] According to the third aspect of the present invention, there is also provided a user experience-aware service function chain intelligent orchestration device, which includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of any one of the above-mentioned methods.

[0039] According to the fourth aspect of the present invention, a storage medium is also provided, which stores a computer program that can be executed by a user experience-aware service function chain intelligent orchestration device. When the computer program runs on the user experience-aware service function chain intelligent orchestration device, the user experience-aware service function chain intelligent orchestration device executes the steps of any one of the above-mentioned methods.

[0040] According to a fifth aspect of the present invention, there is also provided a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0041] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0042] The present invention provides a user experience perception-based service function chain intelligent orchestration method, which solves the problem of insufficient optimization of user experience quality in the existing service function chain orchestration process by introducing an improved algorithm in deep reinforcement learning, thereby achieving efficient utilization of network resources and improvement of user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of a user experience-aware service function chain intelligent orchestration method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0047] Figure 1 A flowchart of a user experience-aware service function chain intelligent orchestration method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the embodiment of the present application provides a user experience-aware service function chain intelligent orchestration method, comprising the following steps:

[0048] Construct a Markov decision model for the service function chain orchestration process, define the state space, action space, transition probability and reward function, initialize the two-layer convolutional neural network, and initialize the experience replay memory library;

[0049] Obtaining network bandwidth, latency information and user request data, encoding them into feature vectors as representations of states and inputting them into the two-layer convolutional neural network to obtain action strategies including actions;

[0050] At each time step, an action is selected and executed based on the current state and action strategy to obtain a new state and observation reward;

[0051] The state, executed actions, and observed rewards are replayed into the memory bank, from which samples are randomly drawn for learning, and the neural network parameters are updated using the gradient descent method. The gradient of each function is evaluated according to the loss function, and the parameters are updated based on the dynamically adjusted learning rate.

[0052] Specifically, a Markov decision model (MDP) is constructed to define the state space, action space, transition probability and reward function. The state space includes network bandwidth, latency information, user requests and resource usage of service function chain nodes; the action space is a collection of optional service function instances or network nodes, which is used to build a service function chain; the transition probability is used to describe the possibility of the current state transferring to the next state through a specific action; the reward function comprehensively considers QoE, operating cost (OPEX) and quality of service (QoS). The design of this reward function not only considers the user's subjective experience, but also takes into account the efficiency of network resource use, providing an optimization target for intelligent orchestration.

[0053] In order to achieve intelligent optimization of reinforcement learning, the embodiment of the present application adopts a two-layer convolutional neural network as the algorithm agent, and initializes the experience replay memory library to store the state, action, reward and transfer information of the agent during the training process. The convolutional neural network takes network state information as input and outputs an action value function to guide the agent to select actions. The input feature vector of the network is generated by encoding the network bandwidth, delay information and user request data. The encoded feature is represented as a 5×5×16 three-dimensional matrix that can fully reflect the network environment and user needs. After the feature matrix is ​​input into the convolutional neural network, the network generates an action value function through calculation and outputs the best action strategy under the current state.

[0054] The agent selects the optimal action based on the current state and action strategy, such as selecting a network node or function instance in the service function chain for orchestration; the agent then executes the action and obtains a new environmental state and a corresponding reward value. These states, actions, rewards, and new state information will be stored in the experience replay memory library as a source of samples for subsequent learning. In order to optimize the performance of the agent, the embodiment of the present application randomly extracts samples from the memory library for training, and uses the gradient descent method to optimize the parameters of the convolutional neural network. The learning rate is dynamically adjusted during the training process, and the convergence of the network is accelerated by minimizing the loss value, so that the network can quickly adapt to user requests of different sizes and complex network environments.

[0055] In the design of the reward function, the quantitative results of QoE and the constraints of operating costs are combined, so that the intelligent agent can not only pursue high QoE when arranging the service function chain, but also avoid the problem of excessive operating costs caused by unreasonable resource allocation. Through the continuous iterative optimization of the reinforcement learning model, it can flexibly adapt to different network environments and demand changes.

[0056] The embodiment of the present application provides a method for intelligent orchestration of service function chains based on user experience perception, which solves the problem of insufficient optimization of user experience quality in the existing service function chain orchestration process by introducing an improved algorithm in deep reinforcement learning, thereby achieving efficient utilization of network resources and improvement of user experience.

[0057] Optionally, in the user experience-aware service function chain intelligent orchestration method provided in the embodiment of the present application, the state space, action space, transition probability and reward function specifically include:

[0058] The state space includes network bandwidth, latency information and user request data, the action space includes a set of all executable operations, the transition probability is the probability of a state executing an action and transitioning to another state, and the reward function is a quantified result of the user experience quality.

[0059] Optionally, the user experience-aware service function chain intelligent orchestration method provided in the embodiment of the present application, wherein the Markov decision model is constructed for the service function chain orchestration process, specifically including:

[0060] At each time step, for the state information S t ∈S, action A executed t ∈A, the reward value R obtained after executing the action t+1 ∈R, the new state S after executing the action t+1 , under the conditions of certain states and actions, the probability of the action occurring at time t is

[0061] p{s * ,r|s,a}=Pr{S t =s * ,R t =r|S t-1 =s,A t-1 =a}

[0062] Among them, s * ∈S is the state at time t, s is the state of the previous jump, and a is the previous action.

[0063] Specifically, the SFC orchestration process involves problems such as embedding the current node and selecting the next node, and the selection of the next node is independent of its previous state. Since the current state in reinforcement learning is independent of all previous states, it conforms to the Markov property, so the SFC orchestration process is modeled by Markov (MDP). The agent interacts with the environment at each step in a discrete time series (t = 0, 1, 2, 3...). At each step, the agent receives the state information S of the environment t ∈S, and based on this, choose an action A t∈A, after each step, as a result of its action, the agent receives a reward value R t+1 ∈R, and is in a new state S t+1 , the MDP and the individuals together produce a sequence or trajectory that looks like this:

[0064] S 0 ,A 0 ,R 1 ,S 1 ,A 1 ,R 2 ,S 2 ,A 2 ,R 3 ,…

[0065] Among them, s * ∈S is the state at time t, s is the state of the previous jump, and a is the previous action. Under the conditions of the previous state and action value, the probability of these values ​​occurring at time t is:

[0066] p{s * ,r|s,a}=Pr{S t =s * ,R t =r|S t-1 =s,A t-1 =a}.

[0067] Optionally, in the user experience-aware service function chain intelligent orchestration method provided in the embodiment of the present application, the quantified result of the user experience quality specifically includes:

[0068] Based on the WFL law and IQX hypothesis, data analysis and fitting of user service performance quality are performed to obtain quantitative results of user experience quality.

[0069] Specifically, the WFL law, also known as the Weber-Fechner Law (WFL), is an important principle in psychophysics that describes the relationship between the intensity of a physical stimulus and its perceived intensity. For the human sensory system, in many cases, this dependence is logarithmic in nature. The WFL law describes QoE as a function under positively correlated QoS (such as bandwidth). In network services, QoS focuses on the differences in service management and service provision from the network perspective, while QoE evaluates the quality of services and the network that supports the services from the perspective of end users. Quantitative QoE research has shown that in several different scenarios, it can be observed that there is a similar logarithmic relationship between the size of QoS parameters of certain positively correlated characteristics of the communication system and the final QoE observed at the user end in appropriate user trials. This basic principle links human perception to the relative change of external stimuli. Formally, the differential perception dP is proportional to the relative change dS / S of the physical stimulus. The relationship between QoS and QoE based on the WFL law can be expressed as follows:

[0070] dQoS∝QoS*dQoE(WFL)

[0071] The purpose of the IQX hypothesis is to describe QoE as a function of a single impairment factor (such as packet loss rate). The IQX hypothesis connects QoE and QoS parameters through an exponential relationship. This formula links the change in QoE relative to QoS to the current QoE level. The matching is simple and its limit behavior is easy to explain. The exponential matching of negatively correlated QoS parameters provided by the IQX hypothesis is better than some logarithmic matching. The relationship between QoS and QoE based on the IQX hypothesis can be expressed as follows:

[0072] dQoE′∝QoE′*dQoS(IQX).

[0073] Optionally, the user experience-aware service function chain intelligent orchestration method provided in the embodiment of the present application specifically includes:

[0074] For the t-th user service performance quality indicator qos of service function chain c, we have

[0075]

[0076]

[0077] Among them, α p , β p , γ p and θ p is a constant parameter;

[0078] Combining the above formulas, we can obtain the user experience quality qoe as the reward function in reinforcement learning:

[0079]

[0080] Specifically, by combining the WFL law and the IQX hypothesis, QoE can be comprehensively quantified through QoS data. On the surface, WFL and IQX give a contradictory relationship between QoE and QoS, but WFL and IQX are applicable to QoS indicators with different tendencies. For positive QoS indicators (the larger the value, the better), the corresponding QoE in the formula can be derived according to WFL, and for negative QoS indicators, the corresponding QoE in the formula can be derived according to IQX. is the t-th QoS indicator of service function chain c. p , β p , γ p ,θ p etc. are constant parameters.

[0081] For the t-th user service performance quality indicator qos of service function chain c, we have

[0082]

[0083] Since the QoS of real networks is a mixture of positively correlated and negatively correlated indicators, the above formula is comprehensively improved. The overall QoE of the service function chain can be derived from the following formula, and the obtained QoE is directly used as a reward in reinforcement learning.

[0084]

[0085] Optionally, in the user experience-aware service function chain intelligent orchestration method provided in the embodiment of the present application, during the service function chain intelligent orchestration process, the operating cost OPEX of loading the remote virtual machine vm stored in the image repository and instantiating the virtual network function vnf is

[0086]

[0087] Where j represents the j-th virtual machine vm, i represents the i-th virtual network function vnf enabled on the j-th virtual machine, Indicates the opex of the virtual machine corresponding to the i-th function startup. Indicates the opex of starting the corresponding vnf instance. If a potential vnf instance is selected for instantiation, its opex is opex normal ;

[0088] Then the opex of service function chain c is the sum of all vnf instances, that is,

[0089]

[0090] Define the immediate reward associated with service function chain c as

[0091]

[0092] For vnf instances participating in the construction of service function chain c, the rewards are evenly distributed, that is,

[0093]

[0094] Specifically, when orchestrating SFC, failure to effectively optimize resource utilization may cause operators to deploy and maintain more VNF instances, thereby increasing operational overhead. The instantiation of new VNF instances will suffer greater cost penalties because it may result in loading remote VM images stored in the image repository and instantiating VNF instances, which is a penalty for rewards. The OPEX of a VNF instance is defined as follows:

[0095]

[0096] Where j represents the j-th virtual machine vm, i represents the i-th virtual network function vnf enabled on the j-th virtual machine, Indicates the opex of the virtual machine corresponding to the i-th function startup. Indicates the OPEX of starting the corresponding VNF instance. Once a potential VNF instance is instantiated and selected, it is a deployed VNF instance, and its OPEX penalty is OPEX normal , which is the normal OPEX (e.g., normal energy consumption). Therefore, the OPEX penalty for chain c is the sum of all VNF instances along c, as follows:

[0097]

[0098] According to the previous definitions of QoE gain, QoS constraint penalty, and OPEX penalty, the immediate reward associated with chain c is defined as follows:

[0099]

[0100] For VNF instances participating in the construction of chain c, the reward of chain c is c Evenly distribute among them, expressed as:

[0101]

[0102] Simulation experiments show that the improved DQN algorithm can significantly improve the user experience quality of the service function chain. Its QoE score is about two percentage points higher than that of the general orchestration method. At the same time, it shows better resource utilization in multiple network nodes and user request scenarios. The introduction of the two-layer convolutional neural network has significant advantages in network environment modeling and state feature extraction. The optimized learning rate adjustment mechanism further improves the convergence speed and stability of the algorithm. Compared with traditional genetic algorithms, ant colony algorithms and heuristic search methods, the orchestration efficiency and performance in complex scenarios are significantly improved. This user experience perception-based intelligent orchestration method for service function chains not only provides new theoretical support for the optimal design of service function chains, but also provides a technical reference for the practical application of future network function virtualization (NFV) and software defined networks (SDN).

[0103] The embodiment of the present application also provides a service function chain intelligent arrangement device, including:

[0104] The model building module is used to build a Markov decision model for the service function chain orchestration process, define the state space, action space, transition probability and reward function, initialize the two-layer convolutional neural network, and initialize the experience replay memory library;

[0105] A strategy acquisition module is used to acquire network bandwidth, latency information and user request data, encode them into feature vectors as state representations and input them into the two-layer convolutional neural network to acquire action strategies including actions;

[0106] The execution module is used to select and execute actions according to the current state and action strategy at each time step to obtain new states and observed rewards;

[0107] The update module is used to replay the state, executed actions and observed rewards into the memory bank, randomly extract samples from it for learning, update the neural network parameters using the gradient descent method, evaluate the gradient of each function according to the loss function, and update the parameters based on the dynamically adjusted learning rate.

[0108] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0109] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0110] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program code.

[0115] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0116] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0117] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A user experience-aware service function chain intelligent arrangement method, characterized in that: include: Construct a Markov decision model for the service function chain orchestration process, define the state space, action space, transition probability and reward function, initialize the two-layer convolutional neural network, and initialize the experience replay memory library; Obtaining network bandwidth, latency information and user request data, encoding them into feature vectors as representations of states and inputting them into the two-layer convolutional neural network to obtain action strategies including actions; At each time step, an action is selected and executed based on the current state and action strategy to obtain a new state and observation reward; The state, executed actions, and observed rewards are replayed into the memory bank, from which samples are randomly drawn for learning, and the neural network parameters are updated using the gradient descent method. The gradient of each function is evaluated according to the loss function, and the parameters are updated based on the dynamically adjusted learning rate.

2. The method for intelligently arranging service function chains based on user experience perception according to claim 1, characterized in that: The state space, action space, transition probability and reward function specifically include: The state space includes network bandwidth, latency information and user request data, the action space includes a set of all executable operations, the transition probability is the probability of a state executing an action and transitioning to another state, and the reward function is a quantified result of the user experience quality.

3. The user experience-aware service function chain intelligent arrangement method according to claim 1, characterized in that: The Markov decision model is constructed for the service function chain arrangement process, specifically including: At each time step, for the state information S t ∈S, action A executed t ∈A, the reward value R obtained after executing the action t+1 ∈R, the new state S after executing the action t+1 , under the conditions of certain states and actions, the probability of the action occurring at time t is p{s * ,r|s,a}=Pr{S t =s*,R t =r|S t-1 =s,A t-1 =a} Among them, s * ∈S is the state at time t, s is the state of the previous jump, and a is the previous action.

4. The method for intelligently arranging service function chains based on user experience perception according to claim 1, characterized in that: The quantitative results of the user experience quality specifically include: Based on the WFL law and IQX hypothesis, data analysis and fitting of user service performance quality are performed to obtain quantitative results of user experience quality.

5. The method for intelligently arranging service function chains based on user experience perception according to claim 4, characterized in that: Specifically include: For the t-th user service performance quality indicator qos of service function chain c, we have Among them, α p , β p , γ p and θ p is a constant parameter; Combining the above formulas, we can obtain the user experience quality qoe as the reward function in reinforcement learning:

6. The method for intelligently arranging service function chains based on user experience perception according to claim 5, characterized in that: During the intelligent orchestration of the service function chain, the operating cost opex of loading the remote virtual machine vm stored in the image repository and instantiating the virtual network function vnf is Where j represents the j-th virtual machine vm, i represents the i-th virtual network function vnf enabled on the j-th virtual machine, Indicates the opex of the virtual machine corresponding to the i-th function startup. Indicates the opex of starting the corresponding vnf instance. If a potential vnf instance is selected for instantiation, its opex is opex normal ; Then the opex of service function chain c is the sum of all vnf instances, that is, Define the immediate reward associated with service function chain c as For vnf instances participating in the construction of service function chain c, the rewards are evenly distributed, that is, 7. A user experience-aware service function chain intelligent arrangement device, characterized in that: include: The model building module is used to build a Markov decision model for the service function chain orchestration process, define the state space, action space, transition probability and reward function, initialize the two-layer convolutional neural network, and initialize the experience replay memory library; A strategy acquisition module is used to acquire network bandwidth, latency information and user request data, encode them into feature vectors as state representations and input them into the two-layer convolutional neural network to acquire action strategies including actions; The execution module is used to select and execute actions according to the current state and action strategy at each time step to obtain new states and observed rewards; The update module is used to replay the state, executed actions and observed rewards into the memory bank, randomly extract samples from it for learning, update the neural network parameters using the gradient descent method, evaluate the gradient of each function according to the loss function, and update the parameters based on the dynamically adjusted learning rate.

8. A user experience-aware service function chain intelligent orchestration device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: It stores a computer program that can be executed by a user experience-aware service function chain intelligent orchestration device. When the computer program runs on the user experience-aware service function chain intelligent orchestration device, the user experience-aware service function chain intelligent orchestration device executes the steps of the method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the user experience-aware service function chain intelligent orchestration method according to any one of claims 1 to 6 are implemented.

Images