High-reliability service function chain deployment method based on dual deep Q network

Abstract

The invention discloses a high-reliability service function chain deployment method based on a dual deep Q network, and the method comprises the following steps: 1, building a satellite network model and an SFC model; step 2, establishing an optimization problem model; step 3, modeling the SFC deployment problem as a Markov decision process MDP, and designing a state space including a topological structure of a satellite network and available computing resource capacity of nodes; designing an action space, wherein the action space comprises a main node and a backup node selected for a virtual network function VNF; a reward function comprehensively considers main path end-to-end time delay and SFC reliability, then a high-reliability SFC deployment algorithm based on DDQN is adopted, and a main satellite node and a backup satellite node are sequentially selected for each VNF through a two-stage deployment strategy, namely a main path stage and a backup path stage. The method aims to cope with challenges brought by deployment of high-reliability service function chains in a satellite network with limited resources and topology dynamic changes to the maximum extent.

Description

Technical Field

[0001] To address the challenges brought about by deploying high-reliability service function chains (SFCs) in satellite networks with resource constraints and dynamic topologies, the present invention models the SFC deployment problem as a Markov decision process (MDP) and proposes a high-reliability service function chain deployment algorithm HRSFCD (High Reliability Service Function Chains Deployment) based on the double deep Q-network (DDQN). Background Art

[0002] With the rapid development of the Internet, in addition to traffic forwarding, satellites also undertake computing tasks. Due to the high cost of satellite launch and operation, it has become a trend to carry multiple computing tasks on the same satellite. At the same time, as one of the 5G application scenarios, high-reliability and low-latency communication places extremely high requirements on the performance of satellite networks. However, the dynamic nature of satellite network topologies and node failures are likely to cause service interruptions. In traditional network architectures based on dedicated hardware, providing backups for satellite nodes is a feasible way to improve service reliability and ensure the normal operation of services. However, more backups will result in more resource consumption.

[0003] Network function virtualization technology decouples network functions from dedicated hardware, enabling satellites to undertake multiple computing tasks and significantly improving resource utilization. Therefore, network services based on satellite platforms not only face the risk of hardware failures but also need to consider the potential problems caused by software failures. Therefore, how to comprehensively consider hardware reliability and software reliability in satellite networks with limited resources and dynamic topologies, design solutions that can improve the reliability of SFCs, and propose deployment strategies to ensure the high reliability of SFCs has become the key to solving the problem.

[0004] When deploying SFCs in satellite networks, since the probability of multiple failure events occurring simultaneously is extremely low, the present invention focuses on the single-point failure scenario, that is, at most one hardware or software failure event occurs, and effectively balances the end-to-end delay and the reliability of SFCs by optimizing the deployment strategy. Summary of the Invention

[0005] To solve the technical problems mentioned in the above background art, the technical solution adopted by the present invention is a highly reliable service function chain deployment method based on a dual deep Q network. The method includes the following steps. Step 1, construct a satellite network model, including satellite nodes, the computing resource capacity of the nodes, the hardware failure probability of the nodes, physical links, and the shortest propagation delay between nodes. Step 2, construct a service function chain model, including SFC requests, VNFs, the maximum tolerable end-to-end delay, the computing resource requirements of VNFs, and the software failure probability of VNFs. Step 3, construct an optimization problem model based on the satellite network model and the service function chain model, including the end-to-end delay model of the main path, the SFC reliability model, and the constraint conditions.

[0006] Further, the specific implementation process of constructing the satellite network model in Step 1 is as follows;

[0007] The satellite network consists of several satellite nodes and the links between the nodes, which is represented by a connected graph G(V, E). Each satellite node has a certain computing resource and has a certain probability of failure. Among them, V = {v1, v2,..., v k ,...v |V|} represents the set of all satellite nodes v k , and |V| represents the number of satellite nodes in the network. represents the set of available computing resource capacities of all satellite nodes at time slot t, where represents the available computing resource capacity of v k at time slot t. Ph = {Ph1, Ph2,... Ph |V|} represents the hardware failure probabilities of all satellite nodes. E represents the set of physical links between satellite nodes. represents the matrix of the shortest propagation delays between satellite nodes at time slot t.

[0008] The specific implementation process of constructing the service function chain model in Step 2 is as follows;

[0009] Define SR = {SR1, SR2,... SR |Sr|} as the set of SFCs requests, and |Sr| represents the number of SFCs. Use the tuple <VNF i , D i , Cd i , Ps i > to represent the request SR i ∈ SR. represents the set of VNFs that make up the request SR i , where f i,j represents the jth VNF. D i represents SR iThe maximum tolerable end-to-end delay. Represents the computing resource requirements of each VNF. Represents the software failure probability of each VNF.

[0010] The specific implementation process of constructing the optimization problem model in step 3 above is as follows;

[0011] Satellite network applications such as real-time communication are sensitive to delay and reliability. High delay will lead to a decline in service quality, and low reliability will increase the risk of service interruption. Therefore, end-to-end delay and reliability are used as the indicators of the optimization problem.

[0012] First, establish an end-to-end delay model for the primary path.

[0013] The primary path is the path composed of the primary satellite nodes of all VNFs and the links between them. Select satellite nodes for each VNF in sequence according to the VNF number. The first deployed VNF is called the starting VNF, denoted by There is one or more ending VNFs. Based on this, the end-to-end delay Dp of the primary path of the SFC request SR i Is defined as the maximum value of the shortest delay between the starting VNF and the ending VNFs. The formula is as follows: i

[0014]

[0015] Where k1, k2 represent satellite nodes, and f i,j Represents the jth VNF in the ith SFC request, and C represents the set composed of the ending VNFs. Is a binary variable, Indicates that the VNF software Is deployed on the primary hardware node Above.

[0016] Secondly, establish an SFC reliability model.

[0017] Define the reliability of SR i As the probability that SR i Does not fail, denoted by Since a primary hardware node and a backup hardware node are selected for each VNF software of SR i , the reliability of SR i Is Related to the reliability of the primary path And the reliability of the backup path .

[0018] (a) Calculation of;

[0019] Reliability of the primary path refers to the probability that all VNF software and all the main hardware nodes where the VNF software is located are working properly. For SR i the probability that all VNF software is working properly is denoted by Rs i and the calculation formula is:

[0020]

[0021] where Ps i,j represents the software failure probability of VNF software f i,j , then 1 - Ps i,j represents the software reliability of f i,j . For SR i the probability that all the main hardware nodes where the VNF software is located are working properly is denoted by Rh i and the calculation formula is:

[0022]

[0023] where Ph k represents the hardware failure probability of node v k , then 1 - Ph k represents the hardware reliability of v k . represents that when different VNFs are deployed to the same node v k , the hardware reliability of v k only participates in the calculation once. Therefore, the main path reliability is calculated as follows:

[0024]

[0025] (b) Calculation;

[0026] Deploying VNF software f i,j on satellite node v k is called a VNF instance If v k is the main node of f i,j , when f i,j has a software failure or v k has a hardware failure resulting in the instance being unable to work properly, then the data stream that should have been processed by the main node v k will be forwarded to the backup node of f i,j for processing. At this time, the backup node of f i,j , the main nodes of other VNF software, and the links between the nodes constitute a backup path of SR i . This backup path formed due to the instance The backup path used in case of failure is represented by. Since a backup node is selected for each VNF software, under the premise of single-point failure mentioned in this paper, SR i has 1 primary path and |VNF i | backup paths. The reliability of the backup path of SR i refers to the sum of the reliabilities of all backup paths of SR refers to SR i The sum of the reliabilities of all backup paths of

[0027] For the backup path There are two reasons for its operation. First, the instance The VNF software f of i,j fails but the primary hardware node v k is normal. In this case, the hardware nodes participating in the operation in the path can be divided into two categories, namely the backup hardware node of software f i,j and the primary hardware nodes where other VNF software is located. Traverse all hardware nodes, and when at least belonging to one of the above two categories of nodes, add the hardware reliability of the node to the calculation. Therefore, the probability that all the hardware on the backup path works properly is expressed as where is used to determine whether it belongs to the primary hardware node where other VNF software is located. is a binary variable, indicating that f i,j is deployed to the backup node at time slot t, and is used in this formula to determine whether it is the backup hardware node of software f i,j . All the VNF software of SR i works properly, and the probability is expressed in the same way as in formula (2). Therefore, in this case, the reliability of the backup path is expressed as:

[0028]

[0029] where represents the probability that all the hardware on the backup path works properly, Rs i represents the probability that all the VNF software on the backup path works properly, and Ps i,j ·(1 - Ph k ) represents the probability that the VNF software f i,j fails but the primary hardware node v k is normal.

[0030] Second, the instance The VNF software f ofi,j Normal but the main hardware node v k fails. In this case, the hardware nodes participating in the work on the path can be divided into two categories, namely the backup hardware nodes of the VNF software with the main node being v k , and the main hardware nodes where other VNF software is located. Since v k fails, all hardware nodes except v k are traversed. When it belongs to at least one of the above two categories of nodes, the hardware reliability of this node is added to the calculation. Therefore, the probability that all the hardware on the backup path works properly is expressed as where is used to determine whether it belongs to the main hardware node where other VNF software is located, is used to determine whether it belongs to the backup hardware node of the VNF software with the main node being v k . The probability that all the VNF software of SR i works properly is expressed in the same way as in Equation (2). Therefore, the reliability of the backup path in this case is represented by , and the calculation formula is as follows:

[0031]

[0032] where represents the probability that all the hardware on the backup path works properly, Rs i represents the probability that all the VNF software on the backup path works properly, and (1 - Ps i,j )·Ph k represents the probability that the VNF software f i,j is normal but the main hardware node v k fails.

[0033] Therefore, the reliability of the backup path is expressed as:

[0034]

[0035] where, represents the reliability of the backup path in the case where the VNF software f i,j fails but the main hardware node v k is normal, represents the reliability of the backup path in the case where the VNF software f i,j is normal but the main hardware node v k fails.

[0036] Furthermore, SR i The reliability of all backup paths It is expressed as:

[0037]

[0038] Finally, SR i Reliability The calculation formula is:

[0039]

[0040] in Indicates SR i The reliability of the primary path, Indicates SR i The reliability of all backup paths.

[0041] However, the deployment of service function chains is subject to some restrictions.

[0042] (a) The computing resources of satellite nodes cannot be oversubscribed by VNFs.

[0043]

[0044] (b) Each VNF can only select one primary satellite node and one backup satellite node.

[0045]

[0046] (c) For any f i,j ∈VNF i , does not allow f i,j The primary and backup nodes are the same.

[0047]

[0048] (d) The end-to-end delay of an SFC cannot exceed its delay constraint.

[0049]

[0050] Finally, the optimization problem model is established.

[0051] To optimize the end-to-end latency and reliability of the primary path of the SFC, the optimization objectives are defined as:

[0052] obj:ω·Dp i +(1-ω)·R i (15)

[0053] Where ω∈[0,1] represents the weight coefficient, Dp i and R irespectively represent the normalized end-to-end delay Dp i and the normalized SFC reliability

[0054] It is necessary to maximize the value of obj:

[0055] max obj(16)

[0056] s.t.:(11)-(15).

[0057] To solve problem (16), the SFC deployment problem is modeled as an MDP, and an algorithm based on DDQN is proposed to solve this problem.

[0058] First, establish the MDP model.

[0059] The SFC deployment problem is a sequential decision-making problem. Therefore, it can be modeled as a Markov decision process, represented as a tuple (S, A, R), where S represents the state of the environment, A represents the actions of the agent, and R represents the reward.

[0060] (a) State: The state S t is composed of the topology structure TS of the satellite network t and the computing

[0061] resource capacity Cr of all satellites t It is expressed as:

[0062] S t =(TS t , Cr t ) (17)

[0063] (b) Action: The agent's deployment of the SFC request is divided into two stages. In the first stage, the primary satellite node is selected for each VNF. In the second stage, the backup satellite node is selected for each VNF. In both stages, the action space of the agent is the set V of all satellite nodes.

[0064] (c) Reward: The reward is divided into stage reward and overall reward. For the stage reward, every time the agent selects a satellite node for a VNF, with this VNF as the end VNF, the end-to-end delay and reliability are calculated respectively through equations (1) and (9), and the reward value is calculated through equation (15). For the overall reward, when the agent has completed the selection of the primary node and the backup node for all VNFs, the sum of the rewards of all previous actions is used as the overall reward. In addition, if the agent's action violates the constraint, a penalty value Ρ=-50 is given as the reward value of the action.

[0065] Then, the proposed HRSFCD algorithm is given.

[0066] In the action space, the allocation decision of satellite nodes is a discrete action. Therefore, this paper uses the DDQN algorithm for processing discrete actions as the basic algorithm. The agent interacts with the satellite network environment to train the Q-network. During the training process, the deployment of an SFC is divided into two stages. In the first stage, the Q-network sequentially selects the primary satellite node for each VNF, and in the second stage, the Q-network sequentially selects the backup satellite node for each VNF.

[0067] Compared with the prior art, the present invention comprehensively considers software reliability and hardware reliability in a dynamic network environment and uses the same Q-network to complete the deployment of the primary node and the backup node respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is an SFC example diagram.

[0069] Figure 2 It is a composition structure diagram of SFC reliability.

[0070] Figure 3 It is a framework diagram of the HRSFCD algorithm. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0071] The present invention will be described in detail below with reference to the drawings and embodiments.

[0072] Satellite network model

[0073] The satellite network is represented by a connected graph G(V, E), where V = {v1, v2,... v |V|} represents the set of all satellite nodes, and |V| represents the number of satellite nodes. represents the set of satellite available computing resource capacities at time slot t, where represents the available computing resource capacity of v k at time slot t. Ph = {Ph1, Ph2,... Ph |V|} represents the hardware failure probabilities of all satellite nodes. E represents the set of physical links between satellite nodes. represents the matrix of the shortest propagation delays between satellite nodes at time slot t. An example of an SFC is as Figure 1 shown.

[0074] Service function chain model

[0075] Define SR = {SR1, SR2,... SR |Sr|} as the set of SFCs requests, and |Sr| represents the number of SFCs. Use the tuple <VNF i , D i , Cd i , Ps i > to represent the request SRi ∈SR. Represents the set of VNFs that make up the request SR i where f i,j represents the j-th VNF. D i Represents SR i The maximum end-to-end delay that SR can tolerate. Represents the computing resource requirements of each VNF. Represents the software failure probability of each VNF.

[0076] (1) Optimization problem model

[0077] Satellite network applications such as real-time communication are sensitive to delay and reliability. High delay will lead to a decline in service quality, and low reliability will increase the risk of service interruption. Therefore, end-to-end delay and reliability are used as the indicators of the optimization problem.

[0078] First, establish the end-to-end delay model of the primary path.

[0079] In the actual scenario, since the probability of system failure is not high, the probability of using backup nodes is small. Therefore, only the primary path is considered, that is, the path composed of the primary satellite nodes of all VNFs and the links between them. Select satellite nodes for each VNF in sequence according to the VNF number. The first deployed VNF is called the starting VNF, denoted by . There is one or more ending VNFs. Based on this, the end-to-end delay Dp of the primary path of the SFC request SR i is defined as the maximum value of the shortest delay between the starting VNF and the ending VNFs. The formula is as follows: i where C represents the set composed of the ending VNFs.

[0080]

[0081] is a binary variable, indicating that the VNF software is deployed on the primary hardware node .

[0082] Secondly, establish the SFC reliability model.

[0083] Define the reliability of SR i as the probability that SR i does not fail, denoted by . Since a primary hardware node and a backup hardware node are selected for each VNF software of SR i , the reliability of SR i ​​ Related to the reliability of the primary path and the reliability of the backup path Since using lasers as inter-satellite links has become a trend in satellite communication, and lasers have high directivity and short wavelengths in space transmission, satellite laser communication can achieve high speed, wide bandwidth, high precision, and high security without electromagnetic spectrum limitations, so only the reliability of satellite nodes is considered. As Figure 2 shown, the reliability of the SFC consists of the primary path reliability, the backup path reliability in the case of single-point failures, and the backup path reliability in the case of multi-point failures, where multi-point failures refer to the situation where 2 or more VNF software or hardware nodes fail simultaneously.

[0084] (a) Calculation of

[0085] Reliability of the primary path Refers to the probability that all VNF software and all the primary hardware nodes where the VNF software is located are all working properly. Denote the probability that all VNF software in SR i works properly as Rs i , and the calculation formula is:

[0086]

[0087] where 1 - Ps i,j represents the software reliability of VNF software f i,j . Denote the probability that all the primary hardware nodes where the VNF software in SR i are all working properly as Rh i , and the calculation formula is:

[0088]

[0089] where 1 - Ph k represents the hardware reliability of node v k . Indicates that when different VNFs are deployed on the same node v k , the hardware reliability of v k is only involved in the calculation once. Therefore, the calculation method of the primary path reliability is as follows:

[0090]

[0091] (b) Calculation of

[0092] Deploying VNF software f i,j on satellite node v k is called a VNF instance If v k is the primary node of f i,j and when a software failure occurs in f i,j or a hardware failure occurs in v k resulting in the instance being unable to work properly, then the data stream that should have been processed by the primary node v k will be forwarded to the backup node of f i,j for processing. At this time, the backup node of f i,j , the primary nodes of other VNF software, and the links between the nodes constitute a backup path of SR i . Denote the backup path used due to the failure of the instance by . Since a backup node is selected for each VNF software, under the premise of single-point failure mentioned in this paper, SR i has 1 primary path and |VNF i | backup paths. The reliability of the backup path of SR i refers to the sum of the reliabilities of all backup paths of SR . i

[0093] For the backup path , there are two reasons for its operation. First, a failure occurs in the VNF software f of the instance i,j but the primary hardware node v k is normal. In this case, the hardware nodes participating in the operation in the path can be divided into two categories, namely the backup hardware node of the software f i,j and the primary hardware nodes where other VNF software is located. Traverse all hardware nodes, and when at least one of the above two categories of nodes is met, add the hardware reliability of that node to the calculation. Therefore, the probability that all the hardware on the backup path is working properly is expressed as where is used to determine whether it belongs to the primary hardware node where other VNF software is located. is a binary variable indicating that f i,j is deployed to the backup node at time slot t and is used in this formula to determine whether it is the backup hardware node of the software f i,j . The probability that all VNF software of SR i is working properly is expressed in the same way as formula (2). Therefore, in this case, the reliability of the backup path is expressed as:[[]]

[0094] ​​

[0095] Second, example The VNF software f of i,j is normal but the main hardware node v k fails. In this case, the hardware nodes participating in the work in the path can be divided into 2 categories, namely the backup hardware nodes of the VNF software with the main node v k , and the main hardware nodes where other VNF software is located. Since v k fails, all hardware nodes except v k are traversed. When it belongs to at least one of the above 2 categories of nodes, the hardware reliability of this node is added to the calculation. Therefore, the probability that all the hardware on the backup path works normally is expressed as where is used to determine whether it belongs to the main hardware node where other VNF software is located, is used to determine whether it belongs to the backup hardware node of the VNF software with the main node v k . The probability that all the VNF software of SR i works normally is expressed in the same way as in Equation (2). Therefore, the reliability of the backup path in this case is represented by , and the calculation formula is as follows:

[0096]

[0097] Therefore, the reliability of the backup path is expressed as:

[0098]

[0099] Furthermore, the reliability i of all the backup paths of SR is expressed as:

[0100]

[0101] Finally, the calculation formula for the reliability i of SR is:

[0102]

[0103] Then, the deployment of the service function chain is restricted by some conditions.

[0104] (a) The computing resources of satellite nodes cannot be over-applied by VNF.

[0105]

[0106] (b) Each VNF can only select one primary satellite node and one backup satellite node.

[0107]

[0108] (c) For any f i,j ∈ VNF i , it is not allowed that the primary node and the backup node of f i,j are the same.

[0109]

[0110] (d) The end-to-end delay of the SFC shall not exceed its delay constraint.

[0111]

[0112] Finally, an optimization problem model is established.

[0113] The research of the present invention aims to optimize the end-to-end delay and reliability of the primary path of the SFC. Therefore, the optimization objective is defined as:

[0114] obj: ω·Dp i +(1 - ω)·R i (15)

[0115] where ω ∈ [0,1] represents the weight coefficient, Dp i and R i respectively represent the normalized end-to-end delay Dp i and the SFC reliability

[0116] It is necessary to maximize the value of obj:

[0117] max obj (16)

[0118] s.t.:(11)-(15).

[0119] (2) Algorithm design

[0120] First, an MDP model is established.

[0121] After the agent selects a satellite node for the VNF according to the environmental state, the available computing resource capacity of the node changes. Then, the agent makes the next action according to the new environment, and repeats the above process until it is processed. Therefore, the deployment problem of SFCs is modeled as a Markov decision process, represented as a tuple (S, A, R), where S represents the state of the environment, A represents the action of the agent, and R represents the reward.

[0122] (a) State: State S t Composed of the topological structure TS of the satellite network t and the computing resource capacity Cr of all satellites

[0123] Denoted as: t

[0124] S t =(TS t , Cr t ) (17)

[0125] (b) Action: The deployment of the SFC request by the agent is divided into two stages. In the first stage, the primary satellite node is selected for each VNF, and in the second stage, the backup satellite node is selected for each VNF. In both stages, the action space of the agent is the set V of all satellite nodes.

[0126] (c) Reward: The reward is divided into stage reward and overall reward. For the stage reward, every time the agent selects a satellite node for a VNF, taking this VNF as the end VNF, the end-to-end delay and reliability are calculated respectively through equations (1) and (9), and the reward value is calculated through equation (15). For the overall reward, when the agent has completed the selection of the primary and backup nodes for all VNFs, the sum of the rewards of all previous actions is used as the overall reward. Additionally, if the action of the agent violates the constraint, a penalty value Ρ = -50 is given as the reward value of the action.

[0127] Then, the proposed HRSFCD algorithm is given.

[0128] In the action space, the allocation decision of satellite nodes is a discrete action. Therefore, this paper adopts the DDQN algorithm for dealing with discrete actions as the basic algorithm. The framework of the HRSFCD algorithm is as Figure 3 shown. The agent interacts with the satellite network environment to train the Q network. During the training process, the deployment of an SFC is divided into two stages. In the first stage, the Q network sequentially selects the primary satellite node for each VNF, and in the second stage, the Q network sequentially selects the backup satellite node for each VNF.

[0129] Algorithm 1 HRSFCD Algorithm

[0130] Input: G(V,E), M t , Ph = {Ph1, Ph2,... Ph |V|}, <VNF i , D i , Cd i , Ps i >[[]]

[0131] Output: Q network​

[0132]

[0133] Lines 2 to 15 of the algorithm complete the work of the first stage, and select the primary satellite node for each VNF in sequence. Lines 16 to 27 of the algorithm complete the work of the second stage, and select the backup satellite node for each VNF in sequence.

Claims

1. A high-reliability service function chain deployment method based on a double deep Q-network, characterized in that: It includes the following steps: Step 1: Establish a satellite network model and an SFC model. The satellite network model and the service function chain model are respectively abstract descriptions of the satellite network environment and the SFC. Step 2: Establish an optimization problem model including the end-to-end delay of the primary path, the reliability of the SFC, and the constraints. Step 3: Model the SFC deployment problem as a Markov decision process MDP, and design the state space, including the topological structure of the satellite network and the available computing resource capacity of the nodes. Design the action space, including the primary node and the backup node selected for the virtual network function VNF; the reward function comprehensively considers the end-to-end delay of the primary path and the reliability of the SFC, and then adopts a high-reliability SFC deployment algorithm based on DDQN. Through a two-stage deployment strategy, namely the primary path stage and the backup path stage, select the primary satellite node and the backup satellite node for each VNF in turn to maximize the objective.

2. The high-reliability service function chain deployment method based on the double deep Q network according to claim 1, characterized in that: The implementation steps of Step 1 are as follows: Step 1.1: Establish a satellite network model The satellite network is represented by a connected graph G(V, E); where V = {v1, v2,... v |V|} represents the set of all satellite nodes, and |V| represents the number of satellite nodes; represents the set of available computing resource capacities of satellites in the t-th time slot, where represents v k 's available computing resource capacity in the t-th time slot; Ph = {Ph1, Ph2,... Ph |V|} represents the hardware failure probabilities of all satellite nodes; E represents the set of physical links between satellite nodes; represents the matrix of the shortest propagation delays between satellite nodes in the t-th time slot; Step 1.2: Establish an SFC model Define SR = {SR1, SR2,... SR |Sr|} as the set of SFCs requests, and |Sr| represents the number of SFCs; use the tuple <VNF i , D i , Cd i , Ps i > to represent the request SR i ∈ SR; represents the set of VNFs that make up the request SR i , where f i,j represents the j-th VNF; D i represents the maximum end-to-end delay that SR i can tolerate; represents the computing resource requirement of each VNF; represents the software failure probability of each VNF.

3. The high-reliability service function chain deployment method based on the double deep Q network according to claim 1, characterized in that: Step 2 includes: Step 2.1: Establish an end-to-end delay model of the primary path; The path consists of the main satellite nodes of all VNFs and the links between them. The satellite nodes are selected for each VNF in sequence according to the VNF sequence number. The first VNF to be deployed is called the starting point VNF. Indicates that there are one or more endpoint VNFs; SFC requests SR i The end-to-end delay Dp of the main path i It is defined as the maximum value of the shortest delay between the starting VNF and the ending VNFs; the formula is as follows: where C represents the set of destination VNFs; is a binary variable, indicating that the VNF software is deployed on the primary hardware node ; Step 2.2: Establish an SFC reliability model; Define the reliability of SR i as the probability that SR i does not fail, denoted by ; Since a primary hardware node and a backup hardware node are respectively selected for all VNF software of SR i , the reliability of SR i is related to the reliability of the primary path and the reliability of the backup path ; The reliability of the SFC consists of the reliability of the primary path, the reliability of the backup path in the case of single point of failure, and the reliability of the backup path in the case of multiple points of failure, where multiple points of failure refer to the situation that 2 or more VNF software or hardware nodes fail simultaneously; Step 2.2.1: Establish a primary path reliability model; Reliability of the main path It refers to the probability that all VNF software and all the main hardware nodes where all VNF software is located are working properly; the probability that all VNF software of SR i works properly is represented by Rs i and the calculation formula is as follows: Among them, 1 - Ps i,j represents the software reliability of VNF software f i,j ; the probability that all the main hardware nodes where all VNF software of SR i is located are all working properly is represented by Rh i , and the calculation formula is: Among them, 1-Ph k represents the hardware reliability of node v k ; represents that when different VNFs are deployed to the same node v k , the hardware reliability of v k only participates in the calculation once; the calculation method of the main path reliability is as follows: Step 2.2.2: Establish a backup path reliability model VNF software f i,j Deployed on satellite node v k It is called VNF instance If v k Yes i,j The master node, when f t,j A software failure or k A hardware failure caused the instance When it fails to work properly, the master node v k The processed data stream will be forwarded to f i,j Processing is performed on the backup node of f i,j The backup nodes of the VNF software and the master nodes of other VNF software and the links between the nodes constitute the SR i A backup path for the instance The backup path used in case of failure Indicates that a backup node is selected for each VNF software, so under the premise of the single point of failure, SR i There is 1 primary path and |VNF i | backup paths; SR i Backup path reliability Refers to SR i The sum of the reliabilities of all backup paths; For the backup path There are two reasons for its operation; firstly, the instance of the VNF software f i,j has a fault but the main hardware node v k is normal; the hardware nodes participating in the operation in the path are divided into two categories, namely the backup hardware nodes of the software f i,j and the main hardware nodes where other VNF software is located; traverse all hardware nodes, and when at least one of the above two types of nodes is met, the hardware reliability of the node is included in the calculation; the probability that all the hardware on the backup path works normally is expressed as where is used to determine whether it belongs to the main hardware node where other VNF software is located; is a binary variable, indicating that f i,j is deployed to the backup node at time slot t, and is used to determine whether it is the backup hardware node of the software f i,j ; the probability that all the VNF software of SR i works normally is expressed in the same way as formula (2); therefore, in this case, the reliability of the backup path is expressed as: Second, the VNF software f i,j is normal but the main hardware node v k has a fault. In this case, the hardware nodes participating in the work in the path are divided into two categories, namely, the backup hardware nodes of the VNF software with the main node as v k , and the main hardware nodes where other VNF software is located; due to the fault of v k , all hardware nodes except v k are traversed. When at least belonging to one of the above two categories of nodes, the hardware reliability of the main node is added to the calculation; the probability that all the hardware on the backup path works normally is expressed as where is used to determine whether it belongs to the main hardware node where other VNF software is located, is used to determine whether it belongs to the backup hardware node of the VNF software with the main node as v k ; the probability that all the VNF software of SR i works normally is expressed in the same way as in Equation (2); the reliability of the backup path is represented by and the calculation formula is as follows: Therefore, the reliability of the backup path is expressed as : Furthermore, the reliability of all backup paths of SR i is expressed as: ​ Finally, SR i reliability The calculation formula is as follows: Step 2.2.3: The restrictions on the deployment of the SFC; (a) The computing resources of the satellite nodes cannot be over-applied by the VNF; (b) Each VNF can only select one primary satellite node and one backup satellite node; (c) For any f i,j ∈ VNF i , it is not allowed that the primary node and the backup node of f i,j are the same; (d) The end-to-end delay of the SFC cannot exceed its delay constraint; Step 2.2.4: Establish an optimization problem model; Optimize the end-to-end delay and reliability of the primary path of the SFC, so the optimization objective is defined as: obj: ω·Dp i +(1 - ω)·R i (15) where ω ∈ [0, 1] represents the weight coefficient, Dp i and R i respectively represent the end-to-end delay Dp i after normalization and the SFC reliability It is necessary to maximize the value of obj: max obj (16) s.t.: (11)-(15).

4. The high-reliability service function chain deployment method based on the double deep Q network according to claim 1, characterized in that: Step 3 includes: Step 3.1 Establish an MDP model; After the agent selects a satellite node for the VNF according to the environmental state, the available computing resource capacity of the node changes, and then the agent makes the next action according to the new environment, repeating until it is processed; model the deployment problem of the SFCs as a Markov decision process, expressed as a tuple (S, A, R), where S represents the state of the environment, A represents the action of the agent, and R represents the reward; (a) Status: Status S t composed of the topological structure TS of the satellite network t and the computing resource capacity Cr of all satellites t constitute; expressed as: S t =(TS t , Cr t )(17) (b) Action: The deployment of the SFC request by the agent is divided into two stages; in the first stage, select the primary satellite node for each VNF, and in the second stage, select the backup satellite node for each VNF. In both stages, the action space of the agent is the set V of all satellite nodes. (c) Rewards: Rewards are divided into stage rewards and overall rewards. For stage rewards, every time the agent selects a satellite node for a VNF and takes this VNF as the end VNF, the end-to-end delay and reliability are calculated respectively through Equations (1) and (9), and the reward value is calculated through Equation (15). For overall rewards, when the agent has completed the selection of primary and backup nodes for all VNFs, the sum of the rewards for all previous actions is taken as the overall reward. Additionally, if the agent's action violates the constraints, a penalty value Ρ = -50 is given as the reward value for the action. Step 3.2 HRSFCD algorithm; In the action space, the allocation decision of satellite nodes is a discrete action, and the DDQN algorithm for dealing with discrete actions is used as the basic algorithm. The agent interacts with the satellite network environment to train the Q network. During the training process, the deployment of an SFC is divided into two stages. In the first stage, the Q network sequentially selects a primary satellite node for each VNF, and in the second stage, the Q network sequentially selects a backup satellite node for each VNF.

Images