Decryption traffic service chain orchestration method based on blockchain and deep reinforcement learning

Abstract

The present invention relates to a decryption traffic service chain orchestration method based on blockchain and deep reinforcement learning, and belongs to the field of information technology. The present invention establishes a non-convex linear mathematical optimization model with the goal of minimizing delay and energy consumption under the condition of network load changes, solves it using a reinforcement learning algorithm, trains agents through interaction with the physical network environment, and finds the optimal physical resource allocation and scheduling strategy for VNF scaling based on the state value of the physical node. The algorithm solves the problem of long service delay and high energy consumption under the condition of limited resources.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and specifically relates to a decryption traffic service chain orchestration method based on blockchain and deep reinforcement learning. Background Art

[0002] With the rapid development of Internet technology, especially in the 5G era, mobile communication services have shown explosive growth, leading to challenges for telecom operators in terms of large-scale device connections, diversified services and network performance requirements. Traditional communication networks are difficult to effectively meet these requirements, so new solutions are needed. Software-defined networking (SDN) and network function virtualization (NFV) technologies have emerged to achieve programming and flexible scheduling of networks by decoupling the control plane and data plane. The European Telecommunications Standards Institute (ETSI) proposed the NFV framework, which consists of virtualized network functions (VNFs), NFV infrastructure (NFVI) and NFV-MANO, and realizes the automated deployment, management and orchestration of network functions. However, in the face of complex network environments, traditional solutions still have problems such as trust, adaptability and real-time performance.

[0003] In order to meet these challenges, this paper proposes a decryption traffic service chain orchestration architecture based on blockchain and deep reinforcement learning. This architecture aims to combine the trust mechanism of blockchain and the adaptability of deep reinforcement learning to achieve real-time network optimization and resource allocation. By registering user information through the blockchain module and automatically adjusting resources using smart contracts, the system effectively improves the efficiency of network resource utilization while protecting user identity and transaction credibility. Under the SDN-VNF network, this architecture can achieve unified satisfaction of user service needs in different scenarios, solving the shortcomings of existing solutions.

[0004] Fu.J. et al. proposed an effective VNF deployment scheme, which consists of two parts: over-deployment and automatic migration to improve resource utilization and ensure on-demand capacity.

[0005] Qu.L. et al. proposed a reliability-aware and delay-constrained routing optimization framework for NFV-enabled data center networks, proposed a heuristic algorithm called greedy-shortest path, and developed an effective routing scheme.

[0006] Nguyen.HT et al. described the scaling problem as an observable Markov decision process with unknown scaling decisions, and applied the reinforcement learning algorithm to the scaling of VNFC, mapping the state space, action space, and reward space one by one to the scaling problem of VNFC.

[0007] Sharma.PK et al. proposed a new blockchain-based software-defined cloud architecture to establish a relationship between service providers and consumers. In the cloud layer, blockchain technology is used to supervise the process of data access and resource allocation. In the fog layer, the IoT network based on SDN and blockchain brings computing resources to the edge, achieving decentralization and complete privacy in the distributed cloud architecture based on blockchain.

[0008] The prior art has the following problems:

[0009] 1. In existing research, in the network virtualization scenario, many studies are based on the condition that the network load is static and unchanged, but ignore the resource limitations of the physical network when the load changes.

[0010] 2. Traditional service chain technology is generally implemented through routing cascade, which requires the creation of multiple subnets to connect multiple VNFs. It is also impossible to select different service chains for different businesses, resulting in poor performance and ease of use.

[0011] 3. Existing technologies mainly focus on the optimization of VNF deployment and resource allocation, and some solutions use machine learning methods such as deep reinforcement learning to deal with dynamic changes. However, these methods may not fully consider the trust mechanism of blockchain and the orchestration of decrypted traffic service chains. Summary of the invention

[0012] 1. Technical issues to be resolved

[0013] The technical problem to be solved by the present invention is how to provide a decryption traffic service chain orchestration method based on blockchain and deep reinforcement learning to solve problems such as trust, adaptability, real-time, security and privacy in resource allocation.

[0014] (II) Technical solution

[0015] In order to solve the above technical problems, the present invention proposes a decryption traffic service chain orchestration method based on blockchain and deep reinforcement learning. The method is applied to a trusted deployment system architecture based on blockchain and deep reinforcement learning. The system includes the following three layers: virtual network layer, trusted physical network layer and intelligent device layer. The method includes the following four stages:

[0016] Phase 1: Execute the login control smart contract, register user information on the blockchain module, request services through the login control protocol, and verify the relevant identity information of the request each time the service is requested;

[0017] Phase 2: Execute the SFC orchestration contract, virtualize the network services at the smart device layer, form a service function chain, and optimize the VNF deployment based on the deep reinforcement learning algorithm to obtain the optimal SFC optimal orchestration solution;

[0018] Phase 3: Based on the optimal SFC orchestration solution obtained by the deep reinforcement learning algorithm in Phase 2, the SDN controller and NFVO are used to complete the placement of VNFs and traffic routing;

[0019] Phase 4: Trigger the smart contract on the blockchain to re-register the transaction on the network. The transaction includes: interconnection, attributes and order of VNFs in SFC, user information and QoS.

[0020] (III) Beneficial effects

[0021] The present invention proposes a decryption traffic service chain orchestration method based on blockchain and deep reinforcement learning, and proposes an algorithm for elastic deployment of VNF based on reinforcement learning. In view of the change of network load, a non-convex linear mathematical optimization model with the goal of minimizing delay and energy consumption is established, and the reinforcement learning algorithm is used to solve it. The agent is trained through interaction with the physical network environment, and the optimal physical resource allocation and scheduling strategy for VNF scaling is found according to the state value of the physical node. The algorithm solves the problem of long service delay and high energy consumption under limited resources.

[0022] The present invention adopts irrelevant routing service chain technology to reduce routing links, and simply splices different VNF ​​groups and different services into multiple service chains, with high performance, simple and flexible configuration. A SFC (Service Function Chaining) service function chain intelligent orchestration algorithm based on DDQN (Double DeepQ-network) is proposed to achieve efficient resource allocation under dynamic load. With the goal of minimizing the cost of orchestration migration, the corresponding mathematical model is established considering the resource constraints of nodes, links, delays, etc., and the solution is solved using a deep reinforcement learning algorithm.

[0023] The present invention builds a trustworthy and automatically adjusted service function chain orchestration architecture based on blockchain algorithm and deep reinforcement learning algorithm, registers user information on the blockchain module, and uses smart contracts to achieve automatic resource adjustment. This system effectively improves the efficiency of network resource utilization, protects user identity, and ensures the credibility of transactions. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Deploy the network architecture for the decrypted traffic service chain VNF;

[0025] Figure 2 It is a work flow chart;

[0026] Figure 3 The overall organizational structure of the system of the present invention;

[0027] Figure 4 It is a system working flow diagram of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0029] Explanation of terms

[0030] (1) Blockchain

[0031] The concept of blockchain was first proposed in 2008, among which the new concepts of smart contracts and smart properties were also widely mentioned. Smart contracts are computer protocols that facilitate, verify or enforce the negotiation or performance of contracts. It provides the ability to directly track and execute complex agreements between parties without human interaction. On the other hand, smart properties are protocols that control ownership through blockchain. Blockchain is a chain data structure consisting of data blocks connected in chronological order. It maintains a growing list of blocks linked to each other. It maintains a consistent ledger through distributed node transactions without central management, and cryptographically guarantees non-quantifiable and unforgeable distributed ledger technology.

[0032] (2) Network virtualization technology

[0033] 1. NFV (Network Function Virtualization, NFV) system: In response to the high deployment cost, high management and maintenance cost, and high failure rate of network function equipment in large-scale heterogeneous networks, the European Telecom Standards Institute (ETSI) proposed a network function virtualization architecture. The basis of NFV technology is virtualization technology and cloud computing technology. Virtualization technology is used to decouple hardware and software from network functions, reduce the dependence of network functions on dedicated hardware devices, and promote the flexibility and scalability of storage, computing and other network resources. At the same time, the deployment cost of new services is also greatly reduced.

[0034] 2. SDN (Software Defined Network, SDN) architecture: In the 5G era, Internet technology is developing rapidly, and the endless mobile communication services are becoming more and more diversified, which has led to the explosive growth of mobile data traffic. The network architecture has become more and more complex and difficult to manage centrally. At the same time, the traditional IP-based network architecture lacks flexible scheduling capabilities in the face of massive services, making the service quality far from meeting user needs. Software-defined networking is an emerging network architecture that has attracted extensive research. SDN is a three-layer separation network architecture that includes an infrastructure layer, a logical control layer, and an application layer. Among them, the bottom layer of the SDN architecture is the infrastructure layer, which is composed of a series of network devices. This layer is mainly responsible for feedback of network configuration information and operation status information and forwarding traffic processing data packets. The control layer is mainly responsible for the centralized management of network resources, obtains global network information through the controller, centrally processes the information and makes decisions, while the top application layer contains many applications. Its main function is to abstract the service functions required by users into a network model and provide network services.

[0035] SDN decouples data control by separating the control plane from the user plane, breaking the tight coupling of traditional IP network architecture. In traditional IP networks, the forwarding of data packets and the operation of routing protocols are all performed by routers. Unlike traditional network architecture, the SDN architecture abstracts data plane network resources, realizes network logic centralization through SDN controllers, and forwards data traffic through switches, which accelerates the rapid deployment and use of emerging business applications. SDN simplifies the complicated steps of traditional router configuration and management, eliminates the concept of distributed network protocols, and can control the network more flexibly. By programming and controlling the rules of the SDN network, the functions of various network services, such as routing, switching, firewalls, etc., can be flexibly implemented.

[0036] (3) Reinforcement Learning

[0037] Reinforcement learning is a machine learning method that focuses on letting computers learn to make optimal decisions through interaction with the environment to maximize cumulative rewards. This learning method is modeled by establishing Markov decision processes (MDPs), which include state space, action space, transition probability, and reward function. The agent selects actions based on its current state to gradually form an optimized policy or value function. The exploration and exploitation strategy is key. The agent needs to try new actions to obtain more information while also using known high-reward actions to gain benefits. In addition, the combination of reinforcement learning and deep learning (deep reinforcement learning) has attracted widespread attention, using neural networks to approximate complex policies or value functions, such as in algorithms such as DQN (Deep Q-network), Actor-Critic, and PPO (Proximal Policy Optimization).

[0038] The present invention belongs to the field of information technology and is specifically applied to the field of network resource management and optimization, especially in large-scale heterogeneous networks. It combines blockchain and deep reinforcement learning technology to solve problems such as trust, adaptability, real-time, security and privacy in resource allocation, and improve network resource utilization efficiency and transaction credibility.

[0039] The present invention proposes an algorithm for elastic deployment of VNF based on reinforcement learning. In view of the change of network load, a non-convex linear mathematical optimization model with the goal of minimizing delay and energy consumption is established, and the reinforcement learning algorithm is used to solve it. The agent is trained through interaction with the physical network environment, and the optimal physical resource allocation and scheduling strategy for VNF scaling is found according to the state value of the physical node. The algorithm solves the problem of long service delay and high energy consumption under limited resources.

[0040] The present invention adopts irrelevant routing service chain technology to reduce routing links, and simply splices different VNF ​​groups and different services into multiple service chains, with high performance, simple and flexible configuration. A SFC (Service Function Chaining) service function chain intelligent orchestration algorithm based on DDQN (Double DeepQ-network) is proposed to achieve efficient resource allocation under dynamic load. With the goal of minimizing the cost of orchestration migration, the corresponding mathematical model is established considering the resource constraints of nodes, links, delays, etc., and the solution is solved using a deep reinforcement learning algorithm.

[0041] The present invention builds a trustworthy and automatically adjusted service function chain orchestration architecture based on blockchain algorithm and deep reinforcement learning algorithm, registers user information on the blockchain module, and uses smart contracts to achieve automatic resource adjustment. This system effectively improves the efficiency of network resource utilization, protects user identity, and ensures the credibility of transactions.

[0042] The decryption traffic service chain is a network service chain composed of different VNFs, which is used to process traffic in the network, especially encrypted traffic. In this context, different service nodes of the decryption traffic service chain correspond to different functions in network security, such as packet capture, decryption and reassembly, content security detection, behavior analysis, encryption and traffic blocking.

[0043] Each service node in the decryption traffic service chain processes the encrypted traffic flowing through the network in a certain order. For example, first use the packet capture service to capture the data packets passing through the network device, then use the decryption and reassembly service to decrypt and reassemble the encrypted data packets, then use the content security detection service to detect potential threat behaviors, and then use the behavior analysis service to analyze traffic patterns and behaviors, and finally perform encryption or traffic blocking operations as needed.

[0044] Each VNF node in the decryption traffic service chain has different functions. By working together, they can coordinate the processing of encrypted traffic flowing through the network, thereby providing more comprehensive security protection. This network architecture can dynamically adjust the combination of decryption service chains according to different threat situations and security requirements to achieve more flexible, efficient and customized network security protection strategies.

[0045] In network virtualization scenarios, physical resources cannot be flexibly allocated in the case of dynamic network loads. It is necessary to migrate VNFs of overloaded physical nodes to other physical nodes with lower loads in real time according to service request requirements, or to scale VNFs to ensure the QoS (Quality of Service) requirements of SFC. For example, in the decrypted traffic scenario, the content security detection service may need to install more filtering and control rules, or handle more applications and users, so it may require more physical resources to handle these requirements. At this time, the VNF corresponding to content security detection needs more physical resources to handle the increased business.

[0046] Decrypting traffic service chain VNF deployment network architecture Figure 1As shown in the figure, the system model is divided into two layers: virtual network layer and physical network layer. The virtual network layer contains an end-to-end SFC service function chain composed of different types of VNFs, and the physical network layer includes a physical network composed of various physical nodes. Each physical node has the function of routing and forwarding, and can be used as a physical entity for VNF instantiation. The VNF on the virtual network layer will be mapped to the corresponding physical nodes of the physical network layer.

[0047] The system will deploy VNFs on physical nodes based on the updated estimated demand data. If the SFC requested by the user occupies a large storage space, the communication overhead is also large. When the initial physical node cannot meet its needs, it will migrate to other physical nodes with sufficient resources to handle these needs.

[0048] Figure 2 The workflow of the VNF deployment module is summarized. In the system initialization state, there is no user request data, so the system counts the composition of all SFCs to represent the estimated demand of VNF. The system will decide on which physical node to deploy VNF based on this estimated demand. After the VNF is deployed, the system will update the parameters according to the deployment situation. These parameters actually represent Figure 2 The deployment feedback value in the reinforcement learning module is then used to learn using the user demand data collected during the cycle and the feedback parameters obtained by deploying the VNF. After learning, the system will output the VNF demand for the next cycle. In the next cycle, the VNF will be deployed on the appropriate physical node based on the estimated VNF demand.

[0049] In response to the trust and adaptation issues faced by VNF resource configuration, the present invention proposes a trusted deployment architecture based on blockchain and deep reinforcement learning, such as Figure 1 As shown in the figure, the whole system mainly includes the following three layers: virtual network layer, trusted physical network layer, and intelligent device layer.

[0050] (1) Smart device layer: The smart device layer is composed of various public infrastructure smart devices. Different business needs require different smart devices. At the same time, smart devices generate a large amount of data, but the processing capacity and resources of local servers are limited. Therefore, when the amount of data is particularly large, it needs to be sent to the cloud for processing.

[0051] (2) Trusted physical network layer: The trusted physical network layer includes a blockchain-based data security module and a DDQN-based VNF deployment module. The blockchain has the ability to provide reliable service management and ensure service consistency. The deep reinforcement learning algorithm places the VNF on the SFC on the appropriate physical node by finding the optimal deployment strategy to optimize resource usage and improve QoS. Therefore, each trusted physical network node contains four main modules: DDQN algorithm module, blockchain encryption module, SDN controller, and NFVO routing orchestration module.

[0052] (3) Virtual network layer: For the data generated by the intelligent device layer, the virtual network layer virtualizes it into different service requests through network slicing. Generally speaking, different service requests are chains composed of different virtual network functions, that is, SFCs. The deployment of VNFs is controlled by the SDN controller NFVO (Network Functions Virtualization Orchestration), and the deployment is optimized by combining the deep reinforcement learning DDQN module to complete the optimal deployment of VNFs.

[0053] like Figure 3 As shown in the figure, the trusted physical network layer includes: users, physical nodes, blockchain, deep reinforcement learning, and SDNC&NFVO. Among them, blockchain and deep reinforcement learning algorithms are the core parts. Blockchain runs through the entire process of VNF deployment, including the registration of resources before and during deployment, and the registration and update of transactions.

[0054] Before VNF deployment: Before providing services, some information of all physical nodes must be registered in the blockchain. By sending the unique identifier of the device and some key attributes to the block, the blockchain forms a resource pool with the information list, and then forms the information into blocks. The information of physical nodes is stored on the block, thus providing security for VNF deployment.

[0055] During VNF deployment: When there is a service request from the intelligent device layer, the blockchain module of the trusted physical network layer will authenticate the user's identity and whether it has permission. At the same time, it will dynamically monitor the resource information and location information of the physical nodes. It will also pass the request information and node information to the DDQN optimization module to calculate the optimal SFC orchestration deployment solution.

[0056] The overall organizational structure of the system is as follows Figure 3 The present invention is based on the system's blockchain and deep reinforcement learning based decryption traffic service chain arrangement method, including four stages, such as the process Figure 4 As shown:

[0057] Phase 1: Execute the login control smart contract, register user information and other information on the blockchain module, and request services through the login control protocol. Verify the relevant identity information of the request each time the service is requested;

[0058] Phase 2: Execute the SFC orchestration contract to virtualize the network services at the smart device layer to form a service function chain, and optimize the VNF deployment based on the deep reinforcement learning algorithm;

[0059] Phase 3: Based on the optimal SFC orchestration solution obtained by the deep reinforcement learning algorithm in Phase 2, the SDN controller and NFVO are used to complete the placement of VNFs and traffic routing;

[0060] Phase 4: Trigger the smart contract on the blockchain to re-register the transaction on the network. The transaction includes: interconnection, attributes and order of VNFs in SFC, user information and QoS.

[0061] The decrypted traffic service chain is a service link in the network that is used to decrypt and analyze encrypted traffic. It provides the function of decrypting traffic by connecting different services in series, thereby helping security teams detect and defend against potential network threats. The decrypted traffic service chain mainly includes the following services:

[0062] 1. Packet Capture: This service is used to capture data packets passing through network devices and transmit them to the next service node.

[0063] 2. Decryption and Reassembly: This service decrypts the encrypted data packets and reassembles them into the original plaintext data packets for subsequent service analysis.

[0064] 3. Content security detection: This service uses security rules and models to detect potential threatening behaviors or malicious activities.

[0065] 4. Behavioral analysis: This service identifies abnormal activities and possible attacks by analyzing traffic patterns and behaviors in the network.

[0066] 5. Encryption: This service re-encrypts traffic that has passed content security inspection and protection to ensure the security of the traffic.

[0067] 6. Traffic blocking: When potential malicious activities or attacks are detected, the service will immediately take blocking measures to prevent the traffic from spreading further into the network. Blocking operations can include closing connections, blocking source IP addresses, sending alert notifications, etc. to improve the security and responsiveness of the network.

[0068] For different threat situations, a variety of service chain combinations can be used. For traffic that needs real-time detection, you can use the service chain of capture, decryption and reassembly, threat detection, encryption or traffic blocking. For traffic that passes content security inspection and protection, that is, no keywords are hit or the action is not blocked, the border security stack will re-encrypt to ensure that the traffic decrypted by the border security stack is still safe when it leaves the border security stack. For traffic that does not pass content security inspection and protection, the border security stack will implement blocking operations to effectively intercept SSL-based encrypted traffic attacks and information leaks. For more complex threat situations, in-depth analysis of traffic behavior is required. You can use the service chain of capture, decryption and reassembly, threat detection, behavior analysis, encryption or traffic blocking to conduct in-depth behavioral analysis of these traffic and identify abnormal activities and possible attack behaviors.

[0069] The present invention adopts DDQN (Double Deep Q-network) reinforcement learning algorithm to realize SFC orchestration deployment. The cost function mainly considers three aspects of cost: resource congestion control, the cost of VNF dynamic migration, and the cost of network reconfiguration after VNF migration. Where V represents the physical node, L represents the physical link. The set of physical nodes is represented by V = {1, 2, ..., v, ..., N}, and the resources of the physical nodes are represented by CPU computing resources and storage capacity, which are defined as CPU v and MEM v The link resource is represented by bandwidth. The maximum transmission rate between node i and node j is the bandwidth, expressed as bw ij At the same time, the transmission delay between node i and node j is expressed as td ij .

[0070] For the service function chain, we define it as a triple s=(i s ,i d ,vnf s ), where i s represents the source node of SFC, i d Indicates the last node of SFC, vnf s Represents a collection of virtual network functions on the SFC, including attributes such as the order of connections between VNFs and the type of VNFs.

[0071] (1) Resource congestion control uses the weighted sum of CPU resource and storage resource usage to describe the current usage of the physical node, which is expressed as:

[0072]

[0073] Among them, w cpu and w memThey represent the weight factors of CPU resource and storage resource utilization, respectively, and w cpu +w mem =1; Indicates whether the virtual network function VNFf on SFC s can be mapped to the physical node v. If successful, it means otherwise r cpu Indicates the CPU usage at time r, r men Indicates the storage resource usage at time r.

[0074] Congestion control of a node can be defined as:

[0075]

[0076] B v It is the node congestion control balance factor.

[0077] Link congestion control can be defined as:

[0078]

[0079] Among them, r bw Indicates the bandwidth usage at time r. Indicates the virtual link on SFC s Whether to map to a physical link If the mapping is successful, otherwise

[0080] In summary, the overall congestion control is expressed as:

[0081] COST co =COST no +COST li

[0082] (2) When the network changes dynamically, it will inevitably affect the service function chain. For example, when the current node is shut down, the VNF deployed on the node in the original service function chain must be migrated to other nodes to meet the demand. Therefore, the cost of VNF dynamic migration is considered in the present invention. The cost of SFC migration can be expressed as:

[0083]

[0084] Among them, h l and h d are the link resource utilization rate and the transmission weight factor respectively, and there is h l +h d =1.

[0085] Bij It is the link congestion control balance factor. When the load is within a certain limit, the balance factor increases linearly with the load. When it exceeds this threshold, the balance factor increases exponentially with the increase of load.

[0086] is a binary variable, indicating the migration status at time t. And at time t+1 This means that VNFf is migrated from physical node i to physical node j.

[0087] (3) Since the SFC is reallocated after migration, the cost of network reconfiguration needs to be considered. We regard the cost of network reconstruction as the cost of rerouting. The state switching on each link requires reconfiguration of services and routes on the servers at both ends of the link. Therefore, the network reconstruction cost is defined as:

[0088]

[0089] in, and They represent the link status at time t+1 and time t respectively.

[0090] Combining the above detailed descriptions of the three costs, the total cost function can be expressed as:

[0091] COST all =COST co +COST mi +COST re

[0092] In the SFC scheduling scheme obtained based on the DDQN reinforcement learning algorithm, the three elements of the reinforcement learning algorithm: state space S, action space A, and reward function R are specifically mapped as follows:

[0093] (1) Action space A

[0094] The agent takes corresponding actions by selecting VNF placement and traffic routing, and changes the state of SFC after taking actions. The action space is represented by the set of action selections for VNF placement and the set of link routing selections, so the action space is defined as:

[0095]

[0096] (2) State space S

[0097] The purpose of studying the SFC mapping problem is to measure the user's satisfaction with the QoS of the SFC service through the delay cost of the SFC. A delay threshold is set for each SFC. When it exceeds this threshold, the state is 0, and when it does not exceed this threshold, the state is 1. Therefore, the state space can be represented by a set of satisfaction. The training goal is to satisfy the QoS of all users so that the set is all 1.

[0098] S={1,0,0,…}

[0099] (3) Reward function R

[0100] When a state transitions to another state by taking certain actions, the network will receive an immediate reward as a return value, which can be expressed as:

[0101] R=COST t -COST t+1

[0102] COST t is the COST at time t all , COST t+1 is the COST at time t+1 all .

[0103] The detailed steps of the SFC scheduling algorithm based on dual deep Q network are shown in Table 1.

[0104] First, initialize the experience memory pool, the weights of the current value neural network and the target value neural network, and the physical network environment: In each training iteration, the action value a is selected according to probability, and the action value a is observed. t Post-SSL encryption and decryption speed x t , and the traffic content abnormality rate σ t , get the next state s t+1 , and (s t ,a t ,x t ,σ t ,s t+1 ) is stored in the memory pool as a set of data, and then gradient descent is used to update its own network weights, and the network weights of the target value neural network are updated every certain period of time. When the loss function value converges, the iteration is terminated.

[0105] Table 1 Decryption traffic service chain orchestration algorithm based on DDQN reinforcement learning

[0106]

[0107]

[0108] Embodiment 1:

[0109] The present invention uses the network simulation tool Opendaylight to simulate network topology and resource allocation for algorithm verification and testing; uses Hyperledger Fabric to build the blockchain; and uses PyTorch to implement and train the deep reinforcement learning algorithm.

[0110] Under different numbers of SFC requests, the delay of the decryption traffic service chain orchestration algorithm based on DDQN reinforcement learning proposed in this invention increases roughly linearly. In the case of 50 SFCs, the average algorithm delay of the algorithm proposed in this invention can reach about 110ms; in the case of 80 SFCs, the average delay can reach about 140ms; in the case of 100 SFCs, the average delay can reach about 180ms.

[0111] Technical effects of the present invention:

[0112] 1. This paper introduces a reinforcement learning algorithm to solve the problem of elastic deployment of VNFs. In view of network load changes, with the goal of minimizing latency and energy consumption, the agent is trained through interaction with the physical network environment to find the optimal VNF scaling and resource allocation strategy. This method can adaptively adjust the deployment to meet the dynamically changing network load requirements.

[0113] 2. This paper proposes a SFC (Service Function Chain) intelligent orchestration algorithm based on deep reinforcement learning. By establishing a mathematical model of resource constraints including nodes, links, delays, etc., and using deep reinforcement learning algorithms such as DDQN (Deep Double Q-Network), efficient resource allocation under dynamic loads can be achieved. This helps to reduce the cost of orchestration migration while ensuring the optimal use of network resources.

[0114] 3. The present invention combines blockchain algorithms with deep reinforcement learning to build a reliable and automatically adjusted service function chain orchestration architecture. By registering user information through the blockchain module and using smart contracts to achieve automatic resource adjustment, the utilization efficiency of network resources is effectively improved, user identity privacy is protected, and the credibility of transactions is ensured.

[0115] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A decryption traffic service chain orchestration method based on blockchain and deep reinforcement learning, characterized in that: The method is applied to a trusted deployment architecture based on blockchain and deep reinforcement learning. The system includes the following three layers: virtual network layer, trusted physical network layer, and smart device layer. The method includes the following four stages: Phase 1: Execute the login control smart contract, register user information on the blockchain module, request services through the login control protocol, and verify the relevant identity information of the request each time the service is requested; Phase 2: Execute the SFC orchestration contract, virtualize the network services at the smart device layer, form a service function chain, and optimize the VNF deployment based on the deep reinforcement learning algorithm to obtain the optimal SFC optimal orchestration solution; Phase 3: Based on the optimal SFC orchestration solution obtained by the deep reinforcement learning algorithm in Phase 2, the SDN controller and NFVO are used to complete the placement of VNFs and traffic routing; Phase 4: Trigger the smart contract on the blockchain to re-register the transaction on the network. The transaction includes: interconnection, attributes and order of VNFs in SFC, user information and QoS; in, In the second stage, the DDQN (Double Deep Q-network) reinforcement learning algorithm is used to implement SFC orchestration deployment. The cost function considers three aspects of cost: resource congestion control, the cost of VNF dynamic migration, and the cost of network reconfiguration after VNF migration. Combining the above three costs, the total cost function is expressed as: COST all =COST co +COST mi +COST re Among them, COST co is the congestion control cost of resources, COST mi COST is the cost of VNF dynamic migration. re Cost of network reconfiguration after VNF migration; Then, based on the above cost function, the DDQN (Double Deep Q-network) reinforcement learning algorithm is used to implement SFC orchestration deployment; Define V to represent a physical node and L to represent a physical link. The set of physical nodes is represented by V = {1, 2, …, v, …, N}. The resources of the physical node are represented by CPU computing resources and storage capacity, which are defined as CPU v and MEM v ; The link resource is represented by bandwidth. The maximum transmission rate between node i and node j is the bandwidth, expressed as bw ij At the same time, the transmission delay between node i and node j is expressed as dt ij ; For a service function chain, it is defined as a triple s=(i s ,i d ,vnf s ), where i s represents the source node of SFC, i d Indicates the last node of SFC, vnf s The VNF type attribute that represents the set of virtual network functions on the SFC, including the order of connections between VNFs; Resource congestion control uses the weighted sum of CPU resource and storage resource usage to describe the current usage of the physical node, and expresses it as: Among them, w cpu and w mem They represent the weight factors of CPU resource and storage resource utilization, respectively, and w cpu +w mem =1; Indicates whether the virtual network function VNFf on SFC s can be mapped to the physical node v. If successful, it means otherwise r cpu Indicates the CPU usage at time r, r men Indicates the storage resource usage at time r; Congestion control of a node can be defined as: B v is the node congestion control balance factor; Link congestion control can be defined as: Among them, r bw represents the bandwidth usage at time r, Indicates the virtual link on SFC s Whether to map to a physical link If the mapping is successful, otherwise In summary, the overall congestion control is expressed as: COST co =COST no +COST li ; When the network changes dynamically, it will inevitably affect the service function chain. The cost of SFC migration is expressed as: Among them, h l and h d are the link resource utilization rate and the transmission weight factor respectively, and there is h l +h d =1; B ij is the link congestion control balance factor. When the load is within a certain limit, the balance factor increases linearly with the load. When it exceeds the threshold, the balance factor increases exponentially with the increase of load. is a binary variable, indicating the migration status at time t. And at time t+1 Then it means that VNFf is migrated from physical node i to physical node j; Since the SFC is reallocated after migration, the cost of network reconfiguration needs to be considered. The cost of network reconstruction is regarded as the cost of rerouting. The state switching on each link requires reconfiguration of services and routes on the servers at both ends of the link. Therefore, the network reconstruction cost is defined as: in, and Respectively represent the link status at time t+1 and time t; Based on the above cost function, the DDQN (Double Deep Q-network) reinforcement learning algorithm is used to implement SFC orchestration deployment, including: For the three elements of the reinforcement learning algorithm: state space S, action space A, and reward function R, the specific mapping is as follows: (1) Action space A The agent takes corresponding actions by selecting VNF placement and traffic routing, and changes the state of SFC after taking actions; the action space is represented by the set of action selections for VNF placement and the set of link routing selections, so the action space is defined as: (2) State space S The purpose of studying the SFC mapping problem is to measure the user's QoS for SFC services by the delay cost of SFC. A delay threshold is set for each SFC. When it exceeds this threshold, the state is 0, and when it does not exceed this threshold, the state is 1. Therefore, the state space is represented by a set of satisfaction. The training goal is to satisfy the QoS of all users so that the set is all 1. S＝{1,0,0,…} (3) Reward function R When a state transitions to another state by taking certain actions, the network will receive an immediate reward as a return value, expressed as: R=COST t -COST t+1 COST t is the COST at time t all , COST t+1 is the COST at time t+1 all ; The detailed steps of the DDQN-based SFC scheduling algorithm include: First, initialize the experience memory pool, the weights of the current value neural network and the target value neural network, and the physical network environment: In each training iteration, the action value a is selected according to probability, and the action value a is observed. t Post-SSL encryption and decryption speed x t , and the traffic content abnormality rate σ t , get the next state s t+1 , and (s t ,a t ,x t ,σ t ,s t+1 ) is stored in the memory pool as a set of data, and then gradient descent is used to update its own network weights, and the network weights of the target value neural network are updated every certain period of time. When the loss function value converges, the iteration is terminated.

2. The decryption traffic service chain arrangement method based on blockchain and deep reinforcement learning as claimed in claim 1 is characterized in that: The smart device layer is composed of various public infrastructure smart devices. When the amount of data is particularly large, it needs to be sent to the cloud for processing.

3. The decryption traffic service chain arrangement method based on blockchain and deep reinforcement learning as claimed in claim 1 is characterized in that: The trusted physical network layer includes a blockchain-based data security module and a DDQN-based VNF deployment module. The blockchain has the ability to provide reliable service management and ensure service consistency. The deep reinforcement learning algorithm places the VNF on the SFC on the appropriate physical node by finding the optimal deployment strategy, thereby optimizing resource usage and improving QoS. Each trusted physical network node contains four main modules: DDQN algorithm module, blockchain encryption module, SDN controller and NFVO routing orchestration module.

4. The decryption traffic service chain arrangement method based on blockchain and deep reinforcement learning as claimed in claim 1 is characterized in that: The virtual network layer virtualizes the data generated by the intelligent device layer into different service requests through network slicing. Different service requests are chains composed of different virtual network functions, namely SFCs. The SDN controller NFVO (Network Functions Virtualization Orchestration) is used to control the deployment of VNFs, and the deployment is optimized in combination with the deep reinforcement learning DDQN module to achieve the optimal deployment of VNFs.

5. The decryption traffic service chain arrangement method based on blockchain and deep reinforcement learning as claimed in claim 1 is characterized in that: The trusted physical network layer includes: users, physical nodes, blockchain, deep reinforcement learning, and SDNC&NFVO; blockchain and deep reinforcement learning algorithms are the core parts. Blockchain runs through the entire VNF deployment process, including the registration of resources before and during deployment, and the registration and update of transactions; Before VNF deployment: Before providing services, all physical node information must be registered in the blockchain. By sending the unique identifier of the device and some key attributes to the block, the blockchain lists the information into a resource pool and then forms the information into blocks. The information of the physical nodes is stored on the block, providing security for VNF deployment. During VNF deployment: When there is a service request from the intelligent device layer, the blockchain module of the trusted physical network layer will authenticate whether the user has the authority. At the same time, it will dynamically monitor the resource information and location information of the physical nodes, and pass the request information and node information to the DDQN optimization module to calculate the optimal SFC orchestration deployment solution.

Images