A large-scale topology arrangement management system for virtual network

Abstract

The application discloses a large-scale topology arrangement management system for a virtual network, which comprises a user interface layer, a network state management layer and a virtual network implementation layer.The user interface layer is used for generating a topology design interface and sending a deployment request carrying complete topology description information to the network state management layer.The network state management layer is used for splitting the virtual topology in the deployment request, calculating the mapping relationship between nodes, links and servers, and respectively constructing corresponding data models to maintain and manage the virtual network topology information and issue deployment instructions to the virtual network implementation layer.The virtual network implementation layer is used for deploying the virtual topology split to each server according to the deployment instructions, and collecting the underlying virtual node operation information and sending the information to the virtual network implementation layer at regular time intervals.The application can guarantee the accuracy of information storage and the efficiency of searching of the virtual topology under large-scale deployment, and greatly reduces the cost of changing the virtual topology in a large-scale scenario.

Description

Technical Field

[0001] This invention relates to the field of virtual network orchestration technology, and more specifically to a large-scale topology orchestration and management system for virtual networks. Background Technology

[0002] Currently, the development of information technologies such as 5G communication, the Internet of Things, and edge computing has presented new demands and challenges for network capacity, transmission speed, and overall architecture. To adapt to the rapidly changing network requirements, universities, research institutions, and enterprises are investing heavily in innovative research and development of network technologies. To avoid potential economic losses and security issues from direct deployment in production environments, researchers often verify and evaluate their innovative research on experimental platforms.

[0003] To meet the verification and innovation needs of various network innovation technologies, many institutions and universities have designed different network innovation simulation platforms. Existing network innovation platforms are mainly divided into testbeds, network simulation platforms, and network simulation platforms. Testbeds are deployed in real environments, primarily using virtual machines or real devices to simulate network nodes, offering good fidelity. However, in large-scale experimental simulation scenarios, testbeds are costly to use, have limited experimental scenarios, lack flexibility, and are difficult to effectively manage large-scale topologies. Network simulation platforms mainly use mathematical models to model and abstract network nodes and links, while using discrete events to simulate network behavior. Although the network topology achieved through modeling can be flexibly expanded to a very large scale, it lacks a real protocol stack, and network behavior is predefined, so network simulation does not possess high fidelity. Simulation platforms fall between the two, primarily using Linux virtualization technology to simulate network elements and links. They are lightweight, have low overhead, and can maintain the fidelity of experimental results. Therefore, they are increasingly favored by experimenters.

[0004] To ensure the accuracy and applicability of results during simulation experiments, researchers often need to conduct tests in large-scale experimental scenarios. For example, in experiments involving the innovation and verification of routing algorithms on production-scale networks, the topology often contains tens of thousands of nodes, and specific configurations are required for different nodes. In some network behavior modeling work, it is necessary to collect and measure network indicator data under large-scale topologies. In addition, in large-scale network experimental scenarios, it is also necessary to minimize the time overhead of topology creation and deletion while ensuring effective management of the virtual topology throughout its lifecycle.

[0005] However, current network simulation platforms cannot provide, or can only partially provide, support for large-scale experimental scenarios for experimenters. To fully meet the needs of supporting large-scale experimental scenarios, the following challenges are mainly faced:

[0006] (1) How to design and store the encoding structure and configuration information of topology information to ensure the accuracy and consistency of topology information and experimental configuration in large-scale scenarios and effectively reduce information storage overhead. In experimental scenarios, topology information can be divided into runtime information and static information. Static information mainly includes the structural information of the network topology, while runtime information mainly includes the information dynamically configured for network nodes and links during network experiments. To achieve efficient and accurate storage of the above topology information in large-scale scenarios, on the one hand, it is necessary to consider modeling the topology structure information and maintaining the consistency of topology structure relationships at the data storage level; on the other hand, while containing the basic necessary information of the topology structure, the topology modeling information should also use an efficient and compact data structure to reduce the resource overhead of data storage space.

[0007] (2) How to efficiently manage the lifecycle of network topology to ensure efficient and reasonable orchestration and management of topology nodes and links in large-scale scenarios. In large-scale scenarios, ensuring efficient and reasonable orchestration of experimental topologies requires consideration of both experimental scenario setup time and physical resource utilization. On the one hand, the cumulative time consumed by creating and destroying numerous nodes is usually long, so parallel acceleration of network element creation and destruction, as well as link creation and destruction, needs to be considered. On the other hand, large-scale deployed topologies are usually distributed across multiple physical servers, so when allocating resources to topology nodes, an efficient resource allocation algorithm is needed to ensure sufficient physical resources are allocated to different virtual nodes while ensuring efficient utilization of physical machine resources.

[0008] Mininet is currently a mainstream network simulation platform. Based on virtualization isolation technologies such as namespaces and cgroups, Mininet simulates common devices like hosts and switches, offering lightweight characteristics while maintaining a certain level of fidelity. Regarding topology information storage, Mininet supports defining topology structures using Python scripts. After the script runs, Mininet uses the memory space allocated by the Python process to store the simulated network's topology information. Due to its Linux container-based nature, Mininet can quickly build virtual network structures, and all created virtual nodes share the same Linux kernel. While this solution is lightweight and resource-efficient, enabling rapid virtual network construction, Mininet has significant limitations. Firstly, Mininet only supports single-machine deployment, and limited by the physical resources of a single machine, it cannot support ultra-large-scale virtual networks. Secondly, because Mininet stores topology information in memory during runtime, fault recovery is difficult, and data loss is likely.

[0009] Kollaps is a container-based network simulation platform proposed by EuroSys. Kollaps uses containers to simulate peer nodes and employs network folding technology to fold the topology into virtual end-to-end connections, primarily for data plane network simulation experiments. Kollaps defines topology connections and traffic characteristics through XML structured files and allows pre-definition of dynamic events during experiments. Kollaps leverages the existing container orchestration tool Docker Swarm to achieve cross-host container orchestration, thus supporting large-scale virtual topologies. While Kollaps can support virtual topologies on the order of thousands, it only supports limited large-scale scenarios. On the one hand, Kollaps only supports simulating end-node network elements and does not simulate internal network elements such as routing devices, switching devices, and controllers. Therefore, it cannot support complex and diverse virtual topologies. On the other hand, Kollaps uses relatively complex XML files to predefine dynamic link attributes in the virtual topology. These attributes are obtained by overlapping calculations of predefined link attributes. In large-scale scenarios, both storage and computation efficiency are low. Furthermore, this event definition can only define end-to-end link attributes and cannot define node attributes, which is not flexible enough and has certain limitations. Summary of the Invention

[0010] To address the aforementioned shortcomings in the prior art, this invention provides a large-scale topology orchestration and management system for virtual networks.

[0011] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0012] A large-scale topology orchestration and management system for virtual networks, comprising:

[0013] The user interface layer is used to generate the topology design interface and send a deployment request carrying complete topology description information to the network state management layer.

[0014] The network state management layer receives deployment requests from the user interface layer, segments the virtual topology in the deployment request, calculates the mapping relationship between nodes, links, and slave servers, and constructs corresponding data models for each. It maintains and manages the virtual network topology information. After completing the above operations, it issues deployment instructions to the virtual network implementation layer. Furthermore, it receives the underlying virtual node running information from the virtual network implementation layer and compares the received underlying virtual node running information with the data model in the Redis database to determine whether the virtual network's running status matches the information recorded in the Redis database.

[0015] The virtual network implementation layer is used to receive deployment instructions from the network status management layer, deploy the virtual topology to each slave server according to the deployment instructions, and periodically collect the underlying virtual node operation information and send it to the virtual network implementation layer.

[0016] Optionally, the step of segmenting the virtual topology in the deployment request, calculating the mapping relationship between nodes, links, and slave servers, and converting it into a data model specifically includes:

[0017] Extract complete topology description information from the deployment request, and then extract the physical resource request information of all virtual nodes in the virtual network from the topology description information;

[0018] The snowflake algorithm is used to calculate the global ID of all virtual nodes. Based on the calculated global ID of all virtual nodes and the physical resource request information, the virtual topology is segmented to obtain the data information of virtual network nodes and links deployed on each slave server in the virtual network implementation layer.

[0019] Based on the data information of each virtual network node and link deployed on the server, construct the node data model, link data model, and virtual LAN extended link data model respectively.

[0020] Optionally, the step of partitioning the virtual topology based on the global IDs and physical resource request information of all virtual nodes specifically includes:

[0021] The virtual network information is simplified into a graph structure composed of points and lines, and a list of resources required by all nodes is maintained;

[0022] A greedy algorithm is used to allocate physical resources on each physical machine based on the physical resource request information.

[0023] The optimization objective is to minimize cross-host links, while the objective constraints are to minimize resource usage and cross-host links. The Metis algorithm is used to segment the virtual topology.

[0024] Alternatively, the method for constructing the node data model is as follows:

[0025] Build a JSON model of the node attributes of virtual nodes in the virtual network;

[0026] The user's experimental project and node name are concatenated into a string combination and used as the index key of the hash table to create an index. The attribute fields and values ​​describing the node attributes in the JSON model are used as the values ​​of the index key of the hash table to form a two-level hash structure.

[0027] Different storage instances are allocated to different users, and the uniqueness of the experimental project name is checked when the user deploys the topology.

[0028] Alternatively, the method for constructing the link data model is as follows:

[0029] Build a JSON model of the node attributes of virtual nodes in the virtual network;

[0030] The combination of user experiment projects and link names is used as the index key of the hash table to create an index, and the attribute fields and values ​​describing node attributes in the JSON model are used as the values ​​of the index key of the hash table to form a two-level hash structure.

[0031] Optionally, the method for constructing the virtual LAN extended link data model is as follows:

[0032] Build a JSON model of the node attributes of virtual nodes in the virtual network;

[0033] The network identification information of user experiment projects and virtual LAN extensions, as well as the information of the corresponding source virtual nodes and virtual software switches / virtual bridges, are combined as the index key of the hash table to create an index. The attribute fields and values ​​describing node attributes in the JSON model are used as the values ​​of the index key of the hash table to form a two-level hash structure.

[0034] Optionally, the network state management layer stores the data model information during the experimental runtime phase in a Redis database that uses memory as a cache, and stores the data model information during the experimental results persistence phase in a disk-based relational database.

[0035] Optionally, when the user completes the experiment and persists the virtual network information, the network state management layer, based on the storage request initiated by the user interface layer, matches all strings in the Redis database that begin with the name of the experiment project according to the string prefix matching method, reads the values ​​corresponding to these index keys, aggregates them, and writes them into the MySQL data to complete the persistent storage of the virtual network experiment information.

[0036] Optionally, the network state management layer models the basic relationship between user information and related experimental information during the persistent storage phase of experimental results, in order to maintain the relationship between user information and experimental information.

[0037] The relationship between users and experiments is maintained by inserting an additional username field into the user experiment project table;

[0038] The experiment and the topology maintain a one-to-many relationship by introducing an additional intermediate table and writing mapping entries of the corresponding primary keys in each table into the intermediate table;

[0039] The user maintains a many-to-many relationship with the Docker image by introducing an additional intermediate table and writing mapping entries for the corresponding primary keys in each table into the intermediate table.

[0040] Optionally, the virtual network implementation layer deploys the virtual topology segmented to each slave server according to the deployment instructions, specifically including:

[0041] Virtual nodes are created and virtual links are established during the topology creation phase;

[0042] During the service startup phase, the routing and forwarding processes running on the nodes are started concurrently.

[0043] The service startup phase adopts a coroutine-based concurrent approach and uses an actor programming model to encapsulate the creation of network components and service startup functions into different actors, which are managed through an actor pool. After an I / O operation request is issued, the coroutine waiting for the I / O operation to finish is asynchronously suspended, and the next operation is executed.

[0044] The present invention has the following beneficial effects:

[0045] (1) This invention provides an efficient storage scheme for supporting virtual topology and related experimental information in large-scale scenarios: the network nodes and nodes in the virtual topology are abstracted and modeled, and a data model is established. Redis and MySQL are used to manage the virtual topology information in stages to ensure the accuracy of virtual topology information storage and the efficiency of retrieval under large-scale deployment.

[0046] (2) This invention provides a virtual network orchestration scheme to support efficient and reliable virtual network in large-scale scenarios: Based on graph cutting theory, an efficient resource calculation and partitioning scheme is designed. Through technologies such as coroutines and multithreading, the deployment process of virtual networks is effectively accelerated. The expected running state of virtual networks is declared through declarative APIs. Data information and underlying network state are synchronized through polling. The consistency between the management model and the actual virtual topology is maintained. This invention maintains virtual topology metadata information down to a single node and link, which greatly reduces the overhead of changing virtual topology in large-scale scenarios. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of a large-scale topology orchestration and management system for virtual networks according to an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the web UI of the user interface layer in an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram illustrating the physical infrastructure deployment of the large-scale network topology orchestration management system in an embodiment of the present invention. Detailed Implementation

[0050] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0051] like Figure 1 As shown, this embodiment of the invention provides a large-scale topology orchestration and management system for virtual networks, including:

[0052] The user interface layer is used to generate the topology design interface and send a deployment request carrying complete topology description information to the network state management layer.

[0053] The network state management layer receives deployment requests from the user interface layer, segments the virtual topology in the deployment request, calculates the mapping relationship between nodes, links, and slave servers, and constructs corresponding data models for each. It maintains and manages the virtual network topology information. After completing the above operations, it issues deployment instructions to the virtual network implementation layer. Furthermore, it receives the underlying virtual node running information from the virtual network implementation layer and compares the received underlying virtual node running information with the data model in the Redis database to determine whether the virtual network's running status matches the information recorded in the Redis database.

[0054] The virtual network implementation layer is used to receive deployment instructions from the network status management layer, deploy the virtual topology to each slave server according to the deployment instructions, and periodically collect the underlying virtual node operation information and send it to the virtual network implementation layer.

[0055] In an optional embodiment of the present invention, the web UI of the user interface layer of the present invention is as follows: Figure 2 As shown, this web UI can be accessed and opened in a browser. Users can design the topology within this web UI. After the user has designed it, they can click the deployment button to send a POST HTTP (HTTP Hypertext Transfer Protocol) request to the master manager (the system management process on the master server). The HTTP payload of this request will carry complete topology description information.

[0056] In an optional embodiment of the present invention, the network state management layer is used to perform graph partitioning calculations on the requested virtual topology, maintain and manage the data model information of the virtual topology, and send virtual topology deployment requests to the slave servers. The system management process on the master server in the network state management layer is the core module of the system.

[0057] This module first maintains a resident HTTP backend process to listen for HTTP requests sent by the user interface layer. Upon receiving a request, it extracts complete topology description information from the request body. Then, it preprocesses the user-submitted topology description file, extracting the physical resource request information for all virtual nodes in the virtual network. This request information includes resource information in three dimensions: CPU, memory, and bandwidth. If the user does not specify these by default, the system will configure default resource information based on the node type. After completing resource calculations, the topology splitting module calls the Snowflake algorithm to calculate the global ID of each node. The final module output is detailed information about the virtual network nodes and links deployed on each slave server. This information is presented as a JSON structured data model. After receiving this information, the master manager writes the virtual network information to a Redis database using a hash table storage format. After writing the information, the master manager sends an HTTP request to the slave server on the host machine, initiating a virtual network deployment request. This deployment request is then processed by the worker manager located in the virtual network implementation layer.

[0058] This involves segmenting the virtual topology in the deployment request, calculating the mapping relationship between nodes, links, and slave servers, and converting it into a data model, specifically including:

[0059] Extract complete topology description information from the deployment request, and then extract the physical resource request information of all virtual nodes in the virtual network from the topology description information;

[0060] The snowflake algorithm is used to calculate the global ID of all virtual nodes. Based on the calculated global ID of all virtual nodes and the physical resource request information, the virtual topology is segmented to obtain the data information of virtual network nodes and links deployed on each slave server in the virtual network implementation layer.

[0061] Based on the data information of each virtual network node and link deployed on the server, construct the node data model, link data model, and virtual LAN extended link data model respectively.

[0062] The virtual topology is segmented based on the global IDs and physical resource request information of all virtual nodes, specifically including:

[0063] The virtual network information is simplified into a graph structure composed of points and lines, and a list of resources required by all nodes is maintained;

[0064] A greedy algorithm is used to allocate physical resources on each physical machine based on the physical resource request information.

[0065] The optimization objective is to minimize cross-host links, while the objective constraints are to minimize resource usage and cross-host links. The Metis algorithm is used to segment the virtual topology.

[0066] To accurately represent the experimental and status information of virtual components in large-scale virtual networks, this invention abstracts and models virtual nodes and virtual links, and establishes a unified data model based on this. This data model uses a JSON structure. JSON is a data structure that uses key:value pairs and can accurately represent all field attributes of the model and their corresponding values. Compared to data structures such as XML, which require additional structured symbols for data definition, JSON results are clearer and more concise.

[0067] This invention designs a network state management model based on the JSON data description language. It abstracts and models virtual network runtime information, node information, and link information, and stores the data model using Redis and MySQL respectively. This invention uses a declarative API to define the expected runtime state of the virtual topology and achieves unified database state of the virtual topology through polling by the underlying monitoring module.

[0068] The method for constructing a node data model according to the present invention is as follows:

[0069] Build a JSON model of the node attributes of virtual nodes in the virtual network;

[0070] The user's experimental project and node name are concatenated into a string combination to create an index for the hash table key. The attribute fields and values ​​describing the node attributes in the JSON model are used as the values ​​for the hash table, forming a two-level hash structure.

[0071] Different storage instances are assigned to different users, and the uniqueness of project_name is checked when a user deploys a topology.

[0072] The main abstract information of the node data model for the virtual topology is shown in Table 1:

[0073] Table 1 Node Data Model

[0074]

[0075]

[0076] Table 1 shows the node data description model established in this invention. This model contains the necessary node information that needs to be maintained during the operation of the virtual topology. The model uses a string concatenation of the user's experimental project and the node name as the key to create an index for the hash table. The corresponding value is the JSON model of the virtual node in the virtual network, which is also a hash table. The JSON model contains attribute fields and values ​​describing node attributes. These attribute fields are also defined using strings, as shown in the table above. This two-level hash structure facilitates users in quickly finding specific node attributes. Furthermore, this system allocates different storage instances to different users and checks the uniqueness of the `project_name` when a user deploys the topology, thus ensuring the uniqueness of the hash table's key value. This also solves the problem of hash collisions caused by duplicate node names in different virtual network topologies. Through the above design, this invention maintains the mapping relationship between basic node attributes and specific values, facilitating the rapid and accurate location and modification of specific node data in large-scale topology deployments.

[0077] This invention uses the Snowflake algorithm to generate a unique container ID for each container node within the server cluster, corresponding to the `container_id` (unique container identifier) ​​field in the table. The algorithm works as follows: The Snowflake algorithm uses 64 bits to generate a unique ID, dividing the 64 bits into four parts. The first part occupies 1 bit, which is always 0 and has no practical function. The second part is the system's current timestamp, occupying 41 bits, accurate to milliseconds, and can hold approximately 69 years of data. The third part is the server machine ID, occupying 10 bits, with the high 5 bits representing the data center ID and the low 5 bits representing the worker node ID, accommodating up to 1024 nodes. The fourth part is the sequence number, occupying 12 bits, which increments continuously from 0 every millisecond, up to a maximum of 4095, generating a total of 4096 IDs. By using this algorithm to generate the unique ID field in the master, the problem of different Docker (container management software) processes on different slave servers generating the same container ID can be avoided. This provides a globally unique identity index for each container node, making it easy to quickly look up and match the container ID in large-scale scenarios to find the actual physical distribution location of the container node.

[0078] The method for constructing the link data model in this invention is as follows:

[0079] Build a JSON model of the node attributes of virtual nodes in the virtual network;

[0080] The combination of user experiment projects and link names is used as the key to create an index for the hash table, and the attribute fields and values ​​describing node attributes in the JSON model are used as the values ​​to form a two-level hash structure.

[0081] The main abstract information of the virtual topology in the link data model is shown in Table 2:

[0082] Table 2 Link Data Model

[0083]

[0084]

[0085] Table 2 shows the link data description model established in this invention. This model contains the necessary link information that needs to be maintained during the operation of the virtual topology. The model uses a combination of user experiment projects and link names as an index, which facilitates users to quickly find specific link attributes in large-scale experimental scenarios and provides good user data isolation. The model uses a two-level hash structure to maintain the mapping relationship between basic link attributes and specific values, which facilitates the quick and accurate location and modification of specific link data in the case of large-scale topology deployment. The model focuses on the most important basic attributes of virtual links in the experiment, including bandwidth, latency, packet loss, etc., and distinguishes between uplink and downlink.

[0086] The method for constructing a virtual local area network extended link data model according to the present invention is as follows:

[0087] Build a JSON model of the node attributes of virtual nodes in the virtual network;

[0088] The user experiment project and the VNI ID information of the virtual LAN extension, as well as the corresponding source virtual node and OVS virtual bridge information, are combined to create an index as the key of the hash table. The attribute fields and values ​​describing the node attributes in the JSON model are used as the values ​​of the hash table to form a two-level hash structure.

[0089] The main abstract information of the virtual topology in the Virtual LAN Extended Link Data Model is shown in Table 3:

[0090] Table 3 Virtual LAN Extended Link Model

[0091]

[0092] Table 3 shows the virtual LAN extension link data description model established in this invention. This model describes the attributes of cross-host links established using virtual LAN extensions during virtual topology operation. This information records in detail the VNI ID information of the virtual LAN extension, as well as the information of the corresponding source virtual node and OVS virtual bridge, which facilitates the maintenance of virtual link information between different hosts. The model uses a hash table to store the above information, which facilitates the quick and accurate modification of virtual LAN extension link information in large-scale virtual network simulation experiments. The model shields users from the deployment details of the underlying virtual network, maintaining the consistency between the underlying virtual topology and the user's logical topology even in large-scale experimental scenarios.

[0093] This invention divides the user's experimental process into two stages: the experimental runtime stage and the experimental result persistent storage stage. The network state management layer stores the data model information of the experimental runtime stage in a Redis database with memory caching, and stores the data model information of the experimental result persistent storage stage in a disk-based relational database.

[0094] When the user completes the experiment and persists the virtual network information, the network state management layer, based on the storage request initiated by the user interface layer, matches all strings in the Redis database that start with the project_name according to the string prefix matching method, reads the value corresponding to these keys, aggregates them, and writes them into the MySQL data, thus completing the persistent storage of the virtual network experiment information.

[0095] During the persistent storage phase of experimental results, the network state management layer models the basic relationship between user information and related experimental information in order to maintain the relationship between user information and experimental information.

[0096] The one-to-one relationship between users and experiments is maintained by inserting an additional user field into the project table;

[0097] The experiment and the topology maintain a one-to-many relationship by introducing an additional intermediate table and writing mapping entries of the corresponding primary keys in each table into the intermediate table;

[0098] The user maintains a many-to-many relationship with the Docker image by introducing an additional intermediate table and writing mapping entries for the corresponding primary keys in each table into the intermediate table.

[0099] Specifically, during the experimental phase, user operations may frequently involve modifying the virtual network topology information, resulting in a large number of I / O operations. Therefore, storing the data model information in a Redis database with in-memory caching will result in faster read and write speeds. To further improve the efficiency of data retrieval, this invention abandons linear data structures such as arrays for storing virtual network-related information and instead uses a hash table to maintain the experimental information of the virtual topology. The time complexity of data retrieval in the hash table is O(1). This hash table uses...<project_name> _<node / link_name> The concatenated string is used as the key, and the value is a JSON structure representing the information of the network component. The JSON structure is also stored based on key-value in terms of expression, and finally forms a two-level hash structure, thereby enabling the search for information of nodes or links in a large-scale virtual topology in O(1) time complexity.

[0100] During the persistent storage phase of experimental results, this invention stores the information of the virtual network in a relational database, MySQL. Compared to the experimental data during the experimental run phase, the information in the persistent storage phase does not involve a large number of read, write, or modification I / O operations. Therefore, the response time for read and write requests is not critical, and these data models can be stored in a disk-based relational database, thereby reducing the resource overhead of memory storage. MySQL also supports storing JSON structured data, so no additional data conversion logic is needed when using MySQL to store virtual network data model information.

[0101] On the other hand, during the persistent storage phase of experimental results, it is also necessary to maintain the relationship between user information and experimental information. This invention abstracts the user information and related experimental information as necessary and models the basic relationships of these data. For example, the relationship between user and experiment is one-to-many, the relationship between experiment and topology is one-to-one, and the relationship between user and Docker image is many-to-many. These data are ultimately stored in MySQL, corresponding to different data tables. The maintenance of the above relationships is based on MySQL indexes. For one-to-one relationships, such as the relationship between user and project, it is mainly achieved by inserting an additional user field into the project table; while for one-to-many and many-to-many relationships, it is mainly achieved by introducing an additional intermediate table and writing the mapping entries of the corresponding primary keys of each table into the intermediate table.

[0102] Large-scale virtual topologies require distributed deployment across multiple servers to ultimately form a flat, layer-2 topology consistent with the user's deployment logic. Before deploying a virtual topology, the physical resources required must be estimated to determine which server to assign virtual nodes to. When a user deploys a virtual topology via the RESTful API implemented in this invention, the invention preprocesses the topology description file submitted by the user. This description file specifies the physical resources consumed by each network node, including resource statistics across three dimensions: CPU, memory, and bandwidth. If the user does not specify these resources by default, the system will configure default resource information based on the node type.

[0103] This invention implements a resource calculation and partitioning scheme based on graph partitioning theory. The purpose of this scheme is to minimize the generation of cross-host links while satisfying the expected physical resource requirements of all virtual nodes. Based on the above requirements, this scheme first simplifies the specific virtual network information into a graph structure composed of points and lines, and maintains a list of resources required by all nodes. According to a greedy algorithm, it ensures that the distribution of physical resources allocated on each physical machine is relatively balanced. On this basis, the minimum number of cross-host links is used as the optimization objective to achieve the final virtual network partitioning. This scheme improves the metis algorithm by introducing objective constraints such as resource usage and the minimum number of cross-host links to the optimization objective, which can better meet the needs of this scenario.

[0104] In an optional embodiment of the present invention, the virtual network implementation layer of the present invention deploys the virtual topology segmented to each slave server according to the deployment instructions, specifically including:

[0105] Virtual nodes are created and virtual links are established during the topology creation phase;

[0106] During the service startup phase, the routing and forwarding processes running on the nodes are started concurrently.

[0107] The service startup phase adopts a coroutine-based concurrent approach and uses an actor programming model to encapsulate the creation of network components and service startup functions into different actors, which are managed through an actor pool. After an I / O operation request is issued, the coroutine waiting for the I / O operation to finish is asynchronously suspended, and the next operation is executed.

[0108] Specifically, this invention utilizes a virtual network implementation layer to deploy and partition the virtual topology onto the slave server, periodically collecting operational information from the underlying virtual nodes and sending it to the master. The core of the underlying virtual topology implementation layer is the slave server manager module.

[0109] This module maintains a resident HTTP backend process to listen for HTTP requests sent by the master manager. Upon receiving a request, the slave server manager extracts description information of the virtual topology deployed on the current slave server from the request payload, including network node and link configuration information. Then, the slave server manager communicates with the resident actor pool process via message passing, creating corresponding actors in the actor pool. These actors handle tasks such as creating network components and starting services, all implemented using coroutines. For collecting underlying network information, the slave server manager runs a separate resident monitoring script. This script communicates with Kubernetes, collecting underlying virtual network runtime information through Kubernetes interface calls. After obtaining this information, the script automatically aggregates the collected information to the master. The master then compares the aggregated information with the data model in the Redis database to check if the overall virtual network's runtime status matches the information recorded in the Redis database. Through this approach, the present invention can efficiently orchestrate and manage virtual networks and achieve virtual network monitoring.

[0110] After completing the topology splitting, this invention will deploy the actual virtual network on the server. This deployment process is divided into two phases: the topology creation phase and the service startup phase.

[0111] The topology creation phase mainly involves building the topology, including creating virtual nodes and virtual links. The service startup phase, on the other hand, involves starting the key processes running on the nodes, such as routing processes and forwarding processes. In the implementation of the operating system, these are all I / O operations, so concurrency can be used to speed up the creation of the virtual network topology and the startup of services.

[0112] Currently, concurrency methods mainly include multithreading, multiprocessing, and coroutines. However, using multithreading and multiprocessing for concurrency can cause significant system overhead, and the number of concurrent connections is limited by the number of operating system kernels. Therefore, this invention uses a coroutine-based concurrency approach. It employs an actor programming model to encapsulate functions such as network component creation and service startup into different actors, and manages these actors through an actor pool. Because it is based on coroutines, it can achieve thousands of concurrent connections on a single thread. After an I / O operation request is issued, the coroutine does not block and wait, but asynchronously suspends the coroutine waiting for the I / O operation to finish and continues to execute the next operation, thereby increasing concurrency and accelerating the construction of the entire virtual network and the startup of services.

[0113] After the virtual network is created and the service is started, this invention also monitors the virtual network's operational status to ensure that the user's virtual network operates in a correct and stable state. Since the virtual network's operational status is stored in a Redis database, this invention uses a scheduled script to cyclically monitor whether the data information in the Redis database is consistent with the actual physical underlying network's operational status. This script communicates with the Kubernetes resident process on the server. Kubernetes is a container manager; compared to the native Docker API, Kubernetes integrates batch container management tools, making the collection of container node information more efficient. After obtaining this information, the automated script aggregates the collected information to the master database. The master then compares the aggregated information with the data model in the Redis database to check whether the overall virtual network's operational status is consistent with the information recorded in the Redis database. Through this approach, this invention can efficiently orchestrate and manage virtual networks and achieve virtual network monitoring.

[0114] When a user completes an experiment and wishes to persistently store the virtual network information, the user needs to initiate a storage request at the user interface layer and submit the request to the master manager. The master manager will match all strings in the Redis database that begin with the project_name based on string prefix matching, read the value corresponding to these keys, aggregate them, and write them into the MySQL database to complete the persistent storage of the virtual network experiment information.

[0115] Deployment environment of the present invention Figure 3 As shown. This invention is mainly divided into master service and slave server service, supporting single-machine or multi-machine deployment (in multi-machine deployment mode, connectivity between physical machines must be guaranteed). This invention can be deployed on any computer running the Linux operating system, or it can be packaged as a Docker image, making deployment flexible and convenient.

[0116] On the server-side Linux operating system, the following software needs to be installed: Python version 3.9.6, Redis version 5.0.2, MySQL version 5.7.0, Open vSwitch version 2.12.0, NGINX version 1.14, and some necessary system dependencies; start the master service and slave server service of the large-scale virtual network orchestration management system on the server respectively, and use Gunicorn to improve the system concurrency.

[0117] 1) First, the experimenter needs to use the GUI visualization interface provided by this invention to design the topology, configure the attributes, and form a structured topology description file. Then, the structured topology description file is sent to the backend service via HTTP service. The server will translate and segment the topology description file, and finally create a virtual topology on the server and initialize the basic functions of the nodes.

[0118] 2) After step 1), experimenters can perform batch operations, experiment configurations, and other operations on nodes in a large-scale virtual network in a visual GUI, and deploy innovative network algorithms.

[0119] 3) Test and optimize the innovative algorithms that need to be verified, and modify the virtual topology on the visual GUI as needed;

[0120] 4) Repeat step 3) to validate and evaluate the performance of the innovative algorithm;

[0121] 5) Organize and analyze the test results to complete the closed-loop process of the entire network innovation.

[0122] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0125] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0126] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A large-scale topology orchestration and management system for virtual networks, characterized in that, include: The user interface layer is used to generate the topology design interface and send a deployment request carrying complete topology description information to the network state management layer. The network state management layer receives deployment requests from the user interface layer, segments the virtual topology in the deployment request, calculates the mapping relationship between nodes, links, and slave servers, and constructs corresponding data models for each. It maintains and manages the virtual network topology information. After completing the above operations, it issues deployment instructions to the virtual network implementation layer. Furthermore, it receives the underlying virtual node running information from the virtual network implementation layer and compares the received underlying virtual node running information with the data model in the Redis database to determine whether the virtual network's running status matches the information recorded in the Redis database. The virtual network implementation layer is used to receive deployment instructions from the network status management layer, deploy the virtual topology to each slave server according to the deployment instructions, and periodically collect the underlying virtual node operation information and send it to the virtual network implementation layer. The process of segmenting the virtual topology in the deployment request, calculating the mapping relationship between nodes, links, and slave servers, and converting it into a data model specifically includes: Extract complete topology description information from the deployment request, and then extract the physical resource request information of all virtual nodes in the virtual network from the topology description information; The snowflake algorithm is used to calculate the global ID of all virtual nodes. Based on the calculated global ID of all virtual nodes and the physical resource request information, the virtual topology is segmented to obtain the data information of virtual network nodes and links deployed on each slave server in the virtual network implementation layer. Based on the data information of each virtual network node and link deployed on the server, construct the node data model, link data model, and virtual LAN extended link data model respectively; The process of partitioning the virtual topology based on the global IDs and physical resource request information of all virtual nodes specifically includes: The virtual network information is simplified into a graph structure composed of points and lines, and a list of resources required by all nodes is maintained; A greedy algorithm is used to allocate physical resources on each physical machine based on the physical resource request information. With the goal of minimizing cross-host links and the constraints of minimizing resource usage and cross-host links, the Metis algorithm is used to segment the virtual topology. When the user completes the experiment and persists the virtual network information, the network state management layer, based on the storage request initiated by the user interface layer, matches all strings in the Redis database that begin with the name of the experiment project according to the string prefix matching method, reads the values ​​corresponding to these index keys, aggregates them, and writes them into the MySQL data to complete the persistent storage of the virtual network experiment information.

2. The large-scale topology orchestration and management system for virtual networks according to claim 1, characterized in that, The method for constructing a node data model is as follows: Build a JSON model of the node attributes of virtual nodes in the virtual network; The user's experimental project and node name are concatenated into a string combination and used as the index key of the hash table to create an index. The attribute fields and values ​​describing the node attributes in the JSON model are used as the values ​​of the index key of the hash table to form a two-level hash structure. Different storage instances are allocated to different users, and the uniqueness of the experimental project name is checked when the user deploys the topology.

3. A large-scale topology orchestration and management system for virtual networks according to claim 1, characterized in that, The method for constructing the link data model is as follows: Build a JSON model of the node attributes of virtual nodes in the virtual network; The combination of user experiment projects and link names is used as the index key of the hash table to create an index, and the attribute fields and values ​​describing node attributes in the JSON model are used as the values ​​of the index key of the hash table to form a two-level hash structure.

4. A large-scale topology orchestration and management system for virtual networks according to claim 1, characterized in that, The method for constructing a virtual LAN extended link data model is as follows: Build a JSON model of the node attributes of virtual nodes in the virtual network; The network identification information of user experiment projects and virtual LAN extensions, as well as the information of the corresponding source virtual nodes and virtual software switches / virtual bridges, are combined as the index key of the hash table to create an index. The attribute fields and values ​​describing node attributes in the JSON model are used as the values ​​of the index key of the hash table to form a two-level hash structure.

5. A large-scale topology orchestration and management system for virtual networks according to claim 1, characterized in that, The network state management layer stores the data model information during the experimental runtime phase in a Redis database that uses memory as a cache, and stores the data model information during the experimental results persistence phase in a disk-based relational database.

6. A large-scale topology orchestration and management system for virtual networks according to claim 1, characterized in that, During the persistent storage phase of experimental results, the network state management layer models the basic relationship between user information and related experimental information to maintain the relationship between user information and experimental information. The relationship between users and experiments is maintained by inserting an additional username field into the user experiment project table; The experiment and the topology maintain a one-to-many relationship by introducing an additional intermediate table and writing mapping entries of the corresponding primary keys in each table into the intermediate table; The user maintains a many-to-many relationship with the Docker image by introducing an additional intermediate table and writing mapping entries for the corresponding primary keys in each table into the intermediate table.

7. A large-scale topology orchestration and management system for virtual networks according to claim 1, characterized in that, The virtual network implementation layer deploys the virtual topology segmented to each slave server according to the deployment instructions, specifically including: Virtual nodes are created and virtual links are established during the topology creation phase; During the service startup phase, the routing and forwarding processes running on the nodes are started concurrently. The service startup phase adopts a coroutine-based concurrent approach and uses an actor programming model to encapsulate the creation of network components and service startup functions into different actors, which are managed through an actor pool. After an I / O operation request is issued, the coroutine waiting for the I / O operation to finish is asynchronously suspended, and the next operation is executed.

Images