Method and system capable of automatically calculating number of IBROCE network equipment

Abstract

The invention relates to the technical field of networks, in particular to a method and system capable of automatically calculating the number of IBROCE network devices, and the method comprises the following steps: a data collection step, a data analysis step, a device number calculation step and a result output step. The method has the beneficial effects that the required IB / ROCE network equipment quantity is accurately calculated, and excessive purchasing or insufficient purchasing of the equipment caused by inaccurate traditional manual estimation is avoided. Excessive procurement can cause a large amount of fund to be occupied by equipment, so that the fund cost and the equipment maintenance cost are increased; and insufficient purchase may cause network performance bottleneck, influence service development and cause potential service loss. Through accurate calculation of the system, the equipment purchase cost can be controlled within a reasonable range, unnecessary capital expenditure is reduced, and meanwhile, the operation and maintenance cost, the energy consumption cost and other related costs of the equipment are reduced.

Description

Technical Field

[0001] The present invention relates to the field of network technology, and specifically provides a method and system for automatically calculating the number of IB / ROCE network devices. Background Art

[0002] In the current era of rapid digital development, the network has become an indispensable infrastructure in various fields. Among them, IB (InfiniBand) and ROCE (RDMA over Converged Ethernet), as high-performance network technologies, have been widely used in scenarios such as data centers, cloud computing, and high-performance computing. In the field of intelligent computing, however, with the continuous expansion of the network scale and the increasing complexity of business requirements, accurately calculating the number of IB / ROCE network devices has become crucial.

[0003] Traditional methods for calculating the number of IB / ROCE network devices mainly rely on manual operations. This requires network administrators to have in-depth knowledge of network architectures, device performance, business traffic, etc., and determine the required number of devices through cumbersome manual calculations and evaluations. This approach is not only inefficient, consuming a large amount of human and time costs, but also easily affected by human factors, resulting in deviations or errors in the calculation results.

[0004] In addition, due to the dynamic changes in the network environment, such as increases or decreases in business volume, equipment upgrades and replacements, etc., traditional methods are difficult to accurately reflect the actual network requirements in real time, which may lead to waste or shortage of network resources, affecting network performance and service quality. For example, when the business volume suddenly increases significantly, if the corresponding network devices are not calculated and added in a timely and accurate manner, problems such as network congestion and increased latency may occur, affecting the normal operation of the business; conversely, if network devices are over-provisioned, it will result in idle resources and increased costs.

[0005] Therefore, there is an urgent need for a system that can automatically, efficiently, and accurately calculate the number of IB / ROCE network devices to meet the needs of modern network development. Summary of the Invention

[0006] The purpose of the present invention is to provide a method and system for automatically calculating the number of IB / ROCE network devices to solve the problems raised in the above background art.

[0007] To achieve the above purpose, the present invention provides the following technical solution: A method for automatically calculating the number of IB / ROCE network devices, the method comprising the following steps:

[0008] Data collection step: Quickly collect various types of data related to IB / ROCE devices in the network;

[0009] Data analysis steps: Use data cleaning techniques to remove noise and outliers from the data, apply feature extraction algorithms to extract key features, and perform classification and clustering analysis on the data through machine learning algorithms;

[0010] Device quantity calculation steps: Based on the results of data analysis, automatically calculate the required IB / ROCE network device quantity through a pre-established mathematical model, and reserve a certain margin considering the scalability and redundancy of the network;

[0011] Result output steps: Present the calculation results in a visual manner and provide a detailed report, including the calculation process, data analysis results, and recommended device configuration plan.

[0012] Preferably, the data analysis steps further include:

[0013] Use an outlier detection algorithm to identify and remove data points that significantly deviate from the normal range;

[0014] Use the principal component analysis (PCA) algorithm to reduce the data dimension and extract the feature vector that best reflects the network device requirements;

[0015] Use the K-Means clustering algorithm to classify the devices in the network into different categories and calculate the device quantity according to the characteristics of different categories.

[0016] Preferably, the mathematical model established in the device quantity calculation steps is based on a linear regression model and a neural network model, and the model is used to establish the relationship between the device quantity and network traffic and device performance factors.

[0017] Preferably, the method further includes an intelligent algorithm application step, which introduces machine learning algorithms, genetic algorithms, and simulated annealing algorithm optimization algorithms to optimize the calculation model, improve the accuracy and adaptability of device quantity calculation, and combine artificial intelligence technology to achieve automated network optimization and fault diagnosis.

[0018] Preferably, the method further includes a network security guarantee step, which encrypts the collected data, establishes a strict access control mechanism, and performs security vulnerability scanning and repair on the system to ensure the security and privacy of the data and the security of the system.

[0019] A system for automatically calculating the number of IBROCE network devices, which is applied to a method for automatically calculating the number of IBROCE network devices. The system includes:

[0020] Data collection module: Quickly collect various types of data related to IB / ROCE devices in the network;

[0021] Data Analysis Module: It uses data cleaning techniques to remove noise and outliers from the data, applies feature extraction algorithms to extract key features, and conducts classification and clustering analysis on the data through machine learning algorithms;

[0022] Device Quantity Calculation Module: Based on the results of data analysis, it automatically calculates the required IB / ROCE network device quantity through a pre-established mathematical model, and reserves a certain margin considering the scalability and redundancy of the network;

[0023] Result Output Module: It presents the calculation results in a visual manner and provides a detailed report, including the calculation process, data analysis results, and recommended device configuration solutions.

[0024] Preferably, the data analysis module further includes:

[0025] Use an outlier detection algorithm to identify and eliminate data points that significantly deviate from the normal range;

[0026] Use the principal component analysis (PCA) algorithm to reduce the data dimension and extract the feature vector that best reflects the network device requirements;

[0027] Use the K-Means clustering algorithm to classify the devices in the network into different categories, and calculate the device quantity according to the characteristics of different categories.

[0028] Preferably, the mathematical model established in the device quantity calculation module is based on a linear regression model and a neural network model, and the model is used to establish the relationship between the device quantity and network traffic and device performance factors.

[0029] Preferably, the system further includes an intelligent algorithm application step, which introduces machine learning algorithms, genetic algorithms, and simulated annealing algorithm optimization algorithms to optimize the calculation model, improve the accuracy and adaptability of device quantity calculation, and combine artificial intelligence technology to achieve automated network optimization and fault diagnosis.

[0030] Preferably, the system further includes a network security guarantee step, which encrypts the collected data, establishes a strict access control mechanism, and conducts security vulnerability scanning and repair on the system to ensure the security and privacy of the data and the security of the system.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] The method and system for automatically calculating the number of IBROCE network devices proposed by the present invention accurately calculate the required amount of IB / ROCE network devices, avoiding over-purchasing or under-purchasing of devices caused by inaccurate traditional manual estimation. Over-purchasing will result in a large amount of funds being idle and occupied by devices, increasing the capital cost and device maintenance cost; under-purchasing may lead to network performance bottlenecks, affecting business operations and causing potential business losses. Through accurate calculation by this system, the device procurement cost can be controlled within a reasonable range, reducing unnecessary capital expenditures, and at the same time reducing related costs such as device operation and maintenance and energy consumption.

[0033] The system can calculate the number of network devices in real-time or quickly. Compared with traditional manual calculation which may take several days or even weeks, this system can provide accurate results within a short time (such as from a few minutes to several hours). This enables network administrators and enterprise decision-makers to quickly make decisions regarding network planning, upgrading, or optimization, seize market opportunities or respond to business changes, greatly shortening the decision-making cycle and improving the flexibility and agility of enterprise operations. The automated calculation process reduces the time for manual data collection, analysis, and calculation, and network operation and maintenance personnel can use the saved time for other important network management tasks, such as network security monitoring, performance optimization, etc., improving the overall work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is the high-speed non-blocking network design architecture diagram of the present invention;

[0035] Figure 2 It is the over-converged switch architecture diagram of the present invention;

[0036] Figure 3 It is the two-layer fat tree architecture diagram of the present invention;

[0037] Figure 4 It is the three-layer fat tree architecture diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] In order to clearly and completely describe the objectives, technical solutions of the present invention, and make the advantages more clear, the following further details the embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, not to limit the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0039] Embodiment 1, the present invention provides a technical solution: a method for automatically calculating the number of IBROCE network devices, the method includes the following steps:

[0040] Data collection step: Rapidly collect various types of data related to IB / ROCE devices in the network.

[0041] Data analysis step: Use data cleaning techniques to remove noise and outliers from the data, apply feature extraction algorithms to extract key features, and perform classification and clustering analysis on the data through machine learning algorithms; This step further includes: Using an outlier detection algorithm to identify and eliminate data points that deviate significantly from the normal range; Using the principal component analysis (PCA) algorithm to reduce the data dimension and extract the feature vectors that best reflect the requirements of network devices; Using the K-Means clustering algorithm to classify the devices in the network into different categories and calculate the device quantity according to the characteristics of different categories.

[0042] Device quantity calculation step: Based on the results of data analysis, automatically calculate the required quantity of IB / ROCE network devices through a pre-established mathematical model, and reserve a certain margin considering the scalability and redundancy of the network; The established mathematical model is based on a linear regression model and a neural network model, and the model is used to establish the relationship between the device quantity and network traffic and device performance factors.

[0043] Result output step: Present the calculation results in a visual manner and provide a detailed report, including the calculation process, data analysis results, and recommended device configuration plan.

[0044] The method further includes an intelligent algorithm application step, which introduces machine learning algorithms, genetic algorithms, simulated annealing algorithms and other optimization algorithms to optimize the calculation model, improve the accuracy and adaptability of device quantity calculation, and combine artificial intelligence technology to achieve automated network optimization and fault diagnosis; The method further includes a network security guarantee step, which encrypts the collected data, establishes a strict access control mechanism, and performs security vulnerability scanning and repair on the system to ensure the security and privacy of the data and the security of the system.

[0045] Example 2, please refer to Figures 1 to 4 On the basis of Example 1, a system for automatically calculating the number of IBROCE network devices is proposed, which is applied to a method for automatically calculating the number of IBROCE network devices. The system includes:

[0046] 1. System architecture design

[0047] Data analysis module:

[0048] Adopt data cleaning techniques to remove noise and outliers from the collected data, ensuring the accuracy and reliability of the data. For example, an outlier detection algorithm can be used to identify and eliminate data points that deviate significantly from the normal range.

[0049] Use feature extraction algorithms to extract key features related to network device quantity calculation from a large amount of data. For example, algorithms such as principal component analysis (PCA) can be used to reduce the data dimension and extract the feature vectors that best reflect network device requirements.

[0050] Based on machine learning algorithms, classify and cluster the data to better understand the device distribution and usage in the network. For example, the K-Means clustering algorithm can be used to divide the devices in the network into different categories and calculate the device quantity according to the characteristics of different categories.

[0051] Calculation model module:

[0052] Establish a mathematical model and calculate the required IB / ROCE network device quantity according to the results of data analysis. For example, based on linear regression models, neural network models, etc., establish a relationship model between device quantity and factors such as network traffic and device performance.

[0053] Consider the scalability and redundancy of the network and reserve a certain margin when calculating the device quantity to cope with future business growth and device failures.

[0054] Combined with the development trends of the intelligent computing industry, such as the application of technologies like artificial intelligence and big data, continuously optimize and adjust the calculation model to improve the accuracy and adaptability of the calculation.

[0055] Result output module:

[0056] Present the calculation results in a visual way, such as through charts, reports, etc., to intuitively display information such as the required IB / ROCE device quantity and distribution in the network.

[0057] Provide a detailed report, including the calculation process, data analysis results, and recommended device configuration plans, etc., to provide a reference for user decision-making.

[0058] Support integration with other network management systems, automatically import the calculation results into network planning and configuration tools, and achieve automated network optimization and upgrade.

[0059] 2 Key technology implementation

[0060] High-performance data processing technology:

[0061] Optimize algorithms and data structures to reduce memory occupancy and calculation time during the calculation process. For example, efficient data structures such as hash tables and binary trees can be used to improve the data search and sorting efficiency.

[0062] Utilize hardware acceleration technologies such as GPU (Graphics Processing Unit) acceleration and FPGA (Field Programmable Gate Array) acceleration to improve the performance of data processing.

[0063] High-speed non-blocking network design:

[0064] According to the requirements of the intelligent computing center architecture, for IB / ROCE networks, high-speed non-blocking networks are required for design. The key to non-blocking network design is to adopt the Fat-Tree network architecture. The downlink and uplink bandwidths of the switch adopt a 1:1 non-convergent design, that is, if there are 64 100Gbps ports in the downlink, then there are also 64 100Gbps ports in the uplink.

[0065] In terms of low-latency network architecture design, the two-layer IB / ROCE network adopts the AI-Pool network solution. In this network solution, 8 access switches form a group to form an AI-Pool. Taking the two-layer switch networking architecture as an example, this network architecture can achieve that the GPU mutual access between different intelligent computing nodes in the same AI-Pool only requires one hop.

[0066] In the AI-Pool network architecture, the network ports with the same number between different intelligent computing nodes need to be connected to the same switch. For example, the No. 1 RDMA network port of intelligent computing node 1, the No. 1 RDMA network port of intelligent computing node 2 until the No. 1 RDMA network port of intelligent computing node P / 2 are all connected to the No. 1 switch.

[0067] Inside the intelligent computing node, the upper-layer communication library performs network matching based on the in-machine network topology to associate the GPU cards with the same number and the network ports with the same number. In this way, the two intelligent computing nodes with the same GPU number can communicate with each other in only one hop.

[0068] Between intelligent computing nodes with different GPU numbers, by means of the Rail Local technology in the NCCL communication library, the bandwidth of the NVSwitch between GPUs in the host can be fully utilized to convert the cross-card communication between multiple machines into the communication between the same GPU cards between multiple machines.

[0069] For the communication between two physical machines across AI-Pools, it needs to pass through the aggregation switch, and there will be 3 hops at this time.

[0070] The scale of GPU cards that the network can carry is related to the port density of the switches adopted and the network architecture. The more network layers, the larger the scale of GPU cards that can be carried, but the number of forwarding hops and latency will also increase, and it is necessary to make a balance in combination with the actual business situation.

[0071] ① Two-layer fat tree architecture

[0072] Eight access switches form an AI-Pool. In the figure, P represents the number of ports on a single switch. A single switch can have a maximum of P / 2 downstream and upstream ports, which means that a single switch can be connected to a maximum of P / 2 servers and P / 2 switches. A two-layer fat tree network can connect to P*P / 2 GPU cards.

[0073] ②Three-layer fat tree architecture

[0074] In the three-layer network architecture, aggregation switch groups and core switch groups will be added. The maximum number of switches in each group is P / 2. The maximum number of aggregation switch groups is 8, and the maximum number of core switch groups is P / 2. The three-layer fat tree network can connect P*(P / 2)*(P / 2)=P*P*P / 4 GPU cards.

[0075] In a three-layer fat-tree network, the maximum number of GPUs that can be accommodated by InfiniBand's 40-port 200Gbps HDR switch is 16,000. This scale of 16,000 GPU cards is also the largest scale network of GPU clusters currently used in China by InfiniBand, and this record is currently held by Baidu.

[0076] Intelligent algorithm application:

[0077] Introduce machine learning algorithms, such as deep learning and reinforcement learning, to perform intelligent analysis and prediction of network data and improve the accuracy and adaptability of equipment quantity calculation.

[0078] The calculation model is optimized using optimization algorithms such as genetic algorithms and simulated annealing algorithms to find the optimal equipment configuration solution.

[0079] Combined with artificial intelligence technology, it can realize automated network optimization and fault diagnosis, and improve network reliability and stability.

[0080] Network security protection:

[0081] The collected data is encrypted to ensure data security and privacy. For example, encryption algorithms such as AES (Advanced Encryption Standard) can be used to encrypt data transmission and storage.

[0082] Establish a strict access control mechanism to limit access to the system and prevent unauthorized users from accessing and modifying system data.

[0083] Scan and repair the system for security vulnerabilities to promptly identify and resolve potential security risks.

[0084] Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatically calculating the number of IBROCE network devices, characterized by: The method comprises the following steps: Data collection steps: Quickly collect various data related to IB / ROCE devices in the network; Data analysis steps: Use data cleaning technology to remove noise and outliers in the data, use feature extraction algorithms to extract key features, and use machine learning algorithms to classify and cluster the data; Equipment quantity calculation steps: Based on the results of data analysis, the required IB / ROCE network equipment quantity is automatically calculated through a pre-established mathematical model, while a certain margin is reserved considering the scalability and redundancy of the network; Result output step: present the calculation results in a visual way and provide a detailed report, including the calculation process, data analysis results and recommended equipment configuration solutions.

2. A method for automatically calculating the number of IBROCE network devices according to claim 1, characterized in that: The data analysis steps further include: Use outlier detection algorithms to identify and remove data points that significantly deviate from the normal range; Use the principal component analysis (PCA) algorithm to reduce the data dimension and extract the feature vector that best reflects the needs of network equipment; Use the K-Means clustering algorithm to classify the devices in the network into different categories, and calculate the number of devices based on the characteristics of different categories.

3. A method for automatically calculating the number of IBROCE network devices according to claim 1, characterized in that: The mathematical model established in the device quantity calculation step is based on a linear regression model and a neural network model. The model is used to establish the relationship between the device quantity and network traffic and device performance factors.

4. The method for automatically calculating the number of IBROCE network devices according to claim 1, characterized in that: The method also includes an intelligent algorithm application step, which introduces a machine learning algorithm, a genetic algorithm, and a simulated annealing algorithm optimization algorithm to optimize the calculation model, improve the accuracy and adaptability of equipment quantity calculation, and combine artificial intelligence technology to achieve automated network optimization and fault diagnosis.

5. The method for automatically calculating the number of IBROCE network devices according to claim 1, characterized in that: The method also includes a network security assurance step, which encrypts the collected data, establishes a strict access control mechanism, and scans and repairs security vulnerabilities in the system to ensure the security and privacy of the data, as well as the security of the system.

6. A system capable of automatically calculating the number of IBROCE network devices, applied to a method capable of automatically calculating the number of IBROCE network devices as described in any one of claims 1 to 5, characterized in that: The system comprises: Data collection module: quickly collect various data related to IB / ROCE devices in the network; Data analysis module: Use data cleaning technology to remove noise and outliers in the data, use feature extraction algorithms to extract key features, and use machine learning algorithms to classify and cluster the data; Equipment quantity calculation module: Based on the results of data analysis, the required IB / ROCE network equipment quantity is automatically calculated through a pre-established mathematical model, while a certain margin is reserved considering the scalability and redundancy of the network; Result output module: presents the calculation results in a visual way and provides a detailed report, including the calculation process, data analysis results and recommended equipment configuration solutions.

7. A system for automatically calculating the number of IBROCE network devices according to claim 6, characterized in that: The data analysis module further includes: Use outlier detection algorithms to identify and remove data points that significantly deviate from the normal range; Use the principal component analysis (PCA) algorithm to reduce the data dimension and extract the feature vector that best reflects the needs of network equipment; Use the K-Means clustering algorithm to classify the devices in the network into different categories, and calculate the number of devices based on the characteristics of different categories.

8. The system for automatically calculating the number of IBROCE network devices according to claim 6, characterized in that: The mathematical model established in the equipment quantity calculation module is based on the linear regression model and the neural network model. The model is used to establish the relationship between the equipment quantity and the network traffic and equipment performance factors.

9. The system for automatically calculating the number of IBROCE network devices according to claim 6, characterized in that: The system also includes an intelligent algorithm application step, which introduces machine learning algorithms, genetic algorithms, and simulated annealing algorithm optimization algorithms to optimize the calculation model, improve the accuracy and adaptability of equipment quantity calculations, and combine artificial intelligence technology to achieve automated network optimization and fault diagnosis.

10. The system for automatically calculating the number of IBROCE network devices according to claim 6, characterized in that: The system also includes a network security assurance step, which encrypts the collected data, establishes a strict access control mechanism, and scans and repairs security vulnerabilities in the system to ensure the security and privacy of the data, as well as the security of the system.

Images