High-speed network data acquisition method, device and storage medium

Abstract

The application provides a high-speed network data collection method, device and storage medium. According to the device information of servers and switches in a cluster, data collection tasks including server data collection tasks, switch data collection tasks and network link collection tasks are determined to provide all-around device index and link data collection with high collection accuracy. In a compact device layout of a micro data center, problems at the link, switch and server network card levels can be accurately located. According to the task type, a data collection module is matched for each data collection task. According to the collection period of the data collection task, the task type and collection parameters are specified to the data collection module at a fixed time to collect data, obtain target data, improve efficiency through timed and concurrent collection and meet real-time monitoring requirements.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of computer network monitoring, and in particular to a high-speed network data collection method, device and storage medium. BACKGROUND

[0002] With the rapid development of large models and artificial intelligence technologies, the demand for computing power is growing explosively, and the scale of computing power centers is expanding. Micro data centers are widely used in enterprise edge computing, small-scale AI training in research institutions, local data processing in smart communities, traffic control, and emergency rescue scenarios due to their controllable cost, flexible deployment, and close proximity to business needs. Even micro data centers frequently exchange massive amounts of data between computing nodes during model training and inference, as well as visual processing, making network communication a potential system bottleneck. To fully release the computing power potential of GPU, NPU, and other computing acceleration cards, and avoid communication delay between cards affecting AI model running efficiency, micro data centers generally use high-speed interconnection network technologies such as wireless bandwidth and Ethernet remote memory direct access.

[0003] The stability and performance of these high-speed networks form the basis for the normal operation of upper-layer AI applications, and have become a key link in the operation and maintenance of micro data centers. However, existing high-speed network monitoring technologies have deficiencies, such as scattered storage of collected data in different systems, low collection efficiency in the case of relatively limited resources in micro data centers, and difficulty in meeting real-time monitoring requirements. SUMMARY

[0004] To overcome the problems in the related art, the present specification provides a high-speed network data collection method, device and storage medium.

[0005] According to a first aspect of an embodiment of the present specification, a method is provided, applied to a micro data center cluster, the method comprising:

[0006] According to the device information of the servers and switches in the cluster, determine data collection tasks including server data collection tasks, switch data collection tasks, and network link collection tasks;

[0007] According to the type of the task, match a data collection module for each of the data collection tasks;

[0008] According to the collection period of the data collection tasks, specify the type of the task and the collection parameters to the data collection module at regular intervals, so that the data collection module collects data to obtain target data;

[0009] Obtain the target data, which is used to monitor the high-speed network status of the micro data center in real time.

[0010] According to the high-speed network data collection method provided in the specification, the task type and collection parameters are specified to the data collection module at a collection period of the data collection task to perform data collection, and target data is obtained, comprising:

[0011] The configuration of the data collection task including the server data collection task and the switch data collection task is analyzed to determine the collection period of the data collection task.

[0012] Tasks with the same collection period are divided into a group, and an independent timing collection process is started for each group of tasks. The task type and collection parameters are specified to the data collection module corresponding to the server data collection task and the switch data collection task, respectively, to perform data collection and obtain target data.

[0013] According to the high-speed network data collection method provided in the specification, the collection period of the data collection task is adjusted according to the change frequency and importance of the collected indicators.

[0014] According to the high-speed network data collection method provided in the specification, the method further comprises:

[0015] The configuration of the network link collection task is analyzed to determine the collection period of the link collection task.

[0016] An independent timing collection process is started for the link collection task, and the task type and collection parameters are specified to the data collection module corresponding to the link collection task.

[0017] According to the high-speed network data collection method provided in the specification, the collection parameters include the number of concurrent query operations.

[0018] The method further comprises:

[0019] Concurrent control is performed on the data collection task to determine the upper limit of the number of query operations performed by the data collection module at the same time.

[0020] According to the high-speed network data collection method provided in the specification, the method further comprises:

[0021] The number of concurrent query operations allowed to be executed simultaneously is dynamically adjusted according to the device load state and historical collection success rate.

[0022] According to the high-speed network data collection method provided in the specification, the method further comprises:

[0023] The execution state of the data collection module performing the data collection task is monitored.

[0024] When an exception occurs, a corresponding processing strategy is executed according to the exception type.

[0025] According to the high-speed network data acquisition method provided in the specification, the target data is obtained, and the target data is used to feed back the high-speed network status of the micro data center in real time, and the method comprises the following steps:

[0026] Obtaining the target data;

[0027] Marking the device source and timestamp of the target data, setting the label and version number, and storing the target data in a unified format into a distributed columnar database for real-time monitoring of the status of the high-speed network of the micro data center.

[0028] According to the high-speed network data acquisition method provided in the specification, the method further comprises:

[0029] The target data comprises device index data and link topology data; the device index data is used to monitor the status of the device in the cluster, and the link topology data is used to monitor the network link status.

[0030] According to the high-speed network data acquisition method provided in the specification, the method is applied to a data acquisition module in a micro data center cluster, and the cluster comprises a server; the acquisition parameters comprise an acquisition period;

[0031] The server data acquisition task is executed by the data acquisition module, and the method comprises:

[0032] Based on the configured acquisition period, the high-speed network device information under the system path is read to obtain the server network card index; the system path is configured as a path for storing high-speed network device files;

[0033] The data of the network card index is sent to the data storage module in a preset format.

[0034] According to the high-speed network data acquisition method provided in the specification, the method further comprises:

[0035] The server network card index is obtained by non-intrusive reading.

[0036] According to the high-speed network data acquisition method provided in the specification, the method is applied to a data acquisition module in a micro data center cluster, and the cluster comprises a switch; the switch data acquisition task is executed by the data acquisition module, and the method comprises:

[0037] Obtaining the switch configuration information corresponding to the switch data acquisition task;

[0038] According to configuration information of the switch, a connection with the switch is established through a network management protocol;

[0039] According to the configured collection period, operation indexes of the switch are collected;

[0040] Data of the operation indexes of the switch are sent to a data storage module in a preset format.

[0041] According to a high-speed network data collection method provided in the present specification, the method is applied to a data collection module in a micro-small data center cluster, the cluster comprising servers and switches, and the switches comprising leaf switches and backbone switches;

[0042] The link collection task is executed by the data collection module, and the method further comprises:

[0043] Server link information is obtained based on a link discovery protocol, and the connection relationship between the servers and the leaf switch ports on the server side is obtained, thereby obtaining the server link information;

[0044] The server link information is structured and processed, and link state changes are recorded;

[0045] The server link information is sent to a data storage module.

[0046] According to a high-speed network data collection method provided in the present specification, the link collection task is executed by the data collection module, and the method further comprises:

[0047] The connection relationship between the leaf switches and the backbone switches is obtained through a network management protocol, thereby obtaining switch link information;

[0048] The switch link information is structured and processed, and link state changes are recorded;

[0049] The switch link information is sent to a data storage module.

[0050] According to a second aspect of the embodiments of the present specification, a device is provided, comprising:

[0051] The device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the high-speed network data collection method according to any one of the above.

[0052] According to a third aspect of the embodiments of the present specification, a computer readable storage medium is provided, comprising:

[0053] The computer program is executed by the processor to implement the high-speed network data collection method according to any one of the above.

[0054] The technical scheme provided by the embodiments of the present specification can include the following beneficial effects:

[0055] In the embodiments of the present specification, according to the device information of the servers and switches in the cluster, the data collection tasks including the server data collection task, the switch data collection task, and the network link collection task are determined, the collection of the full range of device indicators and link data is provided, the collection accuracy is high, and under the compact device layout of the micro data center, the problems at the link, switch and server network card levels can be accurately located. According to the task type, the data collection module is matched for each data collection task; according to the collection period of the data collection task, the task type and the collection parameter are specified to the data collection module at regular intervals, so that the data collection module performs data collection to obtain target data, and the efficiency is improved through regular concurrent collection to meet the real-time monitoring requirement.

[0056] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present specification. BRIEF DESCRIPTION OF DRAWINGS

[0057] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present specification and, together with the specification, serve to explain the principles of the present specification.

[0058] Figure 1 is a data collection system schematic diagram according to an exemplary embodiment of the present specification.

[0059] Figure 2 is a flowchart of a high-speed network data collection method according to an exemplary embodiment of the present specification.

[0060] Figure 3 is another flowchart of a high-speed network data collection method according to an exemplary embodiment of the present specification.

[0061] Figure 4 is a device schematic diagram according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0062] The exemplary embodiments will be described in detail hereinafter with reference to the accompanying drawings. In the following description, the same numbers refer to the same elements throughout the drawings, unless otherwise represented. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present specification. Rather, they are merely examples of devices and methods consistent with some aspects of the present specification, as detailed in the appended claims.

[0063] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0064] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0065] This specification provides a method, device, and computer-readable storage medium for high-speed network data collection. The following describes the embodiments of this specification in detail with reference to the accompanying drawings. The features of the following embodiments and implementations may be combined with each other unless they conflict.

[0066] Existing high-speed network monitoring technology has the following shortcomings: existing network monitoring systems are mainly designed for traditional Ethernet and lack support for the collection of high-speed network-specific indicators; the collection granularity is relatively coarse, and it is difficult to accurately locate problems at the specific network link, switch or server network card level under the compact equipment layout of micro-small data centers; the collected data is stored in different systems in a scattered manner, lacking a unified collection and analysis framework; and in the case of relatively limited resources in micro-small data centers, the collection efficiency is low and it is difficult to meet real-time monitoring needs.

[0067] In order to solve the above technical problems, this specification provides a high-speed network data acquisition method.

[0068] It is designed to comprehensively collect network link, switch, and server network card indicators to achieve all-round monitoring of high-speed networks. It can promptly detect network anomalies, ensure the smooth execution of AI computing tasks, and improve the operating efficiency and stability of micro-data centers.

[0069] like Figure 1 As shown, Figure 1 This is a schematic diagram of a data acquisition system according to an exemplary embodiment of this specification.

[0070] The data acquisition system includes a switch acquisition module, a server acquisition module, a link acquisition module, a data storage module, and a management and scheduling module.

[0071] The micro data center cluster includes servers and switches. The switches include leaf switches and spine switches, wherein the leaf switches belong to an access layer and are directly connected to terminal devices such as servers, and the spine switches belong to a core layer and are fully interconnected with all the leaf switches.

[0072] The switch collection module is configured to establish a management connection with the switches in the cluster, collect various operation indexes of the switches by executing a switch data collection task issued by the management scheduling module, and collect various operation indexes of the switches. The operation indexes include, but are not limited to, port traffic, CPU usage, memory occupation, fan state, power supply state, optical module parameters, and the like. According to the device manufacturer model, a corresponding object identifier mapping table is read to realize compatible collection of switches of different manufacturers. This module supports automatic identification and conversion processing of different data types, including byte arrays, hexadecimal arrays, integer values, and the like, and adds corresponding label information to different types of indexes.

[0075] The switch collection module is configured to establish a management connection with the switches in the cluster, collect various operation indexes of the switches by executing a switch data collection task issued by the management scheduling module, and collect various operation indexes of the switches. The operation indexes include, but are not limited to, port traffic, CPU usage, memory occupation, fan state, power supply state, optical module parameters, and the like. According to the device manufacturer model, a corresponding object identifier mapping table is read to realize compatible collection of switches of different manufacturers. This module supports automatic identification and conversion processing of different data types, including byte arrays, hexadecimal arrays, integer values, and the like, and adds corresponding label information to different types of indexes.

[0073] By monitoring the running state of the switches in real time through the switch collection module, potential failure risks of the switches, such as CPU overload and fan failure, can be found in advance to ensure stable operation of the network core device.

[0074] The server collection module is configured to deploy a collector instance on each server in the micro data center cluster, and is responsible for collecting various operation index data of the server network card, including but not limited to board card ID, firmware version, port state, transmission rate, and error packet number.

[0075] Through the server collection module, detailed data of the server network running state is provided to provide basic information for network performance analysis and fault diagnosis. For example, when a large number of error packets of the server network card are found, the server-side network exception can be quickly located.

[0076] The link collection module is responsible for discovering and maintaining the network topology relationship, which includes server link collection and switch link collection. The server link collection uses LLDP (Link Layer Discovery Protocol) to obtain the connection relationship between the server and the leaf switch; the switch link collection uses SNMP (Simple Network Management Protocol) to obtain the connection relationship between the leaf switch and the spine switch. The link collection module performs structured processing on the collected link information, records the link state changes (device name, port name, link state, and the like), and compares with historical data to realize tracking of link changes.

[0077] The data collected by the link collection module constructs and maintains the network topology of the micro data center, and tracks the link connection state in real time. When the link appears abnormal (such as disconnection, congestion), the fault link is quickly located.

[0078] The data storage module is responsible for the persistent storage and query support of the collected data. In the case of limited resources in the micro data center, the module uses a distributed columnar database to store a large amount of time series index data, and designs optimized table structure and partitioning strategy for different types of data to support efficient writing and querying. The data storage module also implements data version management, identifies the network topology state at different time points through version numbers, and facilitates historical backtracking and change analysis.

[0079] Through the data storage module as the data warehouse of the entire collection system, the safe storage and fast retrieval of data are guaranteed, and reliable data support is provided for network monitoring and analysis.

[0080] The management and scheduling module, as the central controller of the entire collection system, is responsible for configuration management, task scheduling and resource coordination.

[0081] The management and scheduling module provides configuration information for the server collection module, switch collection module and link collection module, including but not limited to system path, object identifier mapping table, link discovery protocol dependency, etc.

[0082] In the running environment of the micro data center, the module maintains the vendor management information base mapping table, sets different collection time intervals according to the importance and frequency of change of the indicators, groups the indicators and creates corresponding timing tasks.

[0083] The management and scheduling module implements a concurrent control mechanism to limit the number of concurrent query operations at the same time, avoiding excessive load on network devices. The module is also responsible for handling abnormal situations during the collection process, including but not limited to connection timeout, data parsing error, device unreachable, etc., to ensure the stable operation of the collection system.

[0084] During data collection, the management and scheduling module issues data collection tasks and configuration information to other data collection modules.

[0085] Through the above data collection system, comprehensive collection of network link, switch indicators and server network card indicators can be realized, the device connection relationship is brought into monitoring through link data, forming a global view of "device-link-system". And through the management scheduling module, the data collection task is uniformly scheduled, a unified collection and analysis framework is constructed, and the collection stability is improved through active control and concurrency. At the same time, under the condition of compact deployment of devices, through fine "port level" indicator collection (such as single network card port state) and "link level" topology tracking, the problem that traditional monitoring is difficult to accurately locate the abnormality is solved, which is especially suitable for efficient operation and maintenance in resource limited scenarios.

[0086] The present specification provides an embodiment of a high-speed network data collection method. The high-speed network data collection method is applied to the above-mentioned data collection system.

[0087] As shown in Figure 2 , the present specification provides an embodiment of a high-speed network data collection method. The high-speed network data collection method is applied to the above-mentioned data collection system. Figure 2 is a flowchart of a method according to an exemplary embodiment of the present specification, comprising the following steps:

[0088] In step 102, according to the device information of servers and switches in the cluster, determine the data collection tasks including server data collection tasks, switch data collection tasks, and network link collection tasks.

[0089] The high-speed network data collection method of the present specification can be applied to data centers, micro-small data centers, and the size of the data center is not specifically limited.

[0090] As an example, the micro-small data center cluster includes servers and switches connected with the servers. Through data collection of servers, switches and network links, comprehensive monitoring of high-speed networks of micro-small data centers is realized.

[0091] The network link includes the link between the server and the switch, and the link between the switch and the switch.

[0092] As an example, the high-speed network data collection method is applied to the management scheduling module in the data collection system.

[0093] In combination with Figure 3 , first, initialize the collector on each server of the micro-small data center cluster, configure the system path, log processor, node name, namespace, mount the dynamic library and dependency of LLDP protocol, and create a new collector instance.

[0094] The system path is a system path of a high-speed network device file storage and is related to a server Linux kernel version. A default system path of the collector is / sys / class / infiniband / , in other words, the operating system will place high-speed network device information and counter information of monitoring indexes in this folder and nested folders thereof by default.

[0095] The log processor is a logger for recording various information, warnings and errors, and can select a standard log, a Prometheus log, etc.

[0096] The node name is a name of the server and is used to identify that the index comes from which server.

[0097] The namespace is a custom string and is used to splice in front of the index to serve as a prefix and distinguish different kinds of indexes.

[0098] The above scheme establishes a basic environment for data collection of the server side and ensures configuration consistency of subsequent operations.

[0099] Then, the management scheduling module starts a manager in a special operation and maintenance management area of the micro data center, configures a database of the manager, an external access port and a query concurrency number.

[0100] As an example, according to device information such as a network card type of a server in the cluster and a vendor model of a switch, a data collection task is set. The data collection task includes a server data collection task, a switch data collection task and a network link collection task, so as to realize comprehensive collection of network link indexes, switch indexes and server network card indexes and promote all-around monitoring of the high-speed network.

[0101] In step 104, according to the task type, a data collection module is matched for each data collection task.

[0102] As an example, the data collection module is a general term of a switch collection module, a server collection module and a link collection module, and each collection module respectively executes a collection task. For example, the switch data collection task is executed by the switch collection module, the server data collection task is executed by the server collection module and the network link collection task is executed by the link collection module.

[0103] As another example, the data collection module includes a switch collection unit, a server collection unit and a link collection unit, and each unit in the data collection module respectively executes a collection task.

[0104] The data collection module is initialized.

[0105] For example, the server collection module is configured with a system path and a log processor related to a server Linux kernel version. The switch collection module is configured with an object identifier (OID) mapping table corresponding to a device vendor model. The link collection module is configured with a link discovery protocol dynamic library and dependencies. The data collection modules are initialized for data collection based on the configurations.

[0106] The execution module and initialization parameters of each data collection task are recorded.

[0107] At step 106, the data collection module is specified with a task type and collection parameters at a collection period of the data collection task for data collection by the data collection module to obtain target data.

[0108] The management scheduling module analyzes each data collection task to determine the collection period of the data collection task. Based on the collection period, the data collection module corresponding to the data collection task with the same collection period is instructed for data collection, and the instruction includes the task type and collection parameters specified for execution by the data collection module. The collection parameters include but are not limited to the collection period, the number of concurrent query operations, connection configuration, etc.

[0109] In some embodiments, the data collection module is specified with a task type and collection parameters at a collection period of the data collection task for data collection by the data collection module to obtain target data, including:

[0110] The configuration of the data collection task including the server data collection task and the switch data collection task is parsed to determine the collection period of the data collection task.

[0111] Tasks with the same collection period are divided into a group, and an independent timing collection process is started for each group of tasks. The data collection module corresponding to the server data collection task and the switch data collection task is specified with a task type and collection parameters for data collection by the data collection module to obtain target data.

[0112] The management scheduling module parses the configuration of the data collection task to determine the collection time interval, i.e., the collection period.

[0113] The vendor management information (OID mapping) is initialized, and for the required object identifier, the index name, label, and collection time interval are defined. For example, the server data collection task and the switch data collection task, such as device index tasks, periodically collect server CPU / memory, switch port traffic, etc. The time interval can be configured (e.g., 10 seconds, 1 minute).

[0114] Grouping the indicators by collection time interval, grouping the indicators of the same collection time interval, the manager traverses all groups and starts a timing task for each group corresponding to the collection time interval, the task interval is the collection time interval of the group, and the task content is the collection of the indicators in the group.

[0115] The grouped tasks are assigned to the corresponding data collection modules, such as server data collection tasks assigned to server collection modules and switch data collection tasks assigned to switch collection modules.

[0116] When the management scheduling module assigns tasks, it specifies the task type and parameters, including but not limited to server collection, switch collection, link collection, etc., and the collection parameters include but are not limited to time interval, indicator type, etc.

[0117] In some embodiments, the collection period of the data collection task is adjusted according to the change frequency and importance of the collected indicators.

[0118] A vendor management information base mapping table is maintained, and server, switch and link collection tasks are grouped by time interval according to the importance and change frequency of the indicators, and a corresponding timing task is started for each group. For example, frequently changing server CPU usage indicators are set as short-interval collection tasks, and slowly changing switch firmware versions are set as long-interval collection tasks.

[0119] In the above manner, multi-task parallel collection can shorten the overall data collection period, reduce the number of interactions between the scheduling module and the collection module, and meet the real-time monitoring needs of micro data centers. At the same time, the same group of tasks triggers collection at the same time, ensuring consistent data timestamps and facilitating subsequent correlation analysis (such as synchronous monitoring of server load and switch traffic).

[0120] In some embodiments, the method further comprises:

[0121] Parsing the configuration of the network link collection task to determine the collection period of the link collection task;

[0122] Starting an independent timing collection process for the link collection task, and specifying the task type and collection parameters to the data collection module corresponding to the link collection task.

[0123] The link collection task needs to be started separately because there is a fundamental difference between its collection logic and device indicator collection:

[0124] Link collection and indicator collection use different interfaces and data structures. Link collection detects connection information through link layer protocols and network management protocols, and then structures and parses it into data including source device ports, target device ports, and link-level data. Indicator collection relies on file system interfaces and network management protocols, and primarily includes indicator values, indicator names, and timestamps. Furthermore, link collection supports both scheduled and instant updates, while indicator collection only involves scheduled updates. Finally, link collection involves two layers of links. The second-layer link depends on the first-layer link, so the second-layer collection must be initiated after the first-layer link collection is updated. In contrast, indicator collection is initiated independently for each indicator.

[0125] Specifically, the configuration of the network link collection task is parsed to determine the collection interval for the Layer 1 link collection task (i.e., the collection period, such as midnight every morning). Based on this interval, a scheduled Layer 2 link collection task is initiated. This task starts after the Layer 1 link collection task is completed (e.g., 1 a.m. every morning). The task collects network link data from leaf switches to spine switches.

[0126] Through the above collection cycle management strategy, a balance between resource utilization and real-time performance is achieved through differentiated configuration.

[0127] In some embodiments, the manager is also responsible for recycling all scheduled tasks. When a service exits unexpectedly, the scheduled task is stopped and an error log is output.

[0128] In some embodiments, the collection parameter includes the number of concurrent query operations; the method further includes:

[0129] Concurrency control is performed on the data acquisition tasks to determine the upper limit of the number of query operations that the data acquisition module can perform at the same time.

[0130] Servers and switches have limited CPU, memory, and network interface resources. A large number of concurrent queries can cause slow device responses and even impact business operations. The scheduling module's core responsibility is to manage the scheduling and distribution of collection tasks, with concurrency control mechanisms as part of task scheduling.

[0131] When the management and scheduling module issues tasks to the data collection module, the module sends query requests (such as those for CPU utilization and port traffic) to the server or switch over the network. Concurrency control limits the number of concurrent queries sent to the device at any one time. This prevents excessive resource consumption at any given moment, accommodating the limited resources of micro-sized data centers. It also avoids excessive load on network devices and prevents data collection failures due to network congestion.

[0132] As an example, limit the number of simultaneous SNMP queries, such as operation threshold set to 10 concurrent connections, limit the maximum 10 query operations at the same time.

[0133] In some embodiments, according to the device load state and the historical acquisition success rate, the number of concurrent query operations allowed to be executed simultaneously is dynamically adjusted.

[0134] Through the above embodiments, the high-speed network data acquisition method can optimize the performance of the micro-small data center high-speed network, the grouping configuration meets different accuracy, concurrent acquisition improves efficiency, data compression reduces storage, and query optimization response is improved.

[0135] In some embodiments, the method further comprises:

[0136] Monitoring the execution state of the data acquisition module executing the data acquisition task;

[0137] When an exception occurs, a corresponding processing strategy is executed according to the exception type.

[0138] The management scheduling module is also responsible for handling abnormal situations during data acquisition, including but not limited to connection timeout, data parsing error, device unreachable, etc. When an abnormal situation occurs, the task is stopped in time, error logs are output, and corresponding retry or alarm measures are taken to ensure the stable operation of the acquisition system.

[0139] In some embodiments, the high-speed network data acquisition method is applied to a data acquisition module in a micro-small data center cluster, the cluster including servers and switches, and the data acquisition module executing a data acquisition task.

[0140] It should be noted that the data acquisition task, including the server data acquisition task, the switch data acquisition task, and the network link acquisition task, is executed in one data acquisition module or in an independent data acquisition module.

[0141] As an example, the data acquisition module is a server acquisition module, and the acquisition parameters include an acquisition period and a system path.

[0142] The server data acquisition task is executed by the data acquisition module, and the method comprises:

[0143] Based on the configured acquisition period, read the high-speed network device information under the system path to obtain the server network card indicators; the system path is configured as a path for storing high-speed network device files;

[0144] The data of the network card indicators is sent to the data storage module in a predetermined format.

[0145] According to the configured system path, read the file directory under the system path through the file system interface to obtain all high-speed network device information. Generally, after the server installs a network device (such as a network card), the operating system recognizes the network device and automatically creates a file directory under the system path, named as the device name. Meanwhile, a plurality of files and a plurality of subdirectories are created under the directory. Among them, the file directory usually has a board card ID, a firmware version and the like. The subdirectory has a plurality of port directories of the device, and each port directory has a counter subdirectory, which contains a plurality of index files.

[0146] If the file directory under the system path is not read through the file system interface, an error of no high-speed network device is returned.

[0147] In the first layer directory under the system path, the directory name is parsed as the name of the device. For each device, the second layer directory is explored to parse the file content to obtain the board card ID and the firmware version information. Then, the device port information is parsed by traversing the subdirectory to obtain the hardware counter and the general counter information, and the port state, the rate, the number of transmission packets, the number of error packets and the like are extracted from the counter. After the parsing of all the nested subdirectories and files under the system path is completed, the collected data is spliced, the node name, the IP address, the device name and the port number are taken as the index labels, the parsed file name is taken as the index name, and the numerical value is converted into a floating point number as the index value, which are reported to the index database.

[0148] Further, the server network card index is obtained through a non-intrusive reading manner.

[0149] The server collection module realizes non-intrusive reading of the InfiniBand or RoCE network card bottom counter through the file system interface, so as to ensure that the collection process does not affect the performance of the network card.

[0150] After the data collection is completed by the server collection module, the data is reported to the data storage module.

[0151] As an example, the data collection module is a switch collection module.

[0152] The method comprises the following steps:

[0153] Obtaining the switch configuration information corresponding to the switch data collection task;

[0154] According to the configuration information of the switch, a connection with the switch is established through a network management protocol;

[0155] Based on the configured collection period, the running index of the switch is collected;

[0156] The data of the operation index of the switch is sent to a data storage module in a preset format.

[0157] The management scheduling module provides object identifier configuration information to the switch collection module. In the process of establishing a connection with the switch, the switch collection module determines the required collection of the switch according to the switch configuration information issued by the management scheduling module. That is, the corresponding object identifier mapping table is read according to the equipment manufacturer model to realize compatible collection of switches of different manufacturers.

[0158] Specifically, the switch collection module queries the network equipment IP and manufacturer model from the database, reads the object identifier of the to-be-collected index from the configuration provided by the management scheduling module according to the equipment manufacturer model, establishes a Simple Network Management Protocol (SNMP) connection with the switch equipment, performs a data collection task at a task time interval set by the management scheduling module, collects various operation indexes of the switch, parses the returned protocol data unit, and returns the collected data to the data storage module.

[0159] The switch collection module supports automatic identification and conversion processing of different data types, including byte array, hexadecimal array, integer value, etc., and adds corresponding label information to indexes of different types.

[0160] Specifically, the switch collection module parses the returned protocol data unit, extracts the index, value, and type of the response data, and the extracted data index is used as part of the reported index label. According to the extracted data type, different processing is performed on the value.

[0161] If it is a byte array, it is converted into a string; if it is a hexadecimal array, it is converted into a decimal number and then into a floating-point number; if it is an integer, an unsigned integer, or the like, it is directly converted into a floating-point number. The parsed index data is labeled and reported.

[0162] The operation index includes but is not limited to port traffic, CPU usage, memory occupation, fan status, power supply status, optical module temperature, etc. Further, the collected indexes include but are not limited to port indexes, CPU indexes, memory indexes, CPU Core indexes, fan indexes, power supply indexes, and optical module indexes.

[0163] Among them, the port index uses IP, device name, port name, and index as labels, the CPU and memory index uses IP and device name as indexes, the CPU Core index uses IP, device name, and index as labels, and the fan, power supply, and optical module index uses IP, device name, index, and serial number as labels.

[0164] As an example, the data collection module is a link collection module, and the switch includes a leaf switch and a backbone switch.

[0165] The link collection includes server link collection and switch link collection.

[0166] The server link collection includes:

[0167] The method further includes:

[0168] The server link information is obtained based on a link discovery protocol.

[0169] The server link information is structured and processed to record link state changes.

[0170] The server link information is sent to a data storage module.

[0171] The management scheduling module sets an execution time for the link collection task of the link collection module, for example, collecting network link data from servers to leaf switches at 0 o'clock every day.

[0172] The link collection module triggers from each server at a regular time, scans devices connected to the server, obtains all opposite device information using an LLDP protocol service, and performs structured analysis on the obtained JSON and regular matching analysis on the obtained string to extract system name, system description, port name, port description, and management IP.

[0173] The system name and description of the device are then filtered to screen out leaf switches and record the collection information of the link from the server to the leaf switch.

[0174] The switch link collection includes:

[0175] The method further includes:

[0176] The connection relationship between the leaf switch and the backbone switch is obtained through a network management protocol to obtain switch link information.

[0177] The switch link information is structured and processed to record link state changes.

[0178] The switch link information is sent to a data storage module.

[0179] The management scheduling module sets the execution time of the link collection task for the link collection module, for example, collecting network link data of leaf switches to spine switches at one o'clock every day, then adding a one o'clock daily timing task,

[0180] The link collection module triggers from each server at a timing to collect network link data of leaf switches to spine switches. All device information of the type leaf is queried from the database, and all leaf switches are queried in a concurrent manner. For each switch, the local device name, the local port name, the remote device name connected to the local port, the remote port name connected to the local port, and the link information are constructed by associating the results of different queries according to the port index, and the link connected to the spine switch is filtered and screened.

[0181] Through the above two link collection processes, the link collection module performs structured processing on the collected link information, records device name, port name, link state, source device port, destination device port, link level, and other information, and compares the latest obtained information with historical data to update version data and realize tracking of link changes. The collected link information is reported to the data storage module.

[0182] Through the above embodiment of data collection, all-around indicators and link data are provided to realize real-time monitoring of high-speed networks, including server network card performance monitoring, switch running state monitoring, network link monitoring, support for high-speed network indicators, and accurate positioning to links, switches and network cards.

[0183] In step 108, the target data is obtained, and the target data is used for real-time monitoring of high-speed network conditions of a micro data center.

[0184] After the data collection module is executed, the data is processed inside the module, time stamps, versions and labels are marked, and then directly stored in the data storage module for real-time analysis of high-speed network conditions of a micro data center.

[0185] In some embodiments, the target data is obtained, and the target data is used for real-time feedback of high-speed network conditions of a micro data center, including:

[0186] The target data is obtained.

[0187] The device source and time stamp of the target data are marked, the label and version number are set, and the target data is stored in a distributed columnar database in a unified format for real-time monitoring of high-speed network conditions of a micro data center.

[0188] As an example, the target data includes device index data and link topology data; the device index data is used to monitor the status of devices in the cluster, and the link topology data is used to monitor the network link status.

[0189] The management scheduling module sends the target data collected by each data collection module to the data storage module for storage, so as to query device information, historical data, etc. from the data storage module for subsequent task configuration and abnormality analysis.

[0190] The data storage module is responsible for persistent storage and query support of collected data. In the case of limited resources in a micro data center, the data storage module uses a distributed columnar database to persistently store a large amount of time series index data, which is the data uploaded by the server collection module, the switch collection module and the link collection module. Optimize the table structure and partition strategy for different types of data, provide data query interface, and support efficient data writing and query.

[0191] The data storage module also implements data version management, which identifies the network topology state at different time points through version numbers, facilitating historical backtracking and change analysis.

[0192] Through the above embodiments, unified storage and analysis of time series data, topology relationship and data version management are realized. Based on these data, the upper application system can realize: abnormality detection: device fault detection, link abnormality alarm, performance problem early warning. Efficient operation and maintenance: unified configuration management, centralized network monitoring, historical data backtracking.

[0193] The present specification provides a high-speed network data collection method, device and computer readable storage medium. According to the device information of servers and switches in the cluster, data collection tasks including server data collection tasks, switch data collection tasks and network link collection tasks are determined, and the collection of all-around device index and link data is provided, which has high collection accuracy and can accurately locate problems at the link, switch and server network card levels under the compact device layout of a micro data center. According to the task type, a data collection module is matched for each data collection task; according to the collection period of the data collection task, the task type and collection parameters are specified to the data collection module at a time, so that the data collection module collects data to obtain target data, and the efficiency is improved through timed and concurrent collection to meet the real-time monitoring requirements.

[0194] Figure 4 An entity structure diagram of a high-speed network data collection device is exemplified as Figure 4As shown, the high-speed network data acquisition device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete mutual communication through the communications bus 840. The processor 810 can invoke the logic instructions in the memory 830 to execute the high-speed network data acquisition method.

[0195] In addition, the logic instructions in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0196] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, and the computer can execute the high-speed network data acquisition method provided by the above-mentioned methods.

[0197] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the high-speed network data acquisition method provided by the above-mentioned methods.

[0198] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0199] Other embodiments of the present description will be apparent to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. The description is intended to cover any alternatives, modifications, and equivalents of the described embodiments that are within the scope of the present description as defined by the appended claims. The specification and examples are intended to be exemplary only and not exhaustive, with the true scope and spirit of the description being indicated by the following claims.

[0200] It is to be understood that the present description is not limited to the precise details of apparatus and methods described herein and as illustrated in the drawings. Various modifications and changes in or from the exact construction and arrangements of parts of the present description as described herein and as illustrated in the drawings can be made by those skilled in the art without departing from the scope of the present description. The scope of the present description is indicated by the appended claims rather than by the foregoing description.

[0201] The above description is intended to be illustrative and not restrictive. Many other modifications within the scope of the described embodiments will become apparent to those skilled in the art upon reading the description. The full scope of the description is set forth in the following claims.

Claims

1. A high-speed network data acquisition method, characterized in that: Applied to a micro-sized data center cluster, the method includes: Determine data collection tasks, including server data collection tasks, switch data collection tasks, and network link data collection tasks, based on the device information of servers and switches in the cluster; Matching a data acquisition module to each of the data acquisition tasks according to the task type; Initiate an independent scheduled collection process for the data collection task according to the collection period of the data collection task, and regularly specify the task type and collection parameters to the data collection module so that the data collection module can collect data and obtain target data; wherein the collection period of the data collection task is adjusted according to the frequency of change and importance of the collected indicators; The target data is obtained, and the target data is used to monitor the high-speed network status of the micro data center in real time.

2. The high-speed network data acquisition method according to claim 1, wherein: According to the collection period of the data collection task, the task type and collection parameters are regularly specified to the data collection module so that the data collection module can collect data and obtain target data, including: Analyze the configuration of data collection tasks, including server data collection tasks and switch data collection tasks, and determine the collection period of the data collection tasks; The tasks with the same collection cycle are divided into a group, and an independent timed collection process is started for each group of tasks. The task type and collection parameters are regularly specified to the data collection modules corresponding to the server data collection tasks and the switch data collection tasks, so that the data collection modules can collect data and obtain target data.

3. The high-speed network data acquisition method according to claim 2, wherein: The method further comprises: Analyze the configuration of the network link collection task and determine the collection period of the link collection task; An independent timed collection process is started for the link collection task, and the task type and collection parameters are regularly specified to the data collection module corresponding to the link collection task.

4. The high-speed network data acquisition method according to claim 1, wherein: The acquisition parameters include the number of concurrent query operations; The method further comprises: Concurrency control is performed on the data acquisition tasks to determine the upper limit of the number of query operations that the data acquisition module can perform at the same time.

5. The high-speed network data acquisition method according to claim 4, wherein: The method further comprises: Dynamically adjust the number of concurrent query operations allowed to be executed simultaneously based on the device load status and historical collection success rate.

6. The high-speed network data acquisition method according to claim 4, wherein: The method further comprises: Monitoring the execution status of the data acquisition module in executing the data acquisition task; When an exception occurs, the corresponding processing strategy is executed according to the exception type.

7. The high-speed network data acquisition method according to claim 1, wherein: The acquiring of the target data, wherein the target data is used to provide real-time feedback on the high-speed network status of the micro data center, includes: acquiring the target data; The target data is marked with a device source and a timestamp, a label and a version number are set, and the target data is stored in a distributed columnar database in a unified format for real-time monitoring of the high-speed network status of the micro data center.

8. The high-speed network data collection method according to claim 7, wherein: The method further comprises: The target data includes device indicator data and link topology data; the device indicator data is used to monitor the status of devices in the cluster, and the link topology data is used to monitor the status of network links.

9. The high-speed network data collection method according to claim 1, wherein: A data acquisition module applied to a micro-data center cluster, wherein the cluster includes a server, and the acquisition parameters include an acquisition period; The server data collection task is performed by the data collection module, and the method includes: Based on the configured collection cycle, high-speed network device information under the system path is read to obtain server network card indicators; the system path is configured as a path for storing high-speed network device files; The data of the network card indicator is sent to the data storage module according to a preset format.

10. The high-speed network data collection method according to claim 9, wherein: The method further comprises: Obtain server network card metrics through non-invasive reading.

11. The high-speed network data collection method according to claim 1, wherein: A data acquisition module is applied to a micro-data center cluster, wherein the cluster includes a switch; and the data acquisition module is used to perform the switch data acquisition task. The method includes: Obtaining switch configuration information corresponding to the switch data collection task; Establishing a connection with the switch through a network management protocol according to the configuration information of the switch; Collecting the operating indicators of the switch based on the configured collection period; The data of the switch's operating indicators are sent to the data storage module in a preset format.

12. The high-speed network data collection method according to claim 1, wherein: A data acquisition module applied to a micro-data center cluster, wherein the cluster includes servers and switches, and the switches include leaf switches and backbone switches; The link collection task is performed by the data collection module, and the method further includes: Acquire a connection relationship between the server and the leaf switch port from the server side based on a link discovery protocol to obtain server link information; Performing structured processing on the server link information and recording link status changes; The server link information is sent to the data storage module.

13. The high-speed network data collection method according to claim 12, wherein: The link collection task is performed by the data collection module, and the method further includes: Obtaining the connection relationship between the leaf switch and the backbone switch through a network management protocol to obtain switch link information; Performing structured processing on the switch link information and recording link status changes; The switch link information is sent to the data storage module.

14. The high-speed network data collection method according to any one of claims 9 to 13, wherein: The data collection tasks including server data collection tasks, switch data collection tasks, and network link collection tasks are executed in one of the data collection modules or in independent data collection modules.

15. A computer device, characterized in that: The method comprises a memory, a processor and a data acquisition program stored in the memory and executable on the processor. When the processor executes the data acquisition program, the steps of the high-speed network data acquisition method according to any one of claims 1 to 14 are implemented.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a data acquisition program, which, when executed, implements the steps of the high-speed network data acquisition method according to any one of claims 1 to 14.

Images