Operation and maintenance index data acquisition and analysis method and system, electronic equipment and storage medium

Abstract

The invention provides an operation and maintenance index data acquisition and analysis method and system, electronic equipment and a storage medium. The operation and maintenance index data acquisition and analysis method for geophysical high-performance calculation comprises the following steps: acquiring operation and maintenance index data based on a high-performance TSDB time sequence database and an export device; processing the collected operation and maintenance index data; and carrying out data analysis and visual display on the processed operation and maintenance index data.

Description

Technical Field

[0001] The present invention belongs to the fields of information technology and petroleum geophysical exploration, and specifically relates to methods for collecting, processing, and displaying operation and maintenance index data closely related to geophysical high-performance computing, and developing an operation and maintenance monitoring system with geophysical industry characteristics based on a mainstream operation and maintenance development framework. Background Art

[0003] Conventional scripted data collection is a way to automatically collect data. It writes program code to simulate human operations, uses web crawler technology to automatically capture data from the Internet, and stores it in a specified database. Compared with traditional manual data collection methods, this method has advantages such as high efficiency, accuracy, strong customizability, and low cost. When using the scripted data collection method, it is necessary to formulate a collection plan, clarify the types of data to be collected and the source websites, and formulate the collection frequency and rules. Then write the corresponding program code according to the collection plan and conduct tests. When running the script program, it is necessary to set the data storage location, monitor the data capture situation in real time, and handle abnormal situations in a timely manner. Finally, the collected data is cleaned, sorted, and analyzed and used for subsequent business decisions.

[0004] However, conventional scripted data collection is not suitable for collecting and processing operation and maintenance index data closely related to geophysical high-performance computing. How to mine the existing operation and maintenance indexes to form valuable data to assist operation and maintenance production has become an urgent problem to be solved. Summary of the Invention

[0005] The present invention designs and develops a brand-new operation and maintenance data collection, monitoring, and display algorithm and system for geophysical high-performance computing scenarios, and solves difficult problems such as insufficient accuracy, insufficient richness of dimensions, low retrieval performance, low secondary development level, and relatively single display means of previous operation and maintenance data from the collection layer, data processing layer, and interactive display layer. The results of the present invention have good uniqueness and innovation.

[0006] To achieve the above object, the present invention provides an operation and maintenance index data collection and analysis method for geophysical high-performance computing, including:

[0007] Collecting operation and maintenance index data based on a high-performance TSDB time series database and an exporter;

[0008] Processing the collected operation and maintenance index data;

[0009] Performing data analysis and visual display on the processed operation and maintenance index data.

[0010] Furthermore, the collected operation and maintenance index data includes basic hardware and site data, operating system data, multi-type device data, and business data.

[0011] Furthermore, multiple dynamic query aggregation statements and custom special function functions for high-performance data at different levels are developed based on the Promql language, and the collected operation and maintenance metric data is processed by combining data feature tagging processing.

[0012] Furthermore, the visual display includes the display of the resource utilization status of the main cluster nodes, the display of the traffic of the computer room network topology, the display of the core switch data, the display of the machine time usage in terms of projects, the display of the scheduling system queue and job running status, the display of the running status of the main commercial software, and the display of the monitoring of the changes in the hardware status.

[0013] Furthermore, the basic hardware data includes: the online status of the server nodes, the parameter status and serial numbers of the installed hardware; the site data includes temperature and humidity, power consumption, branch power load, and 3D visualization modeling data.

[0014] Furthermore, the operating system data includes CPU utilization, GPU utilization, memory utilization, system package and driver version type, disk read and write and load, and process status.

[0015] Furthermore, the multi-type device data includes the network data of the main links and the core and aggregation layer switches in the production network, interface traffic, topology self-discovery data based on the lldp protocol, and the main high-performance storage IO and quota usage data; the business data includes the detailed status data of the queue, jobs, users, work areas, and software types of the scheduling system for the main commercial software in geophysics in multiple dimensions.

[0016] According to another aspect of the present invention, there is provided an operation and maintenance metric data acquisition and analysis system for geophysical high-performance computing, including:

[0017] An acquisition layer, which acquires operation and maintenance metric data based on a high-performance TSDB time series database and an exporter;

[0018] A data processing layer, which processes the acquired operation and maintenance metric data;

[0019] An interactive display layer, which performs data analysis and visual display on the processed operation and maintenance metric data.

[0020] According to another aspect of the present invention, there is provided an electronic device, which includes:

[0021] A memory, which stores executable instructions;

[0022] A processor, which runs the executable instructions in the memory to implement the operation and maintenance metric data acquisition and analysis method for geophysical high-performance computing as described above.

[0023] According to another aspect of the present invention, there is provided a non-transitory computer-readable storage medium having stored thereon a computer program, which when executed by a processor, implements the operation and maintenance index data acquisition and analysis method for geophysical high-performance computing described above.

[0024] The achievements of this patent have currently completed the acquisition, processing, monitoring and display of the operating states of the main business software, computing clusters, network devices, and storage devices in the geophysical exploration institute's cloud computing center, deployed and demonstrated the applications in the monitoring hall and the public office areas of the processing and interpretation center, and achieved good application effects. At the same time, this patent achievement realizes the refined machine-hour calculation at the project dimension, and the relevant technologies and data have been provided to the finance department as the basis for the cost accounting data of some projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] By describing the exemplary embodiments of the present invention in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present invention will become more apparent. Among them, in the exemplary embodiments of the present invention, the same reference numerals generally represent the same components.

[0026] Figure 1 It is a flowchart of the operation and maintenance index data acquisition and analysis method for geophysical high-performance computing according to the present invention.

[0027] Figure 2 It is a monitoring diagram of the GPU cluster service and GPU status according to an embodiment of the present invention.

[0028] Figure 3 It is a monitoring diagram of the CPU cluster and running services according to an embodiment of the present invention.

[0029] Figure 4 It is a visualization diagram of the link load monitoring of the production network core and aggregation layers according to an embodiment of the present invention.

[0030] Figure 5 It is a visualization diagram of the network traffic data of the core switch logical ports according to an embodiment of the present invention.

[0031] Figure 6 It is a visualization diagram of the detailed index data of the cluster nodes according to an embodiment of the present invention.

[0032] Figure 7 It is the queuing situation of the job runs in the scheduling system according to an embodiment of the present invention.

[0033] Figure 8 It is the queue situation of the scheduling system according to an embodiment of the present invention.

[0034] Figure 9 It is the total machine-hours at the project dimension and the machine-hours of the main processing processes according to an embodiment of the present invention.

[0035] Figure 10 Scrolling monitoring and display of detailed operation information according to an embodiment of the present invention. Specific embodiments

[0036] The preferred embodiments of the present invention will be described in more detail below. Although the preferred embodiments of the present invention are described below, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0037] The present invention mainly provides a variety of methods for collecting, processing, and displaying operation and maintenance data for geophysical high-performance computing, and develops a highly available operation and maintenance monitoring visualization system based on these methods. Demand research and development are carried out in many aspects, from multi-type operation and maintenance data collection, tagging processing, aggregation retrieval operations and function operations, operation and maintenance data persistent storage to final visualization design. Based on a self-customized high-performance exporter, concurrent collection of geophysical software and hardware operation and maintenance data information is realized. The operation and maintenance data are processed with multi-dimensional tags according to the actual software and hardware conditions. A simple high-performance retrieval function is used to replace the conventional SQL statement for retrieval query, and display design and development are carried out from multiple dimensions of projects, users, computing resources, network resources, and storage resources.

[0038] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but it is not a limitation of the present invention. It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0039] Embodiment 1

[0040] As Figure 1 shown, this embodiment provides a method for collecting and analyzing operation and maintenance metric data for geophysical high-performance computing, including:

[0041] Collecting operation and maintenance metric data based on a high-performance TSDB time series database and an exporter;

[0042] Processing the collected operation and maintenance metric data;

[0043] Performing data analysis and visualization display on the processed operation and maintenance metric data.

[0044] Furthermore, the collected operation and maintenance metric data includes basic hardware and site data, operating system data, multi-type device data, and business data.

[0045] Furthermore, a variety of dynamic query aggregation statements and custom special function functions for high-performance data at different levels are developed based on the Promql language, and the collected operation and maintenance metric data is processed by combining data feature tagging.

[0046] Furthermore, the visual display includes the display of the resource utilization status of the main cluster nodes, the display of the traffic of the computer room network topology, the display of the core switch data, the display of the machine time usage in terms of projects, the display of the scheduling system queue and job running status, the display of the running status of the main commercial software, and the display of the monitoring of hardware status changes.

[0047] Furthermore, the basic hardware data includes: the online status of the server nodes, the parameter status and serial numbers of the installed hardware; the site data includes temperature and humidity, power consumption, branch power load, and 3D visualization modeling data.

[0048] Furthermore, the operating system data includes CPU utilization, GPU utilization, memory utilization, system package and driver version type, disk read and write and load, and process status.

[0049] Furthermore, the multi-type device data includes the relevant network data of the main links, cores, and aggregation layer switches in the production network, interface traffic, topology self-discovery data based on the lldp protocol, and the main high-performance storage IO and quota usage data; the business data includes the detailed status data of queues, jobs, users, work areas, and software types in the scheduling system of the main commercial software for geophysics in multiple dimensions.

[0050] Embodiment 2

[0051] Refer to Figures 2 - 10 As shown, this embodiment provides an operation and maintenance metric data acquisition and analysis system for geophysical high-performance computing, including:

[0052] An acquisition layer, which acquires operation and maintenance metric data based on a high-performance TSDB time series database and an exporter;

[0053] A data processing layer, which processes the acquired operation and maintenance metric data;

[0054] An interactive display layer, which performs data analysis and visual display on the processed operation and maintenance metric data.

[0055] Based on the original partial visualization display, this embodiment actively adopts the popular Prometheus+Grafana framework and WebGL modeling and rendering technology in the field of operation and maintenance development. It re-upgrades the technical architecture of the operation and maintenance monitoring visualization system, and conducts demand-driven development work in multiple aspects, including multi-type operation and maintenance data collection, tagging processing, aggregation retrieval operations and function calculations, operation and maintenance data persistent storage, and finally visualization design. Starting from the actual geophysical high-performance computing business, it completes the implementation of relevant technologies, achieving a comprehensive improvement in data acquisition, data structure, data source consistency, and interactive visualization of the high-performance operation and maintenance monitoring visualization system. The achievements of the entire project are mainly divided into the following functional layers:

[0056] 1. Collection layer: Based on the high-performance TSDB time series database and exporters, rich operation and maintenance metric data is collected and stored in the project results, which is mainly divided into the following four parts:

[0057] (1) Basic hardware and site data, mainly monitoring high-value clusters, the online status of server nodes, the parameter status of hardware such as installed memory, graphics cards, and hard disks, serial numbers, etc.; site data includes temperature and humidity, power consumption, branch power load, 3D visualization modeling data, etc.;

[0058] (2) Operating system data, including CPU utilization, GPU utilization, memory utilization, system package and driver version types, disk read and write and load, process status, etc.;

[0059] (3) Multi-type device data, different data sources may have different consistency and reliability requirements. Collect network data related to the main links, core and aggregation layer switches in the production network, interface traffic, topology self-discovery data based on the lldp protocol, main high-performance storage IO and quota usage data, etc.;

[0060] (4) Business data, detailed status data of queues, jobs, users, work areas, software types in multiple dimensions for the main commercial software scheduling system in geophysics. An independently developed Prometheus exporter is used for real-time and efficient collection, and data interfaces such as available machine hours, project usage, and alarm push are provided as needed.

[0061] In particular, the exporter is a collector developed specifically for geophysical high-performance computing tasks, with strong innovation in the industry. It can collect and analyze in detail geophysical high-performance data such as the operation status of geophysical business jobs, machine hours, and computing modules.

[0062] The following is the code of the collector for some business software:

[0063] def jobinfo_gather():

[0064] browser = webdriver.Chrome()

[0065] wait = WebDriverWait(browser, 10)

[0066] browser.get("https: / / 192.168.79.27:8181 / ocm / JobList")

[0067] #browser.maximize_window()

[0068] time.sleep(10)

[0069] rc = browser.find_element_by_id('completed_count')

[0070] rc.click()

[0071] time.sleep(10)

[0072] #rc1 = browser.find_element_by_id('yui-dt0-th-tag-liner')

[0073] #rc1.click()

[0074] #time.sleep(10)

[0075] rc1 = browser.find_element_by_id('yui-dt0-th-end_date')

[0076] rc1.click()

[0077] time.sleep(10)

[0078] rc1.click()

[0079] time.sleep(10)

[0080] html = browser.page_source

[0081] soup = BeautifulSoup(html, "html.parser")

[0082] for num in range(3000, 3800):

[0083] row = 'yui-rec' + str(num)

[0084] soup_tr = soup.find("tr", {'id': row})

[0085] soup_tr_tag = soup_tr.find("td", {'headers': 'yui-dt0-th-tag'})

[0086] soup_tr_jobname = soup_tr.find("td", {'headers': 'yui-dt0-th-name'})

[0087] soup_tr_status = soup_tr.find("div", {'class':'status-content'})

[0088] soup_tr_type = soup_tr.find("td", {"headers": 'yui-dt0-th-type'})

[0089] soup_tr_temp = soup_tr.find("td", {"headers": 'yui-dt0-th-template_job_id'})

[0090] soup_tr_node = soup_tr.find("td", {"headers": 'yui-dt0-th-server_name'})

[0091] soup_tr_proj = soup_tr.find("td", {"headers": 'yui-dt0-th-project_name'})

[0092] soup_tr_user = soup_tr.find("td", {"headers": 'yui-dt0-th-user_name'})

[0093] soup_tr_queue = soup_tr.find("td", {"headers": 'yui-dt0-th-collection_date'})

[0094] soup_tr_start = soup_tr.find("td", {"headers": 'yui-dt0-th-start_date'})

[0095] soup_tr_end = soup_tr.find("td", {"headers": 'yui-dt0-th-end_date'})

[0096] soup_tr_elapsed = soup_tr.find("td", {"headers": 'yui-dt0-th-elapsed_time'})

[0097] soup_tr_apps = soup_tr.find("td", {"headers": 'yui-dt0-th-application_name'})

[0098] tag_text = soup_tr_tag.text

[0099] jobname_text = soup_tr_jobname.text

[0100] status_text = soup_tr_status.text

[0101] type_text = soup_tr_type.text

[0102] temp_text = soup_tr_temp.text

[0103] node_text = soup_tr_node.text

[0104] proj_text = soup_tr_proj.text

[0105] user_text = soup_tr_user.text

[0106] queue_text = soup_tr_queue.text

[0107] start_text = soup_tr_start.text

[0108] end_text = soup_tr_end.text

[0109] elapsed_text = soup_tr_elapsed.text

[0110] apps_text = soup_tr_apps.text

[0111] The labeled processing of geophysical high-performance operation and maintenance index data can easily retrieve and display business data quickly by developing custom functions, with relatively high innovation. For example, it can quickly retrieve the number of nodes occupied by different types of business software running in a set of high-performance computing clusters, as well as valid data such as detailed usage status.

[0112] Specifically, the data storage of time-series databases has specific methods and characteristics to adapt to the characteristics of time-series data:

[0113] Data hierarchical storage: Time-series data is usually generated in chronological order and has obvious characteristics of hot and cold data. To improve storage efficiency, time-series databases usually adopt a data hierarchical storage strategy, storing hot data, warm data, and cold data in different media, such as memory, SSD, and SATA / SAS hard disks.

[0114] Data compression: Time-series data usually has the characteristics of continuous time and repeated values. Therefore, effective data compression technologies can be used to reduce the storage space occupancy. For example, timestamps can be used for data aggregation to remove duplicate data, or compression algorithms such as dictionary encoding and run-length encoding can be used to reduce the data size.

[0115] Index design: Time-series databases need to support efficient data query and analysis operations, so appropriate indexes need to be designed to accelerate the query process. For the characteristics of time-series data, methods such as inverted indexes, timestamp indexes, or hash indexes can be adopted.

[0116] Data partitioning: To improve query performance and concurrency, time-series databases usually store data partitions on different nodes or hard disks. The partitioning strategy can be determined according to actual needs and data characteristics, such as partitioning by time range, partitioning by data source, etc.

[0117] Data replication: To ensure data security and reliability, time-series databases usually adopt data replication technologies to replicate data to multiple nodes or hard disks. This can achieve data fault tolerance and load balancing, improving the availability and performance of the system.

[0118] Storage engine selection: The storage engine of time-series databases usually selects columnar storage, such as HBase, etc. Columnar storage can better support aggregation analysis and multidimensional query operations, and at the same time can improve query efficiency.

[0119] Specifically, in the data processing layer, based on the Promql language, a variety of dynamic query aggregation statements and custom special function functions for high-performance data at different levels are developed. Combined with the data feature tagging process, it realizes good storage organization and scalable development capabilities for similar data structures.

[0120] Specifically, in the interactive display layer, based on front-end technologies and open-source frameworks such as Grafana and WebGL, a set of easy-to-use, easy-to-manage, easy-to-expand data analysis and visualization display solutions for various different types have been formed. Currently, the display of resource utilization status of main cluster nodes, the display of computer room network topology traffic, the display of core switch data, the display of machine time usage in terms of projects, the display of the scheduling system queue and job running status, the display of the running status of main commercial software, the display of hardware status change monitoring, etc. have been developed. At the same time, targeted demand display development has been carried out for different users.

[0121] The achievements of this patent apply relatively new technologies in the field of operation and maintenance development, and innovatively integrate with the main business scenarios of the Geophysical Exploration Institute (geophysical high-performance computing, seismic data processing). The monitoring and visualization level of high-performance index data represents the service support ability of the cloud computing center for the entire geophysical exploration business in some aspects. Objective and reliable data display can also provide strong support for the center's development decision-making, visit display and publicity, and also provide a technical foundation for realizing automated and even intelligent operation and maintenance.

[0122] Embodiment 3

[0123] This embodiment provides an electronic device, which includes:

[0124] A memory storing executable instructions;

[0125] A processor that runs the executable instructions in the memory to implement the method for collecting and analyzing operation and maintenance index data for geophysical high-performance computing, and the method includes:

[0126] Collect operation and maintenance index data based on a high-performance TSDB time series database and an exporter;

[0127] Process the collected operation and maintenance index data;

[0128] Perform data analysis and visualization display on the processed operation and maintenance index data.

[0129] Embodiment 4

[0130] This embodiment provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the operation and maintenance index data acquisition and analysis method for geophysical high-performance computing, and the method includes:

[0131] Collect operation and maintenance index data based on a high-performance TSDB time series database and an exporter;

[0132] Process the collected operation and maintenance index data;

[0133] Perform data analysis and visual display on the processed operation and maintenance index data.

[0134] The above computer-readable storage medium includes but is not limited to: optical storage media (such as CD-ROM and DVD), magneto-optical storage media (such as MO), magnetic storage media (such as magnetic tapes or external hard drives), media with built-in rewritable non-volatile memories (such as memory cards), and media with built-in ROMs (such as ROM cartridges).

[0135] In summary, the achievements of the present invention apply relatively new technologies in the field of operation and maintenance development, innovatively integrate with geophysical high-performance computing and seismic data processing. The monitoring and visualization level of high-performance index data represents the service support ability of the cloud computing center for the entire geophysical exploration business in some aspects. Objective and reliable data display can also provide strong support for the center's development decision-making, visit display and publicity, and also provide a technical foundation for realizing automated and even intelligent operation and maintenance.

[0136] The embodiments of the present invention have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments.

Claims

1. A method for collecting and analyzing operation and maintenance metric data for geophysical high-performance computing, characterized in that, it includes: Collecting operation and maintenance metric data based on a high-performance TSDB time series database and exporter; Processing the collected operation and maintenance metric data; Performing data analysis and visual display on the processed operation and maintenance metric data.

2. The method for collecting and analyzing operation and maintenance metric data for geophysical high-performance computing according to claim 1, characterized in that, The collected operation and maintenance metric data includes basic hardware and site data, operating system data, multi-type device data, and business data.

3. The method for collecting and analyzing operation and maintenance metric data for geophysical high-performance computing according to claim 1, characterized in that, Developing various dynamic query aggregation statements and custom special function functions for high-performance data at different levels based on the Promql language, and processing the collected operation and maintenance metric data by combining data feature tagging.

4. The method for collecting and analyzing operation and maintenance metric data for geophysical high-performance computing according to claim 1, characterized in that, The visual display includes the display of the resource utilization status of the main cluster nodes, the display of the traffic of the computer room network topology, the display of the core switch data, the display of the machine time usage in the project dimension, the display of the scheduling system queue and job running status, the display of the running status of the main commercial software, and the display of the monitoring of the hardware status changes.

5. The method for collecting and analyzing operation and maintenance metric data for geophysical high-performance computing according to claim 2, characterized in that, The basic hardware data includes: the online status of the server node number, the parameter status and serial number of the installed hardware; the site data includes temperature and humidity, power consumption, branch power load, and 3D visualization modeling data.

6. The method for collecting and analyzing operation and maintenance metric data for geophysical high-performance computing according to claim 2, characterized in that, The operating system data includes CPU utilization rate, GPU utilization rate, memory utilization rate, system package and driver version type, disk read and write and load, and process status.

7. The method for collecting and analyzing operation and maintenance metric data for geophysical high-performance computing according to claim 2, characterized in that, The multi-type device data includes the relevant network data of the main links, core and aggregation layer switches in the production network, interface traffic, topology self-discovery data based on the lldp protocol, and the main high-performance storage IO and quota usage data; the business data includes the detailed status data of the queue, job, user, work area, and software type of the main commercial software scheduling system for geophysics in multiple dimensions.

8. An operation and maintenance metric data collection and analysis system for geophysical high-performance computing, characterized in that, it includes: A collection layer that collects operation and maintenance metric data based on a high-performance TSDB time series database and exporter; A data processing layer that processes the collected operation and maintenance metric data; An interactive display layer that performs data analysis and visual display on the processed operation and maintenance metric data.

9. An electronic device, characterized in that, the electronic device includes: A memory that stores executable instructions; A processor that runs the executable instructions in the memory to implement the operation and maintenance metric data acquisition and analysis method for geophysical high-performance computing according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, wherein, when the computer program is executed by a processor, it implements the operation and maintenance metric data acquisition and analysis method for geophysical high-performance computing according to any one of claims 1-7.

Images