Large data volume monitoring data analysis method and system

Abstract

The invention provides a large-data-volume monitoring data analysis method and system, and the method comprises the steps: obtaining data parameters, submitting a data parameter analysis request, and transmitting the analysis request to an application programming interface server of a container arrangement platform; according to the resource condition and the scheduling strategy of each node in the container arrangement platform, a data analysis task is selected to be created on a proper node in a containerization mode, the container arrangement platform considers the CPU of the node so as to ensure the utilization of resources, and different analysis requests are distributed to different nodes to be executed; starting a container corresponding to the data analysis task, running a data analysis code in the container to obtain monitoring data in a column database corresponding to the data analysis task, and performing traversal analysis on the monitoring data; and a result obtained after traversal analysis of the monitoring data is stored, and the data analysis task is deleted, so that the real-time performance and the accuracy of data analysis are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a method and system for analyzing large amount of data monitoring. Background Art

[0002] In the field of large-scale data monitoring and data analysis, existing technologies mainly rely on two types of systems: traditional relational database management systems (RDBMS) and big data processing tools based on Hadoop, Spark, etc. It provides transaction processing capabilities and data integrity guarantees, and realizes the storage and calculation of big data through the distributed file system HDFS and MapReduce programming model.

[0003] Traditional RDBMS is known for its powerful transaction processing capabilities and data integrity assurance, but its performance often becomes a bottleneck when faced with large data volume analysis. Since RDBMS usually uses row-based storage, the storage and reading efficiency of data on disk is low, especially when performing complex data analysis queries, the performance degradation is particularly obvious. In addition, the scalability of RDBMS is relatively limited, making it difficult to meet the high concurrency and high throughput requirements in large data volume scenarios.

[0004] Although Hadoop, Spark and other big data processing tools can process massive amounts of data, they are not very good at real-time performance. These tools usually require batch processing, and the data processing delay is high, which cannot meet the real-time requirements of real-time monitoring data analysis. At the same time, the threshold for using these tools is relatively high, requiring users to have certain programming and data processing capabilities. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a large-volume monitoring data analysis method and system for solving the technical problems in the prior art.

[0006] On the one hand, the invention provides the following technical solution, a method for analyzing large amount of data monitoring, the method comprising:

[0007] Obtaining data parameters, submitting an analysis request for the data parameters, and sending the analysis request to an application programming interface server of the container orchestration platform;

[0008] According to the resource conditions and scheduling strategies of each node in the container orchestration platform, select a suitable node to create a containerized data analysis task. The container orchestration platform will consider the remaining CPU, memory, and storage resources of the node to ensure resource utilization. Different analysis requests will be assigned to different nodes for execution.

[0009] Starting a container corresponding to the data analysis task, running a data analysis code in the container to obtain monitoring data in a column-based database corresponding to the data analysis task and performing traversal analysis on the monitoring data;

[0010] The results of the traversal and analysis of the monitoring data are stored, and the data analysis task is deleted, and the resources occupied by it are released.

[0011] Compared with the prior art, the beneficial effects of the present application are as follows: by starting the container corresponding to the data analysis task and running the data analysis code in the container, the parallel execution and dynamic resource management of the data analysis task are realized, the processing efficiency and resource utilization are improved, and the problems of multi-analysis task blocking and resource waste are effectively solved, the real-time and accuracy of data analysis are improved, and the operation and maintenance costs of the system are reduced. At the same time, due to the use of containerization technology, the system has good scalability and portability, and can easily cope with data analysis needs of different scales and complexities. By and sending the analysis request to the application programming interface server of the container orchestration platform, the parallel execution of data analysis tasks is realized, which greatly improves the processing efficiency.

[0012] Furthermore, the step of starting the container corresponding to the data analysis task includes:

[0013] Creating a mirror database, and pulling the corresponding data analysis mirror from the mirror database according to the data analysis task;

[0014] A corresponding container is started based on the data analysis image, wherein the image database includes a client and other data analysis tools.

[0015] Furthermore, the step of obtaining the monitoring data of the data analysis task includes:

[0016] A monitoring node is created to collect and analyze the status and progress of the node when the node runs the data analysis task, and form monitoring data.

[0017] Furthermore, before the step of obtaining the data parameters, the method further includes:

[0018] A Kubernetes cluster, a columnar database, and a PostgreSQL database are created, wherein the Kubernetes cluster consists of multiple nodes, an application programming interface server is running on the nodes, the ClickHouse database is used to store and process large amounts of monitoring data; and the PostgreSQL database is deployed separately on a node in the cluster to store the results of data analysis.

[0019] Furthermore, the step of obtaining data parameters includes:

[0020] A client interface is constructed, and the user selects the data parameters to be analyzed, wherein the data parameters include time range, data type, and analysis algorithm.

[0021] Furthermore, the data analysis task includes a task name, an execution command, resource requirements, and request parameters.

[0022] Furthermore, after the step of selecting to containerize and create a data analysis task on a suitable node, the method further includes:

[0023] Determine whether the node encounters a calculation error or a node failure. If the node encounters a calculation error or a node failure, clean up the node and count the number of cleanups, and re-execute the step of creating a data analysis task request.

[0024] In a second aspect, the invention provides the following technical solution: a large data volume monitoring data analysis system, the system comprising:

[0025] A sending module, used to obtain data parameters, submit an analysis request for the data parameters, and send the analysis request to an application programming interface server of the container orchestration platform;

[0026] A creation module is used to select a containerized creation data analysis task on a suitable node according to the resource status and scheduling strategy of each node in the container orchestration platform, wherein the container orchestration platform will consider the remaining CPU, memory, and storage resources of the node to ensure resource utilization, and different analysis requests will be assigned to different nodes for execution;

[0027] An analysis module is used to start a container corresponding to the data analysis task, and run a data analysis code in the container to obtain monitoring data in a column-based database corresponding to the data analysis task and perform traversal analysis on the monitoring data;

[0028] The deletion module is used to store the results of traversal and analysis of the monitoring data, delete the data analysis task, and release the resources occupied by it.

[0029] In a third aspect, the invention provides the following technical solution: a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the large data volume monitoring data analysis method as described above when executing the computer program.

[0030] In a fourth aspect, the invention provides the following technical solution: a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the large data volume monitoring data analysis method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A flowchart of a large amount of data monitoring and analysis method provided by the first embodiment of the present invention;

[0033] Figure 2 A flowchart of a large data volume monitoring data analysis method provided by the first embodiment of the present invention;

[0034] Figure 3 A structural block diagram of a large data volume monitoring data analysis system provided by a second embodiment of the present invention;

[0035] Figure 4 A schematic diagram of the hardware structure of a computer provided for the third embodiment of the present invention.

[0036] The embodiments of the present invention will be further described below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0037] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout are the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0038] In the description of the embodiments of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0039] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0040] Embodiment 1

[0041] In the first embodiment of the present invention, see Figure 1 to Figure 2 , a large amount of data monitoring data analysis method, comprising the following steps S01 to S03:

[0042] S01, obtaining data parameters, submitting an analysis request for the data parameters, and sending the analysis request to an application programming interface server of a container orchestration platform;

[0043] Specifically, before the step of obtaining the data parameters, the method further includes:

[0044] Create a Kubernetes cluster, a columnar database (ClickHouse database) and a PostgreSQL database, wherein the Kubernetes cluster consists of multiple nodes, an application programming interface server runs on the nodes, the ClickHouse database is used to store and process large amounts of monitoring data; the PostgreSQL database is deployed separately on a node in the cluster to store the results of data analysis.

[0045] Specifically, the step of obtaining data parameters includes:

[0046] A client interface is constructed, and the user selects the data parameters to be analyzed, wherein the data parameters include time range, data type, and analysis algorithm.

[0047] Specifically, the data analysis task includes a task name, an execution command, resource requirements, and request parameters.

[0048] In this embodiment, the client interface provides a data parameter selection function, and the user can select the data parameters to be analyzed according to the needs. After the selection is completed, the client sends the request parameters to the background service program to call the k8s apiserver service (application programming interface server) of Kubernetes (container orchestration platform) to initiate a job (data analysis task) creation request.

[0049] In this embodiment, a client interface (web) is constructed, and users can select the data parameters to be analyzed and submit analysis requests. These parameters can include time range, data type, analysis algorithm, etc. The client encapsulates the user's request into an HTTP request and sends it to the k8sapiserver service of Kubernetes. The user interface is user-friendly and supports multiple parameter configurations to flexibly customize data analysis tasks.

[0050] In this embodiment, first, we need to build an environment that includes a Kubernetes cluster, a columnar database, and a PostgreSQL database. The Kubernetes cluster consists of multiple nodes, including a master node (Master Node) and a worker node (Worker Node). The k8sapiserver service runs on the master node, which is used to receive client requests and manage the entire cluster. The worker node is responsible for running the actual workload. The ClickHouse database is used to store and process large amounts of monitoring data. Its efficient columnar storage and query optimization features can significantly increase the speed of data analysis. The PostgreSQL database is deployed separately on a node in the cluster to store the results of data analysis.

[0051] S02, according to the resource conditions and scheduling strategies of each node in the container orchestration platform, select a suitable node to create a containerized data analysis task, wherein the container orchestration platform will consider the remaining CPU, memory, and storage resources of the node to ensure the best utilization of resources, and different analysis requests will be assigned to different nodes for execution;

[0052] In this embodiment, after receiving the client's request, k8sapiserver will create a new job (data analysis task). This job contains all the necessary information of the data analysis task, such as task name, execution command, resource requirements, request parameters, etc. Then, the Kubernetes scheduler will select a suitable node to run this job based on the resource conditions and scheduling policies of each node in the cluster. When selecting a node, the scheduler will consider the remaining resources of the node, such as CPU, memory, storage, etc., to ensure the best utilization of resources.

[0053] In this embodiment, the background service program first analyzes the data parameters and estimates the size of the computing data and the required computing power. In order to efficiently utilize computing resources, the service program will split the computing task into multiple subtasks according to a specific algorithm, and these subtasks can be executed in parallel to achieve distributed computing. Next, the background service program will call the Kubernetes API to create a job through the k8s apiserver. When creating a job, the service program will specify the required resource requests (Requests) and limits (Limits), which are usually determined based on previous computing power estimates, so as to select the appropriate computing node.

[0054] S03, starting a container corresponding to the data analysis task, running a data analysis code in the container to obtain monitoring data in a column-based database corresponding to the data analysis task and performing traversal analysis on the monitoring data;

[0055] Specifically, the step of starting the container corresponding to the data analysis task includes:

[0056] Creating a mirror database, and pulling the corresponding data analysis mirror from the mirror database according to the data analysis task;

[0057] A corresponding container is started based on the data analysis image, wherein the image database includes a client and other data analysis tools.

[0058] Specifically, the step of obtaining the monitoring data of the data analysis task includes:

[0059] A monitoring node is created to collect and analyze the status and progress of the node when the node runs the data analysis task, and form monitoring data.

[0060] Specifically, after the step of selecting to containerize and create a data analysis task on a suitable node, the method further includes:

[0061] Determine whether the node encounters a calculation error or a node failure. If the node encounters a calculation error or a node failure, clean up the node and count the number of cleanups, and re-execute the step of creating a data analysis task request.

[0062] In this embodiment, once a job is scheduled to a node, the job will start the corresponding container and run the data analysis code in the container. In this process, the data analysis code will obtain the monitoring data that meets the requirements from the ClickHouse database, use ClickHouse's efficient query capabilities to quickly retrieve and process the data, and perform corresponding data analysis. After the analysis results are processed, they are ready to be stored in the result database.

[0063] In this embodiment, the corresponding data analysis image is pulled from the pre-prepared image warehouse according to the computing task. For multi-node distributed computing tasks, in order to effectively monitor the execution of distributed computing tasks, adjacent sub-nodes will be given priority, and the service program will set up one or more monitoring nodes, which are responsible for collecting and analyzing the status and progress of each computing node, and coordinating and communicating when necessary. After completing its own computing task, each sub-node sends the result to the monitoring node. The monitoring node summarizes and verifies the received results to ensure the correctness and completeness of the calculation. In this way, computing time can be effectively saved.

[0064] If a task fails due to a computational error or node failure, the backend service will first clean up the compute nodes and redeploy the task for a limited number of retries. When the maximum number of retries is reached, the error log will be recorded in detail for analysis by the developer.

[0065] After the job is created, it starts to query the monitoring data system for monitoring data that meets the requirements according to the data parameters. After obtaining the data, the job traverses and analyzes the data in the container, and sends the analysis request to the ClickHouse database for data query and analysis. With its high-performance column storage and real-time analysis capabilities, ClickHouse can quickly process large amounts of query requests and significantly improve the efficiency of data analysis.

[0066] S04, storing the results of the traversal analysis of the monitoring data, deleting the data analysis task, and releasing the resources occupied by it.

[0067] In this embodiment, after the analysis is completed, the job or monitoring job will summarize and count the results, and after confirmation, the results will be stored in the Postgresql database. After the storage is completed, the job will be automatically destroyed and the occupied resources will be released. The Kubernetes cluster management system will monitor this process to ensure timely recycling and reuse of resources.

[0068] In this example, when the data analysis task is completed, the Job will store the result file in the PostgreSQL database. The fast storage capability of ClickHouse analysis results ensures that the data can be quickly persisted. After the upload is completed, Kubernetes will automatically destroy the Job and release the resources it occupies, ensuring efficient utilization and dynamic management of cluster resources. This process is automated and does not require manual intervention, ensuring the ease of use and reliability of the system.

[0069] Experimental data:

[0070] In order to verify the effectiveness of this implementation, we conducted a set of comparative experiments. In the experiments, we used the same data set and hardware configuration, and used this implementation and the existing technology to process the same number of data analysis tasks. The experimental results show that in the same amount of time, this implementation can process more tasks, and has lower resource consumption and higher stability. The specific experimental data are as follows:

[0071] As shown in Table 1:

[0072]

[0073] Table 1

[0074] As can be seen from Table 1, when processing the same number of tasks, the average processing time of this embodiment is only half of that of the prior art, and the CPU occupancy rate and memory occupancy rate are also significantly reduced. This proves that this embodiment has higher efficiency and better resource utilization when processing large-scale structure monitoring data analysis tasks.

[0075] It is worth noting that:

[0076] 1. Data analysis task management based on containerization:

[0077] Dynamically create, schedule and manage data analysis jobs through the Kubernetes container orchestration system to ensure that each analysis task runs independently in a resource-isolated container to avoid mutual interference between tasks. The specific method of integrating Kubernetes with data analysis tasks includes the creation, scheduling, resource allocation, monitoring and destruction of jobs.

[0078] 2. ClickHouse as a high-performance data analysis engine:

[0079] Utilize ClickHouse's column storage, vectorized execution, distributed processing and other features to improve the query and analysis performance of large amounts of monitoring data. ClickHouse's specific application scenarios and optimization methods in data monitoring and analysis include but are not limited to data model design, query optimization, indexing strategies, etc.

[0080] 3. Real-time data analysis and result storage:

[0081] Realize rapid analysis of real-time monitoring data and store the analysis results in the database in real time for subsequent query and display. The specific implementation of the trigger mechanism, processing flow and result storage of real-time data analysis, as well as how to ensure the real-time and accuracy of data.

[0082] 4. Dynamic allocation and release of resources:

[0083] Dynamically allocate computing resources according to the needs of data analysis tasks, and release resources after the task is completed to improve resource utilization. The specific implementation of resource allocation algorithm and release mechanism, as well as how to ensure efficient use of resources and system stability.

[0084] 5. Concurrent execution of multiple analysis tasks:

[0085] Supports the simultaneous execution of multiple data analysis tasks to improve overall data processing efficiency. The specific implementation of concurrency control strategies and task scheduling mechanisms, as well as how to ensure coordination and non-interference between multiple analysis tasks and mutual collaboration.

[0086] In this embodiment, the client interface needs to support the selection and submission of data parameters; the Kubernetes cluster needs to support the creation, scheduling and resource management of jobs; the pre-prepared data analysis image needs to contain complete analysis logic and dependencies; the ClickHouse database needs to support high-performance large data storage and real-time analysis; the Postgresql database needs to support efficient data storage and query. The client interface can support a variety of data parameter selection methods (such as single selection, multiple selection, input box, etc.); the Kubernetes cluster can support a variety of scheduling strategies (such as priority scheduling, load balancing scheduling, etc.); the data analysis image can support a variety of data analysis algorithms and models; the clickHouse database can support distributed storage and expansion; the Postgresql database can support distributed storage and expansion. The client interface adopts a graphical method to provide intuitive and easy-to-use data parameter selection functions; the Kubernetes cluster adopts a priority-based scheduling strategy to ensure that high-priority tasks are executed first; the data analysis image adopts high-performance analysis algorithms and models to improve analysis efficiency; the ClickHouse database adopts a distributed storage solution to ensure high availability and scalability of data; the Postgresql database adopts a distributed storage solution to ensure high availability and scalability of data.

[0087] In summary, a large amount of data monitoring data analysis method has the following effects:

[0088] 1. Efficient parallel processing;

[0089] This technical solution realizes the parallel execution of data analysis tasks through the container orchestration and scheduling functions of Kubernetes. Compared with the serial or limited parallel processing methods in the prior art, this technical solution can process multiple data analysis tasks at the same time, greatly improving the processing efficiency. Specifically, the high concurrent query performance of the ClickHouse database combined with the distributed task scheduling of Kubernetes significantly improves the execution speed of analysis tasks in a large data volume environment, thereby showing a significant advantage in processing time.

[0090] 2. Dynamic allocation and release of resources;

[0091] The Kubernetes cluster can dynamically allocate computing resources according to task requirements and automatically release resources after the task is completed, avoiding resource waste. Compared with the static resource allocation method in the prior art, this technical solution can more efficiently utilize system resources and improve resource utilization. ClickHouse's column storage and efficient compression algorithm further reduce the occupancy of storage and computing resources, which significantly optimizes the system's resource utilization efficiency.

[0092] 3. Task isolation and non-interference;

[0093] Through containerization technology, each data analysis task runs in an independent container, achieving isolation between tasks. This avoids mutual influence and interference between different tasks and improves the stability and reliability of the system. Compared with the resource contention problem that may be caused by shared resources in the prior art, this technical solution shows a higher guarantee in task stability and execution reliability.

[0094] 4. Strong scalability;

[0095] This technical solution supports dynamic expansion of the size of the Kubernetes cluster to cope with the growing demand for data analysis. Compared with the fixed-size clusters in the prior art, this technical solution has stronger scalability and flexibility. Combined with the distributed architecture of ClickHouse, this system can smoothly expand nodes and storage capacity without affecting the execution of existing tasks, thereby effectively coping with the continuous growth of data volume and analysis tasks.

[0096] 5. Easy to manage and maintain;

[0097] Kubernetes provides a wealth of monitoring, logging, and troubleshooting tools, making the management and maintenance of data analysis tasks simpler and more efficient. Compared with the complex management and maintenance processes in existing technologies, this technical solution reduces operation and maintenance costs and improves work efficiency. ClickHouse's built-in monitoring and query optimization tools further simplify database performance tuning and troubleshooting, making the management of the entire system more convenient.

[0098] 6. Reduce energy consumption;

[0099] Since this technical solution can dynamically allocate and release resources, it avoids excessive occupation and waste of resources, thereby reducing the energy consumption of the system. Compared with the existing technology, this technical solution has certain advantages in energy conservation and emission reduction. ClickHouse's efficient data compression and query optimization technology reduces the consumption of hardware resources. Combined with Kubernetes' resource scheduling capabilities, the entire system maintains a low energy consumption level while running at high performance.

[0100] This implementation method realizes the parallel execution and dynamic resource management of data analysis tasks through containerization technology, improving processing efficiency and resource utilization. At the same time, through task isolation and result storage mechanism, data consistency and traceability are guaranteed. The efficient storage and query characteristics of ClickHouse database further improve the performance of large-scale data monitoring and analysis, and significantly shorten the data processing time.

[0101] By combining the characteristics of containerization and ClickHouse, efficient execution of data analysis for large-scale monitoring and dynamic management of resources are achieved. This method effectively solves problems such as blocking of multiple analysis tasks and waste of resources, improves the real-time and accuracy of data analysis, and reduces the operation and maintenance costs of the system. At the same time, due to the use of containerization technology, the system has good scalability and portability, and can easily cope with data analysis needs of different scales and complexities.

[0102] The method conceived in this application has been successfully applied to the "intelligent data analysis tool", aiming to improve the performance, efficiency and resource utilization of the tool when processing large amounts of monitoring data analysis.

[0103] Embodiment 2

[0104] like Figure 3 As shown, in a second embodiment of the present invention, a large data volume monitoring data analysis system is provided, and the system includes:

[0105] A sending module 10 is used to obtain data parameters, submit an analysis request for the data parameters, and send the analysis request to an application programming interface server of the container orchestration platform;

[0106] A creation module 20 is used to select a containerized creation data analysis task on a suitable node according to the resource status and scheduling strategy of each node in the container orchestration platform, wherein the container orchestration platform will consider the remaining CPU, memory, and storage resources of the node to ensure resource utilization, and different analysis requests will be assigned to different nodes for execution;

[0107] An analysis module 30 is used to start a container corresponding to the data analysis task, and run a data analysis code in the container to obtain monitoring data in a column-based database corresponding to the data analysis task and perform traversal analysis on the monitoring data;

[0108] The deletion module 40 is used to store the results of the traversal analysis of the monitoring data, delete the data analysis task, and release the resources occupied by it. The system consists of four parts: client, Kubernetes cluster, column database and PostgreSQL database. The client is responsible for the selection and submission of data parameters, the Kubernetes cluster is responsible for the creation, scheduling and resource management of the job, ClickHouse is used for the storage and fast query of large amounts of data, and the Postgresql database is used to store the analysis results.

[0109] The large data volume monitoring data analysis system provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0110] Embodiment 3

[0111] like Figure 4 As shown, in the third embodiment of the present invention, the embodiment of the present invention provides the following technical solution: a computer, comprising a memory 202, a processor 201, and a computer program stored in the memory 202 and executable on the processor 201, wherein the processor 201 implements the large data volume monitoring data analysis method as described above when executing the computer program.

[0112] Specifically, the processor 201 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0113] Among them, the memory 202 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 202 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 202 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 202 may be inside or outside a data processing device. In a specific embodiment, the memory 202 is a non-volatile memory. In a specific embodiment, the memory 202 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (ErasableProgrammable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically ErasableProgrammable Read-Only Memory, EEPROM for short), an electrically alterable ROM (ElectricallyAlterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0114] The memory 202 may be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 201 .

[0115] The processor 201 implements the above-mentioned large-volume monitoring data analysis method by reading and executing computer program instructions stored in the memory 202.

[0116] In some embodiments, the computer may further include a communication interface 203 and a bus 200. Figure 4 As shown, the processor 201, the memory 202, and the communication interface 203 are connected via a bus 200 and communicate with each other.

[0117] The communication interface 203 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 203 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0118] The bus 200 includes hardware, software or both, and couples the components of the computer to each other. The bus 200 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 200 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 200 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0119] Embodiment 4

[0120] In the fourth embodiment of the present invention, in combination with the above-mentioned large data volume monitoring data analysis method, the embodiment of the present invention provides the following technical solution, a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned large data volume monitoring data analysis method is implemented.

[0121] Those skilled in the art will appreciate that the logic and / or steps described in the flowchart as data or in other ways herein, for example, can be considered as a sequenced data table of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses.

[0122] More specific examples of computer-readable media (non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0123] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0124] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for analyzing large amounts of monitoring data. It is characterized in that The method comprises: Obtaining data parameters, submitting an analysis request for the data parameters, and sending the analysis request to an application programming interface server of the container orchestration platform; According to the resource conditions and scheduling strategies of each node in the container orchestration platform, select a suitable node to create a containerized data analysis task. The container orchestration platform will consider the remaining CPU, memory, and storage resources of the node to ensure resource utilization. Different analysis requests will be assigned to different nodes for execution. Starting a container corresponding to the data analysis task, running a data analysis code in the container to obtain monitoring data in a column-based database corresponding to the data analysis task and performing traversal analysis on the monitoring data; The results of the traversal and analysis of the monitoring data are stored, and the data analysis task is deleted, and the resources occupied by it are released.

2. The large data volume monitoring data analysis method according to claim 1, It is characterized in that The step of starting the container corresponding to the data analysis task includes: Creating a mirror database, and pulling the corresponding data analysis mirror from the mirror database according to the data analysis task; A corresponding container is started based on the data analysis image, wherein the image database includes a client and other data analysis tools.

3. The large data volume monitoring data analysis method according to claim 1, It is characterized in that The step of obtaining the monitoring data of the data analysis task includes: A monitoring node is created to collect and analyze the status and progress of the node when the node runs the data analysis task, and form monitoring data.

4. The large data volume monitoring data analysis method according to claim 1, It is characterized in that Before the step of obtaining data parameters, the method further includes: A Kubernetes cluster, a columnar database, and a PostgreSQL database are created, wherein the Kubernetes cluster consists of multiple nodes, an application programming interface server is running on the nodes, the ClickHouse database is used to store and process large amounts of monitoring data; and the PostgreSQL database is deployed separately on a node in the cluster to store the results of data analysis.

5. The large data volume monitoring data analysis method according to claim 1, It is characterized in that The step of obtaining data parameters comprises: A client interface is constructed, and the user selects the data parameters to be analyzed, wherein the data parameters include time range, data type, and analysis algorithm.

6. The large data volume monitoring data analysis method according to claim 1, It is characterized in that The data analysis task includes a task name, an execution command, resource requirements, and request parameters.

7. The large data volume monitoring data analysis method according to claim 1, It is characterized in that After the step of selecting a suitable node to create a data analysis task in a container, the method further includes: Determine whether the node encounters a calculation error or a node failure. If the node encounters a calculation error or a node failure, clean up the node and count the number of cleanups, and re-execute the step of creating a data analysis task request.

8. A large data volume monitoring data analysis system, It is characterized in that The system comprises: A sending module, used to obtain data parameters, submit an analysis request for the data parameters, and send the analysis request to an application programming interface server of the container orchestration platform; A creation module is used to select a containerized creation data analysis task on a suitable node according to the resource status and scheduling strategy of each node in the container orchestration platform, wherein the container orchestration platform will consider the remaining CPU, memory, and storage resources of the node to ensure resource utilization, and different analysis requests will be assigned to different nodes for execution; An analysis module is used to start a container corresponding to the data analysis task, and run a data analysis code in the container to obtain monitoring data in a column-based database corresponding to the data analysis task and perform traversal analysis on the monitoring data; The deletion module is used to store the results of traversal and analysis of the monitoring data, delete the data analysis task, and release the resources occupied by it.

9. A computer comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the large data volume monitoring data analysis method as described in any one of claims 1 to 7 is implemented.

10. A storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the large data volume monitoring data analysis method as described in any one of claims 1 to 7 is implemented.

Images