Mining process detection method, device, computing node and storage medium
By installing monitoring process agents in the operating system, obtaining and comparing the process's resource consumption and microarchitecture parameters, identifying the mining process, solving the problem of high misjudgment rate in the existing technology, and realizing accurate detection of mining processes and Effective protection of resources.
Patent Information
- Application Number
- CN202211520269.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-11-30
AI Technical Summary
The existing mining attack detection methods have a high misjudgment rate, making it difficult to accurately identify mining processes disguised as normal operations, resulting in waste of resources and difficulty in operation and maintenance.
By installing the monitoring process agent in the operating system, obtain the resource consumption parameters and microarchitecture parameters of each process, compare these parameters with the threshold of normal jobs, identify the abnormal process and determine it as a mining process.
It reduces the misjudgment rate of mining processes, accurately detects mining processes disguised as normal operations, saves operation and maintenance personnel's inspection time and protects user resources.
Smart Images

Figure CN116015714B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of mining attack detection, and in particular to a mining process detection method, device, computing node and storage medium. Background Art
[0002] With the rapid development of the information society, various fields of society have put forward new requirements for computer performance. Computer clusters in different forms have well met the needs of high-performance computing, but they also face certain difficulties in operation and maintenance. Mining attacks are the most common security risk in clusters. They usually obtain user authentication information and simulate mining operations as normal operations of the cluster, thereby occupying cluster resources, causing normal operations to run slowly or fail to apply for resources, and a large amount of cluster resources are wasted. At present, mining attack detection mainly uses several indicators summarized in the industry to determine whether there is mining behavior in the computing node. However, this judgment method has a high misjudgment rate and low accuracy in identifying mining attacks. Summary of the invention
[0003] The embodiments of the present application provide a mining process detection method, device, computing node and storage medium, which are beneficial to improving the accuracy of mining attack detection and reducing the misjudgment rate of the mining process.
[0004] In a first aspect, an embodiment of the present application provides a mining process detection method, which is applied to an electronic device, and the method includes:
[0005] Obtaining a first resource consumption parameter and a first micro-architecture parameter of each process in at least one running process;
[0006] Determining an abnormal process from at least one process according to the first resource consumption parameter or the first micro-architecture parameter;
[0007] If the mining detection parameters of the abnormal process meet the determination conditions of mining behavior, the abnormal process is determined to be a mining process.
[0008] It can be seen that in the embodiment of the present application, a monitoring process agent can be installed in the operating system of the electronic device, and the monitoring process agent monitors at least one running process to obtain the first resource consumption parameter and the first micro-architecture parameter of each process, and firstly uses the first resource consumption parameter or the first micro-architecture parameter to determine the abnormal process, and then compares the mining detection parameter with the judgment condition of the mining behavior to determine the mining process from the abnormal process, so as to accurately detect the mining process disguised as the normal operating process, effectively avoid the normal operating process from being misjudged as the mining process, and then help to reduce the misjudgment rate of the mining process.
[0009] In some possible implementations, the first resource consumption parameter includes shaping computing power occupancy data, floating-point computing power occupancy data, memory read and write rate, swap partition occupancy data and network bandwidth occupancy data; the first microarchitecture parameter includes double-precision floating-point calculation speed, single-precision floating-point calculation speed, vectorization unit floating-point calculation speed, scalar unit floating-point calculation speed, vectorization rate, memory bandwidth and number of clock cycles.
[0010] In this implementation, resource consumption parameters and micro-architecture parameters of each process in multiple dimensions are obtained by monitoring the process agent, which is conducive to detecting abnormal processes using these parameters.
[0011] In some possible implementations, determining an abnormal process from at least one process according to the first resource consumption parameter or the first micro-architecture parameter includes:
[0012] For any process A in at least one process, if one or more of the integer computing power usage data, floating-point computing power usage data, memory read / write rate, swap partition usage data, and network bandwidth usage data of process A is greater than or equal to the corresponding resource consumption parameter threshold, process A is determined to be an abnormal process; or,
[0013] If one or more of the double-precision floating-point calculation speed, single-precision floating-point calculation speed, vectorization unit floating-point calculation speed, scalar unit floating-point calculation speed, vectorization rate, memory bandwidth and clock cycle of process A is greater than or equal to the corresponding micro-architecture parameter threshold, process A is determined to be an abnormal process.
[0014] In this embodiment, the first resource consumption parameter is compared with a resource consumption parameter threshold value obtained based on the resource consumption parameter of a normal operating process, or the first micro-architecture parameter is compared with a micro-architecture parameter threshold value obtained based on the micro-architecture parameter of a normal operating process, so that a mining process disguised as a normal operating process can be detected.
[0015] In some possible implementations, before obtaining the first resource consumption parameter, the first micro-architecture parameter, and the mining detection parameter of the target process in at least one running process, the method further includes:
[0016] Get the process name of each of the multiple processes running;
[0017] If the process name contains a mining process name, the process corresponding to the mining process name is determined as the mining process;
[0018] A process that is not determined to be a mining process among the multiple processes is determined as at least one process.
[0019] In this implementation, for multiple processes running in the electronic device, their process names are detected to identify typical mining processes, and for the remaining processes that are not determined to be mining processes, the operations of obtaining the first resource consumption parameter and the first micro-architecture parameter and determining the abnormal process based on the first resource consumption parameter and the first micro-architecture parameter are performed, thereby facilitating the identification of mining processes disguised as normal operating process names.
[0020] In some possible implementations, before obtaining the process name of each of the multiple running processes, the method further includes:
[0021] Performing stress testing on multiple applications to obtain second resource consumption parameters and second micro-architecture parameters of processes of the multiple applications;
[0022] Obtaining a resource consumption parameter threshold according to the second resource consumption parameter;
[0023] A micro-architecture parameter threshold is obtained according to the second micro-architecture parameter.
[0024] In this implementation, by running stress tests of different normal applications, the second resource consumption parameter and the second micro-architecture parameter of the normal operation process are obtained, and then based on the second resource consumption parameter and the second micro-architecture parameter, the corresponding resource consumption parameter threshold and the micro-architecture parameter threshold are obtained respectively, so as to facilitate comparison during subsequent mining process detection.
[0025] In some possible implementations, the electronic device is a computing node in a high-performance computing cluster, and the at least one process is an application process in the high-performance computing cluster.
[0026] In some possible implementations, the method further includes:
[0027] Generate corresponding processing suggestions and warning information for the mining process, and report the processing suggestions and warning information to the management platform of the high-performance computing cluster; or kill the mining process.
[0028] In this implementation, the computing nodes in the high-performance computing cluster can assist the operation and maintenance personnel in managing the application processes in the cluster, for example, when a mining process is detected, corresponding processing suggestions are generated, and the processing suggestions and the alarm information of the process are reported to the management platform for the convenience of operation and maintenance personnel to manage. Alternatively, the computing nodes in the high-performance computing cluster can actively operate and maintain, for example, directly killing or terminating the mining process.
[0029] In a second aspect, an embodiment of the present application provides a mining process detection device, the device comprising an acquisition unit and a processing unit;
[0030] an acquisition unit, configured to acquire a first resource consumption parameter and a first micro-architecture parameter of each process in at least one running process;
[0031] The processing unit is used to determine an abnormal process from at least one process according to the first resource consumption parameter or the first micro-architecture parameter; if the mining detection parameter of the abnormal process meets the determination condition of the mining behavior, the abnormal process is determined as a mining process.
[0032] In some possible implementations, the first resource consumption parameter includes shaping computing power occupancy data, floating-point computing power occupancy data, memory read and write rate, swap partition occupancy data and network bandwidth occupancy data; the first microarchitecture parameter includes double-precision floating-point calculation speed, single-precision floating-point calculation speed, vectorization unit floating-point calculation speed, scalar unit floating-point calculation speed, vectorization rate, memory bandwidth and number of clock cycles.
[0033] In some possible implementations, in determining an abnormal process from at least one process according to the first resource consumption parameter or the first micro-architecture parameter, the processing unit is specifically configured to:
[0034] For any process A in at least one process, if one or more of the integer computing power usage data, floating-point computing power usage data, memory read / write rate, swap partition usage data, and network bandwidth usage data of process A is greater than or equal to the corresponding resource consumption parameter threshold, process A is determined to be an abnormal process; or,
[0035] If one or more of the double-precision floating-point calculation speed, single-precision floating-point calculation speed, vectorization unit floating-point calculation speed, scalar unit floating-point calculation speed, vectorization rate, memory bandwidth and clock cycle of process A is greater than or equal to the corresponding micro-architecture parameter threshold, process A is determined to be an abnormal process.
[0036] In some possible implementations, the processing unit is further configured to:
[0037] Get the process name of each of the multiple processes running;
[0038] If the process name contains a mining process name, the process corresponding to the mining process name is determined as the mining process;
[0039] A process that is not determined to be a mining process among the multiple processes is determined as at least one process.
[0040] In some possible implementations, the processing unit is further configured to:
[0041] Performing stress testing on multiple applications to obtain second resource consumption parameters and second micro-architecture parameters of processes of the multiple applications;
[0042] Obtaining a resource consumption parameter threshold according to the second resource consumption parameter;
[0043] A micro-architecture parameter threshold is obtained according to the second micro-architecture parameter.
[0044] In some possible implementations, the device is applied to a computing node in a high-performance computing cluster, and at least one process is an application process in the high-performance computing cluster.
[0045] In some possible implementations, the processing unit is further used to: generate corresponding processing suggestions for the mining process, and report the processing suggestions and alarm information to the management platform of the high-performance computing cluster; or, kill the mining process.
[0046] It should be noted that the second aspect is a device corresponding to the first aspect, which is used to implement the various method steps provided by the first aspect. For specific implementation details and beneficial effects, please refer to the first aspect.
[0047] In a third aspect, an embodiment of the present application provides a computing node, comprising a processor, a memory, and one or more programs, wherein the processor is connected to the memory, the one or more programs are stored in the memory, and are configured to implement the method in the first aspect when executed by the processor.
[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program for execution by a computing node, and when the computer program is executed, the method in the above-mentioned first aspect is implemented.
[0049] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run by a computing node, the computing node executes the method in the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.
[0051] Figure 1 is a schematic diagram of an HPC cluster;
[0052] Figure 2 A schematic diagram of a system architecture provided for an embodiment of the present application;
[0053] Figure 3 A flowchart of a mining process detection method provided in an embodiment of the present application;
[0054] Figure 4 A schematic diagram of a monitoring process agent obtaining mining detection parameters provided in an embodiment of the present application;
[0055] Figure 5 A flowchart of another mining process detection method is provided for an embodiment of the present application;
[0056] Figure 6 A schematic diagram of resource consumption parameters of multiple applications under a stress test provided in an embodiment of the present application;
[0057] Figure 7 A schematic diagram of the structure of a mining process detection device provided in an embodiment of the present application;
[0058] Figure 8 A schematic diagram of the structure of a computing node provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0060] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0061] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program and / or a computer running on a processor. By way of illustration, both applications running on a terminal device and a terminal device can be components. One or more components can reside in a process and / or an execution thread, and a component can be located on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. Components can, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).
[0062] In order to facilitate the understanding of the embodiments of the present application, further analyze and propose the technical problems to be specifically solved by the present application, the relevant technical solutions of the present application are briefly introduced below.
[0063] For example, Figure 1 As shown, a high performance computing (HPC) cluster usually includes a login node, a computing node, and a storage system. Users transmit account authentication information through encrypted communication, and attackers obtain the account authentication information and simulate mining operations as normal operations of the HPC cluster to perform mining attacks, thereby occupying a large amount of cluster resources. A related technology uses a Python script to detect whether a computing node has mining behavior. Specifically, the port, traffic, memory usage, processor (central processing unit, CPU) usage, and C disk usage (some mining programs are characterized by occupying a large amount of C disk space) are directly detected to find the mining process and kill the mining process. However, HPC itself is a computing-intensive business. It is difficult to accurately determine whether it is a mining process through CPU and memory usage, and not all mining processes will occupy a large amount of C disk resources. Therefore, the above detection method has a high misjudgment rate, and simply and roughly killing the mining process may affect the normal operation in the HPC cluster.
[0064] Based on the above technical problems, the mining process detection method, device, computing node and storage medium provided in the embodiments of the present application are described in detail below in combination with the accompanying drawings.
[0065] See also Figure 2 , Figure 2 A schematic diagram of a system architecture provided in an embodiment of the present application is shown as follows: Figure 2As shown, it includes an HPC cluster, a user and a management platform, wherein the computing node of the HPC cluster includes a scheduler, which is used to schedule computing resources for the user's resource request. A monitoring process agent is installed in the operating system of the computing node. The monitoring process agent is used to monitor all processes in the HPC cluster and obtain the resource consumption parameters and micro-architecture parameters of each process. The monitoring process agent compares the resource consumption parameters and micro-architecture parameters with the characteristics of normal operations to detect abnormal processes, and then compares the mining detection parameters such as the port, traffic, memory usage, CPU usage and C disk usage of the abnormal process with the characteristics of mining behavior to detect the mining process, thereby determining that the HPC cluster is under mining attack and realizing the alarm reporting to the management platform in seconds.
[0066] See also Figure 3 , Figure 3 A flowchart of a mining process detection method provided in an embodiment of the present application is provided. The method can be applied to electronic devices, such as Figure 3 As shown, the method may include steps 301-303:
[0067] 301: Obtain a first resource consumption parameter and a first micro-architecture parameter of each process in at least one running process.
[0068] In an embodiment of the present application, a monitoring process agent can be installed in the operating system of the electronic device, and the monitoring process agent can monitor at least one running process and obtain the first resource consumption parameter and the first micro-architecture parameter of each process. Wherein, the electronic device can be a computing node in an HPC cluster, and at least one process can be an application process in the HPC cluster, for example: the application can be OpenForm (an object-oriented computational fluid dynamics library written in C++ and running under Linux), Gromacs (a molecular dynamics package for studying biological molecular systems), CP2K (an open source first-principles material calculation and simulation software), and so on.
[0069] Since the micro-architecture characteristics of different applications show different characteristics in parameters such as single-precision computing speed, double-precision computing speed, floating-point computing, vectorization rate, memory bandwidth, etc., the characteristics can be used to distinguish the normal application process and the mining application process in at least one process, and the first micro-architecture parameter includes double-precision floating-point computing speed, single-precision floating-point computing speed, vectorization unit floating-point computing speed, standard unit floating-point computing speed, vectorization rate, memory bandwidth and the number of clock cycles (for example: how many clock cycles are occupied). Similarly, the resource consumption characteristics of different applications in parameters such as shaping computing power, floating-point computing power, memory read and write rate, swap occupancy, network bandwidth, etc. will also show different characteristics, and the characteristics can also be used to distinguish the normal application process and the mining application process in at least one process, and the first resource consumption parameter can include shaping computing power occupancy data, floating-point computing power occupancy data, memory read and write rate, swap partition occupancy data and network bandwidth occupancy data. Of course, the first resource consumption parameter and the first micro-architecture parameter can also include more or fewer detailed parameters, and the embodiments of the present application do not make any restrictions. In this implementation, resource consumption parameters and micro-architecture parameters of each process in multiple dimensions are obtained by monitoring the process agent, which is conducive to detecting abnormal processes using these parameters.
[0070] 302: Determine an abnormal process from at least one process according to a first resource consumption parameter or a first micro-architecture parameter.
[0071] In the embodiment of the present application, each first resource consumption parameter in step 301 has a corresponding resource consumption parameter threshold value, and the electronic device compares each first resource consumption parameter with the corresponding resource consumption parameter threshold value. If one or more first resource consumption parameters are greater than or equal to the corresponding resource consumption parameter threshold value, the process A corresponding to the parameter can be determined as an abnormal process. Alternatively, each first micro-architecture parameter in step 301 has a corresponding micro-architecture parameter threshold value, and the electronic device compares each first micro-architecture parameter with the corresponding micro-architecture parameter threshold value. If one or more first micro-architecture parameters are greater than or equal to the corresponding micro-architecture parameter threshold value, the process A corresponding to the parameter can be determined as an abnormal process. Wherein, process A is any one of the at least one process.
[0072] In this embodiment, the first resource consumption parameter is compared with a resource consumption parameter threshold value obtained based on the resource consumption parameter of a normal operating process, or the first micro-architecture parameter is compared with a micro-architecture parameter threshold value obtained based on the micro-architecture parameter of a normal operating process, so that a mining process disguised as a normal operating process can be detected.
[0073] 303: If the mining detection parameter of the abnormal process meets the determination condition of the mining behavior, the abnormal process is determined as a mining process.
[0074] In the embodiment of the present application, for example, the monitoring process agent obtains the first resource consumption parameter and the first micro-architecture parameter of each process in at least one running process, and also obtains the mining detection parameter of each process; or, after determining an abnormal process from at least one process, the monitoring process agent directly obtains the mining detection parameter of the abnormal process. Figure 4 As shown, the electronic device performs port detection, flow detection, memory consumption detection, CPU usage detection, graphics processing unit (GPU) usage detection, etc. through the monitoring process agent, and the mining detection parameters include at least one of the following: port detection parameters, flow detection parameters, memory usage detection parameters, CPU usage detection parameters, and GPU usage detection parameters.
[0075] There is a corresponding judgment condition for each mining detection parameter, for example, the judgment condition of the port detection parameter can be that the process is accessed through a specific port. If the port detection parameter is True, it can be determined that the process is accessed through a specific port. The flow detection parameter can be the total flow calculated based on the sent flow and the received flow. The judgment condition of the flow detection can be greater than or equal to the flow threshold. Similarly, the judgment conditions of the memory usage detection parameter, the CPU usage detection parameter, and the GPU usage detection parameter can also be greater than or equal to the corresponding threshold. If one or more mining detection parameters of the abnormal process meet the judgment conditions of the mining behavior, the electronic device determines the abnormal process as a mining process.
[0076] Exemplarily, the HPC cluster has a corresponding management platform, through which the operation and maintenance personnel can perform operation and maintenance work such as node configuration and process management. The computing node can generate corresponding processing suggestions and alarm information for the mining process and report it to the management platform to assist the operation and maintenance personnel in making associations. For example, the processing suggestion can be to adjust the resource consumption parameter threshold and the micro-architecture parameter threshold, or the processing suggestion can be to kill the mining process, or the processing suggestion can be to obtain its mining detection parameters again at a preset time for further comparison. Exemplarily, the computing node can also directly kill the detected mining process, that is, it has an active operation and maintenance function.
[0077] It can be seen that in the embodiment of the present application, a monitoring process agent can be installed in the operating system of the electronic device, and the monitoring process agent monitors at least one running process to obtain the first resource consumption parameter and the first micro-architecture parameter of each process, first use the first resource consumption parameter or the first micro-architecture parameter to determine the abnormal process, and then compare the mining detection parameter with the judgment condition of the mining behavior to determine the mining process from the abnormal process, so as to accurately detect the mining process disguised as a normal operation process, effectively avoid the normal operation process from being misjudged as a mining process, and then help reduce the misjudgment rate of the mining process. Accurately detecting the mining process is conducive to reducing the resource loss of the HPC cluster, saving the detection time of the operation and maintenance personnel, and protecting user resources.
[0078] See also Figure 5 , Figure 5 A flowchart of another mining process detection method is provided for an embodiment of the present application. The method can be applied to computing nodes in an HPC cluster, such as Figure 5 As shown, the method may include steps 501-505:
[0079] 501: Get the process name of each of the multiple running processes;
[0080] 502: If the process name contains a mining process name, determining the process corresponding to the mining process name as the mining process;
[0081] 503: For at least one process among the multiple processes that is not determined to be a mining process, obtain a first resource consumption parameter and a first micro-architecture parameter of each process in the at least one process;
[0082] 504: Determine an abnormal process from at least one process according to the first resource consumption parameter or the first micro-architecture parameter;
[0083] 505: If the mining detection parameter of the abnormal process meets the determination condition of mining behavior, the abnormal process is determined as a mining process.
[0084] In the embodiment of the present application, the multiple processes refer to all processes running in the HPC cluster. The computing node obtains the process name of each of the multiple processes by monitoring the process agent, and determines whether there is a mining process name among these names. If there is a mining process name, the corresponding process is the mining process, that is, the computing node can detect a typical mining process by monitoring the process agent.
[0085] For at least one process that is not determined to be a mining process among the multiple processes, there may be a mining process disguised as a normal operation process, so it is detected through steps 503-505. The specific implementation of steps 503-505 can be referred to Figure 3 The relevant descriptions in the illustrated embodiments can achieve the same or similar beneficial effects.
[0086] Exemplarily, before obtaining the process name of each process in the plurality of running processes, the method further includes:
[0087] Performing stress testing on multiple applications to obtain second resource consumption parameters and second micro-architecture parameters of processes of the multiple applications;
[0088] Obtaining a resource consumption parameter threshold according to the second resource consumption parameter;
[0089] A micro-architecture parameter threshold is obtained according to the second micro-architecture parameter.
[0090] Among them, the multiple applications can be various HPC applications such as OpenForm, Gromacs, CP2K, VASP (Vienna Ab-initioSimulation Package, a software package for electronic structure calculation and quantum mechanics-molecular dynamics simulation). By running stress tests of different applications, the second resource consumption parameter of the normal application is obtained by monitoring the process agent. For example, in addition to the integer computing power usage data, floating-point computing power usage data, memory read and write rate, swap partition usage data and network bandwidth usage data, the resource consumption parameters may also include disk read and write rate, receiving network usage data, sending network usage data, infinite bandwidth sending bandwidth usage data, infinite bandwidth receiving bandwidth usage data, etc. See Figure 6 For multiple applications running on each computing node, the monitoring process agent can obtain all resource consumption parameters of each application, and then calculate the average value of each resource consumption parameter for multiple applications on the node. Taking computing nodes Node1 and Node2 as an example, assuming that they run 5 applications in total, the average CPU usage of these 5 applications is 48.13%, the average floating-point computing power usage is 24.11%, the average swap partition usage is 0.06%, the average memory usage is 2.02%..., and so on, the average value of all resource consumption parameters of each process in Node1 and Node2 can be obtained, and then the average value can be used as the corresponding resource consumption parameter threshold for all running processes, so as to make abnormal judgments on all processes. For the average values of the above resource consumption parameters obtained at different times, the monitoring process agent can fit them into corresponding waveform graphs ( Figure 6 The waveform diagram of floating-point computing capacity usage data and memory read and write rate is only given as an example, and does not limit the embodiments of the present application in any way) and is displayed to the operation and maintenance personnel. Taking the OpenForm application as an example, the performance under the load pressure of 1-8 nodes is tested, and it can be found that OpenForm has the following characteristics:
[0091] 1. In terms of time consumption, performance increases linearly with the number of nodes;
[0092] 2. In terms of vectorization degree, 4%-5% is low;
[0093] 3. In terms of memory bandwidth, the peak memory read bandwidth reaches 300 GB / s, the average is around 200 GB / s, and the write bandwidth is around 50-100 GB / s, which places high requirements on memory bandwidth;
[0094] 4. From the perspective of IO (input / output), the example data was read in before the test started, and the read IO reached 2200000KB / s. This application is mainly based on read IO.
[0095] The performance parameters under the OpenForm stress test above are examples of resource consumption parameters for normal applications.
[0096] Similarly, for multiple applications, the monitoring process agent can obtain all the second micro-architecture parameters of each application on each computing node, and calculate the average value in each second micro-architecture parameter dimension as the micro-architecture parameter threshold. For example: calculate the average value of the double-precision floating-point computing speed of each process on all computing nodes as the double-precision floating-point computing speed threshold, and so on.
[0097] In this implementation, by running stress tests of different normal applications, the second resource consumption parameter and the second micro-architecture parameter of the normal operation process are obtained, and then based on the second resource consumption parameter and the second micro-architecture parameter, the corresponding resource consumption parameter threshold and the micro-architecture parameter threshold are obtained respectively, so as to facilitate comparison during subsequent mining process detection.
[0098] Exemplarily, after detecting the mining process, the computing node implements assisted operation and maintenance or active operation and maintenance through the monitoring process agent.
[0099] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.
[0100] See also Figure 7 , Figure 7 A schematic diagram of the structure of a mining process detection device 700 provided in an embodiment of the present application. The mining process detection device 700 includes an acquisition unit 701 and a processing unit 702;
[0101] An acquisition unit 701 is used to acquire a first resource consumption parameter and a first micro-architecture parameter of each process in at least one running process;
[0102] The processing unit 702 is used to determine an abnormal process from at least one process according to the first resource consumption parameter or the first micro-architecture parameter; if the mining detection parameter of the abnormal process meets the determination condition of the mining behavior, the abnormal process is determined as a mining process.
[0103] It can be seen that in Figure 7 In the mining process detection device 700 shown, the acquisition unit 701 monitors at least one running process through the monitoring process agent to obtain the first resource consumption parameter and the first micro-architecture parameter of each process. The processing unit 702 first uses the first resource consumption parameter or the first micro-architecture parameter to determine the abnormal process, and then compares the mining detection parameter with the judgment condition of the mining behavior to determine the mining process from the abnormal process, so as to accurately detect the mining process disguised as the normal operation process, effectively avoid the normal operation process from being misjudged as the mining process, and thus help to reduce the misjudgment rate of the mining process.
[0104] In some possible implementations, the first resource consumption parameter includes shaping computing power occupancy data, floating-point computing power occupancy data, memory read and write rate, swap partition occupancy data and network bandwidth occupancy data; the first microarchitecture parameter includes double-precision floating-point calculation speed, single-precision floating-point calculation speed, vectorization unit floating-point calculation speed, scalar unit floating-point calculation speed, vectorization rate, memory bandwidth and number of clock cycles.
[0105] In some possible implementations, in terms of determining an abnormal process from at least one process according to the first resource consumption parameter or the first micro-architecture parameter, the processing unit 702 is specifically configured to:
[0106] For any process A in at least one process, if one or more of the integer computing power usage data, floating-point computing power usage data, memory read / write rate, swap partition usage data, and network bandwidth usage data of process A is greater than or equal to the corresponding resource consumption parameter threshold, process A is determined to be an abnormal process; or,
[0107] If one or more of the double-precision floating-point calculation speed, single-precision floating-point calculation speed, vectorization unit floating-point calculation speed, scalar unit floating-point calculation speed, vectorization rate, memory bandwidth and clock cycle of process A is greater than or equal to the corresponding micro-architecture parameter threshold, process A is determined to be an abnormal process.
[0108] In some possible implementations, the processing unit 702 is further configured to:
[0109] Get the process name of each of the multiple processes running;
[0110] If the process name contains a mining process name, the process corresponding to the mining process name is determined as the mining process;
[0111] A process that is not determined to be a mining process among the multiple processes is determined as at least one process.
[0112] In some possible implementations, the processing unit 702 is further configured to:
[0113] Performing stress testing on multiple applications to obtain second resource consumption parameters and second micro-architecture parameters of processes of the multiple applications;
[0114] Obtaining a resource consumption parameter threshold according to the second resource consumption parameter;
[0115] A micro-architecture parameter threshold is obtained according to the second micro-architecture parameter.
[0116] In some possible implementations, the mining process detection device 700 is applied to a computing node in a high-performance computing cluster, and at least one process is an application process in the high-performance computing cluster.
[0117] In some possible implementations, the processing unit 702 is further used to: generate corresponding processing suggestions for the mining process, and report the processing suggestions and alarm information to the management platform of the high-performance computing cluster; or, kill the mining process.
[0118] The present application embodiment also provides a computing node, which may be a computing node in an HPC cluster. Figure 8 , Figure 8 A schematic diagram of the structure of a computing node 800 provided in an embodiment of the present application, the computing node 800 includes at least a processor 801, a memory 802, an input device 803 and an output device 804, and one or more programs. One or more programs are stored in the memory 802, and the various parts are interconnected by a bus or other means. Among them, the input device 803 and the output device 804 can be the same device, and the input device 803 can be used to receive input from a user or other device, wherein the input device 803 can be a keyboard, a touch screen, a radio frequency receiver, a microphone, etc.; the output device 804 can be used to output the interactive information between the computing node and other devices or users (for example: sending alarm information to the management platform), wherein the output device 804 can be a speaker, a display, a radio frequency transmitter, etc. Figure 8 Each unit shown may be implemented by the processor 801 , memory 802 , input device 803 , and output device 804 described above.
[0119] The memory 802 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), and is used to store relevant computer programs and data.
[0120] The processor 801 may be one or more CPUs. When the processor 801 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.
[0121] It should be noted that the implementation of each operation can also refer to Figure 3 or Figure 5 The corresponding description of the method of any one of the embodiments shown.
[0122] It should be noted that although Figure 8 The computing node 800 shown only shows a processor 801, a memory 802, an input device 803, and an output device 804. However, in the specific implementation process, those skilled in the art should understand that the computing node 800 also includes other devices necessary for normal operation, such as: memory, power supply, application client module, etc. At the same time, according to specific needs, those skilled in the art should understand that the computing node 800 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computing node 800 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 8 All devices shown in .
[0123] The embodiment of the present application further provides a computer-readable storage medium (Memory), which is a memory device in the computing node 800 and is used to store a computer program for device execution. When the computer program is executed on the computing node 800, Figure 3 or Figure 5The method flow shown is implemented. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the computing node 800 and the extended storage medium supported by the computing node 800. The computer-readable storage medium provides a storage space, which stores the operating system of the computing node 800. In addition, one or more computer programs suitable for being loaded and executed by the processor 801 are also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor 801.
[0124] The embodiment of the present application also provides a computer program product, when the computer program product is executed by a device, Figure 3 or Figure 5 The method flow shown is realized.
[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0126] It should be understood that the processor mentioned in the embodiments of the present application may be a CPU, or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0127] It should also be understood that the memory mentioned in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a ROM, a programmable read-only memory (Programmable ROM, PROM), an EPROM, an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM) or a flash memory. The volatile memory can be a RAM, which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (DoubleData Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM).
[0128] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) is integrated in the processor.
[0129] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0130] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0131] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely exemplary. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0133] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium.
[0134] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of this application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0135] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.
[0136] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.
[0137] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A mining process detection method, characterized in that: The method comprises: Obtaining a first resource consumption parameter and a first micro-architecture parameter of each process in at least one running process; determining an abnormal process from the at least one process according to the first resource consumption parameter or the first micro-architecture parameter; If the mining detection parameter of the abnormal process meets the determination condition of mining behavior, the abnormal process is determined as a mining process; The first resource consumption parameter includes shaping computing power occupancy data, floating-point computing power occupancy data, memory read and write rate, swap partition occupancy data and network bandwidth occupancy data; the first microarchitecture parameter includes double-precision floating-point calculation speed, single-precision floating-point calculation speed, vectorized unit floating-point calculation speed, standard unit floating-point calculation speed, vectorized rate, memory bandwidth and number of clock cycles; the mining detection parameter includes at least one of the following: port detection parameter, traffic detection parameter, memory usage detection parameter, CPU usage detection parameter, GPU usage detection parameter.
2. The method according to claim 1, characterized in that The determining an abnormal process from the at least one process according to the first resource consumption parameter or the first micro-architecture parameter includes: For any one process A of the at least one process, if one or more of the integer computing power occupancy data, floating-point computing power occupancy data, memory read / write rate, swap partition occupancy data, and network bandwidth occupancy data of the process A is greater than or equal to the corresponding resource consumption parameter threshold, the process A is determined as the abnormal process; or, If one or more of the double-precision floating-point calculation speed, single-precision floating-point calculation speed, vectorization unit floating-point calculation speed, scalar unit floating-point calculation speed, vectorization rate, memory bandwidth and clock cycle of process A is greater than or equal to the corresponding micro-architecture parameter threshold, then process A is determined as the abnormal process.
3. The method according to claim 1 or 2, characterized in that: Before obtaining the first resource consumption parameter, the first micro-architecture parameter, and the mining detection parameter of the target process in at least one running process, the method further includes: Get the process name of each of the multiple processes running; If a mining process name exists in the process name, determining the process corresponding to the mining process name as the mining process; A process among the multiple processes that is not determined to be a mining process is determined as the at least one process.
4. The method according to claim 3, characterized in that Before obtaining the process name of each process in the plurality of running processes, the method further includes: Performing stress testing on a plurality of applications to obtain second resource consumption parameters and second micro-architecture parameters of processes of the plurality of applications; According to the second resource consumption parameter, obtaining the resource consumption parameter threshold; The micro-architecture parameter threshold is obtained according to the second micro-architecture parameter.
5. The method according to claim 1, characterized in that The mining process detection method is applied to a computing node in a high-performance computing cluster, and the at least one process is an application process in the high-performance computing cluster.
6. The method according to claim 5, characterized in that The method further comprises: Generate corresponding processing suggestions and warning information for the mining process, and report the processing suggestions and the warning information to the management platform of the high-performance computing cluster; or kill the mining process.
7. A mining process detection device, characterized in that: The device comprises an acquisition unit and a processing unit; The acquisition unit is used to acquire a first resource consumption parameter and a first micro-architecture parameter of each process in at least one running process; The processing unit is configured to determine an abnormal process from the at least one process according to the first resource consumption parameter or the first micro-architecture parameter; If the mining detection parameter of the abnormal process meets the determination condition of mining behavior, the abnormal process is determined as a mining process; The first resource consumption parameter includes shaping computing power occupancy data, floating-point computing power occupancy data, memory read and write rate, swap partition occupancy data and network bandwidth occupancy data; the first microarchitecture parameter includes double-precision floating-point calculation speed, single-precision floating-point calculation speed, vectorized unit floating-point calculation speed, standard unit floating-point calculation speed, vectorized rate, memory bandwidth and number of clock cycles; the mining detection parameter includes at least one of the following: port detection parameter, traffic detection parameter, memory usage detection parameter, CPU usage detection parameter, GPU usage detection parameter.
8. A computing node, characterized in that: The system comprises a processor, a memory, and one or more programs, wherein the processor is connected to the memory, the one or more programs are stored in the memory, and are configured to implement the method according to any one of claims 1 to 6 when executed by the processor.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for execution by a computing node, and the computer program implements the method according to any one of claims 1 to 6 when executed.
