A cluster data processing method, device, equipment and medium

Abstract

The present application discloses a cluster data processing method, device, equipment and medium, which relates to the field of computer technology, including: collecting current load data of multiple dimensions of computing nodes; preprocessing the current load data, and classifying and storing the preprocessed load data through message queues of multiple different topics and according to the dimension information of the preprocessed load data; obtaining first data corresponding to the target dimension from the preprocessed load data stored in the message queue of at least one topic according to the current indicator to be analyzed; inputting the first data into the target time series model to output the predicted load data corresponding to the target dimension, and inputting the first data into the target random forest classification model to output the security status score corresponding to the target dimension. The present application improves the traditional information collection and storage technology based on the characteristics of the message queue, and solves the problems of high overhead, slow response and unbalanced IO volume of the traditional technology.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a cluster data processing method, device, equipment and medium. Background Art

[0002] The development of high-performance computing (HPC) technology is an important force in promoting scientific research, industrial innovation and national security. With the continuous growth of computing needs and the rapid evolution of technology, HPC technology has made significant progress in recent years. At the same time, high-performance computing clusters require a real-time data collection and analysis method, as well as a device to respond to and handle cluster security events in real time.

[0003] One of the traditional technologies is to collect and analyze data based on system operation logs, that is, to use special log collection tools (Fluentd, Logstash, rsyslog, etc.) to centrally collect various logs and forward them to the centralized storage and processing platform to achieve real-time monitoring, performance analysis, fault diagnosis and security analysis. The second traditional technology is to collect and analyze data based on cluster management software, that is, cluster management software usually collects and transmits the status information of each node and job in the cluster in real time through daemons (slurmctld, slurmd), and performs job scheduling analysis, resource usage analysis, fault detection and diagnosis, etc.

[0004] However, the above two methods require writing user data to disk and then reading data from disk regularly for analysis. This adds a lot of extra system IO (input / output) overhead to the system. In addition, the regular collection, analysis, and update methods cannot well meet the real-time requirements of the security monitoring system for handling security incidents. The above two methods will cause the IO volume of the entire cluster to be unbalanced, tilting towards a single node or some nodes, which will lead to bottlenecks in the operation of the entire cluster. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a cluster data processing method, device, equipment and medium, which improves the traditional information collection and storage technology based on the characteristics of the message queue, solves the problems of high overhead, slow response and unbalanced IO volume of the traditional technology, and its specific scheme is as follows:

[0006] In a first aspect, the present application discloses a cluster data processing method, comprising:

[0007] Collect current load data of computing nodes in multiple dimensions;

[0008] Preprocessing the current load data according to the preprocessing strategy to obtain preprocessed load data, and classifying and storing the preprocessed load data through message queues of multiple different topics and according to dimension information of the preprocessed load data; each topic corresponds to one dimension;

[0009] Determine a current indicator to be analyzed, and acquire first data corresponding to a target dimension from the preprocessed load data stored in the message queue of at least one topic according to the current indicator to be analyzed;

[0010] The first data is input into a pre-trained target time series model to output predicted load data corresponding to the target dimension, and the first data is input into a pre-trained target random forest classification model to output a safety status score corresponding to the target dimension.

[0011] Optionally, the cluster data processing method further includes:

[0012] Persistently back up and store the pre-processed load data in the message queue in the form of log storage at a preset time interval and through a preset database;

[0013] The second data is obtained from the pre-processed load data stored in the message queue, and the second data is analyzed offline to obtain target analysis data.

[0014] Optionally, before inputting the first data into a pre-trained target time series model, the method further includes:

[0015] Constructing a first training set according to the preprocessed load data in the target time window in the preset database, and determining a time series model to be trained; wherein the time series model to be trained includes an integrated self-attention layer and a position encoding module, the self-attention layer is used to capture the temporal dependency of the data, and the position encoding module is used to provide the position information of the time series;

[0016] The time series model to be trained is trained using the first training set, and the target time series model is obtained when the corresponding first training loss satisfies a first loss condition.

[0017] Optionally, before inputting the first data into a pre-trained target random forest classification model, the method further includes:

[0018] Constructing a second training set according to the preprocessed load data in the preset database, and determining a random forest classification model to be trained; wherein the second training set includes data corresponding to a normal behavior pattern and data corresponding to an abnormal behavior pattern, and different behavior patterns correspond to different safety states;

[0019] The random forest classification model to be trained is trained using the second training set, and the target random forest classification model is obtained when the corresponding second training loss satisfies a second loss condition.

[0020] Optionally, preprocessing the current load data according to a preprocessing strategy to obtain preprocessed load data includes:

[0021] The current load data is marked by a target identifier, missing data in the current load data is supplemented, and the current load data is sampled based on a preset data sampling mechanism to obtain the preprocessed load data.

[0022] Optionally, after inputting the first data into a pre-trained target time series model to output predicted load data corresponding to the target dimension, the method further includes:

[0023] Displaying the predicted load data, and determining whether the predicted load data is greater than a load threshold;

[0024] If the predicted load data is greater than the load threshold, an early warning prompt is issued.

[0025] Optionally, after inputting the first data into a pre-trained target random forest classification model to output a security status score corresponding to the target dimension, the method further includes:

[0026] Determine whether the security status score is greater than a preset score threshold;

[0027] If the security status score is not greater than the preset score threshold, it is determined that the security status score corresponding to the target dimension meets the security event warning condition, and a warning prompt is issued.

[0028] In a second aspect, the present application discloses a cluster data processing device, comprising:

[0029] The data collection module is used to collect the current load data of computing nodes in multiple dimensions;

[0030] A data storage module, used to preprocess the current load data according to a preprocessing strategy to obtain preprocessed load data, and classify and store the preprocessed load data through message queues of multiple different topics and according to dimension information of the preprocessed load data; each topic corresponds to one dimension;

[0031] A data acquisition module, used to determine the current indicator to be analyzed, and acquire first data corresponding to the target dimension from the pre-processed load data stored in the message queue of at least one topic according to the current indicator to be analyzed;

[0032] A data processing module is used to input the first data into a pre-trained target time series model to output predicted load data corresponding to the target dimension, and input the first data into a pre-trained target random forest classification model to output a safety status score corresponding to the target dimension.

[0033] In a third aspect, the present application discloses an electronic device, including:

[0034] Memory, used to store computer programs;

[0035] The processor is used to execute the computer program to implement the cluster data processing method disclosed above.

[0036] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the cluster data processing method disclosed above is implemented.

[0037] It can be seen that the present application proposes a cluster data processing method, including: collecting current load data of multiple dimensions of computing nodes; preprocessing the current load data according to the preprocessing strategy to obtain preprocessed load data, and classifying and storing the preprocessed load data through message queues of multiple different topics and according to the dimension information of the preprocessed load data; each topic corresponds to a dimension; determining the current indicator to be analyzed, and obtaining the first data corresponding to the target dimension from the preprocessed load data stored in the message queue of at least one topic according to the current indicator to be analyzed; inputting the first data into the pre-trained target time series model to output the predicted load data corresponding to the target dimension, and inputting the first data into the pre-trained target random forest classification model to output the security status score corresponding to the target dimension. In summary, since the message queue allows the sender to return immediately after placing the message in the queue without having to wait for the receiver to process the message, the present application stores the preprocessed load data through the message queue, which can improve the response speed and throughput of the system. Furthermore, since the present application no longer needs to use a collection node to complete the data collection of all computing nodes, but stores the pre-processed load data through the message queue, load balancing can be achieved, and the imbalance of the IO volume of the entire cluster can be avoided, resulting in the IO volume tilting toward a single node or part of the nodes. Furthermore, the present application does not need to write the user's data to the disk, and then regularly read the data from the disk for analysis, which greatly reduces the system IO overhead. In other words, the present application improves the traditional information collection and storage technology based on the characteristics of the message queue, and solves the problems of high overhead, slow response and uneven IO volume of traditional technology. In addition, the present application uses the target time series model to extract time-related load characteristics, improves the accuracy of load data prediction, and outputs the security status score of the load data in real time through the target random forest classification model, which can help administrators detect abnormal behavior and potential threats in real time and ensure the safe and stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A flow chart of a cluster data processing method disclosed in this application;

[0040] Figure 2 A schematic diagram of a specific cluster data processing system disclosed in this application;

[0041] Figure 3 A schematic diagram of the structure of a cluster data processing device disclosed in this application;

[0042] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] One of the traditional technologies is to collect and analyze data based on system operation logs. The second traditional technology is to collect and analyze data based on cluster management software. However, the above two methods require writing user data to disk and then regularly reading data from disk for analysis. This form adds a lot of additional system IO overhead to the system. In addition, the regular collection, analysis, and update methods cannot well meet the real-time requirements of the security monitoring system for handling security events. In addition, the above two methods will cause the IO volume of the entire cluster to be unbalanced, tilted towards a single node or some nodes, which will cause the operation of the entire cluster to encounter bottlenecks.

[0045] To this end, the embodiment of the present application proposes a cluster data processing solution, which improves the traditional technology based on the characteristics of the message queue to solve the problems of high overhead, slow response and unbalanced IO volume of the traditional technology.

[0046] The present application embodiment discloses a cluster data processing method, see Figure 1 As shown, the method includes:

[0047] Step S11: Collect current load data of multiple dimensions of computing nodes.

[0048] The problems that the High Performance Computing Center needs to face are as follows: (1) Communication network security issues: Since the remote SSH (Secure Shell) login service needs to be opened and users need to install and compile software, the system must provide sufficient network and system permissions, which makes the system face greater security risks. (2) Cluster computing node security issues: Nodes may crash or be illegally invaded due to user program errors or user account theft. (3) Cluster storage system security issues: The number of files and disk space usage of the storage system need to be continuously monitored. Excessive usage may threaten the overall security of the system. To this end, this embodiment designs a cluster early warning method based on message queue technology. According to the various activity data during the operation of the cluster system, it dynamically monitors data changes, builds an intelligent analysis model, predicts the cluster load in the future, and monitors and warns the security status of the cluster.

[0049] The current load data of multiple dimensions of the computing nodes are collected through the daemon process deployed on the computing nodes. Among them, the computing node (Compute Node) refers to the basic computing unit in the cluster, which is responsible for executing the actual computing tasks. Each computing node is equipped with multiple processors, memory, storage and network interfaces to support high-density parallel computing and data processing. The load data of multiple dimensions includes CPU (Central Processing Unit) load, memory load, network load and I / O load. Among them, the CPU load includes the utilization rate of each core and the total utilization rate, the memory load includes the total memory usage, used memory and free memory, the network load includes network throughput, network delay, packet loss rate, etc., and the I / O load includes disk I / O rate, average I / O time, etc.

[0050] Step S12: preprocess the current load data according to the preprocessing strategy to obtain preprocessed load data, and classify and store the preprocessed load data through message queues of multiple different topics and according to the dimension information of the preprocessed load data; each topic corresponds to one dimension.

[0051] Since the types and semantics of the collected current load data are diverse, there is no unified identification information mechanism, and network transmission may cause disorder or loss, for this reason, this embodiment needs to provide a unified identification mechanism, and make approximate supplements to the missing data, and then perform sampling compression on a large amount of system operation status information. That is, the current load data is preprocessed according to the preprocessing strategy to obtain the preprocessed load data, including: marking the current load data through the target identifier, supplementing the missing data in the current load data, and sampling the current load data based on the preset data sampling mechanism to obtain the preprocessed load data, so as to reduce the storage capacity of the data, thereby improving the validity of the data.

[0052] In one implementation, the random forest method and the K-NearesNeighbor (K-NN) method can be used to implement data preprocessing, data alignment, and supplement missing values ​​for prediction. Take the K-NN method as an example: The K-NN method usually uses the Euclidean distance to measure the distance between data points. and , the Euclidean distance d(x,y) between them is: , using the K-NN regression method to complete missing information includes: 1. Select parameter K for missing data information M: determine the number of neighbors K to be used. 2. Calculate distance: for the data point to be supplemented, calculate its distance from all data points in the training set. 3. Select the nearest K neighbors: based on the calculated distance, select the K data points with the nearest distance. 4. Take the average: calculate the average of these K neighbors and use it as the supplementary value of the data point.

[0053] Furthermore, the pre-processed load data is classified and stored through message queues with multiple different topics and according to the dimension information of the pre-processed load data; each topic corresponds to a dimension. Message queues have the following advantages: (1) Response time: Message queues allow the sender to return immediately after placing a message in the queue without having to wait for the receiver to process the message. This asynchronous mechanism improves the response speed and throughput of the system. (2) Decoupling: Producers and consumers do not need to communicate directly, and messages are transmitted through queues. This decoupling makes the system more scalable and flexible, and producers and consumers can be expanded and maintained independently. (3) Message persistence: Message queues usually support persistent storage of messages. Even if the system fails, the messages will not be lost, ensuring reliable data transmission. (4) Load balancing: Multiple consumers can read messages from the same queue, and the queue can distribute messages to multiple consumers in a balanced manner, thereby achieving load balancing. (5) Traffic peak shaving: In high-concurrency scenarios, message queues can buffer a large number of requests to prevent the system from being impacted by sudden high traffic. Consumers can process messages at their own speed to avoid system overload. (6) Order guarantee: The message queue can ensure that messages are delivered in a certain order, ensuring the order of message processing, which is very important in some scenarios. (7) Retry mechanism: If the consumer fails to process the message, the message queue usually supports automatic retry to ensure that the message is eventually processed successfully. Further, the topics include CPU load topics, memory load topics, network load topics, etc.

[0054] Step S13: determine the current indicator to be analyzed, and obtain first data corresponding to the target dimension from the preprocessed load data stored in the message queue of at least one topic according to the current indicator to be analyzed.

[0055] In this embodiment, the current indicator to be analyzed is determined, and the first data corresponding to the target dimension is obtained from the preprocessed load data stored in the message queue of at least one topic according to the current indicator to be analyzed. By way of example, assuming that the CPU and memory need to be analyzed currently, the CPU load data is obtained from the CPU topic queue, and the memory load data is obtained from the memory topic queue. In this case, the target dimensions are the CPU dimension and the memory dimension, and the first data are the CPU load data and the memory load data.

[0056] Step S14: Input the first data into a pre-trained target time series model to output predicted load data corresponding to the target dimension, and input the first data into a pre-trained target random forest classification model to output a safety status score corresponding to the target dimension.

[0057] In this embodiment, a target time series model and a target random forest classification model are first trained.

[0058] On the one hand, a first training set is constructed based on the pre-processed load data in the target time window in the preset database, and a time series model to be trained (Time Series Transformer, TST) is determined; wherein the time series model to be trained includes an integrated self-attention layer and a position encoding module, the self-attention layer is used to capture the temporal dependency of the data, and the position encoding module is used to provide the position information of the time series. Further, the time series model to be trained is trained using the first training set, and the target time series model is obtained when the corresponding first training loss satisfies the first loss condition.

[0059] On the other hand, a second training set is constructed based on the pre-processed load data in the preset database, and a random forest classification model to be trained is determined; wherein the second training set includes data corresponding to normal behavior patterns and data corresponding to abnormal behavior patterns, and different behavior patterns correspond to different safety states. Further, the random forest classification model to be trained is trained using the second training set, and a target random forest classification model is obtained when the corresponding second training loss satisfies the second loss condition.

[0060] It should be pointed out that the training set used in the training model of this embodiment is constructed based on historical data. In this embodiment, not only the real-time optimization of the traditional model is performed, but also deep learning models with their own characteristics are applied to mine the features and patterns in the historical data. While having better real-time performance, it can also make faster and more accurate judgments based on time series patterns and cluster specificity.

[0061] Further, the first data is input into a pre-trained target time series model to output predicted load data corresponding to the target dimension, and the predicted load data is displayed, and then it is determined whether the predicted load data is greater than the load threshold. If the predicted load data is greater than the load threshold, an early warning prompt is issued. Exemplarily, this embodiment can predict the load conditions for the next 5 minutes or 10 minutes through the target time series model. This embodiment can set the CPU load threshold to 90% and the memory load threshold to 80%. When the predicted load exceeds the set threshold, an alarm is triggered and the system administrator is notified by SMS, email or other means. It should be pointed out that the above-mentioned specific numerical values ​​may vary in different implementation scenarios, and this embodiment does not make specific limitations here.

[0062] Further, the first data is input into a pre-trained target random forest classification model to output a security status score corresponding to the target dimension, and determine whether the security status score is greater than a preset score threshold. If the security status score is not greater than the preset score threshold, it is determined that the security status score corresponding to the target dimension meets the security event warning condition, and a warning prompt is issued. It should be pointed out that random forest is an integrated learning algorithm that performs classification or regression tasks by constructing multiple decision trees and taking the average value. The random forest model has strong robustness, can process high-dimensional data, and is insensitive to noise. It is very suitable for anomaly detection tasks in behavioral analysis. In addition, this embodiment can also use a decision tree to obtain a security status score. The decision tree is a common classification algorithm that can make decisions on input features through a tree structure. The decision tree model is simple and easy to explain, and is suitable for regular processing and identification of specific behaviors.

[0063] In this embodiment, in addition to being able to perform real-time analysis of data, offline analysis of data is also supported. Specifically, the pre-processed load data in the message queue is persistently backed up and stored in the form of log storage at a preset time interval and through a preset database, and then the second data is obtained from the pre-processed load data stored in the message queue, and the second data is analyzed offline to obtain target analysis data.

[0064] Figure 2The main framework of a high-performance cluster security incident response device based on a message queue disclosed in this embodiment specifically includes a data collection module, a preprocessing module, a message organization module and an intelligent analysis module. Specifically, (1) the data collection module is used to collect load information through a lightweight daemon process (MP, monitor process) and send the collected information to the preprocessing module. (2) The preprocessing module is used to clean, align, complete and sample and compress the data, and forward the data to the distributed message queue in the message organization module after preprocessing. (3) The message queue is used to cache the preprocessed data, and message queues of different topics are used to store load data of different dimensions. (4) The information in the queue is periodically backed up to a persistent device in the form of a log, which can be used for offline analysis applications with low real-time requirements. (5) The TST time series model is trained with structured data with timestamps for load prediction of real-time data. (6) The random forest classification model is trained with structured data of the security log for real-time cluster security status judgment and cluster security event monitoring and early warning.

[0065] In summary, this application collects and analyzes cluster data with message queues as the core. In specific analysis, deep learning time series models and classification models based on historical data training are applied to deeply explore the time series and classification features in historical data, so as to more accurately, quickly and efficiently predict cluster load conditions and judge the security status of the cluster. Compared with traditional methods, this application has higher response time, lower IO access pressure, and is easy to achieve load balancing, avoiding cluster IO operation concentration from becoming a bottleneck.

[0066] It can be seen that the present application proposes a cluster data processing method, including: collecting current load data of multiple dimensions of computing nodes; preprocessing the current load data according to the preprocessing strategy to obtain preprocessed load data, and classifying and storing the preprocessed load data through message queues of multiple different topics and according to the dimension information of the preprocessed load data; each topic corresponds to a dimension; determining the current indicator to be analyzed, and obtaining the first data corresponding to the target dimension from the preprocessed load data stored in the message queue of at least one topic according to the current indicator to be analyzed; inputting the first data into the pre-trained target time series model to output the predicted load data corresponding to the target dimension, and inputting the first data into the pre-trained target random forest classification model to output the security status score corresponding to the target dimension. In summary, since the message queue allows the sender to return immediately after placing the message in the queue without having to wait for the receiver to process the message, the present application stores the preprocessed load data through the message queue, which can improve the response speed and throughput of the system. Furthermore, since the present application no longer needs to use a collection node to complete the data collection of all computing nodes, but stores the pre-processed load data through the message queue, load balancing can be achieved, and the imbalance of the IO volume of the entire cluster can be avoided, resulting in the IO volume tilting toward a single node or part of the nodes. Furthermore, the present application does not need to write the user's data to the disk, and then regularly read the data from the disk for analysis, which greatly reduces the system IO overhead. In other words, the present application improves the traditional information collection and storage technology based on the characteristics of the message queue, and solves the problems of high overhead, slow response and uneven IO volume of traditional technology. In addition, the present application uses the target time series model to extract time-related load characteristics, improves the accuracy of load data prediction, and outputs the security status score of the load data in real time through the target random forest classification model, which can help administrators detect abnormal behavior and potential threats in real time and ensure the safe and stable operation of the system.

[0067] Correspondingly, the present application also discloses a cluster data processing device, see Figure 3 As shown, the device comprises:

[0068] The data collection module 11 is used to collect the current load data of multiple dimensions of the computing nodes;

[0069] The data storage module 12 is used to preprocess the current load data according to the preprocessing strategy to obtain preprocessed load data, and classify and store the preprocessed load data through message queues of multiple different topics and according to the dimension information of the preprocessed load data; each topic corresponds to one dimension;

[0070] The data acquisition module 13 is used to determine the current indicator to be analyzed, and acquire first data corresponding to the target dimension from the pre-processed load data stored in the message queue of at least one topic according to the current indicator to be analyzed;

[0071] The data processing module 14 is used to input the first data into a pre-trained target time series model to output predicted load data corresponding to the target dimension, and input the first data into a pre-trained target random forest classification model to output a safety status score corresponding to the target dimension.

[0072] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0073] It can be seen that the present application proposes a cluster data processing method, including: collecting current load data of multiple dimensions of computing nodes; preprocessing the current load data according to the preprocessing strategy to obtain preprocessed load data, and classifying and storing the preprocessed load data through message queues of multiple different topics and according to the dimension information of the preprocessed load data; each topic corresponds to a dimension; determining the current indicator to be analyzed, and obtaining the first data corresponding to the target dimension from the preprocessed load data stored in the message queue of at least one topic according to the current indicator to be analyzed; inputting the first data into the pre-trained target time series model to output the predicted load data corresponding to the target dimension, and inputting the first data into the pre-trained target random forest classification model to output the security status score corresponding to the target dimension. In summary, since the message queue allows the sender to return immediately after placing the message in the queue without having to wait for the receiver to process the message, the present application stores the preprocessed load data through the message queue, which can improve the response speed and throughput of the system. Furthermore, since the present application no longer needs to use a collection node to complete the data collection of all computing nodes, but stores the pre-processed load data through the message queue, load balancing can be achieved, and the imbalance of the IO volume of the entire cluster can be avoided, resulting in the IO volume tilting toward a single node or part of the nodes. Furthermore, the present application does not need to write the user's data to the disk, and then regularly read the data from the disk for analysis, which greatly reduces the system IO overhead. In other words, the present application improves the traditional information collection and storage technology based on the characteristics of the message queue, and solves the problems of high overhead, slow response and uneven IO volume of traditional technology. In addition, the present application uses the target time series model to extract time-related load characteristics, improves the accuracy of load data prediction, and outputs the security status score of the load data in real time through the target random forest classification model, which can help administrators detect abnormal behavior and potential threats in real time and ensure the safe and stable operation of the system.

[0074] Furthermore, an embodiment of the present application also provides an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be considered as any limitation on the scope of use of the present application.

[0075] Figure 4 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a display screen 23, an input and output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the cluster data processing method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0076] In this embodiment, the power supply 26 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 24 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0077] In addition, the memory 22, as a carrier for storing resources, may be a read-only memory, a random access memory, a disk or an optical disk, etc., and the resources stored thereon may include a computer program 221, and the storage method may be temporary storage or permanent storage. In addition to including a computer program that can be used to complete the cluster data processing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 221 may further include a computer program that can be used to complete other specific tasks.

[0078] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the cluster data processing method disclosed above is implemented.

[0079] For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be described in detail here.

[0080] The various embodiments in this application are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0081] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0082] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0083] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0084] The above is a detailed introduction to a cluster data processing method, device, equipment, and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A cluster data processing method, characterized in that: include: Collect current load data of computing nodes in multiple dimensions; Preprocessing the current load data according to the preprocessing strategy to obtain preprocessed load data, and classifying and storing the preprocessed load data through message queues of multiple different topics and according to dimension information of the preprocessed load data; each topic corresponds to one dimension; Determine a current indicator to be analyzed, and acquire first data corresponding to a target dimension from the preprocessed load data stored in the message queue of at least one topic according to the current indicator to be analyzed; Inputting the first data into a pre-trained target time series model to output predicted load data corresponding to the target dimension, and inputting the first data into a pre-trained target random forest classification model to output a safety status score corresponding to the target dimension; Before inputting the first data into the pre-trained target time series model, the method further includes: Constructing a first training set according to the preprocessed load data in the target time window in the preset database, and determining a time series model to be trained; wherein the time series model to be trained includes an integrated self-attention layer and a position encoding module, the self-attention layer is used to capture the temporal dependency of the data, and the position encoding module is used to provide the position information of the time series; Using the first training set to train the time series model to be trained, and obtaining the target time series model when the corresponding first training loss satisfies a first loss condition; Before inputting the first data into the pre-trained target random forest classification model, the method further includes: Constructing a second training set according to the preprocessed load data in the preset database, and determining a random forest classification model to be trained; wherein the second training set includes data corresponding to a normal behavior pattern and data corresponding to an abnormal behavior pattern, and different behavior patterns correspond to different safety states; The random forest classification model to be trained is trained using the second training set, and the target random forest classification model is obtained when the corresponding second training loss satisfies a second loss condition.

2. The cluster data processing method according to claim 1, characterized in that: Also includes: Persistently back up and store the pre-processed load data in the message queue in the form of log storage at a preset time interval and through a preset database; The second data is obtained from the pre-processed load data stored in the message queue, and the second data is analyzed offline to obtain target analysis data.

3. The cluster data processing method according to claim 1, characterized in that: The preprocessing of the current load data according to the preprocessing strategy to obtain preprocessed load data includes: The current load data is marked by a target identifier, missing data in the current load data is supplemented, and the current load data is sampled based on a preset data sampling mechanism to obtain the preprocessed load data.

4. The cluster data processing method according to any one of claims 1 to 3, characterized in that: After inputting the first data into a pre-trained target time series model to output predicted load data corresponding to the target dimension, the method further includes: Displaying the predicted load data, and determining whether the predicted load data is greater than a load threshold; If the predicted load data is greater than the load threshold, an early warning prompt is issued.

5. The cluster data processing method according to any one of claims 1 to 3, characterized in that: After inputting the first data into a pre-trained target random forest classification model to output a security status score corresponding to the target dimension, the method further includes: Determine whether the security status score is greater than a preset score threshold; If the security status score is not greater than the preset score threshold, it is determined that the security status score corresponding to the target dimension meets the security event warning condition, and a warning prompt is issued.

6. A cluster data processing device, characterized in that: include: The data collection module is used to collect the current load data of computing nodes in multiple dimensions; A data storage module, used to preprocess the current load data according to a preprocessing strategy to obtain preprocessed load data, and classify and store the preprocessed load data through message queues of multiple different topics and according to dimension information of the preprocessed load data; each topic corresponds to one dimension; A data acquisition module, used to determine the current indicator to be analyzed, and acquire first data corresponding to the target dimension from the pre-processed load data stored in the message queue of at least one topic according to the current indicator to be analyzed; a data processing module, configured to input the first data into a pre-trained target time series model to output predicted load data corresponding to the target dimension, and input the first data into a pre-trained target random forest classification model to output a safety status score corresponding to the target dimension; Before inputting the first data into the pre-trained target time series model, the method further includes: Constructing a first training set according to the preprocessed load data in the target time window in the preset database, and determining a time series model to be trained; wherein the time series model to be trained includes an integrated self-attention layer and a position encoding module, the self-attention layer is used to capture the temporal dependency of the data, and the position encoding module is used to provide the position information of the time series; Using the first training set to train the time series model to be trained, and obtaining the target time series model when the corresponding first training loss satisfies a first loss condition; Before inputting the first data into the pre-trained target random forest classification model, the method further includes: Constructing a second training set according to the preprocessed load data in the preset database, and determining a random forest classification model to be trained; wherein the second training set includes data corresponding to a normal behavior pattern and data corresponding to an abnormal behavior pattern, and different behavior patterns correspond to different safety states; The random forest classification model to be trained is trained using the second training set, and the target random forest classification model is obtained when the corresponding second training loss satisfies a second loss condition.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the cluster data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the cluster data processing method according to any one of claims 1 to 5 is implemented.

Images