Distributed cluster operation control method and device, electronic equipment and storage medium

By using the encapsulator layer recurrent neural network in a distributed cluster for parameter prediction, the problem of inaccurate manual adjustment timing is solved, automatic adjustment and prediction accuracy are improved, and system stability and resource utilization efficiency are improved.

CN119088535APending Publication Date: 2024-12-06BEIJING JINGDONG QIANSHITECHNOLOGY CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310664530.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-06
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

During the data processing of distributed clusters, it is difficult to accurately grasp the opportunity when manually adjusting the cluster operation parameters, resulting in waste of resources and degradation of system computing performance, affecting system stability and consuming manpower.

Method used

The recurrent neural network with the encapsulator layer is used to train the parameter prediction model. By performing multi-dimensional feature extraction and trend analysis on the cluster processing data and operation data, the cluster target operation data within the future time range is predicted, thereby automatically adjusting the cluster operation parameters. .

Benefits of technology

It improves the prediction accuracy of cluster operation parameters, realizes automatic adjustment, avoids resource waste and performance degradation caused by manual adjustment, saves human resources, and improves the operating stability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088535A_ABST
    Figure CN119088535A_ABST
Patent Text Reader

Abstract

The invention provides a distributed cluster operation control method and device, electronic equipment and a storage medium. Relates to the technical field of data processing. The method comprises the following steps: acquiring cluster processing data and cluster operation data of a distributed cluster in a first time range before a current time point; multi-dimensional feature extraction is carried out on the image; inputting the feature data of the multiple dimensions into a parameter prediction model according to a time sequence, determining cluster target operation data within a second time range after the current time point, and performing corresponding operation on the distributed cluster according to the cluster target operation data to control cluster operation, the parameter prediction model is obtained by training a recurrent neural network with a wrapper layer by using cluster historical data. The problem of resource waste or system calculation performance reduction caused by manual adjustment of cluster operation parameters in related technologies can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a distributed cluster operation control method and device, electronic equipment, and storage medium. Background Art

[0002] In the data processing process of distributed clusters, a business data is usually decomposed into multiple distributed tasks, and multiple distributed tasks are assigned to multiple physical nodes for parallel computing; multiple distributed subtasks can be connected together to form a task. However, for a business, the amount of data in different periods may vary greatly, which requires adaptive adjustments to task parallelism, cluster hardware resources, etc.

[0003] Currently, based on past experience, real-time task stress testing and hardware resource capacity adjustment are usually performed manually on distributed clusters before the data volume may change significantly. This will result in inaccurate timing of adjustments, resulting in resource waste or reduced system computing performance, affecting the stability of system operation and wasting manpower.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the embodiments of the present disclosure is to provide a distributed cluster operation control method and device, electronic device, and storage medium, thereby solving to a certain extent the problem of resource waste or system computing performance degradation caused by manual adjustment of cluster operation parameters in related technologies, which affects the stability of system operation.

[0006] According to a first aspect of the present disclosure, there is provided an operation control method for a distributed cluster, comprising: obtaining cluster processing data and cluster operation data of the distributed cluster within a first time range before a current time point; performing multi-dimensional feature extraction on the cluster processing data and the cluster operation data, respectively, to obtain feature data corresponding to each dimension; inputting the feature data of multiple dimensions into a parameter prediction model in a time series, and determining cluster target operation data within a second time range after the current time point, so that a cluster control platform performs corresponding operations on the distributed cluster according to the cluster target operation data to control cluster operation; the cluster target operation data includes multiple cluster operation parameters, and the parameter prediction model is obtained by training a recurrent neural network with an encapsulator layer using cluster historical data.

[0007] Optionally, the method further includes: in response to the second time range including the first target deadline, determining that the first time range includes the second target deadline; the first target deadline and the second target deadline are respectively time ranges that include a target event.

[0008] Optionally, the multi-dimensional feature extraction of the data includes: performing index processing on the cluster processing data and the cluster operation data according to a first time dimension to obtain first feature data; performing discrete feature extraction on the first feature data to obtain second feature data; performing trend analysis on the cluster processing data and the cluster operation data according to multiple second time dimensions to obtain third feature data corresponding to each second time dimension.

[0009] Optionally, the cluster target operation data includes cluster target resource parameters and cluster target operation parameters, and determining the cluster target operation data within a second time range after the current time point includes: inputting feature data of multiple dimensions into the recurrent neural network in time series to obtain hidden layer output; using an encapsulator layer to perform the same layer processing on each time step of the hidden layer output to obtain cluster target resource parameters and cluster target operation parameters.

[0010] Optionally, the training process of the parameter prediction model includes: using Keras to create a recurrent neural network model with a wrapper layer; inputting a time window parameter and a feature quantity parameter into the network model; obtaining historical cluster data of a historical time period including the target event, and performing multi-dimensional feature extraction on the historical cluster data; inputting the extracted features into the network model to train the network model according to the time window parameter and the feature quantity parameter.

[0011] Optionally, the method further includes: pushing the cluster target operation data to a corresponding database, so as to obtain the latest cluster target operation data by calling a data interface of the database, and adjusting the operation data of the distributed cluster according to the latest cluster target operation data.

[0012] Optionally, obtaining cluster processing data and cluster operation data includes: in response to the triggering of a scheduled task, or in response to the data processing volume of the monitoring cluster being greater than a preset threshold, obtaining cluster processing data and cluster operation data of the distributed cluster within a first time range before and at the current time point.

[0013] According to a second aspect of the present disclosure, there is provided an operation control device for a distributed cluster, the device comprising: an acquisition module, a feature extraction module and a control module, wherein: the acquisition module is used to acquire cluster processing data and cluster operation data of the distributed cluster within a first time range before a current time point; the feature extraction module is used to perform multi-dimensional feature extraction on the cluster processing data and the cluster operation data, respectively, to obtain feature data corresponding to each dimension; the control module is used to input the feature data of multiple dimensions into a parameter prediction model in a time series, and determine the cluster target operation data within a second time range after the current time point, so that a cluster control platform performs corresponding operations on the distributed cluster according to the cluster target operation data to control the cluster operation; the cluster target operation data includes multiple cluster operation parameters, and the parameter prediction model is obtained by training a recurrent neural network with an encapsulator layer using cluster historical data.

[0014] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0015] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a storage device for one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described in any one of the above embodiments.

[0016] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0017] In the distributed cluster operation control method provided by the example implementation of the present disclosure, on the one hand, a parameter prediction model composed of a recurrent neural network with a wrapper layer can be used to implement a many-to-many prediction process from the input of multi-dimensional features of the first time range to the output of multiple cluster operation parameters of the second time range, combining the data features of multiple dimensions of cluster processing data and cluster operation data, improving the prediction accuracy of cluster operation parameters and improving cluster operation stability. On the other hand, the automatic adjustment of cluster operation parameters can be achieved, avoiding the problem of wasting computing resources or system performance degradation caused by manual adjustment, and saving human resources.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0020] Figure 1 An exemplary application scenario architecture diagram of a distributed cluster operation control method and apparatus according to an embodiment of the present disclosure is schematically shown.

[0021] Figure 2 The flowchart of the operation control method of the distributed cluster according to one embodiment of the present disclosure is schematically shown.

[0022] Figure 3 The flowchart of the multi-dimensional feature extraction process according to one embodiment of the present disclosure is schematically shown.

[0023] Figure 4 The figure schematically shows the data processing process of the parameter prediction model according to one embodiment of the present disclosure.

[0024] Figure 5 The flowchart schematically shows the operation control process of the distributed cluster according to one embodiment of the present disclosure.

[0025] Figure 6 The structural block diagram of the operation control device of the distributed cluster according to one embodiment of the present disclosure is schematically shown.

[0026] Figure 7 A block diagram of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0028] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0029] like Figure 1 As shown, an exemplary system diagram of an application scenario of an operation control method and device of a distributed cluster is provided, and the system includes a distributed cluster and a server. The method of this embodiment can be applied to a server. It can be understood that the method can also be applied to a distributed cluster, and can also be applied to a system including a distributed cluster and a server, and is implemented through the interaction between the distributed cluster and the server. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, or a node in a blockchain.

[0030] For example, in the distributed cluster data processing process of the e-commerce platform, the server can be the server corresponding to the offline system, and the server can obtain the cluster processing data and cluster operation data of the offline system at the current time point and within the first time range before it by calling the data repository of the offline system. The server performs multi-dimensional feature extraction on the cluster processing data and cluster operation data respectively to obtain the feature data corresponding to each dimension; the feature data of multiple dimensions are input into the parameter prediction model in time series, the cluster target operation data within the second time range after the current time point is determined, and the cluster target operation data is sent to the distributed cluster, so that the control platform of the distributed cluster performs corresponding operations on the distributed cluster according to the cluster target operation data to control the cluster operation; the cluster target operation data includes multiple cluster operation parameters, and the parameter prediction model is obtained by training a recurrent neural network with a wrapper layer using cluster historical data.

[0031] The operation control method of the distributed cluster provided in the embodiment of the present disclosure may be executed on a server, and accordingly, the operation control device of the distributed cluster is generally arranged in the server.

[0032] The following describes the operation control method of the distributed cluster disclosed in the embodiments of this specification in conjunction with specific embodiments.

[0033] refer to Figure 2 As shown, a distributed cluster operation control method according to an example implementation provided by the present disclosure may include the following steps.

[0034] Step S210: acquiring cluster processing data and cluster operation data of the distributed cluster within a first time range before and at the current time point.

[0035] Step S220 , performing multi-dimensional feature extraction on the cluster processing data and the cluster operation data respectively to obtain feature data corresponding to each dimension.

[0036] Step S230, input the feature data of multiple dimensions into the parameter prediction model in time series, and determine the cluster target operation data within a second time range after the current time point, so that the cluster control platform can perform corresponding operations on the distributed cluster according to the cluster target operation data to control the cluster operation; the cluster target operation data includes multiple cluster operation parameters, and the parameter prediction model is obtained by training a recurrent neural network with a wrapper layer using cluster historical data.

[0037] In the distributed cluster operation control method provided in this example implementation, on the one hand, a parameter prediction model composed of a recurrent neural network with an encapsulator layer can be used to implement a many-to-many prediction process from the input of multi-dimensional features of the first time range to the output of multiple cluster operation parameters of the second time range, combining the data features of multiple dimensions of cluster processing data and cluster operation data, improving the prediction accuracy of cluster operation parameters and improving cluster operation stability. On the other hand, automatic adjustment of cluster operation parameters can be achieved to avoid the problem of wasting computing resources or system performance degradation caused by manual adjustment, saving human resources.

[0038] The following describes each step of the present disclosure in more detail.

[0039] In step S210, cluster processing data and cluster operation data of a distributed cluster within a first time range before a current time point are obtained.

[0040] In this example implementation, cluster processing data refers to data corresponding to processing tasks of a distributed cluster, which may include traffic data, business order data, real-time task throughput data, etc. It may also include other data that affects the stability of cluster operation, which is not limited in this example. For example, for a Flink cluster, cluster processing data may include traffic data of Flink topic partitions, business order data of the cluster, real-time task throughput of the cluster, etc.

[0041] In this example implementation, cluster operation data refers to data that affects the operation speed of the distributed cluster, and may include cluster hardware resource data, hardware resource usage data, real-time task parallelism data, etc. For example, cluster operation data includes cluster CPU, memory, disk and other data, cluster CPU, memory, disk usage, real-time task parallelism, etc. Cluster processing data and cluster operation data may be stored in a corresponding offline database, and data may be obtained from the offline database.

[0042] In this example implementation, the first time range may be preconfigured, for example, a number of days / weeks / months. The first time range may be determined according to the time range to be predicted. For example, for a periodic fluctuation event (such as an e-commerce platform anniversary event), when the time range to be predicted includes the periodic fluctuation event, the time period before the current time point in which the periodic fluctuation event occurs may be selected as the first time range.

[0043] Exemplarily, in response to the second time range including the first target deadline, it is determined that the first time range includes the second target deadline.

[0044] In this example implementation, the second time range refers to the time range to be predicted, the first target deadline refers to the time range after the current time point including the target event, and the second target deadline refers to the time range before the current time point including the target event. The target event refers to a periodic event that causes a large fluctuation in the data processing volume of the distributed cluster, exemplarily, a regular promotion event of an e-commerce platform, such as the Double Eleven promotion or a specific holiday promotion, etc., which is not limited in this example.

[0045] For example, when the second time range includes the time range of the Double Eleven promotion period, the first time range can be determined as the Double Eleven promotion period in recent years and several months / weeks before the current time point.

[0046] For example, in response to the triggering of a scheduled task, cluster processing data and cluster operation data can be obtained to perform operation data prediction. Alternatively, the prediction task can be triggered by monitoring the data processing volume of the application platform and in response to the data processing volume being greater than a preset threshold.

[0047] In this example implementation, the cluster operation parameter prediction task may be triggered by timing, or may be triggered by monitoring the cluster data processing volume, which is not limited in this example.

[0048] In step S220, multi-dimensional feature extraction is performed on the cluster processing data and the cluster operation data respectively to obtain feature data corresponding to each dimension.

[0049] In this example implementation, multi-dimensional feature extraction may include basic data dimensions, data discreteness dimensions, data trend dimensions, and may also include other dimensions such as time dimensions, which are not limited in this example.

[0050] For example, Figure 3 As shown, multi-dimensional feature extraction can be performed through the following steps.

[0051] According to the first time dimension, the cluster processing data and the cluster operation data are respectively indexed to obtain first characteristic data.

[0052] In this example implementation, the first time dimension refers to the statistical dimension of the basic data (the acquired cluster processing data and cluster operation data). For example, the first time dimension can be based on a time interval of one day or several days, and the index processing refers to the statistics of the basic data according to the first time dimension. For example, according to the dimension of each day, the index data corresponding to each feature is counted. The first feature data may include the daily topic partition traffic, order volume, cluster hardware resource volume (such as CPU, memory, disk, etc.), task parallelism, task throughput, etc. of the cluster.

[0053] A discreteness feature is extracted from the first feature data to obtain second feature data.

[0054] In this example implementation, the discreteness feature extraction is the process of extracting the degree of data discreteness from the first feature data. A discreteness index can be configured, and the discreteness index can include one or more, and the second feature data can be determined by calculating each discreteness index. The discreteness index can include standard deviation, mean, coefficient of variation, etc., and can also include other indicators that can characterize the degree of data discreteness, which is not limited in this example. For example, the discreteness index can be set to the coefficient of variation, and the coefficient of variation corresponding to each feature in the first feature data is calculated, and the calculation result is used as the second feature data.

[0055] According to the multiple second time dimensions, trend analysis is performed on the cluster processing data and the cluster operation data respectively to obtain third characteristic data corresponding to each second time dimension.

[0056] In this example implementation, the second time dimension can be specifically set according to the actual situation. For example, the second time dimension can include one month, two months, three months, half a year, one year, etc., which is not limited in this example. The acquired basic data (cluster processing data and cluster operation data) can be subjected to trend analysis of each second time dimension. For example, the month-on-month data of the current month and the previous month, the month-on-month data of the current month and the previous two months can be calculated, and so on, to obtain the third characteristic data.

[0057] Step S230, input the feature data of multiple dimensions into the parameter prediction model in time series, determine the cluster target operation data within a second time range after the current time point, so that the cluster control platform performs corresponding operations on the distributed cluster according to the cluster target operation data to control the cluster operation.

[0058] In this example implementation, each feature data may be formed into an input vector in time series, and multiple feature data of multiple dimensions may form an input feature matrix. For example, the input feature matrix X may include each feature data (including the first feature data, the second feature data, and the third feature data) at the 1st to n-1st moments. The input feature matrix Among them, n represents the time series label, m represents the total number of input features, and x 1n-1 Represents the eigenvalue of the first eigenvariable at the n-1th moment.

[0059] In this example implementation, the parameter prediction model may include a recurrent neural network and a wrapper layer, the recurrent neural network may include a long short-term memory neural network (Long Short-Term Memory, LSTM), the recurrent neural network may include multiple LSTM hidden layer units, the wrapper layer may be a TimeDistributed layer using Keras, and the wrapper layer may apply a layer (such as a fully connected layer) to each time slice of the input. The cluster target operation data may include cluster target resource parameters (CPU, memory, and disk resources required for operation) and cluster target operation parameters (task operator parallelism), which are also output data Y of the parameter prediction model.

[0060] In some embodiments, Figure 4 As shown, the parameter prediction model may include an input layer, a hidden layer, and an output layer. The prediction process of cluster target operation parameters can be implemented through the following steps.

[0061] The feature data of multiple dimensions are input into the recurrent neural network in time series to obtain the hidden layer output.

[0062] In this example implementation, the recurrent neural network may include an input layer and a plurality of LSTM units, such as Figure 4As shown, the recurrent neural network includes two layers of LSTM units. The input data (x0, x1, ...xm) is input into the prediction model through the input layer, and processed by the stacked hidden layer of two LSTM units to obtain the hidden layer output (h0, h1, ...hm). The internal structure of LSTM controls the transmission system through the gated state, remembers the information that needs to be remembered for a long time, and forgets the unimportant information. LSTM can include input gate, forget gate and output gate. The forget gate is used to control whether the information in the memory cell state from the previous moment needs to be forgotten, and can be used to delete useless information; the input gate is used to update the memory cell state information at the current moment, and the output gate is used to determine the output of the memory cell at the current moment.

[0063] The same layer processing is performed on each time step of the hidden layer output using a wrapper layer to obtain the cluster target resource parameters and cluster target operation parameters.

[0064] In this example implementation, the wrapper layer may include a fully connected process in a shared weight mode (with the same weight) for each input time step. The processing result may be processed using a softmax activation function to obtain the final model output (y0, y1, ...yn), i.e., the cluster target resource parameters and the cluster target operation parameters.

[0065] In some embodiments, the parameter prediction model is obtained by training a recurrent neural network with a wrapper layer using cluster historical data, and may specifically include the following steps: using Keras to create a recurrent neural network model with a wrapper layer; inputting time window parameters and feature quantity parameters into the network model; obtaining historical cluster data including the historical time period of the target event, and performing multi-dimensional feature extraction on the historical cluster data; inputting the extracted features into the recurrent neural network model to train the network model according to the time window parameters and feature quantity parameters.

[0066] In this example implementation, a recurrent neural network model with a wrapper layer can be constructed using Keras. The wrapper layer TimeDistributed (time domain sharing function in keras) provided in Keras can be used to encapsulate LSTM. Keras is an open source artificial neural network library written in Python that can be used as a high-level application programming interface for the design, debugging, evaluation, application, and visualization of deep learning models. The time window parameter is used to determine the time slice length of the model, and the feature number parameter is used to determine the number of features of the input data.

[0067] In this example implementation, the historical time period can be determined according to the specific situation, and the historical cluster data (such as traffic, business order volume, real-time throughput, cluster resources and resource usage, etc.) of the corresponding period in recent years can be selected. The network model forward calculates the output of each neuron, reversely calculates the error term of each neuron, calculates the gradient of each weight based on the error term, and updates the weight based on reverse gradient propagation.

[0068] In some embodiments, the method further includes: pushing the cluster target operation data to a corresponding database, so as to obtain the latest cluster target operation data by calling a data interface of the database, and adjusting the operation data of the distributed cluster according to the latest cluster target operation data.

[0069] In this example implementation, the prediction results can be pushed and stored in the corresponding database in real time, and the cluster control platform can obtain the latest prediction results through the external interface service, and adjust the cluster operation parameters according to the prediction results. For example, the cluster control platform can compare the current cluster operation data with the prediction results, and when the two are inconsistent, expand or reduce the cluster or adjust the operator parallelism.

[0070] For example, the implementation process of the distributed cluster operation control method disclosed in the present invention is as follows: Figure 5 As shown, it can be implemented by the following steps.

[0071] The first step is to obtain historical cluster data including the historical time period of the target event from the cluster original database of the distributed cluster, and perform multi-dimensional feature extraction on the historical cluster data;

[0072] In the second step, the extracted historical features are input into the constructed recurrent neural network model with a wrapper layer to train the network model according to the time window parameters and the feature quantity parameters to obtain the parameter prediction model.

[0073] The third step is to obtain, in response to the triggering of the scheduled task, the cluster processing data and cluster running data of the distributed cluster within a first time range before the current time point from the cluster original database.

[0074] The fourth step is to perform multi-dimensional feature extraction on the cluster processing data and cluster operation data respectively to obtain the feature data corresponding to each dimension.

[0075] The fifth step is to input the feature data of multiple dimensions into the parameter prediction model in time series to determine the cluster target operation data within the second time range after the current time point.

[0076] The sixth step is to push the cluster target operation data to the prediction result database corresponding to the model.

[0077] In the seventh step, the cluster control platform calls the latest prediction results from the prediction result database through the data interface.

[0078] In the eighth step, the cluster control platform adjusts the cluster operation data according to the latest prediction results.

[0079] The cluster target operation data prediction process in the above embodiment can be implemented in an offline manner or in an online manner. When implemented in an offline manner, the data involved in the process can be offline data.

[0080] This disclosure aims at the fact that during the periodic promotion of existing e-commerce platforms, the system computing performance cannot meet the peak consumption performance during the promotion and does not generate message backlogs. The operation and maintenance personnel are required to conduct multiple tentative stress tests on the real-time computing tasks of the distributed cluster to ensure that the requirements of the cluster resources and task operator parallelism of the promotion consumption performance are met without wasting computing resources. After the promotion is over, the stress test is adjusted again according to the actual consumption performance. The above process will cause problems such as poor system stability, poor adjustment timeliness and waste of a lot of manpower.

[0081] On the one hand, the present invention uses the wrapper layer to encapsulate LSTM, realizing the many-to-many prediction process of multi-dimensional cluster features to multiple operating parameters, solving the problem of poor accuracy and low efficiency of single-dimensional prediction, while improving the timeliness of operating data adjustment, improving the stability of system operation and saving manpower. On the other hand, by extracting multi-dimensional features as model input data, the fusion process of multi-dimensional data features is realized, further ensuring the accuracy of prediction.

[0082] Furthermore, in this exemplary embodiment, a distributed cluster operation control device 600 is also provided. The distributed cluster operation control device 600 can be applied to a server. Figure 6 As shown, the operation control device 600 of the distributed cluster may include: an acquisition module 610, a feature extraction module 620, and a control module 630. The acquisition module 610 is used to acquire cluster processing data and cluster operation data of the distributed cluster within a first time range before the current time point; the feature extraction module 620 is used to perform multi-dimensional feature extraction on the cluster processing data and the cluster operation data respectively to obtain feature data corresponding to each dimension; the control module 630 is used to input the feature data of multiple dimensions into a parameter prediction model in a time series, and determine the cluster target operation data within a second time range after the current time point, so that the cluster control platform performs corresponding operations on the distributed cluster according to the cluster target operation data to control the cluster operation; the cluster target operation data includes multiple cluster operation parameters, and the parameter prediction model is obtained by training a recurrent neural network with an encapsulator layer using cluster historical data.

[0083] In an exemplary embodiment of the present disclosure, the device 600 may also include a determination module, which may be used to: in response to the second time range including the first target deadline, determine that the first time range includes the second target deadline; the first target deadline and the second target deadline are respectively time ranges that include the target event.

[0084] In an exemplary embodiment of the present disclosure, the feature extraction module 620 can also be used to: perform index processing on the cluster processing data and the cluster operation data according to the first time dimension to obtain first feature data; perform discrete feature extraction on the first feature data to obtain second feature data; perform trend analysis on the cluster processing data and the cluster operation data according to multiple second time dimensions to obtain third feature data corresponding to each second time dimension.

[0085] In an exemplary embodiment of the present disclosure, the cluster target operation data includes cluster target resource parameters and cluster target operation parameters, and the control module 630 can also be used to: input feature data of multiple dimensions into a recurrent neural network in a time series to obtain a hidden layer output; use a wrapper layer to perform the same layer processing on each time step of the hidden layer output to obtain cluster target resource parameters and cluster target operation parameters.

[0086] In an exemplary embodiment of the present disclosure, the device 600 also includes a training module, which can be used to: create a recurrent neural network model with a wrapper layer using Keras; input time window parameters and feature quantity parameters into the network model; obtain historical cluster data of a historical time period including a target event, and perform multi-dimensional feature extraction on the historical cluster data; input the extracted features into the network model to train the network model according to the time window parameters and feature quantity parameters.

[0087] In an exemplary embodiment of the present disclosure, the device 600 also includes a push module, which can be used to: push the cluster target operation data to the corresponding database, so as to obtain the latest cluster target operation data by calling the data interface of the database, and adjust the operation data of the distributed cluster according to the latest cluster target operation data.

[0088] In an exemplary embodiment of the present disclosure, the acquisition module 610 can also be used to: in response to the triggering of a scheduled task, or in response to the data processing volume of the monitoring cluster being greater than a preset threshold, obtain the cluster processing data and cluster operation data of the distributed cluster within a first time range before and at the current time point.

[0089] The specific details of each module or unit in the above-mentioned distributed cluster operation control device have been described in detail in the corresponding distributed cluster operation control method, so they are not repeated here.

[0090] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method in the following embodiment. For example, the electronic device may implement Figure 2 to Figure 5 The various steps shown, etc.

[0091] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0092] Refer to the following Figure 7 hereinafter describes an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0093] like Figure 7As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: the at least one processing unit 710, the at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740.

[0094] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0095] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0096] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0097] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0098] The electronic device 700 may also communicate with one or more external devices 770 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RA identification systems, tape drives, and data backup storage systems, etc.

[0099] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0100] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0101] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in the specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc., which shall all be considered as part of the present disclosure.

[0102] It should be understood that the disclosure disclosed and defined in this specification extends to all alternative combinations of two or more individual features mentioned or evident in the text and / or the accompanying drawings. All these different combinations constitute multiple alternative aspects of the disclosure. The embodiments of this specification illustrate the best mode known for implementing the disclosure and will enable those skilled in the art to utilize the disclosure.

Claims

1. A distributed cluster operation control method, characterized in that: include: Obtain cluster processing data and cluster operation data of the distributed cluster within the first time range before the current time point; Performing multi-dimensional feature extraction on the cluster processing data and the cluster operation data respectively to obtain feature data corresponding to each dimension; The feature data of multiple dimensions are input into the parameter prediction model in time series to determine the cluster target operation data within a second time range after the current time point, so that the cluster control platform can perform corresponding operations on the distributed cluster according to the cluster target operation data to control the cluster operation; the cluster target operation data includes multiple cluster operation parameters, and the parameter prediction model is obtained by training a recurrent neural network with a wrapper layer using cluster historical data.

2. The operation control method of a distributed cluster according to claim 1, characterized in that: The method further comprises: In response to the second time range including the first target deadline, it is determined that the first time range includes the second target deadline; the first target deadline and the second target deadline are time ranges that include a target event respectively.

3. The operation control method of a distributed cluster according to claim 1, characterized in that: The multi-dimensional feature extraction of the data includes: According to the first time dimension, the cluster processing data and the cluster operation data are respectively indexed to obtain first feature data; Performing discreteness feature extraction on the first feature data to obtain second feature data; According to the multiple second time dimensions, trend analysis is performed on the cluster processing data and the cluster operation data respectively to obtain third characteristic data corresponding to each second time dimension.

4. The operation control method of a distributed cluster according to claim 3, characterized in that: The cluster target operation data includes cluster target resource parameters and cluster target operation parameters, and the cluster target operation data within a second time range after the current time point is determined, including: Inputting feature data of multiple dimensions into the recurrent neural network in time series to obtain hidden layer output; The same layer processing is performed on each time step of the hidden layer output by using a wrapper layer to obtain cluster target resource parameters and cluster target operation parameters.

5. The operation control method of a distributed cluster according to claim 4, characterized in that: The training process of the parameter prediction model includes: Create a recurrent neural network model with wrapper layers using Keras; Inputting a time window parameter and a feature quantity parameter into the network model; Acquire historical cluster data of a historical time period including the target event, and perform multi-dimensional feature extraction on the historical cluster data; The extracted features are input into the network model to train the network model according to the time window parameter and the feature quantity parameter.

6. The operation control method of a distributed cluster according to claim 1, characterized in that: The method further comprises: The cluster target operation data is pushed to a corresponding database, so as to obtain the latest cluster target operation data by calling a data interface of the database, and the operation data of the distributed cluster is adjusted according to the latest cluster target operation data.

7. The operation control method of a distributed cluster according to claim 1, characterized in that: The obtaining of cluster processing data and cluster operation data includes: In response to the triggering of the scheduled task, or in response to the data processing volume of the monitoring cluster being greater than a preset threshold, cluster processing data and cluster operation data of the distributed cluster within a first time range before and at a current time point are obtained.

8. A distributed cluster operation control device, characterized in that: The device comprises: An acquisition module, used to acquire cluster processing data and cluster operation data of a distributed cluster within a first time range before a current time point; A feature extraction module, used to perform multi-dimensional feature extraction on the cluster processing data and the cluster operation data respectively, to obtain feature data corresponding to each dimension; A control module is used to input feature data of multiple dimensions into a parameter prediction model in time series, and determine cluster target operation data within a second time range after a current time point, so that a cluster control platform can perform corresponding operations on the distributed cluster according to the cluster target operation data to control cluster operation; the cluster target operation data includes multiple cluster operation parameters, and the parameter prediction model is obtained by training a recurrent neural network with a wrapper layer using cluster historical data.

9. An electronic device, characterized in that: include: one or more processors; as well as A storage device, used to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Power station real-time control and intelligent calculation divide-conquer-collaboration system and method

    CN119966078A