Computing resource adjustment method for data analysis engine

Abstract

The present application provides a method for adjusting computing power resources of a data analysis engine, and relates to the field of cloud computing technology. The method includes obtaining a computing power adjustment configuration file of the data analysis engine to be monitored; determining whether the data analysis engine meets the preset computing power expansion and contraction trigger conditions based on the configuration operation parameter information; if the data analysis engine meets the preset computing power expansion and contraction trigger conditions, then obtaining computing power adjustment information for the data analysis engine; the computing power adjustment information includes the number of instances to be expanded or contracted; and executing computing power adjustment operations for the data analysis engine based on the number of instances to be expanded or contracted. The method of the present application can automatically monitor fluctuations in computing power requests, and adaptively adjust computing power resources based on the fluctuations, thereby making better use of computing power resources.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and more specifically, to a method for adjusting computing resources of a data analysis engine. Background Art

[0002] A data analysis engine is a data analysis tool that can collect, store, calculate, and query and analyze huge amounts of data, and make data more intelligent through deep learning technology and data modeling technology.

[0003] The operation of the data analysis engine depends on its computing resources. At present, some adjustment methods can be used to adjust the computing resources, but how to determine the timing of adjusting the computing resources of the data analysis engine is still a problem that needs to be solved urgently. Summary of the invention

[0004] In view of the deficiencies in the above-mentioned prior art, the present application provides a method for adjusting computing resources of a data analysis engine so as to solve the problems existing in the prior art.

[0005] The technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for adjusting computing resources of a data analysis engine, including:

[0007] Obtaining a computing power adjustment configuration file of a data analysis engine to be monitored, wherein the computing power adjustment configuration file at least includes: configuration operation parameter information of the data analysis engine;

[0008] Determine whether the data analysis engine meets the preset computing power expansion and contraction trigger conditions according to the configuration operation parameter information;

[0009] If the data analysis engine meets the preset computing power expansion and contraction triggering condition, computing power adjustment information for the data analysis engine is obtained, and the computing power adjustment information includes: the number of instances to be expanded or reduced;

[0010] According to the number of instances to be expanded or reduced, a computing power adjustment operation is performed on the data analysis engine.

[0011] In a second aspect, an embodiment of the present application provides a computing resource adjustment device for a data analysis engine, comprising:

[0012] A first acquisition module is used to acquire a computing power adjustment configuration file of a data analysis engine to be monitored, wherein the computing power adjustment configuration file at least includes: configuration operation parameter information of the data analysis engine;

[0013] A determination module, used to determine whether the data analysis engine meets the preset computing power expansion and contraction triggering conditions according to the configuration operation parameter information;

[0014] A second acquisition module is configured to acquire computing power adjustment information for the data analysis engine if the data analysis engine meets the preset computing power expansion and contraction triggering condition, wherein the computing power adjustment information includes: the number of instances to be expanded or contracted;

[0015] An execution module is used to perform a computing power adjustment operation on the data analysis engine according to the number of instances to be expanded or reduced.

[0016] In the third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the program instructions to implement the computing power resource adjustment method of the data analysis engine described in the above embodiment.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the computing power resource adjustment method of the data analysis engine described in the above embodiment is implemented.

[0018] The beneficial effects of the present application are as follows: the present application provides a method for adjusting computing power resources of a data analysis engine, including obtaining a computing power adjustment configuration file of a data analysis engine to be monitored, the computing power adjustment configuration file at least including: configuration and operation parameter information of the data analysis engine; determining whether the data analysis engine meets the preset computing power expansion and contraction triggering conditions according to the configuration and operation parameter information; if the data analysis engine meets the preset computing power expansion and contraction triggering conditions, then obtaining computing power adjustment information for the data analysis engine, the computing power adjustment information including: the number of instances to be expanded and contracted; and executing computing power adjustment operations for the data analysis engine according to the number of instances to be expanded and contracted. The method of the present application can determine whether the data analysis engine meets the preset computing power expansion and contraction triggering conditions, automatically monitor the fluctuation of computing power requests, and adaptively adjust computing power resources according to the fluctuations, thereby making better use of computing power resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1One of the flowcharts of the method for adjusting computing resources of the data analysis engine provided in the embodiment of the present application;

[0021] Figure 2 A second flowchart of a method for adjusting computing resources of a data analysis engine provided in an embodiment of the present application;

[0022] Figure 3 A third flowchart of a method for adjusting computing resources of a data analysis engine provided in an embodiment of the present application;

[0023] Figure 4 A fourth flowchart of a method for adjusting computing resources of a data analysis engine provided in an embodiment of the present application;

[0024] Figure 5 A fifth flowchart of a method for adjusting computing resources of a data analysis engine provided in an embodiment of the present application;

[0025] Figure 6 Flow chart six of the method for adjusting computing resources of a data analysis engine provided in an embodiment of the present application;

[0026] Figure 7 Flow chart of the method for adjusting computing resources of a data analysis engine provided in an embodiment of the present application, No. 7;

[0027] Figure 8 8. Flow chart of the method for adjusting computing resources of a data analysis engine provided in an embodiment of the present application;

[0028] Fig. 9 A ninth flowchart of a method for adjusting computing resources of a data analysis engine provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of the computing power resource adjustment device of the data analysis engine provided in this application;

[0030] Fig.11 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0032] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0033] In addition, the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0035] A data analysis engine is a data analysis tool that provides data collection, data storage, data calculation, and query analysis services, such as online analytical processing (OLAP) databases. Common OLAP databases include StarRocks, Doris, and Clickhouse. Currently, the adjustment of computing resources of data analysis engines depends on the maintenance of staff, which has high operation and maintenance costs and cannot quickly and accurately determine the timing of adjusting computing resources.

[0036] In response to the above problems, an embodiment of the present application provides a method for adjusting computing resources of a data analysis engine. The method can be generated by any electronic device with computing and processing capabilities. The electronic device can be, for example, a terminal-oriented computer device or a back-end server.

[0037] It should be noted that the method provided in this application has a premise that the data analysis engine must support the storage-computing separation architecture, that is, the architecture that supports the decoupling and separation of storage and computing. Cloud storage or storage nodes are responsible for storage, and computing nodes are responsible for computing. Not all OLAP databases support this architecture, so the method described in this application is only applicable to some data analysis engines. This article takes the StarRocks data analysis engine as an example, and the methods described are all applicable to this data analysis engine. For data analysis engines that do not have a storage-computing separation architecture, their data is stored locally. Under this architecture, the data analysis engine will have many restrictions on adjusting computing resources, which is not conducive to the expansion and contraction of computing resources. This application will not elaborate on this.

[0038] The following is a specific example description of the computing power resource adjustment method of the data analysis engine provided in this application through multiple examples in combination with the accompanying drawings.

[0039] Figure 1 One of the flow charts of the method for adjusting computing resources of the data analysis engine provided in the embodiment of the present application is as follows: Figure 1 As shown, the method includes:

[0040] S101. Obtain a computing power adjustment configuration file of a data analysis engine to be monitored.

[0041] In this embodiment, the computing power adjustment configuration file includes at least configuration operating parameter information of the data analysis engine, such as the network bandwidth of the server, the maximum and minimum number of data analysis engine instances, and other information. This information can be obtained from the configuration file of the data analysis engine to be monitored.

[0042] S102. Determine whether the data analysis engine meets the preset computing power expansion and contraction trigger conditions based on the configuration operation parameter information.

[0043] Based on the obtained configuration and operating parameter information, a task thread pool is established. The thread pool is used to execute monitoring tasks for the data analysis engine to be monitored, so as to determine whether the data analysis engine meets the preset computing power expansion and contraction trigger conditions. The monitoring tasks include but are not limited to the monitoring tasks of the data analysis engine operating environment resources and the data analysis engine's own computing resources. The monitoring tasks can be executed every 10 seconds. The preset computing power expansion and contraction trigger conditions are used to indicate whether the data analysis engine needs to be expanded or reduced.

[0044] S103. If the data analysis engine meets the preset computing power expansion and contraction triggering conditions, computing power adjustment information for the data analysis engine is obtained.

[0045] The computing power adjustment information includes the number of instances to be expanded or reduced. When the data analysis engine meets the preset computing power expansion or reduction triggering conditions, the computing power adjustment information for the data analysis engine is obtained. For example, when the data analysis engine meets the preset computing power expansion triggering conditions, the number of instances to be expanded for the data analysis engine is obtained; when the data analysis engine meets the preset computing power reduction triggering conditions, the number of instances to be reduced for the data analysis engine is obtained.

[0046] S104. Execute a computing power adjustment operation for the data analysis engine according to the number of instances to be expanded or reduced.

[0047] In this embodiment, the number of instances to be expanded or reduced includes the number of instances to be expanded, or the number of instances to be reduced, and the computing power adjustment configuration file also includes information of servers to be deployed.

[0048] Among them, the information of the server to be deployed is pre-configured. For example, the server information available for deployment of the instance of the data analysis engine to be monitored can be configured in the configuration file of the data analysis engine to be monitored. How to obtain the server information and usage rights will not be elaborated here. You can purchase the usage rights first, or apply for them dynamically when using the internal maintenance server cluster.

[0049] S104 performs a computing power adjustment operation for the data analysis engine according to the number of instances to be expanded or reduced, specifically including:

[0050] 1. When capacity expansion is required, obtain the usage rights of the target servers to be deployed with the corresponding number of instances based on the number of instances to be expanded and the information of the servers to be deployed, and then upload the deployment installation package of the data analysis engine being monitored to the target servers to be deployed, so as to start the running instances of the corresponding number of instances on the target servers to be deployed, and add the information of the running instances of the corresponding number of instances to the service cluster of the data analysis engine.

[0051] Specifically, when the data analysis engine expansion operation is currently required, the corresponding server information is obtained from the configuration file according to the number of data analysis engine instances to be expanded, or, if the server usage permission is obtained by dynamically applying for server resources, the requested server information is obtained after the application for usage permission is successful. The server information includes the IP address of the server, the port number of the SSH protocol, and the account and password for logging into the server.

[0052] After obtaining the server information, log in to the server through the server information and ssh protocol, upload the pre-prepared data analysis engine deployment installation package to the server, and decompress the installation package, modify the installation package configuration file, start the data analysis engine instance, etc. Different data analysis engine instance configuration file modification content and startup methods are different, so I will not elaborate on them here. After the data analysis engine instances that need to be expanded are started, these instances are added to the data analysis engine service cluster. These newly expanded data analysis engine instances can work together with the original data analysis engine instances to perform analysis tasks.

[0053] Taking the StarRocks data analysis engine as an example, a new data analysis engine instance can be added by executing the command ALTER SYSTEM ADD COMPUTE NODE to the data analysis engine service cluster:<cn_address> :<heartbeat_service_port> "Join the original data analysis engine service cluster. After joining, you can also record the newly expanded data analysis engine instance information, mainly including the deployed server information. At this point, the expansion operation of the data analysis engine has been completed.

[0054] 2. When scaling down is required, remove the information of deployed instances of the same number of instances to be scaled down from the service cluster of the data analysis engine being monitored, then establish a connection with the server corresponding to the deployed instance, and through the connection, remove the deployment file data of the deployed instance on the corresponding server.

[0055] Specifically, according to the number of instances that need to be scaled down, the corresponding data analysis engine instance information is selected, a connection is made to the server described by the instance information via the SSH protocol, and the data analysis engine instance is removed from the data analysis engine service cluster.

[0056] Taking StarRocks as an example, you can execute the command ALTER SYSTEMDROP COMPUTE NODE to the data analysis engine service cluster:<cn_address> :<heartbeat_service_port> " , execute the data analysis engine stop command, stop the data analysis engine instance service running on the server, and remove the file data generated during the installation and deployment of the data analysis engine instance. At this point, the data analysis engine scaling down operation is complete.

[0057] In summary, this embodiment provides a computing power resource adjustment method for a data analysis engine, which can determine whether the data analysis engine meets the preset computing power expansion and contraction trigger conditions, automatically monitor the fluctuation of computing power requests, and adaptively adjust the computing power resources according to the fluctuation, thereby better utilizing the computing power resources.

[0058] In one embodiment, the computing power adjustment configuration file also includes configuration connection information, and the configuration connection information includes address information of the current deployment server and connection information of the current deployment server. Then, according to the configuration operation parameter information, determining whether the data analysis engine meets the preset computing power expansion and contraction triggering condition in S102 may include:

[0059] S201. According to the configuration connection information, execute the computing power resource monitoring operation for the data analysis engine to obtain the computing power resource monitoring data of the data analysis engine.

[0060] By performing a computing resource monitoring operation on the data analysis engine based on the IP address, security authentication information, and login information of the server where the data analysis engine is deployed, the computing resource monitoring data of the data analysis engine can be obtained. The computing resource monitoring data is used to indicate the computing resource usage of the data analysis engine.

[0061] S202. Based on the computing power resource monitoring data and the configuration operation parameter information, the computing power usage of the data analysis engine is evaluated to obtain the computing power evaluation level of the data analysis engine.

[0062] After obtaining the computing power resource monitoring data of the data analysis engine, the computing power occupancy of the data analysis engine is evaluated according to the computing power resource monitoring data and the configuration operation parameter information, and the computing power evaluation level of the data analysis engine can be obtained. The computing power evaluation level may include, for example, idle, good, and busy.

[0063] S203. Determine whether the data analysis engine meets preset computing power expansion and contraction trigger conditions based on the computing power evaluation level of the data analysis engine.

[0064] After obtaining the computing power evaluation level of the data analysis engine, it can be determined whether the data analysis engine meets the preset computing power expansion and contraction trigger conditions based on the computing power evaluation level.

[0065] In this embodiment, the computing power assessment level of the data analysis engine is determined based on the computing power resource monitoring data, so that the computing power assessment level is more in line with the actual situation of the data analysis engine to be monitored, thereby improving the accuracy of determining the timing of computing power resource adjustment.

[0066] In one embodiment, the computing power resource monitoring data may include operating environment resource data and computing resource data. In S201, the computing power resource monitoring operation for the data analysis engine is performed according to the configuration connection information to obtain the computing power resource monitoring data of the data analysis engine, which may include:

[0067] S301. Establish a connection instance with a currently deployed server according to configuration connection information, and execute an operating system command through the connection instance to obtain operating environment resource data of the currently deployed server.

[0068] The connection information of the currently deployed server in the configured connection information includes the IP address, security authentication information and server login information. The currently deployed server is connected according to the SSH protocol (a protocol for remote access to servers) and the server login information to establish a connection instance. Through this connection, the operating system commands can be executed to obtain the operating environment resource data of the currently deployed server, including the server's CPU and network IO usage.

[0069] S302. Establish a database connection with the data analysis engine according to the configuration connection information, and execute a structured query statement for the data analysis engine through the database connection to obtain computing resource data of the data analysis engine.

[0070] The computing resource data of the data analysis engine, that is, the usage of its own computing resources, can be queried and obtained by requesting the data analysis engine to execute SQL statements. There are two ways to execute SQL statements: one is to connect to the data analysis engine based on the JDBC driver and the basic connection information of the data analysis engine. Through the established JDBC connection, SQL query statements can be executed to obtain the query results of SQL statements; the other is that the data analysis engine also supports requesting the execution of SQL statements through the HTTP protocol to obtain the query results of SQL statements.

[0071] On this basis, the computing power occupancy evaluation of the data analysis engine is performed according to the computing power resource monitoring data and the configuration operation parameter information in S202 to obtain the computing power evaluation level of the data analysis engine, which may include:

[0072] S401. Perform an operating environment evaluation on the data analysis engine according to the operating environment resource data and the configuration operating parameter information to obtain an operating environment evaluation score of the data analysis engine.

[0073] The configuration operation parameter information includes the network bandwidth of the currently deployed server, and the operation environment resource data includes the processor idle rate of the currently deployed server and the network traffic data of the currently deployed server.

[0074] In step S401, the operating environment of the data analysis engine is evaluated according to the operating environment resource data and the configuration operating parameter information, so as to obtain the operating environment evaluation score of the data analysis engine, which specifically includes:

[0075] S501. According to the processor idle rate and the evaluation scores corresponding to a plurality of preset idle rate ranges, determine that the evaluation score corresponding to the target idle rate range where the processor idle rate is located is the first evaluation score of the data analysis engine.

[0076] By using the SSH protocol to execute the iostat command in the operating system of the server where the data analysis engine is located, you can obtain the current processor idle rate.

[0077] Then, based on the processor idle rate and the evaluation scores corresponding to the multiple preset idle rate ranges, the evaluation score corresponding to the target idle rate range where the processor idle rate is located is determined, including: 2 points when the total CPU idle rate is less than or equal to 20%, 1 point when greater than 20% and less than or equal to 40%, and no points when greater than 40%. Finally, the evaluation score can be used as the first evaluation score of the data analysis engine.

[0078] It should be noted that when the data analysis engine is deployed in cluster mode, there are multiple data analysis engine instances. In this case, the first evaluation score is obtained by summing up the first evaluation scores of different data analysis engine instances and taking the average value.

[0079] S502. Determine the network occupancy rate of the data analysis engine based on the network bandwidth and network traffic data, and determine the evaluation score corresponding to the target network occupancy rate range where the network occupancy rate is located as the second evaluation score of the data analysis engine based on the network occupancy rate and the evaluation scores corresponding to multiple preset network occupancy rate ranges.

[0080] By executing the sar command in the operating system of the server where the data analysis engine is located through the ssh protocol, you can obtain the current network traffic data. The network traffic data includes the number of bytes sent and received per second of the current network traffic. Then, combined with the server network bandwidth configured in the configuration file, the network uplink occupancy rate and network download occupancy rate can be calculated: Network uplink occupancy rate = network traffic bytes sent per second / (network uplink bandwidth / 8), network download occupancy rate = network traffic bytes downloaded per second / (network download bandwidth / 8).

[0081] The evaluation score corresponding to the target network occupancy range where the network occupancy rate is located is: 2 points when the network uplink occupancy rate is greater than or equal to 90%, 1 point when the network uplink occupancy rate is less than 90% and greater than or equal to 80%, and no points when it is less than 80%. 2 points when the network download occupancy rate is greater than or equal to 90%, 1 point when the network download occupancy rate is less than 90% and greater than or equal to 80%, and no points when it is less than 80%. Finally, the evaluation score can be used as the second evaluation score of the data analysis engine.

[0082] It should be noted that the network IO monitoring result score is the average of the network uplink occupancy score and the network download occupancy score. When there are multiple data analysis engine instances, the network IO monitoring result score is the average of the network IO monitoring scores of different data analysis engine instances.

[0083] In summary, the operating environment evaluation score can be obtained by summing the first evaluation score and the second evaluation score.

[0084] S402. Perform computing resource evaluation on the data analysis engine according to the computing resource data to obtain a computing resource evaluation score of the data analysis engine.

[0085] The computing resource data includes the number of analysis tasks to be executed by the data analysis engine and the execution time of the analysis tasks being executed. The computing resource data of the data analysis engine itself can be obtained by querying the data of the task execution queue statistics table of the data analysis engine itself. Taking starRocks as an example, you can query the currently executing analysis tasks by executing the SHOW PROC ' / current_queries' sql statement, and query the details of the analysis tasks currently in the queue waiting to be executed by executing the SHOW PROCESSLIST sql statement.

[0086] S402, performing computing resource evaluation on the data analysis engine according to the computing resource data to obtain a computing resource evaluation score of the data analysis engine, may include:

[0087] S601. According to the number of analysis tasks to be executed and the evaluation scores corresponding to multiple preset task number ranges, determine that the evaluation score corresponding to the target task number range within which the number of analysis tasks to be executed is located is the third evaluation score of the data analysis engine.

[0088] By requesting the data analysis engine to execute the SHOW PROCESSLIST statement, the number of analysis tasks currently waiting to be executed can be queried, and the evaluation score corresponding to the target task number range of the number of analysis tasks to be executed is determined as the third evaluation score of the data analysis engine, including: if the number of analysis tasks still in the queue is greater than 100, 6 points are obtained; when the number of analysis tasks still in the queue is less than 100 and greater than or equal to 80, 4 points are obtained; when the number of analysis tasks still in the queue is less than 80 and greater than or equal to 50, 2 points are obtained. When the number of analysis tasks still in the queue is less than 50 and greater than or equal to 20, 1 point is obtained.

[0089] S602: According to the execution time of the analysis tasks being executed, determine the number of analysis tasks whose execution time is greater than or equal to a preset execution time threshold as the target task number.

[0090] By requesting the data analysis engine to execute the SHOW PROC ' / current_queries' statement, you can query the details of the currently executing analysis tasks, including the execution time of all the tasks being executed. The number of tasks that have been executed for more than 30 seconds is taken as the target number of tasks. For example, if the number of tasks that have been executed for more than 30 seconds is 10, the target number of tasks is 10.

[0091] S603. Determine a fourth evaluation score of the data analysis engine according to the number of target tasks.

[0092] 0.5 points are awarded for each target task. For example, if there are 10 tasks that take more than 30 seconds to execute, the fourth evaluation score of the data analysis engine is 0.5×10=5 points.

[0093] The computing resource evaluation score may be obtained by summing the third evaluation score and the fourth evaluation score.

[0094] S403. Determine the total evaluation score of the data analysis engine according to the operating environment evaluation score and the computing resource evaluation score.

[0095] After obtaining the operating environment evaluation score and the computing resource evaluation score, add the two together to get the total evaluation score of the data analysis engine.

[0096] S404. Obtain the computing power evaluation level of the data analysis engine according to the total evaluation score of the data analysis engine.

[0097] Finally, the computing power evaluation level of the data analysis engine is obtained based on the total evaluation score of the data analysis engine. For example, when the total evaluation score is greater than or equal to 10 points, the computing power evaluation level is busy, when the total evaluation score is less than 10 points and greater than or equal to 4 points, the computing power evaluation level is good, and when the total evaluation score is less than 6 points, the computing power evaluation level is idle.

[0098] In one embodiment, determining whether the data analysis engine meets the preset computing power expansion and contraction triggering condition according to the computing power evaluation level of the data analysis engine in S203 may include:

[0099] S701. Determine a first total number of times that the computing power evaluation level of the data analysis engine is a first resource usage level within a first preset evaluation number of times.

[0100] The first resource usage level is used to indicate that resources are busy, that is, the computing power evaluation level is busy. The first preset evaluation number may be, for example, 10 times. For example, when 6 of the computing power evaluation levels obtained in the last 10 evaluations are busy, the first total number is determined to be 6 times.

[0101] S702: If the first total number of times is greater than or equal to the first number threshold, it is determined that the data analysis engine meets the preset computing power expansion trigger condition.

[0102] The first number threshold is a preset value, for example, it can be 5 times. When the first total number is greater than or equal to the first number threshold, it is determined that the current computing power resources are obviously insufficient, and the data analysis engine meets the preset computing power expansion trigger conditions, and computing power resources need to be expanded.

[0103] In this case, the number of instances to be scaled up or down includes the number of instances to be expanded, and the step of obtaining the computing power adjustment information for the data analysis engine in S103 includes:

[0104] S703. Determine the sum of the computing resource evaluation scores of the data analysis engines according to the computing resource evaluation scores of the data analysis engines within a first preset number of evaluations.

[0105] The computing resource evaluation scores of the data analysis engine obtained nearly 10 times are summed to obtain the sum of the computing resource evaluation scores of the data analysis engine. For example, the computing resource evaluation scores of the data analysis engine obtained nearly 10 times are 11, 10, 11, 12, 13, 15, 8, 6, 3, 9, there are 6 busy times, and the sum of the scores is 98.

[0106] S704: Determine the number of instances to be expanded according to the sum of the computing resource evaluation scores.

[0107] The number of instances to be expanded can be determined based on the sum of the computing resource evaluation scores, including: if the sum of the computing resource evaluation scores of the past ten times is greater than or equal to 90, then the capacity will be expanded to add 2 new data analysis engine instances; if the sum of the computing resource evaluation scores of the past ten times is less than 90 and greater than or equal to 60, then the capacity will be expanded to add 1 new data analysis engine instance. It should be noted that if the current number of data analysis engine instances reaches the maximum value described in the configuration file, no further expansion will be performed.

[0108] For example, when the computing resource evaluation scores of the data analysis engine obtained in the last 10 times are 11, 10, 11, 12, 13, 15, 8, 6, 3, and 9, there are 6 busy times, and the sum of the scores is 98, the capacity is expanded to add 2 new data analysis engine instances.

[0109] The step of determining whether the data analysis engine meets a preset computing power expansion and contraction trigger condition according to the computing power evaluation level of the data analysis engine further includes:

[0110] S801. Determine a second total number of times that the computing power evaluation level of the data analysis engine is a second resource usage level within a second preset evaluation number of times.

[0111] The second preset evaluation number is greater than the first preset evaluation number, and the second resource usage level is used to indicate that there is idle resource, that is, the computing power evaluation level is idle. The second preset evaluation number can be, for example, 30 times. For example, when 20 of the computing power evaluation levels obtained from the nearly 30 evaluations are idle, the second total number is determined to be 20 times.

[0112] S802. Determine the third total number of times that the computing power evaluation level of the data analysis engine within the second preset evaluation number is the second resource utilization level or the third resource utilization level.

[0113] The third resource usage level is between the second resource usage level and the first resource usage level, indicating that the resource usage status is good, that is, the computing power evaluation level is good. For example, when the computing power evaluation levels obtained in the last 30 evaluations are 20 idle, 7 good, and 3 busy, the third total number is determined to be 20+7=27 times.

[0114] S803. If the second total number is greater than or equal to the second number threshold, and the third total number is greater than or equal to the third number threshold, it is determined that the data analysis engine meets the preset computing power reduction trigger condition.

[0115] The third number threshold is greater than the second number threshold, and the second number threshold is greater than the first number threshold.

[0116] Taking the second number threshold of 18 times and the third number threshold of 25 times as an example, when the second total number is 20 times and the third total number is 27 times, it is determined that the current computing power resources are obviously surplus, and the data analysis engine meets the preset computing power reduction trigger conditions, and computing power resources need to be reduced.

[0117] In this case, the number of instances to be expanded or reduced includes the number of instances to be reduced, and the step of obtaining the computing power adjustment information for the data analysis engine in S103 includes:

[0118] S804: Determine the preset minimum number of instances as the number of instances to be scaled down.

[0119] The preset minimum number of instances is 1, that is, each shrinking only reduces 1 data analysis engine instance. If the current number of data analysis engine instances reaches the minimum value described in the configuration file, no further shrinking will be performed.

[0120] In some scenarios where computing power demand fluctuates regularly, the use of computing power resources is predictable, and capacity expansion and contraction tasks can be performed regularly based on historical computing power usage. For example, in scenarios such as the Double Eleven shopping spree in the e-commerce industry and holiday travel in the transportation industry, the computing power resources that can accommodate normal business requests will become particularly tight at this time, and it is urgent to perform capacity expansion tasks to increase computing power resources.

[0121] In this case, the configuration operation parameter information includes a preset scaling plan, and the preset scaling plan includes: a target time period for scaling, and a preset number of scaling instances.

[0122] Determining whether the data analysis engine meets the preset computing power expansion and contraction triggering conditions according to the configuration operation parameter information in S102 may include:

[0123] S901: Determine whether the current time is in the target time period.

[0124] Determine the target time period for scaling tasks in advance based on historical computing power usage, such as the Double Eleven Shopping Festival, and then determine whether the current time is in the target time period.

[0125] S902: If the current time is in the target time period, determine whether the data analysis engine meets the preset computing power expansion and contraction triggering conditions.

[0126] The target time period includes an expansion target time period corresponding to the expansion task and a contraction target time period corresponding to the contraction task.

[0127] If the current time is in the target time period, it is determined that the data analysis engine meets the preset computing power expansion and contraction trigger conditions, including: if the target time period is the expansion target time period, it is determined that the data analysis engine meets the preset computing power expansion trigger conditions; if the target time period is the contraction target time period, it is determined that the data analysis engine meets the preset computing power contraction trigger conditions.

[0128] In this case, the step of obtaining the computing power adjustment information for the data analysis engine in S103 includes: S903, obtaining the preset number of scaling instances as the number of instances to be scaled. The preset number of scaling instances can be estimated based on historical data. Taking Double Eleven as an example, the preset number of scaling instances can be determined based on historical Double Eleven computing power resource usage data.

[0129] In this embodiment, by regularly scaling up and down, computing power requirements can be better adapted, the environment can be preheated, more cache space can be freed up for data analysis tasks, and better performance and user experience can be provided.

[0130] In summary, this application provides a method for adjusting computing resources of a data analysis engine, which has the following advantages:

[0131] 1. It can automatically monitor the fluctuation of computing power requests. When it detects that computing power resources are being preempted and computing power is insufficient, it will adaptively expand computing power according to the current computing power usage. Or, when it detects that computing power resources are in excess, it will adaptively reduce computing power.

[0132] 2. Better utilization of hardware resources. While saving hardware costs, the adaptive growth of computing power does not require too many people for maintenance. The operation is simple and can reduce the corresponding operation and maintenance conditions and costs.

[0133] Optionally, in the present application, after the computing power resources are expanded or reduced in capacity, the event information including the expansion or reduction operation may also be notified to the alarm person, and the specific workflow steps are as follows:

[0134] S1: Get the alarm person information. Before starting the computing power adaptive adjustment device, the user needs to configure the alarm person information to be notified in the configuration file, including the alarm person's mobile phone number and email address. After the computing power resources are expanded or reduced, the alarm person information is obtained from the configuration file.

[0135] S2: If the alarm person's mobile phone number is configured, the alarm person will be notified of the expansion and contraction warning information by sending a text message through the SMS service, including the time of the warning, the number of data analysis engine instances required to be expanded or contracted, the number of data analysis engine instances currently running, and the usage level of the recent assessment.

[0136] S3: If the alarm person's email address is configured, the alarm person will be notified of the S2 expansion and contraction warning information by email through the email service.

[0137] Optionally, the present application can also generate a computing power resource usage report for summarizing computing power resource monitoring results, count the peak values, valley values, and average values ​​of each monitoring result within the time range to be analyzed according to user requests, and give recommendations on the appropriate number of data analysis engine instances.

[0138] In addition, the computing resource usage report also supports the analysis of the hot periods of task execution within a certain period of time, helping users to recommend models for computing demand and predict future computing demand trends. It also provides theoretical support for users to plan the scheduled expansion and contraction of data analysis engine instance tasks. The specific workflow steps are as follows:

[0139] S11: Provide a computing resource usage report interface, through which users can input the time range parameters to be analyzed to obtain the usage report details of computing resources, and save the monitoring results after each monitoring task is executed. When receiving a user's request to obtain a computing resource usage report interface, filter the corresponding monitoring result data according to the input time range parameters, and group the filtered monitoring result data by time dimension.

[0140] If the time range of the input parameter is within 1 day, the hour is used as the time dimension. If the time range of the input parameter is greater than 1 day and less than 30 days, the day is used as the time dimension. If the time range of the input parameter is greater than 30 days, the month is used as the time dimension. Display the peak, valley, and average values ​​of the monitoring results of each group under the corresponding time dimension, such as displaying the peak, valley, and average values ​​of the CPU idle rate every day. In addition, according to this time dimension, the computing power resource monitoring result score of each group and the number of executions of each group of monitoring tasks are grouped and counted to calculate the computing power resource utilization ratio V. V=score / (number of executions of each group of monitoring tasks*10). When the ratio V is less than 0.1, it is recommended that the number of data analysis engine instances in the group be half of the number of data analysis engine instances during monitoring; when the ratio V is greater than or equal to 0.1 and less than 0.5, it is recommended that the number of data analysis engine instances in the group be three quarters of the number of data analysis engine instances during monitoring; when the ratio V is greater than or equal to 0.5 and less than 1, it is recommended that the number of data analysis engine instances in the group be consistent with the number of data analysis engine instances during monitoring; when the ratio V is greater than 1, it is recommended that the number of data analysis engine instances in the group be three half of the number of data analysis engine instances during monitoring. The calculated recommended number of instances may be a decimal, which is rounded up. The calculated peak, valley, average, and recommended number of data analysis engine instances of each monitoring result are returned to the user.

[0141] S12: Provide a data analysis hotspot period statistics interface to the outside world. Users can use this interface to input the time range parameters to be analyzed to obtain the details of the data analysis hotspots. When the computing power resource statistics report module receives a request for a data analysis hotspot period statistics interface, it filters the corresponding monitoring result data according to the input time range parameters, and groups the filtered monitoring result data by time dimension. The time dimension is consistent with the grouping logic of S1. Count the number of different computing power resource usage levels in each group, and calculate the busy rate ratio M, where M = number of busy levels / number of monitoring task executions. When the ratio M is greater than 0.6, it means that the period is a data analysis hotspot period, and the calculated data analysis hotspot period data is returned to the user.

[0142] The following is a corresponding explanation of the apparatus, device and storage medium for executing the computing resource adjustment method of the data analysis engine provided by any of the above embodiments of the present application. Its specific implementation process and the technical effects produced are the same as those of the corresponding method embodiments mentioned above. For the sake of brief description, for the parts not mentioned in the following embodiments, please refer to the corresponding content in the method embodiments.

[0143] like Fig.10 As shown, the present application provides a computing resource adjustment device for a data analysis engine, comprising:

[0144] The first acquisition module 10 is used to obtain a computing power adjustment configuration file of the data analysis engine to be monitored, and the computing power adjustment configuration file at least includes: configuration operation parameter information of the data analysis engine.

[0145] The determination module 20 is used to determine whether the data analysis engine meets the preset computing power expansion and contraction trigger conditions according to the configuration operation parameter information.

[0146] The second acquisition module 30 is used to obtain computing power adjustment information for the data analysis engine if the data analysis engine meets the preset computing power expansion and contraction triggering conditions. The computing power adjustment information includes: the number of instances to be expanded or reduced.

[0147] The execution module 40 is used to perform a computing power adjustment operation for the data analysis engine according to the number of instances to be expanded or reduced.

[0148] Optionally, the computing power adjustment configuration file also includes: configuration connection information; a determination module 20 is also used to perform a computing power resource monitoring operation on the data analysis engine according to the configuration connection information, and obtain computing power resource monitoring data of the data analysis engine; based on the computing power resource monitoring data and the configuration operation parameter information, perform a computing power occupancy assessment on the data analysis engine to obtain a computing power assessment level of the data analysis engine; based on the computing power assessment level of the data analysis engine, determine whether the data analysis engine meets the preset computing power expansion and contraction trigger conditions.

[0149] Optionally, the configuration connection information includes: the address information of the currently deployed server and the connection information of the currently deployed server; the determination module 20 is also used to establish a connection instance with the currently deployed server based on the configuration connection information, and execute operating system commands through the connection instance to obtain the operating environment resource data of the currently deployed server; establish a database connection with the data analysis engine based on the configuration connection information, and execute structured query statements for the data analysis engine through the database connection to obtain the computing resource data of the data analysis engine; the computing power resource monitoring data includes: operating environment resource data and computing resource data.

[0150] Optionally, based on the computing power resource monitoring data and the configuration operating parameter information; the determination module 20 is also used to perform an operating environment assessment on the data analysis engine based on the operating environment resource data and the configuration operating parameter information to obtain an operating environment assessment score of the data analysis engine; based on the computing resource data, perform a computing resource assessment on the data analysis engine to obtain a computing resource assessment score of the data analysis engine; based on the operating environment assessment score and the computing resource assessment score, determine a total assessment score of the data analysis engine; based on the total assessment score of the data analysis engine, obtain the computing power assessment level of the data analysis engine.

[0151] Optionally, the configuration operating parameter information includes: the network bandwidth of the currently deployed server, and the operating environment resource data includes: the processor idle rate of the currently deployed server, and the network traffic data of the currently deployed server; the determination module 20 is also used to determine that the evaluation score corresponding to the target idle rate range where the processor idle rate is located is the first evaluation score of the data analysis engine based on the processor idle rate and the evaluation scores corresponding to multiple preset idle rate ranges; determine the network occupancy rate of the data analysis engine based on the network bandwidth and network traffic data, and determine that the evaluation score corresponding to the target network occupancy rate range where the network occupancy rate is located is the second evaluation score of the data analysis engine based on the network occupancy rate and the evaluation scores corresponding to multiple preset network occupancy rate ranges; the operating environment evaluation score includes: a first evaluation score and a second evaluation score.

[0152] Optionally, the computing resource data includes: the number of analysis tasks to be executed, and the execution time of the analysis tasks being executed; the determination module 20 is also used to determine, based on the number of analysis tasks to be executed and the evaluation scores corresponding to multiple preset task number ranges, that the evaluation score corresponding to the target task number range within which the number of analysis tasks to be executed is located is the third evaluation score of the data analysis engine; based on the execution time of the analysis tasks being executed, determine the number of analysis tasks whose execution time is greater than or equal to the preset execution time threshold as the target task number; based on the target task number, determine the fourth evaluation score of the data analysis engine, and the computing resource evaluation score includes: the third evaluation score and the fourth evaluation score.

[0153] Optionally, the determination module 20 is further used to determine a first total number of times that the computing power evaluation level of the data analysis engine is a first resource usage level within a first preset evaluation number of times, where the first resource usage level is used to indicate that resources are busy; if the first total number of times is greater than or equal to a first number threshold, then determine whether the data analysis engine meets a preset computing power expansion trigger condition;

[0154] The number of instances to be expanded or reduced includes: the number of instances to be expanded; the second acquisition module 30 is also used to determine the sum of the computing resource evaluation scores of the data analysis engine based on the computing resource evaluation scores of the data analysis engine within the first preset evaluation times; determine the number of instances to be expanded based on the sum of the computing resource evaluation scores.

[0155] Optionally, the determination module 20 is also used to determine the second total number of times that the computing power evaluation level of the data analysis engine is the second resource usage level within a second preset evaluation number of times, wherein the second preset evaluation number of times is greater than the first preset evaluation number of times, and the second resource usage level is used to indicate that there are idle resources; determine the third total number of times that the computing power evaluation level of the data analysis engine is the second resource usage level or the third resource usage level within the second preset evaluation number of times, wherein the third resource usage level is between the second resource usage level and the first resource usage level; if the second total number of times is greater than or equal to the second number threshold, and the third total number of times is greater than or equal to the third number threshold, it is determined that the data analysis engine meets the preset computing power reduction trigger condition; wherein the third number threshold is greater than the second number threshold, and the second number threshold is greater than the first number threshold.

[0156] The number of instances to be expanded or reduced includes: the number of instances to be reduced; the second determination module 30 is further used to determine the preset minimum number of instances as the number of instances to be reduced.

[0157] Optionally, the configuration operating parameter information also includes: a preset scaling plan, the preset scaling plan includes: a target time period for scaling, and a preset number of scaling instances; the determination module 20 is also used to determine whether the current time is in the target time period; if the current time is in the target time period, determine that the data analysis engine meets the preset computing power scaling trigger conditions.

[0158] The second acquisition module 30 is further used to acquire a preset number of scaling instances as the number of instances to be scaled.

[0159] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0160] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0161] like Fig.11As shown, the present application provides an electronic device, including: a processor 100, a storage medium 200 and a bus 300, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to implement the computing power resource adjustment method of the data analysis engine described in the above embodiment.

[0162] The present application also provides a readable storage medium, on which program instructions are stored. When the program instructions are executed by a processor, the computing power resource adjustment method of the data analysis engine described in the above embodiment is implemented.

[0163] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0166] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0167] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for adjusting computing resources of a data analysis engine, characterized in that: include: Obtaining a computing power adjustment configuration file of a data analysis engine to be monitored, wherein the computing power adjustment configuration file at least includes: configuration operation parameter information of the data analysis engine; Determine whether the data analysis engine meets the preset computing power expansion and contraction trigger conditions according to the configuration operation parameter information; If the data analysis engine meets the preset computing power expansion and contraction triggering condition, computing power adjustment information for the data analysis engine is obtained, and the computing power adjustment information includes: the number of instances to be expanded or contracted; According to the number of instances to be expanded or reduced, a computing power adjustment operation is performed on the data analysis engine; The computing power adjustment configuration file also includes: configuration connection information; the determining whether the data analysis engine meets the preset computing power expansion and contraction triggering conditions according to the configuration operation parameter information includes: According to the configuration connection information, a computing power resource monitoring operation is performed on the data analysis engine to obtain computing power resource monitoring data of the data analysis engine; According to the computing power resource monitoring data and the configuration operation parameter information, the computing power occupancy of the data analysis engine is evaluated to obtain a computing power evaluation level of the data analysis engine; Determining whether the data analysis engine meets the preset computing power expansion and contraction triggering condition according to the computing power evaluation level of the data analysis engine; The configuration connection information includes: address information of the currently deployed server and connection information of the currently deployed server; The step of performing a computing resource monitoring operation on the data analysis engine according to the configuration connection information to obtain computing resource monitoring data of the data analysis engine includes: Establishing a connection instance with the currently deployed server according to the configuration connection information, and executing an operating system command through the connection instance to obtain operating environment resource data of the currently deployed server; According to the configuration connection information, a database connection is established with the data analysis engine, and a structured query statement for the data analysis engine is executed through the database connection to obtain computing resource data of the data analysis engine; the computing resource monitoring data includes: the operating environment resource data and the computing resource data; The monitoring data based on the computing power resources and the configuration operation parameter information; The step of evaluating the computing power usage of the data analysis engine to obtain the computing power evaluation level of the data analysis engine includes: Performing an operating environment evaluation on the data analysis engine according to the operating environment resource data and the configuration operating parameter information to obtain an operating environment evaluation score of the data analysis engine; Performing a computing resource evaluation on the data analysis engine according to the computing resource data to obtain a computing resource evaluation score of the data analysis engine; Determining a total evaluation score of the data analysis engine according to the operating environment evaluation score and the computing resource evaluation score; Obtaining a computing power evaluation level of the data analysis engine according to the total evaluation score of the data analysis engine; The determining, according to the computing power evaluation level of the data analysis engine, whether the data analysis engine meets the preset computing power expansion and contraction triggering condition includes: Determine a first total number of times within a first preset number of evaluations that the computing power evaluation level of the data analysis engine is a first resource usage level, where the first resource usage level is used to indicate that resources are busy; If the first total number of times is greater than or equal to the first number threshold, it is determined that the data analysis engine meets the preset computing power expansion trigger condition; The number of instances to be expanded or reduced includes: the number of instances to be expanded; the obtaining of computing power adjustment information for the data analysis engine includes: Determining the sum of the computing resource evaluation scores of the data analysis engines according to the computing resource evaluation scores of the data analysis engines within the first preset evaluation times; The number of instances to be expanded is determined according to the sum of the computing resource evaluation scores.

2. The method according to claim 1, characterized in that The configuration operation parameter information includes: the network bandwidth of the currently deployed server, and the operation environment resource data includes: the processor idle rate of the currently deployed server, and the network traffic data of the currently deployed server; The step of performing an operating environment evaluation on the data analysis engine according to the operating environment resource data and the configuration operating parameter information to obtain an operating environment evaluation score of the data analysis engine includes: According to the processor idle rate and the evaluation scores corresponding to a plurality of preset idle rate ranges, determining the evaluation score corresponding to the target idle rate range where the processor idle rate is located as the first evaluation score of the data analysis engine; Based on the network bandwidth and the network traffic data, the network occupancy of the data analysis engine is determined, and based on the network occupancy and the evaluation scores corresponding to multiple preset network occupancy ranges, the evaluation score corresponding to the target network occupancy range where the network occupancy is located is determined as the second evaluation score of the data analysis engine; the operating environment evaluation score includes: the first evaluation score and the second evaluation score.

3. The method according to claim 1, characterized in that The computing resource data includes: the number of analysis tasks to be executed, and the execution time of the analysis tasks being executed; The step of performing computing resource evaluation on the data analysis engine according to the computing resource data to obtain a computing resource evaluation score of the data analysis engine includes: According to the number of the analysis tasks to be executed and the evaluation scores corresponding to the multiple preset task number ranges, determine that the evaluation score corresponding to the target task number range in which the number of the analysis tasks to be executed is located is the third evaluation score of the data analysis engine; According to the execution time of the analysis task being executed, determining the number of analysis tasks whose execution time is greater than or equal to a preset execution time threshold as the target task number; A fourth evaluation score of the data analysis engine is determined according to the number of target tasks, and the computing resource evaluation score includes: the third evaluation score and the fourth evaluation score.

4. The method according to claim 1, characterized in that: The determining, according to the computing power evaluation level of the data analysis engine, whether the data analysis engine meets the preset computing power expansion and contraction triggering condition further includes: Determine a second total number of times within a second preset evaluation number that the computing power evaluation level of the data analysis engine is a second resource usage level, wherein the second preset evaluation number is greater than the first preset evaluation number, and the second resource usage level is used to indicate that there is idle resources; Determine a third total number of times that the computing power evaluation level of the data analysis engine is the second resource usage level or the third resource usage level within the second preset evaluation number, wherein the third resource usage level is between the second resource usage level and the first resource usage level; If the second total number of times is greater than or equal to the second number threshold, and the third total number of times is greater than or equal to the third number threshold, it is determined that the data analysis engine meets the preset computing power reduction trigger condition; wherein the third number threshold is greater than the second number threshold, and the second number threshold is greater than the first number threshold; The number of instances to be expanded or reduced includes: the number of instances to be reduced; the obtaining of computing power adjustment information for the data analysis engine includes: The preset minimum number of instances is determined to be the number of instances to be scaled down.

5. The method according to claim 1, characterized in that The configuration operation parameter information also includes: a preset expansion and contraction plan, the preset expansion and contraction plan includes: a target time period for expansion and contraction, and a preset number of expansion and contraction instances; the determining whether the data analysis engine meets the preset computing power expansion and contraction triggering condition according to the configuration operation parameter information includes: Determine whether the current time is within the target time period; If the current time is within the target time period, determining that the data analysis engine meets the preset computing power expansion and contraction triggering condition; The obtaining of computing power adjustment information for the data analysis engine includes: The preset number of scaling instances is obtained as the number of instances to be scaled.

6. The method according to claim 1, characterized in that The number of instances to be expanded or reduced includes: the number of instances to be expanded, or the number of instances to be reduced; the computing power adjustment configuration file also includes: information of servers to be deployed; The performing a computing power adjustment operation on the data analysis engine according to the number of instances to be expanded or reduced includes: According to the number of instances to be expanded and the information of the servers to be deployed, obtaining the use rights of the target servers to be deployed corresponding to the number of instances; Upload the deployment installation package of the data analysis engine to the target server to be deployed, start the corresponding number of running instances on the target server to be deployed, and add the information of the corresponding number of running instances to the service cluster of the data analysis engine; or, Remove information of deployed instances of the number of instances to be scaled down from the service cluster of the data analysis engine; A connection is established with a server corresponding to the deployed instance, and deployment file data of the deployed instance on the corresponding server is removed through the connection.

Images