Data processing method, apparatus and system

Abstract

Provided in the embodiments of the present description are a data processing method, apparatus and system. The data processing method is applied to a service platform, which is used for managing and controlling a target service node in a service node cluster. The method comprises: acquiring a resource usage data set of a target service node; on the basis of a data statistics strategy that matches the target service node, compiling data statistics on resource usage data, which corresponds to at least two resource indices, in the resource usage data set, so as to obtain statistic information corresponding to each resource index; on the basis of the statistic information corresponding to each resource index and efficiency calculation information, determining efficiency information corresponding to each resource index; selecting efficiency information corresponding to the resource index corresponding to a calculation dimension, so as to construct calculation efficiency information, and selecting efficiency information corresponding to the resource index corresponding to a storage dimension, so as to construct storage efficiency information; and on the basis of the calculation efficiency information and the storage efficiency information, generating a node update task for the target service node, wherein the node update task is used for optimizing resources of the target service node.

Description

Data processing method, device and system

[0001] This application claims priority to the Chinese patent application filed on March 18, 2024, with application number 202410310049.0 and invention name “Data processing method, device and system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of this specification relate to the field of computer technology, data processing methods, devices and systems. Background Art

[0003] With the advent of the big data era, data computing and storage face enormous challenges. In this environment, cloud computing services have emerged. Service management platforms offer users powerful computing and storage capabilities. They can provide a variety of services, including local storage, local computing, cloud storage, and cloud computing. However, the diverse range of services, the growing number of service providers, and user misjudgment of their needs have led to wasted computing and storage resources.

[0004] In existing technologies, the service management platform usually optimizes computing and storage performance to improve the utilization rate of computing and storage resources. However, this optimization method is difficult and the optimization effect is not significant. Therefore, a more effective data processing method is urgently needed to solve the above problems. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a data processing method is provided, which is applied to a service platform, wherein the service platform is used to manage and control a target service node in a service node cluster, including:

[0007] Obtaining a resource usage dataset of the target service node;

[0008] Based on the data statistics strategy matched by the target service node, data statistics are performed on the resource usage data corresponding to at least two resource indicators in the resource usage data set to obtain statistical information corresponding to each resource indicator;

[0009] Determine the performance information corresponding to each resource indicator based on the statistical information and performance calculation information corresponding to each resource indicator;

[0010] Selecting performance information corresponding to resource indicators corresponding to the computing dimension to construct computing performance information, and selecting performance information corresponding to resource indicators corresponding to the storage dimension to construct storage performance information;

[0011] A node update task of the target service node is generated according to the computing performance information and the storage performance information, and the node update task is used to optimize resources of the target service node.

[0012] According to a second aspect of an embodiment of this specification, a data processing device is provided, which is applied to a service platform, wherein the service platform is used to manage and control a target service node in a service node cluster, including:

[0013] an acquisition module, configured to acquire a resource usage dataset of the target service node;

[0014] a statistics module configured to perform data statistics on the resource usage data corresponding to at least two resource indicators in the resource usage data set based on the data statistics strategy matched by the target service node, and obtain statistical information corresponding to each resource indicator;

[0015] a determination module configured to determine the performance information corresponding to each resource indicator based on the statistical information and performance calculation information corresponding to each resource indicator;

[0016] A construction module is configured to select performance information corresponding to resource indicators corresponding to the computing dimension to construct computing performance information, and select performance information corresponding to resource indicators corresponding to the storage dimension to construct storage performance information;

[0017] A generating module is configured to generate a node updating task of the target service node according to the computing performance information and the storage performance information, wherein the node updating task is used to optimize resources of the target service node.

[0018] According to a third aspect of an embodiment of this specification, there is provided a data processing system, including a service platform and a service node cluster;

[0019] The target service node in the service node cluster is configured to submit a resource usage data set to the service platform in response to a data acquisition request;

[0020] The service platform is configured to perform data statistics on the resource usage data corresponding to at least two resource indicators in the resource usage data set based on the data statistics strategy matched by the target service node, thereby obtaining statistical information corresponding to each resource indicator; determine the performance information corresponding to each resource indicator based on the statistical information and performance calculation information corresponding to each resource indicator; select the performance information corresponding to the resource indicator corresponding to the computing dimension to construct computing performance information, and select the performance information corresponding to the resource indicator corresponding to the storage dimension to construct storage performance information; generate a node update task based on the computing performance information and the storage performance information, and send the node update task to the target service node;

[0021] The target service node is used to optimize local resources associated with the node update task by executing the node update task.

[0022] According to a fourth aspect of the embodiments of this specification, a computing device is provided, including:

[0023] memory and processor;

[0024] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned data processing method are implemented.

[0025] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0026] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0027] The data processing method provided in one embodiment of the present specification is applied to a service platform, which is used to manage and control target service nodes in a service node cluster. Based on the resource usage data set of the target service node collected, the statistical information corresponding to each resource indicator is counted, and then the efficiency information of each resource indicator is generated. The resources of the target service node can be optimized by constructing the computing efficiency information and storage efficiency information. Thereby, the resource utilization rate of the resources provided by the target service node is improved, and the resources of the target service node can be reasonably utilized to avoid resource waste. In addition, for the service provider corresponding to the target service node, the resource utilization of the resources provided by the target service node can be determined based on the computing efficiency information, storage efficiency information, and the statistical information corresponding to each resource indicator, so as to generate a node update task in a targeted manner, and optimize the resources of the target service node by executing the node update task, thereby saving the cost of the service provider while avoiding resource waste and improving the resource utilization rate of the target service node. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a schematic diagram of a processing process of a data processing method provided by one embodiment of this specification;

[0029] FIG2 is a flow chart of a data processing method provided by one embodiment of this specification;

[0030] FIG3 is a flowchart of a data processing method according to an embodiment of the present disclosure;

[0031] FIG4 a is a schematic diagram of a cloud database performance evaluation model provided in a data processing method provided in one embodiment of this specification;

[0032] FIG4 b is a diagram showing the overall architecture of a cloud database performance evaluation method in a data processing method provided in one embodiment of this specification;

[0033] FIG5 is a schematic structural diagram of a data processing device provided by one embodiment of this specification;

[0034] FIG6 is a schematic diagram of the structure of a data processing system provided by one embodiment of this specification;

[0035] FIG7 is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0036] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0037] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0038] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0039] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] First, the terms involved in one or more embodiments of this specification are explained.

[0041] Slow queries: These typically refer to database queries that take a long time to execute and are inefficient. In database management systems, slow queries can become performance bottlenecks, impacting overall system response time and user experience.

[0042] OLAP (Online Analytical Processing): Primarily used to query and analyze large amounts of historical data, observe relationships across multiple dimensions, and perform complex calculations and summaries. Its main functions include querying, analysis, forecasting, and data mining, providing users with flexible data analysis and rapid decision support.

[0043] OLTP (Online Transaction Processing): This is primarily used to process a company's daily transaction data, such as order processing, inventory management, and banking transactions. OLTP emphasizes real-time data processing, supporting concurrent transaction processing and operations such as data insertion, update, and deletion.

[0044] Time window: In data processing and computing, a time window refers to a time range or period used to limit or define the time boundaries of certain operations or events.

[0045] Figure 1 is a schematic diagram of the processing process of a data processing method provided by an embodiment of the present specification; the data processing method is applied to a service platform, and the service platform is used to manage and control the target service node in the service node cluster, as shown in Figure 1, to obtain the resource usage data set of the target service node; determine the data statistics strategy that matches the node type of the target service node. Based on the data statistics strategy that matches the target service node, data statistics are performed on the resource usage data corresponding to at least two resource indicators in the resource usage data set to obtain statistical information corresponding to each resource indicator. The performance information corresponding to each resource indicator is determined based on the statistical information and performance calculation information corresponding to each resource indicator. The performance information corresponding to the resource indicator corresponding to the computing dimension is selected to construct the computing performance information, and the performance information corresponding to the resource indicator corresponding to the storage dimension is selected to construct the storage performance information. A node update task for the target service node is generated based on the computing performance information and the storage performance information, and the node update task is used to optimize the resources of the target service node.

[0046] Based on the collected resource usage data set of the target service node, the statistical information corresponding to each resource indicator is collected, and then the performance information of each resource indicator is generated. The resources of the target service node can be optimized by constructing the computing performance information and storage performance information. This improves the resource utilization rate of the resources provided by the target service node, enables the resources of the target service node to be reasonably utilized, and avoids resource waste. In addition, for the service provider corresponding to the target service node, the resource utilization of the resources provided by the target service node can be determined based on the computing performance information, storage performance information, and the statistical information corresponding to each resource indicator, so as to generate node update tasks in a targeted manner, and optimize the resources of the target service node by executing the node update tasks, thereby saving the cost of the service provider while avoiding resource waste and improving the resource utilization rate of the target service node.

[0047] In this specification, a data processing method is provided. This specification also relates to a data processing system, a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0048] Refer to Figure 2, which shows a flowchart of a data processing method provided according to an embodiment of this specification. The data processing method is applied to a service platform, and the service platform is used to manage and control target service nodes in a service node cluster. The data processing method specifically includes the following steps.

[0049] Step 202: Acquire the resource usage data set of the target service node.

[0050] Specifically, the target service node can be a database instance, a cloud database instance, or other node that can provide data services such as computing services and storage services; the service node cluster is a database cluster, a cloud database cluster, or a cloud computing platform; accordingly, the service platform is used to manage and control the target service nodes in the service node cluster, and can obtain resource usage data of the target service nodes in the service node cluster based on data transmission, and can also plan the computing services and storage services provided by the target service nodes in the service node cluster. The service platform can collect resource usage data of the target service nodes in the service node cluster based on the management and control plug-in. The resource usage data can indicate how the computing services and storage services provided by the target service nodes are used by users, including but not limited to resource utilization, resource usage time, and other data. The usage data of the computing resources provided by the target service node and the usage data of the storage resources constitute the resource usage data set.

[0051] Based on this, when managing and controlling the target service node in the service node cluster, the computing and storage resources provided by the target service node are constantly being used by users. When users use the computing and storage resources provided by the target service node, real user resource usage data can be generated. The computing and storage resource usage data of the target service node can be obtained in real time to form a resource usage dataset.

[0052] Furthermore, considering that the computing resources and storage resources provided by the target service node are used differently in different time periods, the resource usage data of a single resource indicator at a certain moment cannot be used to evaluate the resource usage. Therefore, it is possible to obtain resource usage data corresponding to at least two resource indicators within a certain time interval. The specific implementation is as follows:

[0053] Determine a data acquisition time interval according to the node type of the target service node; obtain resource usage data corresponding to at least two resource indicators generated by the target service node within the data acquisition time interval; and construct the resource usage data set based on the resource usage data corresponding to at least two resource indicators.

[0054] Specifically, the node type of the target service node can represent the event processing type of the target service node. The node type can be an online analytical processing type or an online transaction processing type. The data acquisition time interval is a preset time period for the target service node, which is used to obtain resource usage data generated during the time period. The data acquisition time interval can be set to any time interval such as one hour, one day, or one week. The resource indicator is used to represent the resource usage of the target service node, that is, the resource consumption, which is the performance indicator of the target service node. The resource indicators can be divided into computing resource indicators corresponding to computing resources and storage resource indicators corresponding to storage resources.

[0055] Based on this, determine the node type of the target service node and the data acquisition time interval set for the target service node type. Different node types have different data acquisition time intervals. For target service nodes with daily load fluctuations, a daily data acquisition time interval is selected. Obtain resource usage data corresponding to at least two resource indicators generated by the target service node during the data acquisition time interval. Construct a resource usage dataset based on the resource usage data corresponding to the at least two resource indicators.

[0056] For example, if the target service node is a database instance, the node type is a cloud database. Cloud databases can be online analytical processing databases, online transaction processing databases, or mixed workload databases. A time window is set for the cloud database, i.e., the time interval for data acquisition. Resource usage data such as CPU utilization, memory utilization, and bandwidth utilization generated during the time window is collected for the cloud database.

[0057] In summary, resource usage data corresponding to at least two resource indicators generated by the target service node during the data acquisition time interval is acquired, so that the resource usage dataset constructed based on the resource usage data corresponding to the at least two resource indicators can be used to complete the execution of downstream tasks. This improves the comprehensiveness of data acquisition.

[0058] Step 204: Based on the data statistics strategy matched by the target service node, data statistics are performed on the resource usage data corresponding to at least two resource indicators in the resource usage data set to obtain statistical information corresponding to each resource indicator.

[0059] Specifically, after obtaining the resource usage data set of the target service node as mentioned above, data statistics can be performed on the resource usage data corresponding to at least two resource indicators in the resource usage data set based on the data statistics strategy matched by the target service node to obtain statistical information corresponding to each resource indicator, wherein the data statistics strategy is used to represent the statistical rules for performing statistics on the resource usage data of the target service node. The data statistics strategy can be to perform statistics on the resource usage data of the target service node according to statistical standards such as peak value, mean, median, variance, etc.; the statistical information is the resource data obtained after performing data statistics on the resource usage data of each resource indicator; at least two resource indicators, including: CPU average indicator, memory average indicator, slow query indicator, storage type, storage indicator, storage throughput indicator and service node attribute indicator.

[0060] Based on this, after obtaining the resource usage data set of the target service node as mentioned above, the data statistical strategy matching the target service node is determined, and the data statistical rules corresponding to at least two resource indicators in the resource usage data set are determined based on the data statistical strategy matching the target service node. The resource usage data corresponding to each resource indicator in the resource usage data set are subjected to data statistics according to the data statistical rules corresponding to each resource indicator to obtain statistical information corresponding to each resource indicator.

[0061] In actual applications, the data statistics strategy corresponding to the target service node includes data statistics rules corresponding to each resource indicator of the target service node. The data statistics rules can be statistical standards such as peak value, mean, median, variance, etc. of resource usage data.

[0062] Step 206: Determine the performance information corresponding to each resource indicator according to the statistical information and performance calculation information corresponding to each resource indicator.

[0063] Specifically, in the above-mentioned data statistics strategy based on target service node matching, data statistics are performed on the resource usage data corresponding to at least two resource indicators in the resource usage data set. After obtaining the statistical information corresponding to each resource indicator, the efficiency information corresponding to each resource indicator can be determined based on the statistical information corresponding to each resource indicator and the efficiency calculation information, wherein the efficiency calculation information is used to perform efficiency calculation for the resource indicator in combination with the statistical information corresponding to the resource indicator; the efficiency calculation information includes the resource threshold and resource weight set for the resource indicator, and the resource weight is set according to the proportion of the resource corresponding to the resource indicator in the total resources provided by the target service node. The efficiency information represents the usage of the resource corresponding to the resource indicator, and the efficiency information can be an efficiency score or an efficiency level. The higher the efficiency score, the higher the resource utilization corresponding to the resource indicator and the less resource waste. Correspondingly, the higher the efficiency level, the higher the resource utilization corresponding to the resource indicator and the less resource waste.

[0064] Based on this, in the above-mentioned data statistics strategy based on target service node matching, data statistics are performed on the resource usage data corresponding to at least two resource indicators in the resource usage data set. After obtaining the statistical information corresponding to each resource indicator, the statistical information and efficiency calculation information corresponding to each resource indicator are determined, and the efficiency score or efficiency level corresponding to each resource indicator is calculated based on the statistical information and efficiency calculation information corresponding to each resource indicator.

[0065] Among them, the at least two resource indicators include: CPU average indicator, memory average indicator, slow query indicator, storage type, storage indicator, storage throughput indicator and service node attribute indicator; the CPU average indicator includes but is not limited to CPU average utilization and CPUp95 utilization, the memory average indicator can be memory average utilization, the slow query indicator can be the number of slow queries, the storage type can be different levels of storage forms (such as PL0, PL1 and other different performance levels), the storage indicator can be storage average utilization, the storage throughput indicator can be throughput average utilization, and the service node attribute indicator can be the local disk usage ratio.

[0066] Among them, the resource usage data corresponding to the at least two resource indicators respectively include: the CPU average usage data corresponding to the CPU average indicator, the memory average usage data corresponding to the memory average indicator, the slow query usage data corresponding to the slow query indicator, the storage usage data corresponding to the storage indicator, the storage throughput usage data corresponding to the storage throughput indicator, and the service node attribute usage data corresponding to the service node attribute indicator.

[0067] The statistical information corresponding to the at least two resource indicators respectively includes: CPU average statistical information corresponding to the CPU average indicator, memory average statistical information corresponding to the memory average indicator, slow query statistical information corresponding to the slow query indicator, storage statistical information corresponding to the storage indicator, storage throughput statistical information corresponding to the storage throughput indicator, and service node attribute statistical information corresponding to the service node attribute indicator.

[0068] The performance information corresponding to the at least two resource indicators respectively includes: the CPU average performance information corresponding to the CPU average indicator, the memory average performance information corresponding to the memory average indicator, the slow query performance information corresponding to the slow query indicator, the storage performance information corresponding to the storage indicator, the storage throughput performance information corresponding to the storage throughput indicator, and the service node attribute performance information corresponding to the service node attribute indicator. The CPU average performance information corresponding to the CPU average indicator can be the storage node CPU average utilization score and the computing node CPU average utilization score; the memory average performance information corresponding to the memory average indicator can be the computing node memory average utilization score and the storage node memory average utilization score; the slow query performance information corresponding to the slow query indicator can be the slow query score; the storage performance information corresponding to the storage indicator can be the storage average utilization score; the storage throughput performance information corresponding to the storage throughput indicator can be the storage throughput usage score; and the service node attribute performance information corresponding to the service node attribute indicator can be the local disk management score.

[0069] Furthermore, considering that the resource usage dataset corresponds to at least two resource indicators, and the target service node provides different resource specifications for different resource indicators, it is necessary to combine the weight information corresponding to the resource indicators when determining the performance information corresponding to the resource indicators. The specific implementation is as follows:

[0070] Determine the weight information and threshold information included in the performance calculation information of the target resource indicator; generate the indicator performance information of the target resource indicator based on the statistical information corresponding to the target resource indicator and the threshold information; generate the performance information corresponding to the target resource indicator based on the indicator performance information and the weight information.

[0071] Specifically, the target resource indicator can be any one of at least two resource indicators; the weight information represents the proportion of the resource corresponding to the target resource indicator in the resources provided in the target service node; the threshold information refers to the resource value set for the resource corresponding to the target resource indicator; the indicator performance information represents the resource score of the target resource indicator obtained based on the statistical information and threshold information of the target resource indicator; it is the score of the target resource indicator in the resource dimension; the performance information is the score of the target resource indicator in combination with the proportion of the resource corresponding to the target resource indicator in the resources provided in the target service node.

[0072] Based on this, the weight information and threshold information included in the performance calculation information of the target resource indicator are determined. The statistical resource value is determined based on the statistical information corresponding to the target resource indicator, and the indicator performance information of the target resource indicator is calculated based on the statistical resource value and the resource threshold corresponding to the threshold information. The performance information corresponding to the target resource indicator is calculated based on the indicator performance information and the weight information.

[0073] The indicator performance information can be calculated using the following formula (1):

[0074] Where t represents the threshold information; i represents the target service node; j represents the target resource indicator; S ij Indicates indicator performance information; r ij Represents statistical information. If the statistical resource value corresponding to the target resource indicator's statistical information is greater than or equal to the resource threshold corresponding to the threshold information, the score corresponding to the target resource indicator's indicator performance information is 100. If the statistical resource value corresponding to the target resource indicator's statistical information is less than the resource threshold corresponding to the threshold information, the target resource indicator's indicator performance information is the ratio of the statistical information to the target resource indicator's threshold information. Performance information is the product of the indicator performance information and the weight information.

[0075] In addition, the total performance information of the target service node can be calculated by the following formula (2): S i =∑s ij ×W(r ij ) (2)

[0076] Among them, W(r ij ) represents the weight of each resource indicator, S i Indicates the overall performance information of the target service node.

[0077] The performance information of the service node cluster can also be calculated based on the total performance information of each target service node. The performance information S(Ω) of the service node cluster can be weighted summed based on the total performance information of each target service node and the node weight W(R) of each target service node.

[0078] Continuing with the above example, when at least two resource indicators are determined: CPU average utilization, memory average utilization, number of slow queries, storage average utilization, throughput average utilization, and local disk usage ratio. Any resource indicator can be selected as the target resource indicator to calculate the performance information of the target resource indicator. Determine the CPU average utilization as the target resource indicator. Determine the actual utilization value of the CPU average utilization (90%), the resource threshold corresponding to the CPU average utilization of 80%, and the threshold information corresponding to the CPU average utilization of 85%. Based on the above formula (1), it can be determined that the actual utilization value of the CPU average utilization of 90% is greater than the resource threshold corresponding to the CPU average utilization of 80%, and the performance information corresponding to the CPU average utilization is 100 points.

[0079] In summary, combined with the weight information and threshold information included in the performance calculation information of the target resource indicator, the performance information corresponding to the target resource indicator is gradually calculated, so that the proportion of the target resource indicator in the target service node is used as the influencing factor of the performance information.

[0080] Step 208: Select performance information corresponding to resource indicators corresponding to the computing dimension to construct computing performance information, and select performance information corresponding to resource indicators corresponding to the storage dimension to construct storage performance information.

[0081] Specifically, after determining the performance information corresponding to each resource indicator based on the statistical information and performance calculation information corresponding to each resource indicator, the performance information corresponding to the resource indicator of the corresponding computing dimension can be selected to construct the computing performance information, and the performance information corresponding to the resource indicator of the corresponding storage dimension can be selected to construct the storage performance information, wherein the computing dimension and the storage dimension are two analysis angles for data analysis on the target service node; the computing dimension corresponds to the computing resources of the target service node, and the storage dimension corresponds to the storage resources of the target service node; accordingly, the computing performance information is resource utilization information obtained by integrating the resource usage data corresponding to the resource indicators of the target service node in the computing dimension, indicating the usage of the computing resources of the target service node; the storage performance information is resource utilization information obtained by integrating the resource usage data corresponding to the resource indicators of the target service node in the storage dimension, indicating the usage of the storage resources of the target service node.

[0082] Based on this, after determining the performance information corresponding to each resource indicator based on the statistical information and performance calculation information corresponding to each resource indicator, a resource indicator corresponding to the computing dimension and a resource indicator corresponding to the storage dimension are selected from the at least two resource indicators. Computing performance information is constructed based on the performance information of each resource indicator corresponding to the computing dimension, and storage performance information is constructed based on the performance information of each resource indicator corresponding to the storage dimension. This facilitates subsequent resource optimization of the target service node based on the computing performance information and storage performance information.

[0083] Furthermore, considering that the resource usage data set contains resource usage data corresponding to at least two resource indicators, when there are many resource indicators, the resource indicators can be divided into two categories, namely, resource indicators of the computing dimension and resource indicators of the storage dimension. Therefore, when calculating the performance information, the weight of each resource indicator can also be combined to calculate the storage performance information and the computing performance information separately. The specific implementation is as follows:

[0084] Selecting a computing resource indicator of a computing dimension from at least two resource indicators; performing weighted calculation based on the computing indicator weight and computing performance information of the computing resource indicator to obtain the computing performance information of the computing dimension; wherein, the selecting performance information corresponding to the resource indicator corresponding to the storage dimension to construct storage performance information includes: selecting a storage resource indicator of a storage dimension from at least two resource indicators; performing weighted calculation based on the storage indicator weight and storage performance information of the storage resource indicator to obtain the storage performance information of the storage dimension.

[0085] Specifically, the computing resource metric corresponds to the computing resources provided by the target service node, and the resource usage data corresponding to the computing resource metric can represent the usage of the computing resources. Correspondingly, the storage resource metric corresponds to the storage resources provided by the target service node, and the resource usage data corresponding to the storage resource metric can represent the usage of the storage resources. The computing metric weight is determined based on the proportion of the computing resources provided by the target service node in the total resources; the storage metric weight is determined based on the proportion of the storage resources provided by the target service node in the total resources.

[0086] Based on this, a computing resource indicator of the computing dimension is selected from at least two resource indicators. The computing indicator weight is determined based on the proportion of the computing resources provided by the target service node in the total resources. A weighted calculation is performed based on the computing indicator weight of the computing resource indicator and the computing performance information to obtain the computing performance information of the computing dimension. Correspondingly, a storage resource indicator of the storage dimension is selected from at least two resource indicators. The storage indicator weight is determined based on the proportion of the storage resources provided by the target service node in the total resources. A weighted calculation is performed based on the storage indicator weight of the storage resource indicator and the storage performance information to obtain the storage performance information of the storage dimension.

[0087] Continuing with the above example, after determining at least two resource indicators: average CPU utilization, average memory utilization, number of slow queries, storage type, average storage utilization, average throughput utilization, and local disk usage ratio. Select the computing resource indicators of the computing dimension from the above resource indicators: average CPU utilization, average memory utilization, and number of slow queries. Calculate the performance score corresponding to the computing resource indicator based on the computing indicator weights of each resource indicator and the initial performance score of each indicator obtained by calculation. Correspondingly, select the computing resource indicators of the storage dimension from the above resource indicators: storage type, average storage utilization, average throughput utilization, and local disk usage ratio. Calculate the performance score corresponding to the storage resource indicator based on the storage indicator weights of each resource indicator and the initial performance score of each indicator obtained by calculation.

[0088] In summary, computing performance information is determined in the computing dimension, and storage performance information is determined in the storage dimension, so that resource usage of the target service node can be subsequently obtained in the computing dimension and the storage dimension.

[0089] Step 210: Generate a node update task for the target service node according to the computing performance information and the storage performance information, wherein the node update task is used to optimize resources of the target service node.

[0090] Specifically, after selecting the performance information corresponding to the resource indicators of the corresponding computing dimension to construct the computing performance information, and selecting the performance information corresponding to the resource indicators of the corresponding storage dimension to construct the storage performance information, the node update task of the target service node can be generated according to the computing performance information and the storage performance information. The node update task is used to optimize the resources of the target service node, wherein the node update task is used to optimize the resource allocation of the resources provided by the target node, and optimize the resource scale, resource parameters, and service node specifications provided by the target node in the computing dimension and storage dimension respectively, wherein resource optimization can be to adjust the resource scale and resource parameters.

[0091] Based on this, after selecting the performance information corresponding to the resource indicators in the corresponding computing dimension to construct the computing performance information, and selecting the performance information corresponding to the resource indicators in the corresponding storage dimension to construct the storage performance information, a node update task for the target service node is generated based on the computing performance information and the storage performance information. The node update task can be provided to the service provider as an optimization suggestion. The node update task is used to optimize the resources of the target service node and adjust the resource scale of the resources provided by the target service node.

[0092] Furthermore, considering that optimizing the resources of the target service node will affect the service time of the target service node, it is possible to determine whether the target service node meets the update conditions before optimizing the resources of the target service node. The specific implementation is as follows:

[0093] Node performance information of the target service node is constructed based on the computing performance information and the storage performance information; when it is determined based on the node performance information that the target service node meets the update conditions, a node update task of the target service node is generated based on the computing performance information and the storage performance information.

[0094] Specifically, node performance information can be obtained by performing a weighted calculation on the computing performance information and the storage performance information; the weight corresponding to the computing performance information and the weight corresponding to the storage performance information are determined and then weighted summed. The update condition is used to determine whether the target service node needs to be updated.

[0095] Based on this, the node performance information of the target service node is constructed based on the computing performance information and the storage performance information. When it is determined based on the node performance information that the target service node meets the update conditions, a node update task for the target service node is generated based on the computing performance information and the storage performance information. The update condition can be whether the node performance score corresponding to the node performance information of the target service node is greater than the performance score threshold. If the node performance score is greater than the performance score threshold, it indicates that the resource utilization rate of the target service node is high and the target service node does not need to be updated. If the node performance score is less than or equal to the performance score threshold, it indicates that the resource utilization rate of the target service node is low and the target service node needs to be updated to achieve the purpose of resource optimization of the target service node.

[0096] In actual applications, it is also possible to determine whether the target service node meets the update conditions based on the computing performance information or the storage performance information. In the case that the computing performance information or the storage performance information meets the update conditions, a node update task for the target service node is generated based on the computing performance information or the storage performance information. That is, in the case that the target service node is determined to meet the update conditions based on the computing performance information, a node update task for the target service node is generated based on the computing performance information; in the case that the target service node is determined to meet the update conditions based on the storage performance information, a node update task for the target service node is generated based on the storage performance information. This enables resource optimization of the target service node when any one of the computing and storage dimensions meets the update conditions, thereby reducing the granularity of resource optimization judgment for the target service node.

[0097] Continuing with the previous example, if the compute performance score is 80 and the storage performance score is 76, the target service node's node performance score is calculated using the compute performance weight of 0.6 and the storage performance weight of 0.4: 80 * 0.6 + 76 * 0.4 = 78.4. If the node performance score is less than the performance threshold of 80, resource optimization is required for the target service node by executing the node update task.

[0098] Node performance information of the target service node is constructed; when it is determined based on the node performance information that the target service node meets the update conditions, a node update task of the target service node is generated according to the computing performance information and the storage performance information.

[0099] In summary, when a target service node is determined to meet update conditions based on node performance information, a node update task for the target service node is generated based on the computing and storage performance information. This determines whether the target service node requires an update based on its node performance information, enabling a node-level determination of whether the target service node requires resource optimization. This reduces the number of resource optimization operations performed on the target service node.

[0100] Furthermore, considering that the resource usage dataset corresponds to at least two resource indicators, when generating a node update task, you can generate a node update task based on a single resource indicator or multiple resource indicators. The specific implementation is as follows:

[0101] Compare the computing performance information with reference computing information, and compare the storage performance information with reference storage information; update the node status of the target service node based on the computing information comparison result and the storage information comparison result; when the updated node status is a pending update status, generate a node update task for the target service node based on the performance information corresponding to the target resource indicator in at least two resource indicators.

[0102] Specifically, the reference computing information can be a reference computing score pre-set for the target service node, which is used to determine whether the target service node needs to perform resource optimization in the computing dimension in combination with the computing performance information; the reference storage information can be a reference storage score pre-set for the target service node, which is used to determine whether the target service node needs to perform resource optimization in the storage dimension in combination with the storage performance information; the computing information comparison result indicates the judgment result of whether the target service node needs to perform resource optimization in the computing dimension; the storage information comparison result indicates the judgment result of whether the target service node needs to perform resource optimization in the storage dimension; the node status can be a to-be-updated status, an updated status, or a non-updated status in this round; the target resource indicator can be any one or more resource indicators among at least two resource indicators; the target resource indicator can be selected based on the performance information of the resource indicator, and the resource indicator with a lower performance score corresponding to the performance information is selected as the target resource indicator.

[0103] Based on this, the computing performance information is compared with the reference computing information, and the storage performance information is compared with the reference storage information; the node status of the target service node is updated based on the computing information comparison result and the storage information comparison result; or the node status of the target service node is updated based on the computing information comparison result; or the node status of the target service node is updated based on the storage information comparison result; when the updated node status is a to-be-updated status, the target resource indicator is selected from at least two resource indicators according to the performance information of the resource indicator, and a node update task of the target service node is generated based on the performance information corresponding to the target resource indicator.

[0104] Continuing with the above example, when the computing efficiency score of the computing efficiency information is 80 and the storage efficiency score of the storage efficiency information is 76, the computing efficiency score 80 is compared with the reference computing score 85 corresponding to the reference computing information, and the storage efficiency score 76 is compared with the reference storage score 85 corresponding to the reference storage information. It is determined that the computing efficiency score is less than the reference computing score 85, and the storage efficiency score 76 is less than the reference storage score 85. Then, the node status of the database instance is updated to the pending update status. Among at least two resource indicators, the resource indicator with a lower efficiency score is selected: the average CPU utilization and the average memory utilization as the target resource indicator. Then, a node update task for the database instance is generated based on the average CPU utilization and the average memory utilization.

[0105] To sum up, the node update task of the target service node is generated based on the performance information corresponding to the target resource indicator in at least two resource indicators, and the node update task is generated based on a single resource indicator, or based on multiple resource indicators, thereby improving the flexibility of node update task generation, and also improving the flexibility of resource optimization for the target service node.

[0106] Furthermore, considering that the resource optimization result of the target service node after executing the node update task is unknown, the target service node's detection computing efficiency information and detection storage efficiency information can be determined after executing the node update task. By comparing the computing efficiency information and storage efficiency information of the target service node with the computing efficiency information and storage efficiency information of the target service node before the node update task, it can be determined whether the resources of the target service node are optimized. The specific implementation is as follows:

[0107] When the node update task is completed, the detected computing performance information and the detected storage performance information of the target service node are obtained; the computing performance information is compared with the detected computing performance information, and the storage performance information is compared with the detected storage performance information; when it is determined based on the computing comparison result and the storage comparison result that the resource optimization of the target service node for resource optimization is completed, the optimization information corresponding to the node update task is created for the target service node and recorded.

[0108] Specifically, the detection of computing efficiency information is the efficiency score or efficiency level obtained by performing efficiency calculation on the target service node in the computing dimension after the node update task is completed; the calculation comparison result is the efficiency score or efficiency level obtained by performing efficiency calculation on the target service node in the storage dimension after the node update task is completed; the storage comparison result is the comparison result of the efficiency information before the node update task is executed and the efficiency information after the node update task is executed, and the storage comparison result represents the resource optimization status of the target service node; the optimization information is the resource indicators optimized during the execution of the node update task, as well as the optimization intensity of the resource indicators.

[0109] Based on this, when the node update task is completed, it means that the target service node has completed this round of resource optimization, and the detected computing efficiency information and detected storage efficiency information of the target service node are obtained. The computing efficiency information is compared with the detected computing efficiency information, and the storage efficiency information is compared with the detected storage efficiency information. When it is determined based on the calculation comparison results and the storage comparison results that the resource optimization of the target service node for resource optimization is completed, the optimization information corresponding to the node update task is created for the target service node and recorded. Or, when it is determined based on the calculation comparison results that the resource optimization of the target service node for resource optimization is completed, the optimization information corresponding to the node update task is created for the target service node and recorded. Or, when it is determined based on the storage comparison results that the resource optimization of the target service node for resource optimization is completed, the optimization information corresponding to the node update task is created for the target service node and recorded.

[0110] Continuing with the previous example, after executing the node update task, the database instance's corresponding compute performance score of 82 and storage performance score of 79 are calculated. The compute performance score of 82 is compared with the compute performance score of 80, and the storage performance score of 79 is compared with the storage performance score of 76. The comparison results confirm that the database instance has achieved resource optimization in both the storage and compute dimensions. Optimization information for this round of optimization is generated based on the optimized resource metrics and the comparison results. This provides a reference for subsequent database instance optimization.

[0111] In summary, the resource optimization information of the target service node can be determined based on the calculation and storage of the comparison results. The resource optimization information is recorded and can be used as reference information for the next resource optimization of the target service node.

[0112] The data processing method provided in one embodiment of the present specification is applied to a service platform, which is used to manage and control target service nodes in a service node cluster. Based on the resource usage data set of the target service node collected, the statistical information corresponding to each resource indicator is counted, and then the efficiency information of each resource indicator is generated. The resources of the target service node can be optimized by constructing the computing efficiency information and storage efficiency information. Thereby, the resource utilization rate of the resources provided by the target service node is improved, and the resources of the target service node can be reasonably utilized to avoid resource waste. In addition, for the service provider corresponding to the target service node, the resource utilization of the resources provided by the target service node can be determined based on the computing efficiency information, storage efficiency information, and the statistical information corresponding to each resource indicator, so as to generate a node update task in a targeted manner, and optimize the resources of the target service node by executing the node update task, thereby saving the cost of the service provider while avoiding resource waste and improving the resource utilization rate of the target service node.

[0113] The following further illustrates the data processing method provided in this specification using the application of the data processing method in improving the performance of a cloud database as an example, in conjunction with Figure 3. Figure 3 shows a flow chart of the processing process of a data processing method provided in one embodiment of this specification, specifically including the following steps.

[0114] Step 302: Determine a time window according to the database type of the cloud database instance.

[0115] In cloud database instance scenarios, you can evaluate and optimize cloud database performance. Public cloud service providers offer a range of cloud plug-ins that track and record cloud database instance performance metrics in real time. Key performance indicator data, such as CPU utilization, memory utilization, and bandwidth utilization, can be obtained, exported, or downloaded through the console. This data can be recorded at a granularity from minute to daily, including time series data showing the highest, lowest, and average values ​​within the time period.

[0116] Step 304: Obtain resource usage data corresponding to at least two resource indicators generated by the cloud database instance within the time window.

[0117] At least two resource indicators include, but are not limited to, a CPU average indicator, a memory average indicator, a slow query indicator, a storage type, a storage indicator, a storage throughput indicator, and a service node attribute indicator. As shown in Figure 4a, the cloud database performance evaluation model includes computing performance indicators and storage performance indicators. The CPU average indicator includes, but is not limited to, CPU average utilization and CPU p95 utilization. The memory average indicator can be the average memory utilization. The slow query indicator can be the number of slow queries. The storage type can be different levels of storage (such as PL0, PL1, and other different performance levels). The storage indicator can be the storage average utilization. The storage throughput indicator can be the throughput average utilization. The service node attribute indicator can be the local disk usage percentage.

[0118] Step 306: Construct a resource usage dataset based on the resource usage data corresponding to the at least two resource indicators.

[0119] The resource usage data of each resource indicator obtained are integrated to obtain a resource usage data set.

[0120] Step 308: Based on the data aggregation strategy matched with the cloud database instance, aggregate the resource usage data corresponding to at least two resource indicators in the resource usage data set to obtain aggregation information corresponding to each resource indicator.

[0121] The data aggregation policy indicates the data processing strategy that matches the cloud database instance type. For example, for an OLTP online transaction processing database, according to the data aggregation policy, CPU utilization is aggregated into the P95 peak utilization of the day, memory utilization is aggregated into the daily average utilization, storage utilization is aggregated into the daily average utilization, and storage throughput utilization is aggregated into the P95 peak utilization of the day.

[0122] Step 310: Determine the performance score corresponding to each resource indicator based on the aggregation information and performance calculation information corresponding to each resource indicator.

[0123] Based on the aggregate information corresponding to each resource indicator, the performance score corresponding to each resource indicator can be calculated according to the calculation method provided by the performance evaluation model.

[0124] When the cloud database is OLAP, the data related to the OLAP online analytical processing database performance indicators are shown in Table 1 below:

[0125] Table 1

[0126] Step 312: Calculate the instance performance score of the cloud database instance based on the performance score corresponding to each resource indicator.

[0127] By combining the score weights of various resource indicators, the performance score of the cloud database instance and the performance score of the cloud database cluster can be calculated.

[0128] Step 314: Generate an instance update task for the cloud database instance based on the performance score corresponding to each resource indicator and the instance performance score, so as to optimize resources of the cloud database instance.

[0129] Based on the calculated performance scores corresponding to each resource indicator, the performance score of the cloud database instance, and the performance score of the cloud database cluster, optimization suggestions can be made for the resource indicators that need to be optimized corresponding to the cloud database instance, and then resource optimization can be performed on the resource indicators that need to be optimized.

[0130] Step 316: When the instance update task is completed, obtain the cloud database instance optimization efficiency score.

[0131] Step 318: Compare the optimization efficiency score with the instance efficiency score. If it is determined based on the comparison result that the resource optimization of the cloud database instance is completed, generate and record optimization information corresponding to the instance update task for the cloud database instance.

[0132] After optimizing the resource metrics that require optimization, you can recalculate their performance scores to determine the effectiveness of the optimization, specifically whether costs have been reduced and resource waste has been minimized. Based on the cloud database instance performance evaluation results and specific evaluation metrics, optimization recommendations are provided to reduce cloud database costs and improve efficiency.

[0133] In practical applications, the overall architecture of cloud database performance evaluation is shown in Figure 4b. The cloud database performance evaluation and optimization process involves three key steps: data collection and preprocessing, evaluation model construction and indicator selection, and optimization recommendation generation. During the data collection and preprocessing phase, cloud database resource data is collected. Resource data can correspond to historical indicators, online indicators, and cloud bills. Resource data is aggregated according to established rules and stored in a data warehouse. Based on the cloud database performance evaluation model, the performance score of each resource indicator and the performance score of the cloud database are calculated in combination with the preprocessed resource data. Evaluation results are generated based on the performance scores of each resource indicator and the performance score of the cloud database. During the optimization recommendation phase, targeted optimization solutions are provided based on the evaluation results, and operation and maintenance optimization is implemented to achieve cost savings and optimize system stability.

[0134] In specific implementation, the cloud database performance evaluation model includes a computing performance model and a storage performance model. The computing performance model includes CPU and memory utilization indicators and slow query management indicators. Among them, because slow queries lead to higher peak utilization, instances where slow queries frequently occur may have more resources in an idle state. The main factors causing slow queries may include the complexity of the query itself, the lack of necessary indexes in the database, or the query statements needing further analysis and optimization. Therefore, evaluating the slow query situation in the database instance and guiding users to improve it is an important step to improve database performance and resource utilization efficiency. The storage performance model includes storage utilization indicators and local disk management indicators. Among them, the local disk management indicators can guide users to migrate data from local disks to cloud disks to achieve the goal of reducing costs and increasing efficiency.

[0135] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment. FIG5 shows a schematic diagram of the structure of a data processing device provided in one embodiment of this specification. As shown in FIG5, the device includes:

[0136] According to a second aspect of an embodiment of this specification, a data processing device is provided, which is applied to a service platform, wherein the service platform is used to manage and control a target service node in a service node cluster, including:

[0137] An acquisition module 502 is configured to acquire a resource usage dataset of the target service node;

[0138] A statistics module 504 is configured to perform data statistics on the resource usage data corresponding to at least two resource indicators in the resource usage data set based on the data statistics strategy matched by the target service node, and obtain statistical information corresponding to each resource indicator;

[0139] Determining module 506, configured to determine the performance information corresponding to each resource indicator based on the statistical information and performance calculation information corresponding to each resource indicator;

[0140] A construction module 508 is configured to select performance information corresponding to resource indicators corresponding to the computing dimension to construct computing performance information, and select performance information corresponding to resource indicators corresponding to the storage dimension to construct storage performance information;

[0141] The generating module 510 is configured to generate a node update task of the target service node according to the computing performance information and the storage performance information, wherein the node update task is used to optimize the resources of the target service node.

[0142] In an optional embodiment, the determining module 506 is further configured to:

[0143] The at least two resource indicators include: a CPU average indicator, a memory average indicator, a slow query indicator, a storage indicator, a storage throughput indicator and a service node attribute indicator; wherein, the resource usage data corresponding to the at least two resource indicators respectively include: CPU average usage data corresponding to the CPU average indicator, memory average usage data corresponding to the memory average indicator, slow query usage data corresponding to the slow query indicator, storage usage data corresponding to the storage indicator, storage throughput usage data corresponding to the storage throughput indicator and service node attribute usage data corresponding to the service node attribute indicator; the statistical information corresponding to the at least two resource indicators respectively include: CPU average statistics corresponding to the CPU average indicator The at least two resource indicators respectively correspond to the performance information, including: the CPU average performance information corresponding to the CPU average indicator, the memory average performance information corresponding to the memory average indicator, the slow query performance information corresponding to the slow query indicator, the storage performance information corresponding to the storage indicator, the storage throughput performance information corresponding to the storage throughput indicator, and the service node attribute statistical information corresponding to the service node attribute indicator. The performance information corresponding to the at least two resource indicators includes: the CPU average performance information corresponding to the CPU average indicator, the memory average performance information corresponding to the memory average indicator, the slow query performance information corresponding to the slow query indicator, the storage performance information corresponding to the storage indicator, the storage throughput performance information corresponding to the storage throughput indicator, and the service node attribute performance information corresponding to the service node attribute indicator.

[0144] In an optional embodiment, the determining module 506 is further configured to:

[0145] Determine the weight information and threshold information included in the performance calculation information of the target resource indicator; generate the indicator performance information of the target resource indicator based on the statistical information corresponding to the target resource indicator and the threshold information; generate the performance information corresponding to the target resource indicator based on the indicator performance information and the weight information.

[0146] In an optional embodiment, the building module 508 is further configured to:

[0147] Selecting a computing resource indicator of a computing dimension from at least two resource indicators; performing weighted calculation based on the computing indicator weight and computing performance information of the computing resource indicator to obtain the computing performance information of the computing dimension; wherein, the selecting performance information corresponding to the resource indicator corresponding to the storage dimension to construct storage performance information includes: selecting a storage resource indicator of a storage dimension from at least two resource indicators; performing weighted calculation based on the storage indicator weight and storage performance information of the storage resource indicator to obtain the storage performance information of the storage dimension.

[0148] In an optional embodiment, the generating module 510 is further configured to:

[0149] Node performance information of the target service node is constructed based on the computing performance information and the storage performance information; when it is determined based on the node performance information that the target service node meets the update conditions, a node update task of the target service node is generated based on the computing performance information and the storage performance information.

[0150] In an optional embodiment, the generating module 510 is further configured to:

[0151] Compare the computing performance information with reference computing information, and compare the storage performance information with reference storage information; update the node status of the target service node based on the computing information comparison result and the storage information comparison result; when the updated node status is a pending update status, generate a node update task for the target service node based on the performance information corresponding to the target resource indicator in at least two resource indicators.

[0152] In an optional embodiment, the generating module 510 is further configured to:

[0153] When the node update task is completed, the detected computing performance information and the detected storage performance information of the target service node are obtained; the computing performance information is compared with the detected computing performance information, and the storage performance information is compared with the detected storage performance information; when it is determined based on the computing comparison result and the storage comparison result that the resource optimization of the target service node for resource optimization is completed, the optimization information corresponding to the node update task is created for the target service node and recorded.

[0154] In an optional embodiment, the acquisition module 502 is further configured to:

[0155] Determine a data acquisition time interval according to the node type of the target service node; obtain resource usage data corresponding to at least two resource indicators generated by the target service node within the data acquisition time interval; and construct the resource usage data set based on the resource usage data corresponding to at least two resource indicators.

[0156] The data processing method provided in one embodiment of the present specification is applied to a service platform, which is used to manage and control target service nodes in a service node cluster. Based on the resource usage data set of the target service node collected, the statistical information corresponding to each resource indicator is counted, and then the efficiency information of each resource indicator is generated. The resources of the target service node can be optimized by constructing the computing efficiency information and storage efficiency information. Thereby, the resource utilization rate of the resources provided by the target service node is improved, and the resources of the target service node can be reasonably utilized to avoid resource waste. In addition, for the service provider corresponding to the target service node, the resource utilization of the resources provided by the target service node can be determined based on the computing efficiency information, storage efficiency information, and the statistical information corresponding to each resource indicator, so as to generate a node update task in a targeted manner, and optimize the resources of the target service node by executing the node update task, thereby saving the cost of the service provider while avoiding resource waste and improving the resource utilization rate of the target service node.

[0157] The above is a schematic diagram of a data processing device according to this embodiment. It should be noted that the technical solution of the data processing device and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the above-mentioned data processing method.

[0158] 6 shows a schematic diagram of a data processing system according to an embodiment of the present disclosure. The data processing system 600 includes a service platform 610 and a service node cluster 620 .

[0159] The target service node 622 in the service node cluster 620 is configured to submit a resource usage dataset to the service platform in response to a data acquisition request;

[0160] The service platform 610 is configured to perform data statistics on the resource usage data corresponding to at least two resource indicators in the resource usage data set based on the data statistics strategy matched by the target service node, to obtain statistical information corresponding to each resource indicator; determine the performance information corresponding to each resource indicator based on the statistical information and performance calculation information corresponding to each resource indicator; select the performance information corresponding to the resource indicator corresponding to the computing dimension to construct computing performance information, and select the performance information corresponding to the resource indicator corresponding to the storage dimension to construct storage performance information; generate a node update task based on the computing performance information and the storage performance information, and send the node update task to the target service node 622;

[0161] The target service node 622 is configured to optimize local resources associated with the node update task by executing the node update task.

[0162] In actual applications, the service platform 610 can periodically send data acquisition requests to the target service node 622 to obtain the resource usage dataset of the target service node 622. The service platform 610 can perform resource optimization for the target service node 622 based on the resource usage dataset submitted by the target service node 622 in the service node cluster 620. After receiving the data acquisition request from the service platform 610, the target service node 622 submits the resource usage dataset to the service platform 610 in response to the data acquisition request. The service platform 610 determines a data statistics policy that matches the node type of the target service node 622. Based on the data statistics policy that matches the target service node 622, the service platform 610 performs data statistics on the resource usage data corresponding to at least two resource indicators in the resource usage dataset to obtain statistical information corresponding to each resource indicator. The at least two resource indicators include: a CPU average indicator, a memory average indicator, a slow query indicator, a storage type, a storage indicator, a storage throughput indicator, and a service node attribute indicator. The service platform 610 then determines the performance information corresponding to each resource indicator based on the statistical information corresponding to each resource indicator and the performance calculation information. The performance information corresponding to the resource indicators in the corresponding computing dimension is selected to construct the computing performance information, and the performance information corresponding to the resource indicators in the corresponding storage dimension is selected to construct the storage performance information. A node update task for the target service node 622 is generated based on the computing performance information and the storage performance information, and the node update task is sent to the target service node 622. The target service node 622 executes the node update task to optimize the resources of the target service node 622.

[0163] Furthermore, the service platform 610 is used to determine the weight information and threshold information contained in the performance calculation information of the target resource indicator; generate the indicator performance information of the target resource indicator based on the statistical information corresponding to the target resource indicator and the threshold information; and generate the performance information corresponding to the target resource indicator based on the indicator performance information and the weight information.

[0164] Furthermore, the service platform 610 is used to select a computing resource indicator of a computing dimension from at least two resource indicators; perform weighted calculation based on the computing indicator weight and computing performance information of the computing resource indicator to obtain the computing performance information of the computing dimension; select a storage resource indicator of a storage dimension from at least two resource indicators; perform weighted calculation based on the storage indicator weight and storage performance information of the storage resource indicator to obtain the storage performance information of the storage dimension.

[0165] Furthermore, the service platform 610 is used to construct the node performance information of the target service node 622 based on the computing performance information and the storage performance information; when it is determined based on the node performance information that the target service node 622 meets the update conditions, a node update task of the target service node 622 is generated based on the computing performance information and the storage performance information.

[0166] Furthermore, the service platform 610 is used to compare the computing performance information with the reference computing information, and to compare the storage performance information with the reference storage information; to update the node status of the target service node 622 based on the computing information comparison result and the storage information comparison result; and when the updated node status is a pending update status, to generate a node update task for the target service node 622 based on the performance information corresponding to the target resource indicator in at least two resource indicators.

[0167] Furthermore, the service platform 610 is also used to obtain the detected computing performance information and the detected storage performance information of the target service node 622 when the node update task is completed; compare the computing performance information with the detected computing performance information, and compare the storage performance information with the detected storage performance information; and when it is determined based on the computing comparison results and the storage comparison results that the resource optimization of the target service node 622 for resource optimization is completed, create and record the optimization information corresponding to the node update task for the target service node 622.

[0168] Furthermore, the service platform 610 is used to determine a data acquisition time interval based on the node type of the target service node 622; obtain resource usage data corresponding to at least two resource indicators generated by the target service node 622 within the data acquisition time interval; and construct the resource usage data set based on the resource usage data corresponding to at least two resource indicators.

[0169] In summary, the service platform collects statistical information corresponding to each resource indicator based on the resource usage data set submitted by the target service node, and then generates performance information for each resource indicator. By constructing good computing performance information and storage performance information, the resources of the target service node can be optimized. This improves the resource utilization rate of the resources provided by the target service node, enables the resources of the target service node to be reasonably utilized, and avoids resource waste. In addition, for the service provider corresponding to the target service node, the resource utilization status of the resources provided by the target service node can be determined based on the computing performance information, storage performance information, and the statistical information corresponding to each resource indicator, thereby generating node update tasks in a targeted manner, and optimizing the resources of the target service node by executing the node update tasks, thereby saving the cost of the service provider while avoiding resource waste and improving the resource utilization rate of the target service node.

[0170] The above is a schematic diagram of a data processing system according to this embodiment. It should be noted that the technical solution of the data processing system and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the above-mentioned data processing method.

[0171] Figure 7 shows a block diagram of a computing device 700 according to one embodiment of this specification. Components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0172] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0173] In one embodiment of the present specification, the aforementioned components of computing device 700 and other components not shown in FIG. 7 may also be connected to one another, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 7 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0174] Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 700 may also be a mobile or stationary server.

[0175] The processor 720 is configured to execute the following computer-executable instructions, which implement the steps of the above-mentioned data processing method when executed by the processor.

[0176] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned data processing method are of the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the above-mentioned data processing method.

[0177] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0178] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the above-mentioned data processing method.

[0179] An embodiment of the present specification further provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0180] The above is a schematic solution of a computer program product of this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-mentioned data processing method.

[0181] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0182] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0183] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0184] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0185] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied to a service platform for managing and controlling a target service node in a service node cluster, comprising: Obtaining a resource usage dataset of the target service node; Based on the data statistics strategy matched by the target service node, data statistics are performed on the resource usage data corresponding to at least two resource indicators in the resource usage data set to obtain statistical information corresponding to each resource indicator; Determine the performance information corresponding to each resource indicator based on the statistical information and performance calculation information corresponding to each resource indicator; Selecting performance information corresponding to resource indicators corresponding to the computing dimension to construct computing performance information, and selecting performance information corresponding to resource indicators corresponding to the storage dimension to construct storage performance information; A node update task of the target service node is generated according to the computing performance information and the storage performance information, and the node update task is used to optimize resources of the target service node.

2. The data processing method according to claim 1, wherein the at least two resource indicators include: Average CPU metrics, average memory metrics, slow query metrics, storage metrics, storage throughput metrics, and service node attribute metrics; The resource usage data corresponding to the at least two resource indicators include: The average CPU usage data corresponding to the CPU average indicator, the average memory usage data corresponding to the memory average indicator, the slow query usage data corresponding to the slow query indicator, the storage usage data corresponding to the storage indicator, the storage throughput usage data corresponding to the storage throughput indicator, and the service node attribute usage data corresponding to the service node attribute indicator; The statistical information corresponding to the at least two resource indicators includes: The CPU mean statistical information corresponding to the CPU mean indicator, the memory mean statistical information corresponding to the memory mean indicator, the slow query statistical information corresponding to the slow query indicator, the storage statistical information corresponding to the storage indicator, the storage throughput statistical information corresponding to the storage throughput indicator, and the service node attribute statistical information corresponding to the service node attribute indicator; The performance information corresponding to the at least two resource indicators includes: The CPU average performance information corresponding to the CPU average indicator, the memory average performance information corresponding to the memory average indicator, the slow query performance information corresponding to the slow query indicator, the storage performance information corresponding to the storage indicator, the storage throughput performance information corresponding to the storage throughput indicator, and the service node attribute performance information corresponding to the service node attribute indicator.

3. The data processing method according to claim 1, wherein determining the performance information corresponding to any resource indicator comprises: Determine the weight information and threshold information included in the performance calculation information of the target resource indicator; Generate indicator performance information of the target resource indicator based on the statistical information corresponding to the target resource indicator and the threshold information; The performance information corresponding to the target resource indicator is generated based on the indicator performance information and the weight information.

4. The data processing method according to claim 1, wherein selecting performance information corresponding to resource indicators corresponding to the computing dimensions to construct computing performance information comprises: Select a computing resource indicator of a computing dimension from at least two resource indicators; Perform weighted calculation based on the computing indicator weight and computing efficiency information of the computing resource indicator to obtain the computing efficiency information of the computing dimension; The step of selecting performance information corresponding to resource indicators corresponding to storage dimensions to construct storage performance information includes: Selecting a storage resource indicator of a storage dimension from at least two resource indicators; A weighted calculation is performed based on the storage indicator weight of the storage resource indicator and the storage efficiency information to obtain the storage efficiency information of the storage dimension.

5. The data processing method according to claim 1, wherein generating the node update task of the target service node according to the computing performance information and the storage performance information comprises: Constructing node performance information of the target service node according to the computing performance information and the storage performance information; When it is determined based on the node performance information that the target service node meets an update condition, a node update task for the target service node is generated according to the computing performance information and the storage performance information.

6. The data processing method according to claim 1, wherein generating the node update task of the target service node according to the computing performance information and the storage performance information comprises: comparing the computing performance information with reference computing information, and comparing the storage performance information with reference storage information; Updating the node status of the target service node based on the calculation information comparison result and the storage information comparison result; When the updated node state is a pending update state, a node update task of the target service node is generated based on performance information corresponding to a target resource indicator among the at least two resource indicators.

7. The data processing method according to claim 1, after generating the node update task of the target service node according to the computing performance information and the storage performance information, further comprising: When the node update task is completed, obtaining the detection computing efficiency information and the detection storage efficiency information of the target service node; comparing the computing performance information with the detected computing performance information, and comparing the storage performance information with the detected storage performance information; When it is determined based on the calculation comparison result and the storage comparison result that the resource optimization of the target service node for resource optimization is completed, optimization information corresponding to the node update task is created for the target service node and recorded.

8. The data processing method according to claim 1, wherein obtaining the resource usage dataset of the target service node comprises: Determining a data acquisition time interval according to the node type of the target service node; Obtain resource usage data corresponding to at least two resource indicators generated by the target service node within the data acquisition time interval; The resource usage data set is constructed based on resource usage data corresponding to at least two resource indicators.

9. A data processing device, applied to a service platform, wherein the service platform is used to manage and control target service nodes in a service node cluster, comprising: an acquisition module, configured to acquire a resource usage dataset of the target service node; a statistics module configured to perform data statistics on the resource usage data corresponding to at least two resource indicators in the resource usage data set based on the data statistics strategy matched by the target service node, and obtain statistical information corresponding to each resource indicator; a determination module configured to determine the performance information corresponding to each resource indicator based on the statistical information and performance calculation information corresponding to each resource indicator; A construction module is configured to select performance information corresponding to resource indicators corresponding to the computing dimension to construct computing performance information, and select performance information corresponding to resource indicators corresponding to the storage dimension to construct storage performance information; A generating module is configured to generate a node updating task of the target service node according to the computing performance information and the storage performance information, wherein the node updating task is used to optimize resources of the target service node.

10. A data processing system comprising a service platform and a service node cluster; The target service node in the service node cluster is configured to submit a resource usage data set to the service platform in response to a data acquisition request; The service platform is configured to perform data statistics on the resource usage data corresponding to at least two resource indicators in the resource usage data set based on the data statistics strategy matched by the target service node, thereby obtaining statistical information corresponding to each resource indicator; and determine the efficiency information corresponding to each resource indicator based on the statistical information corresponding to each resource indicator and the efficiency calculation information; Selecting performance information corresponding to resource indicators corresponding to the computing dimension to construct computing performance information, and selecting performance information corresponding to resource indicators corresponding to the storage dimension to construct storage performance information; generating a node update task based on the computing performance information and the storage performance information, and sending the node update task to the target service node; The target service node is used to optimize local resources associated with the node update task by executing the node update task.

11. A computing device comprising: memory and processor; The memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions. When the computer program or instructions are executed by the processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium storing a computer program or instruction, wherein the computer program or instruction, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 8.

13. A computer program product, comprising a computer program or instructions, which implements the steps of the data processing method according to any one of claims 1 to 8 when executed by a processor.

Images