Data processing method and device, server and storage medium
By determining the priority and resource utilization of data processing tasks executed by the storage medium, and forming a task queue and execution plan, the problems of high cost and poor stability of storage medium resource expansion are solved, and the stability and cost reduction of data processing are achieved.
Patent Information
- Application Number
- CN202411179567.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-29
AI Technical Summary
In the prior art, it takes a lot of manpower and time to expand storage media resources, resulting in waste of resources and increased economic costs, and it is easy to cause storage media to collapse during peak periods, affecting the stability of data processing tasks.
By determining the priority and resource utilization of data processing tasks performed by the storage media, putting tasks into the task queue based on priority, and forming the current and next execution plans based on resource utilization and thresholds, ensuring that resource utilization is always within the threshold. , avoid storage media crashes.
It improves the stability of data processing, reduces data processing costs, avoids storage media crashes, and ensures the smooth execution of data processing tasks.
Smart Images

Figure CN119045970A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a data processing method, device, server and storage medium. Background Art
[0002] With the development of computer technology and the popularization of the Internet, people have higher and higher requirements for the real-time execution of data processing tasks. Using a real-time computing engine to perform data processing tasks is a good choice. The real-time computing engine can perform real-time calculations and processing on data, and then write the data to the storage medium, providing data writing services for users to write data to the storage medium, and the storage medium can provide an interface to the outside world, providing data reading services for users to read data from the storage medium. Usually, if multiple data tables are stored in the same storage medium, there will be resource conflicts between multiple data tables and resource conflicts in reading and writing the same data table.
[0003] In the prior art, the resources of the storage medium are usually expanded based on manual experience to solve the resource conflict problem.
[0004] In the process of implementing the present invention, the inventors found that there are at least the following technical problems in the prior art:
[0005] Expanding storage media resources requires a lot of manpower and time, resulting in serious waste of resources and increased economic costs. Summary of the invention
[0006] The present invention provides a data processing method, device, server and storage medium, which improve the stability of data processing and reduce the cost of data processing.
[0007] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising:
[0008] Determining the priority and resource utilization of each data processing task executed based on the storage medium;
[0009] Put each of the data processing tasks into a task queue based on priority, and form a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium;
[0010] Determine the data processing instructions corresponding to each of the data processing tasks in the current execution plan and execute each of the data processing instructions;
[0011] After determining that any data processing task in the current execution plan has been completed, a target data processing task is determined in the next execution plan and the newly added data processing tasks, and the target data processing task is added to the current execution plan until all data processing tasks are executed.
[0012] In a second aspect, an embodiment of the present invention further provides a data processing device, the device comprising:
[0013] A first determination module, used to determine the priority and resource utilization of each data processing task executed based on the storage medium;
[0014] A building module, used to put each of the data processing tasks into a task queue based on priority, and to build a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium;
[0015] An execution module, used to determine the data processing instructions corresponding to each of the data processing tasks in the current execution plan and execute each of the data processing instructions;
[0016] The second determination module is used to determine the target data processing task in the next execution plan and the newly added data processing tasks after determining that any data processing task in the current execution plan has been completed, and add the target data processing task to the current execution plan until all data processing tasks are executed.
[0017] In a third aspect, an embodiment of the present invention further provides a server, the server comprising:
[0018] one or more processors;
[0019] a storage device for storing one or more programs,
[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of the first aspects.
[0021] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the data processing method as described in any one of the first aspects.
[0022] The embodiments of the above invention have the following advantages or beneficial effects:
[0023] An embodiment of the present invention provides a data processing method, which includes: determining the priority and resource utilization of each data processing task executed based on the storage medium; placing each of the data processing tasks into a task queue based on the priority, and forming a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium; determining the data processing instructions corresponding to each of the data processing tasks in the current execution plan and executing each of the data processing instructions; after determining that the execution of any data processing task in the current execution plan has been completed, determining a target data processing task in the next execution plan and a newly added data processing task, and adding the target data processing task to the current execution plan until all data processing tasks are executed. The above technical scheme can first determine the data processing tasks that currently need to be executed based on the storage medium, and secondly, can determine the priority and resource utilization of each data processing task according to the task attributes of each data processing task, and then can put each data processing task into the task queue based on the priority, so as to arrange each data processing task according to the priority, and form at least one data processing task that enters the task queue first, that is, has a high priority and a total resource utilization that is not greater than the resource utilization threshold of the storage medium as the current execution plan, and form other data processing tasks in the task queue as the next execution plan, so as to realize the formation of the current execution plan and the next execution plan, and then, after determining the data processing instructions corresponding to each data processing task in the current execution plan, each data processing instruction can be executed simultaneously. The command is used to realize the simultaneous execution of various data processing tasks that have a high priority ranking and a total resource utilization rate that is not greater than the resource utilization threshold of the storage medium among the data processing tasks executed based on the storage medium, and to suspend the execution of other data processing tasks that are executed based on the storage medium. After determining that the execution of any data processing task in the current execution plan is completed, the target data processing task is determined in the next execution plan and the newly added data processing task, and the target data processing task is added to the current execution plan. While executing the data processing tasks that need to be executed based on the storage medium according to the priority, it is ensured that the total resource utilization rate of the data processing tasks executed simultaneously is always less than the resource utilization threshold of the storage medium, thereby avoiding the collapse of the storage medium, improving the stability of data processing, and reducing the data processing cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;
[0025] Figure 2 A flowchart of another data processing method provided by an embodiment of the present invention;
[0026] Figure 3 A schematic diagram of a data processing system provided by an embodiment of the present invention;
[0027] Figure 4 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;
[0028] Figure 5 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0030] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.
[0031] With the development of computer technology and the popularization of the Internet, people have higher and higher requirements for the real-time execution of data processing tasks. For example, logistics systems have very high timeliness requirements for the execution of data processing tasks in terms of waybill management, monitoring dashboards, cargo tracking, distribution scheduling, and abnormal alarms. When executing data processing tasks based on real-time computing engines, the stability of the storage medium is crucial to the smooth completion of data processing tasks based on the storage medium. In order to make better use of the storage medium, multiple data tables are often stored in the same storage medium, which will cause resource conflicts between multiple data tables and resource conflicts caused by multiple data processing tasks accessing the same data table. During the peak period when the real-time computing engine processes data processing tasks, the above-mentioned resource conflict problems often cause the storage medium to crash, resulting in abnormal data processing tasks.
[0032] In the prior art, in order to solve the above problems, the resources of the storage medium are usually expanded several times or even dozens of times based on manual experience. This method requires a lot of manpower and time, resulting in serious waste of resources and increased economic costs.
[0033] Moreover, for the expanded storage medium, multiple data processing tasks still write data to the same storage medium, and multiple data processing tasks still read the same storage medium to obtain data, which still causes the data processing tasks to affect each other and eventually causes the storage medium to crash.
[0034] Therefore, the present application proposes a data processing method, which improves the stability of data processing and reduces the cost of data processing under the premise that multiple data processing tasks share a storage medium.
[0035] The data processing method provided by the present application will be described in detail below with reference to diagrams and embodiments.
[0036] Figure 1 The flowchart of a data processing method provided by an embodiment of the present invention is applicable to situations where the stability of real-time data processing needs to be improved. The method can be executed by a data processing device, which can be implemented by software and / or hardware. Figure 1 Said method specifically comprises the following steps:
[0037] Step 110: Determine the priority and resource utilization of each data processing task executed based on the storage medium.
[0038] Among them, the storage medium can be understood as a database, the data processing task can be a data writing task and / or a data reading task, and the data processing task performed based on the storage medium can be understood as a data processing task that writes data to the storage medium and / or a data processing task that reads data from the storage medium at the current moment.
[0039] The priority of a data writing task is determined by the level system, data volume coefficient, and writing complexity coefficient of the data writing task, and the priority of a data reading task is determined by the level coefficient, data volume coefficient, and reading complexity coefficient of the data reading task. The level system, data volume coefficient, and writing complexity coefficient of a data writing task can be determined by the task attributes of the data writing task, and the level coefficient, data volume coefficient, and reading complexity coefficient of a data reading task can be determined by the task attributes of the data reading task.
[0040] Therefore, after obtaining the task attributes of the data writing task, the level coefficient, data volume coefficient and writing complexity coefficient of the data writing task can be determined according to the task attributes of the data writing task; after obtaining the task attributes of the data reading task, the level coefficient, data volume coefficient and reading complexity coefficient of the data writing task can be determined according to the task attributes of the data reading task.
[0041] Furthermore, the priority coefficient of the data writing task can be determined based on the level coefficient and level weight, data volume coefficient and data volume weight, and write complexity coefficient and complexity weight of the data writing task; and the priority coefficient of the data reading task can be determined based on the level coefficient and level weight, data volume coefficient and data volume weight, and read complexity coefficient and complexity weight of the data reading task.
[0042] After determining the priority coefficient of each data processing task, the priority of each data processing task may be determined according to the principle that the larger the priority coefficient, the higher the priority.
[0043] The resource utilization of the data writing task is determined by the data volume coefficient, the single data volume coefficient and the writing complexity coefficient of the data writing task, and the resource utilization of the data reading task is determined by the data volume coefficient, the single data volume coefficient and the reading complexity coefficient of the data reading task. Therefore, the resource utilization of the data writing task can be determined based on the data volume coefficient and the data volume weight, the single data volume coefficient and the single data volume weight of the data writing task, and the resource utilization of the data reading task can be determined based on the data volume coefficient and the data volume weight, the single data volume coefficient and the single data volume weight, and the reading complexity coefficient and the complexity weight of the data reading task.
[0044] In an embodiment of the present invention, after determining the data processing tasks that currently need to be executed based on the storage medium, the priority and resource utilization of each data processing task are determined according to the task attributes of each data processing task, so as to facilitate the sorting and combination of each data processing task.
[0045] Step 120: put each of the data processing tasks into a task queue based on priority, and form a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium.
[0046] Specifically, first, each data processing task can be sorted based on priority, and each data processing task can be put into the task queue in descending order according to priority. Secondly, the current execution plan can be formed according to the resource utilization of each data processing task in the task queue and the resource utilization threshold of the storage medium. The formation rule is: the N data processing tasks that enter the task queue first are formed into the current execution plan, and the sum of the resource utilization of the N data processing tasks is not greater than the resource utilization threshold of the storage medium, and N is a natural number.
[0047] Of course, after the current execution plan is formed, other data processing tasks in the task queue may be formed into the next execution plan.
[0048] In an embodiment of the present invention, each data processing task is placed in a task queue based on priority, so that each data processing task is arranged according to priority. At least one data processing task that enters the task queue first, i.e., has a high priority and the sum of its resource utilization is not greater than the resource utilization threshold of the storage medium, is formed into the current execution plan, and other data processing tasks in the task queue are formed into the next execution plan, thereby realizing the formation of the current execution plan and the next execution plan.
[0049] Step 130: Determine the data processing instructions corresponding to each of the data processing tasks in the current execution plan and execute each of the data processing instructions.
[0050] Specifically, the data processing instructions corresponding to each data processing task are determined according to the task attributes of each data processing task in the current execution plan, and each data processing instruction is executed simultaneously, so as to realize the simultaneous execution of each data processing task in the current execution plan.
[0051] In an embodiment of the present invention, after determining the data processing instructions corresponding to each data processing task in the current execution plan, each data processing instruction is executed simultaneously, so as to achieve simultaneous execution of each data processing task with a high priority ranking and a sum of resource utilization rates not greater than the resource utilization threshold of the storage medium among the data processing tasks executed based on the storage medium, and temporarily suspend the execution of other data processing tasks executed based on the storage medium, so as to further achieve stable execution of the data processing tasks and avoid the crash of the storage medium.
[0052] Step 140: After determining that any data processing task in the current execution plan has been completed, determine the target data processing task in the next execution plan and the newly added data processing tasks, and add the target data processing task to the current execution plan until all data processing tasks are executed.
[0053] Specifically, since the execution time of each data processing task in the current execution plan is not necessarily the same, the data processing tasks that are started and executed at the same time may not be completed at the same time. After determining that any data processing task in the current execution plan has been completed, the data processing tasks in the next execution plan and / or the newly added data processing tasks can be added to the current execution plan to achieve continuous execution of the data processing tasks.
[0054] Specifically, the target data processing task can be determined among the data processing tasks and the newly added data processing tasks in the next execution plan. The data processing tasks in the current execution plan and the next execution plan are still sorted according to priority. Therefore, first, each newly added data processing task can be inserted into the next execution plan according to the priority of each newly added data processing task. Secondly, the target data processing task can be determined among the data processing tasks sorted from large to small according to priority. The sum of the resource utilization of each target data processing task and the sum of the resource utilization of the data processing tasks being executed in the current execution plan is not greater than the resource utilization threshold of the storage medium. When each target data processing task is added to the current execution plan and executed together with the data processing tasks that have not yet been executed in the current execution plan, the resource utilization of the storage medium still does not exceed its resource utilization threshold, thereby avoiding the collapse of the storage medium.
[0055] In an embodiment of the present invention, after determining that any data processing task in the current execution plan has been completed, the target data processing task can be determined in the next execution plan and the newly added data processing task, and the target data processing task can be added to the current execution plan, so as to realize the execution of data processing tasks that need to be executed based on the storage medium according to priority, and ensure that the resource utilization of the data processing tasks executed simultaneously is always no greater than the resource utilization threshold of the storage medium, thereby avoiding the collapse of the storage medium.
[0056] The data processing method provided by an embodiment of the present invention includes: determining the priority and resource utilization of each data processing task executed based on a storage medium; placing each of the data processing tasks into a task queue based on the priority, and forming a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium; determining the data processing instructions corresponding to each of the data processing tasks in the current execution plan and executing each of the data processing instructions; after determining that the execution of any data processing task in the current execution plan has been completed, determining the target data processing task in the next execution plan and the newly added data processing task, and adding the target data processing task to the current execution plan until all data processing tasks are executed. The above technical scheme can first determine the data processing tasks that currently need to be executed based on the storage medium, and secondly, can determine the priority and resource utilization of each data processing task according to the task attributes of each data processing task, and then can put each data processing task into the task queue based on the priority, so as to sort each data processing task according to the priority, and form at least one data processing task that enters the task queue first, that is, has a high priority and the sum of resource utilization is not greater than the resource utilization threshold of the storage medium as the current execution plan, and form other data processing tasks in the task queue as the next execution plan, so as to realize the formation of the current execution plan and the next execution plan, and then after determining the data processing instructions corresponding to each data processing task in the current execution plan, each data processing instruction can be executed simultaneously, so as to realize Now, among the data processing tasks executed based on the storage medium, the data processing tasks with high priority ranking and the sum of resource utilization rates not greater than the resource utilization threshold of the storage medium are executed simultaneously, and the execution of other data processing tasks executed based on the storage medium is suspended. After determining that the execution of any data processing task in the current execution plan is completed, the target data processing task is determined in the next execution plan and the newly added data processing task, and the target data processing task is added to the current execution plan for execution. While executing the data processing tasks that need to be executed based on the storage medium according to the priority, it is ensured that the sum of resource utilization rates of the data processing tasks executed simultaneously is always less than the resource utilization threshold of the storage medium, thereby avoiding the collapse of the storage medium, improving the stability of data processing, and reducing the data processing cost.
[0057] Figure 2A flowchart of another data processing method provided by an embodiment of the present invention, which can be applied to situations where the stability of real-time data processing needs to be improved. Based on the above embodiment, the embodiment of the present invention adds "obtaining cluster information of the storage medium, the cluster information including at least one cluster parameter; determining that any of the cluster parameters meets the corresponding parameter conditions, determining that the storage medium meets the optimization conditions, and determining the data processing task to be performed based on the storage medium.", wherein the explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. See Figure 2 , the data processing method provided by the embodiment of the present invention includes:
[0058] Step 210: Obtain cluster information of the storage medium.
[0059] The cluster information includes at least one cluster parameter.
[0060] Cluster information can be understood as attribute information of a cluster where storage media is deployed. Cluster parameters include CPU utilization, memory utilization, and resources allocated to each data processing task.
[0061] Specifically, when the storage medium is running, the cluster information of the storage medium is obtained, and specifically at least one of the CPU utilization, memory utilization, and resources allocated to each data processing task can be obtained. Then, the cluster information can be summarized and converted into a JSON format or a serialized DTO object.
[0062] In the embodiment of the present invention, it is achieved that the cluster information of the storage medium is obtained during the operation of the storage medium.
[0063] Step 220: When it is determined that any of the cluster parameters satisfies a corresponding parameter condition, it is determined that the storage medium satisfies an optimization condition, and the data processing task to be executed based on the storage medium is determined.
[0064] Specifically, the configuration information of the storage medium includes parameter conditions of various cluster parameters, the cluster parameters include CPU utilization, memory utilization and resources allocated to data processing tasks of each task level, and the parameter conditions include maximum CPU utilization, maximum memory utilization and maximum allocatable resources for data processing tasks of each task level. Therefore, when it is determined that the storage medium satisfies any one of the following conditions: CPU utilization is greater than maximum CPU utilization, memory utilization is greater than maximum memory utilization, and resources allocated to any data processing task is greater than the maximum allocatable resources for data processing tasks of its task level, it is determined that the storage medium satisfies the optimization condition, and further it can be determined that the performance of the storage medium is relatively poor and all current data processing tasks cannot be executed based on the storage medium. At this point, it can be determined that the data processing tasks that need to be executed based on the storage medium.
[0065] It should be noted that, when it is determined that the storage medium does not meet the optimization condition, it can be determined that the storage medium has good performance and all current data processing tasks can be performed based on the storage medium. At this time, each data processing task can be directly performed based on the storage medium, or step 230 can be performed to further optimize the execution of each data processing task.
[0066] In the embodiment of the present invention, when it is determined that any cluster parameter satisfies the corresponding parameter condition, it is determined that the storage medium satisfies the optimization condition. At this time, the data processing task that needs to be executed based on the storage medium can be determined, thereby determining the data processing task that needs to be optimized for execution.
[0067] Step 230: Determine the priority and resource utilization of each data processing task executed based on the storage medium.
[0068] In one implementation, step 230 may specifically include:
[0069] Determine the level coefficient, data volume coefficient and complexity coefficient of each data processing task according to the task attributes of each data processing task; determine the priority of each data processing task according to the level coefficient, data volume coefficient and complexity coefficient of each data processing task; determine the resource utilization of each data processing task according to the data volume coefficient and complexity coefficient of each data processing task.
[0070] Further, determining the priority of each of the data processing tasks according to the level coefficient, the data volume coefficient and the complexity coefficient of each of the data processing tasks includes:
[0071] The priority of each of the data processing tasks is determined based on the level coefficient and level weight, data volume coefficient and data volume weight, and complexity coefficient and complexity weight of each of the data processing tasks.
[0072] Further, determining the resource utilization of each of the data processing tasks according to the data volume coefficient and the complexity coefficient of each of the data processing tasks includes:
[0073] Determine the resource utilization of each of the data processing tasks based on the data volume coefficient and data volume weight as well as the complexity coefficient and complexity weight of each of the data processing tasks; or, input the data volume coefficient and the complexity coefficient of each of the data processing tasks into a pre-trained resource utilization determination model so that the resource utilization determination model determines the resource utilization of each of the data processing tasks, wherein the resource utilization model is trained based on the data volume coefficient, complexity coefficient and resource utilization corresponding to multiple sample data processing tasks.
[0074] The task attributes of a data writing task include the allowed delay duration, the allowed backlog of data, the number of data rows written and the number of fields in each row of data, the amount of data written for a single piece of data, the data writing method, the data generation speed, the data writing speed, the data writing interval, the data writing batch, the amount of data in batches, the write parallelism, etc.; the task attributes of a data reading task include the number of data rows read and the number of fields in each row of data, the number and association of data tables read, the read response time, the data reading efficiency, etc.
[0075] The level coefficient, data volume coefficient, write complexity coefficient and read complexity coefficient of the data processing task are determined by the task attributes of the data processing task and the assignment method of the level coefficient and data volume coefficient of the data processing task of various pre-set task attributes, as well as the calculation method of the write complexity coefficient and the read complexity coefficient.
[0076] Data processing tasks include data writing tasks and / or data reading tasks. Data writing tasks can perform logical processing such as windowing, filtering, conversion, processing, aggregation, and diversion on the data in the message queue, and store the processed data in the storage medium. Data reading tasks can provide an interface to the outside world to support other servers to call to read the data stored in the storage medium. In addition, the input parameters of the data reading service can be parameters such as date and dimension, and the reading statement can be SQL / DSL, etc.
[0077] It should be noted that the data writing task can be executed based on the real-time computing engine, and the real-time computing engine can provide an interface, and based on the interface, the task attributes of a batch of data writing tasks or each data writing task can be exposed to customization. For example, the task attributes of a batch of data writing tasks or each data writing task can be exposed based on the RestAPI, and the task attributes exposed based on the RestAPI can be customized according to actual needs.
[0078] After acquiring the data exposed by the real-time computing engine by calling the Rest API, the task attributes of each data writing task can be determined in the acquired data according to the task identifier of each data writing task, and after data calculation, aggregation, filtering, windowing, etc., the processed task attributes are converted into JSON format or serialized DTO objects. When processing the task attributes, the task level of the data writing task can be determined according to the allowed delay time and the allowed backlog data volume of the data writing task, and then the level coefficient, data volume coefficient, and writing complexity coefficient of the data writing task can be determined in combination with the pre-set assignment method of the level coefficient and data volume coefficient of the data writing task and the calculation method of the writing complexity coefficient.
[0079] In actual applications, when the allowable delay market for data writing tasks is less than 2s or the maximum allowable backlog data volume is less than 5w, the task level of the data writing task can be determined as 1; when the allowable delay market for data writing tasks is less than 30s or the maximum allowable backlog data volume is less than 3 million, the task level of the data writing task can be determined as 2; when the allowable delay market for data writing tasks is not less than 30s or the maximum allowable backlog data volume is not less than 5w, the task level of the data writing task can be determined as 3.
[0080] After obtaining the task attributes of the data reading task, you can also perform data calculation, aggregation, filtering, windowing and other processing on the task attributes, and convert the processed task attributes into JSON format or serialized DTO objects. When processing the task attributes, you can determine the task level of the data reading task based on the response time and reading efficiency of the data reading task, and then combine the pre-set assignment method of the level coefficient and data volume coefficient of the data reading task and the calculation method of the reading complexity coefficient to determine the level coefficient, data volume coefficient and reading complexity coefficient of the data reading task.
[0081] For example, when the task level of the data processing task is L0, its level coefficient can be assigned a value of 2; when the task level of the data processing task is L1, its level coefficient can be assigned a value of 1.5; when the task level of the data processing task is L3, its level coefficient can be assigned a value of 0.8. Determine the number of data rows and the number of fields of each row of data corresponding to the data processing task, and assign a value to the data volume coefficient of the data processing task according to the reciprocal of the product of the number of data rows and the number of fields of each row of data. When the product of the number of data rows and the number of fields of each row of data corresponding to the data processing task is 100w, its data volume coefficient is assigned a value of 1, and when the product of the number of data rows and the number of fields of each row of data is 1000w, its data volume coefficient is assigned a value of 0.1. For the data writing task, the writing complexity coefficient of the data writing task can be determined according to the writing frequency and the first weight, the data volume of the single data written and the second weight, the assignment corresponding to the writing method and the third weight. For the data reading task, the reading complexity coefficient of the data reading task can be determined according to the number and association of the data tables read. When the data table to be read is one, the read complexity coefficient is determined to be 1, and when the data tables to be read are two interrelated data tables, the read complexity coefficient is determined to be 1.5. In practical applications, the read complexity can also be determined based on the number of data tables read, the number of hit data table indexes, the number of hit data table partitions, and the number of interrelated data tables, and the read complexity of the data reading task can be determined based on a pre-trained read complexity determination model. The read complexity determination model is obtained based on multiple sample data reading tasks and the number of data tables read by each sample data reading task, the number of hit data table indexes, the number of hit data table partitions, the number of interrelated data tables, and the read complexity training, and is used to determine the read complexity of the data reading task according to the number of data tables read, the number of hit data table indexes, the number of hit data table partitions, and the number of interrelated data tables of the data reading task.
[0082] After determining the level coefficient, data volume coefficient, write complexity coefficient of the data writing task and the level coefficient, data volume coefficient, read complexity coefficient of the data reading task, the priority coefficient of the data writing task can be determined based on the level coefficient and level weight, data volume coefficient and data volume weight, write complexity coefficient and complexity weight of the data writing task, and the priority coefficient of the data reading task can be determined based on the level coefficient and level weight, data volume coefficient and data volume weight, read complexity coefficient and complexity weight of the data reading task.
[0083] Furthermore, the priority of each data processing task can be determined according to the principle that the larger the priority coefficient, the higher the priority.
[0084] In practical applications, the priority of data processing tasks can also be determined by referring to the table level of the data table written by the data writing task and the table level of the data table read by the data reading task. The table level of the data table can be determined according to the allowed delay time and the allowed backlog data volume.
[0085] It can be understood that resource utilization is CPU utilization, memory utilization or IO utilization. The calculation of resource utilization is determined by the data volume coefficient and complexity coefficient of the data processing task. On the one hand, the resource utilization of the data processing task can be determined according to the data volume coefficient and data volume weight of the data processing task as well as the complexity coefficient and complexity weight; on the other hand, the data volume coefficient and complexity coefficient of the data processing task can be input into a pre-trained resource utilization determination model so that the resource utilization model determines the resource utilization of the data processing task according to the data volume coefficient and complexity coefficient of the data processing task. It should be noted that the resource utilization model is obtained by training the data volume coefficient, complexity coefficient and resource utilization corresponding to multiple sample data processing tasks, and is used to determine the resource utilization of the data processing task according to the data volume coefficient and complexity coefficient of the data processing task. In addition, the training of the resource utilization model is obtained by supervised training using a tree algorithm or a regression algorithm, which can determine a more accurate resource utilization of the data processing task.
[0086] In the embodiment of the present invention, the priority and resource utilization of each data processing task are determined according to the task attributes of each data processing task, so as to facilitate the sorting and combination of each data processing task.
[0087] Step 240: put each of the data processing tasks into a task queue based on priority, and form a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium.
[0088] In one implementation, step 240 may specifically include:
[0089] Put each of the data processing tasks into the task queue in descending order of priority; form the first N data processing tasks in the task queue into the current execution plan, wherein the sum of the resource utilization rates of the first N data processing tasks is not greater than the resource utilization threshold, and N is a natural number; and form the remaining data processing tasks in the task queue into the next execution plan.
[0090] The resource utilization may be CPU utilization, memory utilization or IO utilization, and the resource utilization threshold may be maximum CPU utilization, maximum memory utilization or maximum IO utilization.
[0091] Specifically, first, each data processing task can be placed in the task queue in order from high to low priority, and secondly, the resource utilization of each data processing task in the task queue can be accumulated in order from high to low priority, and N data processing tasks whose sum of resource utilization is not greater than the resource utilization threshold of the storage medium are formed into the current execution plan, and the remaining data processing tasks in the task queue are formed into the next execution plan.
[0092] For example, the data processing tasks sorted from high to low in priority are: data reading task 1 (8%), data writing task 1 (2%), data reading task 2 (15%), data reading task 3 (15%), data writing task 2 (11%), data writing task 3 (15%), data writing task 4 (10%), data reading task 4 (15%), data reading task 5 (9%), where the data in () are the CPU utilization rates of each data processing task. When the CPU utilization threshold is 70%, the sum of the CPU utilizations of data reading task 1, data writing task 1, data reading task 2, data reading task 3, data writing task 2, and data writing task 3 is 66%, and the sum of the CPU utilizations of data reading task 1, data writing task 1, data reading task 2, data reading task 3, data writing task 2, data writing task 3, and data writing task 4 is 76%. Therefore, data reading task 1, data writing task 1, data reading task 2, data reading task 3, data writing task 2, and data writing task 3 can be formed into the current execution plan, and data writing task 4, data reading task 4, and data reading task 5 can be formed into the next execution plan.
[0093] In an embodiment of the present invention, each data processing task is placed in a task queue based on priority, so that each data processing task is arranged according to priority. At least one data processing task that enters the task queue first, i.e., has a high priority and a total resource utilization that is not greater than the resource utilization threshold of the storage medium, is formed into the current execution plan, and other data processing tasks in the task queue are formed into the next execution plan, thereby realizing the formation of the current execution plan and the next execution plan.
[0094] Step 250: Determine the data processing instructions corresponding to each of the data processing tasks in the current execution plan and execute each of the data processing instructions.
[0095] In one implementation, determining the data processing instructions corresponding to each of the data processing tasks in the current execution plan includes:
[0096] The data processing instructions corresponding to each of the data processing tasks are determined according to the task attributes of each of the data processing tasks.
[0097] Among them, when the data processing task is a data writing task, the data processing instructions can be determined according to the task attributes including the number of batches, batch threshold, batch interval, and write parallelism. When the data processing task is a data reading task, the data processing instructions can be determined according to the task attributes including the query status.
[0098] Specifically, the data write instruction may be determined based on Table 1.
[0099] Table 1
[0100] Parameter configuration instruction Number of batches sink.buffer-flush.max-rows Batch Threshold sink.buffer-flush.max-size Batch interval sink.buffer-flush.interval Write parallelism sink.parallelism
[0101] After determining that the query status of the data reading task is the current query, it may be determined that the status of the data reading instruction is the execution instruction.
[0102] Furthermore, each data processing instruction in the current execution plan can be executed simultaneously, thereby realizing the simultaneous execution of each data processing task in the current execution plan.
[0103] In an embodiment of the present invention, after determining the data processing instructions corresponding to each data processing task in the current execution plan, each data processing instruction is executed simultaneously, so as to achieve simultaneous execution of each data processing task with a high priority ranking and a sum of resource utilization rates not greater than the resource utilization threshold of the storage medium among the data processing tasks executed based on the storage medium, and temporarily suspend the execution of other data processing tasks executed based on the storage medium, so as to further achieve stable execution of the data processing tasks and avoid the crash of the storage medium.
[0104] Step 260: After determining that any data processing task in the current execution plan has been completed, determine the target data processing task in the next execution plan and the newly added data processing tasks, and add the target data processing task to the current execution plan until all data processing tasks are executed.
[0105] In one implementation, determining the target data processing task in the next execution plan and the newly added data processing task includes:
[0106] Determine the total resource utilization of each of the data processing tasks being executed in the current execution plan, and determine the resource utilization gap based on the resource utilization threshold and the total resource utilization; determine the target data processing task in the next execution plan and the newly added data processing task based on the resource utilization gap.
[0107] Further, determining the target data processing task in the next execution plan and the newly added data processing task according to the resource utilization gap includes:
[0108] Sort the data processing tasks and the newly added data processing tasks in the next execution plan according to priority from high to low; determine the first M data processing tasks as the target data tasks, wherein the sum of the resource utilization of the first M data processing tasks is not greater than the resource utilization gap, and M is a natural number.
[0109] Specifically, after determining that any data processing task in the current execution plan has been completed, the total resource utilization of each data processing task being executed in the current execution plan can be determined, and the resource utilization gap can be determined based on the resource utilization threshold and the total resource utilization. Then, the data processing tasks and the newly added data processing tasks in the next execution plan can be sorted from high to low based on the priority, and the newly added data processing tasks can be inserted into the data processing tasks of the next execution plan sorted from high to low according to the priority of each newly added data processing task, and the priority of each newly added data processing task is less than the priority of the data processing task in front of it, and greater than the priority of the data processing tasks in front and behind. The resource utilization of each data processing task and / or the newly added data processing task in the next execution plan is accumulated from high to low priority, and M data processing tasks whose sum of resource utilization is not greater than the resource utilization gap are determined as target data processing tasks.
[0110] Furthermore, each target data processing task can be added to the current execution plan and executed together with the data processing tasks in the current execution plan that have not yet been completed. When each target data processing task is executed together with the data processing tasks in the current execution plan that have not yet been completed, the resource utilization of the storage medium still does not exceed its resource utilization threshold, thereby avoiding the crash of the storage medium.
[0111] It should be noted that in order to avoid the data processing tasks with higher priority and resource utilization in the next execution plan from being unable to be executed for a long time due to resource limitations, the priority of these data processing tasks can be increased and the waiting time acceptance value can be lowered to ensure priority execution of these data processing tasks.
[0112] In addition, after the data processing tasks and / or newly added data processing tasks in the next execution plan are added as target data processing tasks to the current execution plan, it is necessary to determine the data processing instructions corresponding to the target data processing tasks.
[0113] Specifically, when the target data processing task is a data writing task, the data writing instruction corresponding to the data writing task can be determined according to the processing progress, processing result and processing status of the data writing task, and then the data writing instruction can be continued to be executed based on the processing progress of the data writing task to realize the execution of the data writing task. It should be noted that the processing progress, processing result and processing status of the data writing task can all be obtained by calling RestAPI. RestAPI is an interface pre-set in the real-time computing engine for obtaining the task attributes of the data writing task. The task attributes may include processing progress, processing results, processing status, etc. When the target data processing task is a data reading task, the status of the data reading instruction is changed to an execution instruction to realize the execution of the data reading task.
[0114] In an embodiment of the present invention, after determining that any data processing task in the current execution plan has been completed, the target data processing task can be determined in the next execution plan and the newly added data processing task, and the target data processing task can be added to the current execution plan, so as to execute the data processing tasks that need to be executed based on the storage medium according to the priority, and ensure that the total resource utilization of the data processing tasks executed simultaneously is always no more than the resource utilization threshold of the storage medium, thereby avoiding the collapse of the storage medium.
[0115] The data processing method provided by an embodiment of the present invention includes: obtaining cluster information of the storage medium; determining that the storage medium satisfies the optimization condition when any of the cluster parameters satisfies the corresponding parameter condition, and determining the data processing task executed based on the storage medium; determining the priority and resource utilization of each data processing task executed based on the storage medium; placing each of the data processing tasks into a task queue based on the priority, and forming a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium; determining the data processing instructions corresponding to each of the data processing tasks in the current execution plan and executing each of the data processing instructions; after determining that the execution of any data processing task in the current execution plan has been completed, determining the target data processing task in the next execution plan and the newly added data processing task, and adding the target data processing task to the current execution plan until all data processing tasks are executed. The above technical scheme can firstly obtain the cluster information of the storage medium during the operation of the storage medium. When it is determined that any cluster parameter satisfies the corresponding parameter condition, it is determined that the storage medium satisfies the optimization condition. At this time, the data processing task that currently needs to be executed based on the storage medium can be determined. Secondly, the priority and resource utilization of each data processing task can be determined according to the task attributes of each data processing task. Thirdly, each data processing task can be put into a task queue based on the priority, so that each data processing task can be arranged according to the priority. At least one data processing task that enters the task queue first, that is, has a high priority and the sum of its resource utilization is not greater than the resource utilization threshold of the storage medium, is formed into the current execution plan, and other data processing tasks in the task queue are formed into the next execution plan, so as to realize the formation of the current execution plan and the next execution plan, and then the current execution plan can be determined. After the data processing instructions corresponding to each data processing task in the plan are issued, each data processing instruction is executed simultaneously to achieve simultaneous execution of each data processing task with a high priority ranking and a sum of resource utilization rates not greater than the resource utilization threshold of the storage medium among the data processing tasks executed based on the storage medium, and the execution of other data processing tasks executed based on the storage medium is suspended. After determining that the execution of any data processing task in the current execution plan is completed, the target data processing task is determined in the next execution plan and the newly added data processing task, and the target data processing task is added to the current execution plan. While executing the data processing tasks that need to be executed based on the storage medium according to the priority, it is ensured that the sum of the resource utilization rates of the data processing tasks executed simultaneously is always less than the resource utilization threshold of the storage medium, thereby avoiding the collapse of the storage medium, improving the stability of data processing, and reducing the data processing cost.
[0116] In addition, according to the task attributes of each data processing task and the cluster information of the storage medium, the execution of each data processing task is adaptively adjusted to achieve stable execution of each data processing task under the premise that multiple data processing tasks share the storage medium, which not only saves resources but also ensures the stable execution of each data processing task.
[0117] Figure 3 A schematic diagram of a data processing system provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the data processing system includes a data writing server, a data storage server, a data reading server, a storage monitoring server, a processing monitoring server, an instruction computing server and an instruction execution server. The data writing server is used to write data to a data storage service. The data storage server is used to store data. The data reading service is used to read data from the data storage service. The storage monitoring server is used to obtain cluster information of the data storage service and send the cluster information to the instruction computing service. The processing monitoring server is used to obtain cluster information, data lineage information and data writing information of the data writing service. The data writing information at least includes task attributes of each data writing task executed by the data writing service. The processing monitoring server is also used to obtain data reading information of the data reading service. The data reading information at least includes task attributes of each data reading task executed by the data reading service. , and is also used to send the cluster information, data lineage information and data writing information of the data writing service and the data reading information of the data reading service to the instruction computing service. The instruction computing server is used to determine the current execution plan and the next execution plan according to the task attributes of each data writing task and each data reading task, and determine the data instructions corresponding to each data processing task included in the current execution plan, and send each data instruction to the instruction execution server for execution to realize the execution of the current execution plan. Moreover, at least one data processing task and / or a new data processing task in the next execution plan can be added to the current execution plan, and after determining the data instructions corresponding to each data processing task, each data instruction can be sent to the instruction execution server for execution to realize the continuous execution of data processing tasks until all data processing tasks are executed.
[0118] The data processing system provided by the embodiment of the present invention can execute the data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data processing method.
[0119] Figure 4 The present invention provides a schematic diagram of the structure of a data processing device. The device and the data processing methods of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the data processing device, reference can be made to the embodiments of the above data processing methods.
[0120] The specific structure of the data processing device is as follows: Figure 4 As shown, including:
[0121] A first determination module 410, configured to determine the priority and resource utilization of each data processing task executed based on the storage medium;
[0122] A building module 420, configured to place each of the data processing tasks into a task queue based on a priority, and to build a current execution plan and a next execution plan according to a resource utilization rate of each of the data processing tasks in the task queue and a resource utilization threshold of the storage medium;
[0123] An execution module 430, configured to determine a data processing instruction corresponding to each of the data processing tasks in the current execution plan and execute each of the data processing instructions;
[0124] The second determination module 440 is used to determine the target data processing task in the next execution plan and the newly added data processing tasks after determining that any data processing task in the current execution plan has been completed, and add the target data processing task to the current execution plan until all data processing tasks are executed.
[0125] Based on the above embodiment, the device further includes:
[0126] An acquisition module is used to obtain cluster information of the storage medium, wherein the cluster information includes at least one cluster parameter; when it is determined that any of the cluster parameters satisfies a corresponding parameter condition, it is determined that the storage medium satisfies an optimization condition, and the data processing task executed based on the storage medium is determined.
[0127] Based on the above embodiment, the first determining module 410 is specifically configured to:
[0128] Determine the level coefficient, data volume coefficient and complexity coefficient of each data processing task according to the task attributes of each data processing task; determine the priority of each data processing task according to the level coefficient, data volume coefficient and complexity coefficient of each data processing task; determine the resource utilization of each data processing task according to the data volume coefficient and complexity coefficient of each data processing task.
[0129] Further, determining the priority of each of the data processing tasks according to the level coefficient, the data volume coefficient and the complexity coefficient of each of the data processing tasks includes:
[0130] The priority of each of the data processing tasks is determined based on the level coefficient and level weight, data volume coefficient and data volume weight, and complexity coefficient and complexity weight of each of the data processing tasks.
[0131] Further, determining the resource utilization of each of the data processing tasks according to the data volume coefficient and the complexity coefficient of each of the data processing tasks includes:
[0132] Determine the resource utilization of each of the data processing tasks based on the data volume coefficient and data volume weight as well as the complexity coefficient and complexity weight of each of the data processing tasks; or, input the data volume coefficient and the complexity coefficient of each of the data processing tasks into a pre-trained resource utilization determination model so that the resource utilization determination model determines the resource utilization of each of the data processing tasks, wherein the resource utilization model is trained based on the data volume coefficient, complexity coefficient and resource utilization corresponding to multiple sample data processing tasks.
[0133] Based on the above embodiment, the module 420 is constructed to:
[0134] Put each of the data processing tasks into the task queue in descending order of priority; form the first N data processing tasks in the task queue into the current execution plan, wherein the sum of the resource utilization rates of the first N data processing tasks is not greater than the resource utilization threshold, and N is a natural number; and form the remaining data processing tasks in the task queue into the next execution plan.
[0135] Based on the above embodiment, the execution module 430 is specifically used for:
[0136] The data processing instructions corresponding to each of the data processing tasks are determined according to the task attributes of each of the data processing tasks.
[0137] Based on the above embodiment, the second determining module 440 is specifically configured to:
[0138] After determining that any data processing task in the current execution plan has been completed, determine the total resource utilization of each data processing task being executed in the current execution plan, and determine the resource utilization gap based on the resource utilization threshold and the total resource utilization; determine the target data processing task in the next execution plan and the newly added data processing task based on the resource utilization gap; and add the target data processing task to the current execution plan until all data processing tasks are executed.
[0139] In one implementation, determining the target data processing task in the next execution plan and the newly added data processing task according to the resource utilization gap includes:
[0140] Sort the data processing tasks and the newly added data processing tasks in the next execution plan according to priority from high to low; determine the first M data processing tasks as the target data tasks, wherein the sum of the resource utilization of the first M data processing tasks is not greater than the resource utilization gap, and M is a natural number.
[0141] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data processing method.
[0142] It is worth noting that in the embodiment of the above-mentioned data processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0143] Figure 5 A schematic diagram of the structure of a server provided in an embodiment of the present invention. Figure 5 A block diagram of an exemplary server 5 suitable for implementing embodiments of the present invention is shown. Figure 5 The server 5 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0144] like Figure 5 As shown, the server 5 is in the form of a general purpose computing electronic device. The components of the server 5 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
[0145] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0146] The server 5 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the server 5, including volatile and non-volatile media, removable and non-removable media.
[0147] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The server 5 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0148] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0149] The server 5 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the server 5, and / or any device that enables the server 5 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the server 5 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. Figure 5 As shown, the network adapter 20 communicates with other modules of the server 5 via the bus 18. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the server 5, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0150] The processing unit 16 executes various functional applications and page displays by running the programs stored in the system memory 28, for example, implementing the data processing method provided by the embodiment of the present invention, which includes:
[0151] Determine the priority and resource utilization of each data processing task performed based on the storage medium;
[0152] Put each of the data processing tasks into a task queue based on priority, and form a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium;
[0153] Determine the data processing instructions corresponding to each of the data processing tasks in the current execution plan and execute each of the data processing instructions;
[0154] After determining that any data processing task in the current execution plan has been completed, a target data processing task is determined in the next execution plan and the newly added data processing tasks, and the target data processing task is added to the current execution plan until all data processing tasks are executed.
[0155] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data processing method provided by any embodiment of the present invention.
[0156] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, for example, a data processing method provided by an embodiment of the present invention is implemented. The method includes:
[0157] Determine the priority and resource utilization of each data processing task performed based on the storage medium;
[0158] Put each of the data processing tasks into a task queue based on priority, and form a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium;
[0159] Determine the data processing instructions corresponding to each of the data processing tasks in the current execution plan and execute each of the data processing instructions;
[0160] After determining that any data processing task in the current execution plan has been completed, a target data processing task is determined in the next execution plan and the newly added data processing tasks, and the target data processing task is added to the current execution plan until all data processing tasks are executed.
[0161] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0162] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0163] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0164] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0165] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0166] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A data processing method, characterized in that: The method comprises: Determine the priority and resource utilization of each data processing task performed based on the storage medium; Put each of the data processing tasks into a task queue based on priority, and form a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium; Determine the data processing instructions corresponding to each of the data processing tasks in the current execution plan and execute each of the data processing instructions; After determining that any data processing task in the current execution plan has been completed, a target data processing task is determined in the next execution plan and the newly added data processing tasks, and the target data processing task is added to the current execution plan until all data processing tasks are executed.
2. The data processing method according to claim 1, characterized in that: Also includes: Acquire cluster information of the storage medium, where the cluster information includes at least one cluster parameter; When it is determined that any of the cluster parameters satisfies the corresponding parameter condition, it is determined that the storage medium satisfies the optimization condition, and the data processing task to be executed based on the storage medium is determined.
3. The data processing method according to claim 1, characterized in that: Determine the priority and resource utilization of each data processing task performed based on the storage media, including: Determining a level coefficient, a data volume coefficient and a complexity coefficient of each of the data processing tasks according to the task attributes of each of the data processing tasks; Determining the priority of each of the data processing tasks according to the level coefficient, the data volume coefficient and the complexity coefficient of each of the data processing tasks; The resource utilization of each of the data processing tasks is determined according to the data volume coefficient and the complexity coefficient of each of the data processing tasks.
4. The data processing method according to claim 3, characterized in that: Determining the priority of each of the data processing tasks according to the level coefficient, the data volume coefficient, and the complexity coefficient of each of the data processing tasks includes: The priority of each of the data processing tasks is determined based on the level coefficient and level weight, data volume coefficient and data volume weight, and complexity coefficient and complexity weight of each of the data processing tasks.
5. The data processing method according to claim 3, characterized in that: Determining the resource utilization of each of the data processing tasks according to the data volume coefficient and the complexity coefficient of each of the data processing tasks includes: Determining resource utilization of each of the data processing tasks according to the data volume coefficient and the data volume weight and the complexity coefficient and the complexity weight of each of the data processing tasks; Alternatively, the data volume coefficient and the complexity coefficient of each data processing task are input into a pre-trained resource utilization determination model so that the resource utilization determination model determines the resource utilization of each data processing task, wherein the resource utilization model is trained based on the data volume coefficient, complexity coefficient and resource utilization corresponding to multiple sample data processing tasks.
6. The data processing method according to claim 1, characterized in that: Putting each of the data processing tasks into a task queue based on priority, and forming a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium, including: Putting each of the data processing tasks into the task queue in order from high to low priority; The first N data processing tasks in the task queue are formed into the current execution plan, wherein the sum of resource utilization rates of the first N data processing tasks is not greater than the resource utilization threshold, and N is a natural number; The remaining data processing tasks in the task queue are formed into the next execution plan.
7. The data processing method according to claim 1, characterized in that: Determining the data processing instructions corresponding to each of the data processing tasks in the current execution plan includes: The data processing instructions corresponding to each of the data processing tasks are determined according to the task attributes of each of the data processing tasks.
8. The data processing method according to claim 1, characterized in that: Determining the target data processing task in the next execution plan and the newly added data processing task includes: Determine the total resource utilization of each of the data processing tasks being executed in the current execution plan, and determine the resource utilization gap according to the resource utilization threshold and the total resource utilization; The target data processing task is determined in the next execution plan and the newly added data processing task according to the resource utilization gap.
9. The data processing method according to claim 8, characterized in that: Determining the target data processing task in the next execution plan and the newly added data processing task according to the resource utilization gap includes: Sorting the data processing tasks and the newly added data processing tasks in the next execution plan according to their priorities from high to low; The first M data processing tasks are determined as the target data tasks, wherein the sum of the resource utilizations of the first M data processing tasks is not greater than the resource utilization gap, and M is a natural number.
10. A data processing device, characterized in that: The device comprises: A first determination module, used to determine the priority and resource utilization of each data processing task currently executed based on the storage medium; A building module, used to put each of the data processing tasks into a task queue based on priority, and to build a current execution plan and a next execution plan according to the resource utilization of each of the data processing tasks in the task queue and the resource utilization threshold of the storage medium; An execution module, used to determine the data processing instructions corresponding to each of the data processing tasks in the current execution plan and execute each of the data processing instructions; The second determination module is used to determine the target data processing task in the next execution plan and the newly added data processing tasks after determining that any data processing task in the current execution plan has been completed, and add the target data processing task to the current execution plan until all data processing tasks are executed.
11. A server, characterized in that: The server comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 9.
12. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the data processing method according to any one of claims 1 to 9.
