Data processing method and device and computer storage medium

Abstract

The invention provides a data processing method and device and a computer storage medium. The data processing method comprises the steps that an identification file comprising at least part of historically processed tasks is obtained, and an intermediate calculation result corresponding to each task in the identification file is stored in a target storage; determining a first target task which is not recorded in the identification file and a second target task which is recorded in the identification file in a plurality of to-be-processed tasks corresponding to the current processing stage of the query process; executing each first target task to obtain an intermediate calculation result corresponding to the first target task; the intermediate calculation result corresponding to the second target task is determined according to the intermediate calculation result stored in the target memory, through the data processing method, the processing efficiency of batch processing query is improved, and then the completion period of the batch processing query is shortened to meet the real-time requirement of a service for a data analysis result.

Description

Technical Field

[0001] This application relates to the field of big data analytics, and in particular to a data processing method, apparatus, and computer storage medium. Background Technology

[0002] Batch querying refers to a querying method that processes large amounts of data in a centralized manner according to a pre-defined task logic. As one of the core scenarios of big data analysis, batch querying is widely used in many fields such as data warehouse construction, offline statistical analysis, and data mining modeling. Trino, as a high-performance distributed SQL (Structured Query Language) query engine, has received widespread attention and application in batch querying scenarios.

[0003] The Trino Tardigrade architecture is an upgrade of the traditional Trino, that is, a query architecture obtained by adding capabilities to the original Trino framework. Compared with the original Trino framework, it improves the fault tolerance and stability in batch query scenarios. However, in the batch query scenario of online analytical processing (OLAP), with the explosive growth of business data, the amount of data for each operation is relatively large and the query is relatively complex. The execution time of each processing stage is generally long, which makes the completion cycle of batch query too long and cannot meet the business's real-time requirements for data analysis results. Summary of the Invention

[0004] To address the aforementioned technical problems, this application proposes a data processing method, an apparatus, and a computer storage medium.

[0005] To address the aforementioned technical problems, this application proposes a data processing method, comprising: acquiring an identifier file including at least a portion of historically processed tasks, wherein intermediate calculation results corresponding to each task in the identifier file are stored in a target memory; determining a first target task not recorded in the identifier file and a second target task recorded in the identifier file among a plurality of pending tasks corresponding to the current processing stage of a query process; executing each of the first target tasks to obtain an intermediate calculation result corresponding to the first target task; and determining an intermediate calculation result corresponding to the second target task based on the intermediate calculation results stored in the target memory.

[0006] The method further includes, after executing each of the first target tasks and obtaining the intermediate calculation results corresponding to the first target task, obtaining the task complexity of the first target task and the data volume of the corresponding intermediate calculation results; and determining whether to add the first target task to the identifier file based on the task complexity and the data volume.

[0007] The step of determining whether to add the first target task to the identifier file based on the task complexity and the data volume includes: adding the first target task to the identifier file when the task complexity is greater than a preset complexity threshold and the data volume is less than a preset data volume, and storing the intermediate calculation results corresponding to the first target task in the target memory.

[0008] The process of obtaining the task complexity of the first target task includes: obtaining the task type and metadata information corresponding to the first target task, wherein the metadata information includes the data processing volume corresponding to the first target task; and determining the task complexity based on the task type and the data processing volume.

[0009] The method further includes, after the first target task is added to the identifier file, determining a time threshold corresponding to the first target task based on the task complexity, wherein the time threshold represents the maximum duration for which intermediate calculation results corresponding to the first target task are statically stored in the target memory.

[0010] The method further includes: determining the target duration between the current time and the most recent invocation time of each task in the identifier file, and obtaining the time threshold corresponding to each task in the identifier file; deleting tasks whose target duration exceeds the corresponding time threshold from the identifier file, and deleting the corresponding intermediate calculation results from the target memory.

[0011] Determining the target duration between the current time and the most recent invocation time of each task in the identifier file includes: obtaining the lifecycle value corresponding to each task in the identifier file at the current time; determining the lifecycle value as the corresponding target duration; wherein, when any task in the identifier file is invoked, the lifecycle value corresponding to that task is cleared to zero and recalculated.

[0012] The method further includes: obtaining the call frequency of intermediate calculation results corresponding to each task in the identifier file within a preset time period; deleting tasks with a call frequency less than a frequency threshold from the identifier file, and deleting the corresponding intermediate calculation results from the target memory.

[0013] To address the aforementioned technical problems, this application also proposes a data processing device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the data processing method described above.

[0014] To address the aforementioned technical problems, this application also proposes a computer storage medium for storing program data, which, when executed by a computer, is used to implement the aforementioned data processing method.

[0015] Compared with the prior art, the beneficial effects of this application are as follows: intermediate calculation results of historically processed tasks are stored in the target memory, and the tasks corresponding to the intermediate calculation results stored in the target memory are maintained in the identifier file. Only the first target task that is not recorded in the identifier file needs to be executed. For the second target task that has been calculated and recorded in the identifier file, there is no need to execute the complete calculation process again. The corresponding result can be determined by directly calling the intermediate calculation results that have been persisted in the target memory. This avoids the repeated execution of the same task, reduces the amount of calculation in each processing stage, improves the processing efficiency of batch query, and shortens the completion cycle of batch query to meet the real-time requirements of business for data analysis results. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic diagram of the architecture of Trino Tardigrade in the related technologies provided in this application; Figure 2 This is a flowchart illustrating an embodiment of the data processing method provided in this application; Figure 3 This is a schematic diagram of the improved Trino Tardigrade architecture provided in this application; Figure 4 This is a schematic diagram of the structure of an embodiment of the data processing apparatus provided in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the data processing device provided in this application; Figure 6 This is a schematic diagram of the structure of an embodiment of the computer storage medium provided in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] The data processing method of this application is applied to a data processing apparatus, which can be a server, a terminal device, or a system in which the server and the terminal device cooperate with each other. Accordingly, the various parts of the data processing apparatus, such as each unit, subunit, module, and submodule, can all be set in the server, all in the terminal device, or separately in the server and the terminal device.

[0020] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed server functionality, or as a single software program or software module; no specific limitations are made here.

[0021] The Trino Tardigrade architecture, when processing batch queries, breaks them down into multiple processing stages. Each stage is further broken down into multiple tasks. To ensure retrying after task failure, intermediate computation results for each task in each processing stage are written to a configurable third-party storage medium. The upstream stage calculates the intermediate computation results for each task and writes them to the third-party storage medium. The downstream stage reads the intermediate computation results from the upstream stage from the third-party storage medium and completes its task based on these results. The intermediate computation results for the last stage are returned to the cache in the Coordinator. The Coordinator then aggregates the results to obtain the final query result. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the architecture of Trino Tardigrade in the related technologies provided in this application, such as... Figure 1 As shown, the Coordinator breaks down the overall batch query task into three processing stages (Stage 1, Stage 2, and Stage 3), and each processing stage is further broken down into three tasks: Stage 1 is broken down into Task 1-1, Task 1-2, and Task 1-3; Stage 2 is broken down into Task 2-1, Task 2-2, and Task 2-3; and Stage 3 is broken down into Task 3-1, Task 3-2, and Task 3-3.

[0022] Stage 1 completes Tasks 1-1, 1-2, and 1-3, obtaining the intermediate calculation results for Stage 1 and writing them to Storage (a third-party storage medium). Stage 2 reads the intermediate calculation results from Storage, completes the tasks of its downstream Stage 2 (Tasks 2-1, 2-2, and 2-3) based on these results, obtains the intermediate calculation results for Stage 2, and writes them to Storage. Stage 3 reads the intermediate calculation results from Storage, completes the tasks of its downstream Stage 3 based on these results, and obtains the intermediate calculation results for Stage 3. The intermediate calculation results for Stage 3 are written to the Coordinator's Buffer, and the Coordinator aggregates the intermediate calculation results in the Buffer to obtain the final query result, i.e., the Output.

[0023] In traditional Trino, data is transmitted directly and in real-time over the network between processing stages. If a worker node in the execution of a task fails, the tasks and computation results of its upstream processing stages are lost, and the entire query task must be rerun from the beginning, lacking a fault tolerance mechanism. However, in the Trino Tardigrade architecture, the intermediate computation results of each task in each processing stage are written to a third-party storage medium. When a worker node in the execution of a task fails, it is only necessary to retrieve the tasks and computation results of the upstream processing stages from the third-party storage medium to reschedule the task, without having to go back to the original source data. Compared to traditional Trino, this allows failed processing stage tasks in batch queries to be retried, improving fault tolerance.

[0024] In the existing Trino Tardigrade architecture, each task in each processing stage requires computation when handling batch queries. Especially in online analytical processing (OLAP) batch query scenarios, each task is complex and time-consuming, resulting in an excessively long batch query completion cycle. Furthermore, the intermediate computation results of each processing stage are written to third-party storage media. Compared to the traditional Trino, the Trino Tardigrade architecture requires frequent read and write operations on third-party storage media, further extending the batch query completion cycle.

[0025] Therefore, how to improve the processing efficiency of batch queries and shorten the completion cycle of batch queries has become the technical problem to be solved in this application.

[0026] To address the aforementioned problems, this application proposes a data processing method, for details of which please refer to [link / reference needed]. Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the data processing method provided in this application, as shown below. Figure 2 As shown, the specific steps are as follows: Step S11: Obtain an identification file containing at least a portion of historically processed tasks, wherein intermediate calculation results corresponding to each task in the identification file are stored in a target memory.

[0027] In this application embodiment, the Trino Tardigrade architecture in the prior art is further improved to improve the processing efficiency of batch queries, resulting in the improved Trino Tardigrade. The data processing method provided in this application is based on the improved Trino Tardigrade.

[0028] Please see Figure 3 , Figure 3 This is a schematic diagram of the improved Trino Tardigrade architecture provided in this application, as shown below. Figure 3As shown, when processing batch queries, the Trino Tardigrade architecture breaks down the entire query process into multiple processing stages, and each processing stage is further broken down into multiple tasks to be processed. During the processing of each processing stage, the intermediate calculation results corresponding to each task to be processed in the corresponding processing stage are determined.

[0029] In existing technology, Trino Tardigrade sequentially executes each task in each processing stage, storing the corresponding intermediate computation results in a third-party storage medium. After completing the batch query, the intermediate computation results stored in the third-party storage medium are cleared. That is, the intermediate computation results stored in the third-party storage medium are only stored during the execution of the batch query, not for persistent storage. In the improved Trino Tardigrade, at least a portion of the intermediate computation results during the processing are persistently stored in the third-party storage medium (they remain there indefinitely unless actively cleared, meaning all intermediate computation results are not cleared after the batch query). Simultaneously, an identification file is stored in the third-party storage medium. The tasks recorded in the identification file are those processed within a historical time period, and the intermediate computation results corresponding to all tasks recorded in the identification file are persistently stored in the third-party storage medium, i.e., the target memory.

[0030] When starting the current processing stage, the identification file is first obtained. Based on the identification file, the method for determining the intermediate calculation results of each task to be processed in the current processing stage is determined, thereby obtaining the intermediate calculation results of each task to be processed and completing the task processing of the current processing stage.

[0031] It should be noted that the tasks corresponding to the intermediate computation results of persistent storage are written to the identifier file in the form of metadata. The metadata in the identifier file is used to identify which tables, which parts of the partition data, and what operations were performed by the corresponding task.

[0032] The system determines whether the task to be processed is recorded in the identification file by comparing the metadata in the identification file with the metadata of the task to be processed.

[0033] The target storage device can be any type of disk or memory that provides storage functionality.

[0034] Step S12: Determine the first target task that is not recorded in the identifier file and the second target task that is recorded in the identifier file among the multiple pending tasks corresponding to the current processing stage of the query process.

[0035] In the embodiments of this application, the intermediate calculation result of a certain stage task in a batch query process is very likely to be used by a stage task in another batch query process, or there may be two identical stage tasks in two different batch query processes. Therefore, persistently storing the intermediate calculation results corresponding to historically processed tasks allows different batch query processes to share intermediate calculation results, avoiding the repeated execution of the same task.

[0036] After obtaining the identification file, the multiple pending tasks that need to be processed in the current processing stage are divided into two types: one is the pending task that is not recorded in the identification file, which is called the first target task, and the other is the pending task that is recorded in the identification file, which is called the second target task.

[0037] When determining the intermediate calculation results for each task to be processed, different determination methods are used for the first target task and the second target task.

[0038] Step S13: Execute each of the first target tasks to obtain the intermediate calculation results corresponding to the first target tasks.

[0039] In this application embodiment, if a task to be processed is a first target task that is not recorded in the identifier file, then in the current processing stage, the task to be processed needs to be executed in order to calculate the intermediate settlement result of the task to be processed. That is, after the first target task is determined, each first target task needs to be executed.

[0040] Step S14: Determine the intermediate calculation result corresponding to the second target task based on the intermediate calculation result stored in the target memory.

[0041] In this embodiment of the application, the second target task is a task to be processed that is recorded in the identifier file. Therefore, the second target task is a task that has been processed within a historical time period, and the corresponding intermediate processing results are stored in the target memory. At this time, it is not necessary to re-execute the task to be processed. The intermediate calculation result corresponding to the processing task can be determined by searching among the multiple intermediate calculation results stored in the target memory.

[0042] Through the above embodiments, intermediate calculation results of historically processed tasks are stored in the target memory, and the tasks corresponding to the intermediate calculation results stored in the target memory are maintained in the identifier file. Only the first target task that is not recorded in the identifier file needs to be executed. For the second target task that has been calculated and recorded in the identifier file, there is no need to execute the complete calculation process again. The corresponding result can be determined by directly calling the intermediate calculation results that have been persisted in the target memory. This avoids the repeated execution of the same task, reduces the amount of calculation in each processing stage, improves the processing efficiency of batch query, and shortens the completion cycle of batch query to meet the real-time requirements of business for data analysis results.

[0043] Meanwhile, for the second target task, it is not necessary to execute the complete calculation process again or write the intermediate calculation results corresponding to the second target task back to the target memory, which reduces the write operations to the target memory and further improves the processing efficiency of batch query.

[0044] In an optional embodiment, after executing each of the first target tasks and obtaining the intermediate calculation results corresponding to the first target task, the method further includes: obtaining the task complexity of the first target task and the data volume of the corresponding intermediate calculation results; and determining whether to add the first target task to the identifier file based on the task complexity and the data volume.

[0045] In this embodiment of the application, the first target task is not recorded in the identifier file. Therefore, the corresponding intermediate calculation results are not stored in the target memory. The corresponding intermediate calculation results are obtained by executing the first target task.

[0046] The intermediate calculation results stored in the target memory need to be continuously maintained and updated. For example, after executing a task that has never been executed before, the corresponding intermediate calculation results are stored in the target memory and recorded in the identifier file so that when there is a similar task in the future, it is not necessary to perform the calculation again.

[0047] It should be noted that all intermediate calculation results obtained in the current processing stage will be stored in the target memory to provide input data for the task processing in the next processing stage. However, after the entire batch processing query is completed, the intermediate calculation results corresponding to the tasks not recorded in the identifier file will be cleared. That is, only the intermediate calculation results corresponding to the tasks in the identifier file will be persistently stored.

[0048] If all intermediate computation results for different tasks are persisted in the target memory, the cost of the memory will be too high. Therefore, it is necessary to manage intermediate computation results reasonably, for example, only persisting the intermediate computation results corresponding to tasks with higher complexity.

[0049] In this embodiment, for a first target task for which the corresponding intermediate calculation result is not stored in the target memory, after executing the first target task and obtaining the intermediate calculation result of the first target task, it is determined whether to persist the intermediate calculation corresponding to the first target task, i.e. whether to add the first target task to the identifier file, based on the task complexity and the data volume of the corresponding intermediate calculation result.

[0050] The step of determining whether to add the first target task to the identifier file based on the task complexity and the data volume includes: adding the first target task to the identifier file when the task complexity is greater than a preset complexity threshold and the data volume is less than a preset data volume, and storing the intermediate calculation results corresponding to the first target task in the target memory.

[0051] In this embodiment of the application, the maximum amount of data that can be stored is limited by setting a parameter threshold, i.e., a preset data amount. That is, the amount of data of the intermediate calculation result corresponding to a single task must not exceed the preset data amount, so as to avoid the storage pressure on the target memory caused by the excessive amount of data of a single intermediate calculation result.

[0052] Meanwhile, for a single task, depending on its complexity, tasks with low complexity (less than or equal to a preset complexity threshold) have faster computation speeds. Even if computation is performed every time, intermediate results can be obtained quickly, with minimal impact on the completion cycle. Therefore, the intermediate results do not need to be persisted. However, for tasks with high complexity (greater than a preset complexity threshold), each computation typically takes longer, significantly impacting the completion cycle. If the amount of intermediate results meets the requirements, these intermediate results are persisted, improving the processing efficiency of subsequent tasks without increasing the storage pressure on the target memory.

[0053] In an optional embodiment, obtaining the task complexity of the first target task includes: obtaining the task type and metadata information corresponding to the first target task, wherein the metadata information includes the data processing volume corresponding to the first target task; and determining the task complexity based on the task type and the data processing volume.

[0054] In this embodiment of the application, when determining whether to persistently store the intermediate calculation results of the first target task, it is necessary to first determine the task complexity corresponding to the first target task. Specifically, the task complexity is determined according to the task type and data processing volume of the first target task.

[0055] For example, different task types are quantified into different first complexities, and different data processing volumes are quantified into different second complexities. The first and second complexities of a task are weighted and fused to obtain the corresponding task complexity.

[0056] When a task's task type is association or aggregation, its corresponding initial complexity is relatively high; when a task's task type is addition or subtraction, its corresponding initial complexity is relatively low.

[0057] In an optional embodiment, after the first target task is added to the identifier file, the method further includes: determining a time threshold corresponding to the first target task based on the task complexity, wherein the time threshold represents the maximum duration for which intermediate calculation results corresponding to the first target task are statically stored in the target memory.

[0058] In the embodiments of this application, a time threshold is set for each task that persistently stores intermediate calculation results in the target memory. The time threshold specifies the maximum duration for which the intermediate calculation results corresponding to the task are statically stored in the target memory. The duration of static storage refers to the time elapsed since the intermediate calculation results were last called after being stored in the target memory.

[0059] Therefore, if the first target task is added to the identifier file, that is, the intermediate calculation results corresponding to the first target task are persistently stored in the target memory, a time threshold needs to be set for the first target task. Specifically, the time thresholds for different tasks can be different. Therefore, the corresponding time threshold is determined according to the task complexity. The task complexity and the time threshold are positively correlated. The higher the task complexity of the first target task, the larger its corresponding time threshold is, and vice versa.

[0060] In an optional embodiment, the method further includes: determining the target duration between the current time and the most recent invocation time of each task in the identifier file, and obtaining the time threshold corresponding to each task in the identifier file; deleting tasks whose target duration exceeds the corresponding time threshold from the identifier file, and deleting the corresponding intermediate calculation results from the target memory.

[0061] In this embodiment of the application, in order to further reduce the storage pressure on the target memory, the intermediate calculation results stored persistently in the target memory are managed. Specifically, intermediate calculation results that have not been called for a long time in the target memory are deleted. For each task in the identification file, the duration of static storage corresponding to each task is determined, that is, the target duration between the current time and the most recent call time of each task in the identification file. The target duration is compared with the time threshold corresponding to the task. If the target duration exceeds the time threshold, the corresponding task is deleted from the identification file, and the intermediate calculation results corresponding to the task are also deleted from the target memory.

[0062] Determining the target duration between the current time and the most recent invocation time of each task in the identifier file includes: obtaining the lifecycle value corresponding to each task in the identifier file at the current time; determining the lifecycle value as the corresponding target duration; wherein, when any task in the identifier file is invoked, the lifecycle value corresponding to that task is cleared to zero and recalculated.

[0063] Each task is configured with a lifecycle, which gradually increases over time. However, when a task is invoked, its corresponding lifecycle value is cleared to zero and then recalculated. The real-time value of the task's lifecycle is the target duration of the corresponding task at the current moment.

[0064] In an optional embodiment, the method further includes: obtaining the call frequency of intermediate calculation results corresponding to each task in the identifier file within a preset time period; deleting tasks with call frequencies less than a frequency threshold from the identifier file, and deleting the corresponding intermediate calculation results from the target memory.

[0065] In this embodiment, the intermediate calculation results persistently stored in the target memory are managed by the calling frequency of the intermediate calculation results corresponding to each task. Specifically, the calling frequency of the intermediate calculation results corresponding to each task within a preset time period is determined, the calling frequency is compared with a preset frequency threshold, and tasks with a frequency lower than the frequency threshold are deleted from the identification file, and the intermediate calculation results corresponding to the task are also deleted from the target memory.

[0066] To implement the above data processing method, this application also proposes a data processing apparatus, for details please refer to [link / reference needed]. Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the data processing apparatus provided in this application.

[0067] The data processing device 500 in this embodiment includes: The acquisition module 51 is used to acquire an identification file including at least a portion of historically processed tasks, wherein intermediate calculation results corresponding to each task in the identification file are stored in a target memory; The first determining module 52 is used to determine the first target task that is not recorded in the identifier file and the second target task that is recorded in the identifier file among the multiple pending tasks corresponding to the current processing stage of the query process. Execution module 53 is used to execute each of the first target tasks and obtain the intermediate calculation results corresponding to the first target tasks; The second determining module 54 is used to determine the intermediate calculation result corresponding to the second target task based on the intermediate calculation result stored in the target memory.

[0068] To implement the above data processing method, this application also proposes a data processing device, please refer to the details below. Figure 5 , Figure 5 This is a schematic diagram of an embodiment of the data processing device provided in this application.

[0069] The data processing device 400 in this embodiment includes a processor 41, a memory 42, an input / output device 43, and a bus 44.

[0070] The processor 41, memory 42, and input / output device 43 are respectively connected to the bus 44. The memory 42 stores program data, and the processor 41 is used to execute the program data to implement the data processing method described in the above embodiments.

[0071] In this embodiment, processor 41 can also be referred to as a CPU (Central Processing Unit). Processor 41 may be an integrated circuit chip with signal processing capabilities. Processor 41 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 41 can be any conventional processor.

[0072] This application also provides a computer storage medium; please refer to the following: Figure 6 , Figure 6This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 600 stores a computer program 61, which, when executed by a processor, is used to implement the data processing method of the above embodiment.

[0073] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data processing method, characterized in that, The data processing method includes: Obtain an identification file containing at least a portion of historically processed tasks, wherein intermediate computation results corresponding to each task in the identification file are stored in a target memory; Identify the first target task that is not recorded in the identifier file and the second target task that is recorded in the identifier file among the multiple pending tasks corresponding to the current processing stage of the query process; Execute each of the first target tasks to obtain the intermediate calculation results corresponding to the first target tasks; The intermediate calculation results corresponding to the second target task are determined based on the intermediate calculation results stored in the target memory.

2. The data processing method according to claim 1, characterized in that, After executing each of the first target tasks and obtaining the intermediate calculation results corresponding to the first target tasks, the method further includes: Obtain the task complexity of the first target task and the corresponding amount of intermediate calculation results; Whether to add the first target task to the identifier file is determined based on the task complexity and the amount of data.

3. The data processing method according to claim 2, characterized in that, Determining whether to add the first target task to the identifier file based on the task complexity and the data volume includes: If the task complexity is greater than a preset complexity threshold and the data volume is less than a preset data volume, the first target task is added to the identifier file, and the intermediate calculation results corresponding to the first target task are stored in the target memory.

4. The data processing method according to claim 2, characterized in that, Obtaining the task complexity of the first target task includes: Obtain the task type and metadata information corresponding to the first target task, wherein the metadata information includes the data processing volume corresponding to the first target task; The task complexity is determined based on the task type and the amount of data processed.

5. The data processing method according to claim 2, characterized in that, After the first target task is added to the identifier file, the method further includes: The time threshold corresponding to the first target task is determined based on the task complexity, wherein the time threshold represents the maximum duration for which the intermediate calculation results corresponding to the first target task are statically stored in the target memory.

6. The data processing method according to claim 1, characterized in that, The method further includes: Determine the target duration between the current time and the most recent invocation time of each task in the identifier file, and obtain the time threshold corresponding to each task in the identifier file; Tasks whose target duration exceeds the corresponding time threshold are deleted from the identifier file, and the corresponding intermediate calculation results are deleted from the target memory.

7. The data processing method according to claim 6, characterized in that, Determine the target duration between the current time and the most recent invocation time of each task in the identifier file, including: Obtain the lifecycle value corresponding to each task in the identifier file at the current time; The value of the lifecycle is determined as the corresponding target duration; When any task in the identification file is invoked, the lifecycle value corresponding to that task is cleared to zero and recalculated.

8. The data processing method according to claim 1, characterized in that, The method further includes: Obtain the call frequency of intermediate calculation results corresponding to each task in the identifier file within a preset time period; Tasks whose call frequency is less than the frequency threshold are deleted from the identification file, and the corresponding intermediate calculation results are deleted from the target memory.

9. A data processing device, characterized in that, The data processing device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the data processing method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium is used to store program data, which, when executed by the computer, is used to implement the data processing method as described in any one of claims 1 to 8.

Images