Task processing method and apparatus, electronic device, and medium

Abstract

The present disclosure provides a task processing method and device, electronic equipment and medium, relates to the field of computers, specifically to cloud computing, cloud data computing and cloud database technology, and can be applied in an intelligent cloud scenario. The specific implementation scheme is as follows: at least two original task nodes contained in a to-be-processed task are acquired, and an external access file associated with the original task nodes is determined; a target access file is determined from the external access file, and an auxiliary task node is generated according to the target access file; wherein the target access file is associated with at least two original task nodes; the to-be-processed task is processed according to the auxiliary task node and the original task node; wherein the auxiliary task node is used for locally caching the target access file. The present disclosure achieves the effect of reducing network traffic caused by accessing external files in the task processing process.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computers, in particular to cloud computing, cloud data computing and cloud database technology, which can be applied in an intelligent cloud scenario. In particular, a task processing method and device, electronic equipment and medium are provided. BACKGROUND

[0002] In order to solve the problem of huge network traffic caused by multiple accesses of a computing cluster to a file in a storage cluster, various manufacturers provide file cache services. The file cache service is deployed in the computing cluster and is used to cache the access file in the storage cluster, so as to locally reuse the access file by the computing cluster.

[0003] At present, the cache mode of the file cache service is mostly to cache only the access file frequently used by the computing cluster. SUMMARY

[0004] The present disclosure provides a method, device, electronic equipment and medium for reducing network traffic caused by access of a computing cluster to an external file.

[0005] According to an aspect of the present disclosure, a task processing method is provided, comprising:

[0006] obtaining at least two original task nodes contained in a to-be-processed task, and determining an external access file associated with the original task nodes;

[0007] determining a target access file from the external access file, and generating an auxiliary task node according to the target access file; wherein the target access file is associated with at least two original task nodes;

[0008] processing the to-be-processed task according to the auxiliary task node and the original task node; wherein the auxiliary task node is used to locally cache the target access file.

[0009] According to another aspect of the present disclosure, a task processing device is provided, comprising:

[0010] a file determination module configured to obtain at least two original task nodes contained in a to-be-processed task, and determine an external access file associated with the original task nodes;

[0011] a node generation module configured to determine a target access file from the external access file, and generate an auxiliary task node according to the target access file; wherein the target access file is associated with at least two original task nodes;

[0012] The task processing module is used to process the task to be processed based on the auxiliary task node and the original task node; wherein, the auxiliary task node is used to cache the target access file locally.

[0013] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0014] At least one processor; and

[0015] A memory that is communicatively connected to at least one processor; wherein,

[0016] The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform any of the methods of this disclosure.

[0017] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform any of the methods of this disclosure.

[0018] According to another aspect of this disclosure, a computer program product is provided, including a computer program and a method for the computer program to be executed by a processor according to any of the methods disclosed herein.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a flowchart of a method for processing some tasks according to embodiments of this disclosure;

[0022] Figure 2 This is a flowchart of processing methods for some other tasks disclosed in embodiments of this disclosure;

[0023] Figure 3 This is a flowchart illustrating the processing of some task nodes according to embodiments of this disclosure;

[0024] Figure 4 This is a schematic diagram of the structure of a processing apparatus for some tasks disclosed in the embodiments of this disclosure;

[0025] Figure 5 This is a block diagram of an electronic device used to implement the processing method for the tasks disclosed in the embodiments of this disclosure. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] As the amount of data stored in a cluster increases, it's necessary to expand the cluster nodes. Due to the coupling between computing and storage, both expand simultaneously. However, in real-world business scenarios, the demand for storage and computing varies across different periods; the demand changes elastically. Scaling up and down by binding both together would waste resources and fail to meet the elastic nature of cloud computing. For example, when computing resources are scarce, they need to be expanded separately, while when there are no computing tasks, they can be reduced.

[0028] To address the aforementioned issues, a storage-compute separation architecture has emerged, which divides the cluster into storage clusters and compute clusters, with communication connections between them. This architecture allows big data clusters to fully utilize resources, scale elastically, and is more flexible, better aligning with the characteristics of cloud computing.

[0029] However, because the computing cluster frequently reads or writes data to files in the storage cluster when processing computing tasks, a large number of computing tasks in the computing cluster will undoubtedly generate huge network traffic, resulting in insufficient network bandwidth and affecting the efficiency of task processing.

[0030] To address the issue of massive network traffic caused by multiple file accesses from the storage cluster to the compute cluster, major vendors have provided file caching services. Deployed within the compute cluster, these services utilize the cluster's memory or disk to create a high-speed storage service that caches files accessed from the storage cluster, allowing the compute cluster to reuse these files locally.

[0031] When the compute cluster needs to read a file from the storage cluster, it doesn't read directly from the storage cluster; instead, it reads from the file caching service. If the file caching service finds that the file is already cached, it returns the file directly to the compute cluster, without requiring the compute cluster to read it from the storage cluster again. When the compute cluster needs to write to a file from the storage cluster, it caches the data locally through the file caching service. After the data is cached, it is synchronized to the storage cluster, avoiding frequent writes from the compute cluster to the files in the storage cluster.

[0032] Because computing clusters have limited space available for data storage, most current file caching services only cache frequently accessed files locally. However, these services do not analyze each task individually to determine which files need to be cached. This means that when a task requires access to a file that differs significantly from the cached file, the computing cluster still needs to frequently access the file in the storage cluster, resulting in significant network traffic.

[0033] Figure 1 This is a flowchart illustrating some task processing methods disclosed in embodiments of this disclosure. These embodiments are applicable to situations where a computing cluster processes tasks. The methods in this embodiment can be executed by the task processing apparatus disclosed in these embodiments. The apparatus can be implemented in software and / or hardware and can be integrated into any electronic device with computing capabilities.

[0034] like Figure 1 As shown, the task processing method disclosed in this embodiment may include:

[0035] S101. Obtain at least two original task nodes contained in the task to be processed, and determine the external access files associated with the original task nodes.

[0036] In this context, "tasks to be processed" refers to tasks that need to be processed within the computing cluster. These tasks include, but are not limited to, SQL (Structured Query Language) tasks. Each task to be processed consists of at least two original task nodes, each corresponding to a task processing statement. The task is processed by executing these statements from the original task nodes. When executing the task processing statement of any original task node involves accessing at least one external access file in the storage cluster (e.g., reading or writing to an external access file), that at least one external access file is designated as the external access file associated with that original task node.

[0037] In one implementation, the computing cluster parses the task to be processed, identifying at least two original task nodes contained within the task. It then extracts the task processing statements corresponding to each original task node. These task processing statements represent executable code programs; for example, "insert overwrite table ods.A select xxx from ods.P_Agroup by xxx." represents a task processing statement. Information extraction is performed on the task processing statements, and based on the extraction results, the external access files contained within the task processing statements are determined as the external access files associated with the original task nodes.

[0038] By obtaining at least two original task nodes contained in the task to be processed and determining the external access files associated with the original task nodes, the effect of determining all external access files associated with the task to be processed is achieved, laying the data foundation for subsequently determining the target access file from the external access files.

[0039] S102. Determine the target access file from the external access files, and generate auxiliary task nodes based on the target access file; wherein the target access file is associated with at least two original task nodes.

[0040] In one implementation, the computing cluster determines the number of original task nodes associated with each external access file, and filters the external access files based on the number of nodes, selecting external access files associated with at least two original task nodes as target access files.

[0041] For example, assuming the original task nodes associated with each external access file are shown in the table below, it can be seen that the external access file "ods.P_BC" is associated with original task nodes B and C, a total of two original task nodes, and "ods.A" is associated with original task nodes A, B, C, and D, a total of four original task nodes. Then "ods.P_BC" and "ods.A" will be used as target access files.

[0042] External access file Associated original task node ods.P_A A ods.P_BC B, C ods.P_D D ods.A A, B, C, D ods.B B ods.C C ods.D D

[0043] Since the target access file is associated with at least two original task nodes, it means that at least two original task nodes need to read or write the target access file from the storage cluster when executing task processing statements. In order to reduce the network traffic caused by access between the compute cluster and the storage cluster, the compute cluster generates auxiliary task nodes in the task to be processed based on the target access file. This allows the compute cluster to cache the target access file locally when executing the auxiliary task processing statements of the auxiliary task nodes, so that the target access file can be reused locally on the compute cluster in subsequent processing.

[0044] It is worth noting that for external access files that are associated with only one original task node, since only one original task node needs to read or write the external access file from the storage cluster, there is no need to cache it locally for subsequent reuse, thus avoiding the problem of excessive cached data.

[0045] The target access file is determined from external access files, and auxiliary task nodes are generated based on the target access file. The target access file is associated with at least two original task nodes, which lays the data foundation for the subsequent local caching of the target access file through the auxiliary task nodes.

[0046] S103. Process the task to be processed based on the auxiliary task node and the original task node; wherein, the auxiliary task node is used to cache the target access file locally.

[0047] In one implementation, during the processing of the task to be processed, the computing cluster first executes the auxiliary task processing statements of the auxiliary task node, thereby caching the target access file into the cache file on the local end of the computing cluster.

[0048] Secondly, the task processing statement of the first original task node associated with the target access file is modified, including the following steps: the file identifier of the target access file in the task processing statement is replaced with the file identifier of the cache file.

[0049] Finally, the task processing statements of the first original task node and the task processing statements of the second original task node are executed to complete the processing of the task to be processed.

[0050] This disclosure obtains at least two original task nodes contained in the task to be processed, determines the external access file associated with the original task nodes, determines the target access file from the external access file, and generates an auxiliary task node based on the target access file. The target access file is associated with at least two original task nodes, and the task to be processed is then performed based on the auxiliary task node and the original task nodes. The auxiliary task node is used to locally cache the target access file. Because the auxiliary task node locally caches the target access file, the original task nodes associated with the target access file can reuse access to the target access file locally, eliminating the need to access the target access file from the storage cluster each time, thus reducing network traffic caused by accessing external files during task processing.

[0051] Figure 2 This is a flowchart of processing methods for some other tasks disclosed in the embodiments of this disclosure, which is further optimized and extended based on the above technical solutions, and can be combined with the above optional implementation methods.

[0052] like Figure 2 As shown, the task processing method disclosed in this embodiment may include:

[0053] S201. Obtain at least two original task nodes contained in the task to be processed.

[0054] S202. Parse the task processing statements of the original task node and determine the file identifiers contained in the task processing statements.

[0055] In one implementation, a statement structure parsing tool is used to parse the task processing statement and determine the file identifier contained in the task processing statement. For example, when the task processing statement is an SQL statement, an SQL structure parser is used to parse the SQL statement.

[0056] For example, assuming the task processing statement is "insert overwrite table ods.A select xxx from ods.P_A group by xxx.", the parsing of this task processing statement includes two file identifiers: the file identifier for reading "ods.P_A" and the file identifier for writing "ods.A".

[0057] For example, assuming the task processing statement is "Insert overwrite table ods.B select xxx from ods.A join ods.P_BC on xxx", then parsing this task processing statement will reveal three file identifiers: the file identifier for reading "ods.A", "ods.P_BC", and the file identifier for writing "ods.B".

[0058] S203. Determine the external access file associated with the original task node based on the file identifier.

[0059] In one implementation, the external access file associated with the file identifier contained in the task processing statement is determined as the external access file associated with the original task node based on the file identifier contained in the task processing statement and the association relationship between the external access file and the file identifier.

[0060] For example, based on the access type performed by the original task node on the target access file, the target access file is classified and statistically analyzed, including external read files and external write files. As shown in the table below, original task node A is associated with the external read file "ods.P_A" and the external write file "ods.A"; original task node B is associated with the external read files "ods.A" and "ods.P_BC" and the external write file "ods.B"; original task node C is associated with the external read files "ods.A" and "ods.P_BC" and the external write file "ods.C"; original task node D is associated with the external read files "ods.A" and "ods.P_D" and the external write file "ods.D".

[0061] Original task node External read file External write file A ods.P_A ods.A B ods.A, ods.P_BC ods.B C ods.A, ods.P_BC ods.C D ods.A, ods.P_D ods.D

[0062] By parsing the task processing statements of the original task nodes, the file identifiers contained in the task processing statements are determined, and the external access files associated with the original task nodes are determined based on the file identifiers. This method of determining the external access files associated with each original task node by parsing the file identifiers improves the efficiency and accuracy of determining the external access files.

[0063] S204. Determine the target access file from the external access files, and select at least two original task nodes associated with the target access file as candidate task nodes.

[0064] In one implementation, an external access file associated with at least two original task nodes is selected as the target access file. Furthermore, based on the association between the external access file and the original task nodes, at least two original task nodes associated with the target access file are selected as candidate task nodes.

[0065] For example, continuing with the table above, the external access file "ods.A" is the external write file of the original task node A, and the external read file of the original task nodes B, C, and D. That is, the external access file "ods.A" is associated with four original task nodes, which are used as the target access file, and the original task nodes A, B, C, and D are used as candidate task nodes.

[0066] S205. Based on the dependencies between the original task nodes, determine the target task node from the candidate task nodes.

[0067] In this context, if the execution of one original task node depends on the execution result of another original task node, then the two original task nodes are considered to have a dependency relationship. For example, if the execution of original task nodes B, C, and D depends on original task node A, then original task nodes A, B, C, and D have a dependency relationship.

[0068] In one implementation, the execution priority of each original task node is determined based on the dependencies between the original task nodes, and the target task node is determined from the candidate task nodes based on the execution priority.

[0069] Optionally, S205 includes:

[0070] The execution priority of each candidate task node is determined based on the dependency relationship, and the candidate task node with the highest execution priority is selected as the target task node.

[0071] For example, suppose candidate task node B depends on candidate task node A, and candidate task node C depends on candidate task node B. Then, candidate task node A has the highest execution priority, candidate task node B has the second highest execution priority, and candidate task node C has the lowest execution priority. Therefore, candidate task node A is selected as the target task node.

[0072] By determining the execution priority of each candidate task node based on dependencies, and taking the candidate task node with the highest execution priority as the target task node, a data foundation is laid for the subsequent generation of auxiliary task nodes based on the target task node.

[0073] S206. Based on the access type of the target access file performed by the target task node, generate an auxiliary task node; where the access type includes read type and write type.

[0074] In one implementation, when the access type is read, a new auxiliary task node is created in the previous execution priority of the target task node to locally cache the target access file. When the access type is write, the task processing statement of the target task node is modified to make the target task node an auxiliary task node for local caching of the target access file.

[0075] By using at least two original task nodes associated with the target access file as candidate task nodes, and determining the target task node from the candidate task nodes based on the dependencies between the original task nodes, auxiliary task nodes are generated according to the access type performed by the target task node on the target access file. The access type includes read type and write type, which achieves the effect of adaptively generating auxiliary task nodes based on the access type, ensuring that the auxiliary task nodes can successfully complete the local caching operation of the target access file regardless of the access type.

[0076] Optionally, S206 includes:

[0077] When the access type is read, a new task node is created as an auxiliary task node; the execution priority of the auxiliary task node is higher than that of the target task node; the auxiliary task processing statements of the auxiliary task node are used to cache the target access file in the local cache file.

[0078] In one implementation, if the access type is read, it means that data from the target access file needs to be read when executing the task processing statement of the target task node. Therefore, a new task node is created as an auxiliary task node, and the execution priority of the auxiliary task node is set to be higher than that of the target task node, in order to synchronize the target access file in the storage cluster to a pre-generated cache file on the local end of the computing cluster.

[0079] By creating a new task node as an auxiliary task node when the access type is read, the target access file can be cached locally, ensuring that it can be reused locally later. Furthermore, since the execution priority of the auxiliary task node is higher than that of the target task node, the target task node with the highest execution priority can successfully read the target access file from the local end, ensuring the normal execution of the target task node.

[0080] Optionally, S206 includes:

[0081] When the access type is write, determine the cache file identifier of the cached file in the local terminal and the target file identifier of the target access file; in the task processing statement of the target task node, replace the target file identifier with the cache file identifier to generate an auxiliary task processing statement; based on the auxiliary task processing statement, use the target task node as an auxiliary task node.

[0082] In one implementation, if the access type is write type, the task processing statement of the target task node is obtained, and the target file identifier of the target access file in the task processing statement is replaced with the cache file identifier of the cache file pre-established on the local end of the computing cluster to generate an auxiliary task processing statement. Then, the target task node corresponding to the auxiliary task processing statement is used as the auxiliary task node.

[0083] For example, assuming the access type is write, the cache file identifier of the pre-established cache file on the local compute cluster is "fuse_cache.ods_db_A_1", and the task processing statement of the target task node A is "insert overwrite table ods.A select xxx from ods.P_A group by xxx.", where the target file identifier of the target access file is "ods.A", then "ods.A" is replaced with "fuse_cache.ods_db_A_1", generating the auxiliary task processing statement "insert overwrite table fuse_cache.ods_db_A_1 select xxx from ods.P_A group by xx". Then, the target task node A of the corresponding auxiliary task processing statement is used as the auxiliary task node.

[0084] By replacing the target file identifier with the cache file identifier in the task processing statement of the target task node when the access type is write, an auxiliary task processing statement is generated. Based on the auxiliary task processing statement, the target task node is used as an auxiliary task node. Thus, the target task node after the modified task processing statement is directly used as an auxiliary task node, so that the target task node with the highest execution priority can directly write data to the local cache file when it is executed, so that subsequent reading or writing can be performed directly in the cache file, without having to read or write the target access file from the storage cluster, thus reducing the network traffic generated.

[0085] S207. Process the task to be processed based on the auxiliary task node and the original task node.

[0086] Optionally, S207 includes:

[0087] Determine the cache file identifier of the cached file and the target file identifier of the target access file; replace the target file identifier with the cache file identifier in the task processing statement of the candidate task node to generate the update task processing statement of the candidate task node; execute the auxiliary task processing statement, the update task processing statement, and the task processing statements of other original task nodes besides the candidate task nodes to process the tasks to be processed.

[0088] In one implementation, task processing statements of candidate task nodes associated with the target access file are obtained, and the target file identifier of the target access file in the task processing statement is replaced with the cache file identifier of the cache file to generate an update task processing statement. Based on the dependencies between task nodes in the task to be processed, auxiliary task processing statements of auxiliary task nodes, update task processing statements of candidate task nodes, and task processing statements of other original task nodes are executed to complete the processing of the task to be processed.

[0089] By executing the auxiliary task processing statements of the auxiliary task nodes, the target access file is cached locally. By executing the update task processing statements of the candidate task nodes, the candidate task nodes can directly read or write the target access file on the local end of the computing cluster, without having to read or write the target access file from the storage cluster, thus reducing network traffic. By executing the task processing statements of other original task nodes besides the candidate task nodes, it is ensured that original task nodes not associated with the target access file can still be executed, thus ensuring the accuracy of the processing results of the pending tasks.

[0090] Optionally, following the above embodiment, step S207 may further include:

[0091] The execution priority of each candidate task node is determined based on the dependency relationship, and the candidate task node with the lowest execution priority is designated as the termination task node; a new task node is created as the cleanup task node; the execution priority of the cleanup task node is lower than that of the termination task node; the termination task node is used to clear cache files.

[0092] For example, suppose candidate task node B depends on candidate task node A, and candidate task node C depends on candidate task node B. Then, candidate task node A has the highest execution priority, candidate task node B has the second highest, and candidate task node C has the lowest execution priority. Therefore, candidate task node C is chosen as the termination task node. A new task node is created as a cleanup task node in the next execution priority range of candidate task node C to clear cached files.

[0093] The execution priority of each candidate task node is determined based on dependencies, and the candidate task node with the lowest execution priority is designated as the termination task node. A new task node is created as the cleanup task node. The execution priority of the cleanup task node is lower than that of the termination task node. The termination task node is used to clear cache files, which reduces the storage pressure on the local computing cluster by setting the cleanup task node. Furthermore, since the execution priority of the cleanup task node is lower than that of the termination task node, it is ensured that the cache file is cleared only after all candidate task nodes have accessed the target access file in the cache file, thus ensuring the normal processing of pending tasks.

[0094] Based on the above embodiments, in order to further explain the solutions involved in this disclosure, this disclosure also provides a processing flow of the task processing method in a specific scenario.

[0095] Assume the task to be processed contains original task nodes A, B, C, and D. The corresponding task processing statements are: "insert overwrite table ods.A select xxx from ods.P_A group by xxx.", "insert overwrite table ods.B select xxx from ods.A join ods.P_BC on xxx", "insert overwrite table ods.C select xxx from ods.A join ods.P_BC on xxx", and "insert overwrite table ods.D select xxx from ods.A join ods.P_D on xxx". Original task nodes B, C, and D depend on original task node A.

[0096] 1) Identify the associated external access files.

[0097] Original task node A is associated with the external access files "ods.P_A" and "ods.A". Original task node B is associated with the external access files "ods.A", "ods.P_BC", and "ods.B". Original task node C is associated with the external access files "ods.A", "ods.P_BC", and "ods.C". Original task node D is associated with the external access files "ods.A", "ods.P_D", and "ods.D".

[0098] 2) Determine the target access file from the external access files.

[0099] The external access file "ods.P_A" is associated with the original task node A. The external access file "ods.P_BC" is associated with original task nodes B and C. The external access file "ods.P_D" is associated with the original task node D. The external access file "ods.A" is associated with original task nodes A, B, C, and D. The external access file "ods.B" is associated with the original task node B. The external access file "ods.C" is associated with the original task node C. The external access file "ods.D" is associated with the original task node D.

[0100] Then the external access files “ods.P_BC” and “ods.A” will be used as the target access files.

[0101] 3) Select original task node B and original task node C as candidate task nodes for the target access file "ods.P_BC". Select original task node A, original task node B, original task node C and original task node D as candidate task nodes for the target access file "ods.A".

[0102] 4) Since original task node B and original task node C have the same execution priority, both original task node B and original task node C will be designated as target task nodes for the target access file "ods.P_BC". Furthermore, since original task node A has the highest execution priority, original task node A will be designated as the target task node for the target access file "ods.A".

[0103] 5) Original task nodes B and C perform read operations on the target access file "ods.P_BC". Therefore, a new auxiliary task node E is created to synchronize the target access file "ods.P_BC" from the storage cluster to the local end of the compute cluster. Furthermore, auxiliary task node E has the same execution priority as the original task node A.

[0104] The original task node A performs a write operation on the target access file "ods.A". Assuming the cache file identifier of the pre-established cache file on the local computing cluster is "fuse_cache.ods_db_A_1", then "ods.A" in the task processing statement is replaced with "fuse_cache.ods_db_A_1", generating the auxiliary task processing statement "insert overwritetable fuse_cache.ods_db_A_1 select xxx from ods.P_A group by xx". The original task node A corresponding to the auxiliary task processing statement is then designated as the auxiliary task node.

[0105] 6) Original task nodes B, C, and D are candidate task nodes for the target access file "ods.A", and original task nodes B and C are candidate task nodes for the target access file "ods.P_BC". Assume the cache identifier of the cache file corresponding to the target access file "ods.A" is "fuse_cache.ods_db_A_1", and the cache identifier of the cache file corresponding to the target access file "ods.P_BC" is "fuse_cache.ods_db_P_BC_1".

[0106] The update statement for the original task node B is "insert overwrite table ods.B select xxx from fuse_cache.ods_db_A_1 join fuse_cache.ods_db_P_BC_1 on xxx". The update statement for the original task node C is "insert overwrite table ods.C select xxx from fuse_cache.ods_db_A_1 join fuse_cache.ods_db_P_BC_1 on xxx". The update statement for the original task node D is "insert overwrite table ods.D select xxx from fuse_cache.ods_db_A_1 join ods.P_D on xxx".

[0107] 7) During the processing of the task to be processed, the auxiliary task processing statements of the original task node A and the auxiliary task node E are executed first. Then the update task processing statements of the original task nodes B, C, and D are executed. (There are no other original task nodes besides the candidate task nodes here).

[0108] 8) Original task nodes B and C are the terminating task nodes for the target access file "ods.P_BC". Original task nodes B, C, and D are the terminating task nodes for the target access file "ods.A". Therefore, in the next execution priority of original task nodes B and C, a clearing task node G is set to clear the cache file "fuse_cache.ods_db_P_BC_1". In the next execution priority of original task nodes B, C, and D, a clearing task node H is set to clear the cache file "fuse_cache.ods_db_A_1".

[0109] 9) Complete the processing of the task to be processed.

[0110] The specific execution methods involved in the above processing flow can be referred to the description of the methods in the embodiments of this disclosure, and will not be repeated here.

[0111] In accordance with the above processing flow, this disclosure also provides corresponding accompanying drawings. Figure 3 This is a flowchart illustrating the processing of some task nodes according to embodiments of this disclosure, wherein... Figure 3 'a' represents the task node processing flow before optimization of the task to be processed. Figure 3b represents the optimized task node processing flow after the above processing steps for the task to be processed.

[0112] Figure 4 This is a schematic diagram of the structure of a processing apparatus for some tasks disclosed in embodiments of this disclosure, which can be applied to situations where a computing cluster processes tasks to be processed. The apparatus of this embodiment can be implemented in software and / or hardware and can be integrated on any electronic device with computing capabilities.

[0113] like Figure 4 As shown, the task processing device 40 disclosed in this embodiment may include a file determination module 41, a node generation module 42, and a task processing module 43, wherein:

[0114] The file determination module 41 is used to obtain at least two original task nodes contained in the task to be processed, and to determine the external access file associated with the original task nodes.

[0115] The node generation module 42 is used to determine the target access file from the external access file and generate auxiliary task nodes based on the target access file; wherein the target access file is associated with at least two original task nodes;

[0116] The task processing module 43 is used to process the task to be processed according to the auxiliary task node and the original task node; wherein, the auxiliary task node is used to cache the target access file locally.

[0117] Optional, file determination module 41, specifically used for:

[0118] The task processing statement of the original task node is parsed to determine the file identifier contained in the task processing statement;

[0119] The external access file associated with the original task node is determined based on the file identifier.

[0120] Optionally, the node generation module 42 is specifically used for:

[0121] Select at least two original task nodes associated with the target access file as candidate task nodes;

[0122] Based on the dependencies between the original task nodes, the target task node is determined from the candidate task nodes;

[0123] Based on the access type performed by the target task node on the target access file, an auxiliary task node is generated; wherein, the access type includes read type and write type.

[0124] Optionally, the node generation module 42 is further used for:

[0125] The execution priority of each candidate task node is determined based on the dependency relationship, and the candidate task node with the highest execution priority is taken as the target task node.

[0126] Optionally, the node generation module 42 is further used for:

[0127] When the access type is read, a new task node is created as the auxiliary task node; wherein, the execution priority of the auxiliary task node is higher than that of the target task node; the auxiliary task processing statement of the auxiliary task node is used to cache the target access file in a cache file on the local end.

[0128] Optionally, the node generation module 42 is further used for:

[0129] If the access type is write type, determine the cache file identifier of the cached file in the local terminal, and the target file identifier of the target access file;

[0130] In the task processing statement of the target task node, the target file identifier is replaced with the cache file identifier to generate an auxiliary task processing statement;

[0131] According to the auxiliary task processing statement, the target task node is used as the auxiliary task node.

[0132] Optional, task processing module 43, specifically used for:

[0133] Determine the cache file identifier of the cached file and the target file identifier of the target access file;

[0134] In the task processing statement of the candidate task node, the target file identifier is replaced with the cache file identifier to generate the update task processing statement of the candidate task node;

[0135] The auxiliary task processing statement, the update task processing statement, and the task processing statements of other original task nodes besides the candidate task nodes are executed to process the task to be processed.

[0136] Optionally, the device also includes a cache clearing module, specifically used for:

[0137] The execution priority of each candidate task node is determined based on the dependency relationship, and the candidate task node with the lowest execution priority is selected as the termination task node.

[0138] Create a new task node as a cleanup task node; wherein the execution priority of the cleanup task node is lower than that of the terminate task node; the terminate task node is used to clear the cache file.

[0139] The task processing apparatus 40 disclosed in this embodiment can execute the task processing method disclosed in this embodiment, and has the corresponding functional modules and beneficial effects for executing the method. Content not described in detail in this embodiment can be referred to the description in the method embodiments of this disclosure.

[0140] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0141] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0142] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0143] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0144] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0145] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as task processing methods. For example, in some embodiments, the task processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the task processing method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform task processing methods by any other suitable means (e.g., by means of firmware).

[0146] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0150] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0151] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0152] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0153] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A task processing method, comprising: obtaining at least two original task nodes contained in a to-be-processed task, and determining an external access file associated with the original task nodes; wherein the original task nodes are obtained by parsing a task processing statement of the to-be-processed task; and the external access file is an access file that needs to be read or written from a storage cluster when the task processing statement of the original task nodes is executed; determining a target access file from the external access file according to a number of the original task nodes associated with the external access file, and generating an auxiliary task node in the to-be-processed task according to the target access file; wherein the target access file is associated with at least two original task nodes; processing the to-be-processed task according to the auxiliary task node and the original task nodes; wherein the auxiliary task node is used to locally cache the target access file.

2. The method of claim 1, wherein, The determination of the external access file associated with the original task nodes comprises: parsing the task processing statement of the original task nodes to determine a file identifier contained in the task processing statement; determining the external access file associated with the original task nodes according to the file identifier.

3. The method of claim 1, wherein, The generation of the auxiliary task node according to the target access file comprises: regarding at least two original task nodes associated with the target access file as candidate task nodes; determining a target task node from the candidate task nodes according to a dependency relationship between the original task nodes; generating an auxiliary task node according to an access type of the target access file accessed by the target task node; wherein the access type comprises a read type and a write type.

4. The method of claim 3, wherein, The determination of the target task node from the candidate task nodes according to the dependency relationship between the original task nodes comprises: determining an execution priority of each of the candidate task nodes according to the dependency relationship, and regarding a candidate task node with the highest execution priority as the target task node.

5. The method of claim 3, wherein, The generation of the auxiliary task node according to the access type of the target access file accessed by the target task node comprises: in a case where the access type is the read type, creating a new task node as the auxiliary task node; wherein the execution priority of the auxiliary task node is higher than that of the target task node; and an auxiliary task processing statement of the auxiliary task node is used to cache the target access file in a cache file at a local end.

6. The method of claim 3, wherein, The generation of the auxiliary task node according to the access type of the target access file accessed by the target task node comprises: in a case where the access type is the write type, determining a cache file identifier of a cache file at a local end and a target file identifier of the target access file; replacing the target file identifier with the cache file identifier in a task processing statement of the target task node to generate an auxiliary task processing statement; regarding the target task node as the auxiliary task node according to the auxiliary task processing statement.

7. The method of claim 5 or 6, wherein, The processing of the to-be-processed task according to the auxiliary task node and the original task node comprises: determining a cache file identifier of the cache file and a target file identifier of the target access file; replacing the target file identifier with the cache file identifier in the task processing statement of the candidate task node to generate an updated task processing statement of the candidate task node; executing the auxiliary task processing statement, the updated task processing statement, and the task processing statement of the other original task nodes except the candidate task node to process the to-be-processed task.

8. The method of claim 5 or 6, after the processing of the to-be-processed task according to the auxiliary task node and the original task node, further comprising: determining an execution priority of each of the candidate task nodes according to the dependency relationship, and taking the candidate task node with the lowest execution priority as a terminated task node; creating a new task node as a cleanup task node; wherein the execution priority of the cleanup task node is lower than that of the terminated task node; the terminated task node is used to clean up the cache file.

9. A processing device of a task, comprising: a file determination module configured to acquire at least two original task nodes contained in a to-be-processed task and determine an external access file associated with the original task nodes; wherein the original task node is a task processing statement obtained by parsing the to-be-processed task; and the external access file is an access file that needs to be read or written from a storage cluster when the task processing statement of the original task node is executed; a node generation module configured to determine a target access file from the external access file according to a number of nodes of the original task nodes associated with the external access file, and generate an auxiliary task node in the to-be-processed task according to the target access file; wherein the target access file is associated with at least two original task nodes; a task processing module configured to process the to-be-processed task according to the auxiliary task node and the original task node; wherein the auxiliary task node is used to locally cache the target access file.

10. The apparatus of claim 9, wherein, The file determination module is specifically configured to: parse the task processing statement of the original task node to determine a file identifier contained in the task processing statement; determine the external access file associated with the original task node according to the file identifier.

11. The apparatus of claim 9, wherein, The node generation module is specifically configured to: take at least two original task nodes associated with the target access file as candidate task nodes; determine a target task node from the candidate task nodes according to a dependency relationship between the original task nodes; generate an auxiliary task node according to an access type of the target access file executed by the target task node; wherein the access type comprises a read type and a write type.

12. The apparatus of claim 11, wherein, The node generation module is specifically further configured to: determine an execution priority of each of the candidate task nodes according to the dependency relationship, and take the candidate task node with the highest execution priority as the target task node.

13. The apparatus of claim 11, wherein, The node generation module is further specifically configured to: In a case where the access type is a read type, a new task node is created as the auxiliary task node; the execution priority of the auxiliary task node is higher than that of the target task node; and the auxiliary task processing statement of the auxiliary task node is used to cache the target access file in a cache file on a local end.

14. The apparatus of claim 11, wherein, The node generation module is further specifically configured to: In a case where the access type is a write type, a cache file identifier of a cache file on the local end and a target file identifier of the target access file are determined; In the task processing statement of the target task node, the target file identifier is replaced by the cache file identifier to generate an auxiliary task processing statement; According to the auxiliary task processing statement, the target task node is used as the auxiliary task node.

15. The apparatus of claim 13 or 14, wherein, The task processing module is specifically configured to: determine a cache file identifier of the cache file and a target file identifier of the target access file; replace the target file identifier by the cache file identifier in the task processing statement of the candidate task node to generate an updated task processing statement of the candidate task node; execute the auxiliary task processing statement, the updated task processing statement, and a task processing statement of an original task node other than the candidate task node, so as to process the to-be-processed task.

16. The apparatus of claim 13 or 14, further comprising a cache clearing module, which is specifically configured to: determine execution priorities of the candidate task nodes according to the dependency relationship, and take a candidate task node with the lowest execution priority as a termination task node; creating a new task node as a cleanup task node; wherein the execution priority of the clearing task node is lower than that of the termination task node; and the termination task node is used to clear the cache file.

17. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-8.

19. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-8.

19. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-8.

Images