Recursive task processing method and device, computer equipment and storage medium
By identifying and decomposing recursive tasks into independent tasks and utilizing multiple computing nodes to execute them in parallel, the problems of low computational efficiency and poor scalability of recursive tasks are solved, thus achieving efficient processing of recursive tasks.
Patent Information
- Application Number
- CN202511072153.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-31
- Publication Date
- 2025-12-19
AI Technical Summary
In a big data environment, recursive tasks suffer from low computational efficiency, high resource consumption, and poor scalability, making it difficult to effectively handle recursive relationships in financial and medical scenarios.
By identifying whether a recursive task is an independent task, it is decomposed into multiple recursive subtasks that are executed in parallel, and processed using multiple computing nodes, thus reducing computational complexity.
It improves the computational efficiency and scalability of recursive tasks while reducing computational complexity.
Smart Images
Figure CN121166291A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of data processing, in particular to a processing method and device of recursive task, computer equipment and storage medium. BACKGROUND
[0002] In financial business, referral is an important link and means in customer management, and the referral contains recursive algorithm logic, which leads to a large number of recursive relationships in customer information and other data of financial business, such as risk assessment through interpersonal relationships between users, or abnormal identification of transaction data. In the medical field, long short-term memory network (LSTM) for processing medical images needs to capture time-dependent relationships in image sequences through recursion; when building a medical knowledge graph, nested structures in text data also need to be processed through recursive tasks. However, in the big data environment, the existence of recursive relationships can easily lead to low efficiency, high resource consumption and poor scalability of large-scale data. How to improve the computational efficiency and scalability of processing recursive tasks to improve the efficiency of big data processing in financial and medical scenarios is a problem that needs to be solved in the prior art. SUMMARY
[0003] The main purpose of the present application is to provide a processing method and device of recursive task, computer equipment and storage medium, which aims to improve the computational efficiency and scalability of processing recursive tasks.
[0004] In a first aspect, the present application provides a processing method of recursive task, comprising the following steps:
[0005] Obtaining a preset data processing statement, identifying a recursive task in the data processing statement;
[0006] When the recursive task in the data processing statement is identified, determining whether the recursive task belongs to an independent task based on a preset recursive task analysis algorithm;
[0007] In the case that the recursive task belongs to an independent task, decomposing the recursive task into a plurality of recursive subtasks according to the analysis result;
[0008] Sending the recursive subtasks to a plurality of computing nodes for execution respectively to obtain a plurality of execution subresults;
[0009] Reducing the execution subresults to obtain a task execution result of the recursive task.
[0010] In a second aspect, the present application further provides a processing device of recursive task, comprising:
[0011] The recursive task recognition module is configured to obtain a preset data processing statement and recognize a recursive task in the data processing statement.
[0012] The recursive task analysis module is configured to determine whether the recursive task belongs to an independent task based on a preset recursive task analysis algorithm when the recursive task in the data processing statement is recognized.
[0013] The recursive task decomposition module is configured to decompose the recursive task into a plurality of recursive subtasks according to the analysis result when the recursive task belongs to an independent task.
[0014] The recursive task execution module is configured to send the recursive subtasks to a plurality of computing nodes for execution to obtain a plurality of execution subresults.
[0015] The execution result reduction module is configured to perform reduction processing on the execution subresults to obtain a task execution result of the recursive task.
[0016] In a third aspect, the present application further provides a computer device, which comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the processing method of the recursive task.
[0017] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, wherein the computer program, when executed by a processor, implements the processing method of the recursive task.
[0018] The present application provides a processing method, device and equipment of a recursive task and a computer storage medium. The present application obtains a preset data processing statement, recognizes a recursive task in the data processing statement, determines whether the recursive task belongs to an independent task based on a preset recursive task analysis algorithm when the recursive task in the data processing statement is recognized, decomposes the recursive task into a plurality of recursive subtasks according to the analysis result when the recursive task belongs to an independent task, sends the recursive subtasks to a plurality of computing nodes for execution to obtain a plurality of execution subresults, and performs reduction processing on the execution subresults to obtain a task execution result of the recursive task. Since the recursive task is analyzed, and the recursive task is decomposed into a plurality of subtasks for execution in a plurality of computing nodes when the recursive task is determined to belong to an independent task, the computational complexity of the recursive task is reduced, and the computational efficiency and scalability of processing the recursive task are improved. BRIEF DESCRIPTION OF DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.
[0020] Figure 1 A flowchart of a processing method of a recursive task provided by an embodiment of the present application;
[0021] Figure 2 A use scenario diagram of a processing method of a recursive task provided by an embodiment of the present application;
[0022] Figure 3 A schematic block diagram of a processing device of a recursive task provided by an embodiment of the present application;
[0023] Figure 4 A structural schematic block diagram of a computer device related to an embodiment of the present application. DETAILED DESCRIPTION
[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without any creative effort fall within the protection scope of the present application.
[0025] The flowchart shown in the drawings is only an example, and it is not necessary to include all the contents and operations / steps, and it is not necessary to execute in the described order. For example, some operations / steps can be decomposed, combined or partially combined, so the actual execution order can be changed according to the actual situation.
[0026] The embodiments of the present application provide a processing method and device of a recursive task, a computer device and a computer readable storage medium.
[0027] Some embodiments of the present application will be described in detail below with reference to the drawings. In the case of no conflict, the embodiments described below and the features in the embodiments can be combined with each other.
[0028] Please refer to Figure 1 , Figure 1This is a flowchart illustrating a method for processing recursive tasks, provided as an embodiment of this application. This method can be used in a terminal or a server. The terminal can be an electronic device such as a mobile phone, tablet, laptop, desktop computer, personal digital assistant, or wearable device; the server can be a standalone server, a server cluster, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0029] Please refer to Figure 2 , Figure 2 This is a usage scenario diagram provided by an embodiment of this application. For example... Figure 2 As shown, the recursive task in the identified data processing statement is determined to be an independent task. If the recursive task is an independent task, it is decomposed into recursive subtasks and sent to various computing nodes for parallel execution. The computing nodes can be servers in a distributed computing system, but are not limited to this, and are not restricted here.
[0030] The recursive task processing method provided in this application can process a large number of recursive relationships in customer information and other data in financial business, enabling user risk assessment, anomaly identification in transaction data, etc. It can also be used to capture temporal dependencies in medical images and construct medical knowledge graphs.
[0031] like Figure 1 As shown, the processing method for this recursive task includes steps S101 to S105.
[0032] Step S101: Obtain a preset data processing statement and identify the recursive tasks in the data processing statement.
[0033] For example, data processing statements could be "Structured Query Language" (SQL).
[0034] For example, data processing statements can be transformed into abstract syntax trees (ASTs), and recursive tasks in the data processing statements can be determined by identifying self-referencing nodes in the syntax tree; alternatively, data processing statements can be transformed into directed cyclic graphs, and recursive tasks in the data processing statements can be determined by identifying the cycle structures in the directed cyclic graphs.
[0035] Step S102: When a recursive task is identified in the data processing statement, determine whether the recursive task is an independent task based on a preset recursive task analysis algorithm.
[0036] For example, if there are overlapping sub-problems in the recursive task, that is, the sub-tasks depend on the same set of intermediate states, the recursive task is not an independent task and cannot be completely decoupled into multiple sub-tasks; or the execution result after decomposition into sub-tasks is different from the original execution result of the recursive task, which also indicates that the recursive task is not an independent task. Therefore, the recursive task needs to be analyzed, and only when the recursive task is an independent task, the recursive task is decomposed.
[0037] In some embodiments, when the recursive task in the data processing statement is identified, whether the recursive task belongs to an independent task is determined based on a preset recursive task analysis algorithm, which comprises:
[0038] In the case where there is a preset statement in the data processing statement corresponding to the recursive task, it is determined that the recursive task is a non-independent task;
[0039] In the case where there is no preset statement in the data processing statement corresponding to the recursive task, a preset probe is inserted into the task entry of the recursive task, and in the case where the running result of the preset probe does not meet the preset condition, it is determined that the recursive task is a non-independent task;
[0040] In the case where the running result of the preset probe does not meet the preset condition, whether the recursive task belongs to an independent task is determined based on a preset task analysis model.
[0041] For example, if there are write operations on global variables, singletons, file handles, external databases, or data is passed between recursive calls through mutable objects (list, dict, object fields) in the data processing statement, it indicates that the recursive task obviously shares and modifies data through mutable objects between recursions, and therefore is not an independent task, which is filtered out.
[0042] For example, a small piece of data is preset as a lightweight preset probe, and the recursive task runs the preset probe, and according to the running result of the preset probe, recursive tasks that are not independent tasks are further filtered out.
[0043] For example, the preset probe filtered out by the first two rounds is determined whether it is an independent task by a task analysis model. Specifically, the task analysis model can be a machine learning model, such as a classifier, which extracts code features and running features of the recursive task, inputs the code features and running features into the task analysis model, obtains the classification result of the task analysis model, and determines whether the recursive task is an independent task. The code features include: recursive depth distribution, parameter type (value type / reference type), whether there is a lock, atomic operation; the running features include: cache hit rate, write conflict rate.
[0044] It can be understood that the calculation amount of the three screening operations increases in turn. The obviously non-independent recursive task is filtered out by the identification of the preset statement with a smaller calculation amount first. Then the recursive task is further filtered by the preset probe. Finally, the remaining recursive task is determined by the task analysis model, thereby reducing the calculation amount of the recursive task analysis as much as possible.
[0045] In some embodiments, the preset probe is inserted into the task entry of the recursive task. If the running result of the preset probe does not meet the preset condition, it is determined that the recursive task belongs to a non-independent task, which includes:
[0046] The preset probe is inserted into the task entry of the recursive task at least twice. The read-write address information in the execution process of the recursive task is recorded, and the probe execution result of the preset probe is obtained.
[0047] If the read-write address information and the probe execution result do not meet the preset condition, it is determined that the recursive task belongs to a non-independent task.
[0048] For example, the preset probe can be a small amount of preset data, which can be set according to actual needs, and is not limited herein. By making the recursive task run the preset probe, the read-write address information in the execution process of the recursive task is determined. Whether the recursive task belongs to an independent task is determined according to whether the read-write address information in the execution process of the recursive task overlaps. The probe execution result of the recursive task running the preset probe in different ways is obtained. Whether the recursive task belongs to an independent task is determined according to the probe execution result.
[0049] In some embodiments, the preset probe is inserted into the task entry of the recursive task at least twice. The write address in the execution process of the recursive task and the probe execution result are obtained. If the write address and the probe execution result do not meet the preset condition, it is determined that the recursive task belongs to a non-independent task, which includes:
[0050] The recursive task is decomposed into at least two probe sub-tasks. The preset probe is inserted into the task entry of the probe sub-task to obtain at least two first probe execution results of the probe sub-task.
[0051] The preset probe is inserted into the task entry of the recursive task to obtain a second probe execution result of the recursive task.
[0052] Exemplarily, the recursive task can be divided into two subtasks as probe subtasks, and the number of the probe subtasks is not limited; the probe subtasks are caused to run the preset probe to obtain a first probe execution result; the original recursive task is caused to run the preset probe to obtain a second probe execution result; and whether the recursive task belongs to an independent task is determined by comparing the first probe execution result and the second probe execution result.
[0053] In some embodiments, in the case that the read-write address information and the probe execution result do not meet the preset condition, it is determined that the recursive task belongs to a non-independent task, including:
[0054] In the case that the first probe execution result and the second probe execution result do not match, it is determined that the recursive task belongs to a non-independent task; or,
[0055] In the case that the read address information and the write address information in the read-write address information exist intersection, it is determined that the recursive task belongs to a non-independent task.
[0056] It can be understood that, if the first probe execution result of the recursive task divided into at least two subtasks is different from the second probe execution result of the recursive task as a whole, it indicates that the decomposition into subtasks will affect the execution of the recursive task, and the recursive task can only be executed directly as a whole, and the recursive task does not belong to an independent task.
[0057] Exemplarily, the read-write address information of the recursive task can be recorded by Intel PIN, Valgrind, Python sys.settrace and the like, wherein the read-write address information includes read address information and write address information, and if the read address information and the write address information overlap in the process of execution of the recursive task, that is, the subtasks depend on the same set of intermediate states, the decomposition into subtasks will cause a large amount of repeated calculation, and thus the recursive task does not belong to an independent task.
[0058] In the case that the recursive task belongs to an independent task, the recursive task is divided into multiple recursive subtasks according to the analysis result.
[0059] Exemplarily, the calculation of the recursive task needs to create a new frame (storing local variables, return addresses, etc.) in the call stack, and the stack space of the computer device is limited, especially in the big data scenario. In order to avoid the risk of stack overflow, performance degradation and uncontrollable memory, it is usually necessary to limit the depth of the recursive task, resulting in that in the financial database with recursive relationships such as transfer, the data mining has great implementation difficulty.
[0060] For example, in order to avoid too large calculation overhead caused by single machine computing recursive tasks, in the case that the recursive task belongs to an independent task, the recursive task can be decomposed into multiple recursive subtasks, and no longer recursively to the bottom as the single machine. Specifically, the to-be-computed data required for computing the recursive task can be split into multiple groups of data, and the multiple groups of data and the recursive task are respectively sent to different computing nodes for parallel computing.
[0061] In some embodiments, in the case that the recursive task belongs to an independent task, decomposing the recursive task into multiple recursive subtasks according to the analysis result further includes:
[0062] In the case that the recursive task belongs to an independent task, determining a target decomposition granularity of the recursive task according to the analysis result;
[0063] Based on the decomposition granularity, decomposing the recursive task into multiple recursive subtasks.
[0064] For example, in the process of decomposing the recursive task, the granularity of decomposition determines the calculation amount of each computing node and the communication overhead of each computing node. If the granularity is too large, the calculation amount of each node will be too large; if the granularity is too small, the calculation overhead will be too large. Therefore, the target decomposition granularity of the recursive task needs to be determined according to the analysis result.
[0065] In some embodiments, in the case that the recursive task belongs to an independent task, determining a target decomposition granularity of the recursive task according to the analysis result includes:
[0066] Determining the calculation amount of the recursive task under different decomposition granularities according to the analysis result and the to-be-computed data;
[0067] Determining the communication overhead of the recursive task under different decomposition granularities according to the to-be-computed data;
[0068] Iterating the calculation amount and the communication overhead under different decomposition granularities to determine the target decomposition granularity.
[0069] For example, in the process of analyzing the recursive task, the calculation amount of the recursive task can be determined, and the calculation amount of the recursive task is related to the data amount of the to-be-computed data in addition to the processing logic of the recursive task itself. Therefore, the calculation amount of the recursive task under different decomposition granularities is determined according to the analysis result and the to-be-computed data. For example, the calculation amount of directly executing the recursive task, the calculation amount of decomposing the recursive task into 2 subtasks, and so on. Specifically, the calculation amount can be represented by the calculation time, for example, the calculation amount of directly executing the recursive task is 20s, and the calculation amount of decomposing the recursive task into 2 subtasks is 10s.
[0070] Exemplarily, the communication overhead can also be represented by a communication duration, which is a cumulative duration of the plurality of computing nodes. For example, if the recursive task is divided into two subtasks, the two subtasks are sent to two computing nodes, and the communication overhead is 2s, and so on.
[0071] Exemplarily, the processing efficiency of each decomposition granularity is determined by traversing the computation amount and the communication overhead under different decomposition granularities. For example, different weights can be assigned to the computation amount and the communication overhead to calculate an efficiency score under different granularities, where the smaller the computation amount, the larger the efficiency score, and the smaller the communication overhead, the larger the efficiency score. The decomposition granularity with the highest efficiency score is determined as the target decomposition granularity.
[0072] Step S104: The recursive subtasks are sent to the plurality of computing nodes for execution to obtain a plurality of execution subresults.
[0073] Exemplarily, the computing nodes can be servers in a distributed computing system for performing calculations. Each computing node executes the recursive subtasks in parallel to obtain execution subresults, thereby reducing the duration required for executing the recursive task.
[0074] Step S105: The execution subresults are reduced to obtain a task execution result of the recursive task.
[0075] Exemplarily, the order of the execution subresults is determined according to the order of the data to be calculated allocated to each computing node, and the execution subresults are spliced to obtain the task execution result of the recursive task.
[0076] The processing method of the recursive task provided in the above embodiments includes the following steps: a predetermined data processing statement is obtained, and a recursive task in the data processing statement is identified; when the recursive task in the data processing statement is identified, it is determined whether the recursive task belongs to an independent task based on a predetermined recursive task analysis algorithm; in the case where the recursive task belongs to an independent task, the recursive task is divided into a plurality of recursive subtasks according to the analysis result; the recursive subtasks are sent to a plurality of computing nodes for execution to obtain a plurality of execution subresults; and the execution subresults are reduced to obtain a task execution result of the recursive task. Since the recursive task is analyzed, and the recursive task is divided into a plurality of subtasks for execution in a plurality of computing nodes in the case where the recursive task belongs to an independent task, the computational complexity of the recursive task is reduced, and the computational efficiency and scalability of processing the recursive task are improved.
[0077] Please refer to Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a processing device of a recursive task according to an embodiment of the present application. The processing device of the recursive task can be configured in a server or a terminal, and is used to execute the processing method of the recursive task described above.
[0078] As shown in Figure 3 The processing apparatus of the recursive task includes a recursive task identification module 110, a recursive task analysis module 120, a recursive task decomposition module 130, a recursive task execution module 140, and an execution result reduction module 150.
[0079] The recursive task identification module 110 is configured to obtain a preset data processing statement and identify a recursive task in the data processing statement.
[0080] The recursive task analysis module 120 is configured to determine whether the recursive task belongs to an independent task based on a preset recursive task analysis algorithm when the recursive task in the data processing statement is identified.
[0081] The recursive task decomposition module 130 is configured to decompose the recursive task into a plurality of recursive subtasks according to the analysis result in the case where the recursive task belongs to an independent task.
[0082] The recursive task execution module 140 is configured to send the recursive subtasks to a plurality of computing nodes for execution to obtain a plurality of execution subresults.
[0083] The execution result reduction module 150 is configured to reduce the execution subresults to obtain a task execution result of the recursive task.
[0084] In some embodiments, the determination of whether the recursive task belongs to an independent task based on a preset recursive task analysis algorithm when the recursive task in the data processing statement is identified includes:
[0085] In the case where a preset statement exists in the data processing statement corresponding to the recursive task, it is determined that the recursive task belongs to a non-independent task.
[0086] In the case where a preset statement does not exist in the data processing statement corresponding to the recursive task, a preset probe is inserted into a task entry of the recursive task, and in the case where a running result of the preset probe does not meet a preset condition, it is determined that the recursive task belongs to a non-independent task.
[0087] In the case where the running result of the preset probe does not meet the preset condition, it is determined whether the recursive task belongs to an independent task based on a preset task analysis model.
[0088] In some embodiments, the insertion of the preset probe into the task entry of the recursive task and the determination of whether the recursive task belongs to an independent task in the case where the running result of the preset probe does not meet the preset condition include:
[0089] inserting the preset probe into a task entry of the recursive task at least twice, recording read-write address information in a process of executing the recursive task, and obtaining a probe execution result of the preset probe;
[0090] In a case where the read-write address information and the probe execution result do not meet preset conditions, it is determined that the recursive task belongs to a non-independent task.
[0091] In some embodiments, the at least twice inserting the preset probe into a task entry of the recursive task, obtaining a write address and a probe execution result in a process of executing the recursive task, and in a case where the write address and the probe execution result do not meet preset conditions, determining that the recursive task belongs to a non-independent task, comprises:
[0092] decomposing the recursive task into at least two probe sub-tasks, inserting the preset probe into a task entry of the probe sub-task, and obtaining at least two first probe execution results of the probe sub-task;
[0093] inserting the preset probe into a task entry of the recursive task, and obtaining a second probe execution result of the recursive task.
[0094] In some embodiments, the in a case where the read-write address information and the probe execution result do not meet preset conditions, determining that the recursive task belongs to a non-independent task, comprises:
[0095] In a case where the first probe execution result and the second probe execution result do not match, it is determined that the recursive task belongs to a non-independent task; or,
[0096] In a case where read address information and write address information in the read-write address information exist an intersection, it is determined that the recursive task belongs to a non-independent task.
[0097] In some embodiments, in a case where the recursive task belongs to an independent task, decomposing the recursive task into a plurality of recursive sub-tasks according to the analysis result, further comprises:
[0098] In a case where the recursive task belongs to an independent task, determining a target decomposition granularity of the recursive task according to the analysis result;
[0099] decomposing the recursive task into a plurality of recursive sub-tasks based on the decomposition granularity.
[0100] In some embodiments, the in a case where the recursive task belongs to an independent task, determining a target decomposition granularity of the recursive task according to the analysis result, comprises:
[0101] determining a calculation amount of the recursive task under different decomposition granularities according to the analysis result and to-be-calculated data;
[0102] determining the communication overhead of the recursive task at different decomposition granularities according to the to-be-calculated data;
[0103] performing traversal on the computation amount and the communication overhead at different decomposition granularities to determine the target decomposition granularity.
[0104] It should be noted that, for the convenience and brevity of description, the specific working process of the above-described apparatus and each module and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.
[0105] The method and apparatus of the present application can be used in many general or special-purpose computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.
[0106] Exemplarily, the above-described method and apparatus can be implemented in the form of a computer program that can run on a computer device as shown in Figure 4 .
[0107] Please refer to Figure 4 , Figure 4 for a structural schematic block diagram of a computer device provided by an embodiment of the present application. The computer device can be a server or a terminal.
[0108] As shown in Figure 4 , the computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory can include a storage medium and an internal memory.
[0109] The storage medium can store an operating system and a computer program. The computer program includes program instructions that, when executed, can cause the processor to perform a processing method of any recursive task.
[0110] The processor is configured to provide computing and control capabilities to support the operation of the entire computer device.
[0111] The internal memory provides an environment for the running of a computer program in a storage medium, which, when executed by the processor, can cause the processor to perform any kind of processing method of a recursive task.
[0112] The network interface is used for network communication, such as sending an assigned task. Those skilled in the art can understand that, Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0113] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0114] In one embodiment, the processor is configured to run a computer program stored in the memory to perform the following steps:
[0115] Obtain a preset data processing statement, and identify a recursive task in the data processing statement;
[0116] When the recursive task in the data processing statement is identified, determine whether the recursive task belongs to an independent task based on a preset recursive task analysis algorithm;
[0117] In the case where the recursive task belongs to an independent task, decompose the recursive task into a plurality of recursive subtasks according to the analysis result;
[0118] Send the recursive subtasks to a plurality of computing nodes for execution respectively, to obtain a plurality of execution subresults;
[0119] Reduce the execution subresults to obtain a task execution result of the recursive task.
[0120] In some embodiments, the determining, when the recursive task in the data processing statement is identified, whether the recursive task belongs to an independent task based on a preset recursive task analysis algorithm comprises:
[0121] In a case where a preset statement exists in the data processing statement corresponding to the recursive task, it is determined that the recursive task belongs to a non-independent task.
[0122] In a case where a preset statement does not exist in the data processing statement corresponding to the recursive task, a preset probe is inserted into a task entry of the recursive task, and in a case where a running result of the preset probe does not meet a preset condition, it is determined that the recursive task belongs to a non-independent task.
[0123] In a case where the running result of the preset probe does not meet the preset condition, it is determined, based on a preset task analysis model, whether the recursive task belongs to an independent task.
[0124] In some embodiments, the inserting, into a task entry of the recursive task, a preset probe and determining, in a case where a running result of the preset probe does not meet a preset condition, that the recursive task belongs to a non-independent task comprises:
[0125] The preset probe is inserted into the task entry of the recursive task at least twice, read-write address information in an execution process of the recursive task is recorded, and a probe execution result of the preset probe is obtained.
[0126] In a case where the read-write address information and the probe execution result do not meet a preset condition, it is determined that the recursive task belongs to a non-independent task.
[0127] In some embodiments, the inserting, into a task entry of the recursive task, a preset probe at least twice, obtaining a write address in an execution process of the recursive task and a probe execution result, and determining, in a case where the write address and the probe execution result do not meet a preset condition, that the recursive task belongs to a non-independent task comprises:
[0128] The recursive task is decomposed into at least two probe subtasks, the preset probe is inserted into a task entry of the probe subtask, and at least two first probe execution results of the probe subtask are obtained.
[0129] The preset probe is inserted into the task entry of the recursive task, and a second probe execution result of the recursive task is obtained.
[0130] In some embodiments, the determining, in a case where the read-write address information and the probe execution result do not meet a preset condition, that the recursive task belongs to a non-independent task comprises:
[0131] determining that the recursive task belongs to a dependent task in a case where the first probe execution result and the second probe execution result do not match; or
[0132] determining that the recursive task belongs to a dependent task in a case where there is an intersection between the read address information and the write address information in the read-write address information.
[0133] In some embodiments, in a case where the recursive task belongs to an independent task, the recursive task is decomposed into a plurality of recursive sub-tasks according to the analysis result, and the method further comprises:
[0134] determining a target decomposition granularity of the recursive task according to the analysis result in a case where the recursive task belongs to an independent task;
[0135] decomposing the recursive task into a plurality of recursive sub-tasks based on the decomposition granularity.
[0136] In some embodiments, in a case where the recursive task belongs to an independent task, the target decomposition granularity of the recursive task is determined according to the analysis result, and the method further comprises:
[0137] determining a computation amount of the recursive task under different decomposition granularities according to the analysis result and to-be-computed data;
[0138] determining a communication overhead of the recursive task under different decomposition granularities according to the to-be-computed data;
[0139] performing traversal on the computation amount and the communication overhead under different decomposition granularities to determine the target decomposition granularity.
[0140] It should be noted that, for the convenience and brevity of description, the specific working process of the computer device can be referred to the corresponding process in the foregoing embodiments of the processing method of the recursive task, and will not be described herein.
[0141] The embodiments of the present application further provide a computer readable storage medium, and the computer readable storage medium stores a computer program, the computer program comprises program instructions, and the method realized when the program instructions are executed can refer to each embodiment of the processing method of the recursive task of the present application.
[0142] The computer readable storage medium can be an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device. The computer readable storage medium can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like.
[0143] It should be understood that the terms used herein are merely for the purpose of describing particular embodiments and are not intended to limit the present application. As used in the specification and the appended claims, the singular forms "a," "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise.
[0144] It should also be understood that the term "and / or" as used herein refers to any combination of associated terms, and all possible combinations, and includes these combinations. It should be noted that the terms "comprise", "comprising", or any other variant thereof, are intended to encompass non-exclusive inclusion, so that processes, methods, articles, or systems that include a series of elements include not only those elements, but also other elements not explicitly listed, or other elements inherent in such processes, methods, articles, or systems. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article, or system that includes the element.
[0145] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A method of processing recursive tasks, characterized by, The method comprises: acquiring a preset data processing statement, and identifying a recursive task in the data processing statement; when the recursive task in the data processing statement is identified, determining whether the recursive task belongs to an independent task based on a preset recursive task analysis algorithm; in the case where the recursive task belongs to an independent task, decomposing the recursive task into a plurality of recursive subtasks according to an analysis result; sending the recursive subtasks to a plurality of computing nodes for execution respectively to obtain a plurality of execution subresults; performing reduction processing on each execution subresult to obtain a task execution result of the recursive task.
2. The method of claim 1, wherein, The method comprises: in the case where the recursive task corresponds to a data processing statement containing a preset statement, determining that the recursive task belongs to a non-independent task; in the case where the recursive task corresponds to a data processing statement not containing a preset statement, inserting a preset probe into a task entry of the recursive task, and in the case where a running result of the preset probe does not meet a preset condition, determining that the recursive task belongs to a non-independent task; in the case where the running result of the preset probe does not meet the preset condition, determining whether the recursive task belongs to an independent task based on a preset task analysis model.
3. The method of claim 2, wherein, The method comprises: inserting the preset probe into the task entry of the recursive task at least twice, recording read-write address information in an execution process of the recursive task, and acquiring a probe execution result of the preset probe; in the case where the read-write address information and the probe execution result do not meet a preset condition, determining that the recursive task belongs to a non-independent task.
4. The method of claim 3, wherein, The method comprises: decomposing the recursive task into at least two probe subtasks, inserting the preset probe into a task entry of the probe subtask to obtain at least two first probe execution results of the probe subtask; inserting the preset probe into the task entry of the recursive task to obtain a second probe execution result of the recursive task.
5. The method of claim 4, wherein, The method comprises: in the case where the first probe execution result and the second probe execution result do not match, determining that the recursive task belongs to a non-independent task; or in the case where read address information and write address information in the read-write address information have an intersection, determining that the recursive task belongs to a non-independent task.
6. The method of claim 1, wherein, The recursive task is divided into multiple recursive sub-tasks according to the analysis result in the case that the recursive task belongs to an independent task, and the method further includes: In the case that the recursive task belongs to an independent task, the target decomposition granularity of the recursive task is determined according to the analysis result; Based on the decomposition granularity, the recursive task is divided into multiple recursive sub-tasks.
7. The method of claim 6, wherein, The target decomposition granularity of the recursive task is determined according to the analysis result in the case that the recursive task belongs to an independent task, and the method further includes: The calculation amount of the recursive task under different decomposition granularities is determined according to the analysis result and the to-be-calculated data; The communication overhead of the recursive task under different decomposition granularities is determined according to the to-be-calculated data; The calculation amount and the communication overhead under different decomposition granularities are traversed to determine the target decomposition granularity.
8. A processing device of a recursive task, characterized by, The processing device of the recursive task includes: A recursive task identification module is configured to acquire a preset data processing statement and identify a recursive task in the data processing statement; A recursive task analysis module is configured to determine whether the recursive task belongs to an independent task based on a preset recursive task analysis algorithm when the recursive task in the data processing statement is identified; A recursive task decomposition module is configured to divide the recursive task into multiple recursive sub-tasks according to an analysis result in the case that the recursive task belongs to an independent task; A recursive task execution module is configured to send the recursive sub-tasks to multiple computing nodes for execution to obtain multiple execution sub-results; An execution result reduction module is configured to perform reduction processing on the execution sub-results to obtain a task execution result of the recursive task.
9. A computer device, comprising: The computer device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the processing method of the recursive task according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein when the computer program is executed by the processor, the steps of the processing method of the recursive task according to any one of claims 1 to 7 are implemented.