A distributed task processing method and device, a terminal device and a storage medium

Abstract

The application is suitable for the technical field of distributed task processing, and provides a distributed task processing method and device, terminal equipment and storage medium. In the embodiment of the application, each task in a preset storage space is periodically detected; when it is detected that a target task exists, the calculation marks of each subtask in the target task are detected, and the subtasks without calculation marks are calculated and processed; the target task is a task containing a start mark; when it is detected that each subtask in the target task meets a preset standard, the target task is marked as completed, thereby improving the stability in the task processing process.

Description

Technical Field

[0001] This application belongs to the field of distributed task processing technology, and in particular relates to a distributed task processing method, apparatus, terminal device and storage medium. Background Technology

[0002] With societal development, people are increasingly valuing task processing efficiency. Currently, commonly used task processing methods primarily rely on specific computer clusters. These clusters require MPI interfaces for task allocation and scheduling, necessitating the MPI master process to communicate with nodes via the network to allocate tasks and coordinate the computation of subtasks. After computation, the master process collects the results. However, due to the excessive centralization of task processing by the MPI master process, if a computation node crashes, the tasks assigned to that node cannot be completed, causing other nodes to wait, resulting in low stability during task processing. Summary of the Invention

[0003] This application provides a distributed task processing method, apparatus, terminal device, and storage medium, which can solve the problem of low stability during task processing.

[0004] In a first aspect, embodiments of this application provide a distributed task processing method, including:

[0005] Periodically check each task in the preset storage space;

[0006] When a target task is detected, the calculation markers of each subtask in the target task are checked, and the subtasks without calculation markers are calculated; the target task is a task that includes a start marker.

[0007] When each subtask in the above target task is detected to meet the preset criteria, the above target task is marked as completed.

[0008] Secondly, embodiments of this application provide a distributed task processing apparatus, including:

[0009] The periodic detection module is used to periodically detect each task in the preset storage space;

[0010] The calculation mark detection module is used to detect the calculation marks of each subtask in the target task when the target task is detected, and to perform calculation processing on the subtasks that do not have calculation marks; the target task is a task that includes a start mark.

[0011] The completion marking module is used to mark the target task as completed when it is detected that each subtask in the target task meets the preset criteria.

[0012] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of any of the above-mentioned distributed task processing methods.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described distributed task processing methods.

[0014] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute any of the distributed task processing methods described in the first aspect.

[0015] In this embodiment, by periodically detecting each task in the preset storage space, the terminal device is encouraged to freely detect the tasks that need to be processed. When a target task is detected, the calculation markers of each subtask in the target task are then detected, and the subtasks without calculation markers are processed, thereby improving the calculation speed of the target task. The target task is a task that includes a start marker. When each subtask in the target task meets a preset standard, it indicates that the target task has been processed and a completion marker is then added to the target task. This allows the terminal device to clearly identify the target task as completed when it detects each task in the preset storage space again. By storing tasks in the storage space, the terminal device is encouraged to freely detect the tasks that need to be processed in the storage space. Even if an unexpected situation occurs during the processing of a subtask by a certain terminal device, other terminal devices will process the unprocessed subtasks when the target task is detected, further improving the stability of the task processing process. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the first type of distributed task processing method provided in the embodiments of this application;

[0018] Figure 2 This is a schematic diagram of the second flow of the distributed task processing method provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the structure of the distributed task processing device provided in the embodiments of this application;

[0020] Figure 4 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0023] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0025] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] Figure 1The diagram shown is a flowchart of a distributed task processing method according to an embodiment of this application. The execution subject of this method can be a terminal device, which can be a computing node. A computing node is a computer and its supporting equipment that can be used to complete computing tasks. In a high-performance computing environment, a computing node is generally equipped with at least two physical or logical cores, as well as shared memory and storage disks. This embodiment uses a computing node as an example for illustration. Figure 1 As shown, the above distributed task processing method may include the following steps:

[0027] Step S101: Periodically check each task in the preset storage space.

[0028] In this embodiment, the aforementioned period can be determined according to requirements. For example, it can be set to 5 seconds, meaning that the computing node checks each task in the preset storage space every 5 seconds. The preset storage space is a shareable storage space, meaning that it can provide data storage media for at least two computing nodes participating in task computation to access and read / write simultaneously. The preset storage space includes physical storage devices stored locally and logical storage spaces accessed via the network. The physical storage devices include, but are not limited to, disk arrays and flash drives, while the logical storage spaces include, but are not limited to, cloud storage spaces and virtual data warehouses.

[0029] Understandably, since computing nodes only need to access the preset storage space to perform computational processing on the required tasks, and there is no need for communication or dependency between computing nodes, there is no need for local area network communication between computing nodes, and no need for MPI coordination. Therefore, the task processing no longer exhibits a centralized phenomenon, and there is no concept of a main process or control process. With the preset storage space existing stably, the crash of any computing node will not lead to task computation failure or subtask loss, thereby improving the stability of task processing in parallel computing.

[0030] Step S102: When a target task is detected, the calculation markers of each subtask in the target task are detected, and the subtasks without calculation markers are calculated; the target task is the task containing the start marker.

[0031] In this embodiment, the target task corresponds to a large-scale computing task, equivalent to handling a large computing problem. It can exist in a preset storage space in the format of a folder. When a computing node detects the existence of a target task in the preset storage space, it indicates that there is a target task that needs to be processed by the computing node. The computing node then continues to detect the subtasks in the target task. Each subtask corresponds to breaking down a large computing problem into smaller computing units. By processing each subtask, the computing node can speed up the processing. Furthermore, when at least two computing nodes access the preset storage space simultaneously, these at least two computing nodes can simultaneously process each subtask in the target task that does not have a computing mark. Before processing, it is not necessary to determine which computing node will process the subtask, thereby realizing the automatic allocation and distributed parallel computing of a large number of subtasks in the target task. The subtasks in the preset storage space can be stored in a flattened format in the folder corresponding to the target task. The preset format includes, but is not limited to, folder format, subdirectory format, etc.

[0032] Understandably, this application uses pre-defined storage space to allocate and manage subtasks. This allows computing nodes to determine whether a subtask has been computed by detecting computation markers; this task information is equivalent to the computation status of the subtask. Therefore, direct communication between computing nodes is unnecessary, and complex job scheduling software and MPI programming are not required. This facilitates easy collaboration between computing nodes of different types, locations, and architectures. By setting shell scripts in the computing nodes to run two simple programs (a detection program and a computation processing program), any computing node can participate in the parallel computation of numerous tasks (e.g., various subtasks within a target task). This significantly reduces the cost and time of building computer clusters and lowers maintenance complexity. It is suitable for distributed computing of large tasks when direct management and synchronization between computing nodes via network communication is not feasible or convenient.

[0033] Optionally, when detecting the computation markers of each subtask in the target task and processing subtasks without computation markers, the aforementioned preset storage space can be accessed by at least two computing nodes simultaneously. This allows for more flexible configuration of the subtasks in the target task. For example, based on the task computation speed and cost, the number of computing nodes participating in subtask computation can be reduced during the entire target task processing process, or computing nodes outside the cluster's local area network can be added to participate in the task computation. These computing nodes outside the cluster's local area network only need to have the ability to read and write to the preset storage space to participate. However, the computational capabilities and architectures of different computing nodes may not be the same. Therefore, when considering computation speed and cost, different types of computing nodes need to be considered separately to improve the accuracy of resource configuration. It is understood that computing nodes coordinate by setting computation markers for subtasks, without needing to set up a master process or control process, and without needing to pre-lock a certain number of subtasks for computation. New computing nodes can be added or existing computing nodes can be removed at any time during the computation process. Each computing node runs independently, unaffected by other nodes, thus greatly improving the flexibility of computing resource scheduling.

[0034] Specifically, before detecting the existence of a target task, the task needs to be divided into subtasks, and the computational data required for each subtask should be stored in a corresponding subfolder, i.e., one subfolder corresponds to one subtask. Alternatively, the computational data required for each subtask can be stored in other containers that can describe parallel data structures. This process of dividing the task into subtasks can be performed by the local computer and then uploaded to a preset storage space, or it can be performed by the computing node. The specific division process can be determined based on the characteristics of the task and is not limited here. For example, if the task is the multiplication of a large matrix A*B, then multiplying each row of A by each column of B can be considered a subtask. Correspondingly, the computational data required for each subtask in the subfolder would be one row of A and one column of B.

[0035] In one embodiment, the specific method for detecting the target task is as follows: A computing node detects whether a start marker exists in the task. This start marker can be determined using a target start marker file. If a target start marker file is detected, it indicates that the task has a start marker, and the task is a target task, requiring computation and processing of its various subtasks. It can be understood that the target start marker file is equivalent to a global start instruction for the entire task, informing at least one computing node capable of recognizing the task that it can compute and process the task. Conversely, if a completion marker is detected after processing the task, this completion marker can be determined using a target end marker file. If a target end marker file is detected, it indicates that the target task has been computed and processed. This target end marker file is equivalent to a global end instruction for the entire task, informing at least one computing node that has already recognized or can recognize the task that the task has been processed, thereby preventing computing nodes from repeatedly computing the same task. Both the target start marker file and the target end marker file can be 0-byte files.

[0036] It is understandable that the process of a computing node detecting and marking tasks does not affect its ability to perform other computational tasks. That is, the aforementioned detection and computational processing procedures enable work switching control for the computing node. While the node is performing detection, its computing power can still be used for other purposes, thus avoiding idleness and waste. The computational processing procedure is only initiated after the detection program detects the target task and any subtasks within that target task that do not contain computational markers, allowing for actual computation. The computing node utilizes the detection and computational processing procedures to manage the allocation and release of its computing power.

[0037] Optionally, when a user wants a computing node to compute a certain task, they can control the computing node, host computer, or other terminals to write a target start marker file to the folder of the corresponding task in the preset storage space. The target end marker file will be automatically written after the task computation is completed.

[0038] Furthermore, since the same task may involve multiple calculations, and the calculation data of each subtask in the task used in each calculation is not the same, in order to prevent the calculation node from mistakenly believing that the current task's calculation is complete due to the completion mark of the previous calculation, after detecting the existence of a target task in step S102 above, that is, indicating that there is a start mark in the task, it may also include: the calculation node detecting whether there is a completion mark in the target task, that is, the target end mark file. If the target task has a completion mark, the marking time of the completion mark and the marking time of the start mark are determined. If the marking time of the start mark is after the marking time of the completion mark, it means that the completion mark of the task after the previous processing has not been eliminated, so the completion mark of the target task is canceled, and the target task calculation process is performed again.

[0039] In one embodiment, computing nodes sequentially detect each subtask in the target task in a preset order. Based on the presence or absence of computation markers in the subtasks, the nodes determine which subtasks require computation. If a computation marker is present in the currently detected subtask, the computing node skips it and proceeds to the next subtask. This preset order can be customized according to user needs. For example, it can be sorted by the filenames of the subtasks within the target task. This order ensures that at least two computing nodes capable of processing the target task detect the subtasks in the same order.

[0040] In one embodiment, the detection of calculation markers for each subtask in the target task in step S102 may include: the computing node detects files in the subtask, and if a start marker file and an end marker file are detected in the subtask, it indicates that the subtask has been processed and the subtask has a calculation marker. This causes the computing node to determine that the subtask does not need to be processed when it detects that a calculation marker exists in the subtask, thereby avoiding the task from being repeatedly calculated and processed.

[0041] It is understandable that each time a task is updated, the folders of each subtask within the task will be updated, so there will be no remnants of the end marker files from the previous task's calculation and processing in the subtasks.

[0042] Accordingly, the detection of calculation markers for each subtask in the target task in step S102 above may further include: the computing node detects files in the subtask, and if it detects that there is no start marker file and no end marker file in the subtask, it means that the subtask has not been calculated and processed. Therefore, it is determined that the subtask does not have a calculation marker, so that the computing node performs calculation processing on the subtask when it detects that there is no calculation marker in the subtask, generates a start marker file, so as to prevent the task from being repeatedly calculated and processed because other computing nodes have not found the calculation marker. After the computing node completes the calculation and processing, it writes the end marker file and then continues to detect the next subtask.

[0043] In one embodiment, sometimes a computing node may experience insufficient computing power or crash due to external or internal factors, causing the subtasks it processes to fail. To prevent subtasks from taking too long to compute or failing to compute due to computing node issues, such as... Figure 2 As shown, the calculation of the tags for each subtask in the target task in step S102 above may include:

[0044] Step S201: Detect the files in the subtask.

[0045] Step S202: If a start marker file exists in the subtask but no end marker file exists, then obtain the current time and the write time of the start marker file.

[0046] Step S203: Calculate the time difference between the write time and the current time.

[0047] Step S204: If the time difference is greater than or equal to the preset calculation time threshold, then it is determined that the subtask does not have a calculation marker.

[0048] Step S205: If the time difference is less than the preset calculation time threshold, then it is determined that the subtask has a calculation marker.

[0049] In this embodiment, a computing node can determine that a subtask is being processed by another computing node by the presence of a start marker file and the absence of an end marker file. To prevent excessive processing time for a subtask, the computing node compares the time difference between the current moment and the time the start marker file was written to determine if the task processing time is too long. If the time difference is greater than or equal to a preset computation time threshold, it indicates that the task's computation time is too long, possibly due to a problem with the computing node that previously processed the task. Therefore, it is determined that the subtask lacks a computation marker, and the current computing node takes over the subtask, re-writes the start marker file into the subtask, performs the computation, and writes the end marker file after the computation is complete. Thus, even if a computing node crashes or becomes unresponsive for an extended period during computation, the corresponding computation task can still be detected and completed by other computing nodes, improving the efficiency of the target task's computation and reducing the waiting time of slow or faulty nodes.

[0050] In one embodiment, the calculation processing of the subtask without a calculation mark in step S102 may include: the computing node writes a preset start mark file into the subtask without a calculation mark, then obtains the data file in the subtask without a calculation mark, and performs calculations on the data file according to the executable program in the subtask without a calculation mark; when the calculation is completed, the calculation result is written into the subtask without a calculation mark through the executable program. The calculation result can be stored not only in a preset storage space, but also in a place required by the user through other information transmission methods such as email. Then, a preset end mark file is written into the subtask without a calculation mark, so that the subtask without a calculation mark has a calculation mark through the writing of the start mark file and the end mark file, proving that the subtask has been calculated and processed.

[0051] Step S103: When it is detected that each subtask in the target task meets the preset criteria, the target task is marked as completed.

[0052] In this embodiment, when the computing node detects that the completion status of each subtask in the target task meets the preset standard, it marks the target task as completed, that is, writes it into the target end marker file. Then the computing node can continue to detect the next task to detect whether the next target task exists in the preset storage space.

[0053] In one embodiment, detecting that each subtask in the target task meets a preset standard may include: if the target task meets a first task type, for example, the multiplication of a large matrix A*B, then when the computing node detects that there are calculation markers in each subtask of the target task, it determines that the completion status of each subtask in the target task meets the preset standard corresponding to the first task type.

[0054] In one embodiment, detecting that each subtask in the target task meets a preset standard may include: if the target task conforms to a second task type, for example, deep learning requires generating a large number of synthetic samples (e.g., hundreds of thousands), and applying these synthetic samples to the training process of the neural network does not require completing all computations, for example, generating 90% of the samples is acceptable. If a computing node continuously detects that a preset number of subtasks all have computational markers, then it determines that the completion status of each subtask in the target task meets the preset standard corresponding to the second task type.

[0055] Specifically, the computing node does not need to sequentially detect each subtask in the target task. Instead, it randomly detects a subtask. If the currently detected subtask has a calculation mark, it means that the subtask has been processed, so the counter is incremented, and the next subtask is detected simultaneously. If the currently detected subtask does not have a calculation mark, it means that the subtask has not been processed, and the subtask is processed. After the processing is complete, the next subtask is detected, and the counter is reset to zero. In this embodiment, the computing node can set a threshold, namely the aforementioned preset number. When the counter reaches the preset number, it means that the preset number of continuously detected subtasks have been processed, and the processing of the target task is considered complete. The computing node stops processing the target task and marks the target task as complete. It is understood that the method in this embodiment, which determines whether a task has been completed by continuously detecting the computational markers of a preset number of subtasks, is suitable for scenarios where only a certain proportion of subtasks need to complete the computation to meet the requirements, such as neural network training scenarios. This avoids the process of each subtask being repeatedly detected by at least two computational nodes involved in the computation, thus reducing the read and write burden of the preset storage space.

[0056] In one embodiment, after step S103, it is indicated that the computation of the current target task has ended, but there is still a start mark in the current target task. In order to prevent the computing node from detecting the start mark and repeating the computation process, the computing node can also determine the mark time of the completion mark and the mark time of the start mark, and compare the mark time of the completion mark and the mark time of the start mark. If the mark time of the start mark is before the mark time of the completion mark, it means that both the start mark and the completion mark are the marks corresponding to the current target task, and then the start mark of the target task is canceled.

[0057] In this embodiment, by periodically detecting each task in the preset storage space, the terminal device is encouraged to freely detect the tasks that need to be processed. When a target task is detected, the calculation markers of each subtask in the target task are then detected, and the subtasks without calculation markers are processed, thereby improving the calculation speed of the target task. The target task is a task that includes a start marker. When each subtask in the target task meets a preset standard, it indicates that the target task has been processed and a completion marker is then added to the target task. This allows the terminal device to clearly identify the target task as completed when it detects each task in the preset storage space again. By storing tasks in the storage space, the terminal device is encouraged to freely detect the tasks that need to be processed in the storage space. Even if an unexpected situation occurs during the processing of a subtask by a certain terminal device, other terminal devices will process the unprocessed subtasks when the target task is detected, further improving the stability of the task processing process.

[0058] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0059] Corresponding to the distributed task processing method described above, Figure 3 The diagram shown is a structural schematic of a distributed task processing device according to an embodiment of this application. Figure 3 As shown, the aforementioned distributed task processing device may include:

[0060] The periodic detection module 301 is used to periodically detect each task in the preset storage space.

[0061] The calculation mark detection module 302 is used to detect the calculation marks of each subtask in the target task when the existence of the target task is detected, and to perform calculation processing on the subtasks that do not have calculation marks; the target task is a task that contains a start mark.

[0062] The completion marking module 303 is used to mark the target task as completed when it is detected that each subtask in the target task meets the preset criteria.

[0063] In one embodiment, the above-described calculation marker detection module 302 may include:

[0064] The first file detection unit is used to detect files in the subtask.

[0065] The first marker determination unit is used to determine that a calculation marker exists in a subtask if a start marker file and an end marker file exist in the subtask.

[0066] In one embodiment, the above-described calculation marker detection module 302 may further include:

[0067] The second file detection unit is used to detect files in the subtask.

[0068] The time acquisition unit is used to acquire the current time and the write time of the start marker file if a start marker file exists in the subtask but no end marker file exists.

[0069] The time difference calculation unit is used to calculate the time difference between the write time and the current time.

[0070] The second marker determination unit is used to determine that the subtask does not have a calculation marker if the time difference is greater than or equal to a preset calculation time threshold.

[0071] The third marker determination unit is used to determine that a calculation marker exists for a subtask if the time difference is less than a preset calculation time threshold.

[0072] In one embodiment, the above-described calculation marker detection module 302 may further include:

[0073] The start mark writing unit is used to write a preset start mark file to a subtask that does not have a calculation mark.

[0074] The file calculation unit is used to obtain data files from subtasks that do not have calculation markers and to perform calculations on the data files.

[0075] The end marker writing unit is used to write a preset end marker file to a subtask that does not have a calculation marker after the calculation is completed.

[0076] In one embodiment, the distributed task processing apparatus may further include:

[0077] The standard determination module is used to determine that each subtask in the target task meets the preset standard if a preset number of subtasks are continuously detected and all have calculated tags.

[0078] In one embodiment, the distributed task processing apparatus may further include:

[0079] The first-moment determination module is used to determine the marking time of the completion mark and the marking time of the start mark if the target task has a completion mark.

[0080] The first cancellation module is used to cancel the completion mark of the target task if the mark start time is after the mark completion time.

[0081] In one embodiment, the distributed task processing apparatus may further include:

[0082] The second time determination module is used to determine the marking time when marking is completed and the marking time when marking begins.

[0083] The second cancellation module is used to cancel the start marking of the target task if the marking start time is before the marking completion time.

[0084] In this embodiment, by periodically detecting each task in the preset storage space, the terminal device is encouraged to freely detect the tasks that need to be processed. When a target task is detected, the calculation markers of each subtask in the target task are then detected, and the subtasks without calculation markers are processed, thereby improving the calculation speed of the target task. The target task is a task that includes a start marker. When each subtask in the target task meets a preset standard, it indicates that the target task has been processed and a completion marker is then added to the target task. This allows the terminal device to clearly identify the target task as completed when it detects each task in the preset storage space again. By storing tasks in the storage space, the terminal device is encouraged to freely detect the tasks that need to be processed in the storage space. Even if an unexpected situation occurs during the processing of a subtask by a certain terminal device, other terminal devices will process the unprocessed subtasks when the target task is detected, further improving the stability of the task processing process.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing system embodiments and method embodiments, and will not be repeated here.

[0086] Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0087] like Figure 4 As shown, the terminal device 4 in this embodiment includes: at least one processor 400 ( Figure 4(Only one is shown in the image), a memory 401 connected to the processor 400, and a computer program 402 stored in the memory 401 and executable on at least one processor 400, such as a distributed task processing program. When the processor 400 executes the computer program 402, it implements the steps in the various distributed task processing method embodiments described above, for example... Figure 1 The steps S101 to S103 are shown. Alternatively, when the processor 400 executes the computer program 402, it implements the functions of each module in the above-described device embodiments, for example... Figure 3 The functions of modules 301 to 303 are shown.

[0088] For example, the computer program 402 described above can be divided into one or more modules. One or more of these modules are stored in the memory 401 and executed by the processor 400 to complete this application. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 402 in the terminal device 4. For example, the computer program 402 can be divided into a periodic detection module 301, a mark calculation detection module 302, and a mark completion module 303, with the specific functions of each module as follows:

[0089] The periodic detection module 301 is used to periodically detect each task in the preset storage space;

[0090] The calculation mark detection module 302 is used to detect the calculation marks of each subtask in the target task when the existence of the target task is detected, and to perform calculation processing on the subtasks that do not have calculation marks; the target task is a task that contains a start mark.

[0091] The completion marking module 303 is used to mark the target task as completed when it is detected that each subtask in the target task meets the preset criteria.

[0092] The aforementioned terminal device 4 may include, but is not limited to, a processor 400 and a memory 401. Those skilled in the art will understand that... Figure 4 This is merely an example of terminal device 4 and does not constitute a limitation on terminal device 4. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, buses, etc.

[0093] The processor 400 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0094] In some embodiments, the aforementioned memory 401 may be an internal storage unit of the terminal device 4, such as a hard disk or memory of the terminal device 4. In other embodiments, the aforementioned memory 401 may be an external storage device of the terminal device 4, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 4. Furthermore, the aforementioned memory 401 may include both internal storage units and external storage devices of the terminal device 4. The aforementioned memory 401 is used to store operating systems, applications, boot loaders, data, and other programs, such as the program code of the aforementioned computer programs. The aforementioned memory 401 may also be used to temporarily store data that has been output or will be output.

[0095] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0096] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0097] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0098] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0099] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0102] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A distributed task processing method, characterized in that, include: The computing nodes periodically detect each task in the preset storage space. The preset storage space is a data storage medium that can be accessed and read / written by at least two computing nodes participating in the task computation at the same time. Each task in the preset storage space is stored in the form of a folder, and each subtask is stored in the form of a subfolder under the folder corresponding to the task. When a target task is detected, the calculation markers of each subtask in the target task are detected, and the subtasks without calculation markers are calculated; the target task is a task that includes a start marker. When each subtask in the target task is detected to meet the preset criteria, the target task is marked as completed. The calculation of the tags for each subtask in the target task includes: Detect the files in the subtask; If a start marker file exists in the subtask but no end marker file exists, then the current time and the write time of the start marker file are obtained. The start marker file is a zero-byte file in the subfolder used to identify the start of the subtask calculation. Calculate the time difference between the write time and the current time; If the time difference is greater than or equal to a preset calculation time threshold, then it is determined that the subtask does not have a calculation marker. If the time difference is less than a preset calculation time threshold, then it is determined that the subtask has a calculation flag.

2. The distributed task processing method as described in claim 1, characterized in that, The calculation of the tags for each subtask in the target task includes: Detect the files in the subtask; If a start marker file and an end marker file exist in the subtask, then it is determined that the subtask has a calculation marker.

3. The distributed task processing method as described in claim 1, characterized in that, The computation processing for subtasks without computational markers includes: Write the preset start marker file into the subtask that does not have a calculation marker; Obtain the data file from the subtask where no calculation marker exists, and perform calculations on the data file; Once the calculation is complete, a preset end marker file is written to the subtask that does not have a calculation marker.

4. The distributed task processing method as described in claim 1, characterized in that, Before detecting that each subtask in the target task meets the preset criteria, the process includes: If a preset number of subtasks are continuously detected and all have calculation tags, then each subtask in the target task is determined to meet the preset standard.

5. The distributed task processing method as described in claim 1, characterized in that, After detecting the existence of a target task, it also includes: If the target task has a completion marker, then determine the marker time of the completion marker and the marker time of the start marker; If the start marker is set after the completion marker, the completion marker for the target task is cancelled.

6. The distributed task processing method as described in claim 1, characterized in that, After marking the target task as completed, the process also includes: Determine the marking time of the completion marking and the marking time of the start marking; If the start marker is set before the completion marker, then the start marker for the target task is cancelled.

7. A distributed task processing device, characterized in that, include: The periodic detection module is used for computing nodes to periodically detect each task in the preset storage space. The preset storage space is a data storage medium that can be accessed and read / written by at least two computing nodes participating in task computation at the same time. Each task in the preset storage space is stored in the form of a folder, and each subtask is stored in the form of a subfolder under the folder corresponding to the task. The calculation mark detection module is used to detect the calculation marks of each subtask in the target task when the existence of the target task is detected, and to perform calculation processing on the subtasks that do not have calculation marks; the target task is a task that includes a start mark. The completion marking module is used to mark the target task as completed when it is detected that each subtask in the target task meets the preset criteria. The computational marker detection module includes: The second file detection unit is used to detect files in the subtask; The time acquisition unit is used to acquire the current time and the write time of the start marker file if a start marker file exists in the subtask and no end marker file exists. The start marker file is a zero-byte file in the subfolder used to identify the start of the subtask calculation. The time difference calculation unit is used to calculate the time difference between the write time and the current time; The second marker determination unit is used to determine that the subtask does not have a calculation marker if the time difference is greater than or equal to a preset calculation time threshold. The third marker determination unit is used to determine that a calculation marker exists for a subtask if the time difference is less than a preset calculation time threshold.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a distributed task processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of a distributed task processing method as described in any one of claims 1 to 6.

Images