A task processing method, device, electronic device and storage medium
Through standardized inspection, testing verification and resource adjustment, the task development and execution efficiency and quality problems in the task scheduling system are solved, resource optimization and result guarantee are achieved, and the accuracy and efficiency of task processing are improved.
Patent Information
- Application Number
- CN202310890814.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-19
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-07-19
AI Technical Summary
The existing task scheduling system has shortcomings in task development, execution efficiency optimization and result quality, resulting in uncontrollable resource consumption, high cost, poor data quality, and affecting customer use.
By loading the initial task, performing standardized detection and testing verification, we ensure that the tasks comply with unified specifications; splitting tasks into subtasks at runtime, executing them in sequence or in parallel according to dependencies, and adjusting resource allocation in real time; finally data quality is performed on the task results. Detection, provide corresponding prompts and blocking mechanisms.
It improves the efficiency of task development and verification, reduces the difficulty of collaborative development and maintenance, improves the accuracy and effectiveness of task execution, and ensures the quality of task results.
Smart Images

Figure CN116820727B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a task processing method, device, electronic device and storage medium. Background Art
[0002] The existing big data task scheduling system can already implement scheduling in the time dimension and dependency dimension, and uses a user interface to develop and submit tasks. However, the existing task scheduling system does not provide good support in terms of task development, task execution efficiency optimization, and task result quality. As a result, the existing task scheduling system has problems such as uncontrollable resource consumption, high costs, unguaranteed execution results, poor data quality, and customer impact when processing tasks. Summary of the invention
[0003] The present disclosure provides a task processing method, device, electronic device and storage medium to at least solve the above technical problems existing in the prior art.
[0004] According to a first aspect of the present disclosure, a task processing method is provided, the method comprising: loading an initial task; performing a specification check on the initial task; performing a test verification on the initial task that has passed the specification check, and determining that the initial task that has passed the test verification is a task to be processed; running the task to be processed; when it is detected that the task to be processed has finished running, obtaining the running result of the task to be processed; performing a data quality check on the running result, and providing a prompt for the running result based on the data quality check result, the data quality check comprising: whether the file size, the scale of the data volume and the content of the output of the task to be processed are accurate.
[0005] In one possible implementation, the standard check on the initial task includes: performing a standard check on the initial task based on a preset task specification, the standard check including a grammar check and a format check; if both the grammar check and the format check of the initial task are qualified, it is determined that the standard check of the initial task is qualified; if at least one of the grammar check and the format check of the initial task fails, it is determined that the standard check of the initial task is unqualified.
[0006] In one possible implementation, the method further includes: generating a prompt message for the initial task that fails the specification test, the prompt message including the reason for the failure and the location of the failure, so as to modify the initial task according to the reason and location, and perform a specification test on the modified initial task until the initial task passes the specification test.
[0007] In one possible implementation mode, the initial task that has passed the test verification is determined as a task to be processed, including: if it is detected that the current time meets the start processing time of the initial task, and all the predecessor tasks that have a dependency relationship with the initial task have been completed, the initial task is set to a ready state, and the initial task is determined to be a task to be processed.
[0008] In one possible implementation, the running of the pending task includes: parsing the pending task, splitting the pending task into multiple subtasks, and determining the dependencies between the multiple subtasks; running the subtasks based on the dependencies between the multiple subtasks, with subtasks with dependencies running in the order of the dependencies, and subtasks without dependencies running in parallel.
[0009] In one possible implementation, the method also includes: adjusting the resources allocated to the target task in real time during the running of the task to be processed, the target task being the task to be processed or the subtask, including: determining the amount of resources required to run the target task and the corresponding amount of allocated resources; if the required amount of resources is greater than the amount of allocated resources, allocating a specific amount of new resources to the target task so that the updated amount of allocated resources meets a first resource threshold; if the required amount of resources is less than the amount of allocated resources, recovering a specific amount of resources from the allocated resources of the target task so that the updated amount of allocated resources meets a second resource threshold.
[0010] In one possible implementation, the prompt of the running result based on the data quality detection result includes: if the data quality detection is qualified, it is determined that the pending task is completed, and a prompt of normal task completion is issued; if the data quality detection is unqualified, and it is detected that the post-task has a dependency relationship with the running result of the pending task, the running of the post-task is blocked, and a first prompt of task failure is issued; if the data quality detection is unqualified, and it is detected that the post-task has no dependency relationship with the running result of the pending task, the running of the post-task is not blocked, and a second prompt of task failure is issued.
[0011] According to a second aspect of the present disclosure, a task processing device is provided, the device comprising: a loading module for loading an initial task; a first detection module for performing a specification detection on the initial task; a verification module for testing and verifying the initial task that has passed the specification detection, and determining that the initial task that has passed the test verification is a task to be processed; a running module for running the task to be processed; an acquisition module for acquiring the running result of the task to be processed when detecting that the running of the task to be processed is finished; a second detection module for performing a data quality detection on the running result, and providing a prompt for the running result based on the data quality detection result, the data quality detection comprising: whether the file size, the scale of the data volume and the content of the output of the task to be processed are accurate.
[0012] In one possible implementation, the first detection module is specifically used to perform a specification check on the initial task based on a preset task specification, and the specification check includes a grammar check and a format check; if both the grammar check and the format check of the initial task are qualified, it is determined that the specification check of the initial task is qualified; if at least one of the grammar check and the format check of the initial task fails, it is determined that the specification check of the initial task is unqualified.
[0013] In one possible implementation, the device also includes: a prompt module, which is used to generate a prompt message for the initial task that fails the standard test, and the prompt message includes the reason for the failure and the location of the failure, so as to modify the initial task according to the reason and location, and perform a standard test on the modified initial task until the standard test of the initial task passes.
[0014] In one possible implementation, the verification module is also used to set the initial task to a ready state and determine the initial task as a task to be processed if it is detected that the current time meets the start processing time of the initial task and all predecessor tasks that have a dependency relationship with the initial task have been completed.
[0015] In one possible implementation, the running module includes: a parsing submodule, used to parse the task to be processed, split the task to be processed into multiple subtasks, and determine the dependencies between the multiple subtasks; a running submodule, used to run the subtasks based on the dependencies between the multiple subtasks, subtasks with dependencies are run in the order of the dependencies, and subtasks without dependencies are run in parallel.
[0016] In one possible implementation mode, the device further includes: an adjustment module, which is used to adjust the resources allocated to the target task in real time during the execution of the task to be processed, wherein the target task is the task to be processed or the subtask; the adjustment module includes: a determination submodule, which is used to determine the amount of resources required to run the target task and the corresponding amount of allocated resources; an adjustment submodule, which is used to allocate a specific amount of new resources to the target task if the required amount of resources is greater than the allocated amount of resources, so that the updated amount of allocated resources meets a first resource threshold; if the required amount of resources is less than the allocated amount of resources, reclaim a specific amount of resources from the allocated resources of the target task, so that the updated amount of allocated resources meets a second resource threshold.
[0017] In one possible implementation, the second detection module is further used to determine that the pending task is completed if the data quality test is qualified, and issue a prompt that the task is completed normally; if the data quality test is unqualified, and it is detected that the post-task has a dependency relationship with the running result of the pending task, then the running of the post-task is blocked, and a first prompt of task failure is issued; if the data quality test is unqualified, and it is detected that the post-task has no dependency relationship with the running result of the pending task, then the running of the post-task is not blocked, and a second prompt of task failure is issued.
[0018] According to a third aspect of the present disclosure, there is provided an electronic device, including:
[0019] at least one processor; and
[0020] a memory communicatively connected to the at least one processor; wherein,
[0021] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.
[0022] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.
[0023] The present invention discloses a task processing method, device, electronic device and storage medium, which first loads the initial task; performs a standard detection on the initial task, and tests and verifies the initial task that has passed the standard detection, and determines the initial task that has passed the test verification as a task to be processed; runs the task to be processed; and when it is detected that the task to be processed has finished running, obtains the running result of the task to be processed, and performs a data quality detection on the running result. When this method is applied to process tasks, standard detection of tasks is added in the development stage of tasks, so that tasks submitted to the production environment meet unified standard requirements, reducing the difficulty of collaborative development and maintenance, and testing and verifying the initial task that has passed the standard detection, so as to avoid affecting the normal operation of the production environment when problems occur when the initial task is directly run, thereby improving the efficiency of task development verification, and finally verifying the data quality after the task is completed, thereby improving the accuracy and effectiveness of verification.
[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:
[0026] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.
[0027] Figure 1 The following is a schematic diagram showing the implementation process of a task processing method according to an embodiment of the present disclosure. Figure 1 ;
[0028] Figure 2 The following is a schematic diagram showing the implementation process of a task processing method according to an embodiment of the present disclosure. Figure 2 ;
[0029] Figure 3 A module schematic diagram of a task processing device according to an embodiment of the present disclosure is shown;
[0030] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0031] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0032] Figure 1 The following is a schematic diagram showing the implementation process of a task processing method according to an embodiment of the present disclosure. Figure 1 ,include:
[0033] Step 101, loading the initial task.
[0034] Step 102, perform standard inspection on the initial task.
[0035] During the task development phase, after being loaded into the initial task, the initial task is subjected to specification testing to ensure that the initial task meets the unified specification requirements before operation.
[0036] Step 103 , test and verify the initial tasks that have passed the specification inspection, and determine the initial tasks that have passed the test and verification as tasks to be processed.
[0037] After the initial tasks are tested for specifications, they are submitted to the designated task verification cluster for verification. The environment formed by the designated task verification cluster is consistent with the production environment. Testing and verifying the initial tasks that have passed the specification test can avoid problems when the tasks are submitted directly to the production environment after development, which may affect the normal operation of the production environment.
[0038] During the test verification process, if the initial task runs successfully on the specified task verification cluster, it is determined that the initial task can run normally, otherwise it is determined that the initial task cannot run normally. When it is determined that the initial task can run normally, the test verification result data is queried, and the initial task running result is judged according to the result data to determine whether the task logic of the initial task is accurate. The initial task that runs successfully and has correct running results is determined as a pending task, which is a task that can be run in the production environment.
[0039] Through the test verification process, we can further eliminate the syntax errors, logic errors, and data inaccuracies in the initial tasks that have passed the specification inspection, and avoid abnormalities when running the initial tasks directly in the production environment.
[0040] If the initial task that passes the specification check fails the test verification, a prompt message is generated based on the reason for the failure and the location of the failure. After making corresponding modifications based on the prompt message, the modified initial task is re-checked for specifications, and then tested and verified again after the specification check until the initial task can be successfully run on the specified task verification cluster and the test verification result data is correct.
[0041] Step 104: Run the pending tasks.
[0042] When the specification inspection and test verification of the initial task are qualified, the initial task is determined as a pending task, and the pending task is submitted to the formal environment for operation. In the actual operation process, the dependencies between the pending tasks in the preset time period can be determined, and the corresponding directed acyclic graph can be generated to run the pending tasks according to the dependencies between the pending tasks. The preset time period can be set according to the actual situation, for example, it can be one day or one week.
[0043] Step 105, when it is detected that the execution of the pending task is completed, the execution result of the pending task is obtained.
[0044] When the pending task is finished running, a prompt message indicating the task is finished running is sent. When the prompt message is listened, the running result of the pending task is obtained. When the running result of the pending task is successful, the running result can be tested for data quality. When the running result of the pending task is failed, the pending task is adjusted according to the failure reason prompted in the failure pool and run again.
[0045] Furthermore, a failure threshold may be set. If the number of times that a pending task fails to run reaches the failure threshold, the pending task will no longer run.
[0046] Step 106, perform data quality detection on the operation result, and provide a prompt for the operation result based on the data quality detection result. The data quality detection includes: the file size of the output of the task to be processed, the scale of the output data volume and whether the output content is accurate.
[0047] The data quality test is performed on the running results of the pending tasks according to the preset test standards. The data quality test includes the file size, data volume scale, data volume range and whether the output content is correct. When the data quality test results meet the preset test standards, the task is confirmed to be completed normally and a prompt for normal task completion is generated; when the data quality test results do not meet the preset test standards, the task is confirmed to be not completed normally and a prompt for abnormal task completion is generated.
[0048] The embodiment of the present scheme provides a task processing method, which performs specification detection and test verification on the initial task after loading it, determines the initial task that has passed both the specification detection and test verification as a task to be processed, then runs the task to be processed, obtains the running result of the task to be processed, performs data quality detection on the running result, and prompts the running result based on the data quality detection result. When applying this method to process tasks, a specification check of the task is added during the development phase of the task, so that the tasks submitted to the operating environment meet the unified specification requirements, reducing the difficulty of collaborative development and maintenance, and verifying the data quality after the task is completed, improving the accuracy and effectiveness of the verification.
[0049] In one possible implementation, the initial task is subjected to a standard test, including:
[0050] The initial task is subjected to a specification check based on the preset task specification, and the specification check includes a grammar check and a format check; if both the grammar check and the format check of the initial task are qualified, it is determined that the specification check of the initial task is qualified; if at least one of the grammar check and the format check of the initial task is unqualified, it is determined that the specification check of the initial task is unqualified.
[0051] There are strict standards for task development. During the task development phase, adding task checks can ensure that the initial task meets the specification requirements. Syntax detection includes statement syntax, field naming, etc.; format detection includes writing format, statement specifications, etc. When the initial task meets the corresponding specification requirements in both syntax detection and format detection, the initial task specification detection is considered qualified. If any of the syntax detection and format detection does not meet the corresponding specification requirements, the initial task is considered unqualified.
[0052] In addition, the specification detection can be expanded and adjusted in many aspects according to the actual task type being run. Different specification detections are provided for different task types to detect whether the task meets the specifications.
[0053] In one embodiment, the method further comprises:
[0054] For the initial task that fails the specification test, a prompt message is generated, which includes the reason for the failure and the location of the failure, so that the initial task can be modified according to the reason and location, and the specification test is performed on the modified initial task until the specification test of the initial task passes.
[0055] When the specification check of the initial task fails, the reason for the failure and the location of the failure are determined, and a prompt message is generated to prompt the developer that the specification check of the initial task fails, as well as the specific location and reason for the failure. For example, the prompt message indicates that the format check of the initial task fails, and which part of the statement is not standardized, so that the developer can make corresponding modifications to the initial task according to the prompt message. After that, the modified initial task is subjected to the specification check. If the specification check passes, the subsequent steps are carried out. If the specification check still fails, the modification is continued until the specification check of the initial task passes.
[0056] In one possible implementation, determining an initial task that has passed the test verification as a pending task includes: if it is detected that the current time meets the start processing time of the initial task, and all the predecessor tasks that have a dependency relationship with the initial task have been completed, setting the initial task to a ready state, and determining the initial task as a pending task.
[0057] When a task is running, the corresponding running time will be set. For example, if a task is set to start running at 10 o'clock, it will start running at 10 o'clock. After the specification detection and test verification of the initial task are qualified, if the current time meets the preset start processing time of the initial task, it is determined whether the initial task has a predecessor task. If not, the initial task is set to the ready state and determined as a pending task.
[0058] If the initial task has a predecessor task, determine whether all the predecessor tasks with a dependency relationship with the initial task have been completed. If they have been completed, set the initial task to the ready state and determine that the task is a pending task. If there is a predecessor task that has not been completed, set the initial task to the waiting state until all the predecessor tasks with a dependency relationship with the initial task are completed, and then set the initial task to the ready state.
[0059] If the current time does not meet the preset start processing time of the initial task, the initial task is also set to a waiting state until the current time meets the preset start processing time of the initial task and all the predecessor tasks that have a dependency relationship with the initial task are completed, and the initial task is set to a ready state.
[0060] When it is detected that there are spare resources in the production environment, the tasks in the ready state can be submitted to the production environment for execution.
[0061] In one possible implementation, Figure 2 As shown, run the pending tasks, including:
[0062] Step 201, parsing the task to be processed, splitting the task to be processed into multiple subtasks, and determining the dependency relationship between the multiple subtasks;
[0063] Step 202: based on the dependency relationship between the multiple subtasks, the subtasks are run, the subtasks with dependencies are run in the order of the dependencies, and the subtasks without dependencies are run in parallel.
[0064] Before the task to be processed is officially run, the task to be processed is analyzed and split into multiple subtasks according to the analysis results, and the dependencies between the multiple subtasks are determined. According to the dependencies between the multiple subtasks, a corresponding directed acyclic graph is constructed.
[0065] When running the pending task, it is run according to the dependency relationship between the subtasks. The subtasks with dependencies are run in the order of the dependencies, while the subtasks without dependencies can be run in parallel, thereby speeding up the task execution and avoiding unnecessary waiting during the task running process.
[0066] When splitting a pending task into multiple subtasks, ensure that the dependent parameters between the subtasks remain unchanged to avoid abnormal task operation.
[0067] In addition, if after analyzing the pending task, the pending task cannot be split into subtasks, the pending task can be directly run.
[0068] In one possible implementation, the method further includes: adjusting resources allocated to a target task in real time during the execution of the task to be processed, the target task being the task to be processed or a subtask, including:
[0069] Determine the amount of resources required to run the target task and the corresponding amount of allocated resources;
[0070] If the required amount of resources is greater than the amount of allocated resources, a specific amount of new resources is allocated to the target task so that the updated amount of allocated resources meets the first resource threshold;
[0071] If the required amount of resources is less than the allocated amount of resources, a specific amount of resources is recovered from the allocated amount of resources of the target task so that the updated allocated amount of resources meets the second resource threshold.
[0072] In the process of running pending tasks, the resources allocated to the target task are adjusted in real time. If the pending task is not split into subtasks, the target task is the pending task. If the pending task is split into subtasks, each subtask is the target task.
[0073] First, determine the amount of resources required to run the target task, and compare the required amount of resources with the allocated amount of resources for the target task. If the required amount of resources for the target task is greater than the allocated amount of resources, the target task will fail due to insufficient resources. At this time, a specific amount of new resources is allocated to the target task so that the updated allocated amount of resources for the target task meets the first resource threshold, which is slightly greater than the required amount of resources for the target task to ensure the smooth operation of the target task.
[0074] If the required amount of resources for the target task is less than the allocated amount of resources, it will cause resource occupation. Therefore, a specific amount of resources is recovered from the allocated amount of resources for the target task so that the updated allocated amount of resources meets the second resource threshold, which is slightly larger than the required amount of resources for the target task, to ensure the smooth operation of the target task without occupying resources and causing waste of resources.
[0075] Furthermore, if the required amount of resources for the target task is less than the allocated amount of resources, the allocated amount of resources is compared with a second resource threshold; if the allocated amount of resources is less than the second resource threshold, a specific amount of new resources is allocated to the target task so that the allocated amount of resources for the target task meets the second resource threshold; if the allocated amount of resources is greater than the second resource threshold, a specific amount of resources is recovered from the allocated resources of the target task so that the allocated amount of resources for the target task meets the second resource threshold.
[0076] If it is detected during the running of the target task that the target task fails to run due to insufficient resources, resources can also be reallocated to the target task so that the target task can run successfully.
[0077] By adjusting resources in real time, it is possible to avoid unnecessary waste of resources by setting larger memory or CPU resources in advance without knowing how much resources the task will require.
[0078] In addition, after the task processing is completed, the required amount of resources for the target task running process or the updated amount of allocated resources can be counted, so that when the same task is run next time, the statistical data can be used as a reference for allocating resources.
[0079] In one possible implementation, prompting of the operation result based on the data quality detection result includes:
[0080] If the data quality test is qualified, the pending task is determined to be completed, and a prompt for normal task completion is issued;
[0081] If the data quality test fails, and it is detected that the post-task has a dependency relationship with the running results of the pending task, the running of the post-task is blocked and the first prompt of task failure is issued;
[0082] If the data quality check fails, and it is detected that there is no dependency between the running results of the post-task and the task to be processed, the running of the post-task is insufficient, and a second prompt of task failure is issued.
[0083] Perform data quality inspection on the running results of the task. When all data quality inspection items are qualified, the data quality inspection is determined to be qualified, the pending task is successfully completed, and a prompt for normal task completion is issued. When there is an unqualified situation in the data quality inspection items, the data quality inspection is determined to be unqualified, the pending task is not successfully completed, and a prompt for abnormal task completion is issued.
[0084] And when the data quality test fails, if it is detected that the post-task of the pending task has a dependency relationship with the running result of the pending task, the post-task will be stopped to avoid the running result affecting the running of the post-task; the first prompt of task failure is issued, and the first prompt of task failure is a warning prompt, prompting the developer to correct the pending task as soon as possible. If it is detected that the post-task of the pending task has no dependency relationship with the running result of the pending task, that is, the running result has no effect on the running of the post-task, the post-task can continue to run, but the second prompt of task failure is issued, and the second prompt of task failure is a reminder prompt, prompting the developer to pay attention to the pending task.
[0085] By checking the data quality of the running results of the tasks to be processed, the accuracy and effectiveness of the verification are improved, the blocking or non-blocking of the post-tasks can be achieved, and a prompt effect can be achieved in the event of abnormal data quality detection so that developers can intervene and handle the problem as soon as possible.
[0086] Figure 3 A module schematic diagram of a task processing device according to an embodiment of the present disclosure is shown.
[0087] See also Figure 3 According to the second aspect of the embodiment of the present disclosure, a task processing device is provided, the device comprising: a loading module 301, used to load an initial task; a first detection module 302, used to perform a specification detection on the initial task; a verification module 303, used to test and verify the initial task that has passed the specification detection, and determine that the initial task that has passed the test verification is a task to be processed; a running module 304, used to run the task to be processed; an acquisition module 305, used to acquire the running result of the task to be processed when detecting that the running of the task to be processed is finished; a second detection module 306, used to perform a data quality detection on the running result, and prompt the running result based on the data quality detection result, the data quality detection includes: whether the file size, the scale of the output data and the output content of the task to be processed are accurate.
[0088] In one possible implementation, the first detection module 302 is specifically used to perform a specification check on the initial task based on a preset task specification, and the specification check includes a grammar check and a format check; if both the grammar check and the format check of the initial task are qualified, it is determined that the specification check of the initial task is qualified; if at least one of the grammar check and the format check of the initial task fails, it is determined that the specification check of the initial task is unqualified.
[0089] In one possible implementation mode, the device also includes: a prompt module 307, which is used to generate a prompt message for an initial task that fails the standard test, the prompt message including the reason for the failure and the location of the failure, so as to modify the initial task according to the reason and the location, and perform a standard test on the modified initial task until the initial task passes the standard test.
[0090] In one embodiment, the verification module 303 is also used to set the initial task to a ready state and determine the initial task as a task to be processed if it is detected that the current time meets the start processing time of the initial task and all the predecessor tasks that have a dependency relationship with the initial task have been completed.
[0091] In one embodiment, the running module 304 includes: a parsing sub-module 3041, which is used to parse the task to be processed, split the task to be processed into multiple sub-tasks, and determine the dependency relationship between the multiple sub-tasks; an running sub-module 3042, which is used to run the sub-tasks based on the dependency relationship between the multiple sub-tasks, and the sub-tasks with dependencies are run in the order of the dependencies, and the sub-tasks without dependencies are run in parallel.
[0092] In one possible implementation mode, the device also includes: an adjustment module 308, which is used to adjust the resources allocated to the target task in real time during the execution of the task to be processed, where the target task is a task to be processed or a subtask, and the adjustment module 308 includes: a determination submodule 3081, which is used to determine the amount of resources required to run the target task and the corresponding amount of allocated resources; an adjustment submodule 3082, which is used to allocate a specific amount of new resources to the target task if the required amount of resources is greater than the allocated amount of resources, so that the updated amount of allocated resources meets the first resource threshold; if the required amount of resources is less than the allocated amount of resources, a specific amount of resources is recovered from the allocated resources of the target task, so that the updated amount of allocated resources meets the second resource threshold.
[0093] In one possible implementation, the second detection module 306 is also used to determine that the pending task is completed if the data quality test is qualified, and issue a prompt that the task is completed normally; if the data quality test is unqualified, and it is detected that the running results of the post-task and the pending task are dependent on each other, the running of the post-task is blocked, and a first prompt of task failure is issued; if the data quality test is unqualified, and it is detected that the running results of the post-task and the pending task are not dependent on each other, the running of the post-task is not blocked, and a second prompt of task failure is issued.
[0094] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.
[0095] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0096] like Figure 4 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0097] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0098] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as a task processing method. For example, in some embodiments, a task processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of a task processing method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform a task processing method in any other appropriate manner (e.g., by means of firmware).
[0099] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0100] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0101] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0103] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0104] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0105] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0106] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0107] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A task processing method, characterized in that: The method comprises: Load the initial task; Performing a normative test on the initial task; Performing test verification on the initial tasks that have passed the specification inspection, and determining the initial tasks that have passed the test verification as tasks to be processed; Running the pending tasks; When it is detected that the task to be processed has finished running, obtaining the running result of the task to be processed; Performing data quality detection on the operation result, and providing a prompt for the operation result based on the data quality detection result, wherein the data quality detection includes: whether the file size, the amount of data output and the content of the output of the task to be processed are accurate; The testing and verifying of the initial tasks that have passed the specification detection, and determining the initial tasks that have passed the test verification as tasks to be processed, includes: submitting the initial tasks that have passed the specification detection to a designated task verification cluster for verification, and determining the initial tasks that have successfully run on the designated task verification cluster and have correct running results as tasks to be processed; During the running of the task to be processed, resources allocated to the target task are adjusted in real time, the target task being the task to be processed or a subtask, including: Determine the amount of resources required to run the target task and the corresponding amount of allocated resources; If the required amount of resources is greater than the allocated amount of resources, allocating a specific amount of new resources to the target task so that the updated amount of allocated resources meets a first resource threshold; If the required amount of resources is less than the allocated amount of resources, compare the allocated amount of resources with a second resource threshold; if the allocated amount of resources is less than the second resource threshold, allocate a specific amount of new resources to the target task so that the updated allocated amount of resources meets the second resource threshold; if the allocated amount of resources is greater than the second resource threshold, reclaim a specific amount of resources from the allocated resources of the target task so that the updated allocated amount of resources meets the second resource threshold.
2. The method according to claim 1, characterized in that The performing of standard detection on the initial task includes: Performing a specification check on the initial task based on a preset task specification, wherein the specification check includes a grammar check and a format check; If both the syntax check and the format check of the initial task are qualified, it is determined that the specification check of the initial task is qualified; If at least one of the grammar check and the format check of the initial task fails, it is determined that the specification check of the initial task fails.
3. The method according to claim 1 or 2, characterized in that: The method further includes: For the initial task that fails the standard test, a prompt message is generated, which includes the reason for the failure and the location of the failure, so that the initial task can be modified according to the reason and location, and the standard test is performed on the modified initial task until the standard test of the initial task passes.
4. The method according to claim 1, characterized in that: The initial task of determining whether the test verification is qualified is a task to be processed, including: If it is detected that the current time meets the start processing time of the initial task, and all the predecessor tasks having a dependency relationship with the initial task have been completed, the initial task is set to a ready state and the initial task is determined to be a task to be processed.
5. The method according to claim 1, characterized in that The running of the pending task includes: Parsing the pending task, splitting the pending task into multiple subtasks, and determining dependencies between the multiple subtasks; Based on the dependency relationship between the multiple subtasks, the subtasks are run, the subtasks with dependency relationships are run in the order of the dependency relationships, and the subtasks without dependency relationships are run in parallel.
6. The method according to claim 1, characterized in that The prompting of the operation result based on the data quality detection result includes: If the data quality test is qualified, it is determined that the pending task is completed, and a prompt for normal completion of the task is issued; If the data quality detection fails, and it is detected that the post-task has a dependency relationship with the running result of the task to be processed, the running of the post-task is blocked, and a first prompt of task failure is issued; If the data quality detection fails, and it is detected that the post-task has no dependency relationship with the running result of the task to be processed, the running of the post-task is not blocked, and a second prompt of task failure is issued.
7. A task processing device, characterized in that: The device comprises: Loading module, used to load the initial task; A first detection module, used for performing a standard detection on the initial task; A verification module, used to test and verify the initial tasks that have passed the specification inspection, and determine the initial tasks that have passed the test verification as tasks to be processed; A running module, used for running the tasks to be processed; An acquisition module is used to acquire the running result of the task to be processed when it is detected that the task to be processed has finished running; A second detection module is used to perform data quality detection on the operation result and provide a prompt for the operation result based on the data quality detection result, wherein the data quality detection includes: whether the file size, the amount of data output and the content of the output of the task to be processed are accurate; The verification module is specifically used to submit the initial tasks that have passed the specification detection to the designated task verification cluster for verification, and determine the initial tasks that have been successfully run on the designated task verification cluster and have correct running results as pending tasks; An adjustment module, the adjustment module is used to adjust the resources allocated to the target task in real time during the running of the pending task, the target task being the pending task or subtask; the adjustment module includes: a determination submodule, used to determine the amount of resources required for running the target task and the corresponding amount of allocated resources; an adjustment submodule, if the required amount of resources is greater than the allocated amount of resources, allocating a specific amount of new resources to the target task so that the updated amount of allocated resources meets a first resource threshold; if the required amount of resources is less than the allocated amount of resources, comparing the allocated amount of resources with a second resource threshold, if the allocated amount of resources is less than the second resource threshold, allocating a specific amount of new resources to the target task so that the updated amount of allocated resources meets the second resource threshold; if the allocated amount of resources is greater than the second resource threshold, reclaiming a specific amount of resources from the allocated resources of the target task so that the updated amount of allocated resources meets the second resource threshold.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
