Task execution method and system, storage medium and computer equipment
Create multiple Pods by parsing the description information of the custom task and injecting sharded environment variables into each Pod to perform the corresponding sharded task, solving the problem that the Kubernetes Job native controller cannot perform heavy tasks in sharded , improve task execution efficiency.
Patent Information
- Application Number
- CN202411020897.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-08
AI Technical Summary
The Kubernetes Job native controller cannot perform sharding execution of heavy tasks, resulting in multiple Pods working at the same time but unable to accelerate task execution, and the repeated execution of tasks created by multiple same replica Pods cannot improve efficiency.
By listening to the shard execution type custom tasks sent by the user, parsing the shard environment variable array and container group Pod description information, creating a corresponding number of Pods based on the description information, and injecting shard environment variables into each Pod, so that Each Pod executes matching sharding tasks to realize task sharding processing.
The task sharding processing is implemented, which improves the task execution speed and processing efficiency, and avoids the inefficiency problem caused by slow work of a single Pod or repeated execution of multiple same Pods.
Smart Images

Figure CN118916117A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cloud platform task execution, and in particular to a task execution method, system, storage medium and computer equipment. Background Art
[0002] In the cloud platform, the Kubernetes Job native controller is responsible for batch processing of single tasks and only performs one-time tasks. It ensures that one or more Pods created by executing tasks are successfully completed, and each Pod and the corresponding task Job are the same. Figure 4 Pod is a logical abstract concept, the smallest unit created and managed by Kubernetes. A Pod consists of one or more containers. A Pod can be understood as an application instance. The container of the Pod that provides services is always deployed on a Node. The containers in the Pod share network and storage resources.
[0003] In the process of implementing the present invention, the applicant discovered that there are at least the following problems in the prior art:
[0004] The Kubernetes Job native controller cannot perform sharding on a task. Some heavy tasks may require multiple Pods to execute the tasks quickly. Multiple Pods work at the same time because one Pod works slowly. However, the multiple identical replica Pods created by the native Job control to execute the tasks are repeated tasks and cannot speed up the execution of the tasks. Summary of the invention
[0005] The embodiments of the present invention provide a task execution method, system, storage medium and computer device, which can solve the technical problem in the prior art that multiple Pods work simultaneously, because one Pod works slowly, the multiple identical copy Pods created by the native Job control to execute the task all execute the task repeatedly, and the task execution cannot be accelerated.
[0006] To achieve the above objectives, in a first aspect, an embodiment of the present invention provides a method for executing a task, comprising:
[0007] Listening to a custom task of a slice execution type sent by a user, wherein the custom task is divided into a plurality of slice tasks and the custom task has description information;
[0008] In the case of monitoring a custom task of a shard execution type, parse the shard environment variable array and the container group Pod description information from the description information of the custom task, wherein each shard task in the custom task has a corresponding shard task identifier, the shard environment variable array is composed of each shard task identifier, and the Pod description information includes the maximum number of Pods required to run the custom task;
[0009] Create a corresponding number of Pods according to the Pod description information, and match a unique shard task identifier for each created Pod according to the shard environment variable array;
[0010] Run all created Pods, and execute the shard task corresponding to the matched shard task identifier through each Pod.
[0011] In a second aspect, an embodiment of the present invention provides a task execution system, wherein the task execution system comprises:
[0012] A coordinator, configured to monitor a custom task of a slice execution type sent by a user, wherein the custom task is divided into a plurality of slice tasks and has description information;
[0013] In the case of monitoring a custom task of a shard execution type, parse the shard environment variable array and the container group Pod description information from the description information of the custom task, wherein each shard task in the custom task has a corresponding shard task identifier, the shard environment variable array is composed of each shard task identifier, and the Pod description information includes the maximum number of Pods required to run the custom task;
[0014] Create a corresponding number of Pods according to the Pod description information, and match a unique shard task identifier for each created Pod according to the shard environment variable array;
[0015] The execution unit is used to run all created Pods and execute the shard task corresponding to the matched shard task identifier through each Pod.
[0016] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a computer device, the computer device performs the execution method of the aforementioned task.
[0017] In a fourth aspect, an embodiment of the present invention provides a computer device, including:
[0018] A processor; and a memory arranged to store computer executable instructions, which when executed cause the processor to perform the aforementioned task execution method.
[0019] The above technical solution has the following beneficial effects: without introducing a third-party system, the task to be executed is directly set as a custom task, and the custom task is divided into multiple shard tasks, so that the function of task sharding processing is automatically realized. A corresponding number of Pods are created according to the Pod description information, and the sharded environment variables are respectively injected into different Pod environment variables, thereby creating a corresponding number of Pods, and each Pod executes the shard task corresponding to the matched shard task identifier. By having each Pod execute the shard task corresponding to the matched shard task identifier, the function of sharding the task is realized, which speeds up the task processing speed and improves the processing efficiency. Multiple Pods work at the same time, avoiding the technical problem that one Pod works slowly, or that there are multiple identical Pods but they are repeatedly executed and the task execution cannot be accelerated. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 is a flowchart of a method for executing a task according to an embodiment of the present invention;
[0022] Figure 2 is a structural diagram of a task execution system according to an embodiment of the present invention;
[0023] Figure 3 This is a logical structure diagram of a computer device according to an embodiment of the present invention.
[0024] Figure 4 is a flow chart of a native controller in the prior art;
[0025] Figure 5 is a flow chart of a JobSharding controller according to an embodiment of the present invention;
[0026] Figure 6 is a flowchart of the description information corresponding to the creation of the custom task in an embodiment of the present invention;
[0027] Figure 7 It is a sequential flow chart of the execution of tasks in an embodiment of the present invention. DETAILED DESCRIPTION
[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0029] like Figure 1 As shown, in combination with an embodiment of the present invention, a method for executing a task is provided, comprising:
[0030] S101: monitoring a custom task of a slice execution type sent by a user, wherein the custom task is divided into a plurality of slice tasks and the custom task has description information;
[0031] S102: When a custom task of a shard execution type is monitored, a shard environment variable array and container group Pod description information are parsed from the description information of the custom task, wherein each shard task in the custom task has a corresponding shard task identifier, the shard environment variable array is composed of each shard task identifier, and the Pod description information includes the maximum number of Pods required to run the custom task;
[0032] S103: Create a corresponding number of Pods according to the Pod description information, and match a unique shard task identifier for each created Pod according to the shard environment variable array;
[0033] S104: Run all created Pods, and execute the shard task corresponding to the matched shard task identifier through each Pod.
[0034] Without introducing a third-party system, the task to be executed is directly set as a custom task, and the function of task sharding is automatically realized. A corresponding number of Pods are created according to the Pod description information, and the sharded environment variables are injected into different Pod environment variables respectively, thereby creating a corresponding number of Pods, and each Pod executes the sharded task corresponding to the matched sharded task identifier. Multiple Pods work at the same time, avoiding the technical problem that one Pod works slowly, or that there are multiple identical Pods but they are repeatedly executed and the task execution cannot be accelerated.
[0035] Preferably, the description information of the custom task includes:
[0036] Type parameter, used to set the custom task to a sharding execution type;
[0037] The shard environment variable data parameter is used to add the shard task identifier corresponding to each shard task to form a shard environment variable array;
[0038] The maximum number of Pods parameter is used to set the maximum number of Pods that need to be run for the custom task at a given time. The maximum number of Pods is equal to the length of the shard environment variable array;
[0039] The maximum duration parameter is used to set the maximum duration from the start to the completion of the custom task;
[0040] The retry count parameter is used to set the retry count for each shard task.
[0041] In the embodiment of the present invention, CRD-Kubernetes-Pod-Kubebuilder uses the open source framework Kubebuilder to extend Kubernetes custom resources and implement a coordination controller to achieve the function of creating a job sharding execution. Figure 6 As shown in the figure, the user generates a custom task, in which the JobSharding custom task description is defined through the JobShardingSpec structure to implement the definition of the sharding execution type. Subsequently, the corresponding Pod will be created for the sharding task according to the description of the custom task, all the created Pods will be run, and the sharding task corresponding to the matched sharding task identifier will be executed through each Pod.
[0042] Among them, define the JobSharding custom task, specifically:
[0043] 1. Type parameter JobShardingSpec( Figure 6 001) in the JobSharding task, which is used to describe how to define a custom JobSharding task;
[0044] 2. Sharding environment variable data parameter Sharding: specifies the sharding environment variable data injected by the task for different Pods ( Figure 6 002), the value of the shard environment variable data is an array, and the length of the array should be equal to the number of Parallelisms; wherein, the environment variable refers to the environment variable of the operating system loaded when the Pod is running, which is related to the program running in the Pod, and the configuration required in the program corresponds to the environment variable parameters. For example, a large computing task, if run using a single pod, can only be calculated step by step from beginning to end; if the computing task adopts the method of this application, the computing task is divided into several shard tasks, and the division of the shard tasks is calculated separately through different identifiers, and the identifier is the environment variable.
[0045] The data structure of Sharding is an array. The number of pods to be started depends on the number of values in the array. The number of pods is determined by the number of parallelisms.
[0046] 3. Parallelism: Specifies the maximum number of Pods that a custom task should run at any given time.
[0047] 4. Completions: Specifies the number of successfully completed Pods required to run the custom task, which is usually equal to the maximum number of Pods;
[0048] 5. Maximum duration parameter ActiveDeadlineSeconds: specifies the maximum time from the start to the end of the custom task;
[0049] 6. Retry parameter BackoffLimit: specifies the number of retries for each shard task before marking this task as failed;
[0050] 7. Pod temporary description data Template: describes the Pod information that will be created when executing the task ( Figure 6 003) in the Pod description data format uniformly specified by Kubenetese, such as version;
[0051] 8. Suspend: Specify whether this running task needs to be suspended.
[0052] The description information of the JobSharding custom resource is encoded in a YAML file, and the content is as follows:
[0053]
[0054]
[0055] Preferably, the Pod description information also includes at least one of the following: shard execution type, Pod name, maximum duration, and number of retries, wherein the Pod name is the shard task identifier corresponding to the shard task to be executed by the Pod.
[0056] Preferably, the JobSharding controller, i.e., the admission controller webhook, is used to verify whether the JobSharding custom task submitted by the user meets the requirements. If it meets the requirements, a corresponding number of Pods are created according to the Pod description information, and then all the created Pods are run, and each Pod executes the sharding task corresponding to the matched sharding task identifier, thereby improving the task execution efficiency. The flowchart containing the JobSharding controller is shown in FIG. Figure 5 shown.
[0057] 1. When a user submits a custom JobSharding task, the custom JobSharding task will be submitted to this webhook service. After receiving the request, the spec.sharding of JobSharding will be verified. If the value does not exist or the data format does not conform to the defined data structure, an error message will be directly returned.
[0058] 2. Verify the value corresponding to spec.sharding. If it is a structure serialized as an array, calculate the length of the array. This length must be equal to spce.completions (the completions under the JobSharding custom task) so that the sharding environment variables can be injected into each Pod.
[0059] 3. Verify that spce.completions and spce.parallelism need to be equal.
[0060] Preferably, through the JobSharding coordinator ( Figure 7 003) in the JobSharding custom task ( Figure 7 002) in, including JobSharding custom creation, update, and deletion.
[0061] 1. In the JobSharding coordinator ( Figure 7 003) in the trigger, read the newly created JobSharding custom task (yaml format file) ( Figure 7 Contents of ( Figure 7 001), parse the description information of the custom task corresponding to spec.sharding and store the obtained content into an array.
[0062] Parse the content of spec.template (i.e. Template under the JobShardingSpec structure) to generate the container group Pod description information, such as ( Figure 7The example of 001 in the example generates three Pods. The Pod description information of the three Pods is different. The environment variable data is obtained from spec.sharding. The Pod name information is the index information of the corresponding Pod.
[0063] The array generated after parsing is as follows:
[0064]
[0065]
[0066] 2. After generating the Pod description information, the coordinator will create three Pods in sequence, and finally generate three running Pods, as shown in the figure ( Figure 7 005, 006, and 007), the three Pods have different names and different injected environment variables.
[0067] 4. After the Pod is started, according to different environment variables, the task ( Figure 7 008) for fragmentation processing.
[0068] In summary, if Figure 7 As shown in the figure, a JobSharding custom task is created. According to the description information of the JobSharding resource, the data identifying the sharding environment variable in the spec.sharding value is injected into the corresponding Pod environment variable according to the sharding environment variable index. The Pod executes different sharding tasks according to different environment variables, thereby achieving the purpose of sharding the custom task.
[0069] Preferably, the method for executing the task further includes:
[0070] S105: In the process of executing each shard task of the custom task through all created Pods, the status of the custom task, the start execution time of each shard task, the time when all shard tasks are completed, the number of Pods that successfully execute the corresponding shard tasks, the number of Pods that fail to execute the corresponding shard tasks, and the number of Pods that are currently running are updated according to the execution status, wherein the status of the custom task includes: executing, successful execution, or failed execution, and the status of the shard task includes: executing, successful execution, or failed execution.
[0071] Specifically, the JobShardingStatus structure is used to record the status of JobSharding after executing and coordinating tasks ( Figure 6 004). The specific fields are described as follows:
[0072] 1. Conditions Figure 6 005): Used to record the execution status and information of each Pod after the task is executed. The status (Status), reason (Reason) and message (Message, which refers to the creation information of the Pod) of a single Pod execution task are recorded in the form of an array;
[0073] The status of the custom task includes: being executed, successfully executed, or failed to execute; the status of the shard task includes: being executed, successfully executed, or failed to execute;
[0074] 2. StartTime: the start execution time of each shard task;
[0075] 3. CompletionTime: the time when all sharding tasks are completed;
[0076] 4. Active: The number of sharding tasks being executed at this time
[0077] 5. Succeeded: The number of sharding tasks that have been successfully executed at this time
[0078] 6. Failed: The number of sharding tasks that failed to execute at this time.
[0079] In the process of executing each shard task of the custom task through all the created Pods, various execution situations are recorded to track the task execution situation.
[0080] Preferably, the method for executing the task further includes:
[0081] S106: Detect in real time whether the number of running Pods is the same as the number of unfinished shard tasks;
[0082] S107: When the number of running Pods is less than the number of unexecuted shard tasks, the missing Pods are determined according to the Pod names;
[0083] S108: Recreate the missing Pod, and execute the corresponding unfinished sharding tasks through the recreated Pod.
[0084] Preferably, the method for executing the task further includes:
[0085] S109: When a user creates a new custom task of a shard execution type, the custom task is divided into multiple shard tasks according to the execution logic of the custom task, wherein the shard tasks do not have an execution order dependency relationship. Because the shard tasks do not have an execution order dependency relationship, the custom task can be divided into multiple shard tasks, and each corresponding Pod can be used to simultaneously execute the shard tasks corresponding to the matched shard task identifier, thereby improving the execution completion efficiency.
[0086] like Figure 2 As shown, in combination with an embodiment of the present invention, a task execution system is provided, including:
[0087] The coordinator 21 is used to monitor a custom task of a slice execution type sent by a user, wherein the custom task is divided into a plurality of slice tasks and has description information;
[0088] In the case of monitoring a custom task of a shard execution type, parse the shard environment variable array and the container group Pod description information from the description information of the custom task, wherein each shard task in the custom task has a corresponding shard task identifier, the shard environment variable array is composed of each shard task identifier, and the Pod description information includes the maximum number of Pods required to run the custom task;
[0089] Create a corresponding number of Pods according to the Pod description information, and match a unique shard task identifier for each created Pod according to the shard environment variable array;
[0090] The execution unit 22 is used to run all the created Pods, and execute the shard task corresponding to the matched shard task identifier through each Pod.
[0091] Without introducing a third-party system, the task to be executed is directly set as a custom task to automatically implement the task sharding function. A corresponding number of Pods are created according to the Pod description information, and the sharded environment variables are injected into different Pod environment variables respectively, thereby creating a corresponding number of Pods, each of which executes the sharding task corresponding to the matched sharding task identifier. Multiple Pods work at the same time, avoiding the technical problem that one Pod works slowly, or that there are multiple identical Pods but they are repeatedly executed and the task execution cannot be accelerated.
[0092] Preferably, the description information of the custom task includes:
[0093] Type parameter, used to set the custom task to a sharding execution type;
[0094] The shard environment variable data parameter is used to add the shard task identifier corresponding to each shard task to form a shard environment variable array;
[0095] The maximum number of Pods parameter is used to set the maximum number of Pods that need to be run for the custom task at a given time. The maximum number of Pods is equal to the length of the shard environment variable array;
[0096] The maximum duration parameter is used to set the maximum duration from the start to the completion of the custom task;
[0097] The retry count parameter is used to set the retry count for each shard task.
[0098] In the embodiment of the present invention, CRD-Kubernetes-Pod-Kubebuilder uses the open source framework Kubebuilder to extend Kubernetes custom resources and implement a coordination controller to achieve the function of creating a job sharding execution. Figure 6 As shown in the figure, the user generates a custom task, in which the JobSharding custom task description is defined through the JobShardingSpec structure to implement the definition of the sharding execution type. Subsequently, the corresponding Pod will be created for the sharding task according to the description of the custom task, all the created Pods will be run, and the sharding task corresponding to the matched sharding task identifier will be executed through each Pod.
[0099] Among them, define the JobSharding custom task, specifically:
[0100] 1. Type parameter JobShardingSpec( Figure 6 001) in the JobSharding task, which is used to describe how to define a custom JobSharding task;
[0101] 2. Sharding environment variable data parameter Sharding: specifies the sharding environment variable data injected by the task for different Pods ( Figure 6 002), the value of the shard environment variable data is an array, and the length of the array should be equal to the number of Parallelisms; wherein, the environment variable refers to the environment variable of the operating system loaded when the Pod is running, which is related to the program running in the Pod, and the configuration required in the program corresponds to the environment variable parameters. For example, a large computing task, if run using a single pod, can only be calculated step by step from beginning to end; if the computing task adopts the method of this application, the computing task is divided into several shard tasks, and the division of the shard tasks is calculated separately through different identifiers, and the identifier is the environment variable.
[0102] The data structure of Sharding is an array. The number of pods to be started depends on the number of values in the array. The number of pods is determined by the number of parallelisms.
[0103] 3. Parallelism: Specifies the maximum number of Pods that a custom task should run at any given time.
[0104] 4. Completions: Specifies the number of successfully completed Pods required to run the custom task, which is usually equal to the maximum number of Pods;
[0105] 5. Maximum duration parameter ActiveDeadlineSeconds: specifies the maximum time from the start to the end of the custom task;
[0106] 6. Retry parameter BackoffLimit: specifies the number of retries for each shard task before marking this task as failed;
[0107] 7. Pod temporary description data Template: describes the Pod information that will be created when executing the task ( Figure 6 003) in the Pod description data format uniformly specified by Kubenetese, such as version;
[0108] 8. Suspend: Specify whether this running task needs to be suspended.
[0109] The description information of the JobSharding custom resource is encoded in a YAML file, and the content is as follows:
[0110]
[0111]
[0112] Preferably, the Pod description information also includes at least one of the following: shard execution type, Pod name, maximum duration, and number of retries, wherein the Pod name is the shard task identifier corresponding to the shard task to be executed by the Pod.
[0113] Preferably, the JobSharding controller is the admission controller webhook, which is mainly used to verify whether the JobSharding custom task submitted by the user meets the regulations. If it meets the specifications, a corresponding number of Pods are created according to the Pod description information, and then all the created Pods are run, and each Pod executes the sharding task corresponding to the matched sharding task identifier to improve the task execution efficiency. The flowchart containing the JobSharding controller is shown in FIG. Figure 5 shown.
[0114] 1. When a user submits a custom JobSharding task, the custom JobSharding task will be submitted to this webhook service. After receiving the request, the spec.sharding of JobSharding will be verified. If the value does not exist or the data format does not conform to the defined data structure, an error message will be directly returned.
[0115] 2. Verify the value corresponding to spec.sharding. If it is a structure serialized as an array, calculate the length of the array. This length must be equal to spce.completions (the completions under the JobSharding custom task) so that the sharding environment variables can be injected into each Pod.
[0116] 3. Verify that spce.completions and spce.parallelism need to be equal.
[0117] Preferably, through the JobSharding coordinator ( Figure 7 003) in the JobSharding custom task ( Figure 7 002) in, including JobSharding custom creation, update, and deletion.
[0118] 1. In the JobSharding coordinator ( Figure 7 003) in the trigger, the coordinator's parsing subunit reads the newly created JobSharding custom task (yaml format file) ( Figure 7 Contents of ( Figure 7 001), parse the description information of the custom task corresponding to spec.sharding and store the obtained content into an array.
[0119] Parse the content of spec.template (i.e. Template under the JobShardingSpec structure) to generate the container group Pod description information, such as ( Figure 7 The example of 001 in the example generates three Pods. The Pod description information of the three Pods is different. The environment variable data is obtained from spec.sharding. The Pod name information is the index information of the corresponding Pod.
[0120] The array generated after parsing is as follows:
[0121]
[0122]
[0123] 2. After the Pod description information is generated, the Pod creation subunit of the coordinator will create three Pods in sequence, and finally generate three running Pods, as shown in the figure ( Figure 7 005, 006, and 007), the three Pods have different names and different injected environment variables.
[0124] 4. The Pod execution subunit of the coordinator 21 is used to execute tasks ( Figure 7 008) for fragmentation processing.
[0125] In summary, if Figure 7 As shown in the figure, a JobSharding custom task is created. According to the description information of the JobSharding resource, the data identifying the sharding environment variable in the spec.sharding value is injected into the corresponding Pod environment variable according to the sharding environment variable index. The Pod executes different sharding tasks according to different environment variables, thereby achieving the purpose of sharding the custom task.
[0126] Preferably, the update subunit of the coordinator 21 is used to update the status of the custom task, the start execution time of each shard task, the time when all shard tasks are completed, the number of Pods that successfully execute the corresponding shard tasks, the number of Pods that fail to execute the corresponding shard tasks, and the number of Pods that are currently running according to the execution status in the process of executing each shard task of the custom task through all created Pods, wherein the status of the custom task includes: executing, successful execution, or failed execution, and the status of the shard task includes: executing, successful execution, or failed execution.
[0127] Specifically, the JobShardingStatus structure is used to record the status of JobSharding after executing and coordinating tasks ( Figure 6 004). The specific fields are described as follows:
[0128] 1. Conditions Figure 6 005): Used to record the execution status and information of each Pod after the task is executed. The status (Status), reason (Reason) and message (Message, which refers to the creation information of the Pod) of a single Pod execution task are recorded in the form of an array;
[0129] The status of the custom task includes: being executed, successfully executed, or failed to execute; the status of the shard task includes: being executed, successfully executed, or failed to execute;
[0130] 2. StartTime: the start execution time of each shard task;
[0131] 3. CompletionTime: the time when all sharding tasks are completed;
[0132] 4. Active: The number of sharding tasks being executed at this time
[0133] 5. Succeeded: The number of sharding tasks that have been successfully executed at this time
[0134] 6. Failed: The number of sharding tasks that failed to execute at this time.
[0135] In the process of executing each shard task of the custom task through all the created Pods, various execution situations are recorded to track the task execution situation.
[0136] Preferably, the coordinator 21 further includes a detection subunit and a reconstruction subunit, wherein
[0137] The detection subunit is used to detect in real time whether the number of running Pods is the same as the number of unfinished sharding tasks; when the number of running Pods is less than the number of unfinished sharding tasks, the missing Pod is determined based on the Pod name;
[0138] The rebuild subunit is used to recreate the missing Pod and execute the corresponding unfinished sharding tasks through the recreated Pod.
[0139] Preferably, the method for executing the task further includes:
[0140] S109: When a user creates a new custom task of a shard execution type, the custom task is divided into multiple shard tasks according to the execution logic of the custom task, wherein the shard tasks do not have an execution order dependency relationship. Because the shard tasks do not have an execution order dependency relationship, the custom task can be divided into multiple shard tasks, and each corresponding Pod can be used to simultaneously execute the shard tasks corresponding to the matched shard task identifier, thereby improving the execution completion efficiency.
[0141] In combination with an embodiment of the present invention, a computer-readable storage medium is also provided, which stores one or more programs. When the one or more programs are executed by a computer device, the computer device executes any one of the aforementioned task execution methods.
[0142] like Figure 3 As shown, in combination with an embodiment of the present invention, a computer device is also provided, including:
[0143] A processor; and a memory arranged to store computer executable instructions, which, when executed, cause the processor to perform any of the aforementioned task execution methods.
[0144] The beneficial effects achieved by the embodiments of the present invention are as follows:
[0145] Without introducing a third-party system, the task to be executed is directly set as a custom task, and the custom task is divided into multiple shard tasks, which automatically realizes the function of task sharding processing. Create a corresponding number of Pods according to the Pod description information, and inject the sharded environment variables into different Pod environment variables respectively, so as to create a corresponding number of Pods, and each Pod executes the shard task corresponding to the matched shard task identifier. By having each Pod execute the shard task corresponding to the matched shard task identifier, the function of sharding the task is realized, which speeds up the processing speed of the task and improves the processing efficiency. Multiple Pods work at the same time, avoiding the technical problem that one Pod works slowly, or that there are multiple identical Pods but they are repeatedly executed and the task execution cannot be accelerated. Among them, custom tasks can be realized by using the custom resources and controller mode supported by Kubernetes, and cannot be built separately, and there is no need to introduce a third-party system.
[0146] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.
[0147] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.
[0148] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.
[0149] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".
[0150] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.
[0151] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.
[0152] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.
[0153] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.
[0154] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for executing a task, characterized in that: include: Listening to a custom task of a slice execution type sent by a user, wherein the custom task is divided into a plurality of slice tasks and the custom task has description information; In the case of monitoring a custom task of a shard execution type, parse the shard environment variable array and the container group Pod description information from the description information of the custom task, wherein each shard task in the custom task has a corresponding shard task identifier, the shard environment variable array is composed of each shard task identifier, and the Pod description information includes the maximum number of Pods required to run the custom task; Create a corresponding number of Pods according to the Pod description information, and match a unique shard task identifier for each created Pod according to the shard environment variable array; Run all created Pods, and execute the shard task corresponding to the matched shard task identifier through each Pod.
2. The task execution method according to claim 1, characterized in that: The description information of the custom task includes: Type parameter, used to set the custom task to a sharding execution type; The shard environment variable data parameter is used to add the shard task identifier corresponding to each shard task to form a shard environment variable array; The maximum number of Pods parameter is used to set the maximum number of Pods that need to be run for the custom task at a given time. The maximum number of Pods is equal to the length of the shard environment variable array; The maximum duration parameter is used to set the maximum duration from the start to the completion of the custom task; The retry count parameter is used to set the retry count for each shard task.
3. The method for executing a task according to claim 2, characterized in that: The Pod description information also includes at least one of the following: shard execution type, Pod name, maximum duration, and number of retries, wherein the Pod name is the shard task identifier corresponding to the shard task to be executed by the Pod.
4. The task execution method according to claim 1, characterized in that: Also includes: In the process of executing each shard task of the custom task through all created Pods, the status of the custom task, the start execution time of each shard task, the time when all shard tasks are completed, the number of Pods that successfully execute the corresponding shard tasks, the number of Pods that fail to execute the corresponding shard tasks, and the number of Pods currently running are updated according to the execution status, wherein the status of the custom task includes: executing, successful execution, or failed execution, and the status of the shard task includes: executing, successful execution, or failed execution.
5. The task execution method according to claim 1, characterized in that: Also includes: Real-time detection of whether the number of running Pods is the same as the number of unfinished sharding tasks; When the number of running Pods is less than the number of unfinished shard tasks, the missing Pods are identified based on the Pod names; Recreate the missing Pods and use the recreated Pods to execute the corresponding unfinished sharding tasks.
6. The task execution method according to claim 1, characterized in that: Also includes: When a user newly creates a custom task of a slice execution type, the custom task is divided into a plurality of slice tasks according to the execution logic of the custom task, wherein the slice tasks have no execution order dependency relationship.
7. A task execution system, characterized in that: The execution system of the task includes: A coordinator, configured to monitor a custom task of a slice execution type sent by a user, wherein the custom task is divided into a plurality of slice tasks and has description information; In the case of monitoring a custom task of a shard execution type, parse the shard environment variable array and the container group Pod description information from the description information of the custom task, wherein each shard task in the custom task has a corresponding shard task identifier, the shard environment variable array is composed of each shard task identifier, and the Pod description information includes the maximum number of Pods required to run the custom task; Create a corresponding number of Pods according to the Pod description information, and match a unique shard task identifier for each created Pod according to the shard environment variable array; The execution unit is used to run all created Pods and execute the shard task corresponding to the matched shard task identifier through each Pod.
8. The task execution system according to claim 7, characterized in that: It also includes a task generation unit, The task generation unit is used to set the description information of the custom task, and the description information of the custom task includes: Type parameter, used to set the custom task to a sharding execution type; The shard environment variable data parameter is used to add the shard task identifier corresponding to each shard task to form a shard environment variable array; The maximum number of Pods parameter is used to set the maximum number of Pods that need to be run for the custom task at a given time. The maximum number of Pods is equal to the length of the shard environment variable array; The maximum duration parameter is used to set the maximum duration from the start to the completion of the custom task; The retry count parameter is used to set the retry count for each shard task.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a computer device, the computer device executes the method for executing the task described in any one of claims 1 to 6.
10. A computer device, characterized in that: include: processor; And, a memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the method for performing the task as claimed in any one of claims 1 to 6.
