Universal task configuration updating method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention relates to a general task configuration updating method and device, a storage medium and electronic equipment, and relates to the technical field of data processing. The method is applied to a data platform and comprises the following steps: selecting a current batch of trial tasks from a plurality of to-be-processed tasks; determining whether to execute each pilot task in the current batch of pilot tasks based on the new-version general task configuration according to a first execution success rate when the new-version general task configuration is applied to the executed tasks in the current batch of pilot tasks; when the execution of the current batch of pilot tasks is completed and a second execution success rate when the new-version general task configuration is applied to the current batch of pilot tasks meets a first expected standard, selecting the next batch of pilot tasks; and in response to the fact that all the to-be-processed tasks are selected as pilot tasks and a third execution success rate when the new-version general task configuration is applied to the last batch of pilot tasks meets a second expected standard, updating the old-version general task configuration. According to the invention, online accidents caused by platform configuration upgrading can be avoided.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of data processing. More specifically, embodiments of the present disclosure relate to a method for updating a general task configuration, a device for updating a general task configuration, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section aims to provide background or context for embodiments of the present disclosure. The descriptions herein are not admitted to be prior art merely because they are included in this section.

[0003] A data platform is a development platform used in an enterprise for developing, scheduling, and executing big data tasks, providing a user interface to facilitate users to submit tasks to be processed and perform operation and maintenance management. In addition to the business code of tasks, the data platform also includes the running configurations of tasks. Summary of the Invention

[0004] The running configuration of a task generally includes two parts. The first is the configuration of the task itself, which only affects the current task, and the other is the general task configuration of the platform, which affects all tasks on the platform. Refer to Figure 1 , Figure 1 shows a schematic diagram of the running configuration of tasks in a data platform. Specifically, in the case where there are multiple tasks (for example: Task 1, Task 2, Task 3, Task 4) in the data platform, each task corresponds to a separate task configuration, and the general task configuration of the platform will act on all the above tasks.

[0005] In the related art, after updating the general task configuration of the platform, a gray release method is generally adopted to roll out the general task configuration of the platform. That is, a new general task configuration of the platform is first created, and then it is gradually rolled out in a gray scale according to a ratio. Some tasks are automatically selected to use the general task configuration of the platform. If there are no problems, the gray scale range is gradually expanded until all tasks use the general task configuration of the platform, and the entire rollout process of the general task configuration of the platform is completed.

[0006] Refer to Figure 2 , Figure 2 shows a schematic diagram of how to perform gray box testing on the general task configuration of the platform in the related art. As Figure 2 shown, Task 3 and Task 4 can be selected as gray box testing tasks, and Task 1 and Task 2 are used as non-gray box testing tasks. Thus, when Task 1 and Task 2 are executed, the old version of the general task configuration of the platform is used, while when Task 3 and Task 4 are executed, the new version of the general task configuration of the platform is used.

[0007] However, the above solutions have at least the following problems:

[0008] First, tasks that have been grayscale may still fail during subsequent actual execution due to the online release of the general task configuration of the platform.

[0009] Second, since the grayscale range is fixed, the entire grayscale process takes a certain amount of time. The batch scheduling tasks of the data platform are generally scheduled and executed according to time configuration, and it generally takes at least one day to complete a cycle. In this scenario, it often takes at least one week to complete the entire grayscale coverage from 0 to 100%, and end the online release process of the general task configuration of the entire platform.

[0010] Therefore, there is a great need for a general task configuration update method that can avoid online problems and accidents caused by platform configuration upgrades and reduce the time for platform configuration upgrades.

[0011] In view of this, the present disclosure proposes a new general task configuration update method. Specifically, after the development of the new version of the general task configuration to be released is completed, a gradual grayscale strategy will be adopted for release, that is, select some pilot tasks, execute the above-mentioned part of the pilot tasks using the above-mentioned new version of the general task configuration, and monitor the execution success rate in real time. When the execution success rate reaches the expected standard, continue to select pilot tasks until the success rate of the last batch of pilot tasks meets the expected standard, and then use the new version of the general task configuration to update the old version of the general task configuration, thereby reducing the upgrade risk and avoiding online accidents and problems caused by platform configuration upgrades.

[0012] In this context, the embodiments of the present disclosure are expected to provide a general task configuration update method, a general task configuration update device, a computer-readable storage medium, and an electronic device.

[0013] According to the first aspect of the embodiments of the present disclosure, a general task configuration update method is provided, which is applied to a data platform. The data platform contains multiple tasks to be processed, and the data platform also contains an old version of the general task configuration and a new version of the general task configuration to be released. The method includes:

[0014] Select the current batch of pilot tasks from the multiple tasks to be processed;

[0015] Determine whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks;

[0016] When the current batch of pilot tasks is completed and the second execution success rate when the new version of the general task configuration is applied to the current batch of pilot tasks meets the first expected standard, select the next batch of pilot tasks;

[0017] When the third execution success rate when the new version of the general task configuration is applied to the last batch of pilot tasks satisfies the second expected standard in response to all of the multiple tasks to be processed being selected as the pilot tasks, update the old version of the general task configuration with the new version of the general task configuration.

[0018] In an alternative implementation, selecting the current batch of pilot tasks from the multiple tasks to be processed includes:

[0019] Obtain the difference degree of the new version of the general task configuration relative to the old version of the general task configuration;

[0020] According to the difference degree, find the corresponding relationship between the difference degree and the number of pilot tasks pre-configured, and determine the number of tasks of the current batch of pilot tasks;

[0021] Based on the number of tasks and in combination with the task priority corresponding to each of the tasks to be processed, select the current batch of pilot tasks from the multiple tasks to be processed.

[0022] In an alternative implementation, the selecting the current batch of pilot tasks from the multiple tasks to be processed based on the number of tasks and in combination with the task priority corresponding to each of the tasks to be processed includes:

[0023] Sort the task priorities corresponding to the multiple tasks to be processed in ascending order to form a sorted sequence;

[0024] Starting from the starting position of the sorted sequence, screen out the current batch of pilot tasks that match the number of tasks.

[0025] In an alternative implementation, after selecting the current batch of pilot tasks from the multiple tasks to be processed, the method further includes:

[0026] Execute the starting task in the current batch of pilot tasks based on the new version of the general task configuration;

[0027] If the execution fails, re-execute the starting task in the current batch of pilot tasks based on the old version of the general task configuration.

[0028] In an alternative implementation, determining whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks includes:

[0029] When the first execution success rate meets the third expected standard, call the new version of the general task configuration to execute each pilot task;

[0030] When the first execution success rate does not meet the third expected standard, call the old version of the general task configuration to execute each pilot task.

[0031] In an alternative embodiment, the method further includes:

[0032] When the second execution success rate when the current batch of pilot tasks is completed and the new version of the general task configuration is applied to the current batch of pilot tasks does not meet the first expected standard, repair the new version of the general task configuration.

[0033] In an alternative embodiment, after repairing the new version of the general task configuration, the method further includes:

[0034] Call the repaired new version of the general task configuration to re-execute the current batch of pilot tasks and re-evaluate the second execution success rate.

[0035] According to the second aspect of the embodiments of the present disclosure, there is provided a general task configuration update device disposed in a data platform. The data platform includes a plurality of tasks to be processed, and the data platform further includes an old version of the general task configuration and a new version of the general task configuration to be launched. The device includes:

[0036] A first task selection module for selecting a current batch of pilot tasks from the plurality of tasks to be processed;

[0037] A task execution module for determining whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks;

[0038] A second task selection module for selecting the next batch of pilot tasks when the second execution success rate when the current batch of pilot tasks is completed and the new version of the general task configuration is applied to the current batch of pilot tasks meets the first expected standard;

[0039] A configuration update module for updating the old version of the general task configuration with the new version of the general task configuration in response to the third execution success rate when all the plurality of tasks to be processed are selected as the pilot tasks and the new version of the general task configuration is applied to the last batch of pilot tasks meeting the second expected standard.

[0040] According to the third aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the general task configuration update method described in the first aspect above is implemented.

[0041] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, including: a processor; and a memory for storing executable instructions of the processor; wherein, the processor is configured to execute the general task configuration update method described in the first aspect above by executing the executable instructions.

[0042] For the general task configuration update method, general task configuration update device, computer-readable storage medium and electronic device according to the embodiments of the present disclosure, on the one hand, by gradually selecting pilot tasks and testing the effects of the new version of the general task configuration, potential problems can be identified without affecting the overall business. Whether to continue using the new version of the general task configuration is determined according to the first execution success rate of each batch of pilot tasks. This mechanism allows for quick response and adjustment of strategies to address any possible problems; further, through dual success rate monitoring, that is, the success rate monitoring during the execution of a single pilot task combined with the overall success rate monitoring during the execution of each batch of pilot tasks, the successful execution of each task is guaranteed, thereby avoiding online problems and accidents and reducing the configuration upgrade cost of the platform; on the other hand, by adopting a progressive configuration update deployment method, the impact on user services is reduced, and the consistency and continuity of the user experience are guaranteed. Thus, both the safe introduction of new functions or improvements can be ensured, and the impact on existing services can be minimized. It is a very robust and efficient configuration upgrade strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the drawings, several embodiments of the present disclosure are shown in an exemplary rather than restrictive manner, wherein:

[0044] Figure 1 A schematic diagram showing the running configuration of tasks in a data platform is shown;

[0045] Figure 2 A schematic diagram showing how to perform gray-box testing on the general task configuration of the platform in the related art is shown;

[0046] Figure 3 A schematic flowchart showing the general task configuration update method in the present disclosure is shown;

[0047] Figure 4 A schematic flowchart showing how to select the current batch of pilot tasks from the tasks to be processed in an embodiment of the present disclosure is shown;

[0048] Figure 5 A schematic flowchart showing how to select the current batch of pilot tasks from multiple tasks to be processed based on the number of tasks in combination with the task priority corresponding to each task to be processed in an embodiment of the present disclosure is shown;

[0049] Figure 6 It shows a schematic flowchart for determining whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration when the first execution success rate of the executed tasks in the current batch of pilot tasks according to the new version of the general task configuration is applied;

[0050] Figure 7 It shows a schematic flowchart of how to execute each pilot task in the current batch of pilot tasks;

[0051] Figure 8 It shows a schematic overall flowchart of the general task configuration update method in the embodiments of the present disclosure;

[0052] Figure 9 It shows a schematic diagram of a general task configuration update device according to an embodiment of the present disclosure; and

[0053] Figure 10 It shows a structural diagram of an electronic device according to an embodiment of the present disclosure.

[0054] In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Embodiments

[0055] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and then implement the present disclosure, rather than limiting the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to be able to fully convey the scope of the present disclosure to those skilled in the art.

[0056] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, a device, an equipment, a method, or a computer program product. Therefore, the present disclosure can be specifically implemented in the following forms: completely hardware, completely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0057] According to an embodiment of the present disclosure, there is provided a general task configuration update method, a general task configuration update device, a computer-readable storage medium, and an electronic device.

[0058] In this article, the number of any element in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0059] Below, with reference to several representative embodiments of the present disclosure, the principles and spirit of the present disclosure will be elaborated in detail. Summary of the Invention

[0061] The inventor of the present invention has found that the existing platform general configuration update solutions have low efficiency and high upgrade risks, which are likely to cause online problems and accidents.

[0062] In view of the above, the basic idea of the present disclosure is as follows: to provide a general task configuration update method, a general task configuration update device, a computer-readable storage medium, and an electronic device. On the one hand, by gradually selecting pilot tasks and testing the effects of the new version of the general task configuration, potential problems can be identified without affecting the overall business. Whether to continue using the new version of the general task configuration is determined based on the first execution success rate of each batch of pilot tasks. This mechanism allows for quick response and adjustment of strategies to address any possible problems. Further, through dual success rate monitoring, that is, the success rate monitoring during the execution of a single pilot task combined with the overall success rate monitoring during the execution of each batch of pilot tasks, the successful execution of each task is ensured, thereby avoiding online problems and accidents and reducing the platform's configuration upgrade cost. On the other hand, by adopting a progressive configuration update deployment method, the impact on user services is reduced, and the consistency and continuity of the user experience are ensured. Thus, it can not only ensure the safe introduction of new functions or improvements but also minimize the possible impact on existing operations, which is a very robust and efficient configuration upgrade strategy.

[0063] After introducing the basic principle of the present disclosure, the various non-limiting embodiments of the present disclosure will be specifically introduced below.

[0064] Overview of Application Scenarios

[0065] It should be noted that the following application scenarios are only shown for the convenience of understanding the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0066] The embodiments of the present disclosure support the safe upgrade of the platform general task configuration on the premise of reducing online problems and risks. Specifically, the current batch of pilot tasks can be selected from multiple tasks to be processed first. When the first execution success rate meets the requirements when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks, the next batch of pilot tasks is selected until all multiple tasks to be processed are selected as pilot tasks and the third execution success rate meets the second expected standard when the new version of the general task configuration is applied to the last batch of pilot tasks. Then, the old version of the general task configuration is updated with the new version of the general task configuration. Thus, through dual success rate monitoring, that is, the success rate monitoring during the execution of a single pilot task combined with the overall success rate monitoring during the execution of each batch of pilot tasks, the successful execution of each task is ensured, thereby avoiding online problems and accidents and reducing the platform's configuration upgrade cost.

[0067] Exemplary Method

[0068] Exemplary embodiments of the present disclosure first provide a general task configuration update method. Figure 3 A flowchart showing the general task configuration update method in the present disclosure may include the following steps S310 to S340:

[0069] Step S310: Select a current batch of pilot tasks from multiple tasks to be processed;

[0070] Step S320: Determine whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks;

[0071] Step S330: When the second execution success rate when the new version of the general task configuration is applied to the current batch of pilot tasks meets the first expected standard after the execution of the current batch of pilot tasks is completed, select the next batch of pilot tasks;

[0072] Step S340: In response to that all tasks to be processed are selected as pilot tasks and the third execution success rate when the new version of the general task configuration is applied to the last batch of pilot tasks meets the second expected standard, update the old version of the general task configuration with the new version of the general task configuration.

[0073] The following explains the specific implementation manners of the above steps:

[0074] Before step S310, it should be noted that the method in the present disclosure is applied to a data platform, which refers to a development platform in an enterprise for developing and scheduling the execution of big data tasks, providing a user interface to facilitate users to submit data processing tasks and perform operation and maintenance management. Specifically, the functions of the data platform may include the following aspects:

[0075] ① Task development: Provide a user interface to facilitate users to upload or edit and save data processing tasks;

[0076] ② Task execution: Provide the execution ability of data tasks, including but not limited to popular Spark batch processing tasks and Flink real-time processing tasks currently;

[0077] Among them, Apache Spark is an open source cluster computing framework originally developed by AMPLab at the University of California, Berkeley. Compared with Hadoop's MapReduce, which stores intermediate data on disk after running the work, Spark uses in-memory computing technology, which can analyze and calculate in memory before the data is written to the hard disk. Spark can run programs in memory 100 times faster than Hadoop MapReduce, and even when running programs on the hard disk, Spark can be 10 times faster. In addition to the input parameters for startup, Spark tasks also support a series of configuration transfers. The configuration is in KV form and can be passed through parameters.

[0078] ③Task scheduling: Provides scheduling execution capabilities based on time and task flow dependencies;

[0079] ④Operation and maintenance monitoring: Ability to provide timeout alarm for task failure.

[0080] The data platform may contain multiple tasks to be processed. At the same time, the data platform may also contain old version general task configurations and new version general task configurations to be launched. The old version general task configuration may be the general task configuration currently used by the data platform, and the new version general task configuration may be a new general task configuration formed after adding, deleting, modifying, or optimizing the old version general task configuration. The new version general task configuration needs to be tested before it goes online to avoid potential problems.

[0081] In step S310, a current batch of pilot tasks is selected from a plurality of tasks to be processed.

[0082] In this step, the current batch of pilot tasks can be selected from multiple tasks to be processed.

[0083] In an optional implementation, illustratively, each pending task may correspond to a task identifier, so that the current batch processing pilot task may be selected from the plurality of pending tasks based on the task identifier by a random algorithm.

[0084] In an optional embodiment, reference Figure 4 , Figure 4 A flow chart showing how to select the current batch of pilot tasks from the tasks to be processed in the embodiment of the present disclosure includes steps S401 to S403:

[0085] In step S401, the difference between the new version general task configuration and the old version general task configuration is obtained.

[0086] In this step, the difference degree between the new version of the general task configuration and the old version of the general task configuration can be obtained. Based on this difference degree, the change magnitude of the new version of the general task configuration relative to the old version of the general task configuration can be determined, so as to facilitate configuring the number of pilot tasks selected by circle according to different change magnitudes. Exemplarily, in the case of a small change, the number of pilot tasks can be set to a number greater than the preset threshold, while in the case of a large change, the number of pilot tasks can be set to a number less than the preset threshold, so that the number of pilot tasks can be flexibly configured according to the change magnitude, thereby reducing the probability of configuration problems.

[0087] In addition, since the number of pilot tasks can be set according to requirements, compared with the solution in the related art of automatically selecting tasks according to a fixed number, the present disclosure can customize the number of tasks according to requirements, thereby improving the upgrade efficiency of the general configuration of the platform.

[0088] In step S402, according to the difference degree, find the corresponding relationship between the pre-configured difference degree and the number of pilot tasks, and determine the number of tasks of the current batch of pilot tasks.

[0089] In this step, exemplarily, the corresponding relationship between the pre-configured difference degree and the number of pilot tasks can be found according to the difference degree, and the number of tasks of the current batch of pilot tasks can be determined. Exemplarily, it can be set that when the difference degree is 10%, the number of tasks of the current batch of pilot tasks can be 100, and when the difference degree is 20%, the number of tasks of the current batch of pilot tasks can be 50. The specific corresponding relationship can be set according to actual needs, and the present disclosure does not make special limitations on this.

[0090] In step S403, based on the number of tasks and combined with the task priority corresponding to each task to be processed, select the current batch of pilot tasks from multiple tasks to be processed.

[0091] In this step, based on the number of tasks and combined with the task priority corresponding to each task to be processed, the current batch of pilot tasks can be selected from multiple tasks to be processed. Exemplarily, the task priority can be determined according to the size of the task volume. For example, the smaller the task volume, the lower the task priority, and the larger the task volume, the higher the task priority. Or, the task priority can also be determined according to the importance of the task, etc. For example, the lower the importance of the task, the lower the task priority, and the higher the importance of the task, the higher the task priority. All can be set according to the actual situation, and the present disclosure does not make special limitations on this.

[0092] In an alternative embodiment, refer to Figure 5 , Figure 5The flowchart shows how to select the current batch of pilot tasks from multiple tasks to be processed based on the number of tasks and the task priority corresponding to each task to be processed in the embodiments of the present disclosure, including steps S501 - S502:

[0093] In step S501, the task priorities corresponding to multiple tasks to be processed are sorted in ascending order to form a sorted sequence.

[0094] In this step, the task priorities corresponding to the above - mentioned multiple tasks to be processed can be sorted in ascending order to form a sorted sequence.

[0095] In step S502, starting from the starting position of the sorted sequence, the current batch of pilot tasks that match the number of tasks are selected.

[0096] In this step, starting from the starting position of the sorted sequence, the current batch of pilot tasks that match the above - mentioned number of tasks can be selected. Exemplarily, assuming the number of tasks is 100, the tasks to be processed in the first 100 positions of the sorted sequence can be selected as the above - mentioned current batch of pilot tasks.

[0097] By preferentially selecting the tasks to be processed with lower task priorities as the current batch of pilot tasks, on the one hand, a relatively safe configuration test environment can be provided. Given that tasks with lower priorities usually have less impact on the business process, even if errors or problems occur when testing new configurations on these tasks, it will not seriously affect the overall business operation. Further, given that low - priority tasks may involve fewer resources (such as computing resources, storage space, etc.), it is easier to make experimental adjustments and optimizations. In this way, the performance and stability of the new version of the general configuration can be evaluated without affecting the main business operations. On the other hand, by implementing and adjusting new configurations or processes on low - priority tasks, developers can pre - understand the possible problems and solutions, avoiding the pressure and urgency caused by problems occurring in high - priority tasks.

[0098] Then refer to Figure 3 In step S320, it is determined whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks.

[0099] In this step, after selecting the current batch of pilot tasks from multiple tasks to be processed, for the starting task in the current batch of pilot tasks, the new version of the general task configuration can be called to execute the starting task first. If the execution is successful, the process can jump to the next pilot task in the current batch of pilot tasks. If the execution fails, the starting task in the current batch of pilot tasks can be re-executed based on the old version of the general task configuration to provide a fallback strategy and avoid missed task execution.

[0100] For each pilot task in the current batch of pilot tasks except the starting task, it can be determined whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks. Specifically, reference can be made to Figure 6 , Figure 6 FIG. shows a flowchart of determining whether to execute each pilot task in the current batch of pilot tasks based on the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks in an embodiment of the present disclosure, including step S601-step S602:

[0101] In step S601, when the first execution success rate meets the third expected standard, the new version of the general task configuration is called to execute each pilot task.

[0102] In this step, for each pilot task in the current batch of pilot tasks except the starting task, the first execution success rate when the tasks in the current batch of pilot tasks are executed can be obtained. When the first execution success rate meets the third expected standard (for example: the first execution success rate is greater than 85%, which can be set according to the actual situation and is not specifically limited in the present disclosure), the new version of the general task configuration can be called to execute each pilot task.

[0103] In step S602, when the first execution success rate does not meet the third expected standard, the old version of the general task configuration is called to execute each pilot task.

[0104] In this step, if the above first execution success rate does not meet the above third expected standard, the old version of the general task configuration can be called to execute each pilot task.

[0105] By determining whether to call the new version of the general task configuration to execute each pilot task according to the third expected standard of the success rate, the new version of the general task configuration can be gradually introduced on the premise of ensuring system stability. Once it is found that the platform general task configuration cannot meet the expected success rate, quickly rolling back to the old version can greatly reduce the service interruption time caused by configuration errors and improve the availability of the data platform; it also helps to ensure that only the new version of the general task configuration that has been actually tested and proven to reach a certain quality level will be widely applied, which increases the reliability and stability of the data platform.

[0106] Reference Figure 7 , Figure 7 shows a schematic flowchart of how to execute each pilot task in the current batch of pilot tasks in an embodiment of the present disclosure, including steps S701 - S706:

[0107] In step S701, start;

[0108] In step S702, determine whether the first execution success rate of the executed tasks does not meet the third expected standard;

[0109] If so, enter step S703, and call the old version of the general task configuration to execute the pilot task;

[0110] If not, enter step S704, and call the new version of the general task configuration to execute the pilot task;

[0111] In step S705, determine whether the execution is successful;

[0112] If the execution is not successful, jump to S703;

[0113] If the execution is successful, enter step 706, and end.

[0114] Then refer to Figure 3 , in step S330, when the second execution success rate when the current batch of pilot tasks is executed and the new version of the general task configuration is applied to the current batch of pilot tasks meets the first expected standard, select the next batch of pilot tasks.

[0115] In this step, when the second execution success rate (i.e., the overall execution success rate for the current batch of pilot tasks) when the current batch of pilot tasks is executed and the new version of the general task configuration is applied to the current batch of pilot tasks meets the first expected standard (for example: 100%, which can be set by itself according to the actual situation, and the present disclosure does not make special limitations on this), the next batch of pilot tasks can be selected. The specific process of selecting the next batch of pilot tasks can refer to the above step S310, which will not be elaborated here.

[0116] It should be noted that when the second execution success rate when the current batch of pilot tasks are completed and the new version of the general task configuration is applied to the current batch of pilot tasks does not meet the first expected standard, the new version of the general task configuration can be repaired, so that targeted repairs can be made according to the problems found in the pilot tasks. This can not only solve existing errors, but also discover and improve potential performance bottlenecks, thereby optimizing the entire task configuration. By gradually repairing and improving the platform's general task configuration, the maturity of the new configuration can be steadily improved without affecting the main business processes, thereby reducing the risk of large-scale deployment failures in the future and laying a solid foundation for the full promotion of subsequent new versions.

[0117] After repairing the new version of the general task configuration, the repaired new version of the general task configuration can be called to re-execute the current batch of pilot tasks, and the second execution success rate can be re-evaluated. When the second execution success rate after this re-evaluation meets the above first expected standard, the next batch of pilot tasks can be selected. This can optimize the new version of the general task configuration in a timely manner when problems are discovered to avoid the spread of problems.

[0118] In step S340, in response to multiple tasks to be processed being selected as pilot tasks and the third execution success rate when the new version of the general task configuration is applied to the last batch of pilot tasks meeting the second expected standard, the old version of the general task configuration is updated using the new version of the general task configuration.

[0119] In this step, when the third execution success rate when multiple tasks to be processed in the data platform are selected as pilot tasks and the new version of the general task configuration is applied to the last batch of pilot tasks meets the second expected standard (for example: 100%, which can be set according to the actual situation and is not specifically limited in this disclosure), the old version of the general task configuration can be updated using the new version of the general task configuration.

[0120] This disclosure can reduce the need for manual operations through automated monitoring and intelligent adjustment, and can also discover and solve problems in a timely manner, thereby effectively reducing the cost of upgrading the data platform configuration environment. This solution not only ensures the stability of the data platform, improves resource utilization, but also minimizes the impact on business continuity and user experience. Therefore, it is very suitable for data platform environments that require frequent updates and iterations, and helps to achieve efficient and stable operation and maintenance management.

[0121] Reference Figure 8 , Figure 8 shows the overall flowchart of the general task configuration update method in an embodiment of this disclosure, including step S801-step S809:

[0122] In step S801, start;

[0123] In step S802, obtain the new version of the general task configuration to be launched.

[0124] In step S803, select the pilot tasks by circling.

[0125] In step S804, follow up the second execution success rate.

[0126] In step S805, determine whether the second execution success rate meets the first expected standard.

[0127] If not (does not meet the first expected standard), then enter step S806 to fix the configuration problem and jump back to step S805.

[0128] If so (meets the first expected standard), then enter step S807 to determine whether multiple tasks to be processed are all selected as pilot tasks.

[0129] If not, then expand the gray-scale range and jump back to step S803.

[0130] If so, then enter step S808 to update the old version of the general task configuration with the new version of the general task configuration.

[0131] In step S809, end.

[0132] Based on the above technical solutions, the present disclosure can at least achieve the following technical effects:

[0133] First, the data platform automatically monitors the success rate of the gray-scale configuration execution, eliminating the need for manual real-time monitoring, reducing the consumption of human resources. When the success rate is lower than the expected standard, the data platform will automatically close the relevant pilot tasks, avoiding manual intervention and potential human errors.

[0134] Second, once a problem (such as low success rate) is found in the new version of the general task configuration, immediate measures can be taken, such as closing the task or using the old version of the general task configuration as a fallback, preventing the problem from spreading further. Thus, problematic tasks can be stopped in a timely manner, reducing the risk of large-scale failures or data corruption caused by configuration errors. Also, since problems can be quickly detected and measures taken, large-scale repair work is avoided, reducing the repair cost.

[0135] Third, for the already selected pilot tasks, when the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks does not meet the third expected standard, the old version of the general task configuration will be automatically used to execute the tasks. Thus, even if there are problems with the new version of the general task configuration, the existing business processes can still continue, ensuring that these tasks will not fail or be delayed due to problems with the new version of the general configuration, reducing the impact on daily operations and ensuring business continuity.

[0136] Exemplary Device

[0137] After introducing the general task configuration update method of the exemplary embodiments of the present disclosure, next, reference will be made to Figure 9 describe the general task configuration update device of the exemplary embodiments of the present disclosure.

[0138] Figure 9 FIG. shows a schematic structural diagram of the general task configuration update device in the exemplary embodiment of the present disclosure; the device is disposed in the data platform, the data platform includes a plurality of tasks to be processed, and the data platform further includes an old version of the general task configuration and a new version of the general task configuration to be launched, as Figure 9 shown, the general task configuration update device 900 may include a first task selection module 910, a task execution module 920, a second task selection module 930, and a configuration update module 940. Among them:

[0139] The first task selection module 910 is configured to select the current batch of pilot tasks from the plurality of tasks to be processed;

[0140] The task execution module 920 is configured to determine whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks;

[0141] The second task selection module 930 is configured to select the next batch of pilot tasks when the current batch of pilot tasks is completed and the second execution success rate when the new version of the general task configuration is applied to the current batch of pilot tasks meets the first expected standard;

[0142] The configuration update module 940 is configured to update the old version of the general task configuration with the new version of the general task configuration in response to that all the plurality of tasks to be processed are selected as the pilot tasks and the third execution success rate when the new version of the general task configuration is applied to the last batch of pilot tasks meets the second expected standard.

[0143] In an alternative embodiment, the first task selection module 910 selects the current batch of pilot tasks from the plurality of tasks to be processed, including:

[0144] Obtain the difference degree of the new version of the general task configuration relative to the old version of the general task configuration;

[0145] Find the corresponding relationship between the difference degree and the number of pilot tasks pre-configured according to the difference degree, and determine the number of tasks of the current batch of pilot tasks;

[0146] Select the current batch of pilot tasks from the multiple tasks to be processed based on the number of tasks in combination with the task priorities corresponding to each task to be processed.

[0147] In an optional implementation manner, the first task selection module 910 selects the current batch of pilot tasks from the multiple tasks to be processed based on the number of tasks in combination with the task priorities corresponding to each task to be processed, including:

[0148] Sort the task priorities corresponding to the multiple tasks to be processed in ascending order to form a sorted sequence;

[0149] Starting from the starting position of the sorted sequence, screen out the current batch of pilot tasks that match the number of tasks.

[0150] In an optional implementation manner, after selecting the current batch of pilot tasks from the multiple tasks to be processed, the task execution module 920 is configured to:

[0151] Execute the starting task in the current batch of pilot tasks based on the new version of the general task configuration;

[0152] If the execution fails, re - execute the starting task in the current batch of pilot tasks based on the old version of the general task configuration.

[0153] In an optional implementation manner, the task execution module 920 determines whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks, including:

[0154] When the first execution success rate meets the third expected standard, call the new version of the general task configuration to execute each pilot task;

[0155] When the first execution success rate does not meet the third expected standard, call the old version of the general task configuration to execute each pilot task.

[0156] In an optional implementation manner, the second task selection module 930 is configured to:

[0157] When the current batch of pilot tasks is completed and the second execution success rate when the new version of the general task configuration is applied to the current batch of pilot tasks does not meet the first expected standard, repair the new version of the general task configuration.

[0158] In an optional implementation manner, after repairing the new version of the general task configuration, the second task selection module 930 is configured to:

[0159] Invoke the current batch of pilot tasks again using the repaired new version of the general task configuration, and re-evaluate the second execution success rate.

[0160] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the embodiments of the invention of the above method, and will not be repeated here.

[0161] Exemplary Storage Medium

[0162] The storage medium of the exemplary embodiment of the present disclosure will be described below.

[0163] In this exemplary embodiment, the above method can be implemented by a program product. For example, a portable compact disc read-only memory (CD-ROM) can be used and includes program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited to this. In this document, a readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0164] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0165] The computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium can also be any readable medium other than the readable storage medium, and this readable medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0166] The program code contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the above.

[0167] The program code for performing the operations of the present disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0168] Exemplary Electronic Device

[0169] Reference Figure 10 An electronic device according to an exemplary embodiment of the present disclosure will be described.

[0170] Figure 10 The displayed electronic device 1000 is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0171] As Figure 10 shown, the electronic device 1000 is presented in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processor 1010, at least one memory 1020, a bus 1030 connecting different system components (including the memory 1020 and the processor 1010), and a display unit 1040.

[0172] Among them, the memory stores program code, and the program code can be executed by the processor 1010, so that the processor 1010 executes the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification. For example, the processor 1010 can execute method steps as Figure 2 shown, etc.

[0173] The memory 1020 may include volatile memory, such as a random access memory (RAM) 1021 and / or a cache memory 1022, and may further include a read-only memory (ROM) 1023.

[0174] The memory 1020 may further include a program / utilities 1024 having a set (at least one) of program modules 1025. Such program modules 1025 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0175] The bus 1030 may include a data bus, an address bus, and a control bus.

[0176] The electronic device 1000 may also communicate with one or more external devices 1100 (such as a keyboard, a pointing device, a Bluetooth device, etc.), and such communication may be carried out through the input / output (I / O) interface 1050. The electronic device 1000 further includes a display unit 1040, which is connected to the input / output (I / O) interface 1050 for display. Also, the electronic device 1000 may further communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 1060. As shown in the figure, the network adapter 1060 communicates with other modules of the electronic device 1000 through the bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in combination with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0177] It should be noted that, although several modules or sub-modules of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more of the above-described units / modules may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided and embodied by multiple units / modules.

[0178] In addition, although the operations of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the shown operations must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0179] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of each aspect does not mean that the features in these aspects cannot be combined for benefits. Such division is only for the convenience of description. The present disclosure aims to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A general task configuration update method, characterized in that, Applied to a data platform, where the data platform contains multiple tasks to be processed, and the data platform also contains an old version of the general task configuration and a new version of the general task configuration to be launched. The method includes: Select a current batch of pilot tasks from the multiple tasks to be processed; Determine whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks; When the current batch of pilot tasks is completed and the second execution success rate when the new version of the general task configuration is applied to the current batch of pilot tasks meets the first expected standard, select the next batch of pilot tasks; In response to the third execution success rate when all the multiple tasks to be processed are selected as the pilot tasks and the new version of the general task configuration is applied to the last batch of pilot tasks meeting the second expected standard, update the old version of the general task configuration with the new version of the general task configuration.

2. The method according to claim 1, wherein The step of selecting a current batch of pilot tasks from the multiple tasks to be processed includes: Obtain the difference degree of the new version of the general task configuration relative to the old version of the general task configuration; Find the corresponding relationship between the pre-configured difference degree and the number of pilot tasks according to the difference degree, and determine the number of tasks in the current batch of pilot tasks; Based on the number of tasks and combining the task priorities corresponding to each task to be processed, select the current batch of pilot tasks from the multiple tasks to be processed.

3. The method according to claim 2, wherein The step of selecting the current batch of pilot tasks from the multiple tasks to be processed based on the number of tasks and combining the task priorities corresponding to each task to be processed includes: Sort the task priorities corresponding to the multiple tasks to be processed in ascending order to form a sorted sequence; Starting from the starting position of the sorted sequence, filter out the current batch of pilot tasks that match the number of tasks.

4. The method according to any one of claims 1 to 3, characterized in that, After selecting the current batch of pilot tasks from the multiple tasks to be processed, the method further includes: Execute the starting task in the current batch of pilot tasks based on the new version of the general task configuration; If the execution fails, re-execute the starting task in the current batch of pilot tasks based on the old version of the general task configuration.

5. The method according to claim 1, characterized in that, The step of determining whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks includes: When the first execution success rate meets the third expected standard, call the new version of the general task configuration to execute each pilot task; When the first execution success rate does not meet the third expected standard, call the old version of the general task configuration to execute each pilot task.

6. The method according to claim 1, characterized in that, The method further includes: When the current batch of pilot tasks is completed and the second execution success rate when the new version of the general task configuration is applied to the current batch of pilot tasks does not meet the first expected standard, repair the new version of the general task configuration.

7. The method according to claim 6, characterized in that, After repairing the new version of the general task configuration, the method further includes: Re - execute the current batch of pilot tasks by invoking the repaired new version of the general task configuration, and re - evaluate the second execution success rate.

8. A general task configuration update device, characterized in that, Set in a data platform, the data platform contains multiple tasks to be processed. The data platform also contains an old version of the general task configuration and a new version of the general task configuration to be launched. The device includes: A first task selection module, configured to select a current batch of pilot tasks from the multiple tasks to be processed; A task execution module, configured to determine whether to execute each pilot task in the current batch of pilot tasks based on the new version of the general task configuration according to the first execution success rate when the new version of the general task configuration is applied to the executed tasks in the current batch of pilot tasks; A second task selection module, configured to select the next batch of pilot tasks when the current batch of pilot tasks is completed and the second execution success rate when the new version of the general task configuration is applied to the current batch of pilot tasks meets the first expected standard; A configuration update module, configured to update the old version of the general task configuration with the new version of the general task configuration in response to that all the multiple tasks to be processed are selected as the pilot tasks and the third execution success rate when the new version of the general task configuration is applied to the last batch of pilot tasks meets the second expected standard.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, Including: A processor; And A memory, configured to store the executable instructions of the processor; Wherein, the processor is configured to execute the method according to any one of claims 1 to 7 by executing the executable instructions.

Images