Upgrading method and system of task scheduling system, computer equipment and storage medium

Abstract

The invention provides an upgrading method and system of a task scheduling system, computer equipment and a storage medium, and relates to the technical field of dynamic updation.The system applied to the method comprises a job processing module, a job processing plug-in, a configuration center, a first configuration module and a second configuration module, and the configuration modules are configured and managed by the configuration center. A first configuration module is embedded in a job processing module, a second configuration module is embedded between the job processing module and a job processing plug-in, a configuration center carries out configuration management on the configuration module in combination with a test user set during version upgrading, and the test and upgrading processes among different users are isolated; according to the method, the modules / plug-ins in the scheduling system are tested, the testing and upgrading processes of the different modules / plug-ins are isolated, the upgrading of the modules / plug-ins in the scheduling system is tested in a gray level testing mode, the testing processes of the multiple modules / plug-ins are decoupled through dynamic configuration isolation, and problems can be rapidly positioned when the problems occur in the testing result.

Description

Technical Field

[0001] The present application relates to the field of dynamic update technology, and in particular to an upgrade method, system, computer device and storage medium of a task scheduling system. Background Art

[0002] The task scheduling system in the public cloud platform is used to implement the dispatch, execution and management of data jobs. These systems use a message-driven approach to achieve high flexibility and scalability in the design of distributed system modules. However, for such large-scale scheduling systems that rely on message queues, upgrading in the process of introducing new features, fixing bugs or optimizing performance becomes a challenge, because it is required to consider the massive message-driven sub-modules of different versions stored in the message queue to implement functional verification of the grayscale testing process without interrupting key business processes, while ensuring that the upgrade process does not affect the normal use of users. Therefore, it is necessary to propose a new upgrade method for scheduling systems. Summary of the invention

[0003] The present application provides a task scheduling system upgrade method, system, computer equipment and storage medium, which are used to solve the technical problems existing in the upgrade and testing process of the prior art that affect the normal use of the system.

[0004] In a first aspect, the present application provides an upgrade method for a task scheduling system, the method being applied to an upgrade system of the task scheduling system, the upgrade system of the task scheduling system comprising a configuration center and a task scheduling system, the task scheduling system comprising a job processing module and a job processing plug-in, the job processing module being used to manage, dispatch and execute job request messages, a first configuration module being embedded in the job processing module, the job processing plug-in being used to perform data processing based on the job request message; a second configuration module being embedded between the job processing module and the job processing plug-in; the first configuration module and the second configuration module being configured and managed by the configuration center; the method comprising:

[0005] Obtaining a first current version and a first updated version of a job processing module, and a second current version and a second updated version of a job processing plug-in;

[0006] According to the user characteristics of each user managed by the task scheduling system, all users are divided into a test user set and a non-test user set;

[0007] Generate a configuration result of the first configuration module and a configuration result of the second configuration module according to the test user set, the non-test user set, the first current version, the first updated version, the second current version, and the second updated version;

[0008] According to the configuration result of the first configuration module and the configuration result of the second configuration module, the task scheduling system is run, and the first updated version and the second updated version are tested based on the test user set to obtain the test result.

[0009] In one embodiment, the user characteristics include the business impact of the data processing job, the acceptance of job failure, and whether delayed reprocessing can be accepted; accordingly, according to the user characteristics of each user managed by the task scheduling system, all users are divided into a test user set and a non-test user set, including:

[0010] Define users whose business impact on data processing jobs is lower than a preset impact, whose acceptance of job failure is lower than a preset acceptance, and who can accept delayed reprocessing as test users, and construct a test user set;

[0011] Define the remaining users as non-test users and construct a non-test user set.

[0012] In one embodiment, generating a configuration result of a first configuration module and a configuration result of a second configuration module according to a test user set, a non-test user set, a first current version, a first updated version, a second current version, and a second updated version includes:

[0013] Determine an isolation configuration of the job processing module according to the test user set, the non-test user set, the first current version, and the first updated version;

[0014] Determine a plug-in version information set configuration of the job processing plug-in according to the second current version and the second updated version;

[0015] According to the isolation configuration and the plug-in version information set configuration, the association configuration between the version of the job processing module and the version of the job processing plug-in is determined.

[0016] In one embodiment, determining the isolation configuration of the job processing module according to the test user set, the non-test user set, the first current version, and the first updated version includes:

[0017] The test user set is set as the isolation configuration of the first current version of the job processing module, and the non-test user set is set as the isolation configuration of the first updated version of the job processing module.

[0018] In one embodiment, there are multiple job processing plug-ins; accordingly, determining the plug-in version information set configuration of the job processing plug-in according to the second current version and the second updated version includes:

[0019] For any job processing plug-in, the plug-in version information set configuration of any job processing plug-in is determined according to the second updated version of any job processing plug-in and the second current versions of the remaining job processing plug-ins.

[0020] In one embodiment, the task scheduling system is run according to the first configuration result, and the first updated version and the second updated version are tested based on the test user set to obtain the test results, including:

[0021] The job processing module pulls the job request message and identifies the user type corresponding to the job request message. The user type is used to indicate whether the user belongs to the test user set.

[0022] Acquire the configuration result of the first configuration module and the configuration result of the second configuration module from the configuration center;

[0023] Determine, according to the user type, the configuration result of the first configuration module and the configuration result of the second configuration module, the version information of the job processing module and the version information of the job processing plug-in corresponding to the job request message;

[0024] According to the version information of the job processing module and the version information of the job processing plug-in corresponding to the job request message, the corresponding job processing module and the job processing plug-in are called to process the job request message, and then the job processing result is obtained.

[0025] In one embodiment, the method further comprises:

[0026] When the job processing result meets the preset conditions, the configuration results of the first configuration module and the configuration results of the second configuration module are modified, and all users in the test user set and the non-test user set correspond to the first updated version and the second updated version.

[0027] In a second aspect, the present application further provides an upgrade system for a task scheduling system, characterized in that the upgrade system for the task scheduling system includes a configuration center and a task scheduling system, the task scheduling system includes a job processing module and a job processing plug-in, the job processing module is used to manage, dispatch and execute job request messages, a first configuration module is embedded in the job processing module, and the job processing plug-in is used to perform data processing based on the job request message; a second configuration module is embedded between the job processing module and the job processing plug-in; the first configuration module and the second configuration module are configured and managed by the configuration center;

[0028] The upgrade system of the task scheduling system executes any of the above method steps.

[0029] In a third aspect, the present application further provides a computer device, the computer comprising: a processor, and a memory communicatively connected to the processor;

[0030] Memory stores computer-executable instructions;

[0031] The above method is implemented when the processor executes the computer-executable instructions stored in the memory.

[0032] In a fourth aspect, the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are used to implement the above method when executed by a processor.

[0033] In a fifth aspect, the present application also provides a computer program product, including computer execution instructions, which implement the above method when executed by a processor.

[0034] The present application provides an upgrade method, system, computer device and storage medium for a task scheduling system, wherein the system applied by the method includes a job processing module, a job processing plug-in, a configuration center, a first configuration module and a second configuration module, wherein the first configuration module and the second configuration module are configured and managed by the configuration center, by embedding the first configuration module in the job processing module and embedding the second configuration module between the job processing module and the job processing plug-in, the configuration center configures and manages the first configuration module and the second configuration module in combination with a set of test users during version upgrade, isolates the test and upgrade processes between different users, isolates the test and upgrade processes of different modules / plug-ins, tests the upgrades of modules / plug-ins in the scheduling system in a grayscale test manner, and decouples the test processes of multiple modules / plug-ins through dynamic configuration isolation of the first configuration module and the second configuration module, so that problems can be quickly located when there are problems with the test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] FIG1 is a schematic diagram of the structure of an upgrade system of a task scheduling system provided in one embodiment;

[0037] FIG2 is a flowchart 1 of a method for upgrading a task scheduling system provided in one embodiment;

[0038] FIG3 is a flowchart of a third method for upgrading a task scheduling system provided in one embodiment;

[0039] FIG4 is a fourth flowchart of a method for upgrading a task scheduling system provided in one embodiment;

[0040] FIG5 is a schematic diagram of the structure of an upgrade system of a task scheduling system provided in another embodiment;

[0041] FIG6 is a schematic diagram of a structural change of an upgrade system of a task scheduling system in an update version testing phase provided in one embodiment;

[0042] 7 is a schematic diagram of a structural change of an upgrade system of a task scheduling system in an update version switching operation phase provided in one embodiment;

[0043] FIG8 is a diagram showing the internal structure of a computer device provided in one embodiment.

[0044] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0045] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0046] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0047] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of financial data or user data and other information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals. It should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, models, etc. may be mentioned, and they should be considered as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0048] The task scheduling system in the public cloud platform is used to implement the dispatch, execution and management of data jobs. These systems use a message-driven approach to achieve high flexibility and scalability in the design of distributed system modules. However, for such large-scale scheduling systems that rely on message queues, upgrading in the process of introducing new features, fixing bugs or optimizing performance becomes a challenge, because it is required to consider the massive message-driven sub-modules of different versions stored in the message queue to implement functional verification of the grayscale testing process without interrupting key business processes, while ensuring that the upgrade process does not affect the normal use of users.

[0049] The method provided in the embodiment of the present application can be applied to the upgrade system of the task scheduling system as shown in Figure 1. The upgrade system of the task scheduling system includes a configuration center 100 and a task scheduling system 200. The task scheduling system 200 includes a job processing module 201 and a job processing plug-in 202. The job processing module 201 is used to manage, dispatch and execute job request messages, and the job processing plug-in 202 is used to process data based on job request messages; a first configuration module 203 is embedded in the job processing module 201, and a second configuration module 204 is embedded between the job processing module 201 and the job processing plug-in 202; the first configuration module 203 and the second configuration module 204 are configured and managed by the configuration center 100.

[0050] When the above-mentioned task scheduling system 200 does not need to be upgraded, the normal operation process is: the user submits job information from the front-end page of the data platform (it can also be the front-end page of the task scheduling system, here referring to the producer who sends the job message), and the front-end initiates the job message to the message queue 300; the job processing module 201 pulls the job request message from the message queue job 300, and the job processing module 201 first calls the first configuration module 203 when pulling the job request message. At this time, the first configuration module 203 will pull the configuration result in the configuration center 100, and the job processing module 201 processes the job request message based on the obtained configuration result, identifies the job processing plug-in type and version corresponding to the job request message, and then obtains the corresponding configuration result in the configuration center based on the second configuration module 204, and calls the corresponding job processing plug-in 202 for data processing in combination with the configuration result of the second configuration module 204.

[0051] For the above task scheduling system, in one embodiment, as shown in FIG2 , a method for upgrading the task scheduling system is provided, comprising the following steps:

[0052] Step 202, obtaining a first current version and a first updated version of the job processing module, and a second current version and a second updated version of the job processing plug-in;

[0053] First of all, it should be noted that upgrading the task scheduling system can be upgrading all modules / plug-ins of the entire task scheduling system, or it can refer to upgrading one or several modules / plug-ins. The version before the task scheduling system is upgraded is defined as the current version, which can be expressed as version V1, and the version of the task scheduling system that needs to be upgraded is defined as the updated version, which is expressed as version V2. It can be seen that the current and updated are relative. When the version of the task scheduling system is version V2, version V3 is an updated version relative to version V2.

[0054] The job processing module and the job processing plug-in are two different modules, so the first current version and the second current version, the first updated version and the second updated version refer to completely different things. Normally, the task scheduling system will divide the functions according to the specific processing type of the job information, and each function corresponds to a submodule. For example, in a task scheduling system, the job processing module is divided into a management submodule, a dispatching submodule and an execution submodule according to the function. The job processing plug-in will also have multiple types according to the job requirements, for example, the job processing plug-in includes Spark, MPP, and Flink. Therefore, the version of the job processing module / job processing plug-in can be based on the function for a specific submodule / plug-in version without the need for the entire job processing module and all job processing plug-ins. That is, the first current version and the first updated version of the job processing module mentioned in this step, and the number of the second current version and the second updated version of the job processing plug-in are determined according to the specific structure of the task scheduling system and the number of modules / plug-ins that need to be updated.

[0055] For example, in a task scheduling system, the job processing module is divided into a management submodule, a dispatch submodule, and an execution submodule according to its functions. Currently, the dispatch submodule and the execution submodule need to be upgraded. The first current version of the job processing module includes the V1 version of the dispatch submodule and the V1 version of the execution submodule, and the first updated version includes the V2 version of the dispatch submodule and the V2 version of the execution submodule. The job management submodule does not need to be updated, so its first updated version is empty.

[0056] Step 204, dividing all users into a test user set and a non-test user set according to the user characteristics of each user managed by the task scheduling system;

[0057] As the job scheduling system of the public cloud platform, the task scheduling system needs to schedule and manage any user of the cloud platform after triggering a job request message. Usually, the cloud platform has a large number of users. When performing upgrade tests on the task scheduling system, due to the uncertainty of the test results, if all users are upgraded during use, when the test fails, it will have a huge impact on the use of some users. Therefore, the embodiment of the present application performs upgrade tests in a grayscale test manner.

[0058] The users managed by the task scheduling system are all users of the public cloud platform. User characteristics are used to describe the user's requirements for the job processing and processing results. User characteristics can be set by human judgment, or they can be determined by analyzing the processing process and results of job request messages triggered by users in the past. In the embodiment of the present application, all users are divided into a test user set and a non-test user set based on the user characteristics of each user. In order to ensure the validity of the test results, the division can also be combined with the number of test users. It should be noted that, under normal circumstances, the number of test users in the test user set will be less than the number of non-test users in the non-test user set.

[0059] The test users in the test user set are used to test the updated version. The job request messages triggered by the test users are processed using the updated version of the job processing module and job processing plug-in; and the job request messages triggered by non-test users in the non-test user set are also processed using the current version of the job processing module and job processing plug-in.

[0060] Step 206, generating a configuration result of the first configuration module and a configuration result of the second configuration module according to the test user set, the non-test user set, the first current version, the first updated version, the second current version, and the second updated version;

[0061] The first configuration module is embedded in the job processing module and is used to isolate the message queue and the job processing module. Since the job request message triggered by the test user in the test user set is processed using the updated version of the job processing module and the job processing plug-in; and the job request message triggered by the non-test user in the non-test user set is still processed using the current version of the job processing module and the job processing plug-in, the job processing module needs to call the first configuration module first when pulling the job request message, and determine the processing version of the job request message based on the configuration result of the first configuration module in the configuration center.

[0062] It can be seen that the configuration results of the first configuration module can explain the relationship between test users, non-test users, the first current version and the second current version. Through the configuration results of the first configuration module, the consumers of the message queue corresponding to the first current version and the consumers of the message queue corresponding to the first updated version are determined.

[0063] The second configuration module is embedded between the job processing module and the job processing plug-in, and the consumer of the message queue produced by the job processing module is determined by the configuration result of the second configuration module. Therefore, it is necessary to generate the configuration result of the second configuration module according to the test user set, the non-test user set, the second current version and the second updated version.

[0064] It should be noted that when the job processing module has multiple functional sub-modules that need to be updated, that is, it has multiple first updated versions and a corresponding second updated version of the job processing plug-in, the configuration results of the first configuration module and the configuration results of the second configuration module in the embodiment of the present application will include a separately updated test configuration corresponding to each first updated version / second updated version, so as to perform separate tests on each first updated version / second updated version.

[0065] Step 208: Run the task scheduling system according to the configuration result of the first configuration module and the configuration result of the second configuration module, and test the first updated version and the second updated version based on the test user set to obtain the test result.

[0066] After determining the configuration results of the first configuration module and the configuration results of the second configuration module, the consumer of the message queue can be determined by calling the configuration results of the first configuration module and the configuration results of the second configuration module. At this time, the task scheduling system is run, and the job request message triggered by the non-test user in the message queue calls the current version of the job processing module and the job processing plug-in for data processing, while the job request message triggered by the test user calls the updated version of the job processing module and the job processing plug-in for data processing. By analyzing the job processing results of the job request message triggered by the test user, the test results of the first updated version and the second updated version can be obtained. Among them, when there are multiple first updated versions / second updated versions, the test results include the test results when each first updated version / second updated version is updated separately.

[0067] In the method provided in the above embodiment, the applied system includes a job processing module, a job processing plug-in, a configuration center, a first configuration module and a second configuration module. The first configuration module and the second configuration module are configured and managed by the configuration center. By embedding the first configuration module in the job processing module and embedding the second configuration module between the job processing module and the job processing plug-in, the configuration center configures and manages the first configuration module and the second configuration module in combination with the test user set during version upgrade, isolates the test and upgrade processes between different users, and isolates the test and upgrade processes of different modules / plug-ins. The upgrade of modules / plug-ins in the scheduling system is tested in a grayscale test manner, and through the dynamic configuration isolation of the first configuration module and the second configuration module, the test processes of multiple modules / plug-ins are decoupled, so that problems can be quickly located when problems occur in the test results.

[0068] In one embodiment, the user characteristics include the business impact of the data processing job, the acceptance of job failure, and whether delayed reprocessing can be accepted; accordingly, according to the user characteristics of each user managed by the task scheduling system, all users are divided into a test user set and a non-test user set, including:

[0069] Define users whose business impact on data processing jobs is lower than a preset impact, whose acceptance of job failure is lower than a preset acceptance, and who can accept delayed reprocessing as test users, and construct a test user set;

[0070] Define the remaining users as non-test users and construct a non-test user set.

[0071] The business impact of data processing jobs is used to indicate the impact of data processing on the user's business using the cloud platform. The impact of data processing on the business is mainly reflected in improving the effectiveness of business decisions, improving operational efficiency and reducing costs, increasing online sales and customer groups, etc. Therefore, the business impact of the user's data processing jobs can be determined from the above aspects. The acceptance of job failure is used to measure the user's acceptance of job failure; whether delayed reprocessing can be accepted is used to measure the user's requirements for job processing efficiency.

[0072] In this embodiment, users whose characteristics are that the business impact of data processing jobs is lower than the preset impact level, the acceptance level of job failure is lower than the preset acceptance level, and the users who can accept delayed reprocessing are defined as test users to construct a test user set; the remaining users are defined as non-test users to construct a non-test user set.

[0073] In the method provided in the above embodiment, user characteristics are judged by the business impact of the data processing job, the acceptance of job failure and whether delayed reprocessing can be accepted, and test users are screened out so that the upgrade of the task calling system can be tested based on the test users.

[0074] In one embodiment, as shown in FIG3 , generating a configuration result of a first configuration module and a configuration result of a second configuration module according to a test user set, a non-test user set, a first current version, a first updated version, a second current version, and a second updated version includes:

[0075] Step 302, determining an isolation configuration of the job processing module according to the test user set, the non-test user set, the first current version, and the first updated version;

[0076] Step 304, determining a plug-in version information set configuration of the job processing plug-in according to the second current version and the second updated version;

[0077] Step 306: Determine the association configuration between the version of the job processing module and the version of the job processing plug-in according to the isolation configuration and the plug-in version information set configuration.

[0078] Isolation configuration is used to isolate versions from users. In the task scheduling system, when the job processing module consumes the message queue produced by the upstream, the first configuration module will rebuild the consumer of the upstream message queue through the isolation configuration and remove the filter to consume all messages. The isolation configuration mainly stores the relationship between versions and users. In the isolation configuration, for one version, it can record the user that needs to be isolated, or it can record the user that needs to be isolated.

[0079] For example, in one embodiment, the first current version of the job processing module is V1, the first updated version is V2, the non-test user set is A, and the test user set is B; then, the first current version V1 needs to isolate the test user set B, and the first updated version V2 needs to isolate the non-test user set A. The test user set is set as the isolation configuration of the first current version of the job processing module, and the non-test user set is set as the isolation configuration of the first updated version of the job processing module. Then, the isolation configuration of the first configuration module is:

[0080] {V1: {B}, V2: {A}}.

[0081] The plug-in version information set configuration is used to configure the version of the job processing plug-in. Since there are multiple job processing plug-ins, when multiple job processing plug-ins need to be updated, by configuring the plug-in version information set configuration, the test of each job processing plug-in being updated separately is implemented. Specifically, for any job processing plug-in, the plug-in version information set configuration of any job processing plug-in is determined according to the second updated version of any job processing plug-in and the second current versions of the remaining job processing plug-ins.

[0082] For example, the job processing plug-ins include Spark, Flink-v3, Mpp-v2, and Spark SQL. When only Spark and Flink need to be updated, Spark is upgraded from v1 to v2, and Flink is upgraded from v3 to v4. Before the update, the plug-in version information set is configured as {Spark-v1, Flink-v1, Mpp-v2, SparkSQL-v3}, which is defined as SV1 version. In the testing phase, two new plug-in version information set configurations SV2 are added: {Spark-v2, Flink-v3, Mpp-v2, SparkSQL-v3} and SV3: {Spark-v1, Flink-v4, Mpp-v2, SparkSQL-v3}. Finally, the plugin version information set is configured as {SV1: {Spark-v1, Flink-v3, Mpp-v2, SparkSQL-v3}, SV2: {Spark-v2, Flink-v3, Mpp-v2, SparkSQL-v3}, SV3: {Spark-v1, Flink-v4, Mpp-v2, SparkSQL-v3}}.

[0083] In the above process, several job processing plug-ins were updated to shield each other from influence, and the plug-in version information set configuration includes several plug-in version information sets. Only in this way, during the grayscale test, the upstream and downstream tests of the data processing link will not affect each other's test results because the upstream and downstream links are changed at the same time.

[0084] In addition, the configuration center also needs to add an association configuration between the version of the job processing module and the version of the job processing plug-in. For example, {V2_Spark: SV2, V2_Flink: SV3} indicates that when the job processing module V2 version processes messages, if Spark needs to be tested, the corresponding plug-in version information set is configured as SV2. In the above implementation, the association configuration is: {V1_Spark: SV1, V1_Flink: SV1, V2_Spark: SV2, V2_Flink: SV3}, which is the association configuration when testing through the test users in the test user set. After the grayscale test is completed, the association configuration should be modified to {V2_Spark: SV4, V2_Flink: SV4}, where SV4: {Spark-v2, Flink-v4, Mpp-v2, SparkSQL-v3}}.

[0085] In the method provided in the above embodiment, the configuration is modified in real time according to the updated version and the scope of the test users in combination with dynamic configuration, and the update of each module / plug-in is tested according to the dynamic configuration, thereby achieving decoupling of the test processes between different modules / plug-ins, and being able to quickly locate the problem when an error occurs in the test result of one module / plug-in.

[0086] In one embodiment, as shown in FIG. 4 , the task scheduling system is run according to the first configuration result, and the first updated version and the second updated version are tested based on the test user set to obtain test results, including:

[0087] Step 402, the job processing module pulls the job request message and identifies the user type corresponding to the job request message, where the user type is used to indicate whether the user belongs to the test user set;

[0088] Step 404, obtaining the configuration result of the first configuration module and the configuration result of the second configuration module from the configuration center;

[0089] Step 406, determining the version information of the job processing module and the version information of the job processing plug-in corresponding to the job request message according to the user type, the configuration result of the first configuration module and the configuration result of the second configuration module;

[0090] Step 408, according to the version information of the job processing module and the version information of the job processing plug-in corresponding to the job request message, call the corresponding job processing module and the job processing plug-in to process the job request message, and obtain the job processing result.

[0091] The user type is used to determine whether the user who triggered the job request message is a test user. If it is a test user, the downstream consumers of the job request message are reconstructed according to the configuration results of the configuration center, and the job request message is isolated from the first current version and the second current version. If it is a non-test user, the job request message is processed by the first current version and the second current version.

[0092] In order to facilitate understanding of the embodiments of the present application, the following examples are used as the basis for explanation.

[0093] As shown in FIG5 , an upgrade system for a task scheduling system is provided, the system includes a Nacos configuration center 100 and a task scheduling system 200, the task scheduling system 200 includes a job processing module 201, a job execution plug-in 202, a first configuration module 203 and a second configuration module 204, wherein the job processing module 201 includes a job management submodule 2011, a job dispatching submodule 2012, and a job execution submodule 2013, and the first configuration module 203 includes a first configuration submodule 2031, a second configuration submodule 2032, and a third configuration submodule 2033.

[0094] The non-test user set is A, and the test user set is B. Upgrade requirements: The first current version of the job processing module is V1, and the first updated version is V2; the job processing plug-ins include {Spark-v1, Flink-v1, Mpp-v2, SparkSQL-v3}, among which Spark is upgraded from v1 to v2, and Flink is upgraded from v3 to v4.

[0095] During the test, the isolation configuration in the Nacos configuration center 100 is: {V1: {B}, V2: {A}}, indicating that the first current version V1 needs to isolate the relevant job information of the test user B, and the first updated version V2 needs to isolate the relevant job information of the non-test user A.

[0096] At the same time, when the data processing plug-ins to be changed are only Spark and Flink, after the plug-in deployment is completed, it is necessary to first configure the plug-in version information set from the outside to the Nacos configuration center:

[0097] {SV2: {Spark-v2, Flink-v3, Mpp-v2, SparkSQL-v3}, SV3: {Spark-v1, Flink-v4, Mpp-v2, SparkSQL-v3}, SV4: {Spark-v2, Flink-v4, Mpp-v2, SparkSQL-v3}};

[0098] The final configuration corresponds to {SV1: {Spark-v1, Flink-v3, Mpp-v2, SparkSQL-v3}, SV2: {Spark-v2, Flink-v3, Mpp-v2, SparkSQL-v3}, SV3: {Spark-v1, Flink-v4, Mpp-v2, SparkSQL-v3}};

[0099] Nacos configuration center also needs to add associated configurations, such as {V2_Spark: SV2, V2_Flink: SV3}. Finally, the associated configuration between the version of the job processing module and the version of the job processing plug-in is {V1_Spark: SV1, V1_Flink: SV1, V2_Spark: SV2, V2_Flink: SV3}. After the grayscale test is completed, it will correspond to {V2_Spark: SV4, V2_Flink: SV4}.

[0100] The operation of the upgrade system of the task scheduling system shown in FIG5 includes three stages: a current version operation stage, an updated version test stage, and an updated version switching operation stage, wherein the current version operation stage includes:

[0101] S1: The user submits job information from the front-end page of the data platform. The front-end initiates a job message to the message queue, and then the job management submodule 2011 pulls the job request information for consumption. The subject of the message queue is the job request, which corresponds to the message queue in the upper left corner of the job management submodule 2011.

[0102] S2: When pulling messages, the job management submodule 2011 will first call the first configuration submodule 2031. At this time, the first configuration submodule 2031 will pull the isolation configuration of the Nacos configuration center 100, check that the isolation configuration is empty, and the first configuration submodule 2031 will rebuild the consumer and remove the filter to consume all messages. At the same time, the job management submodule 2011 will report service information to the Nacos configuration center 100. The service information includes heartbeat time, service version, service description information, etc.;

[0103] S3: the job management submodule 2011 will parse the job information and store the job instance information in the message queue of the job dispatching submodule 2012, which will be consumed by the downstream job dispatching submodule 2012;

[0104] S4: The job dispatching submodule 2012 will similarly call the second configuration submodule 2032 first, and will also check that the isolation configuration is empty. The second configuration submodule 2032 will rebuild the consumer and remove the filter to consume all messages; at the same time, the job dispatching submodule 2012 will report service information to the Nacos configuration center 100, and the service information includes heartbeat time, service version, service description information, etc.;

[0105] S5: The job dispatching submodule 2012 compares the priority and job type of all currently received job information to check whether the job running resources are sufficient. If sufficient, the job instance information is stored in the job execution message queue to be consumed by the job execution submodule 2013;

[0106] S6: Similarly, the job execution submodule 2013 will first call the third configuration submodule 2033, and will also check that the isolation configuration is empty. The third configuration submodule 2033 will rebuild the consumer and remove the filter to consume all messages; at the same time, the job execution submodule 2013 will report service information to the Nacos configuration center 100, and the service information includes heartbeat time, service version, service description information, etc.;

[0107] S7: The job execution submodule 2013 first parses the data processing logic in the job instance information, identifies the version of the corresponding job processing plug-in, and then calls the second configuration module 204 to obtain the plug-in version information set configuration and associated configuration of the Nacos configuration center 100, and obtains the corresponding plug-in version information; since the current version is in the running stage, there is only one plug-in corresponding to the Nacos configuration center 100, so the current version of the job processing plug-in program is directly called to perform data processing;

[0108] S8: The job execution submodule 2013 will wait for the job processing plug-in to complete the processing, and then write the job processing result into the message queue, which will be consumed by the job management submodule 2011; the subject of this message queue is the job processing result, which corresponds to the message queue in the upper right corner of the job management submodule 2011 in FIG5 ;

[0109] S9: Similarly, the job management submodule 2011 will first call the first configuration submodule 2031, and will also check that the isolation configuration is empty. The first configuration submodule 2031 will rebuild the consumer and remove the filter to consume all messages;

[0110] S10: The job management submodule 2011 will parse the job processing results and check the job frequency information. If the job needs to be turned over to the next business date, it will do the corresponding processing and continue to put the job instance information into the message queue, and then go to step S4; otherwise, it will not continue processing and put the job instance information into the persistent storage corresponding to the historical information.

[0111] In this corresponding stage, the function of the first configuration module is to obtain the isolation configuration and rebuild the consumer client of the message queue corresponding to the module, and the function of the second configuration module is to obtain the plug-in version information and return the correct plug-in version information.

[0112] It should be made clear that the first configuration module and the second configuration module are collectively referred to as dynamic configuration modules. The dynamic configuration module does not directly consume or directly drive the downstream modules, but is consumed and driven by the core modules of the task scheduling system. The dynamic configuration module mainly encapsulates the relevant logic to ensure that the logic related to the Nacos configuration center, grayscale upgrade, and scheduling system messages involved in the upgrade process does not intrude into the relevant implementation of the task scheduling system. This application decouples this.

[0113] At the service change point, the job request messages of non-test users are run normally by the current version, and the job request messages of test users are run by the updated version to implement grayscale testing, and the architecture diagram is shown in Figure 6. In the updated version testing phase, the process of the above steps S4-S8 is changed, and the job dispatching submodule 2012 and the job execution submodule 2013 are set to have new logics L1 and L2 respectively, then the above steps specifically include:

[0114] S4': Both V1 and V2 of the job dispatch submodule 2012 will report service information to the Nacos configuration center 100. The service information includes heartbeat time, service version, service description information, etc. When the second configuration submodule 2032 of the upstream module monitors that there are multiple downstream modules, it will add an additional version identifier to the message. At the same time, V1 of the job dispatch submodule 2012 will first call the second configuration submodule 2032 to obtain the isolation configuration from the Nacos configuration center 100 in real time, and check that the version V1 in the isolation configuration corresponds to user B. Then, the consumer of the V1 version of the job dispatch submodule 2012 is rebuilt and the message that the user is not B and the version is V1 is set to filter the V1 version so that the V1 version only consumes the message produced by the upstream of user A version V1; the same is true for V2 of the job dispatch submodule 2012. The second configuration submodule 2032 will rebuild the V2 version of the consumer and set the filter user is not A and the version is V2.

[0115] S5': the job dispatching submodule 2012 compares the priority and job type of all currently received job information. Compared with the V1 version of the job dispatching submodule 2012, the V2 version will additionally execute the new logic L1, and then check whether the job running resources are sufficient. If sufficient, the job instance information will be stored in the job execution message queue, and consumed by the job execution submodule 2013;

[0116] S6': V1 and V2 of the job execution submodule 2013 will report service information to the Nacos configuration center 100. The service information includes heartbeat time, service version, service description information, etc. When the third configuration submodule 2033 of the upstream module monitors that there are multiple downstream modules, it will add an additional version identifier to the message; at the same time, V1 of the job execution submodule 2013 will first call the third configuration submodule 2033, obtain the isolation configuration from the Nacos configuration center 100 in real time, check that the version V1 in the isolation configuration corresponds to user B, and then rebuild the consumer of the V1 version of the job execution submodule 2013 and set to filter the message whose user is not B and whose version is V1 so that the V1 version only consumes the message of user A; the same is true for V2 of the job execution submodule 2013. The third configuration submodule 2033 will rebuild the consumer of the V2 version and set to filter the message whose user is not A and whose version is V2;

[0117] S7': The job execution submodule 2013 first parses the data processing logic in the job instance information. Compared with the V1 version of the job execution submodule 2013, the V2 version will additionally execute the new logic L2 and identify the version of the corresponding job processing plug-in. Then, the second configuration module 204 is called to obtain the plug-in version information set configuration and associated configuration of the Nacos configuration center 100, and obtain the corresponding plug-in version information. For example, if it is a Spark plug-in, then the V1 of the job execution submodule 2013 corresponds to SV1, and the corresponding Spark plug-in version called is v1; and the V2 of the job execution submodule 2013 corresponds to SV2, and the corresponding Spark plug-in version is v2; if the Spark plug-in has completed a separate grayscale test during the test process, the configuration is modified to change the Spark plug-in of the V2 of the job execution submodule 2013 to SV4 for joint testing;

[0118] S8': the job execution submodule 2013 will wait for the job processing plug-in to complete the processing, and then write the job processing result into the message queue, which will be consumed by the job management submodule 2011.

[0119] After completing the above test process, determine whether the test results meet expectations. If the job processing results meet the preset conditions, modify the configuration results of the first configuration module and the configuration results of the second configuration module, and make all users in the test user set and the non-test user set correspond to the first update version and the second update version. Specifically, in conjunction with the above example, the structure diagram of the update version switching operation stage is shown in Figure 7, and steps S4H and S6 are different, specifically:

[0120] S4'': V2 of the job dispatch submodule 2012 will report service information to the Nacos configuration center 100, while V1 will no longer report. The service information includes heartbeat time, service version, service description information, etc. At the same time, V1 of the job dispatch submodule 2012 will first call the second configuration submodule 2032, obtain the isolation configuration from the Nacos configuration center 100 in real time, check that the version V1 in the isolation configuration corresponds to users A and B, and then rebuild the consumers of the V1 version of the job dispatch submodule 2012 and set to filter messages whose users are not A and B, which directly leads to the V1 version no longer consuming messages; the same is true for V2 of the job dispatch submodule 2012. The second configuration submodule 2032 will rebuild the consumers of the V2 version and set to filter messages whose users are not A.

[0121] S6'': V2 of the job execution submodule 2013 will report service information to the Nacos configuration center 100, while V1 will no longer report. The service information includes heartbeat time, service version, service description information, etc.; at the same time, V1 of the job execution submodule 2013 will first call the third configuration submodule 2033, obtain the isolation configuration from the Nacos configuration center 100 in real time, and check that the version V1 in the isolation configuration corresponds to users A and B, and the users corresponding to the message are all A or B, so it is equivalent to version V1 not processing messages, while V2 of the job execution submodule 2013 processes all messages.

[0122] When all current versions no longer process messages, especially when it is found that no jobs are running version V1, the V1 versions of the job execution submodule and the job dispatch submodule can be directly closed.

[0123] In the upgrade system and method of the task scheduling system provided in the above embodiments, the first problem is to determine the scope of the upgrade. According to the characteristics of the task scheduling system that supports multiple users and multiple projects, the scope of the grayscale upgrade is determined, which can reduce the scope of impact of the upgrade failure. The general practice is to limit the scope to users or projects that have no business impact, which avoids the impact of the upgrade on business interruption; the second problem is to ensure that the current old version system processes the current old version job. According to the grayscale upgrade combined with grayscale testing, the functions of the new version are only verified for users or projects that have no business impact, and the data processing jobs outside the upgrade scope are still processed by the current old version during the upgrade process, which also avoids the interruption of the upgrade, resulting in the problem that the data processing jobs with business impact are not triggered at the predefined time point; the third problem is the matching of upstream and downstream module versions. With the help of the characteristics of dynamic configuration, this makes the current Each module of the old version of the scheduling system can parse the message content of the message communication in real time, match the version configuration in the message with the isolation configuration dynamically issued in real time, so as to determine the version of the downstream module or plug-in that the message will drive. This also ensures that the current old version of the service will not process the new version of the job during operation, and makes the versions of upstream and downstream services consistent and will not cross, which greatly reduces the probability of problems due to incompatibility between the old and new versions; finally, there is the problem of real-time configuration taking effect. The present application embeds a dynamic configuration module in the task scheduling system. This module can dynamically switch the processing logic according to the isolation configuration issued by the dynamic configuration center in real time. This can support the new version to gradually expand the upgrade scope to all users and projects, and the processing scope of the current old version is simultaneously reduced to no processing. Gradual switching according to test results can help achieve non-stop upgrades on the user side.

[0124] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0125] Based on the same inventive concept, an embodiment of the present application also provides an upgrade device for a task scheduling system. The implementation solution for solving the problem provided by the device is similar to the implementation solution recorded in the above method. Therefore, the specific limitations in the upgrade device embodiments of one or more task scheduling systems provided below can be found in the above limitations on the upgrade method of the task scheduling system, and will not be repeated here.

[0126] As shown in FIG. 8 , an embodiment of the present application provides a computer device 800 , which includes a memory 801 and a processor 802 .

[0127] The memory 801 is used to store computer-executable instructions executable by the processor;

[0128] The processor 802 implements each step of the method in the above embodiment when executing the computer execution instruction. For details, please refer to the relevant description in the above method embodiment.

[0129] Optionally, the memory 801 can be independent or integrated with the processor 802. When the memory 801 is independently provided, the computer device further includes a bus for connecting the memory 801 and the processor 802.

[0130] The embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, each step of the method in the above embodiment is implemented.

[0131] The embodiment of the present application also provides a computer program product, including computer executable instructions, which implement the various steps of the method in the above embodiment when executed by a processor.

[0132] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer execution instructions in the computer program are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0133] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for upgrading a task scheduling system, characterized in that: The method is applied to an upgrade system of a task scheduling system, wherein the upgrade system of the task scheduling system includes a configuration center and a task scheduling system, wherein the task scheduling system includes a job processing module and a job processing plug-in, wherein the job processing module is used to manage, dispatch and execute job request messages, wherein a first configuration module is embedded in the job processing module, and the job processing plug-in is used to perform data processing based on the job request message; and a second configuration module is embedded between the job processing module and the job processing plug-in; The first configuration module and the second configuration module are configured and managed by the configuration center; the method includes: Obtaining a first current version and a first updated version of the job processing module, and a second current version and a second updated version of the job processing plug-in; According to the user characteristics of each user managed by the task scheduling system, all users are divided into a test user set and a non-test user set; Generate a configuration result of the first configuration module and a configuration result of the second configuration module according to the test user set, the non-test user set, the first current version, the first updated version, the second current version, and the second updated version; According to the configuration result of the first configuration module and the configuration result of the second configuration module, the task scheduling system is run, and the first updated version and the second updated version are tested based on the test user set to obtain test results.

2. The method according to claim 1, characterized in that The user characteristics include the business impact of the data processing job, the acceptance of job failure, and whether delayed reprocessing can be accepted; accordingly, according to the user characteristics of each user managed by the task scheduling system, all users are divided into a test user set and a non-test user set, including: Define users whose business impact on data processing jobs is lower than a preset impact, whose acceptance of job failure is lower than a preset acceptance, and who can accept delayed reprocessing as test users, and construct a test user set; Define the remaining users as non-test users and construct a non-test user set.

3. The method according to claim 1, characterized in that The generating, according to the test user set, the non-test user set, the first current version, the first updated version, the second current version, and the second updated version, a configuration result of the first configuration module and a configuration result of the second configuration module comprises: Determining an isolation configuration of the job processing module according to the test user set, the non-test user set, the first current version, and the first updated version; Determining a plug-in version information set configuration of the job processing plug-in according to the second current version and the second updated version; According to the isolation configuration and the plug-in version information set configuration, the association configuration between the version of the job processing module and the version of the job processing plug-in is determined.

4. The method according to claim 3, characterized in that The determining, according to the test user set, the non-test user set, the first current version, and the first updated version, the isolation configuration of the job processing module includes: The test user set is set as the isolation configuration of the first current version of the job processing module, and the non-test user set is set as the isolation configuration of the first updated version of the job processing module.

5. The method according to claim 3, characterized in that: There are multiple job processing plug-ins; accordingly, determining the plug-in version information set configuration of the job processing plug-in according to the second current version and the second updated version includes: For any job processing plug-in, the plug-in version information set configuration of the any job processing plug-in is determined according to the second updated version of the any job processing plug-in and the second current versions of the remaining job processing plug-ins.

6. The method according to claim 1, characterized in that The step of running the task scheduling system according to the first configuration result, testing the first updated version and the second updated version based on the test user set, and obtaining the test result includes: The job processing module pulls the job request message and identifies a user type corresponding to the job request message, where the user type indicates whether the user belongs to a test user set; Acquire the configuration result of the first configuration module and the configuration result of the second configuration module from the configuration center; Determine, according to the user type, the configuration result of the first configuration module, and the configuration result of the second configuration module, the version information of the job processing module and the version information of the job processing plug-in corresponding to the job request message; According to the version information of the job processing module and the version information of the job processing plug-in corresponding to the job request message, the corresponding job processing module and the job processing plug-in are called to process the job request message, and then the job processing result is obtained.

7. The method according to claim 6, characterized in that The method further comprises: When the job processing result meets the preset conditions, the configuration result of the first configuration module and the configuration result of the second configuration module are modified, and all users in the test user set and the non-test user set correspond to the first updated version and the second updated version.

8. An upgrade system for a task scheduling system, characterized in that: The upgrade system of the task scheduling system includes a configuration center and a task scheduling system. The task scheduling system includes a job processing module and a job processing plug-in. The job processing module is used to manage, dispatch and execute job request messages. A first configuration module is embedded in the job processing module. The job processing plug-in is used to perform data processing based on the job request message. A second configuration module is embedded between the job processing module and the job processing plug-in. The first configuration module and the second configuration module are configured and managed by the configuration center. The task scheduling system upgrade system executes the method steps as claimed in any one of claims 1 to 7.

9. A computer device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The method comprises computer-executable instructions, which implement the method according to any one of claims 1 to 7 when executed by a processor.

Images