Scheduling script generation method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a scheduling script generation method and device, a storage medium and electronic equipment, and the method comprises the steps: determining a target algorithm corresponding to a business demand in a preset algorithm library, and generating a scheduling job rule according to the target algorithm and the business demand, the scheduling job rule at least comprises a transmission path, a processing logic and a scheduling parameter of a scheduling job corresponding to the service demand; configuring a first mapping relation between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list, and configuring a second mapping relation between source field information in the source table and target field information in the target table according to the scheduling job rule to generate a column mapping list, the source field information is used for indicating each row of data in the source table, and the target field information is used for indicating each row of data in the target table; and generating a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list.

Description

Technical Field

[0001] The present application relates to the field of data processing and integration technology, and in particular to a method and device for generating a scheduling script, a storage medium, and an electronic device. Background Art

[0002] Currently, cross-platform data exchange methods vary. In a microservices architecture, immediate responses primarily rely on external interface calls, often using a one-to-one API call approach. Acquired data is then processed based on the API's requests and responses. Batch exchange primarily relies on highly productized tools, with one-to-one batch job development. The cross-platform data extraction, transformation, and loading (ETL) process is complex.

[0003] Therefore, there is a problem in the related art that the process of cross-platform data scheduling (data extraction, conversion and loading) is relatively complicated.

[0004] With regard to the complex process of cross-platform data scheduling (data extraction, conversion and loading) in related technologies, no effective solution has been proposed so far.

[0005] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the Invention

[0006] The embodiments of the present application provide a method and device for generating a scheduling script, a storage medium, and an electronic device to at least solve the problem of the complex process of cross-platform data scheduling (data extraction, conversion, and loading) in related technologies.

[0007] According to one embodiment of the present application, a method for generating a scheduling script is provided, comprising: determining a target algorithm corresponding to a business requirement in a preset algorithm library, and generating a scheduling job rule based on the target algorithm and the business requirement, wherein the business requirement is used to indicate that the target data in the source database of the source system is scheduled to the target database of the target system, and the scheduling job rule includes at least one of the following: a transmission path of the scheduling job corresponding to the business requirement, a processing logic corresponding to the scheduling job, and a scheduling parameter corresponding to the scheduling job; configuring a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list, and configuring a second mapping relationship between source field information in the source table and target field information in the target table according to the scheduling job rule to generate a column mapping list, wherein the source field information is used to indicate each row of data in the source table, and the target field information is used to indicate each row of data in the target table; generating a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list.

[0008] In an exemplary embodiment, generating a scheduling operation rule based on the target algorithm and the business requirements includes: parsing the business requirements, and determining the transmission path, the processing logic and the scheduling parameters through the parsed business requirements and the target algorithm; generating the scheduling operation rule based on at least one of the transmission path, the scheduling parameters and the processing logic.

[0009] In an exemplary embodiment, the transmission path, the processing logic and the scheduling parameters are determined by the parsed business requirements and the target algorithm, including: determining the transmission path corresponding to the scheduling job according to the parsed business requirements; determining a resource allocation strategy for allocating resources required to execute the scheduling job according to the target algorithm and the business requirements, and determining scheduling configuration information corresponding to the scheduling job according to the target algorithm and the business requirements, wherein the scheduling configuration information includes at least one of the following: scheduling frequency, scheduling time window and whether the scheduling job is executed periodically; determining the scheduling parameters according to the resource allocation strategy and the scheduling configuration information; determining the data flow corresponding to the scheduling job and the data conversion rules for converting the target data from the source database to the target database according to the target algorithm and the business requirements, and determining the processing logic according to the data conversion rules.

[0010] In an exemplary embodiment, a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system is configured according to the scheduling job rule to generate a table mapping list, including: determining the source table in the source system according to the scheduling job rule, and determining the target table in the target system, wherein the source table at least includes: data stored in the source database, and the target table at least includes: data stored in the target database; determining the first mapping information between the source table and the target table through a low-code tool, wherein the first mapping information includes at least one of the following: a first identifier of the source table, a second identifier of the target table, a source system parameter of the source system, and a target system parameter of the target system; configuring the first mapping relationship between the source table and the target table according to the first mapping information to generate the table mapping list.

[0011] In an exemplary embodiment, a second mapping relationship between source field information in the source table and target field information in the target table is configured according to the scheduling job rules to generate a column mapping list, including: determining a first field list in the source table to be configured to the target table according to the scheduling job rules, wherein the source field information is used to indicate the field information contained in the first field list; allocating a second field list to the first field list in the target table through a low-code tool, wherein the target field information is used to indicate the field information contained in the second field list; determining a conversion rule for converting fields in the first field list into fields in the second field list, and determining the second mapping information between the source field information and the target field information according to the conversion rule; configuring the second mapping relationship between the source table and the target table according to the second mapping information to generate the column mapping list.

[0012] In an exemplary embodiment, a scheduling script for executing the scheduling job is generated based on the table mapping list and the column mapping list, including: generating an interface calling code for extracting the target data in the source table and a first code for scheduling the target data to the target table based on the table mapping list; generating a second code for mapping the first field list in the source table to the second field list in the target table based on the column mapping list; and generating the scheduling script based on the first code, the second code, and the interface calling code.

[0013] In an exemplary embodiment, after generating a scheduling script for executing the scheduling job based on the table mapping list and the column mapping list, the method further includes: generating execution configuration information for executing the scheduling job based on the scheduling script, wherein the execution configuration information includes at least one of the following: the execution time and execution frequency of the scheduling job; configuring execution nodes for the scheduling job and determining the dependency relationship between each execution node; generating a scheduling job rule for the scheduling job based on the execution configuration information, the execution nodes and the dependency relationship, and determining the triggering condition of the scheduling job rule; and executing the scheduling job based on the scheduling job rule when it is determined that the triggering condition is triggered.

[0014] According to another embodiment of the present application, a device for generating a scheduling script is provided, including: a determination module for determining a target algorithm corresponding to a business requirement in a preset algorithm library, and generating a scheduling job rule based on the target algorithm and the business requirement, wherein the business requirement is used to indicate that the target data in the source database of the source system is scheduled to the target database of the target system, and the scheduling job rule includes at least one of the following: a transmission path of the scheduling job corresponding to the business requirement, a processing logic corresponding to the scheduling job, and a scheduling parameter corresponding to the scheduling job; a configuration module for configuring a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list, and configuring a second mapping relationship between source field information in the source table and target field information in the target table according to the scheduling job rule to generate a column mapping list, wherein the source field information is used to indicate each row of data in the source table, and the target field information is used to indicate each row of data in the target table; a generation module for generating a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list.

[0015] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0016] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] According to another embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps of any of the above method embodiments when executed by a processor.

[0018] Through the embodiments of the present application, the target algorithm corresponding to the business needs is determined in the preset algorithm library, and then the scheduling operation rules are generated according to the target algorithm and the business needs; the first mapping relationship between the source table corresponding to the source system and the target table corresponding to the target system is configured according to the scheduling operation rules to generate a table mapping list, and the second mapping relationship between the source field information in the source table and the target field information in the target table is configured according to the scheduling operation rules to generate a column mapping list; and then a scheduling script for executing the scheduling operation corresponding to the business needs is generated according to the table mapping list and the column mapping list. According to the embodiments of the present application, the problem of the complex process of cross-platform data scheduling (data extraction, conversion and loading) in the related technology can be solved, and then the process of cross-platform (source system and target system) data scheduling can be simplified by generating a scheduling script. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is a hardware structure block diagram of a computer terminal device for a method for generating a scheduling script according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of a method for generating a scheduling script according to an embodiment of the present application;

[0023] Figure 3 This is an architectural diagram of a cross-platform, enterprise-level ETL low-code R&D integrated tool according to an optional embodiment of the present application;

[0024] Figure 4 is a flowchart of the parsing process of the low-code tool according to an optional embodiment of the present application;

[0025] Figure 5 This is a schematic diagram corresponding to a cross-platform ETL script according to an optional embodiment of the present application;

[0026] Figure 6 is a schematic diagram corresponding to a node configuration operation according to an optional embodiment of the present application;

[0027] Figure 7 is a schematic diagram of performing a configuration operation according to an optional embodiment of the present application;

[0028] Figure 8 It is a structural block diagram of a device for generating a scheduling script according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0031] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal device or a similar computing device. Taking running on a computer terminal device as an example, Figure 1 This is a hardware structure diagram of a computer terminal device for a method of generating a scheduling script according to an embodiment of the present application. Figure 1 As shown, the computer terminal device may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MPU or a programmable logic device FPGA) and a memory 104 for storing data. The computer terminal device may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal device. For example, the computer terminal device may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0032] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for generating a scheduling script in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal device via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0033] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a communication provider of a computer terminal device. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] In this embodiment, a method for generating a scheduling script is provided. Figure 2 is a flow chart of a method for generating a scheduling script according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0035] Step S202: Determine a target algorithm corresponding to the business requirement in a preset algorithm library, and generate a scheduling operation rule based on the target algorithm and the business requirement, wherein the business requirement is used to instruct the scheduling of target data in the source database of the source system to the target database of the target system, and the scheduling operation rule includes at least one of the following: a transmission path of the scheduling operation corresponding to the business requirement, processing logic corresponding to the scheduling operation, and scheduling parameters corresponding to the scheduling operation;

[0036] Step S204: configuring a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling operation rule to generate a table mapping list, and configuring a second mapping relationship between source field information in the source table and target field information in the target table according to the scheduling operation rule to generate a column mapping list, wherein the source field information is used to indicate each row of data in the source table, and the target field information is used to indicate each row of data in the target table;

[0037] Step S206: Generate a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list.

[0038] Through the above steps, the target algorithm corresponding to the business needs is determined in the preset algorithm library, and then the scheduling operation rules are generated according to the target algorithm and the business needs; the first mapping relationship between the source table corresponding to the source system and the target table corresponding to the target system is configured according to the scheduling operation rules to generate a table mapping list, and the second mapping relationship between the source field information in the source table and the target field information in the target table is configured according to the scheduling operation rules to generate a column mapping list; and then a scheduling script for executing the scheduling operation corresponding to the business needs is generated according to the table mapping list and the column mapping list. According to the embodiment of the present application, the problem of the complex process of cross-platform data scheduling (data extraction, conversion and loading) in the related technology can be solved, and then the process of cross-platform (source system and target system) data scheduling can be simplified by generating a scheduling script.

[0039] Optionally, the step S202 of generating a scheduling operation rule according to the target algorithm and the business requirements includes:

[0040] 1) Analyze the business requirements;

[0041] Understandably, at the outset of generating scheduling rules, it's necessary to understand and analyze business requirements, such as data sources, targets, processing methods, and timeframes. For example, a company might need to extract data from a sales table in a Hive database, cleanse and convert the data, and then load it into a sales summary table in a Data Warehouse Service (DWS) database. The data must be updated daily between 2:00 AM and 3:00 AM.

[0042] When analyzing business needs, low-code tools will read information such as data sources, data targets, data processing methods, etc. from configuration documents or interfaces, and convert this information into processable data structures.

[0043] 2) Determine the transmission path, the processing logic, and the scheduling parameters based on the analyzed business requirements and the target algorithm; specifically:

[0044] Determine the transmission path corresponding to the scheduling job according to the parsed business needs; determine the resource allocation strategy for allocating resources required to execute the scheduling job according to the target algorithm and the business needs, and determine the scheduling configuration information corresponding to the scheduling job according to the target algorithm and the business needs, wherein the scheduling configuration information includes at least one of the following: scheduling frequency, scheduling time window and whether the scheduling job is executed periodically; determine the scheduling parameters according to the resource allocation strategy and the scheduling configuration information; determine the data flow corresponding to the scheduling job and the data conversion rules for converting the target data from the source database to the target database according to the target algorithm and the business needs, and determine the processing logic according to the data conversion rules.

[0045] The transmission path determines the data flow path based on the data source and target information in the business requirements. For example, this involves determining whether to use Hive Structured Query Language (HiveSQL) for data extraction or Spark SQL (Spark is a distributed computing system) for data transformation, and ultimately whether to load data into the target system through a Common Data Model (CDM) job or DWS SQL.

[0046] Processing logic: Based on the target algorithm and business requirements, determine the specific steps for data processing, such as using algorithms such as Append, Delta, Full_Dump, Upsert, and Replace to transform the data. For example, if the business requirement is to update the data in the target table daily, the processing logic might use the "Append" algorithm to add the new sales data to the target table.

[0047] Scheduling parameters: Determine the scheduling details of the job, such as the scheduling frequency (end of day or low frequency), the scheduling time window (specific execution time period), and whether to execute periodically. For example, set the scheduling frequency to end of day, the scheduling time window to 2:00 AM to 3:00 AM, and the periodic execution to daily.

[0048] 3) Generate the scheduling operation rule according to at least one of the transmission path, the scheduling parameters and the processing logic.

[0049] It is understandable that after determining the transmission path, processing logic, and scheduling parameters, low-code tools can automatically generate scheduling job rules. Scheduling job rules will guide the specific execution of ETL jobs, including when to execute, how to execute, and the order and dependencies of execution.

[0050] For example, when generating a scheduling job rule, a low-code tool might create a job name similar to "SYN-SALES_HIVE_D," where "SYN" stands for data synchronization, "SALES" is the English name of the target table, "HIVE" is the source database, and "D" indicates the end-of-day scheduling frequency. Furthermore, low-code tools can automatically generate resource allocation and scheduling parameters based on resource allocation policies and scheduling configurations, such as setting the number of concurrent subtasks, the number of extractors, and job priority. Job rules also include descriptions of data flows and data conversion rules to ensure that data follows predefined formats and logic during transmission.

[0051] Optionally, the step S204 of configuring a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list includes:

[0052] 1) determining the source table in the source system and the target table in the target system according to the scheduling job rule, wherein the source table includes at least data stored in the source database, and the target table includes at least data stored in the target database;

[0053] As you can understand, scheduling job rules define the data transfer path and processing method from the source system to the target system. Based on scheduling job rules, the source and destination of data can be automatically identified. For example, a scheduling job rule specifies extracting data from the sales table (source table) in the Hive database and then loading it into the sales summary table (target table) in the DWS database.

[0054] 2) determining, by a low-code tool, first mapping information between the source table and the target table, wherein the first mapping information includes at least one of the following: a first identifier of the source table, a second identifier of the target table, a source system parameter of the source system, and a target system parameter of the target system;

[0055] Among them, the first mapping information is the bridge to establish the association between the source table and the target table. The low-code tool can automatically identify and determine the mapping relationship between the source table and the target table based on the scheduling job rules and business needs. The first mapping information contains at least the first identifier of the source table (such as the source table name "sales table"), the second identifier of the target table (such as the target table name "sales summary table"), the parameters of the source system (such as the connection information of the Hive database) and the parameters of the target system (such as the connection information of the DWS database).

[0056] For example, suppose a scheduled job rule is executed based on "SYN-SALES_HIVE_D," where "SYN" represents data synchronization, "SALES" is the source table name, "HIVE" is the source database identifier, and "D" represents the end-of-day scheduling frequency. Low-code tools can identify and configure these identifiers and parameters, such as: the source table's first identifier: "Sales Table"; the target table's second identifier: "Sales Summary Table"; the source system's source system parameters: Hive database connection information, including the server address, port number, database name, username, and password; the target system's target system parameters: DWS database connection information, also including the server address, port number, database name, username, and password.

[0057] 3) configuring a first mapping relationship between the source table and the target table according to the first mapping information to generate the table mapping list.

[0058] The configuration of the first mapping relationship is based on the first mapping information. By identifying the tables, fields, and data types involved in the source and target systems, the low-code tool can automatically generate a table mapping list to guide the actual conversion and transfer of data. For example, the table mapping list will list in detail which source table fields correspond to which target table fields, the field type conversion rules, and whether data cleansing is required.

[0059] Optionally, the step S204 of configuring a second mapping relationship between the source field information in the source table and the target field information in the target table according to the scheduling job rule to generate a column mapping list includes:

[0060] 1) determining a first field list in the source table to be configured in the target table according to the scheduling job rule, wherein the source field information is used to indicate field information included in the first field list; specifically:

[0061] Low-code tools can determine the specific fields in the source table that need to be transferred to the target table based on scheduling rules and business requirements. For example, if the business requirement is to synchronize transaction data from the Hive source table to the DWS target table at the end of each day, the low-code tool can identify all relevant transaction information fields from the source table to form the first field list.

[0062] 2) allocating a second field list to the first field list in the target table using a low-code tool, wherein the target field information is used to indicate field information included in the second field list;

[0063] That is, after the first field list is determined, the low-code tool can automatically or assist developers in finding or creating equivalent fields in the target table to form the second field list.

[0064] 3) determining a conversion rule for converting fields in the first field list into fields in the second field list, and determining second mapping information between the source field information and the target field information according to the conversion rule;

[0065] To ensure the correct transfer of data between the source and target tables, low-code tools can determine the conversion rules between fields based on data types, business logic, and possible data aggregation requirements. For example, for the "amount" field, if the target table requires the total transaction amount per customer per day, the conversion rule may involve aggregating and summing this field by customer identifier (ID) and date. These conversion rules and field mapping relationships together constitute the second mapping information.

[0066] 4) configuring a second mapping relationship between the source table and the target table according to the second mapping information to generate the column mapping list.

[0067] That is, the low-code tool can generate a specific column mapping list based on the second mapping information, and record in detail the mapping relationship and conversion logic between the source table fields and the target table fields.

[0068] Optionally, generating a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list in step S206 includes:

[0069] 1) generating, according to the table mapping list, an interface calling code for extracting the target data from the source table and a first code for dispatching the target data to the target table;

[0070] That is, based on the previously generated table mapping list, the low-code tool can automatically generate the interface call code for extracting data from the source table and the first code for dispatching this data to the target table. The first code ensures the correct acquisition and import of data based on the configured transmission path and processing logic.

[0071] 2) generating a second code for mapping the first field list in the source table to the second field list in the target table according to the column mapping list;

[0072] That is, the low-code tool can also generate a second code for field mapping and data conversion based on the column mapping list. The second code ensures the correspondence between the source table fields and the target table fields, and processes the data according to the preset conversion rules.

[0073] 3) Generate the scheduling script according to the first code, the second code and the interface calling code.

[0074] That is, low-code tools can also combine the first code and the second code to generate a complete scheduling script. The scheduling script contains all the necessary steps for data extraction, data transformation, and data loading, and can automate the entire ETL process.

[0075] Optionally, after generating a scheduling script for executing the scheduled job according to the table mapping list and the column mapping list in step S206, the method further includes:

[0076] 1) generating execution configuration information for executing the scheduled job according to the scheduling script, wherein the execution configuration information includes at least one of the following: the execution time and execution frequency of the scheduled job;

[0077] Among them, the above-mentioned execution configuration information defines key parameters such as the execution environment, time and frequency of the job. Low-code tools can automatically generate execution configuration information based on the scheduling script to ensure that the ETL job can be automatically executed according to the preset schedule without manual intervention. For example: Assume that the scheduling script is used to synchronize the "sales" table data in Hive (a data warehouse) to the "sales_summary" (sales summary) table in DWS at the end of the day, then the execution configuration information may include: Execution time: 4 am every day; Execution frequency: end of day (once a day).

[0078] 2) configuring execution nodes for the scheduled job and determining the dependency relationship between each execution node;

[0079] The above-mentioned execution nodes refer to the various processing tasks or steps in the ETL process, such as data extraction, data conversion, data loading, etc. Low-code tools can automatically configure execution nodes for each scheduled job, and at the same time, determine the dependencies between these nodes based on the logical order of the job to ensure the correctness and efficiency of the data processing process. For example, a scheduled job may include the following nodes: Data extraction node: extracts data from the "sales" table in Hive; data cleaning node: cleans and preprocesses the extracted data; data aggregation node: performs summary calculations on the cleaned data; data loading node: loads the summarized data into the "sales_summary" table in DWS.

[0080] The dependency relationship between the above execution nodes can be: after the data extraction node is completed, the data cleaning node is triggered, after the data cleaning node is completed, the data aggregation node is triggered, and finally after the data aggregation node is completed, the data loading node is triggered.

[0081] 3) generating a scheduling operation rule for the scheduling operation according to the execution configuration information, the execution node, and the dependency relationship, and determining a triggering condition for the scheduling operation rule;

[0082] That is, the scheduling job rules combine the execution configuration information, execution nodes, and dependencies, and are a complete job execution plan. Low-code tools can organize this information into rules, such as defining the conditions under which the job is started, the execution order of the job, and the handling strategy when errors are encountered. For example: the scheduling job rules may be: Trigger condition: the current time reaches 4 a.m. every day; execution order: data extraction → data cleaning → data aggregation → data loading; error handling: if the data extraction node fails, the entire job is paused and waits for manual restart. These scheduling job rules are defined and edited through configuration files, database records, or the interface of the low-code tool to ensure that the job can be executed according to the established process and schedule.

[0083] 4) When it is determined that the trigger condition is triggered, executing the scheduling operation based on the scheduling operation rule.

[0084] Once the trigger conditions are met, low-code tools can automatically execute the corresponding scheduled jobs according to the scheduling job rules. Low-code tools can monitor time and, when the preset time point is reached, initiate the job according to the rules, extracting data from the source system, performing data processing and transformation, and finally loading it into the target system. The entire process is completed automatically without human intervention.

[0085] For example, at 4 a.m., when the tool detects that the trigger conditions have been met, it automatically executes the scheduled job according to the following steps: It starts the data extraction node to extract the previous day's sales data from the Hive "sales" table; it starts the data cleaning node to clean the extracted data, such as removing blank records and outliers; it starts the data aggregation node to summarize the total sales amount according to business needs, such as by customer ID and date; and it starts the data loading node to load the aggregated data into the DWS "sales_summary" table, completing the end-of-day data synchronization job.

[0086] In this way, low-code tools not only simplify the configuration of scheduling jobs, but also realize the automated execution of jobs, improve the efficiency and reliability of data processing, and reduce the burden of manual management.

[0087] In order to better understand the process of the above-mentioned scheduling script generation method, the implementation method flow of the above-mentioned scheduling script generation method is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of this application.

[0088] The optional embodiment of this application proposes a cross-platform, enterprise-level ETL low-code R&D integrated tool or method. This tool can unify interface specifications, improve interface usability, enhance interface template adaptability, strengthen cross-platform exchange automation capabilities, improve interface reusability and reduce code redundancy, allowing developers to improve development efficiency and facilitate maintenance during the development process. Based on an enterprise-level development platform, this tool can effectively integrate multiple functions such as cross-platform data exchange, ETL processes, batch scheduling, signal triggering, and automatic labeling, and complete development through configuration documents, greatly improving development efficiency.

[0089] The optional embodiment of this application is developed around a set of algorithm rule bases, which explains how to parse through a script program based on the rule base and parameter configuration, and generate the code of the interface program through the template code and the parsed parameters. It can quickly generate the interface program in a low-code manner, and at the same time generate the corresponding scheduling tasks, which can automatically complete the configuration of the time and frequency of cross-platform interaction, further reducing the coding workload and improving development efficiency. At the same time, the business logic of the interface program can be supplemented by a business development module, which can be customized and implemented by developers to further improve scalability.

[0090] Figure 3 This is an architectural diagram of a cross-platform enterprise-level ETL low-code R&D integrated tool according to an optional embodiment of the present application, such as Figure 3 As shown:

[0091] A cross-platform enterprise-level ETL low-code R&D integrated tool in an optional embodiment of the present application includes: a source system, a data center, and a target system;

[0092] The data middle platform obtains data from the source system through low-code tools, performs ETL on the data in the source system, and then loads the data into the target system to realize data scheduling.

[0093] Among them, the low-code tool includes a rule engine, which includes: algorithm library, mapping library, rule library, configuration library and parameter library.

[0094] Figure 4 This is a flowchart of the parsing process of the low-code tool according to an optional embodiment of the present application, such as Figure 4 As shown:

[0095] Step S401, receiving parameters passed in by the rule engine;

[0096] Step S402, generating an interface program according to the parameters;

[0097] Step S403: Release the data middle platform interface program.

[0098] Among them, the algorithm library in the rule engine can include several types of algorithms used to generate scheduling jobs, which can be adapted to most data exchange scenarios and can be further supplemented if there are special scenarios.

[0099] based on Figure 3 The optional embodiment of the present application describes how to understand the rule engine template in general, familiarize yourself with the template content, filling specifications, the relationship between the template and the interface, etc., so that developers can quickly understand it.

[0100] The main contents of the optional embodiment of this application include eight parts: algorithm description, generation rules, table mapping operations, column mapping operations, cross-platform ETL script operations, job execution operations, node configuration operations, and execution configuration operations. While ensuring full adaptability, it also minimizes technical debt caused by excessive customization, improves interface usability, and lowers the development threshold.

[0101] 1. Algorithm description:

[0102] The above rule engine includes an algorithm library.

[0103] The algorithms contained in the algorithm library can be further explained, specifically:

[0104] Table 1 is an introduction table of algorithms in the algorithm library according to an optional embodiment of the present application, as shown in Table 1:

[0105] Table 1

[0106]

[0107]

[0108] The above algorithm is implemented by a script, which is parsed according to the rule engine template and generates an interface program based on the parameters, and then publishes the generated interface program for actual application.

[0109] 2. Generate rules:

[0110] The generated rules are the above rule base.

[0111] The rule base may include: scheduling job algorithm templates (ie, scheduling job rules).

[0112] Table 2 is a schematic diagram of a scheduling algorithm template according to an optional embodiment of the present application, as shown in Table 2:

[0113] Table 2

[0114]

[0115]

[0116]

[0117] The job name can be determined based on the above-mentioned scheduling job algorithm template, and the job name format is "algorithm classification_target name_job frequency". After filling in the algorithm classification, target name and job frequency, the scheduling job will be automatically generated. According to the above scheduling job algorithm template, corresponding interface code can be generated based on different development platforms. The scheduling job algorithm template proposed in the optional embodiment of the present application is a standard development template, which further improves the standardization and maintainability of interface development and enhances the quality of interface development.

[0118] The mapping library may include: table mapping (i.e., table mapping list) and column mapping (i.e., column mapping list), where:

[0119] 3. Table mapping operation:

[0120] A table mapping consists of related elements of a source table and a target table.

[0121] Source table information includes: source storage medium, source system, source table schema (SCHEMA), source table name, data file exchange platform path, data file character set, data format, column delimiter, whether the data file has a header, update instructions, etc.

[0122] The target table includes: identifier (ID), job name (read job name directly), algorithm type, job frequency, algorithm name, target-storage medium, target system, target table SCHEMA, target table subject, target table name, and target table Chinese name.

[0123] These table elements are filled in the second row of the table mapping. The first, third and fourth rows are their corresponding English names, filling rules and data types respectively.

[0124] 4. Column mapping operation:

[0125] A column mapping consists of related elements from the source and target tables.

[0126] Source table related elements include: SCHEMA, source table Chinese name, source table English name, source field Chinese name, source field English name, source field data type, mapping rules, remarks, association rules, and filtering rules;

[0127] The relevant elements of the target table include: table number, job name, column number, target table SCHEMA, table Chinese name, target table name, field Chinese name, field English name, field data type, whether it is a primary key, whether it is a step key, and code value description.

[0128] These table elements are filled in the second row of the column mapping, and the first, third and fourth rows are their corresponding English names, filling rules and data types respectively.

[0129] 5. Cross-platform ETL script operation:

[0130] Figure 5 This is a schematic diagram corresponding to a cross-platform ETL script according to an optional embodiment of the present application, such as Figure 5 As shown:

[0131] The table elements of the cross-platform ETL script are:

[0132] ID, job name, job type, source connection name, source connection type, destination connection name, destination connection type, fromJobConfig.useSql (job configuration - use SQL), fromJobConfig.schemaName (source job configuration - schema name), fromJobConfig.tableName (source job configuration - table name), fromJobConfig.incrMigration (source job configuration - incremental migration), fromJobConfig.allowNullValueInPartitionColumn (source job configuration - allow NULL values ​​in partition columns), fromJobConfig.cdc (source job configuration - change data capture), fromJobConfig.creatOutTable (source job configuration - create output table), fro mJobConfig.usePartition (source job configuration - use partition), toJobConfig.extendeFields (target job configuration - extended fields), toJobfig.schemaName (target job configuration - schema name), toJobfig.tablePreparation (target job configuration - table preparation), toJobfig.tableName (target job configuration - table name), toJobfig.compression (target job configuration - compression), toJobfig.useStageTable (target job configuration - use staging table), toJobfig.shouldClearTable (target job configuration - whether to clear the table), toJobfig.beforeImportType (target job configuration - pre-import type), throttl ingConfig.concurrentSubJob (current limiting configuration - number of concurrent subtasks), throttlingConfig.numExtractors (current limiting configuration - number of extractors), throttlingConfig.numSplits (current limiting configuration - number of splits), throttlingConfig.spl itRetryTime (current limiting configuration - split retry time), throttlingConfig.submitToCluster (current limiting configuration - submit to cluster), throttlingConfig.numLoaders (current limiting configuration - number of loaders), throttlingConfig.recordDirtyData (current limiting configuration - record dirty data), throttlingConfig.writeToLink (throttling configuration - write connection), throttlingConfig.maxErrorRecords (throttling configuration - maximum number of error records), throttlingConfig.throttling (throttling configuration - current limiting), throttlingConfig.byteRate (throttling configuration - byte rate), schedulerConfig.isSchedulerJob (scheduler configuration - whether it is a scheduled job), schedulerConfig.disposableType (scheduler configuration - disposal type), transformConfig.trans (transformation configuration - transformation), smnConfig.isNeedNotification (notification configuration - whether notification is required), retryJobConfig.retryJobType (retry job configuration - retry job type), groupJobConfig.groupId (job group configuration - group ID), groupJobConfig.groupName (job group configuration - group name).

[0133] in, Figure 5 The fifth row explains the entry rules for each column. For example, the job name must be between 1 and 240 characters long. Job types include table / file migration, whole database migration, and scenario migration. The source connection type corresponds to the following connection parameters: relational database connection, OBS connection (object storage service connection), HDFS connection (Hadoop Distributed File System connection), HBase connection (HBase database connection), CloudTable connection (Cloud Table Storage Service connection), Hive connection (Hive data warehouse connection), ATP / SFTP connection (secure file transfer protocol connection), MongoDB connection (MongoDB database connection), Redis / DCS connection (Redis or distributed cache service connection), Elasticsearch cloud search service connection, DLI connection (Data Lake Insight connection), HTTP / HTTPS connection (Hypertext Transfer Protocol / Secure Hypertext Transfer Protocol connection, which currently has no connection parameters), and DMC Kafka connection (Data Management Platform Kafka connection). The destination connection name is the connection name corresponding to the connection created through the interface. The connection parameters corresponding to the destination connection type can be the same as those corresponding to the source connection type. The contents of the sixth and seventh rows are examples.

[0134] In the database schema or tablespace, for example, "public". This parameter supports the configuration of wildcards (*) to export all databases that begin with a prefix or end with a suffix. For example, SCHEMA* means exporting all databases that begin with "SCHEMA". *SCHEMA means exporting all databases that end with "SCHEMA". *SCHEMA* means exporting all tables that contain the string "SCHEMA". Select BDM (Bulk Data Management), ADM_WDM (Advanced Data Management for Wide Data Models), WDM (Wide Data Model), or BDM_TMP (Bulk Data Migration Temporary). Select the group to which the job belongs. The default group is "DEFAULT".

[0135] 6. Execute the operation:

[0136] The table elements for executing a job include ID, job name (directly read the job name), object name, low-frequency time, job type, job path in the directory tree, label, job priority, empty run, job parameter definition, resource, single node flag, user cache, version, scheduling type, horizontal axis position, vertical axis position, proxy service, user-defined field, and instance timeout.

[0137] The job name format is "algorithm classification-target table name_job frequency". The algorithm classification is used as a prefix. The prefixes corresponding to the four algorithm types are shown in Table 3:

[0138] Table 3

[0139] Algorithm Type Prefix Data Extraction SYN Data Conversion TRS Data synchronization SYN Data offloading ULD

[0140] The operating frequency is used as a suffix. The suffixes corresponding to the two frequencies are shown in Table 4:

[0141] Table 4

[0142] Operation frequency suffix End of Day D low frequency L

[0143] For example, the job name is TRS-S088_PROD_TERMINATE_REG-D, where TRS is the prefix, - is the connector, S088_PROD_TERMINATE_REG is the target table name, - is the connector, and D is the suffix.

[0144] 7. Node configuration operation:

[0145] Figure 6 is a schematic diagram corresponding to the node configuration operation according to an optional embodiment of the present application, such as Figure 6 As shown:

[0146] The table elements of node configuration include ID, job type, node name, node type, list of previous node names that this node depends on, polling node execution result interval, node failure strategy, input and output information, maximum node execution time, number of node failure retries, failure retry interval, resources, script name, script type, detailed description, database name, script parameters, stored procedure or script, dry run script, CDM job type, CDM cluster name, node horizontal axis position on the job canvas, and node vertical axis position on the job canvas.

[0147] in, Figure 6 Lines 5 to 10 of are examples.

[0148] 8. Perform configuration operations:

[0149] Figure 7 is a schematic diagram of performing a configuration operation according to an optional embodiment of the present application, such as Figure 7 As shown:

[0150] The table elements of the execution configuration include the sequence number, job name, scheduling start time, scheduling end time, Cron expression, time zone information corresponding to the Cron expression, interval type, whether to skip its own dependencies, the current number of connections, whether it depends on the execution result of the previous run cycle task of this job, and a list of dependent job names.

[0151] in, Figure 8 Lines 5 to 28 of are examples.

[0152] In summary, the optional embodiment of the present application is based on a low-code rule library to adapt to a variety of scenarios, provide a variety of commonly used templates, and support the addition of custom templates. While continuously improving the template library, it is also improving the flexibility of the system, shortening the development cycle, lowering the threshold for developers to use, and can quickly get started through simple training. Product personnel can directly combine the desired pages through the system, and then the developers will improve it, reducing the investment in coding time, trial and error costs, and communication costs; and support page data source configuration, which can realize dynamic binding between page elements and data; in addition, it encapsulates high-quality code to reduce the high operation and maintenance costs caused by code quality in the later stage. The interface development method and tool of the cross-platform ETL low-code integrated research and development of the optional embodiment of the present application configures a variety of cross-platform ETL methods, further improving the adaptability of the method; at the same time, it takes into account the trigger mechanism of real-time access and batch scheduling, further improving the automation capability of cross-platform ETL. Specifically, it has the following technical effects: unified interface development specifications, improving interface development quality; improving interface usability, lowering the development threshold; improving the adaptability level of low-code development templates; strengthening the automation capability of cross-platform data exchange; improving interface reusability, and reducing code redundancy.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0154] In this embodiment, a device for generating a scheduling script is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments. The details already described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0155] Figure 8 is a structural block diagram of a device for generating a scheduling script according to an embodiment of the present application, such as Figure 8 As shown, the device includes:

[0156] A determination module 82 is configured to determine a target algorithm corresponding to a business requirement in a preset algorithm library, and generate a scheduling operation rule based on the target algorithm and the business requirement, wherein the business requirement is used to instruct the scheduling of target data in a source database of a source system to a target database of a target system, and the scheduling operation rule includes at least one of the following: a transmission path of the scheduling operation corresponding to the business requirement, processing logic corresponding to the scheduling operation, and scheduling parameters corresponding to the scheduling operation;

[0157] a configuration module 84 configured to configure a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling operation rule to generate a table mapping list, and to configure a second mapping relationship between source field information in the source table and target field information in the target table according to the scheduling operation rule to generate a column mapping list, wherein the source field information is used to indicate each row of data in the source table, and the target field information is used to indicate each row of data in the target table;

[0158] The generating module 86 is configured to generate a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list.

[0159] Through the above-mentioned device, the target algorithm corresponding to the business demand is determined in the preset algorithm library, and then the scheduling operation rules are generated according to the target algorithm and the business demand; the first mapping relationship between the source table corresponding to the source system and the target table corresponding to the target system is configured according to the scheduling operation rules to generate a table mapping list, and the second mapping relationship between the source field information in the source table and the target field information in the target table is configured according to the scheduling operation rules to generate a column mapping list; and then a scheduling script for executing the scheduling operation corresponding to the business demand is generated according to the table mapping list and the column mapping list. According to the embodiment of the present application, the problem of the complex process of cross-platform data scheduling (data extraction, conversion and loading) in the related technology can be solved, and then the process of cross-platform (source system and target system) data scheduling can be simplified by generating a scheduling script.

[0160] In an exemplary embodiment, the determination module 82 is also used to parse the business requirements, and determine the transmission path, the processing logic and the scheduling parameters through the parsed business requirements and the target algorithm; and generate the scheduling operation rules based on at least one of the transmission path, the scheduling parameters and the processing logic.

[0161] In an exemplary embodiment, the determination module 82 is also used to determine the transmission path corresponding to the scheduling job based on the parsed business needs; determine the resource allocation strategy for allocating resources required to execute the scheduling job based on the target algorithm and the business needs, and determine the scheduling configuration information corresponding to the scheduling job based on the target algorithm and the business needs, wherein the scheduling configuration information includes at least one of the following: scheduling frequency, scheduling time window and whether the scheduling job is executed periodically; determine the scheduling parameters based on the resource allocation strategy and the scheduling configuration information; determine the data flow corresponding to the scheduling job and the data conversion rules for converting the target data from the source database to the target database based on the target algorithm and the business needs, and determine the processing logic based on the data conversion rules.

[0162] In an exemplary embodiment, the configuration module 84 is also used to determine the source table in the source system and determine the target table in the target system according to the scheduling job rules, wherein the source table includes at least: data stored in the source database, and the target table includes at least: data stored in the target database; determine the first mapping information between the source table and the target table through a low-code tool, wherein the first mapping information includes at least one of the following: a first identifier of the source table, a second identifier of the target table, source system parameters of the source system, and target system parameters of the target system; configure the first mapping relationship between the source table and the target table according to the first mapping information to generate the table mapping list.

[0163] In an exemplary embodiment, the configuration module 84 is also used to determine the first field list in the source table to be configured to the target table according to the scheduling job rules, wherein the source field information is used to indicate the field information contained in the first field list; assign a second field list to the first field list in the target table through a low-code tool, wherein the target field information is used to indicate the field information contained in the second field list; determine the conversion rules for converting the fields in the first field list into the fields in the second field list, and determine the second mapping information between the source field information and the target field information according to the conversion rules; configure the second mapping relationship between the source table and the target table according to the second mapping information to generate the column mapping list.

[0164] In an exemplary embodiment, the generation module 86 is further used to generate an interface calling code for extracting the target data from the source table and a first code for scheduling the target data to the target table based on the table mapping list; generate a second code for mapping the first field list in the source table to the second field list in the target table based on the column mapping list; and generate the scheduling script based on the first code, the second code and the interface calling code.

[0165] In an exemplary embodiment, the generation module 86 is also used to generate execution configuration information for executing the scheduled job based on the scheduling script, wherein the execution configuration information includes at least one of the following: the execution time and execution frequency of the scheduled job; configuring execution nodes for the scheduled job and determining the dependency relationship between each execution node; generating a scheduling job rule for the scheduled job based on the execution configuration information, the execution nodes and the dependency relationship, and determining the triggering condition of the scheduling job rule; and executing the scheduled job based on the scheduling job rule when it is determined that the triggering condition is triggered.

[0166] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0167] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above method embodiments when run.

[0168] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0169] S1. Determine a target algorithm corresponding to a business requirement in a preset algorithm library, and generate a scheduling operation rule based on the target algorithm and the business requirement, wherein the business requirement is used to instruct to schedule target data in a source database of a source system to a target database of a target system, and the scheduling operation rule includes at least one of the following: a transmission path of the scheduling operation corresponding to the business requirement, processing logic corresponding to the scheduling operation, and scheduling parameters corresponding to the scheduling operation;

[0170] S2: configuring a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list; and configuring a second mapping relationship between source field information in the source table and target field information in the target table according to the scheduling job rule to generate a column mapping list, wherein the source field information is used to indicate each row of data in the source table, and the target field information is used to indicate each row of data in the target table;

[0171] S3: Generate a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list.

[0172] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0173] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0174] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0175] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0176] S1. Determine a target algorithm corresponding to a business requirement in a preset algorithm library, and generate a scheduling operation rule based on the target algorithm and the business requirement, wherein the business requirement is used to instruct to schedule target data in a source database of a source system to a target database of a target system, and the scheduling operation rule includes at least one of the following: a transmission path of the scheduling operation corresponding to the business requirement, processing logic corresponding to the scheduling operation, and scheduling parameters corresponding to the scheduling operation;

[0177] S2: configuring a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list; and configuring a second mapping relationship between source field information in the source table and target field information in the target table according to the scheduling job rule to generate a column mapping list, wherein the source field information is used to indicate each row of data in the source table, and the target field information is used to indicate each row of data in the target table;

[0178] S3: Generate a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list.

[0179] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0180] An embodiment of the present application further provides another computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above method embodiments are implemented.

[0181] An embodiment of the present application also provides a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above method embodiments.

[0182] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0183] S1. Determine a target algorithm corresponding to a business requirement in a preset algorithm library, and generate a scheduling operation rule based on the target algorithm and the business requirement, wherein the business requirement is used to instruct to schedule target data in a source database of a source system to a target database of a target system, and the scheduling operation rule includes at least one of the following: a transmission path of the scheduling operation corresponding to the business requirement, processing logic corresponding to the scheduling operation, and scheduling parameters corresponding to the scheduling operation;

[0184] S2: configuring a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list; and configuring a second mapping relationship between source field information in the source table and target field information in the target table according to the scheduling job rule to generate a column mapping list, wherein the source field information is used to indicate each row of data in the source table, and the target field information is used to indicate each row of data in the target table;

[0185] S3: Generate a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list.

[0186] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0187] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0188] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for generating a scheduling script, characterized in that: include: Determining a target algorithm corresponding to the business requirement in a preset algorithm library, and generating a scheduling operation rule based on the target algorithm and the business requirement, wherein the business requirement is used to instruct to schedule target data in a source database of a source system to a target database of a target system, and the scheduling operation rule includes at least one of the following: a transmission path of the scheduling operation corresponding to the business requirement, processing logic corresponding to the scheduling operation, and scheduling parameters corresponding to the scheduling operation; configuring a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list, and configuring a second mapping relationship between source field information in the source table and target field information in the target table according to the scheduling job rule to generate a column mapping list, wherein the source field information is used to indicate each row of data in the source table, and the target field information is used to indicate each row of data in the target table; A scheduling script for executing the scheduled job is generated according to the table mapping list and the column mapping list.

2. The method for generating a scheduling script according to claim 1, wherein: Generating scheduling operation rules according to the target algorithm and the business requirements includes: Parsing the business requirements, and determining the transmission path, the processing logic, and the scheduling parameters based on the parsed business requirements and the target algorithm; The scheduling operation rule is generated according to at least one of the transmission path, the scheduling parameters, and the processing logic.

3. The method for generating a scheduling script according to claim 2, wherein: Determining the transmission path, the processing logic, and the scheduling parameters based on the parsed business requirements and the target algorithm includes: Determine the transmission path corresponding to the scheduled job according to the analyzed business requirements; Determining a resource allocation strategy for allocating resources required to execute the scheduled job based on the target algorithm and the business requirements, and determining scheduling configuration information corresponding to the scheduled job based on the target algorithm and the business requirements, wherein the scheduling configuration information includes at least one of the following: a scheduling frequency, a scheduling time window, and whether to execute the scheduled job periodically; Determining the scheduling parameters according to the resource allocation strategy and the scheduling configuration information; The data flow corresponding to the scheduling job and the data conversion rules for converting the target data from the source database to the target database are determined according to the target algorithm and the business requirements, and the processing logic is determined according to the data conversion rules.

4. The method for generating a scheduling script according to claim 1, wherein: Configuring a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list includes: Determining the source table in the source system and determining the target table in the target system according to the scheduling job rule, wherein the source table includes at least data stored in the source database, and the target table includes at least data stored in the target database; Determining, by a low-code tool, first mapping information between the source table and the target table, wherein the first mapping information includes at least one of the following: a first identifier of the source table, a second identifier of the target table, a source system parameter of the source system, and a target system parameter of the target system; A first mapping relationship between the source table and the target table is configured according to the first mapping information to generate the table mapping list.

5. The method for generating a scheduling script according to claim 1, wherein: Configuring a second mapping relationship between the source field information in the source table and the target field information in the target table according to the scheduling job rule to generate a column mapping list includes: Determine, according to the scheduling job rule, a first field list in the source table to be configured in the target table, wherein the source field information is used to indicate field information included in the first field list; Allocating a second field list to the first field list in the target table using a low-code tool, wherein the target field information is used to indicate field information included in the second field list; Determining a conversion rule for converting fields in the first field list into fields in the second field list, and determining second mapping information between the source field information and the target field information according to the conversion rule; A second mapping relationship between the source table and the target table is configured according to the second mapping information to generate the column mapping list.

6. The method for generating a scheduling script according to claim 1, wherein: Generating a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list includes: Generate, according to the table mapping list, an interface calling code for extracting the target data from the source table and a first code for dispatching the target data to the target table; generating, according to the column mapping list, a second code for mapping a first field list in the source table to a second field list in the target table; The scheduling script is generated according to the first code, the second code and the interface calling code.

7. The method for generating a scheduling script according to claim 1, wherein: After generating a scheduling script for executing the scheduled job according to the table mapping list and the column mapping list, the method further includes: Generate execution configuration information for executing the scheduled job according to the scheduling script, wherein the execution configuration information includes at least one of the following: the execution time and execution frequency of the scheduled job; Configuring execution nodes for the scheduled job and determining the dependency relationship between each execution node; Generating a scheduling operation rule for the scheduling operation according to the execution configuration information, the execution node and the dependency relationship, and determining a triggering condition for the scheduling operation rule; When it is determined that the trigger condition is triggered, the scheduled job is executed based on the scheduled job rule.

8. A device for generating a scheduling script, characterized in that: include: a determination module, configured to determine a target algorithm corresponding to a business requirement in a preset algorithm library, and generate a scheduling operation rule based on the target algorithm and the business requirement, wherein the business requirement is used to instruct the scheduling of target data in a source database of a source system to a target database of a target system, and the scheduling operation rule includes at least one of the following: a transmission path of the scheduling operation corresponding to the business requirement, processing logic corresponding to the scheduling operation, and scheduling parameters corresponding to the scheduling operation; a configuration module, configured to configure a first mapping relationship between a source table corresponding to the source system and a target table corresponding to the target system according to the scheduling job rule to generate a table mapping list, and to configure a second mapping relationship between source field information in the source table and target field information in the target table according to the scheduling job rule to generate a column mapping list, wherein the source field information is used to indicate each row of data in the source table, and the target field information is used to indicate each row of data in the target table; A generating module is used to generate a scheduling script for executing the scheduling job according to the table mapping list and the column mapping list.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the method according to any one of claims 1 to 7 is executed when the program is executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images